Overview
Introduction
A single misplaced rule in robots.txt can accidentally block search engines from an entire site section, or fail to block the admin area it was meant to protect - and because the file is just plain text with no validation, these mistakes ship silently until someone notices traffic dropping or sensitive paths turning up crawlable.
This tool parses a robots.txt file the way a crawler would: grouping User-agent blocks, reading their Allow/Disallow rules, and resolving which rule actually applies to a specific path, so you can check a rule's effect before it goes live rather than after.
What Is Robots.txt Tester?
A robots.txt parser and path tester that reads a fetched or pasted robots.txt file, breaks it into its User-agent groups, and lets you test whether a given path is allowed or disallowed for a chosen crawler.
It implements the same longest-match precedence rule documented by Google and formalized in RFC 9309, including wildcard (*) and end-anchor ($) pattern support, so the result matches what a real crawler would decide.
How Robots.txt Tester Works
The file is split into groups by consecutive User-agent lines, each followed by its Allow/Disallow/Sitemap directives. When you test a path, the tool first finds the group matching your chosen user-agent exactly, falling back to the wildcard (*) group if there's no exact match, per the standard.
Every rule in the matched group is converted into a pattern (respecting * wildcards and a trailing $ end-anchor) and tested against your path. Among all matching rules, the longest one wins; ties go to Allow. The result explains which rule won and why, not just a pass/fail badge.
When To Use Robots.txt Tester
Before deploying a new or edited robots.txt, to confirm a Disallow rule actually blocks the paths you intend without accidentally catching more than expected (a common mistake with overly broad wildcards).
When debugging why a page isn't being indexed, to rule out (or confirm) that robots.txt is blocking the crawler from fetching it in the first place.
Often used alongside Robots.txt Generator and Meta Tag Checker.
Features
Advantages
- Implements the actual longest-match conflict resolution rule crawlers use, not a simplified first-match approximation that could give a wrong answer.
- Works on any site via the paste fallback, not just domains that happen to send permissive CORS headers.
- Shows every rule considered and why one won, so you understand the outcome instead of just trusting a verdict.
Limitations
- This tests robots.txt syntax and precedence only; it can't tell you whether a crawler actually respects robots.txt at all (well-behaved crawlers do, but nothing technically stops a bad actor from ignoring it).
- A live fetch only works when the target domain sends permissive CORS headers, which most ordinary websites don't; use the paste fallback for anything else.
Examples
Best Practices & Notes
Best Practices
- Test every new or edited Disallow rule against both the paths you intend to block and a few you don't, since wildcard patterns are easy to write more broadly than intended.
- Keep a wildcard (User-agent: *) group in every robots.txt as a fallback, since a crawler with no matching group defaults to allowing everything.
- Remember robots.txt is publicly readable - don't rely on Disallow to hide sensitive paths from humans, only to guide well-behaved crawlers.
Developer Notes
Precedence uses raw pattern character length (excluding a trailing $) as the specificity score, matching Google's documented algorithm; rule order in the file has no bearing on the outcome once parsed into groups.
Robots.txt Tester Use Cases
- Verifying a Disallow rule blocks the intended admin or staging paths before deploying it to production
- Debugging why a specific page isn't getting crawled by checking it against the live robots.txt
- Auditing a site migration's robots.txt for accidental overly broad Disallow rules inherited from a staging environment
Common Mistakes
- Writing "Disallow: /" in a staging environment's robots.txt and then deploying that same file to production unchanged, blocking the entire site from search engines.
- Assuming Disallow prevents indexing entirely, when it only prevents crawling; an already-linked disallowed URL can still appear in search results without a description.
Tips
- Test with an empty path ("/") first to confirm the site isn't accidentally blocked entirely, then test specific paths you care about.
- If a path you expect to be blocked shows as allowed, check for a more specific Allow rule elsewhere in the same group; it likely wins on length.