robots.txt Tester

How to use

1. Paste your robots.txt into the upper box.
2. Add (user-agent, path) test cases — set UA to '*' to test the catch-all group.
3. Read the verdict and the rule that produced it on the right.

Frequently asked questions

How does longest-match work?

Within the matching UA group, the rule with the longest pattern that still matches the path wins. Allow and Disallow are weighed by pattern length only — an Allow with a longer pattern beats a Disallow with a shorter one.

What does $ at the end of a pattern mean?

It anchors the pattern to the end of the URL path. /foo$ matches /foo exactly, not /foo/bar. Useful for blocking specific file extensions like /*.pdf$.

Does a Disallow rule keep a page out of search results?

Not reliably. It stops well-behaved crawlers from fetching the page, but a URL that other sites link to can still be listed, because the crawler never reads the page and so never sees a noindex tag on it. To keep something out of an index, allow it to be crawled and return noindex, or put it behind authentication.

Does one robots.txt cover my subdomains?

No. The file applies to exactly one scheme, host and port, so example.com and www.example.com each need their own, and so does a different port. It also has to sit at the root of that host — a robots.txt inside a subdirectory is simply not read.

Related tools