robots.txt Tester

Implements the matching rules used by Googlebot / Bingbot: groups are keyed by user-agent, the most-specific UA wins (exact > substring > *), and within a group the longest matching pattern wins (Allow ties beat Disallow). Supports the * wildcard and $ end-of-path anchor. Surfaces every Sitemap: declaration too, so you can sanity-check that your sitemap URLs are reachable.

The longest-match rule is where most surprises live, because it ignores the order the lines are written in. A Disallow: /admin/ followed by Allow: /admin/public/ leaves the deeper path crawlable no matter which line comes first, and moving the lines around changes nothing. Ties are resolved toward the least restrictive rule, so a path matched by an Allow and a Disallow of identical length is crawlable. Everything after a # is discarded before matching, which means a rule commented out mid-line stops applying silently.

A robots file is scoped to one scheme, host and port. The file served at https://example.com/robots.txt says nothing about http://example.com, about www.example.com, or about a subdomain, each of which needs its own. Paths are matched case-sensitively even where the server itself is not, so /Admin/ and /admin/ are separate rules.

The most consequential limit is what the file governs at all. Disallow stops a crawler from fetching a URL; it does not remove that URL from search results. A blocked page that other sites link to can still be indexed, listed with no description because the crawler was never allowed to read it. Keeping a page out of the index needs a noindex response header or meta tag, and that in turn requires the page to stay crawlable, so blocking it in robots.txt actively prevents the removal you wanted.