canaicrawl Get early access
RFC 9309 · 30 bots · runs in your browser

robots.txt tester

Paste a robots.txt or load one from any site, enter a URL path, and see which line decides for each of 30 crawlers: Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and the rest. Every line is also checked for the mistakes crawlers trip over.

Or paste a file below. Testing happens in your browser; loading a site’s file is one request to it, and nothing is stored.

Result for /

Your file, line by line

How crawlers read robots.txt

  1. Pick a group. A crawler looks for the group whose User-agent line names it, matching the name case-insensitively and ignoring any version (Googlebot/2.1 counts as Googlebot). If no group names it, it uses User-agent: *. If there’s neither, everything is allowed. Groups that name the same crawler are combined. Applebot falls back to the Googlebot group, and Bingbot to an old msnbot group, before *.
  2. Find the matching rules. Each Allow and Disallow path is compared with the start of the URL path, including the query string. * matches anything and a final $ means the URL must end there.
  3. The most specific rule wins. The longest matching path decides; on a tie, Allow wins. With no matching rule, the URL is allowed. /robots.txt itself is always allowed.

A crawler that gets a 4xx answer for robots.txt treats the whole site as allowed; a 5xx answer or a failed request makes it stay away. robots.txt controls crawling, not indexing: a blocked page can still appear in search results when other pages link to it. To build a file from scratch, use the robots.txt generator; to see what each bot actually gets from your server, run the full check.

Questions

How do I test my robots.txt?

Load your site’s robots.txt above, or paste the file, then enter the path you care about, such as /blog/my-post. The tester applies the rules the way RFC 9309 and the big crawlers do and shows, for each of the bots it lists, whether that path is allowed and which line decided. Every line is also checked for mistakes.

Is there still a robots.txt tester in Google Search Console?

No. Google retired the robots.txt Tester in Search Console in late 2023 and replaced it with the robots.txt report, which shows the robots.txt files Google fetched for your site and any errors in them. To see whether Google may crawl one URL, use URL Inspection in Search Console, or test the path here.

Which rule wins when Allow and Disallow both match?

The rule with the longest path wins, because it is the most specific. When an Allow and a Disallow rule are equally long, the Allow rule wins. The order of the lines in the file doesn’t matter.

Does User-agent: * apply to Googlebot when there is a Googlebot group?

No. A crawler follows only the group that names it most specifically. Once a group names Googlebot, Googlebot ignores the User-agent: * group entirely, so repeat any general rules you still want it to obey.

Do * and $ work in robots.txt?

Yes, in paths. * matches any sequence of characters and $ at the end of a rule anchors it to the end of the URL, so Disallow: /*.pdf$ blocks URLs ending in .pdf. They don’t work in user-agent names.

Is robots.txt case-sensitive?

Paths are: Disallow: /Private doesn’t block /private. Directive names (Disallow, disallow) and user-agent names (Googlebot, googlebot) are not.