Free · no signup · 30 bots

Can ChatGPT, Claude and Google actually read your site?

We read your robots.txt, look for llms.txt and indexing signals, and request your page as each AI crawler, search engine and SEO bot.

Try a live report:stripe.com github.com bbc.com theverge.com

acme-store.comexample report

78/ 100

Search engines can read your site. 2 bots can’t.27 allowed 1 token 2 blocked

GooglebotAllowed · 200
OAI-SearchBotAllowed · 200
ClaudeBotAllowed · 200
GPTBotBlocked · robots.txt
Google-ExtendedAllowed · token
BytespiderBlocked · 403

robots.txt foundllms.txt missingsignals none

01 · robots.txtWhich bots your rules allowrobots.txt rules
02 · live requestThe HTTP status each bot getsLive request per bot
03 · signalsllms.txt, noindex, noai and TDMllms.txt and signals

How the check works

robots.txt rules

We fetch /robots.txt and apply it the way crawlers do (RFC 9309): the group naming the bot wins, then User-agent: *, and the longest matching path rule decides.

A live request per bot

For every bot the rules allow, we request your page with that bot’s user agent and record the status, time and size. This is how you catch firewall and CDN blocks.

llms.txt and signals

We look for /llms.txt, a short Markdown file that points AI tools to your most useful pages, and read the page’s indexing signals: noindex, noai, TDM-Reservation, Content-Signal and your sitemap.

Adding an llms.txt

Put a plain Markdown file at https://your-site.com/llms.txt. Start with a heading for your site and a one-line summary, then list the pages you’d want an AI assistant to read first. Serve it as text/plain or text/markdown, not as an HTML page. The llms.txt generator writes one for you and checks an existing file. To decide which bots to allow or block, use the robots.txt generator.

# Acme Store
> Handmade kitchenware, shipped across the EU.

## Docs
- [Shipping & returns](https://acme-store.com/shipping): costs, times, how to return
- [Size guide](https://acme-store.com/sizes): measurements for every product

## Optional
- [Company history](https://acme-store.com/about)

Questions

Why check robots.txt and a live request?

robots.txt is what you ask bots to do. The live request shows what your server actually does. Firewalls, CDNs and bot-protection services often refuse crawlers that robots.txt allows, and nothing in robots.txt shows it.

How is the score worked out?

Pick a goal first: open to everything, search and AI answers without training, or search engines only. Only the bots in that goal count. Each group of bots has a weight (search engines 40, AI search 25, user-triggered fetchers 10, AI training 10, SEO tools 5), the groups in your goal share 90 points in proportion to their weights, and each group earns its share by the fraction of its bots that can read the page. Having an llms.txt adds the last 10. Bots outside your goal are still checked and shown, but blocking or allowing them doesn’t move the score.

Why is there no score for some sites?

When a site refuses even our plain browser request, we can only read robots.txt. A live 403 then tells you nothing about the bot, so those bots are marked unverified and no score is shown, rather than a number that looks earned. Your firewall or CDN logs show what each bot really gets.

Why are some bots marked “token”?

Google-Extended and Applebot-Extended aren’t crawlers. They are names in robots.txt that tell Google and Apple whether they may use pages their normal crawlers fetched for AI training. There is nothing to request, so only the rules are checked.

Can a 403 here be wrong?

Yes. We send each request from our server with the bot’s user agent, not from the bot operator’s IP addresses. Some firewalls let a verified bot in but refuse the same user agent from anywhere else. If a bot shows as refused, check your firewall or CDN logs for that bot.

What are the “signals”?

Things beyond robots.txt that change what bots do with a page: a noindex or nofollow directive in the X-Robots-Tag header or a robots meta tag, the noai and noimageai directives, a TDM-Reservation header that reserves text-and-data-mining rights, Content-Signal lines in robots.txt, and whether a sitemap is declared. A page can be crawlable and still be kept out of search results by a noindex.

Are results cached?

Yes, for ten minutes, so a shared link opens instantly. After you change robots.txt or a firewall rule, press “Re-check” to run a fresh scan.

Guides

BlockingHow to block AI crawlersA practical guide to blocking GPTBot, ClaudeBot, CCBot, Bytespider and other AI crawlers with robots.txt and firewall rules, which bots to keep, and how to verify the block works.9 min read robots.txtrobots.txt for AI crawlersHow robots.txt really works under RFC 9309: groups, precedence, wildcards, status codes, the AI bot tokens for OpenAI, Anthropic, Google, Apple and others, and the mistakes that silently break it.11 min read llms.txtWhat is llms.txtWhat the llms.txt proposal is, what the format looks like, which AI systems actually read it, and an honest view of whether adding one changes anything.7 min read llms.txtllms.txt examplesReal llms.txt files from Stripe, Cloudflare, Vercel, Sephora, Etsy and others, what each does well, the mistakes to avoid, and six templates you can copy.10 min read Indexingnoindex vs robots.txtThe difference between blocking crawling and blocking indexing, when to use robots.txt, noindex and X-Robots-Tag, and what the noai, TDM-Reservation and Content-Signal directives do.8 min read PolicyContent-Signal explainedWhat the Content-Signal lines in robots.txt mean, where they come from, how search=yes, ai-input=no and ai-train=no are interpreted, and what they do and do not enforce.6 min read CDNCloudflare blocking GooglebotHow Bot Fight Mode, managed challenges, WAF rules and the AI-bot toggle end up refusing Googlebot and other crawlers, how to see it in Cloudflare and Search Console, and how to fix it without turning protection off.9 min read

Monitoring early access

A robots.txt deploy, a new CDN rule or a bot-protection update can lock crawlers out without anyone noticing. Monitoring will re-run this check every week and email you when a result changes.

It isn’t open yet. Leave your email and the site, and we’ll write to you when it is. Nothing is charged and nothing is set up until then.