canaicrawl Get early access
6 guides · reviewed 30 September 2026

Guides

How crawler access really works: robots.txt under RFC 9309, what each AI operator’s bot does, the directives that keep pages out of search or out of training, and how CDNs end up blocking the bots you need. Written to be checked against the operators’ own documentation, and dated so you can tell how fresh they are.

Blocking How to block AI crawlers: robots.txt, firewall rules and what each one costs you A practical guide to blocking GPTBot, ClaudeBot, CCBot, Bytespider and other AI crawlers with robots.txt and firewall rules, which bots to keep, and how to verify the block works. 9 min read · reviewed 30 September 2026 robots.txt robots.txt for AI crawlers: the complete guide How robots.txt really works under RFC 9309: groups, precedence, wildcards, status codes, the AI bot tokens for OpenAI, Anthropic, Google, Apple and others, and the mistakes that silently break it. 11 min read · reviewed 30 September 2026 llms.txt What is llms.txt, and does it matter? What the llms.txt proposal is, what the format looks like, which AI systems actually read it, and an honest view of whether adding one changes anything. 7 min read · reviewed 30 September 2026 Indexing noindex vs robots.txt: crawling, indexing and the AI signals in between The difference between blocking crawling and blocking indexing, when to use robots.txt, noindex and X-Robots-Tag, and what the noai, TDM-Reservation and Content-Signal directives do. 8 min read · reviewed 30 September 2026 Policy Content-Signal explained: Cloudflare’s robots.txt policy for search, AI input and AI training What the Content-Signal lines in robots.txt mean, where they come from, how search=yes, ai-input=no and ai-train=no are interpreted, and what they do and do not enforce. 6 min read · reviewed 30 September 2026 CDN Why Cloudflare blocks Googlebot, and how to check whether it is happening to you How Bot Fight Mode, managed challenges, WAF rules and the AI-bot toggle end up refusing Googlebot and other crawlers, how to see it in Cloudflare and Search Console, and how to fix it without turning protection off. 9 min read · reviewed 30 September 2026