30 bots · 5 groups

Bot directory

Every bot canaicrawl checks, who runs it, and the exact user agent we send. Each name links to a page on what the bot does and how to block or allow it. To build a full robots.txt, use the generator. To allow one bot your User-agent: * rules block, give it its own group:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Allow: /

This list is also published as JSON at /bots.json (CORS enabled, cached for an hour) with every bot’s token, user agent, group and documentation link, for scripts, firewalls and other tools. Please link back if you use it.

A visitor claiming to be Googlebot or Bingbot? Anyone can send those user agents; verify the IP against Google’s and Microsoft’s published lists. Full reference pages: Googlebot user agent, Bingbot.

Seeing canaicrawl in your logs? Someone ran this checker on your site: one visit as canaicrawl/1.0 (and one as a browser if that was refused), one as each bot above, plus robots.txt, llms.txt and sitemap.xml. A weekly request for only robots.txt and llms.txt is our check of the web’s most-linked sites for Who blocks AI. Questions: info@canaicrawl.com.

AI search · AI answers

Crawlers that build the indexes AI assistants answer from and cite. Blocking one keeps you out of its answers.

BotOperatorUser agent we test withDocs
OAI-SearchBot OpenAI Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot Docs
PerplexityBot Perplexity Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) Docs
Claude-SearchBot Anthropic Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com) Docs
DuckAssistBot DuckDuckGo DuckAssistBot/1.0; (+http://duckduckgo.com/duckassistbot.html) Docs
Amazonbot Amazon Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Docs
YouBot You.com Mozilla/5.0 (compatible; YouBot (+http://www.you.com)) Docs

AI training · Model training

Crawlers that collect pages to train models. Blocking them does not affect search or AI answers.

BotOperatorUser agent we test withDocs
GPTBot OpenAI Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot Docs
ClaudeBot Anthropic Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) Docs
Google-Extended Google robots.txt token only, never sent as a user agent Docs
CCBot Common Crawl CCBot/2.0 (https://commoncrawl.org/faq/) Docs
Applebot-Extended Apple robots.txt token only, never sent as a user agent Docs
GoogleOther Google GoogleOther Docs
Meta-ExternalAgent Meta meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) Docs
Bytespider ByteDance Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com) none

User-triggered fetchers · Live user request

Fetchers that load a page because a person asked an assistant about it. Blocking one means the assistant can’t open your pages on request.

BotOperatorUser agent we test withDocs
ChatGPT-User OpenAI Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot Docs
Claude-User Anthropic Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Docs
Perplexity-User Perplexity Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Docs
MistralAI-User Mistral Mozilla/5.0 (compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots) Docs
Meta-ExternalFetcher Meta meta-externalfetcher/1.1 Docs

SEO tools · Backlink & SEO index

Crawlers behind backlink and SEO tools. Many sites block them to save bandwidth; it has no effect on search.

BotOperatorUser agent we test withDocs
AhrefsBot Ahrefs Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) Docs
SemrushBot Semrush Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html) Docs
MJ12bot Majestic Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) Docs
DotBot Moz Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com) Docs