Bot directory · AI training

AI training · Anthropic

ClaudeBot

ClaudeBot is the crawler Anthropic uses to collect web content that could help train its Claude models. Blocking it in robots.txt is how Anthropic asks site owners to opt out of training, and it doesn’t stop Claude’s search or the fetches Claude makes for its users, which use their own tokens.

Checked against Anthropic’s documentation on 19 October 2026

Operator
Anthropic
Purpose
Model training. Crawlers that collect pages to train models. Blocking them does not affect search or AI answers.
robots.txt token
ClaudeBot
We test with
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
a string with the ClaudeBot token, as commonly seen in server logs. Anthropic doesn’t publish the full string; robots.txt and firewall rules only need the token.
Documentation
support.claude.com/en/articles/8896518

What ClaudeBot does

ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.

support.claude.com

When a site restricts ClaudeBot access, it signals that the site’s future materials should be excluded from our AI model training datasets.

support.claude.com

Anthropic runs three bots, each with its own token:

BotWhat it doesModel training
ClaudeBotCollects web content that could contribute to training Anthropic’s modelsYes
Claude-SearchBotIndexes pages to improve Claude’s search resultsNot described as training
Claude-UserOpens a page when a Claude user asks a question that needs itNot described as training

The ClaudeBot user agent

Anthropic names its bots’ robots.txt tokens but doesn’t publish their full user agent strings. Match on the token, which appears in every request: ClaudeBot, case-insensitively. Strings quoted elsewhere are observations, not Anthropic’s documentation.

The token is unique among Anthropic’s bots: a rule matching claudebot doesn’t catch the other two.

How ClaudeBot reads robots.txt

Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.

support.claude.com

ClaudeBot IP addresses and how to verify them

Anthropic publishes one list of addresses for all three of its bots, so a match tells you a request came from Anthropic, not which bot sent it; the user agent says that. Anthropic documents no reverse DNS names.

ListCoversIPv4IPv6Dated
claude.com/crawling/bots.jsonClaudeBot, Claude-SearchBot and Claude-User3807 October 2026

Anthropic recommends robots.txt over IP blocking for opting out of ClaudeBot: “Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.” If you block its addresses, keep /robots.txt open to them.

Check an IP now

Paste an address from your logs. We match it against Anthropic’s current list, along with the lists of the other operators that publish them.

Block ClaudeBot at your firewall or CDN

robots.txt asks; a firewall enforces. A rule on the user agent refuses ClaudeBot even if it doesn’t honour robots.txt. Match the name case-insensitively and anywhere in the string, never a full string or a version.

Cloudflare

In Security → WAF → Custom rules, create a rule with this expression and the action Block. Cloudflare’s contains is case-sensitive, hence lower().

(lower(http.user_agent) contains "claudebot")

Cloudflare also has one-click controls: AI bot policies, which block AI crawlers by what they do (search, agent or training), and AI Crawl Control, which lets you allow or block crawlers one by one and shows which of them fetch your pages.

nginx

# in the http block
map $http_user_agent $blocked_bot {
    default 0;
    ~*claudebot 1;
}

# in the server block
if ($blocked_bot) { return 403; }

Anthropic prefers robots.txt for opting out: blocking its requests outright also stops ClaudeBot reading your robots.txt, so it can’t see that you opted out.

Block ClaudeBot

Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt. This keeps your future pages out of Anthropic’s training data. It doesn’t affect Claude-SearchBot or Claude-User.

User-agent: ClaudeBot
Disallow: /

Allow ClaudeBot when everything else is blocked

A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.

User-agent: ClaudeBot
Allow: /

User-agent: *
Disallow: /

Does ClaudeBot reach your site?

The checker applies your robots.txt the way ClaudeBot does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on ClaudeBot.

Questions about ClaudeBot

What is ClaudeBot?

ClaudeBot is Anthropic’s web crawler. Anthropic says it collects web content that could potentially contribute to training its generative AI models, and that restricting it signals your future content should be excluded from its training datasets.

Does ClaudeBot respect robots.txt?

Yes. Anthropic says its bots honour robots.txt and support the non-standard Crawl-delay directive. Add the rules to the robots.txt of every subdomain you want to opt out.

Does blocking ClaudeBot stop Claude reading my site?

No. Claude’s search uses Claude-SearchBot, and pages Claude opens for its users come from Claude-User. Each has its own robots.txt token.

What IP addresses does ClaudeBot use?

Anthropic publishes one list for all its bots at claude.com/crawling/bots.json: 38 IPv4 ranges and no IPv6 in October 2026. A match shows a request came from Anthropic; the user agent says which bot.

Should I block ClaudeBot by IP?

Anthropic advises against relying on it: blocking its addresses can stop it reading your robots.txt, so the opt-out may not work. Use robots.txt, and keep /robots.txt reachable if you add firewall rules.

Also run by Anthropic: Claude-SearchBot, Claude-User.

Other ai training: GPTBot, Google-Extended, CCBot, Applebot-Extended, GoogleOther, Meta-ExternalAgent, Bytespider.

Build a complete file with the robots.txt generator.