Bot directory · User-triggered fetchers
Claude-User
Claude-User is the user agent Claude uses when someone asks it a question and it needs to read a web page to answer. Unlike the user-triggered fetchers of OpenAI and Perplexity, Anthropic doesn’t exempt it from robots.txt, so a disallow in robots.txt stops it.
- Operator
- Anthropic
- Purpose
- Live user request. Fetchers that load a page because a person asked an assistant about it. Blocking one means the assistant can’t open your pages on request.
- robots.txt token
Claude-User- We test with
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)
a string with theClaude-Usertoken. Anthropic doesn’t publish the full string.- Documentation
- support.claude.com/en/articles/8896518
What Claude-User does
Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.
support.claude.com
Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site’s visibility for user-directed web search.
support.claude.com
| Bot | What it does | Model training |
|---|---|---|
ClaudeBot | Collects web content that could contribute to training Anthropic’s models | Yes |
Claude-SearchBot | Indexes pages to improve Claude’s search results | Not described as training |
Claude-User | Opens a page when a Claude user asks a question that needs it | Not described as training |
The Claude-User user agent
Anthropic names its bots’ robots.txt tokens but doesn’t publish their full user agent strings. Match on the token, which appears in every request: Claude-User, case-insensitively. Strings quoted elsewhere are observations, not Anthropic’s documentation.
The token is unique among Anthropic’s bots: a rule matching claude-user doesn’t catch the other two.
How Claude-User reads robots.txt
- All three bots honour robots.txt. Anthropic doesn’t exempt Claude-User, unlike OpenAI and Perplexity for their user-triggered fetchers.
- Crawl-delay works. Anthropic supports it to slow crawling, though it isn’t part of the robots.txt standard.
- Each subdomain has its own robots.txt. Anthropic asks you to add the rules to every subdomain you want to opt out.
- No CAPTCHA solving. Anthropic says its bots respect anti-circumvention technology and won’t try to get past CAPTCHAs.
Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.
support.claude.com
Claude-User IP addresses and how to verify them
Anthropic publishes one list of addresses for all three of its bots, so a match tells you a request came from Anthropic, not which bot sent it; the user agent says that. Anthropic documents no reverse DNS names.
| List | Covers | IPv4 | IPv6 | Dated |
|---|---|---|---|---|
| claude.com/crawling/bots.json | ClaudeBot, Claude-SearchBot and Claude-User | 38 | 0 | 7 October 2026 |
Anthropic recommends robots.txt over IP blocking for opting out of Claude-User: “Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.” If you block its addresses, keep /robots.txt open to them.
Check an IP now
Paste an address from your logs. We match it against Anthropic’s current list, along with the lists of the other operators that publish them.
Block Claude-User at your firewall or CDN
robots.txt asks; a firewall enforces. Blocking Claude-User means Claude can’t read your pages when its users ask about them. Match the name case-insensitively and anywhere in the string, never a full string or a version.
Cloudflare
In Security → WAF → Custom rules, create a rule with this expression and the action Block. Cloudflare’s contains is case-sensitive, hence lower().
(lower(http.user_agent) contains "claude-user")
Cloudflare also has one-click controls: AI bot policies, which block AI crawlers by what they do (search, agent or training), and AI Crawl Control, which lets you allow or block crawlers one by one and shows which of them fetch your pages.
nginx
# in the http block
map $http_user_agent $blocked_bot {
default 0;
~*claude-user 1;
}
# in the server block
if ($blocked_bot) { return 403; }
Block Claude-User
Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt. Anthropic honours this for Claude-User too: Claude then can’t open your pages when someone asks about them.
User-agent: Claude-User
Disallow: /
Allow Claude-User when everything else is blocked
A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.
User-agent: Claude-User
Allow: /
User-agent: *
Disallow: /
Does Claude-User reach your site?
The checker applies your robots.txt the way Claude-User does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on Claude-User.
Questions about Claude-User
What is Claude-User?
Claude-User is the user agent Claude uses to open a web page when a user’s question needs it. It doesn’t crawl on its own.
Does Claude-User respect robots.txt?
Yes. Anthropic says its bots honour robots.txt and doesn’t make an exception for Claude-User, unlike OpenAI’s ChatGPT-User and Perplexity’s Perplexity-User, which may not follow it.
What happens if I block Claude-User?
Anthropic says it prevents Claude retrieving your content in response to a user’s query, which may reduce your site’s visibility for user-directed web search.
What IP addresses does Claude-User use?
The addresses in Anthropic’s list at claude.com/crawling/bots.json, which it publishes for all three of its bots: 38 IPv4 ranges and no IPv6 in October 2026.
Also run by Anthropic: Claude-SearchBot, ClaudeBot.
Other user-triggered fetchers: ChatGPT-User, Perplexity-User, MistralAI-User, Meta-ExternalFetcher.
Build a complete file with the robots.txt generator.