Claude-SearchBot
Claude-SearchBot is the crawler Anthropic uses to index pages so Claude’s web search can find and cite them. Blocking it can make your site less visible in Claude’s answers. It is separate from ClaudeBot, Anthropic’s training crawler, so you can stay findable in Claude without opting in to training.
- Operator
- Anthropic
- Purpose
- AI answers. Crawlers that build the indexes AI assistants answer from and cite. Blocking one keeps you out of its answers.
- robots.txt token
Claude-SearchBot- We test with
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com)
a string with theClaude-SearchBottoken. Anthropic doesn’t publish the full string.- Documentation
- support.claude.com/en/articles/8896518
What Claude-SearchBot does
Claude-SearchBot navigates the web to improve search result quality for users.
support.claude.com
Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site’s visibility and accuracy in user search results.
support.claude.com
| Bot | What it does | Model training |
|---|---|---|
ClaudeBot | Collects web content that could contribute to training Anthropic’s models | Yes |
Claude-SearchBot | Indexes pages to improve Claude’s search results | Not described as training |
Claude-User | Opens a page when a Claude user asks a question that needs it | Not described as training |
The Claude-SearchBot user agent
Anthropic names its bots’ robots.txt tokens but doesn’t publish their full user agent strings. Match on the token, which appears in every request: Claude-SearchBot, case-insensitively. Strings quoted elsewhere are observations, not Anthropic’s documentation.
The token is unique among Anthropic’s bots: a rule matching claude-searchbot doesn’t catch the other two.
How Claude-SearchBot reads robots.txt
- All three bots honour robots.txt. Anthropic doesn’t exempt Claude-User, unlike OpenAI and Perplexity for their user-triggered fetchers.
- Crawl-delay works. Anthropic supports it to slow crawling, though it isn’t part of the robots.txt standard.
- Each subdomain has its own robots.txt. Anthropic asks you to add the rules to every subdomain you want to opt out.
- No CAPTCHA solving. Anthropic says its bots respect anti-circumvention technology and won’t try to get past CAPTCHAs.
Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.
support.claude.com
Claude-SearchBot IP addresses and how to verify them
Anthropic publishes one list of addresses for all three of its bots, so a match tells you a request came from Anthropic, not which bot sent it; the user agent says that. Anthropic documents no reverse DNS names.
| List | Covers | IPv4 | IPv6 | Dated |
|---|---|---|---|---|
| claude.com/crawling/bots.json | ClaudeBot, Claude-SearchBot and Claude-User | 38 | 0 | 7 October 2026 |
Anthropic recommends robots.txt over IP blocking for opting out of Claude-SearchBot: “Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.” If you block its addresses, keep /robots.txt open to them.
Check an IP now
Paste an address from your logs. We match it against Anthropic’s current list, along with the lists of the other operators that publish them.
Block Claude-SearchBot at your firewall or CDN
robots.txt asks; a firewall enforces. Blocking Claude-SearchBot can make your site less visible in Claude’s search answers, so most sites that want AI visibility allow it. Match the name case-insensitively and anywhere in the string, never a full string or a version.
Cloudflare
In Security → WAF → Custom rules, create a rule with this expression and the action Block. Cloudflare’s contains is case-sensitive, hence lower().
(lower(http.user_agent) contains "claude-searchbot")
Cloudflare also has one-click controls: AI bot policies, which block AI crawlers by what they do (search, agent or training), and AI Crawl Control, which lets you allow or block crawlers one by one and shows which of them fetch your pages.
nginx
# in the http block
map $http_user_agent $blocked_bot {
default 0;
~*claude-searchbot 1;
}
# in the server block
if ($blocked_bot) { return 403; }
Block Claude-SearchBot
Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt. Blocking Claude-SearchBot may reduce how visible and accurately described your site is in Claude’s search results. To opt out of training only, block ClaudeBot.
User-agent: Claude-SearchBot
Disallow: /
Allow Claude-SearchBot when everything else is blocked
A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.
User-agent: Claude-SearchBot
Allow: /
User-agent: *
Disallow: /
Does Claude-SearchBot reach your site?
The checker applies your robots.txt the way Claude-SearchBot does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on Claude-SearchBot.
Questions about Claude-SearchBot
What is Claude-SearchBot?
Claude-SearchBot is the crawler Anthropic uses to improve the quality of search results in Claude. It indexes pages so Claude can find and cite them when it searches the web.
What happens if I block Claude-SearchBot?
Anthropic says blocking it prevents your content being indexed for search optimisation, which may reduce your site’s visibility and accuracy in Claude’s search results.
Is Claude-SearchBot used for training?
Anthropic describes ClaudeBot, not Claude-SearchBot, as the bot that collects content for training. Block ClaudeBot to opt out of training while staying in Claude’s search.
How do I verify Claude-SearchBot?
Match the IP against Anthropic’s list at claude.com/crawling/bots.json, which covers all three of its bots, and check that the user agent contains Claude-SearchBot.
Also run by Anthropic: ClaudeBot, Claude-User.
Other ai search: OAI-SearchBot, PerplexityBot, DuckAssistBot, Amazonbot, YouBot.
Build a complete file with the robots.txt generator.