Bot directory · AI search

AI search · Anthropic

Claude-SearchBot

Claude-SearchBot is the crawler Anthropic uses to index pages so Claude’s web search can find and cite them. Blocking it can make your site less visible in Claude’s answers. It is separate from ClaudeBot, Anthropic’s training crawler, so you can stay findable in Claude without opting in to training.

Checked against Anthropic’s documentation on 19 October 2026

Operator
Anthropic
Purpose
AI answers. Crawlers that build the indexes AI assistants answer from and cite. Blocking one keeps you out of its answers.
robots.txt token
Claude-SearchBot
We test with
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com)
a string with the Claude-SearchBot token. Anthropic doesn’t publish the full string.
Documentation
support.claude.com/en/articles/8896518

What Claude-SearchBot does

Claude-SearchBot navigates the web to improve search result quality for users.

support.claude.com

Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site’s visibility and accuracy in user search results.

support.claude.com
BotWhat it doesModel training
ClaudeBotCollects web content that could contribute to training Anthropic’s modelsYes
Claude-SearchBotIndexes pages to improve Claude’s search resultsNot described as training
Claude-UserOpens a page when a Claude user asks a question that needs itNot described as training

The Claude-SearchBot user agent

Anthropic names its bots’ robots.txt tokens but doesn’t publish their full user agent strings. Match on the token, which appears in every request: Claude-SearchBot, case-insensitively. Strings quoted elsewhere are observations, not Anthropic’s documentation.

The token is unique among Anthropic’s bots: a rule matching claude-searchbot doesn’t catch the other two.

How Claude-SearchBot reads robots.txt

Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.

support.claude.com

Claude-SearchBot IP addresses and how to verify them

Anthropic publishes one list of addresses for all three of its bots, so a match tells you a request came from Anthropic, not which bot sent it; the user agent says that. Anthropic documents no reverse DNS names.

ListCoversIPv4IPv6Dated
claude.com/crawling/bots.jsonClaudeBot, Claude-SearchBot and Claude-User3807 October 2026

Anthropic recommends robots.txt over IP blocking for opting out of Claude-SearchBot: “Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.” If you block its addresses, keep /robots.txt open to them.

Check an IP now

Paste an address from your logs. We match it against Anthropic’s current list, along with the lists of the other operators that publish them.

Block Claude-SearchBot at your firewall or CDN

robots.txt asks; a firewall enforces. Blocking Claude-SearchBot can make your site less visible in Claude’s search answers, so most sites that want AI visibility allow it. Match the name case-insensitively and anywhere in the string, never a full string or a version.

Cloudflare

In Security → WAF → Custom rules, create a rule with this expression and the action Block. Cloudflare’s contains is case-sensitive, hence lower().

(lower(http.user_agent) contains "claude-searchbot")

Cloudflare also has one-click controls: AI bot policies, which block AI crawlers by what they do (search, agent or training), and AI Crawl Control, which lets you allow or block crawlers one by one and shows which of them fetch your pages.

nginx

# in the http block
map $http_user_agent $blocked_bot {
    default 0;
    ~*claude-searchbot 1;
}

# in the server block
if ($blocked_bot) { return 403; }

Block Claude-SearchBot

Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt. Blocking Claude-SearchBot may reduce how visible and accurately described your site is in Claude’s search results. To opt out of training only, block ClaudeBot.

User-agent: Claude-SearchBot
Disallow: /

Allow Claude-SearchBot when everything else is blocked

A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.

User-agent: Claude-SearchBot
Allow: /

User-agent: *
Disallow: /

Does Claude-SearchBot reach your site?

The checker applies your robots.txt the way Claude-SearchBot does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on Claude-SearchBot.

Questions about Claude-SearchBot

What is Claude-SearchBot?

Claude-SearchBot is the crawler Anthropic uses to improve the quality of search results in Claude. It indexes pages so Claude can find and cite them when it searches the web.

What happens if I block Claude-SearchBot?

Anthropic says blocking it prevents your content being indexed for search optimisation, which may reduce your site’s visibility and accuracy in Claude’s search results.

Is Claude-SearchBot used for training?

Anthropic describes ClaudeBot, not Claude-SearchBot, as the bot that collects content for training. Block ClaudeBot to opt out of training while staying in Claude’s search.

How do I verify Claude-SearchBot?

Match the IP against Anthropic’s list at claude.com/crawling/bots.json, which covers all three of its bots, and check that the user agent contains Claude-SearchBot.

Also run by Anthropic: ClaudeBot, Claude-User.

Other ai search: OAI-SearchBot, PerplexityBot, DuckAssistBot, Amazonbot, YouBot.

Build a complete file with the robots.txt generator.