canaicrawl Get early access

Who blocks AI · #1098 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does sophos.com block AI crawlers?

Yes. sophos.com’s robots.txt blocks 1 of the 19 AI crawlers we track, including Amazonbot. It also blocks PetalBot among search engines.

GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed
robots.txt
www.sophos.com/robots.txt · 104 lines
llms.txt
found
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedline 55 · Allow: / under User-agent: Googlebot
BingbotMicrosoftAllowedline 40 · Allow: / under User-agent: Bingbot
DuckDuckBotDuckDuckGoAllowedno group applies
ApplebotAppleAllowedline 55 · Allow: / under User-agent: Googlebot
PetalBotHuaweiBlockedline 100 · Disallow: / under User-agent: PetalBot
YandexBotYandexAllowedno group applies
BaiduspiderBaiduAllowedno group applies

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIAllowedline 67 · Allow: / under User-agent: OAI-SearchBot
PerplexityBotPerplexityAllowedline 70 · Allow: / under User-agent: PerplexityBot
Claude-SearchBotAnthropicAllowedno group applies
DuckAssistBotDuckDuckGoAllowedno group applies
AmazonbotAmazonBlockedline 82 · Disallow: / under User-agent: Amazonbot
YouBotYou.comAllowedline 97 · Allow: / under User-agent: YouBot

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIAllowedline 61 · Allow: / under User-agent: GPTBot
ClaudeBotAnthropicAllowedline 37 · Allow: / under User-agent: ClaudeBot
Google-ExtendedGoogleAllowedline 58 · Allow: / under User-agent: Google-Extended
CCBotCommon CrawlAllowedno group applies
Applebot-ExtendedAppleAllowedno group applies
GoogleOtherGoogleAllowedno group applies
Meta-ExternalAgentMetaAllowedno group applies
BytespiderByteDanceAllowedno group applies

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIAllowedline 49 · Allow: / under User-agent: ChatGPT-User
Claude-UserAnthropicAllowedno group applies
Perplexity-UserPerplexityAllowedno group applies
MistralAI-UserMistralAllowedno group applies
Meta-ExternalFetcherMetaAllowedno group applies

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedno group applies
SemrushBotSemrushAllowedno group applies
MJ12botMajesticAllowedno group applies
DotBotMozAllowedno group applies
View www.sophos.com/robots.txt · 104 lines
1# robots.txt for www.sophos.com (generated November 2025)
2
3# Disallow internal or non-public endpoints (update these for content stack)
4#User-agent: *
5#Disallow: /admin/
6#Disallow: /internal/
7#Disallow: /private/
8#Disallow: /error/
9
10# Allow all other content, including documentation, downloads and product pages.
11Allow: /
12
13# Allow the LLM policy file
14Allow: /llms.txt
15
16# Sitemaps for supported locales
17Sitemap: https://www.sophos.com/en-us/sitemap.xml
18Sitemap: https://www.sophos.com/en-gb/sitemap.xml
19Sitemap: https://www.sophos.com/de-de/sitemap.xml
20Sitemap: https://www.sophos.com/es-es/sitemap.xml
21Sitemap: https://www.sophos.com/it-it/sitemap.xml
22Sitemap: https://www.sophos.com/fr-fr/sitemap.xml
23Sitemap: https://www.sophos.com/ja-jp/sitemap.xml
24Sitemap: https://www.sophos.com/pt-br/sitemap.xml
25Sitemap: https://www.sophos.com/zh-cn/sitemap.xml
26
27# video sitemap
28Sitemap: https://www.sophos.com/video-sitemap.xml
29
30
31# Allow AI Search and Agent Crawlers
32
33User-agent: Anthropic-ai
34Allow: /
35
36User-agent: ClaudeBot
37Allow: /
38
39User-agent: Bingbot
40Allow: /
41
42User-agent: BingAI
43Allow: /
44
45User-agent: BingPreview
46Allow: /
47
48User-agent: ChatGPT-User
49Allow: /
50
51User-agent: CopilotBot
52Allow: /
53
54User-agent: Googlebot
55Allow: /
56
57User-agent: Google-Extended
58Allow: /
59
60User-agent: GPTBot
61Allow: /
62
63User-agent: LlamaBot
64Allow: /
65
66User-agent: OAI-SearchBot
67Allow: /
68
69User-agent: PerplexityBot
70Allow: /
71
72User-agent: xAI-Bot
73Allow: /
74
75User-agent: Amzn-SearchBot
76Allow: /
77
78User-agent: Amzn-User
79Allow: /
80
81User-agent: Amazonbot
82Disallow: /
83
84User-agent: FirecrawlAgent
85Allow: /
86
87User-agent: AndiBot
88Allow: /
89
90User-agent: ExaBot
91Allow: /
92
93User-agent: PhindBot
94Allow: /
95
96User-agent: YouBot
97Allow: /
98
99User-agent: PetalBot
100Disallow: /
101
102# Disallow specific LLM crawler
103User-agent: DeepSeekBot
104Disallow: /

What do the bots actually get?

This page reads robots.txt, which states what sophos.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on sophos.com Test a path in the robots.txt tester

History

Questions

Does sophos.com block GPTBot?

No. sophos.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does sophos.com block ClaudeBot?

No. sophos.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does sophos.com block Google-Extended?

No. sophos.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does sophos.com block PerplexityBot?

No. sophos.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find sophos.com?

Yes. sophos.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.