Who blocks AI · #345 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does theconversation.com block AI crawlers?
Yes. theconversation.com’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks Applebot and PetalBot among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- theconversation.com/robots.txt · 216 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Partly blocked | line 34 · Disallow: /metrics/ line 35 · Disallow: /auth/ and 11 more |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Blocked | line 215 · Disallow: / under User-agent: Applebot |
PetalBot | Huawei | Blocked | line 215 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 215 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 215 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 215 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 215 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 215 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 215 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 215 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 215 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 215 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 215 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 215 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 215 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Blocked | line 215 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 215 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 215 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 215 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 215 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 215 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 215 · Disallow: / under User-agent: meta-externalfetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View theconversation.com/robots.txt · 216 lines
1User-Agent: *
2Disallow: /metrics/
3Disallow: /auth/
4Disallow: /content/
5Disallow: /comments/*
6Disallow: /sign_in?
7Disallow: /sign_up?
8Disallow: /become-an-author?
9Disallow: /profiles/*/articles
10Disallow: /profiles/*/activities
11Disallow: /*/search?
12Disallow: /experts/search?
13Disallow: /experts/search/*
14Disallow: /pb/*
15
16Sitemap: https://theconversation.com/sitemap.xml
17Sitemap: https://theconversation.com/africa/sitemap_news.xml
18Sitemap: https://theconversation.com/au/sitemap_news.xml
19Sitemap: https://theconversation.com/br/sitemap_news.xml
20Sitemap: https://theconversation.com/ca/sitemap_news.xml
21Sitemap: https://theconversation.com/ca-fr/sitemap_news.xml
22Sitemap: https://theconversation.com/catalan/sitemap_news.xml
23Sitemap: https://theconversation.com/es/sitemap_news.xml
24Sitemap: https://theconversation.com/europe/sitemap_news.xml
25Sitemap: https://theconversation.com/fr/sitemap_news.xml
26Sitemap: https://theconversation.com/global/sitemap_news.xml
27Sitemap: https://theconversation.com/id/sitemap_news.xml
28Sitemap: https://theconversation.com/nz/sitemap_news.xml
29Sitemap: https://theconversation.com/uk/sitemap_news.xml
30Sitemap: https://theconversation.com/us/sitemap_news.xml
31
32User-agent: bingbot
33Crawl-delay: 10
34Disallow: /metrics/
35Disallow: /auth/
36Disallow: /content/
37Disallow: /comments/*
38Disallow: /sign_in?
39Disallow: /sign_up?
40Disallow: /become-an-author?
41Disallow: /profiles/*/articles
42Disallow: /profiles/*/activities
43Disallow: /*/search?
44Disallow: /experts/search?
45Disallow: /experts/search/*
46Disallow: /pb/*
47
48User-agent: AddSearchBot
49User-agent: AgentTimes
50User-agent: AI2Bot
51User-agent: AI2Bot-DeepResearchEval
52User-agent: Ai2Bot-Dolma
53User-agent: aiHitBot
54User-agent: AIWebIndex
55User-agent: AliyunSecBot/Aliyun
56User-agent: amazon-kendra
57User-agent: amazon-QBusiness
58User-agent: Amazonbot
59User-agent: AmazonBuyForMe
60User-agent: Amzn-SearchBot
61User-agent: Amzn-User
62User-agent: Andibot
63User-agent: Anomura
64User-agent: anthropic-ai
65User-agent: ApifyBot
66User-agent: ApifyWebsiteContentCrawler
67User-agent: Applebot
68User-agent: Applebot-Extended
69User-agent: Aranet-SearchBot
70User-agent: atlassian-bot
71User-agent: Autn-WKOOP
72User-agent: Awario
73User-agent: AzureAI-SearchBot
74User-agent: barkrowler
75User-agent: bedrockbot
76User-agent: bigsur.ai
77User-agent: Bravebot
78User-agent: Brightbot
79User-agent: Brightbot 1.0
80User-agent: BuddyBot
81User-agent: Bytespider
82User-agent: CCBot
83User-agent: Channel3Bot
84User-agent: ChatGLM-Spider
85User-agent: ChatGPT Agent
86User-agent: ChatGPT-User
87User-agent: Claude-Code
88User-agent: Claude-SearchBot
89User-agent: Claude-User
90User-agent: Claude-Web
91User-agent: ClaudeBot
92User-agent: Cloudflare-AutoRAG
93User-agent: CloudVertexBot
94User-agent: Code
95User-agent: cohere-ai
96User-agent: cohere-training-data-crawler
97User-agent: Cotoyogi
98User-agent: CragCrawler
99User-agent: Crawl4AI
100User-agent: Crawlspace
101User-agent: Cursor
102User-agent: Datenbank Crawler
103User-agent: DeepSeekBot
104User-agent: Devin
105User-agent: Diffbot
106User-agent: dobot
107User-agent: DuckAssistBot
108User-agent: Echobot Bot
109User-agent: EchoboxBot
110User-agent: ExaBot
111User-agent: ExaSearchBot
112User-agent: FacebookBot
113User-agent: Factset_spyderbot
114User-agent: FirecrawlAgent
115User-agent: FriendlyCrawler
116User-agent: GeistHaus-PageFetcher
117User-agent: Gemini-Deep-Research
118User-agent: Google-Agent
119User-agent: Google-CloudVertexBot
120User-agent: Google-Extended
121User-agent: Google-Firebase
122User-agent: Google-Gemini-CLI
123User-agent: Google-NotebookLM
124User-agent: GoogleAgent-Mariner
125User-agent: GoogleAgent-URLContext
126User-agent: GoogleOther
127User-agent: GoogleOther-Image
128User-agent: GoogleOther-Video
129User-agent: GPTBot
130User-agent: HawaiiBot
131User-agent: HenkBot
132User-agent: iAskBot
133User-agent: iaskspider
134User-agent: iaskspider/2.0
135User-agent: ICC-Crawler
136User-agent: ImagesiftBot
137User-agent: imageSpider
138User-agent: img2dataset
139User-agent: ISSCyberRiskCrawler
140User-agent: kagi-fetcher
141User-agent: Kangaroo Bot
142User-agent: Kimi-User
143User-agent: KlaviyoAIBot
144User-agent: KunatoCrawler
145User-agent: laion-huggingface-processor
146User-agent: LAIONDownloader
147User-agent: LCC
148User-agent: Lightpanda
149User-agent: LinerBot
150User-agent: Linguee Bot
151User-agent: LinkupBot
152User-agent: Manus-User
153User-agent: meta-externalagent
154User-agent: meta-externalfetcher
155User-agent: meta-webindexer
156User-agent: MistralAI-User
157User-agent: MistralAI-User/1.0
158User-agent: Mozilla-Tabstack
159User-agent: MyCentralAIScraperBot
160User-agent: NagetBot
161User-agent: netEstate Imprint Crawler
162User-agent: newsai
163User-agent: NotebookLM
164User-agent: NovaAct
165User-agent: OAI-SearchBot
166User-agent: omgili
167User-agent: omgilibot
168User-agent: OpenAI
169User-agent: opencode
170User-agent: Operator
171User-agent: PanguBot
172User-agent: Panscient
173User-agent: panscient.com
174User-agent: Perplexity-User
175User-agent: PerplexityBot
176User-agent: PetalBot
177User-agent: PhindBot
178User-agent: Poggio-Citations
179User-agent: Poseidon Research Crawler
180User-agent: QualifiedBot
181User-agent: Querit-SearchBot
182User-agent: QueritBot
183User-agent: QuillBot
184User-agent: quillbot.com
185User-agent: Reflectionbot
186User-agent: SBIntuitionsBot
187User-agent: Scrapy
188User-agent: SemrushBot-OCOB
189User-agent: SemrushBot-SWA
190User-agent: Shap-User
191User-agent: ShapBot
192User-agent: Sidetrade indexer bot
193User-agent: Spider
194User-agent: TavilyBot
195User-agent: Terra Cotta
196User-agent: TerraCotta
197User-agent: Thinkbot
198User-agent: TikTokSpider
199User-agent: Timpibot
200User-agent: TongyiBot
201User-agent: Trae
202User-agent: TwinAgent
203User-agent: UseAI
204User-agent: VelenPublicWebCrawler
205User-agent: WARDBot
206User-agent: Webzio-Extended
207User-agent: wpbot
208User-agent: WRTNBot
209User-agent: YaK
210User-agent: YandexAdditional
211User-agent: YandexAdditionalBot
212User-agent: YiyanBot
213User-agent: YouBot
214User-agent: ZanistaBot
215Disallow: /
216
What do the bots actually get?
This page reads robots.txt, which states what theconversation.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on theconversation.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does theconversation.com block GPTBot?
Yes. theconversation.com’s robots.txt disallows GPTBot from the whole site (line 215: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does theconversation.com block ClaudeBot?
Yes. theconversation.com’s robots.txt disallows ClaudeBot from the whole site (line 215: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does theconversation.com block Google-Extended?
Yes. theconversation.com’s robots.txt disallows Google-Extended from the whole site (line 215: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does theconversation.com block PerplexityBot?
Yes. theconversation.com’s robots.txt disallows PerplexityBot from the whole site (line 215: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find theconversation.com?
Not through its own crawler. theconversation.com’s robots.txt disallows OAI-SearchBot from the whole site (line 215: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.