Who blocks AI · #549 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does vox.com block AI crawlers?
Yes. vox.com’s robots.txt blocks 15 of the 19 AI crawlers we track, including ClaudeBot, Google-Extended, PerplexityBot and 12 more. It also blocks PetalBot and Baiduspider among search engines.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.vox.com/robots.txt · 542 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | line 6 · Allow: / under User-agent: Applebot |
PetalBot | Huawei | Blocked | line 367 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Blocked | line 67 · Disallow: / under User-agent: Baiduspider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Blocked | line 362 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 122 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 202 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Allowed | line 3 · Allow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 507 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | line 9 · Allow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 137 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 237 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 107 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 62 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 292 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 97 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 117 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 127 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 357 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 317 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 297 · Disallow: / under User-agent: meta-externalfetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 17 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Blocked | line 417 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.vox.com/robots.txt · 542 lines
1
2User-agent: Amazonbot
3Allow: /
4
5User-agent: Applebot
6Allow: /
7
8User-agent: GPTBot
9Allow: /
10
11User-agent: A6-Indexer
12Disallow: /
13Allow: /sp/
14Allow: /strategist
15
16User-agent: AhrefsBot
17Disallow: /
18Allow: /sp/
19Allow: /strategist
20
21User-agent: AI2Bot
22Disallow: /
23Allow: /sp/
24Allow: /strategist
25
26User-agent: AI2Bot-DeepResearchEval
27Disallow: /
28Allow: /sp/
29Allow: /strategist
30
31User-agent: Ai2Bot-Dolma
32Disallow: /
33Allow: /sp/
34Allow: /strategist
35
36User-agent: aiHitBot
37Disallow: /
38Allow: /sp/
39Allow: /strategist
40
41User-agent: Anchor Browser
42Disallow: /
43Allow: /sp/
44Allow: /strategist
45
46User-agent: anthropic-ai
47Disallow: /
48Allow: /sp/
49Allow: /strategist
50
51User-agent: ApifyBot
52Disallow: /
53Allow: /sp/
54Allow: /strategist
55
56User-agent: ApifyWebsiteContentCrawler
57Disallow: /
58Allow: /sp/
59Allow: /strategist
60
61User-agent: Applebot-Extended
62Disallow: /
63Allow: /sp/
64Allow: /strategist
65
66User-agent: Baiduspider
67Disallow: /
68Allow: /sp/
69Allow: /strategist
70
71User-agent: betaBot
72Disallow: /
73Allow: /sp/
74Allow: /strategist
75
76User-agent: BLEXBot
77Disallow: /
78Allow: /sp/
79Allow: /strategist
80
81User-agent: botify
82Disallow: /
83Allow: /sp/
84Allow: /strategist
85
86User-agent: brands-bot-logo
87Disallow: /
88Allow: /sp/
89Allow: /strategist
90
91User-agent: Brightbot
92Disallow: /
93Allow: /sp/
94Allow: /strategist
95
96User-agent: Bytespider
97Disallow: /
98Allow: /sp/
99Allow: /strategist
100
101User-agent: calculon
102Disallow: /
103Allow: /sp/
104Allow: /strategist
105
106User-agent: CCBot
107Disallow: /
108Allow: /sp/
109Allow: /strategist
110
111User-agent: ChatGLM-Spider
112Disallow: /
113Allow: /sp/
114Allow: /strategist
115
116User-agent: ChatGPT-User
117Disallow: /
118Allow: /sp/
119Allow: /strategist
120
121User-agent: Claude-SearchBot
122Disallow: /
123Allow: /sp/
124Allow: /strategist
125
126User-agent: Claude-User
127Disallow: /
128Allow: /sp/
129Allow: /strategist
130
131User-agent: Claude-Web
132Disallow: /
133Allow: /sp/
134Allow: /strategist
135
136User-agent: ClaudeBot
137Disallow: /
138Allow: /sp/
139Allow: /strategist
140
141User-agent: CloudVertexBot
142Disallow: /
143Allow: /sp/
144Allow: /strategist
145
146User-agent: cohere-ai
147Disallow: /
148Allow: /sp/
149Allow: /strategist
150
151User-agent: cohere-training-data-crawler
152Disallow: /
153Allow: /sp/
154Allow: /strategist
155
156User-agent: collection@infegy.com
157Disallow: /
158Allow: /sp/
159Allow: /strategist
160
161User-agent: Cotoyogi
162Disallow: /
163Allow: /sp/
164Allow: /strategist
165
166User-agent: Crawl4AI
167Disallow: /
168Allow: /sp/
169Allow: /strategist
170
171User-agent: CrunchBot
172Disallow: /
173Allow: /sp/
174Allow: /strategist
175
176User-agent: deepcrawl
177Disallow: /
178Allow: /sp/
179Allow: /strategist
180
181User-agent: DeepSeekBot
182Disallow: /
183Allow: /sp/
184Allow: /strategist
185
186User-agent: Diffbot
187Disallow: /
188Allow: /sp/
189Allow: /strategist
190
191User-agent: DirBuster
192Disallow: /
193Allow: /sp/
194Allow: /strategist
195
196User-agent: DTS
197Disallow: /
198Allow: /sp/
199Allow: /strategist
200
201User-agent: DuckAssistBot
202Disallow: /
203Allow: /sp/
204Allow: /strategist
205
206User-agent: EasouSpider
207Disallow: /
208Allow: /sp/
209Allow: /strategist
210
211User-agent: FacebookBot
212Disallow: /
213Allow: /sp/
214Allow: /strategist
215
216User-agent: FirecrawlAgent
217Disallow: /
218Allow: /sp/
219Allow: /strategist
220
221User-agent: FriendlyCrawler
222Disallow: /
223Allow: /sp/
224Allow: /strategist
225
226User-agent: GlutenCrawler
227Disallow: /
228Allow: /sp/
229Allow: /strategist
230
231User-agent: Google-CloudVertexBot
232Disallow: /
233Allow: /sp/
234Allow: /strategist
235
236User-agent: Google-Extended
237Disallow: /
238Allow: /sp/
239Allow: /strategist
240
241User-agent: GrapeshotCrawler
242Disallow: /
243Allow: /sp/
244Allow: /strategist
245
246User-agent: iaskspider
247Disallow: /
248Allow: /sp/
249Allow: /strategist
250
251User-agent: ICC-Crawler
252Disallow: /
253Allow: /sp/
254Allow: /strategist
255
256User-agent: ImagesiftBot
257Disallow: /
258Allow: /sp/
259Allow: /strategist
260
261User-agent: img2dataset
262Disallow: /
263Allow: /sp/
264Allow: /strategist
265
266User-agent: JackCrawl
267Disallow: /
268Allow: /sp/
269Allow: /strategist
270
271User-agent: Jetslide
272Disallow: /
273Allow: /sp/
274Allow: /strategist
275
276User-agent: linkapedia_bot
277Disallow: /
278Allow: /sp/
279Allow: /strategist
280
281User-agent: Mediatoolkitbot
282Disallow: /
283Allow: /sp/
284Allow: /strategist
285
286User-agent: MegaIndex
287Disallow: /
288Allow: /sp/
289Allow: /strategist
290
291User-agent: meta-externalagent
292Disallow: /
293Allow: /sp/
294Allow: /strategist
295
296User-agent: meta-externalfetcher
297Disallow: /
298Allow: /sp/
299Allow: /strategist
300
301User-agent: Microsoft BITS
302Disallow: /
303Allow: /sp/
304Allow: /strategist
305
306User-agent: Microsoft URL Control
307Disallow: /
308Allow: /sp/
309Allow: /strategist
310
311User-agent: Microsoft-WebDAV-MiniRedir
312Disallow: /
313Allow: /sp/
314Allow: /strategist
315
316User-agent: MistralAI-User
317Disallow: /
318Allow: /sp/
319Allow: /strategist
320
321User-agent: news-please
322Disallow: /
323Allow: /sp/
324Allow: /strategist
325
326User-agent: Novellum
327Disallow: /
328Allow: /sp/
329Allow: /strategist
330
331User-agent: NTENTbot
332Disallow: /
333Allow: /sp/
334Allow: /strategist
335
336User-agent: Nutch
337Disallow: /
338Allow: /sp/
339Allow: /strategist
340
341User-agent: omgili
342Disallow: /
343Allow: /sp/
344Allow: /strategist
345
346User-agent: Omgilibot
347Disallow: /
348Allow: /sp/
349Allow: /strategist
350
351User-agent: OpenLinkProfiler
352Disallow: /
353Allow: /sp/
354Allow: /strategist
355
356User-agent: Perplexity-User
357Disallow: /
358Allow: /sp/
359Allow: /strategist
360
361User-agent: PerplexityBot
362Disallow: /
363Allow: /sp/
364Allow: /strategist
365
366User-agent: PetalBot
367Disallow: /
368Allow: /sp/
369Allow: /strategist
370
371User-agent: Primalbot
372Disallow: /
373Allow: /sp/
374Allow: /strategist
375
376User-agent: python-requests
377Disallow: /
378Allow: /sp/
379Allow: /strategist
380
381User-agent: Re-Animator
382Disallow: /
383Allow: /sp/
384Allow: /strategist
385
386User-agent: Riddler
387Disallow: /
388Allow: /sp/
389Allow: /strategist
390
391User-agent: SafeDNSBot
392Disallow: /
393Allow: /sp/
394Allow: /strategist
395
396User-agent: SBIntuitionsBot
397Disallow: /
398Allow: /sp/
399Allow: /strategist
400
401User-agent: Scrapy
402Disallow: /
403Allow: /sp/
404Allow: /strategist
405
406User-agent: SearchAtlas
407Disallow: /
408Allow: /sp/
409Allow: /strategist
410
411User-agent: SemanticScholarBot
412Disallow: /
413Allow: /sp/
414Allow: /strategist
415
416User-agent: SemrushBot
417Disallow: /
418Allow: /sp/
419Allow: /strategist
420
421User-agent: SentiBot
422Disallow: /
423Allow: /sp/
424Allow: /strategist
425
426User-agent: Sogou
427Disallow: /
428Allow: /sp/
429Allow: /strategist
430
431User-agent: Spider-Pig
432Disallow: /
433Allow: /sp/
434Allow: /strategist
435
436User-agent: sqlmap
437Disallow: /
438Allow: /sp/
439Allow: /strategist
440
441User-agent: StorygizeBot
442Disallow: /
443Allow: /sp/
444Allow: /strategist
445
446User-agent: Test Certificate Info
447Disallow: /
448Allow: /sp/
449Allow: /strategist
450
451User-agent: TikTokSpider
452Disallow: /
453Allow: /sp/
454Allow: /strategist
455
456User-agent: Timpibot
457Disallow: /
458Allow: /sp/
459Allow: /strategist
460
461User-agent: Turnitin
462Disallow: /
463Allow: /sp/
464Allow: /strategist
465
466User-agent: uipbot
467Disallow: /
468Allow: /sp/
469Allow: /strategist
470
471User-agent: vBSEO
472Disallow: /
473Allow: /sp/
474Allow: /strategist
475
476User-agent: VegeBot
477Disallow: /
478Allow: /sp/
479Allow: /strategist
480
481User-agent: VelenPublicWebCrawler
482Disallow: /
483Allow: /sp/
484Allow: /strategist
485
486User-agent: Web Link Validator
487Disallow: /
488Allow: /sp/
489Allow: /strategist
490
491User-agent: WebCruiser
492Disallow: /
493Allow: /sp/
494Allow: /strategist
495
496User-agent: webzio-extended
497Disallow: /
498Allow: /sp/
499Allow: /strategist
500
501User-agent: Xenu
502Disallow: /
503Allow: /sp/
504Allow: /strategist
505
506User-agent: YouBot
507Disallow: /
508Allow: /sp/
509Allow: /strategist
510
511User-agent: *
512Disallow: /wp-admin/
513Allow: /wp-admin/admin-ajax.php
514
515User-agent: Googlebot-News
516Disallow: /ad/
517Disallow: /sponsored/
518Disallow: /sp/
519
520User-agent: *
521Allow: /sp/
522Disallow: /admin
523Disallow: /newfanshot
524Disallow: /users/*/replies
525Disallow: /users/*/comments
526Disallow: /login
527Disallow: /account
528Disallow: /auth/*
529Disallow: /chorus_auth
530Disallow: /sso
531Disallow: /search
532Disallow: /the-highlight$
533Disallow: /share$
534Disallow: /share/*
535Disallow: /share?*
536
537# Google news sitemap
538Sitemap: https://www.vox.com/sitemaps/google_news
539
540# Sitemap archive
541Sitemap: https://www.vox.com/sitemaps
542
What do the bots actually get?
This page reads robots.txt, which states what vox.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on vox.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does vox.com block GPTBot?
No. vox.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does vox.com block ClaudeBot?
Yes. vox.com’s robots.txt disallows ClaudeBot from the whole site (line 137: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does vox.com block Google-Extended?
Yes. vox.com’s robots.txt disallows Google-Extended from the whole site (line 237: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does vox.com block PerplexityBot?
Yes. vox.com’s robots.txt disallows PerplexityBot from the whole site (line 362: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find vox.com?
Yes. vox.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.