Who blocks AI · #1324 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does adweek.com block AI crawlers?
Yes. adweek.com’s robots.txt blocks 1 of the 19 AI crawlers we track, including YouBot. It also blocks Baiduspider among search engines.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.adweek.com/robots.txt · 553 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no group applies |
DuckDuckBot | DuckDuckGo | Allowed | no group applies |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no group applies |
YandexBot | Yandex | Allowed | no group applies |
Baiduspider | Baidu | Blocked | line 31 · Disallow: / under User-agent: Baiduspider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no group applies |
PerplexityBot | Perplexity | Allowed | no group applies |
Claude-SearchBot | Anthropic | Allowed | no group applies |
DuckAssistBot | DuckDuckGo | Allowed | no group applies |
Amazonbot | Amazon | Allowed | no group applies |
YouBot | You.com | Blocked | line 544 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no group applies |
ClaudeBot | Anthropic | Allowed | no group applies |
Google-Extended | Allowed | line 2 · Allow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Allowed | no group applies |
Applebot-Extended | Apple | Allowed | no group applies |
GoogleOther | Allowed | no group applies | |
Meta-ExternalAgent | Meta | Allowed | no group applies |
Bytespider | ByteDance | Allowed | no group applies |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no group applies |
Claude-User | Anthropic | Allowed | no group applies |
Perplexity-User | Perplexity | Allowed | no group applies |
MistralAI-User | Mistral | Allowed | no group applies |
Meta-ExternalFetcher | Meta | Allowed | no group applies |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no group applies |
SemrushBot | Semrush | Allowed | no group applies |
MJ12bot | Majestic | Allowed | no group applies |
DotBot | Moz | Allowed | no group applies |
View www.adweek.com/robots.txt · 553 lines
1User-agent: Google-Extended
2Allow: /
3
4User-agent: Googlebot
5Allow: /wp-content/plugins/
6Allow: /wp-content/themes/
7Allow: /wp-content/uploads/
8
9User-agent: AdsBot-Google
10Allow: /wp-content/plugins/
11Allow: /wp-content/themes/
12
13User-agent: Googlebot-Image
14Allow: /wp-content/plugins/
15Allow: /wp-content/themes/
16Allow: /wp-content/uploads/
17
18User-agent: Twitterbot
19Allow: /sponsored/
20Allow: /partner-articles/
21
22Sitemap: https://www.adweek.com/sitemap.xml
23Sitemap: https://www.adweek.com/sitemap-1.xml
24Sitemap: https://www.adweek.com/tvnewser/sitemap.xml
25Sitemap: https://www.adweek.com/tvspy/sitemap.xml
26Sitemap: https://www.adweek.com/agencyspy/sitemap.xml
27Sitemap: https://www.adweek.com/lostremote/sitemap.xml
28
29User-agent: Baiduspider
30# https://darkvisitors.com/agents/baiduspider
31Disallow: /
32
33
34# START KNOWN AGENTS BLOCK
35# ---------------------------
36# Scraper
37# https://knownagents.com/agents/008
38
39User-agent: 008
40Disallow: /
41
42# Scraper
43# https://knownagents.com/agents/adstxtcrawler
44
45User-agent: AdsTxtCrawler
46Disallow: /
47
48# AI Data Provider
49# https://knownagents.com/agents/agentdatabot
50
51User-agent: AgentDataBot
52Disallow: /
53
54# Undocumented AI Agent
55# https://knownagents.com/agents/anthropic-ai
56
57User-agent: anthropic-ai
58Disallow: /
59
60# AI Data Provider
61# https://knownagents.com/agents/apifybot
62
63User-agent: ApifyBot
64Disallow: /
65
66# AI Data Provider
67# https://knownagents.com/agents/apifywebsitecontentcrawler
68
69User-agent: ApifyWebsiteContentCrawler
70Disallow: /
71
72# Undocumented AI Agent
73# https://knownagents.com/agents/aranet-searchbot
74
75User-agent: Aranet-SearchBot
76Disallow: /
77
78# Archiver
79# https://knownagents.com/agents/archive-it
80
81User-agent: Archive-It
82Disallow: /
83
84# Archiver
85# https://knownagents.com/agents/archive-org-bot
86
87User-agent: archive.org_bot
88Disallow: /
89
90# Archiver
91# https://knownagents.com/agents/arquivo-web-crawler
92
93User-agent: Arquivo-web-crawler
94Disallow: /
95
96# AI Data Provider
97# https://knownagents.com/agents/artsdata-crawler
98
99User-agent: artsdata-crawler
100Disallow: /
101
102# Archiver
103# https://knownagents.com/agents/authory
104
105User-agent: Authory
106Disallow: /
107
108# AI Data Provider
109# https://knownagents.com/agents/bixelbot
110
111User-agent: BixelBot
112Disallow: /
113
114# Archiver
115# https://knownagents.com/agents/bl-uk-lddc-bot
116
117User-agent: bl.uk_lddc_bot
118Disallow: /
119
120# Archiver
121# https://knownagents.com/agents/bne-es-bot
122
123User-agent: bne.es_bot
124Disallow: /
125
126# Archiver
127# https://knownagents.com/agents/bnf-fr-bot
128
129User-agent: bnf.fr_bot
130Disallow: /
131
132# AI Data Provider
133# https://knownagents.com/agents/bravebot
134
135User-agent: Bravebot
136Disallow: /
137
138# AI Data Provider
139# https://knownagents.com/agents/brightbot
140
141User-agent: Brightbot
142Disallow: /
143
144# Undocumented AI Agent
145# https://knownagents.com/agents/claude-web
146
147User-agent: Claude-Web
148Disallow: /
149
150# AI Data Provider
151# https://knownagents.com/agents/cloudflarebrowserrenderingcrawler
152
153User-agent: CloudflareBrowserRenderingCrawler
154Disallow: /
155
156# Undocumented AI Agent
157# https://knownagents.com/agents/cohere-ai
158
159User-agent: cohere-ai
160Disallow: /
161
162# AI Data Provider
163# https://knownagents.com/agents/cragcrawler
164
165User-agent: CragCrawler
166Disallow: /
167
168# Undocumented AI Agent
169# https://knownagents.com/agents/crawl4ai
170
171User-agent: Crawl4AI
172Disallow: /
173
174# AI Data Provider
175# https://knownagents.com/agents/crawlspace
176
177User-agent: Crawlspace
178Disallow: /
179
180# Archiver
181# https://knownagents.com/agents/cuch-org-archive-bot
182
183User-agent: CUCH.org Archive Bot
184Disallow: /
185
186# Scraper
187# https://knownagents.com/agents/dcrawl
188
189User-agent: dcrawl
190Disallow: /
191
192# AI Data Provider
193# https://knownagents.com/agents/desearchbot
194
195User-agent: DesearchBot
196Disallow: /
197
198# AI Data Provider
199# https://knownagents.com/agents/diffbot
200
201User-agent: Diffbot
202Disallow: /
203
204# AI Data Provider
205# https://knownagents.com/agents/exabot
206
207User-agent: ExaBot
208Disallow: /
209
210# AI Data Provider
211# https://knownagents.com/agents/firecrawlagent
212
213User-agent: FirecrawlAgent
214Disallow: /
215
216# Undocumented AI Agent
217# https://knownagents.com/agents/grok-agent
218
219User-agent: grok-agent
220Disallow: /
221
222# Undocumented AI Agent
223# https://knownagents.com/agents/grokbot
224
225User-agent: GrokBot
226Disallow: /
227
228# Scraper
229# https://knownagents.com/agents/helloworkjobpostingbot
230
231User-agent: HelloworkJobPostingBot
232Disallow: /
233
234# AI Data Provider
235# https://knownagents.com/agents/henkbot
236
237User-agent: HenkBot
238Disallow: /
239
240# Archiver
241# https://knownagents.com/agents/heritrix
242
243User-agent: heritrix
244Disallow: /
245
246# Scraper
247# https://knownagents.com/agents/httrack
248
249User-agent: HTTrack
250Disallow: /
251
252# Scraper
253# https://knownagents.com/agents/httrack-3-0
254
255User-agent: HTTrack 3.0
256Disallow: /
257
258# Archiver
259# https://knownagents.com/agents/ia-archiver
260
261User-agent: ia_archiver
262Disallow: /
263
264# Archiver
265# https://knownagents.com/agents/ia-archiver-web-archive-org
266
267User-agent: ia_archiver-web.archive.org
268Disallow: /
269
270# Archiver
271# https://knownagents.com/agents/iabot
272
273User-agent: IABot
274Disallow: /
275
276# Undocumented AI Agent
277# https://knownagents.com/agents/iaskbot
278
279User-agent: iAskBot
280Disallow: /
281
282# Undocumented AI Agent
283# https://knownagents.com/agents/iaskspider
284
285User-agent: iaskspider
286Disallow: /
287
288# Scraper
289# https://knownagents.com/agents/indeedjobbot
290
291User-agent: IndeedJobBot
292Disallow: /
293
294# Archiver
295# https://knownagents.com/agents/internet-archive
296
297User-agent: Internet Archive
298Disallow: /
299
300# AI Data Provider
301# https://knownagents.com/agents/keenable-user
302
303User-agent: Keenable-User
304Disallow: /
305
306# AI Data Provider
307# https://knownagents.com/agents/keenablebot
308
309User-agent: KeenableBot
310Disallow: /
311
312# Scraper
313# https://knownagents.com/agents/krawlerbot
314
315User-agent: KrawlerBot
316Disallow: /
317
318# Undocumented AI Agent
319# https://knownagents.com/agents/kunatocrawler
320
321User-agent: KunatoCrawler
322Disallow: /
323
324# Scraper
325# https://knownagents.com/agents/metainspector
326
327User-agent: MetaInspector
328Disallow: /
329
330# Scraper
331# https://knownagents.com/agents/metajobbot
332
333User-agent: MetaJobBot
334Disallow: /
335
336# Archiver
337# https://knownagents.com/agents/mirrorweb
338
339User-agent: mirrorweb
340Disallow: /
341
342# AI Data Provider
343# https://knownagents.com/agents/mozilla-tabstack
344
345User-agent: Mozilla-Tabstack
346Disallow: /
347
348# Archiver
349# https://knownagents.com/agents/netarkivindsamling
350
351User-agent: netarkivindsamling
352Disallow: /
353
354# Scraper
355# https://knownagents.com/agents/newspaper
356
357User-agent: newspaper
358Disallow: /
359
360# Archiver
361# https://knownagents.com/agents/nicecrawler
362
363User-agent: Nicecrawler
364Disallow: /
365
366# Scraper
367# https://knownagents.com/agents/nutch
368
369User-agent: Nutch
370Disallow: /
371
372# Scraper
373# https://knownagents.com/agents/nyt-scraping
374
375User-agent: nyt_scraping
376Disallow: /
377
378# Archiver
379# https://knownagents.com/agents/oddlybot
380
381User-agent: OddlyBot
382Disallow: /
383
384# Scraper
385# https://knownagents.com/agents/offline-explorer
386
387User-agent: Offline Explorer
388Disallow: /
389
390# Scraper
391# https://knownagents.com/agents/openindexspider
392
393User-agent: OpenindexSpider
394Disallow: /
395
396# Scraper
397# https://knownagents.com/agents/potions
398
399User-agent: Potions
400Disallow: /
401
402# AI Data Provider
403# https://knownagents.com/agents/quartr-crawler
404
405User-agent: Quartr Crawler
406Disallow: /
407
408# AI Data Provider
409# https://knownagents.com/agents/querit-searchbot
410
411User-agent: Querit-SearchBot
412Disallow: /
413
414# AI Data Provider
415# https://knownagents.com/agents/queritbot
416
417User-agent: QueritBot
418Disallow: /
419
420# Scraper
421# https://knownagents.com/agents/scrapy
422
423User-agent: Scrapy
424Disallow: /
425
426# AI Data Provider
427# https://knownagents.com/agents/scrybot
428
429User-agent: ScryBot
430Disallow: /
431
432# Scraper
433# https://knownagents.com/agents/serverhunterspider
434
435User-agent: ServerHunterSpider
436Disallow: /
437
438# AI Data Provider
439# https://knownagents.com/agents/shapbot
440
441User-agent: ShapBot
442Disallow: /
443
444# Archiver
445# https://knownagents.com/agents/smarshbot
446
447User-agent: SmarshBot
448Disallow: /
449
450# AI Data Provider
451# https://knownagents.com/agents/sofyabot
452
453User-agent: SofyaBot
454Disallow: /
455
456# Undocumented AI Agent
457# https://knownagents.com/agents/somethinggbot
458
459User-agent: SomethinggBot
460Disallow: /
461
462# Archiver
463# https://knownagents.com/agents/special-archiver
464
465User-agent: special_archiver
466Disallow: /
467
468# Scraper
469# https://knownagents.com/agents/statsdronebot
470
471User-agent: StatsDroneBot
472Disallow: /
473
474# AI Data Provider
475# https://knownagents.com/agents/talarionfrontiercrawler
476
477User-agent: TalarionFrontierCrawler
478Disallow: /
479
480# AI Data Provider
481# https://knownagents.com/agents/tavilybot
482
483User-agent: TavilyBot
484Disallow: /
485
486# AI Data Provider
487# https://knownagents.com/agents/terra-cotta
488
489User-agent: Terra Cotta
490Disallow: /
491
492# AI Data Provider
493# https://knownagents.com/agents/terracotta
494
495User-agent: TerraCotta
496Disallow: /
497
498# Scraper
499# https://knownagents.com/agents/thinkbot
500
501User-agent: Thinkbot
502Disallow: /
503
504# Archiver
505# https://knownagents.com/agents/turnitin
506
507User-agent: Turnitin
508Disallow: /
509
510# Scraper
511# https://knownagents.com/agents/web-image-collection-research
512
513User-agent: web-image-collection-research
514Disallow: /
515
516# Undocumented AI Agent
517# https://knownagents.com/agents/wrtnbot
518
519User-agent: WRTNBot
520Disallow: /
521
522# Undocumented AI Agent
523# https://knownagents.com/agents/xai-grok
524
525User-agent: xAI-Grok
526Disallow: /
527
528# Archiver
529# https://knownagents.com/agents/xy-archive-compliance
530
531User-agent: XY-Archive-Compliance
532Disallow: /
533
534# Archiver
535# https://knownagents.com/agents/xy-archive-compliance-archiver
536
537User-agent: XY-Archive-Compliance-Archiver
538Disallow: /
539
540# AI Data Provider
541# https://knownagents.com/agents/youbot
542
543User-agent: YouBot
544Disallow: /
545
546# Undocumented AI Agent
547# https://knownagents.com/agents/zenphonybot
548
549User-agent: ZenphonyBot
550Disallow: /
551# ---------------------------
552# END KNOWN AGENTS BLOCK
553
What do the bots actually get?
This page reads robots.txt, which states what adweek.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on adweek.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does adweek.com block GPTBot?
No. adweek.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does adweek.com block ClaudeBot?
No. adweek.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does adweek.com block Google-Extended?
No. adweek.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does adweek.com block PerplexityBot?
No. adweek.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find adweek.com?
Yes. adweek.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.