Who blocks AI · #944 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does nymag.com block AI crawlers?
Yes. nymag.com’s robots.txt blocks 15 of the 19 AI crawlers we track, including ClaudeBot, Google-Extended, PerplexityBot and 12 more. It also blocks PetalBot and Baiduspider among search engines.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- nymag.com/robots.txt · 562 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | line 59 · Allow: / under User-agent: Applebot |
PetalBot | Huawei | Blocked | line 420 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Blocked | line 120 · Disallow: / under User-agent: Baiduspider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Blocked | line 415 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 175 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 255 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Allowed | line 56 · Allow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 560 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | line 62 · Allow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 190 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 290 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 160 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 115 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 345 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 150 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 170 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 180 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 410 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 370 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 350 · Disallow: / under User-agent: meta-externalfetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 70 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Blocked | line 470 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View nymag.com/robots.txt · 562 lines
1User-agent: Mediapartners-Google
2Disallow:
3
4Sitemap: https://nymag.com/sitemap.xml
5Sitemap: https://nymag.com/sitemaps-cq/sitemap.xml
6Sitemap: https://nymag.com/news.xml
7Sitemap: https://nymag.com/strategist/sitemap.xml
8Sitemap: https://nymag.com/intelligencer/sitemap.xml
9Sitemap: https://nymag.com/fashion-sitemaps/fashionlabels-sitemap.xml
10Sitemap: https://nymag.com/fashion-sitemaps/fashionshows-sitemap.xml
11Sitemap: https://nymag.com/scienceofus-sitemaps/scienceofus-sitemap.xml
12
13User-agent: TruliaBot
14Disallow: /
15
16User-agent: Twitterbot
17Disallow: /amp/*
18Disallow: /intelligencer/amp/*
19Disallow: /strategist/amp/*
20
21# Facebook's link-preview crawler reads only its own group, so this allows it everything
22# (empty Disallow); gift links (?view_token=) must stay fetchable to unfurl on Facebook.
23User-agent: facebookexternalhit
24Disallow:
25
26User-agent: *
27Disallow: /coral-talk/
28Disallow: /daily/grubstreet/
29Disallow: /daily/southflorida/
30Disallow: /fashion/lookbook/38917/
31Disallow: /fashion/search/models/
32Disallow: /jesse/
33Disallow: /marketplace/
34Disallow: /news/intelligencer/16951/
35Disallow: /newsletter/
36Disallow: /*/newsletter/
37Disallow: /nymag/columns/intelligencer/features/16951/
38Disallow: /nymag/letters/14872/
39Disallow: /search.html
40Disallow: /search?*
41Disallow: /srch
42Disallow: /srch/
43Disallow: /strategist/search
44Disallow: /tags/newsletters/
45Disallow: /temp/
46Disallow: /test/
47Disallow: /urr/
48Disallow: *?origSession=*
49Disallow: *?q=*
50Disallow: *?view_token=*
51
52User-agent: Mediapartners-Google*
53Disallow:
54
55User-agent: Amazonbot
56Allow: /
57
58User-agent: Applebot
59Allow: /
60
61User-agent: GPTBot
62Allow: /
63
64User-agent: A6-Indexer
65Disallow: /
66Allow: /sp/
67Allow: /strategist
68
69User-agent: AhrefsBot
70Disallow: /
71Allow: /sp/
72Allow: /strategist
73
74User-agent: AI2Bot
75Disallow: /
76Allow: /sp/
77Allow: /strategist
78
79User-agent: AI2Bot-DeepResearchEval
80Disallow: /
81Allow: /sp/
82Allow: /strategist
83
84User-agent: Ai2Bot-Dolma
85Disallow: /
86Allow: /sp/
87Allow: /strategist
88
89User-agent: aiHitBot
90Disallow: /
91Allow: /sp/
92Allow: /strategist
93
94User-agent: Anchor Browser
95Disallow: /
96Allow: /sp/
97Allow: /strategist
98
99User-agent: anthropic-ai
100Disallow: /
101Allow: /sp/
102Allow: /strategist
103
104User-agent: ApifyBot
105Disallow: /
106Allow: /sp/
107Allow: /strategist
108
109User-agent: ApifyWebsiteContentCrawler
110Disallow: /
111Allow: /sp/
112Allow: /strategist
113
114User-agent: Applebot-Extended
115Disallow: /
116Allow: /sp/
117Allow: /strategist
118
119User-agent: Baiduspider
120Disallow: /
121Allow: /sp/
122Allow: /strategist
123
124User-agent: betaBot
125Disallow: /
126Allow: /sp/
127Allow: /strategist
128
129User-agent: BLEXBot
130Disallow: /
131Allow: /sp/
132Allow: /strategist
133
134User-agent: botify
135Disallow: /
136Allow: /sp/
137Allow: /strategist
138
139User-agent: brands-bot-logo
140Disallow: /
141Allow: /sp/
142Allow: /strategist
143
144User-agent: Brightbot
145Disallow: /
146Allow: /sp/
147Allow: /strategist
148
149User-agent: Bytespider
150Disallow: /
151Allow: /sp/
152Allow: /strategist
153
154User-agent: calculon
155Disallow: /
156Allow: /sp/
157Allow: /strategist
158
159User-agent: CCBot
160Disallow: /
161Allow: /sp/
162Allow: /strategist
163
164User-agent: ChatGLM-Spider
165Disallow: /
166Allow: /sp/
167Allow: /strategist
168
169User-agent: ChatGPT-User
170Disallow: /
171Allow: /sp/
172Allow: /strategist
173
174User-agent: Claude-SearchBot
175Disallow: /
176Allow: /sp/
177Allow: /strategist
178
179User-agent: Claude-User
180Disallow: /
181Allow: /sp/
182Allow: /strategist
183
184User-agent: Claude-Web
185Disallow: /
186Allow: /sp/
187Allow: /strategist
188
189User-agent: ClaudeBot
190Disallow: /
191Allow: /sp/
192Allow: /strategist
193
194User-agent: CloudVertexBot
195Disallow: /
196Allow: /sp/
197Allow: /strategist
198
199User-agent: cohere-ai
200Disallow: /
201Allow: /sp/
202Allow: /strategist
203
204User-agent: cohere-training-data-crawler
205Disallow: /
206Allow: /sp/
207Allow: /strategist
208
209User-agent: collection@infegy.com
210Disallow: /
211Allow: /sp/
212Allow: /strategist
213
214User-agent: Cotoyogi
215Disallow: /
216Allow: /sp/
217Allow: /strategist
218
219User-agent: Crawl4AI
220Disallow: /
221Allow: /sp/
222Allow: /strategist
223
224User-agent: CrunchBot
225Disallow: /
226Allow: /sp/
227Allow: /strategist
228
229User-agent: deepcrawl
230Disallow: /
231Allow: /sp/
232Allow: /strategist
233
234User-agent: DeepSeekBot
235Disallow: /
236Allow: /sp/
237Allow: /strategist
238
239User-agent: Diffbot
240Disallow: /
241Allow: /sp/
242Allow: /strategist
243
244User-agent: DirBuster
245Disallow: /
246Allow: /sp/
247Allow: /strategist
248
249User-agent: DTS
250Disallow: /
251Allow: /sp/
252Allow: /strategist
253
254User-agent: DuckAssistBot
255Disallow: /
256Allow: /sp/
257Allow: /strategist
258
259User-agent: EasouSpider
260Disallow: /
261Allow: /sp/
262Allow: /strategist
263
264User-agent: FacebookBot
265Disallow: /
266Allow: /sp/
267Allow: /strategist
268
269User-agent: FirecrawlAgent
270Disallow: /
271Allow: /sp/
272Allow: /strategist
273
274User-agent: FriendlyCrawler
275Disallow: /
276Allow: /sp/
277Allow: /strategist
278
279User-agent: GlutenCrawler
280Disallow: /
281Allow: /sp/
282Allow: /strategist
283
284User-agent: Google-CloudVertexBot
285Disallow: /
286Allow: /sp/
287Allow: /strategist
288
289User-agent: Google-Extended
290Disallow: /
291Allow: /sp/
292Allow: /strategist
293
294User-agent: GrapeshotCrawler
295Disallow: /
296Allow: /sp/
297Allow: /strategist
298
299User-agent: iaskspider
300Disallow: /
301Allow: /sp/
302Allow: /strategist
303
304User-agent: ICC-Crawler
305Disallow: /
306Allow: /sp/
307Allow: /strategist
308
309User-agent: ImagesiftBot
310Disallow: /
311Allow: /sp/
312Allow: /strategist
313
314User-agent: img2dataset
315Disallow: /
316Allow: /sp/
317Allow: /strategist
318
319User-agent: JackCrawl
320Disallow: /
321Allow: /sp/
322Allow: /strategist
323
324User-agent: Jetslide
325Disallow: /
326Allow: /sp/
327Allow: /strategist
328
329User-agent: linkapedia_bot
330Disallow: /
331Allow: /sp/
332Allow: /strategist
333
334User-agent: Mediatoolkitbot
335Disallow: /
336Allow: /sp/
337Allow: /strategist
338
339User-agent: MegaIndex
340Disallow: /
341Allow: /sp/
342Allow: /strategist
343
344User-agent: meta-externalagent
345Disallow: /
346Allow: /sp/
347Allow: /strategist
348
349User-agent: meta-externalfetcher
350Disallow: /
351Allow: /sp/
352Allow: /strategist
353
354User-agent: Microsoft BITS
355Disallow: /
356Allow: /sp/
357Allow: /strategist
358
359User-agent: Microsoft URL Control
360Disallow: /
361Allow: /sp/
362Allow: /strategist
363
364User-agent: Microsoft-WebDAV-MiniRedir
365Disallow: /
366Allow: /sp/
367Allow: /strategist
368
369User-agent: MistralAI-User
370Disallow: /
371Allow: /sp/
372Allow: /strategist
373
374User-agent: news-please
375Disallow: /
376Allow: /sp/
377Allow: /strategist
378
379User-agent: Novellum
380Disallow: /
381Allow: /sp/
382Allow: /strategist
383
384User-agent: NTENTbot
385Disallow: /
386Allow: /sp/
387Allow: /strategist
388
389User-agent: Nutch
390Disallow: /
391Allow: /sp/
392Allow: /strategist
393
394User-agent: omgili
395Disallow: /
396Allow: /sp/
397Allow: /strategist
398
399User-agent: Omgilibot
400Disallow: /
401Allow: /sp/
402Allow: /strategist
403
404User-agent: OpenLinkProfiler
405Disallow: /
406Allow: /sp/
407Allow: /strategist
408
409User-agent: Perplexity-User
410Disallow: /
411Allow: /sp/
412Allow: /strategist
413
414User-agent: PerplexityBot
415Disallow: /
416Allow: /sp/
417Allow: /strategist
418
419User-agent: PetalBot
420Disallow: /
421Allow: /sp/
422Allow: /strategist
423
424User-agent: Primalbot
425Disallow: /
426Allow: /sp/
427Allow: /strategist
428
429User-agent: python-requests
430Disallow: /
431Allow: /sp/
432Allow: /strategist
433
434User-agent: Re-Animator
435Disallow: /
436Allow: /sp/
437Allow: /strategist
438
439User-agent: Riddler
440Disallow: /
441Allow: /sp/
442Allow: /strategist
443
444User-agent: SafeDNSBot
445Disallow: /
446Allow: /sp/
447Allow: /strategist
448
449User-agent: SBIntuitionsBot
450Disallow: /
451Allow: /sp/
452Allow: /strategist
453
454User-agent: Scrapy
455Disallow: /
456Allow: /sp/
457Allow: /strategist
458
459User-agent: SearchAtlas
460Disallow: /
461Allow: /sp/
462Allow: /strategist
463
464User-agent: SemanticScholarBot
465Disallow: /
466Allow: /sp/
467Allow: /strategist
468
469User-agent: SemrushBot
470Disallow: /
471Allow: /sp/
472Allow: /strategist
473
474User-agent: SentiBot
475Disallow: /
476Allow: /sp/
477Allow: /strategist
478
479User-agent: Sogou
480Disallow: /
481Allow: /sp/
482Allow: /strategist
483
484User-agent: Spider-Pig
485Disallow: /
486Allow: /sp/
487Allow: /strategist
488
489User-agent: sqlmap
490Disallow: /
491Allow: /sp/
492Allow: /strategist
493
494User-agent: StorygizeBot
495Disallow: /
496Allow: /sp/
497Allow: /strategist
498
499User-agent: Test Certificate Info
500Disallow: /
501Allow: /sp/
502Allow: /strategist
503
504User-agent: TikTokSpider
505Disallow: /
506Allow: /sp/
507Allow: /strategist
508
509User-agent: Timpibot
510Disallow: /
511Allow: /sp/
512Allow: /strategist
513
514User-agent: Turnitin
515Disallow: /
516Allow: /sp/
517Allow: /strategist
518
519User-agent: uipbot
520Disallow: /
521Allow: /sp/
522Allow: /strategist
523
524User-agent: vBSEO
525Disallow: /
526Allow: /sp/
527Allow: /strategist
528
529User-agent: VegeBot
530Disallow: /
531Allow: /sp/
532Allow: /strategist
533
534User-agent: VelenPublicWebCrawler
535Disallow: /
536Allow: /sp/
537Allow: /strategist
538
539User-agent: Web Link Validator
540Disallow: /
541Allow: /sp/
542Allow: /strategist
543
544User-agent: WebCruiser
545Disallow: /
546Allow: /sp/
547Allow: /strategist
548
549User-agent: webzio-extended
550Disallow: /
551Allow: /sp/
552Allow: /strategist
553
554User-agent: Xenu
555Disallow: /
556Allow: /sp/
557Allow: /strategist
558
559User-agent: YouBot
560Disallow: /
561Allow: /sp/
562Allow: /strategist
What do the bots actually get?
This page reads robots.txt, which states what nymag.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on nymag.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does nymag.com block GPTBot?
No. nymag.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does nymag.com block ClaudeBot?
Yes. nymag.com’s robots.txt disallows ClaudeBot from the whole site (line 190: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does nymag.com block Google-Extended?
Yes. nymag.com’s robots.txt disallows Google-Extended from the whole site (line 290: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does nymag.com block PerplexityBot?
Yes. nymag.com’s robots.txt disallows PerplexityBot from the whole site (line 415: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find nymag.com?
Yes. nymag.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.