Who blocks AI · #69 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does nytimes.com block AI crawlers?
Yes. nytimes.com’s robots.txt blocks 16 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 13 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.nytimes.com/robots.txt · 386 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 17 · Disallow: /ads/ line 18 · Disallow: /adx/bin/ and 91 more | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Partly blocked | line 17 · Disallow: /ads/ line 18 · Disallow: /adx/bin/ and 91 more |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 266 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 279 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 183 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 204 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Partly blocked | line 146 · Disallow: /wirecutter/ |
YouBot | You.com | Blocked | line 318 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 233 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 180 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 230 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 174 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 152 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 246 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 171 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 177 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 186 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 282 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Blocked | line 250 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.nytimes.com/robots.txt · 386 lines
1# New York Times content is made available for your personal, non-commercial
2# use subject to our Terms of Service here:
3# https://help.nytimes.com/hc/en-us/articles/115014893428-Terms-of-Service.
4# Use of any device, tool, or process designed to data mine or scrape the content
5# using automated means is prohibited without prior written permission from
6# The New York Times Company. Prohibited uses include but are not limited to:
7# (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in
8# the Digital Single Market;
9# (2) the development of any software, machine learning, artificial intelligence (AI),
10# and/or large language models (LLMs);
11# (3) creating or providing archived or cached data sets containing our content to others; and/or
12# (4) any commercial purposes.
13# Contact https://nytlicensing.com/contact/ for assistance.
14
15User-agent: *
16User-agent: Googlebot
17Disallow: /ads/
18Disallow: /adx/bin/
19Disallow: /athletic/wp/wp-admin/
20Allow: /athletic/wp/wp-admin/admin-ajax.php
21Disallow: /athletic/async-*
22Disallow: /athletic/search/*
23Allow: /athletic/search/$
24Disallow: /athletic/checkout/
25Disallow: /athletic/checkout?plan_id*
26Allow: /athletic/checkout/$
27Disallow: /athletic/checkout2*
28Disallow: /athletic/login/
29Disallow: /athletic/login?login_source*
30Disallow: /athletic/login?ref_page*
31Allow: /athletic/login/$
32Disallow: /athletic/login2/
33Disallow: /athletic/login2?login_source*
34Disallow: /athletic/login2?ref_page*
35Allow: /athletic/login2/$
36Disallow: /athletic/report/
37Disallow: /athletic/*/discuss/*
38Allow: /athletic/live-blogs/discuss/
39Allow: /athletic/nfl/game/discuss/
40Disallow: /athletic/register/
41Disallow: /athletic/register?welcome_redirect*
42Disallow: /athletic/register2/
43Disallow: /athletic/register2?welcome_redirect*
44Disallow: /athletic/betmgm-redirect*
45Disallow: /athletic/cdn-cgi/
46Disallow: /athletic/verizon/*
47Disallow: /athletic/forgot-password/*
48Disallow: /athletic/forgot-password2/*
49Disallow: /athletic/amp-social-login*
50Disallow: /athletic/track-analytics/
51Disallow: /athletic/amp-auth/
52Disallow: /athletic/rss-feed/
53Disallow: /athletic/*?*rss=1
54Disallow: /athletic/global-color-test.php
55Disallow: /athletic/global-font-test.php
56Disallow: /athletic/graphql*
57Disallow: /athletic/api*
58Disallow: /athletic/ip*
59Disallow: /athletic/call-set-cookie-with-context/*
60Disallow: /athletic/get-current-user/
61Disallow: /athletic/pv.json
62Disallow: /athletic/following-feed-test/*
63Disallow: /athletic*/boxscore/*
64Disallow: /athletic/feed-test/
65Disallow: /athletic*/signed-mp3-redirect-url/*
66Disallow: /athletic/embedded-interactive/*
67Disallow: /card/panel/
68Disallow: /panel/
69Disallow: /puzzles/leaderboards/invite/*
70Disallow: /svc
71Allow: /svc/crosswords
72Allow: /svc/games
73Allow: /svc/letter-boxed
74Allow: /svc/spelling-bee
75Allow: /svc/wordle
76Allow: /svc/connections
77Allow: /svc/sudoku
78Allow: /svc/strands
79Allow: /svc/pips
80Disallow: /video/embedded/*
81Disallow: /search
82Disallow: /multiproduct/
83Disallow: /hd/
84Disallow: /inyt/
85Disallow: /*?*query=
86Disallow: /*.pdf$
87Disallow: /*?*login=
88Disallow: /*?*campaignId=
89Disallow: /*?*mcubz=
90Disallow: /*?*smprod=
91Disallow: /*?*ProfileID=
92Disallow: /*?*ListingID=
93Disallow: /*?*campaign_id=
94Disallow: /*?*hybrid=
95Disallow: /*?*entry=
96Disallow: /*?*embed=
97Disallow: /*?ls=
98Disallow: /*?*&ls=
99Disallow: /v1/initialize*
100Disallow: /v1/rgstr*
101Disallow: /wirecutter/wp-admin/
102Disallow: /wirecutter/*.zip$
103Disallow: /wirecutter/*.csv$
104Disallow: /wirecutter/deals/beta
105Disallow: /wirecutter/data-requests
106Disallow: /wirecutter/search
107Disallow: /wirecutter/*?s=
108Disallow: /wirecutter/*&xid=
109Disallow: /wirecutter/*?q=
110Disallow: /wirecutter/*?l=
111Disallow: /wirecutter/*?merchant=
112Disallow: /wirecutter/out/
113Disallow: /search
114Disallow: /subscription/*?*source=
115Disallow: /subscription/*?*onboarded=
116Disallow: /*?*smid=
117Disallow: /*?*partner=
118Disallow: /*?*utm_source=
119Allow: /wirecutter/*?*utm_source=
120Allow: /ads/public/
121Allow: /svc/news/v3/all/pshb.rss
122Allow: /wirecutter/reviews/*?*utm_source=
123Allow: /wirecutter/blog/*?*utm_source=
124Allow: /wirecutter/lists/*?*utm_source=
125Allow: /wirecutter/gifts/*?*utm_source=
126
127
128# Googlebot Specific Rules
129
130User-agent: Googlebot
131Disallow: /athletic*adgroupid*
132Disallow: /athletic*campaignid*
133Disallow: /athletic*ad_id*
134Disallow: /athletic*access_token*
135Disallow: /athletic*amp_reader_id*
136Disallow: /athletic*/?source=*
137Disallow: /athletic/*?*embed=1
138
139
140# Disallow Rules
141
142User-agent: AliyunSecBot
143Disallow: /
144
145User-agent: Amazonbot
146Disallow: /wirecutter/
147
148User-agent: anthropic-ai
149Disallow: /
150
151User-agent: Applebot-Extended
152Disallow: /
153
154User-agent: archive.org_bot
155Disallow: /
156
157User-agent: AudigentAdBot
158Disallow: /
159
160User-agent: AwarioRssBot
161User-agent: AwarioSmartBot
162Disallow: /
163
164User-agent: BLEXBot
165Disallow: /
166
167User-agent: Brightbot
168Disallow: /
169
170User-agent: Bytespider
171Disallow: /
172
173User-agent: CCBot
174Disallow: /
175
176User-agent: ChatGPT-User
177Disallow: /
178
179User-agent: ClaudeBot
180Disallow: /
181
182User-agent: Claude-SearchBot
183Disallow: /
184
185User-agent: Claude-User
186Disallow: /
187
188User-agent: Claude-Web
189Disallow: /
190
191User-agent: cohere-ai
192Disallow: /
193
194User-agent: DataForSeoBot
195Disallow: /
196
197User-agent: Diffbot
198Disallow: /
199
200User-agent: Diffbot-User
201Disallow: /
202
203User-agent: DuckAssistBot
204Disallow: /
205
206User-agent: EchoboxBot
207Disallow: /
208
209User-agent: ExaBot
210Disallow: /
211
212User-agent: ExaSearchBot
213Disallow: /
214
215User-agent: FacebookBot
216Disallow: /
217
218User-agent: FirecrawlAgent
219Disallow: /
220
221User-agent: FriendlyCrawler
222Disallow: /
223
224User-agent: Google-CloudVertexBot
225Disallow: /
226Allow: /wirecutter/
227Allow: /athletic/
228
229User-agent: Google-Extended
230Disallow: /
231
232User-agent: GPTBot
233Disallow: /
234
235User-agent: ImagesiftBot
236Disallow: /
237
238User-agent: Jetslide
239Disallow: /
240
241User-agent: magpie-crawler
242Disallow: /
243
244User-agent: Meta-ExternalAgent
245User-agent: meta-externalagent
246Disallow: /
247
248User-agent: Meta-ExternalFetcher
249User-agent: meta-externalfetcher
250Disallow: /
251
252User-agent: Meta-WebIndexer
253User-agent: meta-webindexer/1.1
254Disallow: /
255
256User-agent: MyCentralAIScraperBot
257Disallow: /
258
259User-agent: NewsNow
260Disallow: /
261
262User-agent: news-please
263Disallow: /
264
265User-agent: OAI-SearchBot
266Disallow: /
267
268User-agent: omgili
269Disallow: /
270
271User-agent: omgilibot
272Disallow: /
273
274User-agent: peer39_crawler
275User-agent: peer39_crawler/1.0
276Disallow: /
277
278User-agent: PerplexityBot
279Disallow: /
280
281User-agent: Perplexity-User
282Disallow: /
283
284User-agent: Poseidon Research Crawler
285Disallow: /
286
287User-agent: quillbot.com
288Disallow: /
289
290User-agent: Quora-Bot
291Disallow: /
292
293User-agent: Scrapy
294Disallow: /
295
296User-agent: SeekrBot
297Disallow: /
298
299User-agent: SeznamHomepageCrawler
300Disallow: /
301
302User-agent: ShapBot
303Disallow: /
304
305User-agent: TaraGroup Intelligent Bot
306Disallow: /
307
308User-agent: Timpibot
309Disallow: /
310
311User-agent: TurnitinBot
312Disallow: /
313
314User-agent: ViennaTinyBot
315Disallow: /
316
317User-agent: YouBot
318Disallow: /
319
320
321# Ad Bot Rules
322
323User-agent: AmazonAdBot
324Allow: /
325
326
327# Social Bot Rules
328
329User-agent: facebookexternalhit
330Allow: /*?*smid=
331
332User-agent: Facebot
333Allow: /*?*smid=
334
335User-agent: Twitterbot
336Allow: /*?*smid=
337
338User-agent: RedditBot
339Allow: /*?*smid=
340
341User-agent: LinkedInBot
342Allow: /*?*smid=
343
344User-agent: Bluesky
345Allow: /*?*smid=
346
347User-agent: TelegramBot
348Allow: /*?*smid=
349
350User-agent: WhatsApp
351Allow: /*?*smid=
352
353User-agent: Discordbot
354Allow: /*?*smid=
355
356User-agent: Mastodon
357Allow: /*?*smid=
358
359# Sitemaps
360
361Sitemap: https://www.nytimes.com/sitemaps/new/news.xml.gz
362Sitemap: https://www.nytimes.com/sitemaps/new/sitemap.xml.gz
363Sitemap: https://www.nytimes.com/sitemaps/new/collections.xml.gz
364Sitemap: https://www.nytimes.com/sitemaps/new/video.xml.gz
365Sitemap: https://www.nytimes.com/sitemaps/new/cooking.xml.gz
366Sitemap: https://www.nytimes.com/sitemaps/new/recipe-collects.xml.gz
367Sitemap: https://www.nytimes.com/sitemaps/new/regions.xml
368Sitemap: https://www.nytimes.com/sitemaps/new/best-sellers.xml
369Sitemap: https://www.nytimes.com/sitemaps/new/subscription-landing-pages.xml
370Sitemap: https://www.nytimes.com/sitemaps/new/weather.xml.gz
371Sitemap: https://www.nytimes.com/sitemaps/new/espanol.xml.gz
372Sitemap: https://www.nytimes.com/sitemaps/new/espanol-collects.xml.gz
373Sitemap: https://www.nytimes.com/wirecutter/sitemapindex.xml
374Sitemap: https://www.nytimes.com/athletic/sitemap-videos-index.xml
375Sitemap: https://www.nytimes.com/athletic/sitemap-authors.xml
376Sitemap: https://www.nytimes.com/athletic/sitemap-verticals.xml
377Sitemap: https://www.nytimes.com/athletic/sitemap-teams.xml
378Sitemap: https://www.nytimes.com/athletic/sitemap-cities.xml
379Sitemap: https://www.nytimes.com/athletic/sitemap-players.xml
380Sitemap: https://www.nytimes.com/athletic/sitemap-tags.xml
381Sitemap: https://www.nytimes.com/athletic/sitemap-stats.xml
382Sitemap: https://www.nytimes.com/athletic/sitemap-schedule.xml
383Sitemap: https://www.nytimes.com/athletic/sitemap-roster.xml
384Sitemap: https://www.nytimes.com/athletic/sitemap.xml
385Sitemap: https://www.nytimes.com/games-assets/v2/assets/sitemap/games.xml
386
What do the bots actually get?
This page reads robots.txt, which states what nytimes.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on nytimes.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does nytimes.com block GPTBot?
Yes. nytimes.com’s robots.txt disallows GPTBot from the whole site (line 233: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does nytimes.com block ClaudeBot?
Yes. nytimes.com’s robots.txt disallows ClaudeBot from the whole site (line 180: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does nytimes.com block Google-Extended?
Yes. nytimes.com’s robots.txt disallows Google-Extended from the whole site (line 230: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does nytimes.com block PerplexityBot?
Yes. nytimes.com’s robots.txt disallows PerplexityBot from the whole site (line 279: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find nytimes.com?
Not through its own crawler. nytimes.com’s robots.txt disallows OAI-SearchBot from the whole site (line 266: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.