Who blocks AI · #411 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does nypost.com block AI crawlers?
Yes. nypost.com’s robots.txt blocks 11 of the 19 AI crawlers we track, including ClaudeBot, Google-Extended, PerplexityBot and 8 more. It also blocks PetalBot among search engines.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- nypost.com/robots.txt · 358 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 285 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Blocked | line 162 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 135 · Disallow: / under User-agent: claude-searchbot |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 216 · Disallow: / under User-agent: amazonbot |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Blocked | line 144 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 150 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 132 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 228 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 195 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 129 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Blocked | line 138 · Disallow: / under User-agent: claude-user |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Blocked | line 198 · Disallow: / under User-agent: meta-externalfetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 207 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View nypost.com/robots.txt · 358 lines
1# NOTICE: Collection of content and other data on nypost.com through
2# automated means is prohibited unless you have express written
3# permission from NYP Holdings, Inc. and may only be conducted for the
4# limited purpose contained in said permission.
5#
6# NYP Holdings, Inc. Terms of Use may be found at
7# https://nypost.com/terms/
8#
9# If you would like to apply for permission to license the
10# intellectual property and/or other materials of NYP Holdings, Inc.'s
11# brands, please contact us at ip-permissions@nypost.com.
12
13User-agent: *
14Disallow: /wp-admin/
15Allow: /wp-admin/admin-ajax.php
16Disallow: /wp-json/
17Disallow: /wp-login.php
18User-agent: *
19Disallow: /tag/credible/
20Disallow: /personal-finance/
21Disallow: /banking/
22Disallow: /credit-cards/
23Disallow: /loans/
24Disallow: /personal-loans/
25Disallow: /refinance-student-loans/
26Disallow: /student-loans/
27Disallow: /mortgages/
28Disallow: /home-equity/
29Disallow: /mortgage-rates/
30Disallow: /mortgage-refinance/
31Disallow: /account/
32Allow: /account/subscribe
33
34User-agent: *
35Disallow: /search/
36Disallow: /horoscope/
37Disallow: /*?x=
38Disallow: /*?action=
39Disallow: /*&action=
40Disallow: /*?adid=
41Disallow: /*&adid=
42Disallow: /*?bitrate=
43Disallow: /*&bitrate=
44Disallow: /*?channel=
45Disallow: /*&channel=
46Disallow: /*?clipid=
47Disallow: /*&clipid=
48Disallow: /*?contentCollection=
49Disallow: /*&contentCollection=
50Disallow: /*?contentId=
51Disallow: /*&contentId=
52Disallow: /*?ei=
53Disallow: /*&ei=
54Disallow: /*?format=
55Disallow: /*&format=
56Disallow: /*?height=
57Disallow: /*&height=
58Disallow: /*?iframe=
59Disallow: /*&iframe=
60Disallow: /*?mediaId=
61Disallow: /*&mediaId=
62Disallow: /*?module=
63Disallow: /*&module=
64Disallow: /*?monthnum=
65Disallow: /*&monthnum=
66Disallow: /*?pgtype=
67Disallow: /*&pgtype=
68Disallow: /*?preview=
69Disallow: /*&preview=
70Disallow: /*?priority=
71Disallow: /*&priority=
72Disallow: /*?ref=
73Disallow: /*&ref=
74Disallow: /*?referrer=
75Disallow: /*&referrer=
76Disallow: /*?sa=
77Disallow: /*&sa=
78Disallow: /*?section=
79Disallow: /*§ion=
80Disallow: /*?share=
81Disallow: /*&share=
82Disallow: /*?TB_iframe=
83Disallow: /*&TB_iframe=
84Disallow: /*?TBL_Prod100Page=
85Disallow: /*&TBL_Prod100Page=
86Disallow: /*?theme_preview=
87Disallow: /*&theme_preview=
88Disallow: /*?usg=
89Disallow: /*&usg=
90Disallow: /*?ved=
91Disallow: /*&ved=
92Disallow: /*?version=
93Disallow: /*&version=
94Disallow: /*?VID=
95Disallow: /*&VID=
96Disallow: /*?vxBitrate=
97Disallow: /*&vxBitrate=
98Disallow: /*?vxChannel=
99Disallow: /*&vxChannel=
100Disallow: /*?vxClipId=
101Disallow: /*&vxClipId=
102Disallow: /*?vxSiteId=
103Disallow: /*&vxSiteId=
104Disallow: /*?width=
105Disallow: /*&width=
106Disallow: /*?year=
107Disallow: /*&year=
108Disallow: /*.xml?uuid=
109Disallow: /*?post_type=feedback&p=
110Disallow: /*?wptouch_preview_theme=
111Disallow: /*>
112Disallow: /?attachment_id=
113Disallow: /affiliate/
114Disallow: /ai_log/
115Disallow: /*?*pg99.asia*
116Disallow: /*&*pg99.asia*
117Disallow: /*?*gg155.cn*
118Disallow: /*&*gg155.cn*
119Disallow: /*?*To66.Asia*
120Disallow: /*&*To66.Asia*
121
122User-agent: *
123Disallow: /new-york-post-instagram-feed/
124Disallow: /new-york-post-tiktok-feed/
125Disallow: /alexa-new-york-post-instagram-feed/
126Disallow: /alexa-new-york-post-tiktok-feed/
127
128User-agent: Bytespider
129Disallow: /
130
131User-agent: CCBot
132Disallow: /
133
134User-agent: claude-searchbot
135Disallow: /
136
137User-agent: claude-user
138Disallow: /
139
140User-agent: Claude-Web
141Disallow: /
142
143User-agent: ClaudeBot
144Disallow: /
145
146User-agent: FacebookBot
147Disallow: /
148
149User-agent: Google-Extended
150Disallow: /
151
152User-agent: MJ12bot
153Disallow: /
154
155User-agent: NewsNow
156Disallow: /
157
158User-agent: Perplexity-ai
159Disallow: /
160
161User-agent: PerplexityBot
162Disallow: /
163
164User-agent: PiplBot
165Disallow: /
166
167User-agent: Shap-User
168Disallow: /
169
170User-agent: ShapBot
171Disallow: /
172
173User-agent: anthropic-ai
174Disallow: /
175
176User-agent: cohere-ai
177Disallow: /
178
179User-agent: ia_archiver
180Disallow: /
181
182User-agent: magpie-crawler
183Disallow: /
184
185User-agent: news-please
186Disallow: /
187
188User-agent: omgili
189Disallow: /
190
191User-agent: omgilibot
192Disallow: /
193
194User-agent: meta-externalagent
195Disallow: /
196
197User-agent: meta-externalfetcher
198Disallow: /
199
200User-agent: TaraGroup Intelligent Bot
201Disallow: /
202
203User-agent: A6-Indexer
204Disallow: /
205
206User-agent: AhrefsBot
207Disallow: /
208
209User-agent: Amazon-kendra
210Disallow: /
211
212User-agent: Amazon-qbusiness
213Disallow: /
214
215User-agent: amazonbot
216Disallow: /
217
218User-agent: amzn-searchbot
219Disallow: /
220
221User-agent: amzn-user
222Disallow: /
223
224User-agent: anthropic-ai
225Disallow: /
226
227User-agent: Applebot-Extended
228Disallow: /
229
230User-agent: archive.org_bot
231Disallow: /
232
233User-agent: AwarioRssBot
234Disallow: /
235
236User-agent: AwarioSmartBot
237Disallow: /
238
239User-agent: BLP_bbot
240Disallow: /
241
242User-agent: claritybot
243Disallow: /
244
245User-agent: cohere-ai
246Disallow: /
247
248User-agent: DataForSeoBot
249Disallow: /
250
251User-agent: Diffbot
252Disallow: /
253
254User-agent: EasouSpider
255Disallow: /
256
257User-agent: ecoResearch
258Disallow: /
259
260User-agent: FriendlyCrawler
261Disallow: /
262
263User-agent: Genieo
264Disallow: /
265
266User-agent: ImagesiftBot
267Disallow: /
268
269User-agent: Meltwater
270Disallow: /
271
272User-agent: Meta-ExternalAgent
273Disallow: /
274
275User-agent: meta-externalagent
276Disallow: /
277
278User-agent: peer39_crawler
279Disallow: /
280
281User-agent: peer39_crawler/1.0
282Disallow: /
283
284User-agent: PetalBot
285Disallow: /
286
287User-agent: psbot
288Disallow: /
289
290User-agent: R6_CommentReader
291Disallow: /
292
293User-agent: Scrapy
294Disallow: /
295
296User-agent: Seekr
297Disallow: /
298
299User-agent: Spinn3r
300Disallow: /
301
302User-agent: TurnitinBot
303Disallow: /
304
305User-agent: WebVac
306Disallow: /
307
308User-agent: WebZip
309Disallow: /
310
311Disallow: /5850/
312# Sitemap archive
313Sitemap: https://nypost.com/sitemap-1865.xml
314Sitemap: https://nypost.com/sitemap-1999.xml
315Sitemap: https://nypost.com/sitemap-2000.xml
316Sitemap: https://nypost.com/sitemap-2001.xml
317Sitemap: https://nypost.com/sitemap-2002.xml
318Sitemap: https://nypost.com/sitemap-2003.xml
319Sitemap: https://nypost.com/sitemap-2004.xml
320Sitemap: https://nypost.com/sitemap-2005.xml
321Sitemap: https://nypost.com/sitemap-2006.xml
322Sitemap: https://nypost.com/sitemap-2007.xml
323Sitemap: https://nypost.com/sitemap-2008.xml
324Sitemap: https://nypost.com/sitemap-2009.xml
325Sitemap: https://nypost.com/sitemap-2010.xml
326Sitemap: https://nypost.com/sitemap-2011.xml
327Sitemap: https://nypost.com/sitemap-2012.xml
328Sitemap: https://nypost.com/sitemap-2013.xml
329Sitemap: https://nypost.com/sitemap-2014.xml
330Sitemap: https://nypost.com/sitemap-2015.xml
331Sitemap: https://nypost.com/sitemap-2016.xml
332Sitemap: https://nypost.com/sitemap-2017.xml
333Sitemap: https://nypost.com/sitemap-2018.xml
334Sitemap: https://nypost.com/sitemap-2019.xml
335Sitemap: https://nypost.com/sitemap-2020.xml
336Sitemap: https://nypost.com/sitemap-2021.xml
337Sitemap: https://nypost.com/sitemap-2022.xml
338Sitemap: https://nypost.com/sitemap-2023.xml
339Sitemap: https://nypost.com/sitemap-2024.xml
340Sitemap: https://nypost.com/sitemap-2025.xml
341Sitemap: https://nypost.com/sitemap-2026.xml
342
343
344User-agent: FontReport
345Allow: /
346
347# Sitemaps
348Sitemap: https://nypost.com/news-sitemap.xml
349
350# Additional sitemaps
351Sitemap: https://nypost.com/sitemap-nypost-edition.xml
352Sitemap: https://nypost.com/sitemap-nypost-section.xml
353Sitemap: https://nypost.com/sitemap-nypost-post_tag.xml
354Sitemap: https://nypost.com/sitemap-nypost-authors.xml
355Sitemap: https://nypost.com/sitemap-nypost-pages.xml
356Sitemap: https://nypost.com/sitemap-nypost-sportstat.xml
357
358
What do the bots actually get?
This page reads robots.txt, which states what nypost.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on nypost.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does nypost.com block GPTBot?
No. nypost.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does nypost.com block ClaudeBot?
Yes. nypost.com’s robots.txt disallows ClaudeBot from the whole site (line 144: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does nypost.com block Google-Extended?
Yes. nypost.com’s robots.txt disallows Google-Extended from the whole site (line 150: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does nypost.com block PerplexityBot?
Yes. nypost.com’s robots.txt disallows PerplexityBot from the whole site (line 162: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find nypost.com?
Yes. nypost.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.