Who blocks AI · #992 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does nydailynews.com block AI crawlers?
Yes. nydailynews.com’s robots.txt blocks 18 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 15 more. It also blocks Applebot, PetalBot and 2 more among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.nydailynews.com/robots.txt · 230 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Blocked | line 36 · Disallow: / under User-agent: Applebot |
PetalBot | Huawei | Blocked | line 165 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Blocked | line 195 · Disallow: / under User-agent: YandexBot |
Baiduspider | Baidu | Blocked | line 45 · Disallow: / under User-agent: Baiduspider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 150 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 159 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 69 · Disallow: / under User-agent: Claude-Searchbot |
DuckAssistBot | DuckDuckGo | Blocked | line 96 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 27 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 198 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 108 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 66 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 105 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 57 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 39 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 135 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 51 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 63 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 72 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 162 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 144 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 138 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.nydailynews.com/robots.txt · 230 lines
1# Our Content is made available for your personal, non-commercial use subject to our Terms of Service here: https://www.tribpub.com/terms-and-conditions/
2# Any other uses are prohibited, including but not limited to:
3# (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in the Digital Single Market;
4# (2) use of any Content or information available on the Site for purposes of retrieval augmented generation, grounding, training, or development of machine learning models, algorithms, or artificial intelligence (AI) systems, or to generate substitute content or develop any products, services, or technology;
5# (3) caching or archiving the Content; and/or
6# (4) any commercial purposes.
7# Contact us at https://www.nydailynews.com/contact-us/ for assistance.
8
9User-agent: *
10Disallow: /wp-admin/
11Allow: /wp-admin/admin-ajax.php
12Disallow: /cgi-bin/
13Disallow: /wp-includes/
14Disallow: /xmlrpc.php
15Disallow: /wp-content/plugins/
16Disallow: /wp-content/cache/
17Disallow: /trackback/
18Disallow: /comments/
19
20User-agent: AI2Bot
21Disallow: /
22
23User-agent: Ai2Bot-Dolma
24Disallow: /
25
26User-agent: Amazonbot
27Disallow: /
28
29User-agent: Amzn-SearchBot
30Disallow: /
31
32User-agent: anthropic-ai
33Disallow: /
34
35User-agent: Applebot
36Disallow: /
37
38User-agent: Applebot-Extended
39Disallow: /
40
41User-agent: Archive.org_bot
42Disallow: /
43
44User-agent: Baiduspider
45Disallow: /
46
47User-agent: Brightbot 1.0
48Disallow: /
49
50User-Agent: Bytespider
51Disallow: /
52
53User-agent: carbon-umbrella-bot
54Disallow: /
55
56User-agent: CCBot
57Disallow: /
58
59User-agent: ChatGLM-Spider
60Disallow: /
61
62User-agent: ChatGPT-User
63Disallow: /
64
65User-agent: ClaudeBot
66Disallow: /
67
68User-agent: Claude-Searchbot
69Disallow: /
70
71User-agent: Claude-User
72Disallow: /
73
74User-agent: Claude-Web
75Disallow: /
76
77User-agent: Cohere-training-data-crawler
78Disallow: /
79
80User-agent: Cotoyogi
81Disallow: /
82
83User-agent: Dalvik
84Disallow: /
85
86User-agent: Datenbank Crawler
87Disallow: /
88
89User-agent: Diffbot
90Disallow: /
91
92User-agent: DocSpotBot
93Disallow: /
94
95User-agent: DuckAssistBot
96Disallow: /
97
98User-agent: FacebookBot
99Disallow: /
100
101User-agent: GnowitNewsbot
102Disallow: /
103
104User-agent: Google-Extended
105Disallow: /
106
107User-agent: GPTBot
108Disallow: /
109
110User-agent: grok-crawler
111Disallow: /
112
113User-agent: ia_archiver-web.archive.org
114Disallow: /
115
116User-agent: ICC-Crawler
117Disallow: /
118
119User-agent: imageSpider
120Disallow: /
121
122User-agent: Kangaroo Bot
123Disallow: /
124
125User-agent: laion-huggingface-processor
126Disallow: /
127
128User-agent: LCC
129Disallow: /
130
131User-agent: Mantisbot
132Disallow: /
133
134User-agent: Meta-ExternalAgent
135Disallow: /
136
137User-agent: Meta-ExternalFetcher
138Disallow: /
139
140User-agent: Meta-WebIndexer
141Disallow: /
142
143User-agent: MistralAI-User
144Disallow: /
145
146User-agent: netEstate Imprint Crawler
147Disallow: /
148
149User-agent: OAI-SearchBot
150Disallow: /
151
152User-agent: omgili
153Disallow: /
154
155User-agent: PanguBot
156Disallow: /
157
158User-agent: PerplexityBot
159Disallow: /
160
161User-agent: Perplexity-User
162Disallow: /
163
164User-agent: PetalBot
165Disallow: /
166
167User-agent: SBIntuitionsBot
168Disallow: /
169
170User-agent: SirdataBot
171Disallow: /
172
173User-agent: Slurp
174Crawl-delay: 10
175
176User-agent: Spider
177Disallow: /
178
179User-agent: Thinkbot
180Disallow: /
181
182User-agent: Timpibot
183Disallow: /
184
185User-agent: VelenPublicWebCrawler
186Disallow: /
187
188User-agent: Vivaldi
189Disallow: /
190
191User-agent: Webzio-extended
192Disallow: /
193
194User-agent: YandexBot
195Disallow: /
196
197User-agent: YouBot
198Disallow: /
199
200User-agent: AmazonAdBot
201Disallow:
202
203User-agent: facebookexternalhit
204Disallow:
205
206User-agent: grapeshot
207Disallow:
208
209User-agent: Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/w4mwnpbXf3MFAbxOkJRw; +http://www.echobox.com)
210Disallow:
211
212User-agent: Twitterbot
213Disallow:
214
215User-agent: Mozilla/5.0 (compatible; NewsRoom.BI/0.1; +http://www.newsroom.bi/bot.html)
216Allow: /
217User-agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Booldog/1.0)
218Allow: /
219User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36(KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 (compatible; mrfCompass-Booldog/1.0)
220Allow: /
221User-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; mrfCompass-Marshall/1.0)
222Allow: /
223User-agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA51N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Mobile Safari/537.36 (compatible; mrfCompass-Jukebox/1.0)
224Allow: /
225# Sitemap archive
226Sitemap: https://www.nydailynews.com/sitemap.xml
227
228Sitemap: https://www.nydailynews.com/image-sitemap.xml
229
230
What do the bots actually get?
This page reads robots.txt, which states what nydailynews.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on nydailynews.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
- 2 November 2026: started blocking Meta-ExternalFetcher.
Questions
Does nydailynews.com block GPTBot?
Yes. nydailynews.com’s robots.txt disallows GPTBot from the whole site (line 108: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does nydailynews.com block ClaudeBot?
Yes. nydailynews.com’s robots.txt disallows ClaudeBot from the whole site (line 66: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does nydailynews.com block Google-Extended?
Yes. nydailynews.com’s robots.txt disallows Google-Extended from the whole site (line 105: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does nydailynews.com block PerplexityBot?
Yes. nydailynews.com’s robots.txt disallows PerplexityBot from the whole site (line 159: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find nydailynews.com?
Not through its own crawler. nydailynews.com’s robots.txt disallows OAI-SearchBot from the whole site (line 150: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.