Who blocks AI · #480 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does elpais.com block AI crawlers?
Yes. elpais.com’s robots.txt blocks 10 of the 19 AI crawlers we track, including ClaudeBot, PerplexityBot, CCBot and 7 more. It also blocks PetalBot and Baiduspider among search engines.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- elpais.com/robots.txt · 526 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 2 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Blocked | line 87 · Disallow: / under User-agent: Baiduspider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Blocked | line 201 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 42 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 66 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 12 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Blocked | line 36 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Blocked | line 60 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Blocked | line 30 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Blocked | line 39 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 204 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 75 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 333 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View elpais.com/robots.txt · 526 lines
1User-agent: PetalBot
2Disallow: /
3
4# Bloque bots global confirmado 20260505
5User-agent: bedrock-agentcore-browser
6Disallow: /
7
8User-agent: Amazon-Bot
9Disallow: /
10
11User-agent: Amazonbot
12Disallow: /
13
14User-agent: AmazonProductDiscoveryBot
15Disallow: /
16
17User-agent: Amzn-SearchBot
18Disallow: /
19
20User-agent: Amzn-User
21Disallow: /
22
23User-agent: Arquivo-web-crawler
24Disallow: /
25
26User-agent: bnf.fr_bot
27Disallow: /
28
29User-agent: Bytespider
30Disallow: /
31
32User-agent: Claude-Extension
33Disallow: /
34
35User-agent: ClaudeBot
36Disallow: /
37
38User-agent: Claude-User
39Disallow: /
40
41User-agent: Claude-SearchBot
42Disallow: /
43
44User-agent: Comet
45Disallow: /
46
47User-agent: CloudFlare-AlwaysOnline
48Disallow: /
49
50User-agent: Cloudflare-Healthchecks
51Disallow: /
52
53User-agent: CloudflareBrowserRendering
54Disallow: /
55
56User-agent: Cloudflare-Radar
57Disallow: /
58
59User-agent: CCBot
60Disallow: /
61
62User-agent: DeepSeekBot
63Disallow: /
64
65User-agent: DuckAssistBot
66Disallow: /
67
68User-agent: archive.org_bot
69Disallow: /
70
71User-agent: ia_archiver
72Disallow: /
73
74User-agent: MistralAI-User
75Disallow: /
76
77User-agent: Moreover
78Disallow: /
79
80User-agent: Adbeat
81Disallow: /
82
83User-agent: Argus
84Disallow: /
85
86User-agent: Baiduspider
87Disallow: /
88
89User-agent: Baiduspider-image
90Disallow: /
91
92User-agent: Bitdefender
93Disallow: /
94
95User-agent: Blogtrottr
96Disallow: /
97
98User-agent: BloombergBot
99Disallow: /
100
101User-agent: BoardReader
102Disallow: /
103
104User-agent: checkmy.ws
105Disallow: /
106
107User-agent: Clickagy
108Disallow: /
109
110User-agent: CookieBot
111Disallow: /
112
113User-agent: cXensebot
114Disallow: /
115
116User-agent: DataObserver
117Disallow: /
118
119User-agent: DomainStatsBot
120Disallow: /
121
122User-agent: Embedly
123Disallow: /
124
125User-agent: Feedspotbot
126Disallow: /
127
128User-agent: FlipboardProxy
129Disallow: /
130
131User-agent: Getintent
132Disallow: /
133
134User-agent: Google-Apps-Script
135Disallow: /
136
137User-agent: GoogleDrive
138Disallow: /
139
140User-agent: Grapevine6
141Disallow: /
142
143User-agent: Gumgum
144Disallow: /
145
146User-agent: HaloBot
147Disallow: /
148
149User-agent: HubSpot Crawler
150Disallow: /
151
152User-agent: IbouBot
153Disallow: /
154
155User-agent: Inout
156Disallow: /
157
158User-agent: LinerBot
159Disallow: /
160
161User-agent: MJ12bot
162Disallow: /
163
164User-agent: Manus
165Disallow: /
166
167User-agent: Meltwater
168Disallow: /
169
170User-agent: Metajob
171Disallow: /
172
173User-agent: Muckrack
174Disallow: /
175
176User-agent: NetSeer
177Disallow: /
178
179User-agent: Netvibes
180Disallow: /
181
182User-agent: Onyphe
183Disallow: /
184
185User-agent: OpenClaw
186Disallow: /
187
188User-agent: Opoint
189Disallow: /
190
191User-agent: OutbrainBot
192Disallow: /
193
194User-agent: PlumAnalytics
195Disallow: /
196
197User-agent: ProSitemaps
198Disallow: /
199
200User-agent: PerplexityBot
201Disallow: /
202
203User-agent: Perplexity-User
204Disallow: /
205
206User-agent: proximic
207Disallow: /
208
209User-agent: PubMatic
210Disallow: /
211
212User-agent: Quantcastbot
213Disallow: /
214
215User-agent: SearchConcepts
216Disallow: /
217
218User-agent: Semantic-Visions
219Disallow: /
220
221User-agent: SentiBot
222Disallow: /
223
224User-agent: Sindup
225Disallow: /
226
227User-agent: Sirdata
228Disallow: /
229
230User-agent: SmartologyBot
231Disallow: /
232
233User-agent: Superfeedr
234Disallow: /
235
236User-agent: surlybot
237Disallow: /
238
239User-agent: Synaptive
240Disallow: /
241
242User-agent: TinEye
243Disallow: /
244
245User-agent: Trendiction
246Disallow: /
247
248User-agent: TurnitinBot
249Disallow: /
250
251User-agent: Ubermetrics
252Disallow: /
253
254User-agent: Vebidoo
255Disallow: /
256
257User-agent: Webcorp
258Disallow: /
259
260User-agent: Weborama
261Disallow: /
262
263User-agent: WebPageTest
264Disallow: /
265
266User-agent: YandexAccessibilityBot
267Disallow: /
268
269User-agent: YandexDirect
270Disallow: /
271
272User-agent: YandexDirectDyn
273Disallow: /
274
275User-agent: YaDirectFetcher
276Disallow: /
277
278User-agent: YandexFavicons
279Disallow: /
280
281User-agent: YandexMetrika
282Disallow: /
283
284User-agent: YandexMobileBot
285Disallow: /
286
287User-agent: YandexNews
288Disallow: /
289
290User-agent: YandexRCA
291Disallow: /
292
293User-agent: YandexRenderResourcesBot
294Disallow: /
295
296User-agent: YandexUserproxy
297Disallow: /
298
299# OAI-AdsBot se mantiene permitido y no se bloquea.
300
301# Parametros transversales
302User-agent: *
303Disallow: /*core_dmena
304
305User-agent: *
306Disallow: /pruebas/
307Disallow: /publicidad/
308Disallow: /notificarelacionadas
309Disallow: /*.swf$
310Disallow: /eskupTSUpdate
311Disallow: /ThreadeskupSimple
312Disallow: /OuteskupSimple
313# Anadidos 20250325 por oscarf
314Disallow: /Ineskup
315Disallow: /Outeskup
316Disallow: /OuteskupSimpleMustache
317Disallow: /Msgeskup
318Disallow: /Profileeskup
319Disallow: /Threadeskup
320
321# Disallow: /.well-known/amphtml/
322Disallow: /t/
323Disallow: *?preview=true$
324
325# Bots bloqueados
326User-agent: 008
327Disallow: /
328User-agent: AddThis.com
329Disallow: /
330User-agent: admantx
331Disallow: /
332User-agent: AhrefsBot
333Disallow: /
334User-agent: BDCbot
335Disallow: /
336User-agent: Bender
337Disallow: /
338User-agent: BIXOCRAWLER
339Disallow: /
340User-agent: bl.uk_lddc_bot
341Disallow: /
342User-agent: BLEXBot
343Disallow: /
344User-agent: BUbiNG
345Disallow: /
346User-agent: Cliqzbot
347Disallow: /
348User-agent: CNCDialer
349Disallow: /
350User-agent: crawler4j
351Disallow: /
352User-agent: CrystalSemanticsBot
353Disallow: /
354User-agent: CyberAlert
355Disallow: /
356User-agent: DigExt
357Disallow: /
358User-agent: discobot
359Disallow: /
360User-agent: discoverybot
361Disallow: /
362User-agent: dloader
363Disallow: /
364User-agent: dloader(NaverRobot)
365Disallow: /
366User-agent: DOC
367Disallow: /
368User-agent: Download Ninja
369Disallow: /
370User-agent: DTS Agent
371Disallow: /
372User-agent: Exabot
373Disallow: /
374User-agent: Ezooms
375Disallow: /
376User-agent: FairShare
377Disallow: /
378User-agent: Fetch
379Disallow: /
380User-agent: Flamingo_SearchEngine
381Disallow: /
382User-agent: Genieo
383Disallow: /
384User-agent: Gigabot
385Disallow: /
386User-agent: grub-client
387Disallow: /
388User-agent: Heritrix
389Disallow: /
390User-agent: heritrix/3.3.0
391Disallow: /
392User-agent: HTTrack
393Disallow: /
394User-agent: integromedb
395Disallow: /
396User-agent: IstellaBot
397Disallow: /
398User-agent: JikeSpider
399Disallow: /
400User-agent: Jyxobot
401Disallow: /
402User-agent: k2spider
403Disallow: /
404User-agent: Kimengi
405Disallow: /
406User-agent: Kimengi/nineconnections.com
407Disallow: /
408User-agent: larbin
409Disallow: /
410User-agent: LexxeBot/1.0
411Disallow: /
412User-agent: libwww
413Disallow: /
414User-agent: linko
415Disallow: /
416User-agent: Livelapbot
417Disallow: /
418User-agent: magpie-crawler
419Disallow: /
420User-agent: Maxthon
421Disallow: /
422User-agent: MetaURI
423Disallow: /
424User-agent: Microsoft.URL.Control
425Disallow: /
426User-agent: Moreoverbot
427Disallow: /
428User-agent: MSIECrawler
429Disallow: /
430User-agent: nabot
431Disallow: /
432User-agent: NaverBot
433Disallow: /
434User-agent: NerdByNature.Bot
435Disallow: /
436User-agent: netEstate NE Crawler
437Disallow: /
438User-agent: NetSeer crawler
439Disallow: /
440User-agent: Newscan
441Disallow: /
442User-agent: NextGenSearchBot
443Disallow: /
444User-agent: NPBot
445Disallow: /
446User-agent: nutch
447Disallow: /
448User-agent: Offline Explorer
449Disallow: /
450User-agent: omgilibot
451Disallow: /
452User-agent: Orthogaffe
453Disallow: /
454User-agent: PiplBot
455Disallow: /
456User-agent: Pixray-Seeker
457Disallow: /
458User-agent: psbot
459Disallow: /
460User-agent: QuerySeekerSpider
461Disallow: /
462User-agent: seokicks
463Disallow: /
464User-agent: SEOkicks-Robot
465Disallow: /
466User-agent: SiteBot
467Disallow: /
468User-agent: SiteBot/0.1
469Disallow: /
470User-agent: sitecheck.internetseer.com
471Disallow: /
472User-agent: SiteSnagger
473Disallow: /
474User-agent: Slurp
475Disallow: /
476User-agent: sogou
477Disallow: /
478User-agent: Sosospider
479Disallow: /
480User-agent: Spinn3r
481Disallow: /
482User-agent: Teleport
483Disallow: /
484User-agent: TeleportPro
485Disallow: /
486User-agent: trendictionbot
487Disallow: /
488User-agent: trovitBot
489Disallow: /
490User-agent: UbiCrawler
491Disallow: /
492User-agent: uMBot-LN
493Disallow: /
494User-agent: UnisterBot
495Disallow: /
496User-agent: UniversalFeedParser
497Disallow: /
498User-agent: WBSearchBot
499Disallow: /
500User-agent: WebCopier
501Disallow: /
502User-agent: WebReaper
503Disallow: /
504User-agent: WebStripper
505Disallow: /
506User-agent: WebZIP
507Disallow: /
508User-agent: WeSEE:Search
509Disallow: /
510User-agent: wget
511Disallow: /
512User-agent: Wotbot
513Disallow: /
514User-agent: wotbox
515Disallow: /
516User-agent: Xenu
517Disallow: /
518User-agent: Yasni
519Disallow: /
520User-agent: Zao
521Disallow: /
522User-agent: Zealbot
523Disallow: /
524User-agent: ZyBORG
525Disallow: /
526
What do the bots actually get?
This page reads robots.txt, which states what elpais.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on elpais.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does elpais.com block GPTBot?
No. elpais.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does elpais.com block ClaudeBot?
Yes. elpais.com’s robots.txt disallows ClaudeBot from the whole site (line 36: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does elpais.com block Google-Extended?
No. elpais.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does elpais.com block PerplexityBot?
Yes. elpais.com’s robots.txt disallows PerplexityBot from the whole site (line 201: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find elpais.com?
Yes. elpais.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.