Who blocks AI · #1185 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does elmundo.es block AI crawlers?
Yes. elmundo.es’s robots.txt blocks 18 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 15 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.elmundo.es/robots.txt · 385 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 371 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 371 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 371 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 371 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 371 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 371 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 371 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 371 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 371 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 371 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 371 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 371 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 371 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 371 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 371 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Blocked | line 371 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 371 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Blocked | line 371 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 28 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Blocked | line 82 · Disallow: / under User-agent: dotbot |
View www.elmundo.es/robots.txt · 385 lines
1User-agent: *
2Disallow: /1998/
3Disallow: /2002/
4Disallow: /s/
5Disallow: /cgi-bin/
6Disallow: /perl/
7Disallow: /foros/
8Disallow: /yodona/hemeroteca/
9Disallow: /metropoli/hemeroteca/
10Disallow: /eventos/en-directo/*.json
11Disallow: /eventos/en-directo/resources.html*
12Disallow: /noticias/envia_noticia.html
13Disallow: /enviaropinion.html?*
14Disallow: /micuenta/
15
16Disallow: */pruebas-abierto/*
17Disallow: */elmundo/hemeroteca/*/*/*/*/*
18Allow: */elmundo/hemeroteca/*/*/*/*/index*
19
20# Bots poco fiables o que gastan recursos
21User-agent: 008
22Disallow: /
23
24User-agent: AddThis.com
25Disallow: /
26
27User-agent: AhrefsBot
28Disallow: /
29
30User-agent: BDCbot
31Disallow: /
32
33User-agent: Bender
34Disallow: /
35
36User-agent: BIXOCRAWLER
37Disallow: /
38
39User-agent: bl.uk_lddc_bot
40Disallow: /
41
42User-agent: BLEXBot
43Disallow: /
44
45User-agent: BUbiNG
46Disallow: /
47
48User-agent: Cliqzbot
49Disallow: /
50
51User-agent: CNCDialer
52Disallow: /
53
54User-agent: crawler4j
55Disallow: /
56
57User-agent: CrystalSemanticsBot
58Disallow: /
59
60User-agent: CyberAlert
61Disallow: /
62
63User-agent: DigExt
64Disallow: /
65
66User-agent: discobot
67Disallow: /
68
69User-agent: discoverybot
70Disallow: /
71
72User-agent: dloader
73Disallow: /
74
75User-agent: dloader(NaverRobot)
76Disallow: /
77
78User-agent: DOC
79Disallow: /
80
81User-agent: dotbot
82Disallow: /
83
84User-agent: Download Ninja
85Disallow: /
86
87User-agent: DTS Agent
88Disallow: /
89
90User-agent: Exabot
91Disallow: /
92
93User-agent: Ezooms
94Disallow: /
95
96User-agent: FairShare
97Disallow: /
98
99User-agent: Fetch
100Disallow: /
101
102User-agent: Flamingo_SearchEngine
103Disallow: /
104
105User-agent: Genieo
106Disallow: /
107
108User-agent: Gigabot
109Disallow: /
110
111User-agent: grub-client
112Disallow: /
113
114User-agent: Heritrix
115Disallow: /
116
117User-agent: heritrix/3.3.0
118Disallow: /
119
120User-agent: HTTrack
121Disallow: /
122
123User-agent: ia_archiver
124Disallow: /
125
126User-agent: integromedb
127Disallow: /
128
129User-agent: IstellaBot
130Disallow: /
131
132User-agent: JikeSpider
133Disallow: /
134
135User-agent: Jyxobot
136Disallow: /
137
138User-agent: k2spider
139Disallow: /
140
141User-agent: Kimengi
142Disallow: /
143
144User-agent: Kimengi/nineconnections.com
145Disallow: /
146
147User-agent: larbin
148Disallow: /
149
150User-agent: LexxeBot/1.0
151Disallow: /
152
153User-agent: libwww
154Disallow: /
155
156User-agent: linko
157Disallow: /
158
159User-agent: Livelapbot
160Disallow: /
161
162User-agent: magpie-crawler
163Disallow: /
164
165User-agent: Maxthon
166Disallow: /
167
168User-agent: MetaURI
169Disallow: /
170
171User-agent: Microsoft.URL.Control
172Disallow: /
173
174User-agent: MJ12bot
175Disallow: /
176
177User-agent: Moreover
178Disallow: /
179
180User-agent: Moreoverbot
181Disallow: /
182
183User-agent: MSIECrawler
184Disallow: /
185
186User-agent: nabot
187Disallow: /
188
189User-agent: NaverBot
190Disallow: /
191
192User-agent: NerdByNature.Bot
193Disallow: /
194
195User-agent: netEstate NE Crawler
196Disallow: /
197
198User-agent: NetSeer crawler
199Disallow: /
200
201User-agent: Newscan
202Disallow: /
203
204User-agent: NextGenSearchBot
205Disallow: /
206
207User-agent: NPBot
208Disallow: /
209
210User-agent: nutch
211Disallow: /
212
213User-agent: Offline Explorer
214Disallow: /
215
216User-agent: Orthogaffe
217Disallow: /
218
219User-agent: PiplBot
220Disallow: /
221
222User-agent: Pixray-Seeker
223Disallow: /
224
225User-agent: proximic
226Disallow: /
227
228User-agent: psbot
229Disallow: /
230
231User-agent: QuerySeekerSpider
232Disallow: /
233
234User-agent: rogerbot
235Disallow: /
236
237User-agent: seokicks
238Disallow: /
239
240User-agent: SEOkicks-Robot
241Disallow: /
242
243User-agent: SiteBot
244Disallow: /
245
246User-agent: SiteBot/0.1
247Disallow: /
248
249User-agent: sitecheck.internetseer.com
250Disallow: /
251
252User-agent: SiteSnagger
253Disallow: /
254
255User-agent: Slurp
256Disallow: /
257
258User-agent: sogou
259Disallow: /
260
261User-agent: Sosospider
262Disallow: /
263
264User-agent: spbot
265Disallow: /
266
267User-agent: Spinn3r
268Disallow: /
269
270User-agent: Teleport
271Disallow: /
272
273User-agent: TeleportPro
274Disallow: /
275
276User-agent: trendictionbot
277Disallow: /
278
279User-agent: trovitBot
280Disallow: /
281
282User-agent: TurnitinBot
283Disallow: /
284
285User-agent: UbiCrawler
286Disallow: /
287
288User-agent: uMBot-LN
289Disallow: /
290
291User-agent: UnisterBot
292Disallow: /
293
294User-agent: UniversalFeedParser
295Disallow: /
296
297User-agent: WBSearchBot
298Disallow: /
299
300User-agent: WebCopier
301Disallow: /
302
303User-agent: WebReaper
304Disallow: /
305
306User-agent: WebStripper
307Disallow: /
308
309User-agent: WebZIP
310Disallow: /
311
312User-agent: WeSEE:Search
313Disallow: /
314
315User-agent: wget
316Disallow: /
317
318User-agent: Wotbot
319Disallow: /
320
321User-agent: wotbox
322Disallow: /
323
324User-agent: Xenu
325Disallow: /
326
327User-agent: Yasni
328Disallow: /
329
330User-agent: Zao
331Disallow: /
332
333User-agent: Zealbot
334Disallow: /
335
336User-agent: ZyBORG
337Disallow: /
338
339# Bots IA: bloqueo total
340User-agent: GPTBot
341User-agent: ChatGPT-User
342User-agent: OAI-SearchBot
343User-agent: ClaudeBot
344User-agent: Claude-User
345User-agent: Claude-SearchBot
346User-agent: anthropic-ai
347User-agent: claude-web
348User-agent: Google-Extended
349User-agent: Google-CloudVertexBot
350User-agent: Applebot-Extended
351User-agent: Meta-ExternalAgent
352User-agent: Meta-ExternalFetcher
353User-agent: PerplexityBot
354User-agent: Perplexity-User
355User-agent: Amazonbot
356User-agent: Bytespider
357User-agent: CCBot
358User-agent: cohere-ai
359User-agent: cohere-training-data-crawler
360User-agent: MistralAI-User
361User-agent: DuckAssistBot
362User-agent: AI2Bot
363User-agent: Ai2Bot-Dolma
364User-agent: Diffbot
365User-agent: YouBot
366User-agent: Timpibot
367User-agent: ImagesiftBot
368User-agent: omgili
369User-agent: omgilibot
370User-agent: PanguBot
371Disallow: /
372Allow: /uestudio/
373Allow: /ofertas-regalos/
374
375# Bots publicitarios permitidos
376User-agent: AmazonAdBot
377Allow: /
378
379User-agent: admantx
380Allow: /
381
382# Sitemaps
383Sitemap: https://www.elmundo.es/sitemaps/directoeditorial_news_a3f3222a2fa5a98b79804fa04675a139.xml
384Sitemap: https://www.elmundo.es/sitemaps/videoct_news_ba306c30e4f41fc8f7a5e2d166e42a3b.xml
385
What do the bots actually get?
This page reads robots.txt, which states what elmundo.es asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on elmundo.es Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
- 2 November 2026: started blocking Claude-SearchBot; started blocking DuckAssistBot; started blocking Amazonbot; started blocking YouBot; started blocking ClaudeBot; started blocking Applebot-Extended; started blocking Meta-ExternalAgent; started blocking Bytespider; started blocking Claude-User; started blocking Perplexity-User; started blocking MistralAI-User; started blocking Meta-ExternalFetcher.
Questions
Does elmundo.es block GPTBot?
Yes. elmundo.es’s robots.txt disallows GPTBot from the whole site (line 371: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does elmundo.es block ClaudeBot?
Yes. elmundo.es’s robots.txt disallows ClaudeBot from the whole site (line 371: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does elmundo.es block Google-Extended?
Yes. elmundo.es’s robots.txt disallows Google-Extended from the whole site (line 371: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does elmundo.es block PerplexityBot?
Yes. elmundo.es’s robots.txt disallows PerplexityBot from the whole site (line 371: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find elmundo.es?
Not through its own crawler. elmundo.es’s robots.txt disallows OAI-SearchBot from the whole site (line 371: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.