Who blocks AI · #524 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does spiegel.de block AI crawlers?
Yes. spiegel.de’s robots.txt blocks 12 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, OAI-SearchBot and 9 more. It also blocks PetalBot and Baiduspider among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- www.spiegel.de/robots.txt · 314 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | line 2 · Allow: / under User-agent: * | |
Bingbot | Microsoft | Allowed | line 2 · Allow: / under User-agent: * |
DuckDuckBot | DuckDuckGo | Allowed | line 2 · Allow: / under User-agent: * |
Applebot | Apple | Allowed | line 2 · Allow: / under User-agent: * |
PetalBot | Huawei | Blocked | line 110 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | line 2 · Allow: / under User-agent: * |
Baiduspider | Baidu | Blocked | line 230 · Disallow: / under User-agent: Baiduspider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 24 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Allowed | line 2 · Allow: / under User-agent: * |
Claude-SearchBot | Anthropic | Blocked | line 60 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Allowed | line 2 · Allow: / under User-agent: * |
Amazonbot | Amazon | Allowed | line 2 · Allow: / under User-agent: * |
YouBot | You.com | Blocked | line 120 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 78 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 82 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | line 2 · Allow: / under User-agent: * | |
CCBot | Common Crawl | Blocked | line 118 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 100 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | line 2 · Allow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Blocked | line 96 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 236 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 36 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 72 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Allowed | line 2 · Allow: / under User-agent: * |
MistralAI-User | Mistral | Blocked | line 102 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Allowed | line 2 · Allow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 136 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Blocked | line 140 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | line 2 · Allow: / under User-agent: * |
DotBot | Moz | Blocked | line 178 · Disallow: / under User-agent: dotbot |
View www.spiegel.de/robots.txt · 314 lines
1User-agent: *
2Allow: /
3Disallow: /*CR-Dokumentation.pdf$
4
5User-agent: emetriqContextualBot
6Allow: /
7
8User-agent: Mozilla/5.0 (compatible; OGDWCtxCrawler)
9Allow: /
10
11User-agent: AmazonAdBot
12Allow: /
13
14# TLP-6507: Testweise Freischaltung der OpenAI-Suchcrawler fuer ausgewaehlte Bereiche
15User-agent: OAI-SearchBot
16Allow: /ausland/
17Allow: /partnerschaft/
18Allow: /gesundheit/
19Allow: /familie/
20Allow: /reise/
21Allow: /psychologie/
22Allow: /stil/
23Allow: /tests/
24Disallow: /
25
26# TLP-6507: Testweise Freischaltung der OpenAI-Suchcrawler fuer ausgewaehlte Bereiche
27User-agent: ChatGPT-User
28Allow: /ausland/
29Allow: /partnerschaft/
30Allow: /gesundheit/
31Allow: /familie/
32Allow: /reise/
33Allow: /psychologie/
34Allow: /stil/
35Allow: /tests/
36Disallow: /
37
38# TLP-7042: Testweise Freischaltung Claude-Web fuer ausgewaehlte Bereiche
39User-agent: Claude-Web
40Allow: /ausland/
41Allow: /partnerschaft/
42Allow: /gesundheit/
43Allow: /familie/
44Allow: /reise/
45Allow: /psychologie/
46Allow: /stil/
47Allow: /tests/
48Disallow: /
49
50# TLP-7042: Testweise Freischaltung Claude-SearchBot fuer ausgewaehlte Bereiche
51User-agent: Claude-SearchBot
52Allow: /ausland/
53Allow: /partnerschaft/
54Allow: /gesundheit/
55Allow: /familie/
56Allow: /reise/
57Allow: /psychologie/
58Allow: /stil/
59Allow: /tests/
60Disallow: /
61
62# TLP-7042: Testweise Freischaltung Claude-User fuer ausgewaehlte Bereiche
63User-agent: Claude-User
64Allow: /ausland/
65Allow: /partnerschaft/
66Allow: /gesundheit/
67Allow: /familie/
68Allow: /reise/
69Allow: /psychologie/
70Allow: /stil/
71Allow: /tests/
72Disallow: /
73
74# ==========================================
75# KI-CRAWLER & KI-TRAINING (SPERRE)
76# ==========================================
77User-agent: GPTBot
78Disallow: /
79User-agent: anthropic-ai
80Disallow: /
81User-agent: ClaudeBot
82Disallow: /
83User-agent: CloudVertexBot
84Disallow: /
85User-agent: cohere-ai
86Disallow: /
87User-agent: cohere-training-data-crawler
88Disallow: /
89User-agent: cohere-training-data-collector
90Disallow: /
91User-agent: DeepSeekBot
92Disallow: /
93User-agent: DeepSeek
94Disallow: /
95User-agent: Meta-ExternalAgent
96Disallow: /
97User-agent: FacebookBot
98Disallow: /
99User-agent: Applebot-Extended
100Disallow: /
101User-agent: MistralAI-User
102Disallow: /
103User-agent: AI2Bot
104Disallow: /
105User-agent: Kangaroo Bot
106Disallow: /
107User-agent: PanguBot
108Disallow: /
109User-agent: PetalBot
110Disallow: /
111User-agent: Devin
112Disallow: /
113User-agent: bigsur.ai
114Disallow: /
115User-agent: LinerBot
116Disallow: /
117User-agent: CCBot
118Disallow: /
119User-agent: YouBot
120Disallow: /
121User-agent: YouBot-Search
122Disallow: /
123User-agent: iaskspider
124Disallow: /
125User-agent: YoudaoBot
126Disallow: /
127User-agent: DeepL-Translate
128Disallow: /
129User-agent: DeepL-Bot
130Disallow: /
131
132# ==========================================
133# SEO-CRAWLER (SPERRE)
134# ==========================================
135User-agent: AhrefsBot
136Disallow: /
137User-agent: AhrefsSiteAudit
138Disallow: /
139User-agent: SemrushBot
140Disallow: /
141User-agent: SemrushBot-SA
142Disallow: /
143User-agent: SemrushBot-BA
144Disallow: /
145User-agent: SemrushBot-SI
146Disallow: /
147User-agent: SemrushBot-SWA
148Disallow: /
149User-agent: SiteAuditBot
150Disallow: /
151User-agent: SplitSignalBot
152Disallow: /
153User-agent: xovi
154Disallow: /
155User-agent: XoviBot
156Disallow: /
157User-agent: Seobility
158Disallow: /
159User-agent: SeobilityBot
160Disallow: /
161User-agent: RyteBot
162Disallow: /
163User-agent: SEOkicks
164Disallow: /
165User-agent: SEOkicks-Robot
166Disallow: /
167User-agent: Searchmetrics
168Disallow: /
169User-agent: SearchmetricsBot
170Disallow: /
171User-agent: audisto
172Disallow: /
173User-agent: audisto-essential
174Disallow: /
175User-agent: MJ12bot
176Disallow: /
177User-agent: dotbot
178Disallow: /
179User-agent: rogerbot
180Disallow: /
181User-agent: Ezooms
182Disallow: /
183User-agent: barkrowler
184Disallow: /
185User-agent: serpstatbot
186Disallow: /
187User-agent: BLEXBot
188Disallow: /
189User-agent: DataForSeoBot
190Disallow: /
191User-agent: MegaIndex
192Disallow: /
193User-agent: lumar
194Disallow: /
195User-agent: deepcrawl
196Disallow: /
197User-agent: oncrawl
198Disallow: /
199User-agent: sitebulb
200Disallow: /
201User-agent: botify
202Disallow: /
203User-agent: Siteimprove
204Disallow: /
205User-agent: Siteimprove Crawl
206Disallow: /
207User-agent: MozDotNet
208Disallow: /
209User-agent: Cocolyzebot
210Disallow: /
211User-agent: Raven
212Disallow: /
213
214# ==========================================
215# MEDIENBEOBACHTUNG, AGGREGATOREN & SCRAPER
216# ==========================================
217User-agent: Meltwater
218Disallow: /
219User-agent: NewsNow
220Disallow: /
221User-agent: Webzio-Extended
222Disallow: /
223User-agent: magpie-crawler
224Disallow: /
225User-agent: omgili
226Disallow: /
227User-agent: omgilibot
228Disallow: /
229User-agent: Baiduspider
230Disallow: /
231User-agent: Yeti
232Disallow: /
233User-agent: sentibot
234Disallow: /
235User-agent: Bytespider
236Disallow: /
237User-agent: SirdataBot
238Disallow: /
239User-agent: LCC
240Disallow: /
241User-agent: TurnitinBot
242Disallow: /
243User-agent: ImagesiftBot
244Disallow: /
245User-agent: Timpibot
246Disallow: /
247User-agent: Diffbot
248Disallow: /
249User-agent: Landau-Media-Spider
250Disallow: /
251
252
253# ==========================================
254# ARCHIV-CRAWLER (SPERRE)
255# ==========================================
256User-agent: archive.org_bot
257Disallow: /
258User-agent: WaybackSavePage
259Disallow: /
260User-agent: heritrix
261Disallow: /
262User-agent: Special_Archiver
263Disallow: /
264User-agent: ia_archiver-web.archive.org
265Disallow: /
266User-agent: ia_archiver2
267Disallow: /
268User-agent: ia_archiver
269Disallow: /
270
271# ==========================================
272# HISTORISCHE BOTS & MASSEN-DOWNLOADER
273# ==========================================
274User-agent: Bloodhound
275Disallow: /
276User-agent: cydralspider
277Disallow: /
278User-agent: downloadexpress
279Disallow: /
280User-agent: gammaSpider
281Disallow: /
282User-agent: ObjectsSearch
283Disallow: /
284User-agent: Pimptrain
285Disallow: /
286User-agent: wapspider
287Disallow: /
288User-agent: WebZinger
289Disallow: /
290User-agent: Fasterfox
291Disallow: /
292
293# ==========================================
294# SCRAPING FRAMEWORKS & UTILITIES
295# ==========================================
296User-agent: Scrapy
297Disallow: /
298User-agent: HTTPBannerDetection
299Disallow: /
300User-agent: Wget
301Disallow: /
302
303# Sitemaps
304Sitemap: https://www.spiegel.de/sitemaps/news-de.xml
305Sitemap: https://www.spiegel.de/sitemaps/videos/sitemap.xml
306Sitemap: https://www.spiegel.de/plus/sitemap.xml
307Sitemap: https://www.spiegel.de/sitemap.xml
308
309# Legal notice: spiegel.de expressly reserves the right to use its content for commercial text and data mining (§ 44b Urheberrechtsgesetz).
310# The use of robots or other automated means to access spiegel.de or collect or mine data without the express permission of spiegel.de is strictly prohibited.
311# spiegel.de may, in its discretion, permit certain automated access to certain spiegel.de pages,
312# If you would like to apply for permission to crawl spiegel.de, collect or use data, please email syndication@spiegel.de
313
314
What do the bots actually get?
This page reads robots.txt, which states what spiegel.de asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on spiegel.de Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does spiegel.de block GPTBot?
Yes. spiegel.de’s robots.txt disallows GPTBot from the whole site (line 78: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does spiegel.de block ClaudeBot?
Yes. spiegel.de’s robots.txt disallows ClaudeBot from the whole site (line 82: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does spiegel.de block Google-Extended?
No. spiegel.de’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does spiegel.de block PerplexityBot?
No. spiegel.de’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find spiegel.de?
Not through its own crawler. spiegel.de’s robots.txt disallows OAI-SearchBot from the whole site (line 24: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.