Who blocks AI · #911 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does welt.de block AI crawlers?
Yes. welt.de’s robots.txt blocks 9 of the 19 AI crawlers we track, including PerplexityBot, CCBot, Applebot-Extended and 6 more. It also blocks PetalBot and Baiduspider among search engines.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotBlockedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- www.welt.de/robots.txt · 140 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 88 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Blocked | line 88 · Disallow: / under User-agent: Baiduspider |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Blocked | line 88 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Blocked | line 88 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 88 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 88 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Blocked | line 88 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 88 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Blocked | line 88 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Blocked | line 88 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Blocked | line 88 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 88 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Blocked | line 88 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Blocked | line 88 · Disallow: / under User-agent: dotbot |
View www.welt.de/robots.txt · 140 lines
1# The Facebook Crawler
2User-agent: Facebot
3Allow: /
4
5User-agent: AhrefsBot
6User-agent: AI2Bot
7User-agent: Ai2Bot-Dolma
8User-agent: aiHitBot
9User-agent: Amazonbot
10User-agent: Andibot
11User-agent: Applebot-Extended
12User-agent: Awario
13User-agent: Baiduspider
14User-agent: bedrockbot
15User-agent: BLEXBot
16User-agent: Brightbot 1.0
17User-agent: Bytespider
18User-agent: CCBot
19User-agent: cohere-ai
20User-agent: cohere-training-data-crawler
21User-agent: Cotoyogi
22User-agent: Crawlspace
23User-agent: DataForSeoBot
24User-agent: Datenbank Crawler
25User-agent: Devin
26User-agent: Diffbot
27User-agent: dotbot
28User-agent: DuckAssistBot
29User-agent: Echobot Bot
30User-agent: EchoboxBot
31User-agent: Factset_spyderbot
32User-agent: FirecrawlAgent
33User-agent: Flamingo_SearchEngine
34User-agent: FriendlyCrawler
35User-agent: ia_archiver
36User-agent: iaskspider/2.0
37User-agent: ICC-Crawler
38User-agent: ImagesiftBot
39User-agent: img2dataset
40User-agent: ISSCyberRiskCrawler
41User-agent: Kangaroo Bot
42User-agent: LCC
43User-agent: magpie-crawler
44User-agent: MistralAI-User
45User-agent: MistralAI-User/1.0
46User-agent: MJ12bot
47User-agent: MyCentralAIScraperBot
48User-agent: netEstate Imprint Crawler
49User-agent: NovaAct
50User-agent: omgili
51User-agent: omgilibot
52User-agent: Operator
53User-agent: PanguBot
54User-agent: Panscient
55User-agent: panscient.com
56User-agent: Perplexity-User
57User-agent: PerplexityBot
58User-agent: PetalBot
59User-agent: PhindBot
60User-agent: Poseidon Research Crawler
61User-agent: QualifiedBot
62User-agent: QuillBot
63User-agent: quillbot.com
64User-agent: SBIntuitionsBot
65User-agent: Scrapy
66User-agent: SemrushBot
67User-agent: SemrushBot-OCOB
68User-agent: SemrushBot-SA
69User-agent: SemrushBot-SWA
70User-agent: sentibot
71User-agent: Sidetrade indexer bot
72User-agent: SirdataBot
73User-agent: sogou spider
74User-agent: SummalyBot
75User-agent: TerraCotta
76User-agent: Thinkbot
77User-agent: TikTokSpider
78User-agent: Timpibot
79User-agent: TurnitinBot
80User-agent: VelenPublicWebCrawler
81User-agent: WARDBot
82User-agent: Webzio-Extended
83User-agent: wpbot
84User-agent: YandexAdditional
85User-agent: YandexAdditionalBot
86User-agent: Yeti
87User-agent: YouBot
88Disallow: /
89
90
91User-agent: *
92Disallow: /channels-extern/
93Disallow: /sportdaten/
94Disallow: /testgpr/
95Disallow: /boerse/data/
96Disallow: /partner/
97Disallow: /reisetipps/
98Disallow: /z/
99Disallow: /appl/
100Disallow: /woa/
101Disallow: /am-sonntag/vorproduktion/
102Disallow: /audiofiles/
103Disallow: /out-of-home/
104Disallow: /immobilien/expose
105Disallow: /suche
106Disallow: /onward/
107Disallow: /api/
108Disallow: /*?config
109Disallow: /*?config=newsmli_bloomberg2
110Disallow: /*.xmli
111Disallow: /*?service=ajax
112Disallow: /*?service=Ajax
113Disallow: /*?ajax
114Disallow: /*?ajax&wid
115Disallow: /*?config=print
116Disallow: /*?config=articleidfromurl
117Disallow: /*?config=endscreen
118Disallow: /*?config=iframewelt
119Disallow: /*?config=langeslestueck
120Disallow: /*?config=latest_videos
121Disallow: /*?config=menu_home
122Disallow: /*?config=mostviewed_videos
123Disallow: /*?config=recommended_videos
124Disallow: /*?config=regioarticlemarginal
125Disallow: /*?config=zoom
126Disallow: /*?config=zoomopener
127Disallow: /*?noredirect=true&config=standalone
128Disallow: /*?config=standalone
129Disallow: /*?wtmc=XING
130Disallow: /*?config=articleidfromurl
131Disallow: /*?print=yes
132Disallow: /*?tabPane
133Disallow: /video/embeded/
134Disallow: /img/*-wWIDTH*.jpg
135Disallow: /assets/public/fonts/
136
137
138Sitemap: https://www.welt.de/sitemaps/newssitemap/newssitemap.xml
139Sitemap: https://www.welt.de/sitemaps/sitemap/sitemap.xml
140Sitemap: https://www.welt.de/sitemaps/videositemap/videositemap.xml
What do the bots actually get?
This page reads robots.txt, which states what welt.de asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on welt.de Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does welt.de block GPTBot?
No. welt.de’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does welt.de block ClaudeBot?
No. welt.de’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does welt.de block Google-Extended?
No. welt.de’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does welt.de block PerplexityBot?
Yes. welt.de’s robots.txt disallows PerplexityBot from the whole site (line 88: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find welt.de?
Yes. welt.de’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.