Who blocks AI · #1078 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does faz.net block AI crawlers?
Yes. faz.net’s robots.txt blocks 16 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 13 more. It also blocks PetalBot among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.faz.net/robots.txt · 201 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 178 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 190 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 106 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Blocked | line 142 · Disallow: / under User-agent: Claude-SearchBot |
DuckAssistBot | DuckDuckGo | Blocked | line 163 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 154 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 109 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 79 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 115 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 92 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 89 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 112 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 121 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 97 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 194 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Blocked | line 145 · Disallow: / under User-agent: Claude-User |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Blocked | line 172 · Disallow: / under User-agent: MistralAI-User |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.faz.net/robots.txt · 201 lines
1# robots.txt updated 2026-10-30
2
3User-agent: *
4Disallow: /json/
5Disallow: /cmlink/
6Disallow: /premiumContent
7Disallow: /*.pdf
8Disallow: /js/adobe/at.js
9Disallow: /redaktion?cache
10Disallow: *mostclicked.gfn
11Disallow: /;suche=artikelsuche/suche/
12Disallow: /suche/
13Disallow: *?sort=
14Disallow: /dist/scripts/main_faz-comment.js
15Disallow: /html-ajax*
16Disallow: /livewire*
17Disallow: *atarget.js
18Disallow: */sport-ergebnisse/*/ma*/$
19Disallow: */sport-ergebnisse/*/co*/$
20Disallow: /api/collect/
21Disallow: /podcast-service/
22Disallow: */event/widgets/
23Disallow: */event/*/live/
24Disallow: */event/*/html/
25Disallow: /source/
26Disallow: /iq/
27
28Disallow: /allesbeste/*?s=*
29Disallow: /allesbeste/*&s=*
30Disallow: */asset-comment-info/*
31Disallow: /membership/*
32Disallow: /kaufkompass/wp-admin/
33Disallow: /kaufkompass/newsticker/
34Disallow: /kaufkompass/wp-content/ab_api_cache/*
35Disallow: /kaufkompass/search/
36Disallow: /kaufkompass/*&preview=*
37Allow: /kaufkompass/wp-admin/admin-ajax.php
38
39User-agent: Meltwater
40Disallow: /
41
42User-agent: NewsNow
43Disallow: /
44
45User-agent: Bloodhound
46Disallow: /
47
48User-agent: cydralspider
49Disallow: /
50
51User-agent: downloadexpress
52Disallow: /
53
54User-agent: gammaSpider
55Disallow: /
56
57User-agent: ObjectsSearch
58Disallow: /
59
60User-agent: Pimptrain
61Disallow: /
62
63User-agent: Raven
64Disallow: /
65
66User-agent: wapspider
67Disallow: /
68
69User-agent: WebZinger
70Disallow: /
71
72User-agent: Fasterfox
73Disallow: /
74
75User-agent: sentibot
76Disallow: /
77
78User-agent: GPTBot
79Disallow: /
80Allow: /kaufkompass/
81
82User-Agent: omgili
83Disallow: /
84
85User-Agent: omgilibot
86Disallow: /
87
88User-agent: CCBot
89Disallow: /
90
91User-agent: Google-Extended
92Disallow: /
93Allow: /source/
94Allow: /*-accg-
95
96User-agent: Bytespider
97Disallow: /
98
99User-agent: anthropic-ai
100Disallow: /
101
102User-agent: Diffbot
103Disallow: /
104
105User-agent: PerplexityBot
106Disallow: /
107
108User-agent: YouBot
109Disallow: /
110
111User-agent: Applebot-Extended
112Disallow: /
113
114User-agent: ClaudeBot
115Disallow: /
116
117User-agent: Claude-Web
118Disallow: /
119
120User-agent: meta-externalagent
121Disallow: /
122
123User-agent: facebookexternalhit
124Disallow: /
125
126User-agent: meta-webindexer
127Disallow: /
128
129User-agent: FacebookBot
130Disallow: /
131
132User-agent: DeepSeekBot
133Disallow: /
134
135User-agent: DeepSeek
136Disallow: /
137
138user-agent: GoogleOther
139allow: /source/
140
141User-Agent: Claude-SearchBot
142Disallow: /
143
144User-Agent: Claude-User
145Disallow: /
146
147User-Agent: cohere-ai
148Disallow: /
149
150User-Agent: cohere-training-data-crawler
151Disallow: /
152
153User-Agent: Amazonbot
154Disallow: /
155
156User-Agent: Timpibot
157Disallow: /
158
159User-Agent: AI2Bot
160Disallow: /
161
162User-Agent: DuckAssistBot
163Disallow: /
164
165User-Agent: Kangaroo Bot
166Disallow: /
167
168User-Agent: PanguBot
169Disallow: /
170
171User-Agent: MistralAI-User
172Disallow: /
173
174User-Agent: Devin
175Disallow: /
176
177User-Agent: PetalBot
178Disallow: /
179
180User-Agent: bigsur.ai
181Disallow: /
182
183User-Agent: CloudVertexBot
184Disallow: /
185
186User-Agent: LinerBot
187Disallow: /
188
189User-Agent: OAI-SearchBot
190Disallow: /
191Allow: /kaufkompass/
192
193User-Agent: ChatGPT-User
194Disallow: /
195Allow: /kaufkompass/
196
197# Legal notice: faz.net expressly reserves the right to use its content for commercial text and data mining (§44 b UrhG).
198# The use of robots or other automated means to access faz.net or collect or mine data without the express permission of faz.net is strictly prohibited.
199# faz.net may, in its discretion, permit certain automated access to certain faz.net pages.
200# If you would like to apply for permission to crawl faz.net, collect or use data, please email nutzungsrechte@faz.de.
201
What do the bots actually get?
This page reads robots.txt, which states what faz.net asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on faz.net Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does faz.net block GPTBot?
Yes. faz.net’s robots.txt disallows GPTBot from the whole site (line 79: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does faz.net block ClaudeBot?
Yes. faz.net’s robots.txt disallows ClaudeBot from the whole site (line 115: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does faz.net block Google-Extended?
Yes. faz.net’s robots.txt disallows Google-Extended from the whole site (line 92: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does faz.net block PerplexityBot?
Yes. faz.net’s robots.txt disallows PerplexityBot from the whole site (line 106: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find faz.net?
Not through its own crawler. faz.net’s robots.txt disallows OAI-SearchBot from the whole site (line 190: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.