Who blocks AI · #154 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does statista.com block AI crawlers?
Yes. statista.com’s robots.txt blocks 14 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 11 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.statista.com/robots.txt · 408 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Blocked | line 372 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Blocked | line 372 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Blocked | line 372 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 372 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 372 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 372 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 372 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 372 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 372 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 372 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Blocked | line 372 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 372 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Blocked | line 372 · Disallow: / under User-agent: Perplexity-User |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Blocked | line 372 · Disallow: / under User-agent: Meta-ExternalFetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Partly blocked | line 195 · Disallow: /ads/ line 196 · Disallow: /xml/ and 66 more |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.statista.com/robots.txt · 408 lines
1# Automated access to this website by web crawlers is allowed only for the use in public search engines (e.g. Google or Bing).
2# Automated data collection, data mining or mirroring (scraping) of any content on this website for any other purpose than public search engines without explicit authorization is strictly prohibited.
3# Irresponsible or illegal access may be blocked at any time, and we reserve the right to take legal action against such activities.
4#
5User-agent: *
6#
7# Legacy URLs
8Disallow: /ads/
9Disallow: /xml/
10Disallow: /berichte/
11Disallow: /blog/
12Disallow: /data/
13Disallow: /download/
14Disallow: /hinweis/datenleck/
15Disallow: /important/dataleak/
16Disallow: /important/fuite-donnees/
17Disallow: /importante/fuga-de-datos/
18Disallow: /favorit-hinzufuegen/
19Disallow: /add-favorite/
20Disallow: /ajouter-favoris/
21Disallow: /anadir-busqueda/
22Disallow: /favorit-loeschen/
23Disallow: /remove-favorite/
24Disallow: /supprimer-favoris/
25Disallow: /eliminar-busqueda/
26Disallow: /eliminar-favoritos/
27#
28# Forms
29Disallow: /statistik/report-content/
30Disallow: /statistics/report-content/
31Disallow: /statistiques/report-content/
32Disallow: /estadisticas/report-content/
33#
34# Products (06/24 Update)
35# 03/25 companies discontinued & removed
36Disallow: /perspectiva/
37Disallow: /perspective/
38Disallow: /global-consumer-survey/tool/
39#
40# Pagination (04/22)
41Disallow: /statistics/popular/p/
42Disallow: /statistik/top/p/
43Disallow: /statistiques/populaires/p/
44Disallow: /estadisticas/populares/p/
45# Update 03/25
46Disallow: /statistics/recommendations/?p=
47Disallow: /estadisticas/recomendaciones/?p=
48Disallow: /statistiques/recommandees/?p=
49Disallow: /statistik/empfehlungen/?p=
50Disallow: /statistics/forecast/p/
51Disallow: /estadisticas/prevision/p/
52Disallow: /statistiques/prevision/p/
53Disallow: /statistik/prognose/p/
54Disallow: /statistics/updated/?p=
55Disallow: /estadisticas/actualizadas/?p=
56Disallow: /statistiques/mis-a-jour/?p=
57Disallow: /statistik/aktualisiert/?p=
58Disallow: /statistics/new/p/
59Disallow: /estadisticas/nuevas/p/
60Disallow: /statistiques/nouveau/p/
61Disallow: /statistik/neu/p/
62#
63# Editors (08/23 opened for crawler)
64#
65# Study search (04/22)
66Disallow: /studies-and-reports/*?q=
67Disallow: /studies-and-reports/*&q=
68Disallow: /studien-und-reports/*?q=
69Disallow: /studien-und-reports/*&q=
70#
71# Personal information
72Disallow: /profile/
73Disallow: /profil/
74Disallow: /perfil/
75Disallow: /login/password/
76Disallow: /login/passwort_vergessen/
77Disallow: /acceso/contrasena/
78Disallow: /login/resend-confirmation-link
79Disallow: /login/bestaetigungslink-erneut-senden
80Disallow: /login/reenviar-enlace-de-confirmacion
81Disallow: /login/renvoyer-le-lien-de-confirmation
82Disallow: /direct/purchase/
83Disallow: /request/custom-solution/
84#
85# Internal search (04/22)
86Disallow: /search/
87Disallow: /statistik/suche/
88Disallow: /buscar/
89Disallow: /recherche/
90Disallow: /serp
91#
92# Research AI (07/24)
93Disallow: /research-ai/
94Allow: /research-ai/$
95#
96User-agent: YisouSpider
97User-agent: Sogou web spider/4.0
98User-agent: Sogou web spider
99User-agent: sogou spider
100#
101# Legacy URLs
102Disallow: /ads/
103Disallow: /xml/
104Disallow: /berichte/
105Disallow: /blog/
106Disallow: /data/
107Disallow: /download/
108Disallow: /hinweis/datenleck/
109Disallow: /important/dataleak/
110Disallow: /important/fuite-donnees/
111Disallow: /importante/fuga-de-datos/
112Disallow: /favorit-hinzufuegen/
113Disallow: /add-favorite/
114Disallow: /ajouter-favoris/
115Disallow: /anadir-busqueda/
116Disallow: /favorit-loeschen/
117Disallow: /remove-favorite/
118Disallow: /supprimer-favoris/
119Disallow: /eliminar-busqueda/
120Disallow: /eliminar-favoritos/
121#
122# Forms
123Disallow: /statistik/report-content/
124Disallow: /statistics/report-content/
125Disallow: /statistiques/report-content/
126Disallow: /estadisticas/report-content/
127#
128# Products (06/24 Update)
129# 03/25 companies discontinued & removed
130Disallow: /perspectiva/
131Disallow: /perspective/
132Disallow: /global-consumer-survey/tool/
133#
134# Pagination (04/22)
135Disallow: /statistics/popular/p/
136Disallow: /statistik/top/p/
137Disallow: /statistiques/populaires/p/
138Disallow: /estadisticas/populares/p/
139# Update 03/25
140Disallow: /statistics/recommendations/?p=
141Disallow: /estadisticas/recomendaciones/?p=
142Disallow: /statistiques/recommandees/?p=
143Disallow: /statistik/empfehlungen/?p=
144Disallow: /statistics/forecast/p/
145Disallow: /estadisticas/prevision/p/
146Disallow: /statistiques/prevision/p/
147Disallow: /statistik/prognose/p/
148Disallow: /statistics/updated/?p=
149Disallow: /estadisticas/actualizadas/?p=
150Disallow: /statistiques/mis-a-jour/?p=
151Disallow: /statistik/aktualisiert/?p=
152Disallow: /statistics/new/p/
153Disallow: /estadisticas/nuevas/p/
154Disallow: /statistiques/nouveau/p/
155Disallow: /statistik/neu/p/
156#
157# Editors (08/23 opened for crawler)
158#
159# Study search (04/22)
160Disallow: /studies-and-reports/*?q=
161Disallow: /studies-and-reports/*&q=
162Disallow: /studien-und-reports/*?q=
163Disallow: /studien-und-reports/*&q=
164#
165# Personal information
166Disallow: /profile/
167Disallow: /profil/
168Disallow: /perfil/
169Disallow: /login/password/
170Disallow: /login/passwort_vergessen/
171Disallow: /acceso/contrasena/
172Disallow: /login/resend-confirmation-link
173Disallow: /login/bestaetigungslink-erneut-senden
174Disallow: /login/reenviar-enlace-de-confirmacion
175Disallow: /login/renvoyer-le-lien-de-confirmation
176Disallow: /direct/purchase/
177Disallow: /request/custom-solution/
178#
179# Internal search (04/22)
180Disallow: /search/
181Disallow: /statistik/suche/
182Disallow: /buscar/
183Disallow: /recherche/
184Disallow: /serp
185#
186# Research AI (07/24)
187Disallow: /research-ai/
188Allow: /research-ai/$
189#
190# Slowdown
191Crawl-delay: 10
192#
193User-agent: AhrefsBot
194# Legacy URLs
195Disallow: /ads/
196Disallow: /xml/
197Disallow: /berichte/
198Disallow: /blog/
199Disallow: /data/
200Disallow: /download/
201Disallow: /hinweis/datenleck/
202Disallow: /important/dataleak/
203Disallow: /important/fuite-donnees/
204Disallow: /importante/fuga-de-datos/
205Disallow: /favorit-hinzufuegen/
206Disallow: /add-favorite/
207Disallow: /ajouter-favoris/
208Disallow: /anadir-busqueda/
209Disallow: /favorit-loeschen/
210Disallow: /remove-favorite/
211Disallow: /supprimer-favoris/
212Disallow: /eliminar-busqueda/
213Disallow: /eliminar-favoritos/
214#
215# Forms
216Disallow: /statistik/report-content/
217Disallow: /statistics/report-content/
218Disallow: /statistiques/report-content/
219Disallow: /estadisticas/report-content/
220#
221# Products (06/24 Update)
222# 03/25 companies discontinued & removed
223Disallow: /perspectiva/
224Disallow: /perspective/
225Disallow: /global-consumer-survey/tool/
226#
227# Pagination (04/22)
228Disallow: /statistics/popular/p/
229Disallow: /statistik/top/p/
230Disallow: /statistiques/populaires/p/
231Disallow: /estadisticas/populares/p/
232# Update 03/25
233Disallow: /statistics/recommendations/?p=
234Disallow: /estadisticas/recomendaciones/?p=
235Disallow: /statistiques/recommandees/?p=
236Disallow: /statistik/empfehlungen/?p=
237Disallow: /statistics/forecast/p/
238Disallow: /estadisticas/prevision/p/
239Disallow: /statistiques/prevision/p/
240Disallow: /statistik/prognose/p/
241Disallow: /statistics/updated/?p=
242Disallow: /estadisticas/actualizadas/?p=
243Disallow: /statistiques/mis-a-jour/?p=
244Disallow: /statistik/aktualisiert/?p=
245Disallow: /statistics/new/p/
246Disallow: /estadisticas/nuevas/p/
247Disallow: /statistiques/nouveau/p/
248Disallow: /statistik/neu/p/
249#
250# Editors (08/23 opened for crawler)
251#
252# Study search (04/22)
253Disallow: /studies-and-reports/*?q=
254Disallow: /studies-and-reports/*&q=
255Disallow: /studien-und-reports/*?q=
256Disallow: /studien-und-reports/*&q=
257#
258# Personal information
259Disallow: /profile/
260Disallow: /profil/
261Disallow: /perfil/
262Disallow: /login/password/
263Disallow: /login/passwort_vergessen/
264Disallow: /acceso/contrasena/
265Disallow: /login/resend-confirmation-link
266Disallow: /login/bestaetigungslink-erneut-senden
267Disallow: /login/reenviar-enlace-de-confirmacion
268Disallow: /login/renvoyer-le-lien-de-confirmation
269Disallow: /direct/purchase/
270Disallow: /request/custom-solution/
271#
272# Internal search (04/22)
273Disallow: /search/
274Disallow: /statistik/suche/
275Disallow: /buscar/
276Disallow: /recherche/
277Disallow: /serp
278#
279# Research AI (07/24)
280Disallow: /research-ai/
281Allow: /research-ai/$
282#
283# Slowdown
284Crawl-delay: 2
285#
286# Scraping protection
287User-agent: sitecheck.internetseer.com
288User-agent: Zealbot
289User-agent: MSIECrawler
290User-agent: SiteSnagger
291User-agent: WebStripper
292User-agent: WebCopier
293User-agent: Fetch
294User-agent: Offline Explorer
295User-agent: Teleport
296User-agent: TeleportPro
297User-agent: WebZIP
298User-agent: linko
299User-agent: HTTrack
300User-agent: Microsoft.URL.Control
301User-agent: Xenu
302User-agent: larbin
303User-agent: libwww
304User-agent: ZyBORG
305User-agent: Download Ninja
306User-agent: Brightbot 1.0
307Disallow: /
308#
309# Disallow selected LLMs and other bots/crawlers (update 05/26)
310#
311# Amazon https://developer.amazon.com/amazonbot
312User-agent: Amazonbot
313#
314# Anthropic https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
315User-agent: anthropic-ai
316User-agent: Claude-Web
317User-agent: ClaudeBot
318#
319# OpenAI https://developers.openai.com/api/docs/bots
320User-agent: Operator
321User-agent: GPTBot
322#
323# Cohere https://knownagents.com/operators/cohere
324User-agent: cohere-training-data-crawler
325User-agent: cohere-ai
326#
327# Facebook/Meta https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
328User-agent: Meta-ExternalFetcher
329User-agent: Meta-ExternalAgent
330User-agent: FacebookBot
331#
332# Google https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
333User-agent: Google-Extended
334User-agent: GoogleOther
335User-agent: GoogleOther-Video
336User-agent: GoogleOther-Image
337User-agent: Google-CloudVertexBot
338#
339# Perplexity https://docs.perplexity.ai/docs/resources/perplexity-crawlers#perplexity-crawlers
340User-agent: PerplexityBot
341User-agent: Perplexity-User
342#
343# Other
344User-agent: AI2Bot
345User-agent: Ai2Bot-Dolma
346User-agent: Applebot-Extended
347User-agent: Aranet-SearchBot
348User-agent: archive.org_bot
349User-agent: Arquivo-web-crawler
350User-agent: Bytespider
351User-agent: CCBot
352User-agent: Cincraw
353User-agent: Diffbot
354User-agent: DuckAssistBot
355User-agent: FriendlyCrawler
356User-agent: ia_archiver
357User-agent: ia_archiver-web.archive.org
358User-agent: ImagesiftBot
359User-agent: img2dataset
360User-agent: Kangaroo LLM
361User-agent: LinerBot
362User-agent: Magpie-crawler
363User-agent: omgili
364User-agent: omgilibot
365User-agent: PanguBot
366User-agent: Peer39_crawler
367User-agent: TerraCotta
368User-agent: Timpibot
369User-agent: VelenPublicWebCrawler
370User-agent: Webzio-Extended
371User-agent: YouBot
372Disallow: /
373Allow: /$
374Allow: /statistik/info$
375Allow: /kontakt/
376Allow: /contact/
377Allow: /contacto/
378Allow: /accounts/
379Allow: /solutions/
380Allow: /getting-started/
381Allow: /infografik/
382Allow: /chartoftheday/
383Allow: /chart/
384Allow: /graphique-du-jour/
385Allow: /infographie/
386Allow: /grafico-del-dia/
387Allow: /grafico/
388#
389Allow: /karriere/
390Allow: /working-at-statista/
391Allow: /business/
392Allow: /impressum/
393Allow: /imprint/
394Allow: /lexikonListe/
395Allow: /lexikon/
396Allow: /statistics-glossary/
397Allow: /presse/
398Allow: /press/
399Allow: /prensa/
400Allow: /research-ai/$
401Allow: /webinar$
402Allow: /webinar/
403Allow: /site/
404#
405# Sitemaps all languages
406Sitemap: https://www.statista.com/sitemap/
407Sitemap: https://de.statista.com/sitemap/
408
What do the bots actually get?
This page reads robots.txt, which states what statista.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on statista.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does statista.com block GPTBot?
Yes. statista.com’s robots.txt disallows GPTBot from the whole site (line 372: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does statista.com block ClaudeBot?
Yes. statista.com’s robots.txt disallows ClaudeBot from the whole site (line 372: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does statista.com block Google-Extended?
Yes. statista.com’s robots.txt disallows Google-Extended from the whole site (line 372: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does statista.com block PerplexityBot?
Yes. statista.com’s robots.txt disallows PerplexityBot from the whole site (line 372: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find statista.com?
Yes. statista.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.