Who blocks AI · #1105 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does repubblica.it block AI crawlers?
Yes. repubblica.it’s robots.txt blocks 9 of the 19 AI crawlers we track, including GPTBot, Google-Extended, PerplexityBot and 6 more. It also blocks Applebot and PetalBot among search engines.
GPTBotBlockedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.repubblica.it/robots.txt · 287 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | line 2 · Allow: / under User-agent: * | |
Bingbot | Microsoft | Allowed | line 2 · Allow: / under User-agent: * |
DuckDuckBot | DuckDuckGo | Allowed | line 2 · Allow: / under User-agent: * |
Applebot | Apple | Blocked | line 284 · Disallow: / under User-agent: AppleBot |
PetalBot | Huawei | Blocked | line 88 · Disallow: / under User-agent: PetalBot |
YandexBot | Yandex | Allowed | line 2 · Allow: / under User-agent: * |
Baiduspider | Baidu | Allowed | line 2 · Allow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | line 2 · Allow: / under User-agent: * |
PerplexityBot | Perplexity | Blocked | line 92 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | line 2 · Allow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Blocked | line 138 · Disallow: / under User-agent: DuckAssistBot |
Amazonbot | Amazon | Allowed | line 94 · Allow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 280 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 74 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Allowed | line 2 · Allow: / under User-agent: * |
Google-Extended | Blocked | line 76 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 78 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 286 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | line 2 · Allow: / under User-agent: * | |
Meta-ExternalAgent | Meta | Allowed | line 2 · Allow: / under User-agent: * |
Bytespider | ByteDance | Blocked | line 114 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 2 · Allow: / under User-agent: * |
Claude-User | Anthropic | Allowed | line 2 · Allow: / under User-agent: * |
Perplexity-User | Perplexity | Allowed | line 2 · Allow: / under User-agent: * |
MistralAI-User | Mistral | Allowed | line 2 · Allow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Blocked | line 186 · Disallow: / under User-agent: meta-externalfetcher |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 96 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Blocked | line 238 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | line 2 · Allow: / under User-agent: * |
DotBot | Moz | Blocked | line 134 · Disallow: / under User-agent: dotbot |
View www.repubblica.it/robots.txt · 287 lines
1User-agent: *
2Allow: /
3Disallow: /ultimora/24ore/nazionale/news-dettaglio/3688417
4Disallow: /ultimora/24ore/AUTO-IN-CAMBIO-DI-DROGA-13-ARRESTI-DELLA-POLIZIA-DI-PESCARA-(2)/news-dettaglio/3746479
5Disallow: /sport/ciclismo/2023/10/24/news/doping_ciclismo_virtuale_azzurro_zanasca_rischia_una_lunga_squalifica-418660589/
6Disallow: /spettacoli/people/2023/09/07/news/kanye_west_e_bianca_censori_in_italia_la_lunga_estate_calda_degli_scandali-413629176/<
7Disallow: /speciale/
8Disallow: /social/sites/repubblica/d/boxes/shares/
9Disallow: /servizi/
10Disallow: /ricerca/
11Disallow: /politica/2017/01/19/news/il_fratello_del_ministro_alfano_rischia_il_posto_e_costera_all_erario_mezzo_milione_di_euro[1]156371739/
12Disallow: /persone/
13Disallow: /online/cronaca/veronaecstasy/bologna/bologna.html
14Disallow: /online/
15Disallow: /news/ired/ultimora/cronaca/rep_cronaca_n_3487677.html
16Disallow: /java
17Disallow: /ipad/
18Disallow: /interstitial/
19Disallow: /indici/
20Disallow: /il-gusto/2022/04/27/news/frode_fiscale_bancarotta_fraudolenta_arrestato_giovanni_maspero_ristorante_i_tigli_in_theoria-347110162/
21Disallow: /embed/*
22Disallow: /economia/miojob/
23Disallow: /economia/2013/01/31/news/mps_verso_sequestro_conservativo- 51615470/
24Disallow: /dal-quotidiano/
25Disallow: /corporate/privacy/index_static.html?
26Disallow: /cgi-bin/back.cgi
27Disallow: /blaize/datalayer
28Disallow: /audio/i-daily-di-repubblica
29Disallow: /audio-test-app
30Disallow: /api/checkpw-v1/premium/amp/pingback*
31Disallow: /api/*
32Disallow: /annunci/
33Disallow: /2010/
34Disallow: /2009/
35Disallow: /2008/
36Disallow: /2007/
37Disallow: /2006/05/gallerie/politica/farabutti-foto-lettori-95/esterne191030071909103240_big.jpg
38Disallow: /2006/05/gallerie/politica/farabutti-foto-lettori-95/esterne191030071909103240.jpg
39Disallow: /2006/
40Disallow: /2005/
41Disallow: /2004/
42Disallow: /2003/
43Disallow: /coccodrilli/
44Disallow: /swf
45Disallow: /serietv/schede/cerca/?query=
46Disallow: /online3
47Disallow: /mobile/
48Disallow: /ztest/
49Disallow: /dossier/sapori/guide-espresso*
50Disallow: /dossier/motori*
51Disallow: /topics/*?p=
52Disallow: /mobile-rep/
53Disallow: /tv/
54Disallow: /static/__sdemo__/
55Disallow: /quotidiano
56#
57Sitemap: https://www.repubblica.it/sitemap-n.xml
58#
59Sitemap: https://video.repubblica.it/sitemap-v-day.xml
60Sitemap: https://video.repubblica.it/sitemap-v.xml
61Sitemap: https://www.repubblica.it/sitemap-italian-tech-n.xml
62Sitemap: https://www.repubblica.it/sitemap-il-gusto-n.xml
63Sitemap: https://www.repubblica.it/sitemap-salute-n.xml
64Sitemap: https://www.repubblica.it/sitemap-green-and-blue-n.xml
65#
66Sitemap: https://www.repubblica.it/sitemap-dirette-sport-n.xml
67#
68Sitemap: https://www.repubblica.it/sitemap.xml
69Sitemap: https://www.repubblica.it/sitemap-italian-tech.xml
70Sitemap: https://www.repubblica.it/sitemap-il-gusto.xml
71Sitemap: https://www.repubblica.it/sitemap-salute.xml
72Sitemap: https://www.repubblica.it/sitemap-green-and-blue.xml
73User-agent: GPTBot
74Disallow: /
75User-agent: Google-Extended
76Disallow: /
77User-agent: CCBot
78Disallow: /
79User-agent: anthropic-ai
80Disallow: /
81User-agent: Omgilibot
82Disallow: /
83User-agent: FacebookBot
84Disallow: /
85User-agent: cohere-ai
86Disallow: /
87User-agent: PetalBot
88Disallow: /
89User-agent: Yandex
90Disallow: /
91User-agent: PerplexityBot
92Disallow: /
93User-agent: Amazonbot
94Allow: /
95User-agent: AhrefsBot
96Disallow: /
97User-agent: archive.org_bot
98Disallow: /
99User-agent: ArchiveBot
100Disallow: /
101User-agent: Arquivo-web-crawler
102Disallow: /
103User-agent: AwarioRssBot
104Disallow: /
105User-agent: AwarioSmartBot
106Disallow: /
107User-agent: BacklinkCrawler
108Disallow: /
109User-agent: BIXOCRAWLER
110Disallow: /
111User-agent: BLEXBot
112Disallow: /
113User-agent: Bytespider
114Disallow: /
115User-agent: Cliqzbot
116Disallow: /
117User-agent: ConveraCrawler
118Disallow: /
119User-agent: crawler4j
120Disallow: /
121User-agent: CrystalSemanticsBot
122Disallow: /
123User-agent: DataForSeoBot
124Disallow: /
125User-agent: Diffbot
126Disallow: /
127User-agent: discobot
128Disallow: /
129User-agent: discoverybot
130Disallow: /
131User-agent: dloader(NaverRobot)
132Disallow: /
133User-agent: dotbot
134Disallow: /
135User-agent: Download Ninja
136Disallow: /
137User-agent: DuckAssistBot
138Disallow: /
139User-agent: europarchive.org
140Disallow: /
141User-agent: Exabot
142Disallow: /
143User-agent: ExtractorPro
144Disallow: /
145User-agent: Fetch
146Disallow: /
147User-agent: Flamingo_SearchEngine
148Disallow: /
149User-agent: FriendlyCrawler
150Disallow: /
151User-agent: Gigabot
152Disallow: /
153User-agent: GumGum Bot
154Disallow: /
155User-agent: ia_archiver
156Disallow: /
157User-agent: ia_archiver-web.archive.org
158Disallow: /
159User-agent: ImagesiftBot
160Disallow: /
161User-agent: IstellaBot
162Disallow: /
163User-agent: Jetbot
164Disallow: /
165User-agent: JikeSpider
166Disallow: /
167User-agent: Jyxobot
168Disallow: /
169User-agent: k2spider
170Disallow: /
171User-agent: Kangaroo Bot
172Disallow: /
173User-agent: kbcrawl
174Disallow: /
175User-agent: LexxeBot/1.0
176Disallow: /
177User-agent: LinkArchiver
178Disallow: /
179User-agent: LinkextractorPro
180Disallow: /
181User-agent: Livelapbot
182Disallow: /
183User-agent: magpie-crawler
184Disallow: /
185User-agent: meta-externalfetcher
186Disallow: /
187User-agent: MJ12bot
188Disallow: /
189User-agent: Moreoverbot
190Disallow: /
191User-agent: MSIECrawler
192Disallow: /
193User-agent: nabot
194Disallow: /
195User-agent: NaverBot
196Disallow: /
197User-agent: NerdByNature.Bot
198Disallow: /
199User-agent: netEstate NE Crawler
200Disallow: /
201User-agent: nNetSeer crawler
202Disallow: /
203User-agent: NextGenSearchBot
204Disallow: /
205User-agent: Nicecrawler
206Disallow: /
207User-agent: NPBot
208Disallow: /
209User-agent: nutch
210Disallow: /
211User-agent: Openbot
212Disallow: /
213User-agent: PanguBot
214Disallow: /
215User-agent: peer39_crawler
216Disallow: /
217User-agent: peer39_crawler/1.0
218Disallow: /
219User-agent: PiplBot
220Disallow: /
221User-agent: Pixray-Seeker
222Disallow: /
223User-agent: Primalbot
224Disallow: /
225User-agent: psbot
226Disallow: /
227User-agent: QuerySeekerSpider
228Disallow: /
229User-agent: Quora-Bot
230Disallow: /
231User-agent: rogerbot
232Disallow: /
233User-agent: Scrapy
234Disallow: /
235User-agent: searchpreview
236Disallow: /
237User-agent: SemrushBot
238Disallow: /
239User-agent: SEOENGBot
240Disallow: /
241User-agent: SEOkicks-Robot
242Disallow: /
243User-agent: SiteBot
244Disallow: /
245User-agent: SiteBot/0.1
246Disallow: /
247User-agent: Slurp
248Disallow: /
249User-agent: Sosospider
250Disallow: /
251User-agent: spbot
252Disallow: /
253User-agent: Timpibot
254Disallow: /
255User-agent: trendictionbot
256Disallow: /
257User-agent: trovitBot
258Disallow: /
259User-agent: True_Robot
260Disallow: /
261User-agent: TurnitinBot
262Disallow: /
263User-agent: UbiCrawler
264Disallow: /
265User-agent: uMBot-LN
266Disallow: /
267User-agent: URL_Spider_Pro
268Disallow: /
269User-agent: verticalsearch
270Disallow: /
271User-agent: WBSearchBot
272Disallow: /
273User-agent: web-archive-net.com.bot
274Disallow: /
275User-agent: WeSEE:Search
276Disallow: /
277User-agent: Wotbot
278Disallow: /
279User-agent: YouBot
280Disallow: /
281User-agent: Zealbot
282Disallow: /
283User-agent: AppleBot
284Disallow: /
285User-agent: Applebot-Extended
286Disallow: /
287
What do the bots actually get?
This page reads robots.txt, which states what repubblica.it asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on repubblica.it Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does repubblica.it block GPTBot?
Yes. repubblica.it’s robots.txt disallows GPTBot from the whole site (line 74: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does repubblica.it block ClaudeBot?
No. repubblica.it’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does repubblica.it block Google-Extended?
Yes. repubblica.it’s robots.txt disallows Google-Extended from the whole site (line 76: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does repubblica.it block PerplexityBot?
Yes. repubblica.it’s robots.txt disallows PerplexityBot from the whole site (line 92: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find repubblica.it?
Yes. repubblica.it’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.