Who blocks AI · #12 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does wikipedia.org block AI crawlers?
No. wikipedia.org’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- en.wikipedia.org/robots.txt · 712 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Partly blocked | line 153 · Disallow: /w/ line 154 · Disallow: /api/ and 426 more |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View en.wikipedia.org/robots.txt · 712 lines
1# robots.txt for http://www.wikipedia.org/ and friends
2#
3# Please note: There are a lot of pages on this site, and there are
4# some misbehaved spiders out there that go _way_ too fast. If you're
5# irresponsible, your access to the site may be blocked.
6#
7
8# Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN
9# and ignoring 429 ratelimit responses, claims to respect robots:
10# http://mj12bot.com/
11User-agent: MJ12bot
12Disallow: /
13
14# advertising-related bots:
15User-agent: Mediapartners-Google*
16Disallow: /
17
18# Wikipedia work bots:
19User-agent: IsraBot
20Disallow:
21
22User-agent: Orthogaffe
23Disallow:
24
25# Crawlers that are kind enough to obey, but which we'd rather not have
26# unless they're feeding search engines.
27User-agent: UbiCrawler
28Disallow: /
29
30User-agent: DOC
31Disallow: /
32
33User-agent: Zao
34Disallow: /
35
36# Some bots are known to be trouble, particularly those designed to copy
37# entire sites. Please obey robots.txt.
38User-agent: sitecheck.internetseer.com
39Disallow: /
40
41User-agent: Zealbot
42Disallow: /
43
44User-agent: MSIECrawler
45Disallow: /
46
47User-agent: SiteSnagger
48Disallow: /
49
50User-agent: WebStripper
51Disallow: /
52
53User-agent: WebCopier
54Disallow: /
55
56User-agent: Fetch
57Disallow: /
58
59User-agent: Offline Explorer
60Disallow: /
61
62User-agent: Teleport
63Disallow: /
64
65User-agent: TeleportPro
66Disallow: /
67
68User-agent: WebZIP
69Disallow: /
70
71User-agent: linko
72Disallow: /
73
74User-agent: HTTrack
75Disallow: /
76
77User-agent: Microsoft.URL.Control
78Disallow: /
79
80User-agent: Xenu
81Disallow: /
82
83User-agent: larbin
84Disallow: /
85
86User-agent: libwww
87Disallow: /
88
89User-agent: ZyBORG
90Disallow: /
91
92User-agent: Download Ninja
93Disallow: /
94
95# Misbehaving, requests much too fast
96User-agent: fast
97Disallow: /
98
99# Sorry, wget in its recursive mode is a frequent problem.
100# Please read the man page and use it properly; there is a
101# --wait option you can use to set the delay between hits,
102# for instance.
103User-agent: wget
104Disallow: /
105
106# The 'grub' distributed client has been *very* poorly behaved.
107User-agent: grub-client
108Disallow: /
109
110# Doesn't follow robots.txt anyway, but...
111User-agent: k2spider
112Disallow: /
113
114# Hits many times per second, not acceptable
115# http://www.nameprotect.com/botinfo.html
116User-agent: NPBot
117Disallow: /
118
119# A capture bot, downloads gazillions of pages with no public benefit
120# http://www.webreaper.net/
121User-agent: WebReaper
122Disallow: /
123
124# Per their statement, semrushbot respects crawl-delay directives
125# We want them to overall stay within reasonable request rates to
126# the backend (20 rps); keeping in mind that the crawl-delay will
127# be applied by site and not globally by the bot, 5 seconds seem
128# like a reasonable approximation
129User-agent: SemrushBot
130Crawl-delay: 5
131
132#
133# Friendly, low-speed bots are welcome viewing article pages, but not
134# dynamically-generated pages please.
135#
136# Inktomi's "Slurp" can read a minimum delay between hits; if your
137# bot supports such a thing using the 'Crawl-delay' or another
138# instruction, please let us know.
139#
140# There is a special exception for API mobileview to allow dynamic
141# mobile web & app views to load section content.
142# These views aren't HTTP-cached but use parser cache aggressively
143# and don't expose Special-pages and other uncached content.
144#
145# Another exception is for REST API documentation, located at
146# /api/rest_v1/?doc.
147#
148User-agent: *
149Allow: /w/api.php?action=mobileview&
150Allow: /w/load.php?
151Allow: /api/rest_v1/?doc
152Allow: /w/rest.php/site/v1/sitemap
153Disallow: /w/
154Disallow: /api/
155Disallow: /trap/
156Disallow: /wiki/Special:
157Disallow: /wiki/Spezial:
158Disallow: /wiki/Spesial:
159Disallow: /wiki/Special%3A
160Disallow: /wiki/Spezial%3A
161Disallow: /wiki/Spesial%3A
162#
163# ar:
164Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search
165Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch
166#
167# dewiki:
168# T6937
169# sensible deletion and meta user discussion pages:
170Disallow: /wiki/Wikipedia:L%C3%B6schkandidaten/
171Disallow: /wiki/Wikipedia:Löschkandidaten/
172Disallow: /wiki/Wikipedia:Vandalensperrung/
173Disallow: /wiki/Wikipedia:Benutzersperrung/
174Disallow: /wiki/Wikipedia:Vermittlungsausschuss/
175Disallow: /wiki/Wikipedia:Administratoren/Probleme/
176Disallow: /wiki/Wikipedia:Adminkandidaturen/
177Disallow: /wiki/Wikipedia:Qualitätssicherung/
178Disallow: /wiki/Wikipedia:Qualit%C3%A4tssicherung/
179# 4937#5
180Disallow: /wiki/Wikipedia:Vandalismusmeldung/
181Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/
182Disallow: /wiki/Wikipedia:Löschprüfung/
183Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/
184Disallow: /wiki/Wikipedia:Administratoren/Notizen/
185Disallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/
186Disallow: /wiki/Wikipedia:L%C3%B6schpr%C3%BCfung/
187# T14111
188Disallow: /wiki/Wikipedia:Checkuser/
189Disallow: /wiki/Wikipedia_Diskussion:Checkuser/
190Disallow: /wiki/Wikipedia_Diskussion:Adminkandidaturen/
191# T15961
192Disallow: /wiki/Wikipedia:Spam-Blacklist-Log
193Disallow: /wiki/Wikipedia%3ASpam-Blacklist-Log
194Disallow: /wiki/Wikipedia_Diskussion:Spam-Blacklist-Log
195Disallow: /wiki/Wikipedia_Diskussion%3ASpam-Blacklist-Log
196#
197# enwiki:
198# Folks get annoyed when VfD discussions end up the number 1 google hit for
199# their name. See T6776
200Disallow: /wiki/Wikipedia:Articles_for_deletion/
201Disallow: /wiki/Wikipedia%3AArticles_for_deletion/
202Disallow: /wiki/Wikipedia:Votes_for_deletion/
203Disallow: /wiki/Wikipedia%3AVotes_for_deletion/
204Disallow: /wiki/Wikipedia:Pages_for_deletion/
205Disallow: /wiki/Wikipedia%3APages_for_deletion/
206Disallow: /wiki/Wikipedia:Miscellany_for_deletion/
207Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/
208Disallow: /wiki/Wikipedia:Miscellaneous_deletion/
209Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/
210Disallow: /wiki/Wikipedia:Copyright_problems
211Disallow: /wiki/Wikipedia%3ACopyright_problems
212Disallow: /wiki/Wikipedia:Protected_titles/
213Disallow: /wiki/Wikipedia%3AProtected_titles/
214# T15398
215Disallow: /wiki/Wikipedia:WikiProject_Spam/
216Disallow: /wiki/Wikipedia%3AWikiProject_Spam/
217# T16075
218Disallow: /wiki/MediaWiki:Spam-blacklist
219Disallow: /wiki/MediaWiki%3ASpam-blacklist
220Disallow: /wiki/MediaWiki_talk:Spam-blacklist
221Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist
222# T13261
223Disallow: /wiki/Wikipedia:Requests_for_arbitration/
224Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/
225Disallow: /wiki/Wikipedia:Requests_for_comment/
226Disallow: /wiki/Wikipedia%3ARequests_for_comment/
227Disallow: /wiki/Wikipedia:Requests_for_adminship/
228Disallow: /wiki/Wikipedia%3ARequests_for_adminship/
229# T12288
230Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/
231Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/
232Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/
233Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/
234Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/
235Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/
236Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/
237Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/
238Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/
239Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/
240# T16793
241Disallow: /wiki/Wikipedia:Changing_username
242Disallow: /wiki/Wikipedia%3AChanging_username
243Disallow: /wiki/Wikipedia:Changing_username/
244Disallow: /wiki/Wikipedia%3AChanging_username/
245Disallow: /wiki/Wikipedia_talk:Changing_username
246Disallow: /wiki/Wikipedia_talk%3AChanging_username
247Disallow: /wiki/Wikipedia_talk:Changing_username/
248Disallow: /wiki/Wikipedia_talk%3AChanging_username/
249#
250# eswiki:
251# T8746
252Disallow: /wiki/Wikipedia:Consultas_de_borrado/
253Disallow: /wiki/Wikipedia%3AConsultas_de_borrado/
254#
255# fiwiki:
256# T10695
257Disallow: /wiki/Wikipedia:Poistettavat_sivut
258Disallow: /wiki/K%C3%A4ytt%C3%A4j%C3%A4:
259Disallow: /wiki/Käyttäjä:
260Disallow: /wiki/Keskustelu_k%C3%A4ytt%C3%A4j%C3%A4st%C3%A4:
261Disallow: /wiki/Keskustelu_käyttäjästä:
262Disallow: /wiki/Wikipedia:Yll%C3%A4pit%C3%A4j%C3%A4t/
263Disallow: /wiki/Wikipedia:Ylläpitäjät/
264#
265# hewiki:
266Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93:Search
267Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93%3ASearch
268#T11517
269Disallow: /wiki/ויקיפדיה:רשימת_מועמדים_למחיקה/
270Disallow: /wiki/ויקיפדיה%3Aרשימת_מועמדים_למחיקה/
271Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/
272Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/
273Disallow: /wiki/ויקיפדיה:ערכים_לא_קיימים_ומוגנים
274Disallow: /wiki/ויקיפדיה%3Aערכים_לא_קיימים_ומוגנים
275Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
276Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
277Disallow: /wiki/ויקיפדיה:דפים_לא_קיימים_ומוגנים
278Disallow: /wiki/ויקיפדיה%3Aדפים_לא_קיימים_ומוגנים
279Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
280Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
281#
282# huwiki:
283Disallow: /wiki/Speci%C3%A1lis:Search
284Disallow: /wiki/Speci%C3%A1lis%3ASearch
285#
286# itwiki:
287# T7545
288Disallow: /wiki/Wikipedia:Pagine_da_cancellare
289Disallow: /wiki/Wikipedia%3APagine_da_cancellare
290Disallow: /wiki/Wikipedia:Utenti_problematici
291Disallow: /wiki/Wikipedia%3AUtenti_problematici
292Disallow: /wiki/Wikipedia:Vandalismi_in_corso
293Disallow: /wiki/Wikipedia%3AVandalismi_in_corso
294Disallow: /wiki/Wikipedia:Amministratori
295Disallow: /wiki/Wikipedia%3AAmministratori
296Disallow: /wiki/Wikipedia:Proposte_di_cancellazione_semplificata
297Disallow: /wiki/Wikipedia%3AProposte_di_cancellazione_semplificata
298Disallow: /wiki/Categoria:Da_cancellare_subito
299Disallow: /wiki/Categoria%3ADa_cancellare_subito
300Disallow: /wiki/Wikipedia:Sospette_violazioni_di_copyright
301Disallow: /wiki/Wikipedia%3ASospette_violazioni_di_copyright
302Disallow: /wiki/Categoria:Da_controllare_per_copyright
303Disallow: /wiki/Categoria%3ADa_controllare_per_copyright
304Disallow: /wiki/Progetto:Rimozione_contributi_sospetti
305Disallow: /wiki/Progetto%3ARimozione_contributi_sospetti
306Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright
307Disallow: /wiki/Categoria%3ADa_cancellare_subito_per_violazione_integrale_copyright
308Disallow: /wiki/Progetto:Cococo
309Disallow: /wiki/Progetto%3ACococo
310Disallow: /wiki/Discussioni_progetto:Cococo
311Disallow: /wiki/Discussioni_progetto%3ACococo
312#
313# jawiki
314Disallow: /wiki/%E7%89%B9%E5%88%A5:Search
315Disallow: /wiki/%E7%89%B9%E5%88%A5%3ASearch
316# T7239
317Disallow: /wiki/Wikipedia:%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/
318Disallow: /wiki/Wikipedia%3A%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/
319Disallow: /wiki/Wikipedia:%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC
320Disallow: /wiki/Wikipedia%3A%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC
321# nowiki
322# T13432
323Disallow: /wiki/Bruker:
324Disallow: /wiki/Bruker%3A
325Disallow: /wiki/Brukerdiskusjon
326Disallow: /wiki/Wikipedia:Administratorer
327Disallow: /wiki/Wikipedia%3AAdministratorer
328Disallow: /wiki/Wikipedia-diskusjon:Administratorer
329Disallow: /wiki/Wikipedia-diskusjon%3AAdministratorer
330Disallow: /wiki/Wikipedia:Sletting
331Disallow: /wiki/Wikipedia%3ASletting
332Disallow: /wiki/Wikipedia-diskusjon:Sletting
333Disallow: /wiki/Wikipedia-diskusjon%3ASletting
334#
335# plwiki
336# T10067
337Disallow: /wiki/Wikipedia:Strony_do_usuni%C4%99cia
338Disallow: /wiki/Wikipedia%3AStrony_do_usuni%C4%99cia
339Disallow: /wiki/Wikipedia:Do_usuni%C4%99cia
340Disallow: /wiki/Wikipedia%3ADo_usuni%C4%99cia
341Disallow: /wiki/Wikipedia:SDU/
342Disallow: /wiki/Wikipedia%3ASDU/
343Disallow: /wiki/Wikipedia:Strony_podejrzane_o_naruszenie_praw_autorskich
344Disallow: /wiki/Wikipedia%3AStrony_podejrzane_o_naruszenie_praw_autorskich
345#
346# ptwiki:
347# T7394
348Disallow: /wiki/Wikipedia:Páginas_para_eliminar/
349Disallow: /wiki/Wikipedia:P%C3%A1ginas_para_eliminar/
350Disallow: /wiki/Wikipedia%3AP%C3%A1ginas_para_eliminar/
351Disallow: /wiki/Wikipedia_Discussão:Páginas_para_eliminar/
352Disallow: /wiki/Wikipedia_Discuss%C3%A3o:P%C3%A1ginas_para_eliminar/
353Disallow: /wiki/Wikipedia_Discuss%C3%A3o%3AP%C3%A1ginas_para_eliminar/
354#
355# rowiki:
356# T14546
357Disallow: /wiki/Wikipedia:Pagini_de_%C5%9Fters
358Disallow: /wiki/Wikipedia%3APagini_de_%C5%9Fters
359Disallow: /wiki/Discu%C5%A3ie_Wikipedia:Pagini_de_%C5%9Fters
360Disallow: /wiki/Discu%C5%A3ie_Wikipedia%3APagini_de_%C5%9Fters
361#
362# ruwiki:
363Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5:Search
364Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5%3ASearch
365#
366# svwiki:
367# T12229
368Disallow: /wiki/Wikipedia%3ASidor_f%C3%B6reslagna_f%C3%B6r_radering
369Disallow: /wiki/Wikipedia:Sidor_f%C3%B6reslagna_f%C3%B6r_radering
370Disallow: /wiki/Wikipedia:Sidor_föreslagna_för_radering
371Disallow: /wiki/Användare
372Disallow: /wiki/Anv%C3%A4ndare
373Disallow: /wiki/Användardiskussion
374Disallow: /wiki/Anv%C3%A4ndardiskussion
375Disallow: /wiki/Wikipedia:Skyddade_sidnamn
376Disallow: /wiki/Wikipedia%3ASkyddade_sidnamn
377# T13291
378Disallow: /wiki/Wikipedia:Sidor_som_bör_raderas
379Disallow: /wiki/Wikipedia:Sidor_som_b%C3%B6r_raderas
380Disallow: /wiki/Wikipedia%3ASidor_som_b%C3%B6r_raderas
381#
382# zhwiki:
383# T7104
384Disallow: /wiki/Wikipedia:删除投票/侵权
385Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8/%E4%BE%B5%E6%9D%83
386Disallow: /wiki/Wikipedia:删除投票和请求
387Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8%E5%92%8C%E8%AF%B7%E6%B1%82
388Disallow: /wiki/Category:快速删除候选
389Disallow: /wiki/Category:%E5%BF%AB%E9%80%9F%E5%88%A0%E9%99%A4%E5%80%99%E9%80%89
390Disallow: /wiki/Category:维基百科需要翻译的文章
391Disallow: /wiki/Category:%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E9%9C%80%E8%A6%81%E7%BF%BB%E8%AF%91%E7%9A%84%E6%96%87%E7%AB%A0
392#
393# sister projects
394#
395# enwikinews:
396# T7340
397Disallow: /wiki/Portal:Prepared_stories/
398Disallow: /wiki/Portal%3APrepared_stories/
399#
400# itwikinews
401# T11138
402Disallow: /wiki/Wikinotizie:Richieste_di_cancellazione
403Disallow: /wiki/Wikinotizie:Sospette_violazioni_di_copyright
404Disallow: /wiki/Categoria:Da_cancellare_subito
405Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright
406Disallow: /wiki/Wikinotizie:Storie_in_preparazione
407#
408# enwikiquote:
409# T17095
410Disallow: /wiki/Wikiquote:Votes_for_deletion/
411Disallow: /wiki/Wikiquote%3AVotes_for_deletion/
412Disallow: /wiki/Wikiquote_talk:Votes_for_deletion/
413Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion/
414Disallow: /wiki/Wikiquote:Votes_for_deletion_archive/
415Disallow: /wiki/Wikiquote%3AVotes_for_deletion_archive/
416Disallow: /wiki/Wikiquote_talk:Votes_for_deletion_archive/
417Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion_archive/
418#
419# enwikibooks
420Disallow: /wiki/Wikibooks:Votes_for_deletion
421#
422
423
424Sitemap: https://en.wikipedia.org/w/rest.php/site/v1/sitemap/0
425
426#
427#
428#----------------------------------------------------------#
429#
430#
431#
432#
433# Localisable part of robots.txt for en.wikipedia.org
434#
435# Edit at https://en.wikipedia.org/w/index.php?title=MediaWiki:Robots.txt&action=edit
436# Don't add newlines here. All rules set here are active for every user-agent.
437#
438# Please check any changes using a syntax validator
439# Enter https://en.wikipedia.org/robots.txt as the URL to check.
440#
441# https://phabricator.wikimedia.org/T16075
442Disallow: /wiki/MediaWiki:Spam-blacklist
443Disallow: /wiki/MediaWiki%3ASpam-blacklist
444Disallow: /wiki/MediaWiki_talk:Spam-blacklist
445Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist
446Disallow: /wiki/Wikipedia:WikiProject_Spam
447Disallow: /wiki/Wikipedia_talk:WikiProject_Spam
448#
449# Folks get annoyed when XfD discussions end up the number 1 google hit for
450# their name.
451# https://phabricator.wikimedia.org/T16075
452Disallow: /wiki/Wikipedia:Articles_for_deletion
453Disallow: /wiki/Wikipedia%3AArticles_for_deletion
454Disallow: /wiki/Wikipedia:Votes_for_deletion
455Disallow: /wiki/Wikipedia%3AVotes_for_deletion
456Disallow: /wiki/Wikipedia:Pages_for_deletion
457Disallow: /wiki/Wikipedia%3APages_for_deletion
458Disallow: /wiki/Wikipedia:Miscellany_for_deletion
459Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion
460Disallow: /wiki/Wikipedia:Miscellaneous_deletion
461Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion
462Disallow: /wiki/Wikipedia:Categories_for_discussion
463Disallow: /wiki/Wikipedia%3ACategories_for_discussion
464Disallow: /wiki/Wikipedia:Templates_for_deletion
465Disallow: /wiki/Wikipedia%3ATemplates_for_deletion
466Disallow: /wiki/Wikipedia:Redirects_for_discussion
467Disallow: /wiki/Wikipedia%3ARedirects_for_discussion
468Disallow: /wiki/Wikipedia:Deletion_review
469Disallow: /wiki/Wikipedia%3ADeletion_review
470Disallow: /wiki/Wikipedia:WikiProject_Deletion_sorting
471Disallow: /wiki/Wikipedia%3AWikiProject_Deletion_sorting
472Disallow: /wiki/Wikipedia:Files_for_deletion
473Disallow: /wiki/Wikipedia%3AFiles_for_deletion
474Disallow: /wiki/Wikipedia:Files_for_discussion
475Disallow: /wiki/Wikipedia%3AFiles_for_discussion
476Disallow: /wiki/Wikipedia:Possibly_unfree_files
477Disallow: /wiki/Wikipedia%3APossibly_unfree_files
478#
479# https://phabricator.wikimedia.org/T12288
480Disallow: /wiki/Wikipedia_talk:Articles_for_deletion
481Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion
482Disallow: /wiki/Wikipedia_talk:Votes_for_deletion
483Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion
484Disallow: /wiki/Wikipedia_talk:Pages_for_deletion
485Disallow: /wiki/Wikipedia_talk%3APages_for_deletion
486Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion
487Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion
488Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion
489Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion
490Disallow: /wiki/Wikipedia_talk:Templates_for_deletion
491Disallow: /wiki/Wikipedia_talk%3ATemplates_for_deletion
492Disallow: /wiki/Wikipedia_talk:Categories_for_discussion
493Disallow: /wiki/Wikipedia_talk%3ACategories_for_discussion
494Disallow: /wiki/Wikipedia_talk:Deletion_review
495Disallow: /wiki/Wikipedia_talk%3ADeletion_review
496Disallow: /wiki/Wikipedia_talk:WikiProject_Deletion_sorting
497Disallow: /wiki/Wikipedia_talk%3AWikiProject_Deletion_sorting
498Disallow: /wiki/Wikipedia_talk:Files_for_deletion
499Disallow: /wiki/Wikipedia_talk%3AFiles_for_deletion
500Disallow: /wiki/Wikipedia_talk:Files_for_discussion
501Disallow: /wiki/Wikipedia_talk%3AFiles_for_discussion
502Disallow: /wiki/Wikipedia_talk:Possibly_unfree_files
503Disallow: /wiki/Wikipedia_talk%3APossibly_unfree_files
504#
505Disallow: /wiki/Wikipedia:Copyright_problems
506Disallow: /wiki/Wikipedia%3ACopyright_problems
507Disallow: /wiki/Wikipedia_talk:Copyright_problems
508Disallow: /wiki/Wikipedia_talk%3ACopyright_problems
509Disallow: /wiki/Wikipedia:Suspected_copyright_violations
510Disallow: /wiki/Wikipedia%3ASuspected_copyright_violations
511Disallow: /wiki/Wikipedia_talk:Suspected_copyright_violations
512Disallow: /wiki/Wikipedia_talk%3ASuspected_copyright_violations
513Disallow: /wiki/Wikipedia:Contributor_copyright_investigations
514Disallow: /wiki/Wikipedia%3AContributor_copyright_investigations
515Disallow: /wiki/Wikipedia:Contributor_copyright_investigations
516Disallow: /wiki/Wikipedia%3AContributor_copyright_investigations
517Disallow: /wiki/Wikipedia_talk:Contributor_copyright_investigations
518Disallow: /wiki/Wikipedia_talk%3AContributor_copyright_investigations
519Disallow: /wiki/Wikipedia_talk:Contributor_copyright_investigations
520Disallow: /wiki/Wikipedia_talk%3AContributor_copyright_investigations
521Disallow: /wiki/Wikipedia:Protected_titles
522Disallow: /wiki/Wikipedia%3AProtected_titles
523Disallow: /wiki/Wikipedia_talk:Protected_titles
524Disallow: /wiki/Wikipedia_talk%3AProtected_titles
525Disallow: /wiki/Wikipedia:Articles_for_creation
526Disallow: /wiki/Wikipedia%3AArticles_for_creation
527Disallow: /wiki/Wikipedia_talk:Articles_for_creation
528Disallow: /wiki/Wikipedia_talk%3AArticles_for_creation
529Disallow: /wiki/Wikipedia_talk:Article_wizard
530Disallow: /wiki/Wikipedia_talk%3AArticle_wizard
531#
532# https://phabricator.wikimedia.org/T13261
533Disallow: /wiki/Wikipedia:Requests_for_arbitration
534Disallow: /wiki/Wikipedia%3ARequests_for_arbitration
535Disallow: /wiki/Wikipedia_talk:Requests_for_arbitration
536Disallow: /wiki/Wikipedia_talk%3ARequests_for_arbitration
537Disallow: /wiki/Wikipedia:Requests_for_comment
538Disallow: /wiki/Wikipedia%3ARequests_for_comment
539Disallow: /wiki/Wikipedia_talk:Requests_for_comment
540Disallow: /wiki/Wikipedia_talk%3ARequests_for_comment
541Disallow: /wiki/Wikipedia:Requests_for_adminship
542Disallow: /wiki/Wikipedia%3ARequests_for_adminship
543Disallow: /wiki/Wikipedia_talk:Requests_for_adminship
544Disallow: /wiki/Wikipedia_talk%3ARequests_for_adminship
545#
546# https://phabricator.wikimedia.org/T14111
547Disallow: /wiki/Wikipedia:Requests_for_checkuser
548Disallow: /wiki/Wikipedia%3ARequests_for_checkuser
549Disallow: /wiki/Wikipedia_talk:Requests_for_checkuser
550Disallow: /wiki/Wikipedia_talk%3ARequests_for_checkuser
551#
552# https://phabricator.wikimedia.org/T15398
553Disallow: /wiki/Wikipedia:WikiProject_Spam
554Disallow: /wiki/Wikipedia%3AWikiProject_Spam
555#
556# https://phabricator.wikimedia.org/T16793
557Disallow: /wiki/Wikipedia:Changing_username
558Disallow: /wiki/Wikipedia%3AChanging_username
559Disallow: /wiki/Wikipedia:Changing_username
560Disallow: /wiki/Wikipedia%3AChanging_username
561Disallow: /wiki/Wikipedia_talk:Changing_username
562Disallow: /wiki/Wikipedia_talk%3AChanging_username
563Disallow: /wiki/Wikipedia_talk:Changing_username
564Disallow: /wiki/Wikipedia_talk%3AChanging_username
565#
566Disallow: /wiki/Wikipedia:Administrators%27_noticeboard
567Disallow: /wiki/Wikipedia%3AAdministrators%27_noticeboard
568Disallow: /wiki/Wikipedia_talk:Administrators%27_noticeboard
569Disallow: /wiki/Wikipedia_talk%3AAdministrators%27_noticeboard
570Disallow: /wiki/Wikipedia:Community_sanction_noticeboard
571Disallow: /wiki/Wikipedia%3ACommunity_sanction_noticeboard
572Disallow: /wiki/Wikipedia_talk:Community_sanction_noticeboard
573Disallow: /wiki/Wikipedia_talk%3ACommunity_sanction_noticeboard
574Disallow: /wiki/Wikipedia:Bureaucrats%27_noticeboard
575Disallow: /wiki/Wikipedia%3ABureaucrats%27_noticeboard
576Disallow: /wiki/Wikipedia_talk:Bureaucrats%27_noticeboard
577Disallow: /wiki/Wikipedia_talk%3ABureaucrats%27_noticeboard
578#
579Disallow: /wiki/Wikipedia:Sockpuppet_investigations
580Disallow: /wiki/Wikipedia%3ASockpuppet_investigations
581Disallow: /wiki/Wikipedia_talk:Sockpuppet_investigations
582Disallow: /wiki/Wikipedia_talk%3ASockpuppet_investigations
583#
584Disallow: /wiki/Wikipedia:Neutral_point_of_view/Noticeboard
585Disallow: /wiki/Wikipedia%3ANeutral_point_of_view/Noticeboard
586Disallow: /wiki/Wikipedia_talk:Neutral_point_of_view/Noticeboard
587Disallow: /wiki/Wikipedia_talk%3ANeutral_point_of_view/Noticeboard
588#
589Disallow: /wiki/Wikipedia:No_original_research/noticeboard
590Disallow: /wiki/Wikipedia%3ANo_original_research/noticeboard
591Disallow: /wiki/Wikipedia_talk:No_original_research/noticeboard
592Disallow: /wiki/Wikipedia_talk%3ANo_original_research/noticeboard
593#
594Disallow: /wiki/Wikipedia:Fringe_theories/Noticeboard
595Disallow: /wiki/Wikipedia%3AFringe_theories/Noticeboard
596Disallow: /wiki/Wikipedia_talk:Fringe_theories/Noticeboard
597Disallow: /wiki/Wikipedia_talk%3AFringe_theories/Noticeboard
598#
599Disallow: /wiki/Wikipedia:Conflict_of_interest/Noticeboard
600Disallow: /wiki/Wikipedia%3AConflict_of_interest/Noticeboard
601Disallow: /wiki/Wikipedia_talk:Conflict_of_interest/Noticeboard
602Disallow: /wiki/Wikipedia_talk%3AConflict_of_interest/Noticeboard
603#
604Disallow: /wiki/Wikipedia:Long-term_abuse
605Disallow: /wiki/Wikipedia%3ALong-term_abuse
606Disallow: /wiki/Wikipedia_talk:Long-term_abuse
607Disallow: /wiki/Wikipedia_talk%3ALong-term_abuse
608Disallow: /wiki/Wikipedia:Long_term_abuse
609Disallow: /wiki/Wikipedia%3ALong_term_abuse
610Disallow: /wiki/Wikipedia_talk:Long_term_abuse
611Disallow: /wiki/Wikipedia_talk%3ALong_term_abuse
612#
613Disallow: /wiki/Wikipedia:Wikiquette_assistance
614Disallow: /wiki/Wikipedia%3AWikiquette_assistance
615#
616Disallow: /wiki/Wikipedia:Abuse_reports
617Disallow: /wiki/Wikipedia%3AAbuse_reports
618Disallow: /wiki/Wikipedia_talk:Abuse_reports
619Disallow: /wiki/Wikipedia_talk%3AAbuse_reports
620Disallow: /wiki/Wikipedia:Abuse_response
621Disallow: /wiki/Wikipedia%3AAbuse_response
622Disallow: /wiki/Wikipedia_talk:Abuse_response
623Disallow: /wiki/Wikipedia_talk%3AAbuse_response
624#
625Disallow: /wiki/Wikipedia:Reliable_sources/Noticeboard
626Disallow: /wiki/Wikipedia%3AReliable_sources/Noticeboard
627Disallow: /wiki/Wikipedia_talk:Reliable_sources/Noticeboard
628Disallow: /wiki/Wikipedia_talk%3AReliable_sources/Noticeboard
629#
630Disallow: /wiki/Wikipedia:Suspected_sock_puppets
631Disallow: /wiki/Wikipedia%3ASuspected_sock_puppets
632Disallow: /wiki/Wikipedia_talk:Suspected_sock_puppets
633Disallow: /wiki/Wikipedia_talk%3ASuspected_sock_puppets
634#
635Disallow: /wiki/Wikipedia:Biographies_of_living_persons/Noticeboard
636Disallow: /wiki/Wikipedia%3ABiographies_of_living_persons/Noticeboard
637Disallow: /wiki/Wikipedia_talk:Biographies_of_living_persons/Noticeboard
638Disallow: /wiki/Wikipedia_talk%3ABiographies_of_living_persons/Noticeboard
639Disallow: /wiki/Wikipedia:Biographies_of_living_persons%2FNoticeboard
640Disallow: /wiki/Wikipedia%3ABiographies_of_living_persons%2FNoticeboard
641Disallow: /wiki/Wikipedia_talk:Biographies_of_living_persons%2FNoticeboard
642Disallow: /wiki/Wikipedia_talk%3ABiographies_of_living_persons%2FNoticeboard
643#
644Disallow: /wiki/Wikipedia:Content_noticeboard
645Disallow: /wiki/Wikipedia%3AContent_noticeboard
646Disallow: /wiki/Wikipedia_talk:Content_noticeboard
647Disallow: /wiki/Wikipedia_talk%3AContent_noticeboard
648#
649Disallow: /wiki/Template:Editnotices
650Disallow: /wiki/Template%3AEditnotices
651#
652Disallow: /wiki/Wikipedia:Arbitration
653Disallow: /wiki/Wikipedia%3AArbitration
654Disallow: /wiki/Wikipedia_talk:Arbitration
655Disallow: /wiki/Wikipedia_talk%3AArbitration
656#
657Disallow: /wiki/Wikipedia:Arbitration_Committee
658Disallow: /wiki/Wikipedia%3AArbitration_Committee
659Disallow: /wiki/Wikipedia_talk:Arbitration_Committee
660Disallow: /wiki/Wikipedia_talk%3AArbitration_Committee
661#
662Disallow: /wiki/Wikipedia:Arbitration_Committee_Elections
663Disallow: /wiki/Wikipedia%3AArbitration_Committee_Elections
664Disallow: /wiki/Wikipedia_talk:Arbitration_Committee_Elections
665Disallow: /wiki/Wikipedia_talk%3AArbitration_Committee_Elections
666#
667Disallow: /wiki/Wikipedia:Mediation_Committee
668Disallow: /wiki/Wikipedia%3AMediation_Committee
669Disallow: /wiki/Wikipedia_talk:Mediation_Committee
670Disallow: /wiki/Wikipedia_talk%3AMediation_Committee
671#
672Disallow: /wiki/Wikipedia:Mediation_Cabal/Cases
673Disallow: /wiki/Wikipedia%3AMediation_Cabal/Cases
674#
675Disallow: /wiki/Wikipedia:Requests_for_bureaucratship
676Disallow: /wiki/Wikipedia%3ARequests_for_bureaucratship
677Disallow: /wiki/Wikipedia_talk:Requests_for_bureaucratship
678Disallow: /wiki/Wikipedia_talk%3ARequests_for_bureaucratship
679#
680Disallow: /wiki/Wikipedia:Administrator_review
681Disallow: /wiki/Wikipedia%3AAdministrator_review
682Disallow: /wiki/Wikipedia_talk:Administrator_review
683Disallow: /wiki/Wikipedia_talk%3AAdministrator_review
684#
685Disallow: /wiki/Wikipedia:Editor_review
686Disallow: /wiki/Wikipedia%3AEditor_review
687Disallow: /wiki/Wikipedia_talk:Editor_review
688Disallow: /wiki/Wikipedia_talk%3AEditor_review
689#
690Disallow: /wiki/Wikipedia:Article_Incubator
691Disallow: /wiki/Wikipedia%3AArticle_Incubator
692Disallow: /wiki/Wikipedia_talk:Article_Incubator
693Disallow: /wiki/Wikipedia_talk%3AArticle_Incubator
694#
695Disallow: /wiki/Category:Noindexed_pages
696Disallow: /wiki/Category%3ANoindexed_pages
697#
698# User sandboxes for modules and Template Styles are placed in these subpages for testing
699#
700Disallow: /wiki/Module:Sandbox
701Disallow: /wiki/Module%3ASandbox
702Disallow: /wiki/Template:TemplateStyles_sandbox
703Disallow: /wiki/Template%3ATemplateStyles_sandbox
704# Adminship and arbitration discussions are blocked from search engines, so blocking newer adminship processes (as of 2024) as well
705Disallow: /wiki/Wikipedia:Administrator_recall
706Disallow: /wiki/Wikipedia%3AAdministrator_recall
707Disallow: /wiki/Wikipedia_talk:Administrator_recall
708Disallow: /wiki/Wikipedia_talk%3AAdministrator_recall
709Disallow: /wiki/Wikipedia:Administrator_elections
710Disallow: /wiki/Wikipedia%3AAdministrator_elections
711Disallow: /wiki/Wikipedia_talk:Administrator_elections
712Disallow: /wiki/Wikipedia_talk%3AAdministrator_elections
What do the bots actually get?
This page reads robots.txt, which states what wikipedia.org asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on wikipedia.org Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does wikipedia.org block GPTBot?
No. wikipedia.org’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does wikipedia.org block ClaudeBot?
No. wikipedia.org’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does wikipedia.org block Google-Extended?
No. wikipedia.org’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does wikipedia.org block PerplexityBot?
No. wikipedia.org’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find wikipedia.org?
Yes. wikipedia.org’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.