Who blocks AI · #940 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does wikibooks.org block AI crawlers?
No. wikibooks.org’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- en.wikibooks.org/robots.txt · 503 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Partly blocked | line 153 · Disallow: /w/ line 154 · Disallow: /api/ and 254 more |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View en.wikibooks.org/robots.txt · 503 lines
1# robots.txt for http://www.wikipedia.org/ and friends
2#
3# Please note: There are a lot of pages on this site, and there are
4# some misbehaved spiders out there that go _way_ too fast. If you're
5# irresponsible, your access to the site may be blocked.
6#
7
8# Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN
9# and ignoring 429 ratelimit responses, claims to respect robots:
10# http://mj12bot.com/
11User-agent: MJ12bot
12Disallow: /
13
14# advertising-related bots:
15User-agent: Mediapartners-Google*
16Disallow: /
17
18# Wikipedia work bots:
19User-agent: IsraBot
20Disallow:
21
22User-agent: Orthogaffe
23Disallow:
24
25# Crawlers that are kind enough to obey, but which we'd rather not have
26# unless they're feeding search engines.
27User-agent: UbiCrawler
28Disallow: /
29
30User-agent: DOC
31Disallow: /
32
33User-agent: Zao
34Disallow: /
35
36# Some bots are known to be trouble, particularly those designed to copy
37# entire sites. Please obey robots.txt.
38User-agent: sitecheck.internetseer.com
39Disallow: /
40
41User-agent: Zealbot
42Disallow: /
43
44User-agent: MSIECrawler
45Disallow: /
46
47User-agent: SiteSnagger
48Disallow: /
49
50User-agent: WebStripper
51Disallow: /
52
53User-agent: WebCopier
54Disallow: /
55
56User-agent: Fetch
57Disallow: /
58
59User-agent: Offline Explorer
60Disallow: /
61
62User-agent: Teleport
63Disallow: /
64
65User-agent: TeleportPro
66Disallow: /
67
68User-agent: WebZIP
69Disallow: /
70
71User-agent: linko
72Disallow: /
73
74User-agent: HTTrack
75Disallow: /
76
77User-agent: Microsoft.URL.Control
78Disallow: /
79
80User-agent: Xenu
81Disallow: /
82
83User-agent: larbin
84Disallow: /
85
86User-agent: libwww
87Disallow: /
88
89User-agent: ZyBORG
90Disallow: /
91
92User-agent: Download Ninja
93Disallow: /
94
95# Misbehaving, requests much too fast
96User-agent: fast
97Disallow: /
98
99# Sorry, wget in its recursive mode is a frequent problem.
100# Please read the man page and use it properly; there is a
101# --wait option you can use to set the delay between hits,
102# for instance.
103User-agent: wget
104Disallow: /
105
106# The 'grub' distributed client has been *very* poorly behaved.
107User-agent: grub-client
108Disallow: /
109
110# Doesn't follow robots.txt anyway, but...
111User-agent: k2spider
112Disallow: /
113
114# Hits many times per second, not acceptable
115# http://www.nameprotect.com/botinfo.html
116User-agent: NPBot
117Disallow: /
118
119# A capture bot, downloads gazillions of pages with no public benefit
120# http://www.webreaper.net/
121User-agent: WebReaper
122Disallow: /
123
124# Per their statement, semrushbot respects crawl-delay directives
125# We want them to overall stay within reasonable request rates to
126# the backend (20 rps); keeping in mind that the crawl-delay will
127# be applied by site and not globally by the bot, 5 seconds seem
128# like a reasonable approximation
129User-agent: SemrushBot
130Crawl-delay: 5
131
132#
133# Friendly, low-speed bots are welcome viewing article pages, but not
134# dynamically-generated pages please.
135#
136# Inktomi's "Slurp" can read a minimum delay between hits; if your
137# bot supports such a thing using the 'Crawl-delay' or another
138# instruction, please let us know.
139#
140# There is a special exception for API mobileview to allow dynamic
141# mobile web & app views to load section content.
142# These views aren't HTTP-cached but use parser cache aggressively
143# and don't expose Special-pages and other uncached content.
144#
145# Another exception is for REST API documentation, located at
146# /api/rest_v1/?doc.
147#
148User-agent: *
149Allow: /w/api.php?action=mobileview&
150Allow: /w/load.php?
151Allow: /api/rest_v1/?doc
152Allow: /w/rest.php/site/v1/sitemap
153Disallow: /w/
154Disallow: /api/
155Disallow: /trap/
156Disallow: /wiki/Special:
157Disallow: /wiki/Spezial:
158Disallow: /wiki/Spesial:
159Disallow: /wiki/Special%3A
160Disallow: /wiki/Spezial%3A
161Disallow: /wiki/Spesial%3A
162#
163# ar:
164Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search
165Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch
166#
167# dewiki:
168# T6937
169# sensible deletion and meta user discussion pages:
170Disallow: /wiki/Wikipedia:L%C3%B6schkandidaten/
171Disallow: /wiki/Wikipedia:Löschkandidaten/
172Disallow: /wiki/Wikipedia:Vandalensperrung/
173Disallow: /wiki/Wikipedia:Benutzersperrung/
174Disallow: /wiki/Wikipedia:Vermittlungsausschuss/
175Disallow: /wiki/Wikipedia:Administratoren/Probleme/
176Disallow: /wiki/Wikipedia:Adminkandidaturen/
177Disallow: /wiki/Wikipedia:Qualitätssicherung/
178Disallow: /wiki/Wikipedia:Qualit%C3%A4tssicherung/
179# 4937#5
180Disallow: /wiki/Wikipedia:Vandalismusmeldung/
181Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/
182Disallow: /wiki/Wikipedia:Löschprüfung/
183Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/
184Disallow: /wiki/Wikipedia:Administratoren/Notizen/
185Disallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/
186Disallow: /wiki/Wikipedia:L%C3%B6schpr%C3%BCfung/
187# T14111
188Disallow: /wiki/Wikipedia:Checkuser/
189Disallow: /wiki/Wikipedia_Diskussion:Checkuser/
190Disallow: /wiki/Wikipedia_Diskussion:Adminkandidaturen/
191# T15961
192Disallow: /wiki/Wikipedia:Spam-Blacklist-Log
193Disallow: /wiki/Wikipedia%3ASpam-Blacklist-Log
194Disallow: /wiki/Wikipedia_Diskussion:Spam-Blacklist-Log
195Disallow: /wiki/Wikipedia_Diskussion%3ASpam-Blacklist-Log
196#
197# enwiki:
198# Folks get annoyed when VfD discussions end up the number 1 google hit for
199# their name. See T6776
200Disallow: /wiki/Wikipedia:Articles_for_deletion/
201Disallow: /wiki/Wikipedia%3AArticles_for_deletion/
202Disallow: /wiki/Wikipedia:Votes_for_deletion/
203Disallow: /wiki/Wikipedia%3AVotes_for_deletion/
204Disallow: /wiki/Wikipedia:Pages_for_deletion/
205Disallow: /wiki/Wikipedia%3APages_for_deletion/
206Disallow: /wiki/Wikipedia:Miscellany_for_deletion/
207Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/
208Disallow: /wiki/Wikipedia:Miscellaneous_deletion/
209Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/
210Disallow: /wiki/Wikipedia:Copyright_problems
211Disallow: /wiki/Wikipedia%3ACopyright_problems
212Disallow: /wiki/Wikipedia:Protected_titles/
213Disallow: /wiki/Wikipedia%3AProtected_titles/
214# T15398
215Disallow: /wiki/Wikipedia:WikiProject_Spam/
216Disallow: /wiki/Wikipedia%3AWikiProject_Spam/
217# T16075
218Disallow: /wiki/MediaWiki:Spam-blacklist
219Disallow: /wiki/MediaWiki%3ASpam-blacklist
220Disallow: /wiki/MediaWiki_talk:Spam-blacklist
221Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist
222# T13261
223Disallow: /wiki/Wikipedia:Requests_for_arbitration/
224Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/
225Disallow: /wiki/Wikipedia:Requests_for_comment/
226Disallow: /wiki/Wikipedia%3ARequests_for_comment/
227Disallow: /wiki/Wikipedia:Requests_for_adminship/
228Disallow: /wiki/Wikipedia%3ARequests_for_adminship/
229# T12288
230Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/
231Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/
232Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/
233Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/
234Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/
235Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/
236Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/
237Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/
238Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/
239Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/
240# T16793
241Disallow: /wiki/Wikipedia:Changing_username
242Disallow: /wiki/Wikipedia%3AChanging_username
243Disallow: /wiki/Wikipedia:Changing_username/
244Disallow: /wiki/Wikipedia%3AChanging_username/
245Disallow: /wiki/Wikipedia_talk:Changing_username
246Disallow: /wiki/Wikipedia_talk%3AChanging_username
247Disallow: /wiki/Wikipedia_talk:Changing_username/
248Disallow: /wiki/Wikipedia_talk%3AChanging_username/
249#
250# eswiki:
251# T8746
252Disallow: /wiki/Wikipedia:Consultas_de_borrado/
253Disallow: /wiki/Wikipedia%3AConsultas_de_borrado/
254#
255# fiwiki:
256# T10695
257Disallow: /wiki/Wikipedia:Poistettavat_sivut
258Disallow: /wiki/K%C3%A4ytt%C3%A4j%C3%A4:
259Disallow: /wiki/Käyttäjä:
260Disallow: /wiki/Keskustelu_k%C3%A4ytt%C3%A4j%C3%A4st%C3%A4:
261Disallow: /wiki/Keskustelu_käyttäjästä:
262Disallow: /wiki/Wikipedia:Yll%C3%A4pit%C3%A4j%C3%A4t/
263Disallow: /wiki/Wikipedia:Ylläpitäjät/
264#
265# hewiki:
266Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93:Search
267Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93%3ASearch
268#T11517
269Disallow: /wiki/ויקיפדיה:רשימת_מועמדים_למחיקה/
270Disallow: /wiki/ויקיפדיה%3Aרשימת_מועמדים_למחיקה/
271Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/
272Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/
273Disallow: /wiki/ויקיפדיה:ערכים_לא_קיימים_ומוגנים
274Disallow: /wiki/ויקיפדיה%3Aערכים_לא_קיימים_ומוגנים
275Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
276Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
277Disallow: /wiki/ויקיפדיה:דפים_לא_קיימים_ומוגנים
278Disallow: /wiki/ויקיפדיה%3Aדפים_לא_קיימים_ומוגנים
279Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
280Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D
281#
282# huwiki:
283Disallow: /wiki/Speci%C3%A1lis:Search
284Disallow: /wiki/Speci%C3%A1lis%3ASearch
285#
286# itwiki:
287# T7545
288Disallow: /wiki/Wikipedia:Pagine_da_cancellare
289Disallow: /wiki/Wikipedia%3APagine_da_cancellare
290Disallow: /wiki/Wikipedia:Utenti_problematici
291Disallow: /wiki/Wikipedia%3AUtenti_problematici
292Disallow: /wiki/Wikipedia:Vandalismi_in_corso
293Disallow: /wiki/Wikipedia%3AVandalismi_in_corso
294Disallow: /wiki/Wikipedia:Amministratori
295Disallow: /wiki/Wikipedia%3AAmministratori
296Disallow: /wiki/Wikipedia:Proposte_di_cancellazione_semplificata
297Disallow: /wiki/Wikipedia%3AProposte_di_cancellazione_semplificata
298Disallow: /wiki/Categoria:Da_cancellare_subito
299Disallow: /wiki/Categoria%3ADa_cancellare_subito
300Disallow: /wiki/Wikipedia:Sospette_violazioni_di_copyright
301Disallow: /wiki/Wikipedia%3ASospette_violazioni_di_copyright
302Disallow: /wiki/Categoria:Da_controllare_per_copyright
303Disallow: /wiki/Categoria%3ADa_controllare_per_copyright
304Disallow: /wiki/Progetto:Rimozione_contributi_sospetti
305Disallow: /wiki/Progetto%3ARimozione_contributi_sospetti
306Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright
307Disallow: /wiki/Categoria%3ADa_cancellare_subito_per_violazione_integrale_copyright
308Disallow: /wiki/Progetto:Cococo
309Disallow: /wiki/Progetto%3ACococo
310Disallow: /wiki/Discussioni_progetto:Cococo
311Disallow: /wiki/Discussioni_progetto%3ACococo
312#
313# jawiki
314Disallow: /wiki/%E7%89%B9%E5%88%A5:Search
315Disallow: /wiki/%E7%89%B9%E5%88%A5%3ASearch
316# T7239
317Disallow: /wiki/Wikipedia:%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/
318Disallow: /wiki/Wikipedia%3A%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/
319Disallow: /wiki/Wikipedia:%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC
320Disallow: /wiki/Wikipedia%3A%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC
321# nowiki
322# T13432
323Disallow: /wiki/Bruker:
324Disallow: /wiki/Bruker%3A
325Disallow: /wiki/Brukerdiskusjon
326Disallow: /wiki/Wikipedia:Administratorer
327Disallow: /wiki/Wikipedia%3AAdministratorer
328Disallow: /wiki/Wikipedia-diskusjon:Administratorer
329Disallow: /wiki/Wikipedia-diskusjon%3AAdministratorer
330Disallow: /wiki/Wikipedia:Sletting
331Disallow: /wiki/Wikipedia%3ASletting
332Disallow: /wiki/Wikipedia-diskusjon:Sletting
333Disallow: /wiki/Wikipedia-diskusjon%3ASletting
334#
335# plwiki
336# T10067
337Disallow: /wiki/Wikipedia:Strony_do_usuni%C4%99cia
338Disallow: /wiki/Wikipedia%3AStrony_do_usuni%C4%99cia
339Disallow: /wiki/Wikipedia:Do_usuni%C4%99cia
340Disallow: /wiki/Wikipedia%3ADo_usuni%C4%99cia
341Disallow: /wiki/Wikipedia:SDU/
342Disallow: /wiki/Wikipedia%3ASDU/
343Disallow: /wiki/Wikipedia:Strony_podejrzane_o_naruszenie_praw_autorskich
344Disallow: /wiki/Wikipedia%3AStrony_podejrzane_o_naruszenie_praw_autorskich
345#
346# ptwiki:
347# T7394
348Disallow: /wiki/Wikipedia:Páginas_para_eliminar/
349Disallow: /wiki/Wikipedia:P%C3%A1ginas_para_eliminar/
350Disallow: /wiki/Wikipedia%3AP%C3%A1ginas_para_eliminar/
351Disallow: /wiki/Wikipedia_Discussão:Páginas_para_eliminar/
352Disallow: /wiki/Wikipedia_Discuss%C3%A3o:P%C3%A1ginas_para_eliminar/
353Disallow: /wiki/Wikipedia_Discuss%C3%A3o%3AP%C3%A1ginas_para_eliminar/
354#
355# rowiki:
356# T14546
357Disallow: /wiki/Wikipedia:Pagini_de_%C5%9Fters
358Disallow: /wiki/Wikipedia%3APagini_de_%C5%9Fters
359Disallow: /wiki/Discu%C5%A3ie_Wikipedia:Pagini_de_%C5%9Fters
360Disallow: /wiki/Discu%C5%A3ie_Wikipedia%3APagini_de_%C5%9Fters
361#
362# ruwiki:
363Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5:Search
364Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5%3ASearch
365#
366# svwiki:
367# T12229
368Disallow: /wiki/Wikipedia%3ASidor_f%C3%B6reslagna_f%C3%B6r_radering
369Disallow: /wiki/Wikipedia:Sidor_f%C3%B6reslagna_f%C3%B6r_radering
370Disallow: /wiki/Wikipedia:Sidor_föreslagna_för_radering
371Disallow: /wiki/Användare
372Disallow: /wiki/Anv%C3%A4ndare
373Disallow: /wiki/Användardiskussion
374Disallow: /wiki/Anv%C3%A4ndardiskussion
375Disallow: /wiki/Wikipedia:Skyddade_sidnamn
376Disallow: /wiki/Wikipedia%3ASkyddade_sidnamn
377# T13291
378Disallow: /wiki/Wikipedia:Sidor_som_bör_raderas
379Disallow: /wiki/Wikipedia:Sidor_som_b%C3%B6r_raderas
380Disallow: /wiki/Wikipedia%3ASidor_som_b%C3%B6r_raderas
381#
382# zhwiki:
383# T7104
384Disallow: /wiki/Wikipedia:删除投票/侵权
385Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8/%E4%BE%B5%E6%9D%83
386Disallow: /wiki/Wikipedia:删除投票和请求
387Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8%E5%92%8C%E8%AF%B7%E6%B1%82
388Disallow: /wiki/Category:快速删除候选
389Disallow: /wiki/Category:%E5%BF%AB%E9%80%9F%E5%88%A0%E9%99%A4%E5%80%99%E9%80%89
390Disallow: /wiki/Category:维基百科需要翻译的文章
391Disallow: /wiki/Category:%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E9%9C%80%E8%A6%81%E7%BF%BB%E8%AF%91%E7%9A%84%E6%96%87%E7%AB%A0
392#
393# sister projects
394#
395# enwikinews:
396# T7340
397Disallow: /wiki/Portal:Prepared_stories/
398Disallow: /wiki/Portal%3APrepared_stories/
399#
400# itwikinews
401# T11138
402Disallow: /wiki/Wikinotizie:Richieste_di_cancellazione
403Disallow: /wiki/Wikinotizie:Sospette_violazioni_di_copyright
404Disallow: /wiki/Categoria:Da_cancellare_subito
405Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright
406Disallow: /wiki/Wikinotizie:Storie_in_preparazione
407#
408# enwikiquote:
409# T17095
410Disallow: /wiki/Wikiquote:Votes_for_deletion/
411Disallow: /wiki/Wikiquote%3AVotes_for_deletion/
412Disallow: /wiki/Wikiquote_talk:Votes_for_deletion/
413Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion/
414Disallow: /wiki/Wikiquote:Votes_for_deletion_archive/
415Disallow: /wiki/Wikiquote%3AVotes_for_deletion_archive/
416Disallow: /wiki/Wikiquote_talk:Votes_for_deletion_archive/
417Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion_archive/
418#
419# enwikibooks
420Disallow: /wiki/Wikibooks:Votes_for_deletion
421#
422
423
424Sitemap: https://en.wikibooks.org/w/rest.php/site/v1/sitemap/0
425
426#
427#
428#----------------------------------------------------------#
429#
430#
431#
432 # <!-- Please do not remove the space at the start of this line, it breaks the rendering. http://www.robotstxt.org/orig.html says spaces before comments are OK. --><source lang="robots">
433# robots.txt for http://en.wikibooks.org/
434# Edit at http://en.wikibooks.org/w/index.php?title=MediaWiki:Robots.txt&action=edit
435# Don't add newlines here. All rules set here are active for every user-agent.
436#
437# Please check any changes using a syntax validator such as http://tool.motoricerca.info/robots-checker.phtml
438# Enter http://en.wikibooks.org/robots.txt as the URL to check.
439#
440# Don't index anything in the MediaWiki namespace
441Disallow: /wiki/MediaWiki:
442Disallow: /wiki/MediaWiki%3A
443#
444# Don't index anything in the Transwiki namespace
445Disallow: /wiki/Transwiki:
446Disallow: /wiki/Transwiki%3A
447#
448# Don't index discussions
449Disallow: /wiki/Category_talk%3A
450Disallow: /wiki/Category_talk:
451Disallow: /wiki/Cookbook_talk%3A
452Disallow: /wiki/Cookbook_talk:
453Disallow: /wiki/File_talk%3A
454Disallow: /wiki/File_talk:
455Disallow: /wiki/Help_talk%3A
456Disallow: /wiki/Help_talk:
457Disallow: /wiki/MediaWiki_talk%3A
458Disallow: /wiki/MediaWiki_talk:
459Disallow: /wiki/Subject_talk%3A
460Disallow: /wiki/Subject_talk:
461Disallow: /wiki/Talk%3A
462Disallow: /wiki/Talk:
463Disallow: /wiki/Template_talk%3A
464Disallow: /wiki/Template_talk:
465Disallow: /wiki/Transwiki_talk%3A
466Disallow: /wiki/Transwiki_talk:
467Disallow: /wiki/User_talk%3A
468Disallow: /wiki/User_talk:
469Disallow: /wiki/Wikibooks%3AFeatured_books/Nominations
470Disallow: /wiki/Wikibooks%3AFeatured_books/Nominations/
471Disallow: /wiki/Wikibooks%3AReading_room
472Disallow: /wiki/Wikibooks%3AReading_room/
473Disallow: /wiki/Wikibooks%3ARequests_for_deletion
474Disallow: /wiki/Wikibooks%3ARequests_for_deletion/
475Disallow: /wiki/Wikibooks%3ARequests_for_import
476Disallow: /wiki/Wikibooks%3ARequests_for_import/
477Disallow: /wiki/Wikibooks%3ARequests_for_permissions
478Disallow: /wiki/Wikibooks%3ARequests_for_permissions/
479Disallow: /wiki/Wikibooks%3ARequests_for_renaming
480Disallow: /wiki/Wikibooks%3ARequests_for_renaming/
481Disallow: /wiki/Wikibooks:Featured_books/Nominations
482Disallow: /wiki/Wikibooks:Featured_books/Nominations/
483Disallow: /wiki/Wikibooks:Reading_room
484Disallow: /wiki/Wikibooks:Reading_room/
485Disallow: /wiki/Wikibooks:Requests_for_deletion
486Disallow: /wiki/Wikibooks:Requests_for_deletion/
487Disallow: /wiki/Wikibooks:Requests_for_import
488Disallow: /wiki/Wikibooks:Requests_for_import/
489Disallow: /wiki/Wikibooks:Requests_for_permissions
490Disallow: /wiki/Wikibooks:Requests_for_permissions/
491Disallow: /wiki/Wikibooks:Requests_for_renaming
492Disallow: /wiki/Wikibooks:Requests_for_renaming/
493Disallow: /wiki/Wikibooks_talk%3A
494Disallow: /wiki/Wikibooks_talk:
495Disallow: /wiki/Wikijunior_talk%3A
496Disallow: /wiki/Wikijunior_talk:
497#
498Disallow: /wiki/Category:Noindexed_pages
499Disallow: /wiki/Category%3ANoindexed_pages
500#
501Disallow: /wiki/Template:Editnotices/
502Disallow: /wiki/Template%3AEditnotices/
503# </source><!--leave this line alone-->
What do the bots actually get?
This page reads robots.txt, which states what wikibooks.org asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on wikibooks.org Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does wikibooks.org block GPTBot?
No. wikibooks.org’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does wikibooks.org block ClaudeBot?
No. wikibooks.org’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does wikibooks.org block Google-Extended?
No. wikibooks.org’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does wikibooks.org block PerplexityBot?
No. wikibooks.org’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find wikibooks.org?
Yes. wikibooks.org’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.