Who blocks AI · #89 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does who.int block AI crawlers?
Yes. who.int’s robots.txt blocks 1 of the 19 AI crawlers we track, including CCBot. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- www.who.int/robots.txt · 1,064 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no group applies | |
Bingbot | Microsoft | Allowed | no group applies |
DuckDuckBot | DuckDuckGo | Allowed | no group applies |
Applebot | Apple | Allowed | no group applies |
PetalBot | Huawei | Allowed | no group applies |
YandexBot | Yandex | Allowed | no group applies |
Baiduspider | Baidu | Allowed | no group applies |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no group applies |
PerplexityBot | Perplexity | Allowed | no group applies |
Claude-SearchBot | Anthropic | Allowed | no group applies |
DuckAssistBot | DuckDuckGo | Allowed | no group applies |
Amazonbot | Amazon | Allowed | no group applies |
YouBot | You.com | Allowed | no group applies |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no group applies |
ClaudeBot | Anthropic | Allowed | no group applies |
Google-Extended | Allowed | no group applies | |
CCBot | Common Crawl | Blocked | line 132 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | no group applies |
GoogleOther | Allowed | no group applies | |
Meta-ExternalAgent | Meta | Allowed | no group applies |
Bytespider | ByteDance | Allowed | no group applies |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no group applies |
Claude-User | Anthropic | Allowed | no group applies |
Perplexity-User | Perplexity | Allowed | no group applies |
MistralAI-User | Mistral | Allowed | no group applies |
Meta-ExternalFetcher | Meta | Allowed | no group applies |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 28 · Disallow: / under User-agent: AhrefsBot |
SemrushBot | Semrush | Blocked | line 734 · Disallow: / under User-agent: SemrushBot |
MJ12bot | Majestic | Allowed | no group applies |
DotBot | Moz | Blocked | line 224 · Disallow: / under User-agent: Dotbot |
View www.who.int/robots.txt · 1,064 lines
1### Version Information #
2###################################################
3### Version: V3.2018.05.828
4### Updated: Tue May 8 11:37:04 SAST 2018
5### Bad Bot Count: 527
6###################################################
7### Version Information ##
8
9User-agent: 360Spider
10Disallow:/
11User-agent: 404checker
12Disallow:/
13User-agent: 404enemy
14Disallow:/
15User-agent: 80legs
16Disallow:/
17User-agent: Abonti
18Disallow:/
19User-agent: Aboundex
20Disallow:/
21User-agent: Acunetix
22Disallow:/
23User-agent: ADmantX
24Disallow:/
25User-agent: AfD-Verbotsverfahren
26Disallow:/
27User-agent: AhrefsBot
28Disallow:/
29User-agent: AIBOT
30Disallow:/
31User-agent: AiHitBot
32Disallow:/
33User-agent: Aipbot
34Disallow:/
35User-agent: Alexibot
36Disallow:/
37User-agent: Alligator
38Disallow:/
39User-agent: AllSubmitter
40Disallow:/
41User-agent: AlphaBot
42Disallow:/
43User-agent: Anarchie
44Disallow:/
45User-agent: Apexoo
46Disallow:/
47User-agent: ASPSeek
48Disallow:/
49User-agent: Asterias
50Disallow:/
51User-agent: Attach
52Disallow:/
53User-agent: autoemailspider
54Disallow:/
55User-agent: BackDoorBot
56Disallow:/
57User-agent: Backlink-Ceck
58Disallow:/
59User-agent: backlink-check
60Disallow:/
61User-agent: BacklinkCrawler
62Disallow:/
63User-agent: BackStreet
64Disallow:/
65User-agent: BackWeb
66Disallow:/
67User-agent: Badass
68Disallow:/
69User-agent: Bandit
70Disallow:/
71User-agent: Barkrowler
72Disallow:/
73User-agent: BatchFTP
74Disallow:/
75User-agent: Battleztar Bazinga
76Disallow:/
77User-agent: BBBike
78Disallow:/
79User-agent: BDCbot
80Disallow:/
81User-agent: BDFetch
82Disallow:/
83User-agent: BetaBot
84Disallow:/
85User-agent: Bigfoot
86Disallow:/
87User-agent: Bitacle
88Disallow:/
89User-agent: Blackboard
90Disallow:/
91User-agent: Black Hole
92Disallow:/
93User-agent: BlackWidow
94Disallow:/
95User-agent: BLEXBot
96Disallow:/
97User-agent: Blow
98Disallow:/
99User-agent: BlowFish
100Disallow:/
101User-agent: Boardreader
102Disallow:/
103User-agent: Bolt
104Disallow:/
105User-agent: BotALot
106Disallow:/
107User-agent: Brandprotect
108Disallow:/
109User-agent: Brandwatch
110Disallow:/
111User-agent: Bubing
112Disallow:/
113User-agent: Buddy
114Disallow:/
115User-agent: BuiltBotTough
116Disallow:/
117User-agent: BuiltWith
118Disallow:/
119User-agent: Bullseye
120Disallow:/
121User-agent: BunnySlippers
122Disallow:/
123User-agent: BuzzSumo
124Disallow:/
125User-agent: Calculon
126Disallow:/
127User-agent: CATExplorador
128Disallow:/
129User-agent: CazoodleBot
130Disallow:/
131User-agent: CCBot
132Disallow:/
133User-agent: Cegbfeieh
134Disallow:/
135User-agent: CheeseBot
136Disallow:/
137User-agent: CherryPicker
138Disallow:/
139User-agent: ChinaClaw
140Disallow:/
141User-agent: Chlooe
142Disallow:/
143User-agent: Claritybot
144Disallow:/
145User-agent: Cliqzbot
146Disallow:/
147User-agent: Cloud mapping
148Disallow:/
149User-agent: coccocbot-web
150Disallow:/
151User-agent: Cogentbot
152Disallow:/
153User-agent: cognitiveseo
154Disallow:/
155User-agent: Collector
156Disallow:/
157User-agent: com.plumanalytics
158Disallow:/
159User-agent: Copier
160Disallow:/
161User-agent: CopyRightCheck
162Disallow:/
163User-agent: Copyscape
164Disallow:/
165User-agent: Cosmos
166Disallow:/
167User-agent: Craftbot
168Disallow:/
169User-agent: crawler4j
170Disallow:/
171User-agent: crawler.feedback
172Disallow:/
173User-agent: CrazyWebCrawler
174Disallow:/
175User-agent: Crescent
176Disallow:/
177User-agent: CSHttp
178Disallow:/
179User-agent: Curious
180Disallow:/
181User-agent: Custo
182Disallow:/
183User-agent: DatabaseDriverMysqli
184Disallow:/
185User-agent: DataCha0s
186Disallow:/
187User-agent: DBLBot
188Disallow:/
189User-agent: demandbase-bot
190Disallow:/
191User-agent: Demon
192Disallow:/
193User-agent: Deusu
194Disallow:/
195User-agent: Devil
196Disallow:/
197User-agent: Digincore
198Disallow:/
199User-agent: DigitalPebble
200Disallow:/
201User-agent: DIIbot
202Disallow:/
203User-agent: Dirbuster
204Disallow:/
205User-agent: Disco
206Disallow:/
207User-agent: Discobot
208Disallow:/
209User-agent: Discoverybot
210Disallow:/
211User-agent: DittoSpyder
212Disallow:/
213User-agent: DnyzBot
214Disallow:/
215User-agent: DomainAppender
216Disallow:/
217User-agent: DomainCrawler
218Disallow:/
219User-agent: DomainSigmaCrawler
220Disallow:/
221User-agent: DomainStatsBot
222Disallow:/
223User-agent: Dotbot
224Disallow:/
225User-agent: Download Wonder
226Disallow:/
227User-agent: Dragonfly
228Disallow:/
229User-agent: Drip
230Disallow:/
231User-agent: DTS Agent
232Disallow:/
233User-agent: EasyDL
234Disallow:/
235User-agent: Ebingbong
236Disallow:/
237User-agent: eCatch
238Disallow:/
239User-agent: ECCP/1.0
240Disallow:/
241User-agent: Ecxi
242Disallow:/
243User-agent: EirGrabber
244Disallow:/
245User-agent: EMail Siphon
246Disallow:/
247User-agent: EMail Wolf
248Disallow:/
249User-agent: EroCrawler
250Disallow:/
251User-agent: evc-batch
252Disallow:/
253User-agent: Evil
254Disallow:/
255User-agent: Exabot
256Disallow:/
257User-agent: Express WebPictures
258Disallow:/
259User-agent: ExtLinksBot
260Disallow:/
261User-agent: Extractor
262Disallow:/
263User-agent: ExtractorPro
264Disallow:/
265User-agent: Extreme Picture Finder
266Disallow:/
267User-agent: EyeNetIE
268Disallow:/
269User-agent: Ezooms
270Disallow:/
271User-agent: FDM
272Disallow:/
273User-agent: FemtosearchBot
274Disallow:/
275User-agent: FHscan
276Disallow:/
277User-agent: Fimap
278Disallow:/
279User-agent: Firefox/7.0
280Disallow:/
281User-agent: FlashGet
282Disallow:/
283User-agent: Flunky
284Disallow:/
285User-agent: Foobot
286Disallow:/
287User-agent: fq
288Disallow:/
289User-agent: Freeuploader
290Disallow:/
291User-agent: FrontPage
292Disallow:/
293User-agent: Fyrebot
294Disallow:/
295User-agent: GalaxyBot
296Disallow:/
297User-agent: Genieo
298Disallow:/
299User-agent: GermCrawler
300Disallow:/
301User-agent: Getintent
302Disallow:/
303User-agent: GetRight
304Disallow:/
305User-agent: GetWeb
306Disallow:/
307User-agent: Gigablast
308Disallow:/
309User-agent: Gigabot
310Disallow:/
311User-agent: G-i-g-a-b-o-t
312Disallow:/
313User-agent: Go-Ahead-Got-It
314Disallow:/
315User-agent: Gotit
316Disallow:/
317User-agent: GoZilla
318Disallow:/
319User-agent: Go!Zilla
320Disallow:/
321User-agent: Grabber
322Disallow:/
323User-agent: GrabNet
324Disallow:/
325User-agent: Grafula
326Disallow:/
327User-agent: GrapeFX
328Disallow:/
329User-agent: GrapeshotCrawler
330Disallow:/
331User-agent: GridBot
332Disallow:/
333User-agent: GT::WWW
334Disallow:/
335User-agent: Haansoft
336Disallow:/
337User-agent: HaosouSpider
338Disallow:/
339User-agent: Harvest
340Disallow:/
341User-agent: Havij
342Disallow:/
343User-agent: HEADMasterSEO
344Disallow:/
345User-agent: Heritrix
346Disallow:/
347User-agent: Hloader
348Disallow:/
349User-agent: HMView
350Disallow:/
351User-agent: HTMLparser
352Disallow:/
353User-agent: HTTP::Lite
354Disallow:/
355User-agent: HTTrack
356Disallow:/
357User-agent: Humanlinks
358Disallow:/
359User-agent: HybridBot
360Disallow:/
361User-agent: Iblog
362Disallow:/
363User-agent: IDBot
364Disallow:/
365User-agent: Id-search
366Disallow:/
367User-agent: IlseBot
368Disallow:/
369User-agent: Image Fetch
370Disallow:/
371User-agent: Image Sucker
372Disallow:/
373User-agent: IndeedBot
374Disallow:/
375User-agent: Indy Library
376Disallow:/
377User-agent: InfoNaviRobot
378Disallow:/
379User-agent: InfoTekies
380Disallow:/
381User-agent: instabid
382Disallow:/
383User-agent: Intelliseek
384Disallow:/
385User-agent: InterGET
386Disallow:/
387User-agent: Internet Ninja
388Disallow:/
389User-agent: InternetSeer
390Disallow:/
391User-agent: internetVista monitor
392Disallow:/
393User-agent: ips-agent
394Disallow:/
395User-agent: Iria
396Disallow:/
397User-agent: IRLbot
398Disallow:/
399User-agent: Iskanie
400Disallow:/
401User-agent: IstellaBot
402Disallow:/
403User-agent: JamesBOT
404Disallow:/
405User-agent: Jbrofuzz
406Disallow:/
407User-agent: JennyBot
408Disallow:/
409User-agent: JetCar
410Disallow:/
411User-agent: JikeSpider
412Disallow:/
413User-agent: JOC Web Spider
414Disallow:/
415User-agent: Joomla
416Disallow:/
417User-agent: Jorgee
418Disallow:/
419User-agent: JustView
420Disallow:/
421User-agent: Jyxobot
422Disallow:/
423User-agent: Kenjin Spider
424Disallow:/
425User-agent: Keyword Density
426Disallow:/
427User-agent: Kozmosbot
428Disallow:/
429User-agent: Lanshanbot
430Disallow:/
431User-agent: Larbin
432Disallow:/
433User-agent: LeechFTP
434Disallow:/
435User-agent: LeechGet
436Disallow:/
437User-agent: LexiBot
438Disallow:/
439User-agent: Lftp
440Disallow:/
441User-agent: LibWeb
442Disallow:/
443User-agent: Libwhisker
444Disallow:/
445User-agent: Lightspeedsystems
446Disallow:/
447User-agent: Likse
448Disallow:/
449User-agent: Linkdexbot
450Disallow:/
451User-agent: LinkextractorPro
452Disallow:/
453User-agent: LinkpadBot
454Disallow:/
455User-agent: LinkScan
456Disallow:/
457User-agent: LinksManager
458Disallow:/
459User-agent: LinkWalker
460Disallow:/
461User-agent: LinqiaMetadataDownloaderBot
462Disallow:/
463User-agent: LinqiaRSSBot
464Disallow:/
465User-agent: LinqiaScrapeBot
466Disallow:/
467User-agent: Lipperhey
468Disallow:/
469User-agent: Litemage_walker
470Disallow:/
471User-agent: Lmspider
472Disallow:/
473User-agent: LNSpiderguy
474Disallow:/
475User-agent: Ltx71
476Disallow:/
477User-agent: lwp-request
478Disallow:/
479User-agent: LWP::Simple
480Disallow:/
481User-agent: lwp-trivial
482Disallow:/
483User-agent: Magnet
484Disallow:/
485User-agent: Mag-Net
486Disallow:/
487User-agent: magpie-crawler
488Disallow:/
489User-agent: Mail.RU_Bot
490Disallow:/
491User-agent: Majestic12
492Disallow:/
493User-agent: MarkMonitor
494Disallow:/
495User-agent: MarkWatch
496Disallow:/
497User-agent: Masscan
498Disallow:/
499User-agent: Mass Downloader
500Disallow:/
501User-agent: Mata Hari
502Disallow:/
503User-agent: MauiBot
504Disallow:/
505User-agent: Meanpathbot
506Disallow:/
507User-agent: mediawords
508Disallow:/
509User-agent: MegaIndex.ru
510Disallow:/
511User-agent: Metauri
512Disallow:/
513User-agent: MFC_Tear_Sample
514Disallow:/
515User-agent: Microsoft Data Access
516Disallow:/
517User-agent: Microsoft URL Control
518Disallow:/
519User-agent: MIDown tool
520Disallow:/
521User-agent: MIIxpc
522Disallow:/
523User-agent: Mister PiX
524Disallow:/
525User-agent: MJ12bot
526Disallow:/
527User-agent: Mojeek
528Disallow:/
529User-agent: Morfeus Fucking Scanner
530Disallow:/
531User-agent: MSFrontPage
532Disallow:/
533User-agent: MSIECrawler
534Disallow:/
535User-agent: Msrabot
536Disallow:/
537User-agent: MS Web Services Client Protocol
538Disallow:/
539User-agent: muhstik-scan
540Disallow:/
541User-agent: Musobot
542Disallow:/
543User-agent: Name Intelligence
544Disallow:/
545User-agent: Nameprotect
546Disallow:/
547User-agent: Navroad
548Disallow:/
549User-agent: NearSite
550Disallow:/
551User-agent: Needle
552Disallow:/
553User-agent: Nessus
554Disallow:/
555User-agent: NetAnts
556Disallow:/
557User-agent: Netcraft
558Disallow:/
559User-agent: netEstate NE Crawler
560Disallow:/
561User-agent: NetLyzer
562Disallow:/
563User-agent: NetMechanic
564Disallow:/
565User-agent: NetSpider
566Disallow:/
567User-agent: Nettrack
568Disallow:/
569User-agent: Net Vampire
570Disallow:/
571User-agent: Netvibes
572Disallow:/
573User-agent: NetZIP
574Disallow:/
575User-agent: NextGenSearchBot
576Disallow:/
577User-agent: Nibbler
578Disallow:/
579User-agent: NICErsPRO
580Disallow:/
581User-agent: Niki-bot
582Disallow:/
583User-agent: Nikto
584Disallow:/
585User-agent: NimbleCrawler
586Disallow:/
587User-agent: Ninja
588Disallow:/
589User-agent: Nmap
590Disallow:/
591User-agent: NPbot
592Disallow:/
593User-agent: Nutch
594Disallow:/
595User-agent: NYU
596Disallow:/
597User-agent: oBot
598Disallow:/
599User-agent: Octopus
600Disallow:/
601User-agent: Offline Explorer
602Disallow:/
603User-agent: Offline Navigator
604Disallow:/
605User-agent: Openfind
606Disallow:/
607User-agent: OpenLinkProfiler
608Disallow:/
609User-agent: Openvas
610Disallow:/
611User-agent: OrangeBot
612Disallow:/
613User-agent: OrangeSpider
614Disallow:/
615User-agent: OutclicksBot
616Disallow:/
617User-agent: OutfoxBot
618Disallow:/
619User-agent: PageAnalyzer
620Disallow:/
621User-agent: Page Analyzer
622Disallow:/
623User-agent: PageGrabber
624Disallow:/
625User-agent: page scorer
626Disallow:/
627User-agent: PageScorer
628Disallow:/
629User-agent: Panscient
630Disallow:/
631User-agent: Papa Foto
632Disallow:/
633User-agent: Pavuk
634Disallow:/
635User-agent: pcBrowser
636Disallow:/
637User-agent: PECL::HTTP
638Disallow:/
639User-agent: PeoplePal
640Disallow:/
641User-agent: PHPCrawl
642Disallow:/
643User-agent: Picscout
644Disallow:/
645User-agent: Picsearch
646Disallow:/
647User-agent: PictureFinder
648Disallow:/
649User-agent: Pimonster
650Disallow:/
651User-agent: Pi-Monster
652Disallow:/
653User-agent: Pixray
654Disallow:/
655User-agent: PleaseCrawl
656Disallow:/
657User-agent: plumanalytics
658Disallow:/
659User-agent: Pockey
660Disallow:/
661User-agent: POE-Component-Client-HTTP
662Disallow:/
663User-agent: Probethenet
664Disallow:/
665User-agent: ProPowerBot
666Disallow:/
667User-agent: ProWebWalker
668Disallow:/
669User-agent: Psbot
670Disallow:/
671User-agent: Pump
672Disallow:/
673User-agent: PyCurl
674Disallow:/
675User-agent: QueryN Metasearch
676Disallow:/
677User-agent: RankActive
678Disallow:/
679User-agent: RankActiveLinkBot
680Disallow:/
681User-agent: RankFlex
682Disallow:/
683User-agent: RankingBot
684Disallow:/
685User-agent: RankingBot2
686Disallow:/
687User-agent: Rankivabot
688Disallow:/
689User-agent: RankurBot
690Disallow:/
691User-agent: RealDownload
692Disallow:/
693User-agent: Reaper
694Disallow:/
695User-agent: RebelMouse
696Disallow:/
697User-agent: Recorder
698Disallow:/
699User-agent: RedesScrapy
700Disallow:/
701User-agent: ReGet
702Disallow:/
703User-agent: RepoMonkey
704Disallow:/
705User-agent: Ripper
706Disallow:/
707User-agent: RocketCrawler
708Disallow:/
709User-agent: Rogerbot
710Disallow:/
711User-agent: SalesIntelligent
712Disallow:/
713User-agent: SBIder
714Disallow:/
715User-agent: ScanAlert
716Disallow:/
717User-agent: Scanbot
718Disallow:/
719User-agent: scan.lol
720Disallow:/
721User-agent: Scrapy
722Disallow:/
723User-agent: Screaming
724Disallow:/
725User-agent: ScreenerBot
726Disallow:/
727User-agent: Searchestate
728Disallow:/
729User-agent: SearchmetricsBot
730Disallow:/
731User-agent: Semrush
732Disallow:/
733User-agent: SemrushBot
734Disallow:/
735User-agent: SEOkicks
736Disallow:/
737User-agent: SEOlyticsCrawler
738Disallow:/
739User-agent: Seomoz
740Disallow:/
741User-agent: SEOprofiler
742Disallow:/
743User-agent: seoscanners
744Disallow:/
745User-agent: SEOstats
746Disallow:/
747User-agent: sexsearcher
748Disallow:/
749User-agent: Seznam
750Disallow:/
751User-agent: SeznamBot
752Disallow:/
753User-agent: Shodan
754Disallow:/
755User-agent: Siphon
756Disallow:/
757User-agent: SISTRIX
758Disallow:/
759User-agent: Sitebeam
760Disallow:/
761User-agent: SiteExplorer
762Disallow:/
763User-agent: Siteimprove
764Disallow:/
765User-agent: SiteLockSpider
766Disallow:/
767User-agent: SiteSnagger
768Disallow:/
769User-agent: SiteSucker
770Disallow:/
771User-agent: Site Sucker
772Disallow:/
773User-agent: Sitevigil
774Disallow:/
775User-agent: Slackbot-LinkExpanding
776Disallow:/
777User-agent: SlySearch
778Disallow:/
779User-agent: SmartDownload
780Disallow:/
781User-agent: SMTBot
782Disallow:/
783User-agent: Snake
784Disallow:/
785User-agent: Snapbot
786Disallow:/
787User-agent: Snoopy
788Disallow:/
789User-agent: SocialRankIOBot
790Disallow:/
791User-agent: Sogou web spider
792Disallow:/
793User-agent: Sosospider
794Disallow:/
795User-agent: Sottopop
796Disallow:/
797User-agent: SpaceBison
798Disallow:/
799User-agent: Spammen
800Disallow:/
801User-agent: SpankBot
802Disallow:/
803User-agent: Spanner
804Disallow:/
805User-agent: Spbot
806Disallow:/
807User-agent: Spinn3r
808Disallow:/
809User-agent: SputnikBot
810Disallow:/
811User-agent: Sqlmap
812Disallow:/
813User-agent: Sqlworm
814Disallow:/
815User-agent: Sqworm
816Disallow:/
817User-agent: Steeler
818Disallow:/
819User-agent: Stripper
820Disallow:/
821User-agent: Sucker
822Disallow:/
823User-agent: Sucuri
824Disallow:/
825User-agent: SuperBot
826Disallow:/
827User-agent: SuperHTTP
828Disallow:/
829User-agent: Surfbot
830Disallow:/
831User-agent: SurveyBot
832Disallow:/
833User-agent: Suzuran
834Disallow:/
835User-agent: Swiftbot
836Disallow:/
837User-agent: sysscan
838Disallow:/
839User-agent: Szukacz
840Disallow:/
841User-agent: T0PHackTeam
842Disallow:/
843User-agent: T8Abot
844Disallow:/
845User-agent: tAkeOut
846Disallow:/
847User-agent: Teleport
848Disallow:/
849User-agent: TeleportPro
850Disallow:/
851User-agent: Telesoft
852Disallow:/
853User-agent: Telesphoreo
854Disallow:/
855User-agent: Telesphorep
856Disallow:/
857User-agent: The Intraformant
858Disallow:/
859User-agent: TheNomad
860Disallow:/
861User-agent: TightTwatBot
862Disallow:/
863User-agent: Titan
864Disallow:/
865User-agent: Toata
866Disallow:/
867User-agent: Toweyabot
868Disallow:/
869User-agent: Trendiction
870Disallow:/
871User-agent: Trendictionbot
872Disallow:/
873User-agent: trendiction.com
874Disallow:/
875User-agent: trendiction.de
876Disallow:/
877User-agent: True_Robot
878Disallow:/
879User-agent: Turingos
880Disallow:/
881User-agent: Turnitin
882Disallow:/
883User-agent: TurnitinBot
884Disallow:/
885User-agent: TwengaBot
886Disallow:/
887User-agent: Twice
888Disallow:/
889User-agent: Typhoeus
890Disallow:/
891User-agent: UnisterBot
892Disallow:/
893User-agent: URLy.Warning
894Disallow:/
895User-agent: URLy Warning
896Disallow:/
897User-agent: Vacuum
898Disallow:/
899User-agent: Vagabondo
900Disallow:/
901User-agent: VB Project
902Disallow:/
903User-agent: VCI
904Disallow:/
905User-agent: VeriCiteCrawler
906Disallow:/
907User-agent: VidibleScraper
908Disallow:/
909User-agent: Virusdie
910Disallow:/
911User-agent: VoidEYE
912Disallow:/
913User-agent: Voil
914Disallow:/
915User-agent: Voltron
916Disallow:/
917User-agent: Wallpapers/3.0
918Disallow:/
919User-agent: WallpapersHD
920Disallow:/
921User-agent: WASALive-Bot
922Disallow:/
923User-agent: WBSearchBot
924Disallow:/
925User-agent: Webalta
926Disallow:/
927User-agent: WebAuto
928Disallow:/
929User-agent: Web Auto
930Disallow:/
931User-agent: WebBandit
932Disallow:/
933User-agent: WebCollage
934Disallow:/
935User-agent: Web Collage
936Disallow:/
937User-agent: WebCopier
938Disallow:/
939User-agent: WEBDAV
940Disallow:/
941User-agent: WebEnhancer
942Disallow:/
943User-agent: Web Enhancer
944Disallow:/
945User-agent: WebFetch
946Disallow:/
947User-agent: Web Fetch
948Disallow:/
949User-agent: WebFuck
950Disallow:/
951User-agent: Web Fuck
952Disallow:/
953User-agent: WebGo IS
954Disallow:/
955User-agent: WebImageCollector
956Disallow:/
957User-agent: WebLeacher
958Disallow:/
959User-agent: WebmasterWorldForumBot
960Disallow:/
961User-agent: webmeup-crawler
962Disallow:/
963User-agent: WebPix
964Disallow:/
965User-agent: Web Pix
966Disallow:/
967User-agent: WebReaper
968Disallow:/
969User-agent: WebSauger
970Disallow:/
971User-agent: Web Sauger
972Disallow:/
973User-agent: Webshag
974Disallow:/
975User-agent: WebsiteExtractor
976Disallow:/
977User-agent: WebsiteQuester
978Disallow:/
979User-agent: Website Quester
980Disallow:/
981User-agent: Webster
982Disallow:/
983User-agent: WebStripper
984Disallow:/
985User-agent: WebSucker
986Disallow:/
987User-agent: Web Sucker
988Disallow:/
989User-agent: WebWhacker
990Disallow:/
991User-agent: WebZIP
992Disallow:/
993User-agent: WeSEE
994Disallow:/
995User-agent: Whack
996Disallow:/
997User-agent: Whacker
998Disallow:/
999User-agent: Whatweb
1000Disallow:/
1001User-agent: Who.is Bot
1002Disallow:/
1003User-agent: Widow
1004Disallow:/
1005User-agent: WinHTTrack
1006Disallow:/
1007User-agent: WiseGuys Robot
1008Disallow:/
1009User-agent: WISENutbot
1010Disallow:/
1011User-agent: Wonderbot
1012Disallow:/
1013User-agent: Woobot
1014Disallow:/
1015User-agent: Wotbox
1016Disallow:/
1017User-agent: Wprecon
1018Disallow:/
1019User-agent: WPScan
1020Disallow:/
1021User-agent: WWW-Collector-E
1022Disallow:/
1023User-agent: WWW-Mechanize
1024Disallow:/
1025User-agent: WWW::Mechanize
1026Disallow:/
1027User-agent: WWWOFFLE
1028Disallow:/
1029User-agent: x09Mozilla
1030Disallow:/
1031User-agent: x22Mozilla
1032Disallow:/
1033User-agent: Xaldon_WebSpider
1034Disallow:/
1035User-agent: Xaldon WebSpider
1036Disallow:/
1037User-agent: Xenu
1038Disallow:/
1039User-agent: xpymep1.exe
1040Disallow:/
1041User-agent: YoudaoBot
1042Disallow:/
1043User-agent: Zade
1044Disallow:/
1045User-agent: Zauba
1046Disallow:/
1047User-agent: zauba.io
1048Disallow:/
1049User-agent: Zermelo
1050Disallow:/
1051User-agent: Zeus
1052Disallow:/
1053User-agent: zgrab
1054Disallow:/
1055User-agent: Zitebot
1056Disallow:/
1057User-agent: ZmEu
1058Disallow:/
1059User-agent: ZumBot
1060Disallow:/
1061User-agent: ZyBorg
1062Disallow:/
1063
1064Sitemap: http://www.who.int/sitemaps/sitemapindex.xml
What do the bots actually get?
This page reads robots.txt, which states what who.int asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on who.int Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does who.int block GPTBot?
No. who.int’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does who.int block ClaudeBot?
No. who.int’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does who.int block Google-Extended?
No. who.int’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does who.int block PerplexityBot?
No. who.int’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find who.int?
Yes. who.int’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.