Who blocks AI · #1141 in the Majestic Million

robots.txt · checked 16 Nov 2026

Does unhcr.org block AI crawlers?

No. unhcr.org’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed
robots.txt
www.unhcr.org/robots.txt · 164 lines
llms.txt
none
Checked
16 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedno rule matches the home page
BingbotMicrosoftAllowedno rule matches the home page
DuckDuckBotDuckDuckGoAllowedno rule matches the home page
ApplebotAppleAllowedno rule matches the home page
PetalBotHuaweiAllowedno rule matches the home page
YandexBotYandexAllowedno rule matches the home page
BaiduspiderBaiduAllowedno rule matches the home page

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIAllowedno rule matches the home page
PerplexityBotPerplexityAllowedno rule matches the home page
Claude-SearchBotAnthropicAllowedno rule matches the home page
DuckAssistBotDuckDuckGoAllowedno rule matches the home page
AmazonbotAmazonAllowedno rule matches the home page
YouBotYou.comAllowedno rule matches the home page

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIAllowedno rule matches the home page
ClaudeBotAnthropicAllowedno rule matches the home page
Google-ExtendedGoogleAllowedno rule matches the home page
CCBotCommon CrawlAllowedno rule matches the home page
Applebot-ExtendedAppleAllowedno rule matches the home page
GoogleOtherGoogleAllowedno rule matches the home page
Meta-ExternalAgentMetaAllowedno rule matches the home page
BytespiderByteDanceAllowedno rule matches the home page

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIAllowedno rule matches the home page
Claude-UserAnthropicAllowedno rule matches the home page
Perplexity-UserPerplexityAllowedno rule matches the home page
MistralAI-UserMistralAllowedno rule matches the home page
Meta-ExternalFetcherMetaAllowedno rule matches the home page

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedno rule matches the home page
SemrushBotSemrushAllowedno rule matches the home page
MJ12botMajesticAllowedno rule matches the home page
DotBotMozAllowedno rule matches the home page
View www.unhcr.org/robots.txt · 164 lines
1#
2# robots.txt
3#
4# This file is to prevent the crawling and indexing of certain parts
5# of your site by web crawlers and spiders run by sites like Yahoo!
6# and Google. By telling these "robots" where not to go on your site,
7# you save bandwidth and server resources.
8#
9# This file will be ignored unless it is at the root of your host:
10# Used: http://example.com/robots.txt
11# Ignored: http://example.com/site/robots.txt
12#
13# For more information about the robots.txt standard, see:
14# http://www.robotstxt.org/robotstxt.html
15
16User-agent: *
17# CSS, JS, Images
18Allow: /core/*.css$
19Allow: /core/*.css?
20Allow: /core/*.js$
21Allow: /core/*.js?
22Allow: /core/*.avif
23Allow: /core/*.gif
24Allow: /core/*.jpg
25Allow: /core/*.jpeg
26Allow: /core/*.png
27Allow: /core/*.svg
28Allow: /core/*.webp
29Allow: /profiles/*.css$
30Allow: /profiles/*.css?
31Allow: /profiles/*.js$
32Allow: /profiles/*.js?
33Allow: /profiles/*.avif
34Allow: /profiles/*.gif
35Allow: /profiles/*.jpg
36Allow: /profiles/*.jpeg
37Allow: /profiles/*.png
38Allow: /profiles/*.svg
39Allow: /profiles/*.webp
40# Directories
41Disallow: /core/
42Disallow: /profiles/
43# Files
44Disallow: /README.md
45Disallow: /composer/Metapackage/README.txt
46Disallow: /composer/Plugin/ProjectMessage/README.md
47Disallow: /composer/Plugin/Scaffold/README.md
48Disallow: /composer/Plugin/VendorHardening/README.txt
49Disallow: /composer/Template/README.txt
50Disallow: /modules/README.txt
51Disallow: /sites/README.txt
52Disallow: /themes/README.txt
53Disallow: /web.config
54# Paths (clean URLs)
55Disallow: /admin/
56Disallow: /comment/reply/
57Disallow: /filter/tips
58Disallow: /node/add/
59Disallow: /search
60Disallow: /search/
61Disallow: /buscar
62Disallow: /buscar/
63Disallow: /user/register
64Disallow: /user/password
65Disallow: /user/login
66Disallow: /user/logout
67Disallow: /translation-not-found
68# Paths (no clean URLs)
69Disallow: /index.php/admin/
70Disallow: /index.php/comment/reply/
71Disallow: /index.php/filter/tips
72Disallow: /index.php/node/add/
73Disallow: /index.php/search
74Disallow: /index.php/search/
75Disallow: /index.php/user/password
76Disallow: /index.php/user/register
77Disallow: /index.php/user/login
78Disallow: /index.php/user/logout
79# Oembed media
80Disallow: /media/oembed
81Disallow: /index.php/media/oembed
82Disallow: /index.php/*/media/oembed
83Disallow: /*/media/oembed
84# sites
85Disallow: /fr-fr/en/
86Disallow: /es-es/en/
87Disallow: /mx/en/
88Disallow: /de/en/
89Disallow: /at/en/
90Disallow: /ch/en/
91Disallow: /br/en/
92
93# search page
94Disallow: /search
95Disallow: /*/search
96# pdf viewer
97Disallow: /libraries/pdf.js/web/viewer.html
98Disallow: /*/libraries/pdf.js/web/viewer.html
99# site health
100Disallow: /system-status
101Disallow: /*/system-status
102
103# sitemap
104Sitemap: https://www.unhcr.org/sitemap.xml
105Sitemap: https://www.acnur.org/sitemap.xml
106Sitemap: https://www.unhcr.org/africa/sitemap.xml
107Sitemap: https://www.unhcr.org/asia/sitemap.xml
108Sitemap: https://www.unhcr.org/au/sitemap.xml
109Sitemap: https://www.acnur.org/es-es/sitemap.xml
110Sitemap: https://www.unhcr.org/fr-fr/sitemap.xml
111Sitemap: https://www.unhcr.org/ie/sitemap.xml
112Sitemap: https://www.unhcr.org/in/sitemap.xml
113Sitemap: https://www.acnur.org/mx/sitemap.xml
114Sitemap: https://www.unhcr.org/my/sitemap.xml
115Sitemap: https://www.unhcr.org/uk/sitemap.xml
116Sitemap: https://www.unhcr.org/us/sitemap.xml
117Sitemap: https://www.unhcr.org/europe/sitemap.xml
118Sitemap: https://www.unhcr.org/kr/sitemap.xml
119Sitemap: https://www.unhcr.org/hk/sitemap.xml
120Sitemap: https://www.unhcr.org/rw/sitemap.xml
121Sitemap: https://www.unhcr.org/ng/sitemap.xml
122Sitemap: https://www.unhcr.org/de/sitemap.xml
123Sitemap: https://www.unhcr.org/at/sitemap.xml
124Sitemap: https://www.unhcr.org/ch/sitemap.xml
125Sitemap: https://www.unhcr.org/af/sitemap.xml
126Sitemap: https://www.unhcr.org/ir/sitemap.xml
127Sitemap: https://www.unhcr.org/tr/sitemap.xml
128Sitemap: https://www.unhcr.org/ua/sitemap.xml
129Sitemap: https://www.unhcr.org/jp/sitemap.xml
130Sitemap: https://www.unhcr.org/lb/sitemap.xml
131Sitemap: https://www.unhcr.org/be/sitemap.xml
132Sitemap: https://www.unhcr.org/pk/sitemap.xml
133Sitemap: https://www.unhcr.org/id/sitemap.xml
134Sitemap: https://www.unhcr.org/pl/sitemap.xml
135Sitemap: https://www.unhcr.org/eg/sitemap.xml
136Sitemap: https://www.unhcr.org/gr/sitemap.xml
137Sitemap: https://www.unhcr.org/ph/sitemap.xml
138Sitemap: https://www.unhcr.org/ro/sitemap.xml
139Sitemap: https://www.unhcr.org/ru/sitemap.xml
140Sitemap: https://www.unhcr.org/th/sitemap.xml
141Sitemap: https://www.unhcr.org/ke/sitemap.xml
142Sitemap: https://www.unhcr.org/nl/sitemap.xml
143Sitemap: https://www.unhcr.org/it/sitemap.xml
144Sitemap: https://www.acnur.org/br/sitemap.xml
145Sitemap: https://www.unhcr.org/bd/sitemap.xml
146Sitemap: https://www.unhcr.org/jo/sitemap.xml
147Sitemap: https://www.unhcr.org/kw/sitemap.xml
148Sitemap: https://www.unhcr.org/qa/sitemap.xml
149Sitemap: https://www.unhcr.org/sa/sitemap.xml
150Sitemap: https://www.unhcr.org/central-asia/sitemap.xml
151Sitemap: https://www.unhcr.org/cn/sitemap.xml
152Sitemap: https://www.unhcr.org/iq/sitemap.xml
153Sitemap: https://www.unhcr.org/sy/sitemap.xml
154Sitemap: https://www.unhcr.org/bg/sitemap.xml
155Sitemap: https://www.unhcr.org/cy/sitemap.xml
156Sitemap: https://www.unhcr.org/md/sitemap.xml
157Sitemap: https://www.unhcr.org/rs/sitemap.xml
158Sitemap: https://www.unhcr.org/hu/sitemap.xml
159Sitemap: https://www.unhcr.org/am/sitemap.xml
160Sitemap: https://www.unhcr.org/by/sitemap.xml
161Sitemap: https://www.unhcr.org/sk/sitemap.xml
162Sitemap: https://www.unhcr.org/nordic-and-baltic/sitemap.xml
163Sitemap: https://www.unhcr.org/il/sitemap.xml
164Sitemap: https://www.unhcr.org/co/sitemap.xml

What do the bots actually get?

This page reads robots.txt, which states what unhcr.org asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on unhcr.org Test a path in the robots.txt tester

History

Questions

Does unhcr.org block GPTBot?

No. unhcr.org’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 16 November 2026.

Does unhcr.org block ClaudeBot?

No. unhcr.org’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 16 November 2026.

Does unhcr.org block Google-Extended?

No. unhcr.org’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 16 November 2026.

Does unhcr.org block PerplexityBot?

No. unhcr.org’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 16 November 2026.

Can ChatGPT search find unhcr.org?

Yes. unhcr.org’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 16 November 2026.