Who blocks AI · #1038 in the Majestic Million

robots.txt · checked 16 Nov 2026

Does hawaii.edu block AI crawlers?

No. hawaii.edu’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed
robots.txt
hawaii.edu/robots.txt · 90 lines
llms.txt
none
Checked
16 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedno rule matches the home page
BingbotMicrosoftAllowedno rule matches the home page
DuckDuckBotDuckDuckGoAllowedno rule matches the home page
ApplebotAppleAllowedno rule matches the home page
PetalBotHuaweiAllowedno rule matches the home page
YandexBotYandexAllowedno rule matches the home page
BaiduspiderBaiduPartly blockedline 37 · Disallow: /bjs line 38 · Disallow: /bls and 23 more

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIAllowedno rule matches the home page
PerplexityBotPerplexityAllowedno rule matches the home page
Claude-SearchBotAnthropicAllowedno rule matches the home page
DuckAssistBotDuckDuckGoAllowedno rule matches the home page
AmazonbotAmazonAllowedno rule matches the home page
YouBotYou.comAllowedno rule matches the home page

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIAllowedno rule matches the home page
ClaudeBotAnthropicAllowedno rule matches the home page
Google-ExtendedGoogleAllowedno rule matches the home page
CCBotCommon CrawlAllowedno rule matches the home page
Applebot-ExtendedAppleAllowedno rule matches the home page
GoogleOtherGoogleAllowedno rule matches the home page
Meta-ExternalAgentMetaAllowedno rule matches the home page
BytespiderByteDanceAllowedno rule matches the home page

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIAllowedno rule matches the home page
Claude-UserAnthropicAllowedno rule matches the home page
Perplexity-UserPerplexityAllowedno rule matches the home page
MistralAI-UserMistralAllowedno rule matches the home page
Meta-ExternalFetcherMetaAllowedno rule matches the home page

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedno rule matches the home page
SemrushBotSemrushAllowedno rule matches the home page
MJ12botMajesticAllowedno rule matches the home page
DotBotMozAllowedno rule matches the home page
View hawaii.edu/robots.txt · 90 lines
1# robots.txt for http://www.hawaii.edu/
2
3User-agent: *
4Disallow: /tip2008
5Disallow: /dir/pdf
6Disallow: /institutionalresearch
7Disallow: /sls/sls
8Disallow: /bjs
9Disallow: /bls
10Disallow: /bwiki
11Disallow: /casl2
12Disallow: /cip
13Disallow: /dw
14Disallow: /ecafe
15Disallow: /epmt
16Disallow: /filedrop
17Disallow: /institutionalresearch
18Disallow: /irodr
19Disallow: /its/ws/holiday
20Disallow: /itsadmin
21Disallow: /jira
22Disallow: /leave
23Disallow: /odsdd
24Disallow: /payroll/paystub
25Disallow: /pfstest
26Disallow: /policy/docs/temp
27Disallow: /prof
28Disallow: /profileService
29Disallow: /taps
30Disallow: /testaging
31Disallow: /transferdatabase
32Disallow: /trvl
33Disallow: /workflow
34Crawl-delay: 3
35
36User-agent: Baiduspider
37Disallow: /bjs
38Disallow: /bls
39Disallow: /bwiki
40Disallow: /casl2
41Disallow: /cip
42Disallow: /dw
43Disallow: /ecafe
44Disallow: /epmt
45Disallow: /filedrop
46Disallow: /institutionalresearch
47Disallow: /irodr
48Disallow: /its/ws/holiday
49Disallow: /itsadmin
50Disallow: /jira
51Disallow: /leave
52Disallow: /odsdd
53Disallow: /payroll/paystub
54Disallow: /pfstest
55Disallow: /prof
56Disallow: /profileService
57Disallow: /taps
58Disallow: /testaging
59Disallow: /transferdatabase
60Disallow: /trvl
61Disallow: /workflow
62
63User-agent: Baiduspider-image
64Disallow: /bjs
65Disallow: /bls
66Disallow: /bwiki
67Disallow: /casl2
68Disallow: /cip
69Disallow: /dw
70Disallow: /ecafe
71Disallow: /epmt
72Disallow: /filedrop
73Disallow: /institutionalresearch
74Disallow: /irodr
75Disallow: /its/ws/holiday
76Disallow: /itsadmin
77Disallow: /jira
78Disallow: /leave
79Disallow: /odsdd
80Disallow: /payroll/paystub
81Disallow: /pfstest
82Disallow: /prof
83Disallow: /profileService
84Disallow: /taps
85Disallow: /testaging
86Disallow: /transferdatabase
87Disallow: /trvl
88Disallow: /workflow
89
90

What do the bots actually get?

This page reads robots.txt, which states what hawaii.edu asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on hawaii.edu Test a path in the robots.txt tester

History

Questions

Does hawaii.edu block GPTBot?

No. hawaii.edu’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 16 November 2026.

Does hawaii.edu block ClaudeBot?

No. hawaii.edu’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 16 November 2026.

Does hawaii.edu block Google-Extended?

No. hawaii.edu’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 16 November 2026.

Does hawaii.edu block PerplexityBot?

No. hawaii.edu’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 16 November 2026.

Can ChatGPT search find hawaii.edu?

Yes. hawaii.edu’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 16 November 2026.