canaicrawl Get early access

Who blocks AI · #818 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does wa.gov block AI crawlers?

Yes. wa.gov’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks Googlebot, Bingbot and 5 more among search engines.

GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked
robots.txt
wa.gov/robots.txt · 10 lines
llms.txt
none
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleBlockedline 2 · Disallow: / under User-agent: *
BingbotMicrosoftBlockedline 2 · Disallow: / under User-agent: *
DuckDuckBotDuckDuckGoBlockedline 2 · Disallow: / under User-agent: *
ApplebotAppleBlockedline 2 · Disallow: / under User-agent: *
PetalBotHuaweiBlockedline 2 · Disallow: / under User-agent: *
YandexBotYandexBlockedline 2 · Disallow: / under User-agent: *
BaiduspiderBaiduBlockedline 2 · Disallow: / under User-agent: *

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIBlockedline 2 · Disallow: / under User-agent: *
PerplexityBotPerplexityBlockedline 2 · Disallow: / under User-agent: *
Claude-SearchBotAnthropicBlockedline 2 · Disallow: / under User-agent: *
DuckAssistBotDuckDuckGoBlockedline 2 · Disallow: / under User-agent: *
AmazonbotAmazonBlockedline 2 · Disallow: / under User-agent: *
YouBotYou.comBlockedline 2 · Disallow: / under User-agent: *

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIBlockedline 2 · Disallow: / under User-agent: *
ClaudeBotAnthropicBlockedline 2 · Disallow: / under User-agent: *
Google-ExtendedGoogleBlockedline 2 · Disallow: / under User-agent: *
CCBotCommon CrawlBlockedline 2 · Disallow: / under User-agent: *
Applebot-ExtendedAppleBlockedline 2 · Disallow: / under User-agent: *
GoogleOtherGoogleBlockedline 2 · Disallow: / under User-agent: *
Meta-ExternalAgentMetaBlockedline 2 · Disallow: / under User-agent: *
BytespiderByteDanceBlockedline 2 · Disallow: / under User-agent: *

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIBlockedline 2 · Disallow: / under User-agent: *
Claude-UserAnthropicBlockedline 2 · Disallow: / under User-agent: *
Perplexity-UserPerplexityBlockedline 2 · Disallow: / under User-agent: *
MistralAI-UserMistralBlockedline 2 · Disallow: / under User-agent: *
Meta-ExternalFetcherMetaBlockedline 2 · Disallow: / under User-agent: *

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsBlockedline 2 · Disallow: / under User-agent: *
SemrushBotSemrushBlockedline 2 · Disallow: / under User-agent: *
MJ12botMajesticBlockedline 2 · Disallow: / under User-agent: *
DotBotMozBlockedline 2 · Disallow: / under User-agent: *
View wa.gov/robots.txt · 10 lines
1User-Agent: *
2Disallow: /
3
4User-Agent: SiteimproveBot
5Disallow:
6
7User-Agent: LiferayBot
8Disallow:
9
10Sitemap: https://wa.gov:443/sitemap.xml

What do the bots actually get?

This page reads robots.txt, which states what wa.gov asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on wa.gov Test a path in the robots.txt tester

History

Questions

Does wa.gov block GPTBot?

Yes. wa.gov’s robots.txt disallows GPTBot from the whole site (line 2: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does wa.gov block ClaudeBot?

Yes. wa.gov’s robots.txt disallows ClaudeBot from the whole site (line 2: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does wa.gov block Google-Extended?

Yes. wa.gov’s robots.txt disallows Google-Extended from the whole site (line 2: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does wa.gov block PerplexityBot?

Yes. wa.gov’s robots.txt disallows PerplexityBot from the whole site (line 2: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find wa.gov?

Not through its own crawler. wa.gov’s robots.txt disallows OAI-SearchBot from the whole site (line 2: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.