Who blocks AI · #1134 in the Majestic Million
Does gohugo.io block AI crawlers?
No. gohugo.io has no robots.txt, so every crawler may crawl it, AI crawlers included.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- gohugo.io/robots.txt · answers 404: no robots.txt
- llms.txt
- none
- Checked
- 16 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no robots.txt | |
Bingbot | Microsoft | Allowed | no robots.txt |
DuckDuckBot | DuckDuckGo | Allowed | no robots.txt |
Applebot | Apple | Allowed | no robots.txt |
PetalBot | Huawei | Allowed | no robots.txt |
YandexBot | Yandex | Allowed | no robots.txt |
Baiduspider | Baidu | Allowed | no robots.txt |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no robots.txt |
PerplexityBot | Perplexity | Allowed | no robots.txt |
Claude-SearchBot | Anthropic | Allowed | no robots.txt |
DuckAssistBot | DuckDuckGo | Allowed | no robots.txt |
Amazonbot | Amazon | Allowed | no robots.txt |
YouBot | You.com | Allowed | no robots.txt |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no robots.txt |
ClaudeBot | Anthropic | Allowed | no robots.txt |
Google-Extended | Allowed | no robots.txt | |
CCBot | Common Crawl | Allowed | no robots.txt |
Applebot-Extended | Apple | Allowed | no robots.txt |
GoogleOther | Allowed | no robots.txt | |
Meta-ExternalAgent | Meta | Allowed | no robots.txt |
Bytespider | ByteDance | Allowed | no robots.txt |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no robots.txt |
Claude-User | Anthropic | Allowed | no robots.txt |
Perplexity-User | Perplexity | Allowed | no robots.txt |
MistralAI-User | Mistral | Allowed | no robots.txt |
Meta-ExternalFetcher | Meta | Allowed | no robots.txt |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no robots.txt |
SemrushBot | Semrush | Allowed | no robots.txt |
MJ12bot | Majestic | Allowed | no robots.txt |
DotBot | Moz | Allowed | no robots.txt |
What do the bots actually get?
This page reads robots.txt, which states what gohugo.io asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on gohugo.io Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does gohugo.io block GPTBot?
No. gohugo.io has no robots.txt, so nothing stops GPTBot. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 16 November 2026.
Does gohugo.io block ClaudeBot?
No. gohugo.io has no robots.txt, so nothing stops ClaudeBot. ClaudeBot is Anthropic’s crawler for training data. Checked 16 November 2026.
Does gohugo.io block Google-Extended?
No. gohugo.io has no robots.txt, so nothing stops Google-Extended. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 16 November 2026.
Does gohugo.io block PerplexityBot?
No. gohugo.io has no robots.txt, so nothing stops PerplexityBot. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 16 November 2026.
Can ChatGPT search find gohugo.io?
Yes. gohugo.io has no robots.txt, so nothing stops OAI-SearchBot. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 16 November 2026.