canaicrawl Get early access

Who blocks AI · #642 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does ccc.de block AI crawlers?

No. ccc.de’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed
robots.txt
www.ccc.de/robots.txt · 6 lines
llms.txt
none
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedno group applies
BingbotMicrosoftAllowedno group applies
DuckDuckBotDuckDuckGoAllowedno group applies
ApplebotAppleAllowedno group applies
PetalBotHuaweiAllowedno group applies
YandexBotYandexAllowedno group applies
BaiduspiderBaiduAllowedno group applies

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIAllowedno group applies
PerplexityBotPerplexityAllowedno group applies
Claude-SearchBotAnthropicAllowedno group applies
DuckAssistBotDuckDuckGoAllowedno group applies
AmazonbotAmazonAllowedno group applies
YouBotYou.comAllowedno group applies

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIAllowedno group applies
ClaudeBotAnthropicAllowedno group applies
Google-ExtendedGoogleAllowedno group applies
CCBotCommon CrawlAllowedno group applies
Applebot-ExtendedAppleAllowedno group applies
GoogleOtherGoogleAllowedno group applies
Meta-ExternalAgentMetaAllowedno group applies
BytespiderByteDanceAllowedno group applies

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIAllowedno group applies
Claude-UserAnthropicAllowedno group applies
Perplexity-UserPerplexityAllowedno group applies
MistralAI-UserMistralAllowedno group applies
Meta-ExternalFetcherMetaAllowedno group applies

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedno group applies
SemrushBotSemrushAllowedno group applies
MJ12botMajesticAllowedno group applies
DotBotMozAllowedno group applies
View www.ccc.de/robots.txt · 6 lines
1# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
2#
3# To ban all spiders from the entire site uncomment the next two lines:
4# User-Agent: *
5# Disallow: /
6

What do the bots actually get?

This page reads robots.txt, which states what ccc.de asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on ccc.de Test a path in the robots.txt tester

History

Questions

Does ccc.de block GPTBot?

No. ccc.de’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does ccc.de block ClaudeBot?

No. ccc.de’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does ccc.de block Google-Extended?

No. ccc.de’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does ccc.de block PerplexityBot?

No. ccc.de’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find ccc.de?

Yes. ccc.de’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.