Bot directory · AI training

AI training · Common Crawl

CCBot

CCBot is the crawler of Common Crawl, a non-profit that publishes a free, open archive of the web. Anyone can download that archive, and many AI models have been trained on it, so blocking CCBot is a common way to keep future pages out of AI training sets that draw on it.

Checked against Common Crawl’s documentation on 19 October 2026

Operator
Common Crawl
Purpose
Model training. Crawlers that collect pages to train models. Blocking them does not affect search or AI answers.
robots.txt token
CCBot
We test with
CCBot/2.0 (https://commoncrawl.org/faq/)
Common Crawl’s documented string, above.
Documentation
commoncrawl.org/ccbot

What CCBot does

Common Crawl is a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone.

commoncrawl.org

Common Crawl itself doesn’t train models; it publishes the data, and others use it. Blocking CCBot keeps your future pages out of new crawls, not out of archives already published. Common Crawl also runs an opt-out ledger.

The CCBot user agent

RequestUser agent string
PagesCCBot/2.0 (https://commoncrawl.org/faq/)

Common Crawl says it may increment the version. Older crawls used CCBot/1.0. Match on CCBot. Common Crawl warns that other crawlers falsely identify themselves as CCBot, so verify before you trust it.

How CCBot reads robots.txt

CCBot is an automated crawler, checking first the robots.txt, and if crawling a page is allowed, fetches pages using HTTP GET requests.

commoncrawl.org

CCBot IP addresses and how to verify them

Common Crawl documents both checks: CCBot’s IPv4 addresses have reverse DNS names in *.crawl.commoncrawl.org that resolve back to the address (not yet for IPv6), and it publishes its ranges as JSON.

ListCoversIPv4IPv6Dated
index.commoncrawl.org/ccbot.jsonCCBot4111 August 2026
$ host 18.97.14.84
84.14.97.18.in-addr.arpa domain name pointer 18-97-14-84.crawl.commoncrawl.org.

Check an IP now

Paste an address from your logs. We match it against Common Crawl’s current list and run the reverse and forward DNS check, along with the lists of the other operators that publish them.

Block CCBot at your firewall or CDN

robots.txt asks; a firewall enforces. A rule on the user agent refuses CCBot even if it doesn’t honour robots.txt. Match the name case-insensitively and anywhere in the string, never a full string or a version.

Cloudflare

In Security → WAF → Custom rules, create a rule with this expression and the action Block. Cloudflare’s contains is case-sensitive, hence lower().

(lower(http.user_agent) contains "ccbot")

Cloudflare also has one-click controls: AI bot policies, which block AI crawlers by what they do (search, agent or training), and AI Crawl Control, which lets you allow or block crawlers one by one and shows which of them fetch your pages.

nginx

# in the http block
map $http_user_agent $blocked_bot {
    default 0;
    ~*ccbot 1;
}

# in the server block
if ($blocked_bot) { return 403; }

Block CCBot

Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt. This keeps your future pages out of Common Crawl’s archive, and so out of AI training sets built from new crawls. Archives already published stay as they are.

User-agent: CCBot
Disallow: /

Allow CCBot when everything else is blocked

A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.

User-agent: CCBot
Allow: /

User-agent: *
Disallow: /

Does CCBot reach your site?

The checker applies your robots.txt the way CCBot does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on CCBot.

Questions about CCBot

What is CCBot?

CCBot is the crawler of Common Crawl, a non-profit that publishes a free, open archive of the web that anyone can use. Many AI models have been trained on Common Crawl data.

Does CCBot respect robots.txt?

Yes. Common Crawl says CCBot checks robots.txt first, only fetches pages it allows, obeys Crawl-delay and periodically checks for changes.

Does blocking CCBot remove my site from existing AI training data?

No. It stops future crawls. Archives Common Crawl already published, and models trained on them, are unaffected; Common Crawl’s opt-out ledger is its route for other requests.

How do I verify CCBot?

Reverse DNS for its IPv4 addresses ends in crawl.commoncrawl.org and resolves back to the same address, and Common Crawl publishes its ranges at index.commoncrawl.org/ccbot.json (4 IPv4 ranges and 1 IPv6 range in October 2026).

Other ai training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, GoogleOther, Meta-ExternalAgent, Bytespider.

Build a complete file with the robots.txt generator.