CCBot
CCBot is the crawler of Common Crawl, a non-profit that publishes a free, open archive of the web. Anyone can download that archive, and many AI models have been trained on it, so blocking CCBot is a common way to keep future pages out of AI training sets that draw on it.
- Operator
- Common Crawl
- Purpose
- Model training. Crawlers that collect pages to train models. Blocking them does not affect search or AI answers.
- robots.txt token
CCBot- We test with
CCBot/2.0 (https://commoncrawl.org/faq/)
Common Crawl’s documented string, above.- Documentation
- commoncrawl.org/ccbot
What CCBot does
Common Crawl is a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone.
commoncrawl.org
Common Crawl itself doesn’t train models; it publishes the data, and others use it. Blocking CCBot keeps your future pages out of new crawls, not out of archives already published. Common Crawl also runs an opt-out ledger.
The CCBot user agent
| Request | User agent string |
|---|---|
| Pages | CCBot/2.0 (https://commoncrawl.org/faq/) |
Common Crawl says it may increment the version. Older crawls used CCBot/1.0. Match on CCBot. Common Crawl warns that other crawlers falsely identify themselves as CCBot, so verify before you trust it.
How CCBot reads robots.txt
CCBot is an automated crawler, checking first the robots.txt, and if crawling a page is allowed, fetches pages using HTTP GET requests.
commoncrawl.org
- Crawl-delay is obeyed. Common Crawl says CCBot follows it.
- Changes are picked up on the next check. Common Crawl says it periodically checks whether robots.txt has changed.
- nofollow links aren’t followed.
CCBot IP addresses and how to verify them
Common Crawl documents both checks: CCBot’s IPv4 addresses have reverse DNS names in *.crawl.commoncrawl.org that resolve back to the address (not yet for IPv6), and it publishes its ranges as JSON.
| List | Covers | IPv4 | IPv6 | Dated |
|---|---|---|---|---|
| index.commoncrawl.org/ccbot.json | CCBot | 4 | 1 | 11 August 2026 |
$ host 18.97.14.84
84.14.97.18.in-addr.arpa domain name pointer 18-97-14-84.crawl.commoncrawl.org.
Check an IP now
Paste an address from your logs. We match it against Common Crawl’s current list and run the reverse and forward DNS check, along with the lists of the other operators that publish them.
Block CCBot at your firewall or CDN
robots.txt asks; a firewall enforces. A rule on the user agent refuses CCBot even if it doesn’t honour robots.txt. Match the name case-insensitively and anywhere in the string, never a full string or a version.
Cloudflare
In Security → WAF → Custom rules, create a rule with this expression and the action Block. Cloudflare’s contains is case-sensitive, hence lower().
(lower(http.user_agent) contains "ccbot")
Cloudflare also has one-click controls: AI bot policies, which block AI crawlers by what they do (search, agent or training), and AI Crawl Control, which lets you allow or block crawlers one by one and shows which of them fetch your pages.
nginx
# in the http block
map $http_user_agent $blocked_bot {
default 0;
~*ccbot 1;
}
# in the server block
if ($blocked_bot) { return 403; }
Block CCBot
Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt. This keeps your future pages out of Common Crawl’s archive, and so out of AI training sets built from new crawls. Archives already published stay as they are.
User-agent: CCBot
Disallow: /
Allow CCBot when everything else is blocked
A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.
User-agent: CCBot
Allow: /
User-agent: *
Disallow: /
Does CCBot reach your site?
The checker applies your robots.txt the way CCBot does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on CCBot.
Questions about CCBot
What is CCBot?
CCBot is the crawler of Common Crawl, a non-profit that publishes a free, open archive of the web that anyone can use. Many AI models have been trained on Common Crawl data.
Does CCBot respect robots.txt?
Yes. Common Crawl says CCBot checks robots.txt first, only fetches pages it allows, obeys Crawl-delay and periodically checks for changes.
Does blocking CCBot remove my site from existing AI training data?
No. It stops future crawls. Archives Common Crawl already published, and models trained on them, are unaffected; Common Crawl’s opt-out ledger is its route for other requests.
How do I verify CCBot?
Reverse DNS for its IPv4 addresses ends in crawl.commoncrawl.org and resolves back to the same address, and Common Crawl publishes its ranges at index.commoncrawl.org/ccbot.json (4 IPv4 ranges and 1 IPv6 range in October 2026).
Other ai training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, GoogleOther, Meta-ExternalAgent, Bytespider.
Build a complete file with the robots.txt generator.