Quick answer: CCBot is the crawler run by Common Crawl, a nonprofit that publishes a free, public archive of web pages. That archive has been used to train many language models. CCBot respects robots.txt, so blocking it keeps your future pages out of new crawls, but it cannot remove pages already published in past archives.
- Operator
- Common Crawl
- Type
- Open dataset crawler
- robots.txt token
CCBot- Follows robots.txt
- Yes
- Used for
- Building Common Crawl's free public web archive, widely used to train language models
- User agent
CCBot/2.0 (https://commoncrawl.org/faq/)- How to verify
- Match the user agent; Common Crawl crawls from cloud infrastructure, so IP ownership alone proves little
What is CCBot?
Common Crawl releases large snapshots of the web that anyone can download. Researchers and AI companies have used those snapshots as a core ingredient in training data, which is why CCBot matters even though it powers no product of its own.
That also makes it unusual: allowing CCBot is not a relationship with one AI company, it is contributing to a dataset many parties may use.
What happens if you block CCBot?
A Disallow for CCBot stops it from including your pages in future crawls. Previously released archives already contain whatever was crawled before, and copies of those archives exist in many places, so the effect is forward-looking only.
Should you allow CCBot?
If you want the widest possible presence in future models, including open models, allow it. If you opt out of training, blocking CCBot is one of the most effective single lines, because its data reaches many model builders at once.
Check whether CCBot can reach your site
Enter your domain or a specific page. The checker reads your live robots.txt, picks the group CCBot would use, and applies the longest matching rule, the same way major crawlers resolve conflicts.
robots.txt rules for CCBot
Put each group in the robots.txt file at the root of your domain. User agent names are matched without regard to case, and the most specific matching group wins.
User-agent: CCBot
Allow: /User-agent: CCBot
Disallow: /User-agent: CCBot
Disallow: /account/
Disallow: /checkout/
Allow: /The rule most people get wrong: once CCBot has its own group, it ignores everything under User-agent: *. If your wildcard group disallows paths such as /admin/, repeat those lines inside the CCBot group too.
How to verify real CCBot traffic
Match on the CCBot token. Common Crawl runs its crawler on cloud infrastructure, so checking IP ownership is less decisive than it is for Google or Bing. Genuine CCBot fetches robots.txt and honors it, which is the behaviour to check for.
To see how often it visits, count requests by user agent in your access log. On a typical Nginx server:
grep -i "CCBot" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -cCCBot compared with similar agents
| Agent | Operator | Type | Follows robots.txt |
|---|---|---|---|
| CCBot | Common Crawl | Open dataset crawler | Yes |
See all 20 agents side by side in the AI crawler directory.
CCBot FAQ
See every AI crawler signal at once
The free AI SEO audit checks robots.txt access for 12 AI crawlers, plus llms.txt, schema, content and metadata, in a few seconds.
Run the free AI SEO audit