CCBot
CautionUser-agent: CCBot
Category: Data Scrapers
SEO impact if blocked: 2/10
Common Crawl
1 crawler operated by Common Crawl
https://commoncrawl.org/Who Common Crawl is, for site owners
Common Crawl operates 1 crawler in our directory, filed under Data Scrapers. Its crawl behaviour is rated caution.
Blocking Common Crawl carries an SEO impact score of 2/10. Blocking impact is rated: Low - No SEO ranking impact.
Our recommendation for Common Crawl: Consider blocking based on your content strategy.
Common Crawl at a glance
| Operator | Common Crawl |
|---|---|
| Crawler in directory | 1 |
| Categories | Data Scrapers |
| SEO impact if blocked | 2/10 |
| Crawl behaviour | 1 rated caution |
| Website | commoncrawl.org |
| Official crawler docs | Common Crawl crawler documentation |
Crawler operated by Common Crawl
How to block or allow Common Crawl in robots.txt
This rule covers the single crawler Common Crawl operates. Add it to the robots.txt file at the root of your domain.
Block Common Crawl completely
User-agent: CCBot
Disallow: /
Allow Common Crawl explicitly
User-agent: CCBot
Allow: /
robots.txt is a request, not an enforcement mechanism. A crawler that ignores it needs a block at the server or CDN layer. Rules are case-sensitive on the path and matched against the user-agent token, not the full user-agent string.
Frequently asked questions about Common Crawl
What crawler does Common Crawl run?
Common Crawl operates 1 crawler in our directory: CCBot (user-agent CCBot).
Should I block Common Crawl?
Consider blocking based on your content strategy. The SEO impact score for blocking is 2/10, where 10 means blocking costs you the most visibility.
How do I block Common Crawl in robots.txt?
Add the following to robots.txt at your domain root: User-agent: CCBot then Disallow: /.
Does blocking Common Crawl hurt my search rankings?
Blocking impact is rated: Low - No SEO ranking impact. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Common Crawl does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.