Cloudflare AutoRAG
SafeUser-agent: Cloudflare-AutoRAG
Category: AI & LLM Bots
SEO impact if blocked: 2/10
Cloudflare
1 crawler operated by Cloudflare
https://www.cloudflare.com/Who Cloudflare is, for site owners
Cloudflare operates 1 crawler in our directory, filed under AI & LLM Bots. Its crawl behaviour is rated safe.
Blocking Cloudflare carries an SEO impact score of 2/10. Blocking impact is rated: Low - No SEO ranking impact.
Our recommendation for Cloudflare: Generally safe to allow; provides legitimate crawling value.
Cloudflare at a glance
| Operator | Cloudflare |
|---|---|
| Crawler in directory | 1 |
| Categories | AI & LLM Bots |
| SEO impact if blocked | 2/10 |
| Crawl behaviour | 1 rated safe |
| Website | www.cloudflare.com |
| Official crawler docs | Cloudflare crawler documentation |
Crawler operated by Cloudflare
How to block or allow Cloudflare in robots.txt
This rule covers the single crawler Cloudflare operates. Add it to the robots.txt file at the root of your domain.
Block Cloudflare completely
User-agent: Cloudflare-AutoRAG
Disallow: /
Allow Cloudflare explicitly
User-agent: Cloudflare-AutoRAG
Allow: /
robots.txt is a request, not an enforcement mechanism. A crawler that ignores it needs a block at the server or CDN layer. Rules are case-sensitive on the path and matched against the user-agent token, not the full user-agent string.
Frequently asked questions about Cloudflare
What crawler does Cloudflare run?
Cloudflare operates 1 crawler in our directory: Cloudflare AutoRAG (user-agent Cloudflare-AutoRAG).
Should I block Cloudflare?
Generally safe to allow; provides legitimate crawling value. The SEO impact score for blocking is 2/10, where 10 means blocking costs you the most visibility.
How do I block Cloudflare in robots.txt?
Add the following to robots.txt at your domain root: User-agent: Cloudflare-AutoRAG then Disallow: /.
Does blocking Cloudflare hurt my search rankings?
Blocking impact is rated: Low - No SEO ranking impact. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Cloudflare does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.