Anthropic

6 crawlers operated by Anthropic

Who Anthropic is, for site owners

Anthropic operates 6 crawlers in our directory, filed under AI & LLM Bots. Of those 6, 6 rated safe for crawl behaviour.

Blocking Anthropic carries an SEO impact score of 2/10 across its crawlers. Every crawler here carries the same blocking impact: Low - No SEO ranking impact.

Our recommendation for Anthropic, set by its least reassuring crawler (anthropic-ai, rated safe): Generally safe to allow; provides legitimate crawling value.

Anthropic at a glance

Operator	Anthropic
Crawlers in directory	6
Categories	AI & LLM Bots
SEO impact if blocked	2/10
Crawl behaviour	6 rated safe
Website	anthropic.com
Official crawler docs	Anthropic crawler documentation

Crawlers operated by Anthropic

How to block or allow Anthropic in robots.txt

These rules cover all 6 crawlers Anthropic operates. Add them to the robots.txt file at the root of your domain.

Block Anthropic completely

User-agent: anthropic-ai
User-agent: Claude-Code
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Claude-Web
User-agent: ClaudeBot
Disallow: /

Allow Anthropic explicitly

User-agent: anthropic-ai
User-agent: Claude-Code
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Claude-Web
User-agent: ClaudeBot
Allow: /

robots.txt is a request, not an enforcement mechanism. A crawler that ignores it needs a block at the server or CDN layer. Rules are case-sensitive on the path and matched against the user-agent token, not the full user-agent string.

Frequently asked questions about Anthropic

What crawlers does Anthropic run?

Anthropic operates 6 crawlers in our directory: anthropic-ai (user-agent anthropic-ai), Claude-Code (user-agent Claude-Code), Claude-SearchBot (user-agent Claude-SearchBot), Claude-User (user-agent Claude-User), Claude-Web (user-agent Claude-Web), ClaudeBot (user-agent ClaudeBot).

Should I block Anthropic?

Generally safe to allow; provides legitimate crawling value. That guidance follows this operator's least reassuring crawler, anthropic-ai, rated safe, because allowing an operator allows every crawler it runs. The SEO impact score for blocking is 2/10, where 10 means blocking costs you the most visibility.

How do I block Anthropic in robots.txt?

Add the following to robots.txt at your domain root: User-agent: anthropic-ai then User-agent: Claude-Code then User-agent: Claude-SearchBot then User-agent: Claude-User then User-agent: Claude-Web then User-agent: ClaudeBot then Disallow: /. One Disallow line covers all 6 user-agent tokens.

Does blocking Anthropic hurt my search rankings?

Every crawler here carries the same blocking impact: Low - No SEO ranking impact. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Anthropic does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.