AI Crawler Check
Free Bot Analysis Tool
Check
Home / Directory / AI & LLM Bots

AI & LLM Bots Directory

AI training crawlers collecting content for LLM datasets. Blocking has zero SEO impact.

69 total
54 safe
13 caution
2 aggressive

AddSearch AI

Safe
User-agent: AddSearchBot
Operator: AddSearch

AI2Bot

Safe
User-agent: AI2Bot
Operator: Allen Institute for AI

Ai2Bot-Dolma

Caution
User-agent: Ai2Bot-Dolma
Operator: Allen Institute for AI

aiHit Data

Caution
User-agent: aiHitBot
Operator: aiHit

Amazon Buy For Me

Safe
User-agent: AmazonBuyForMe
Operator: Amazon

Amazon User Agent

Safe
User-agent: Amzn-User
Operator: Amazon

Andi Search

Safe
User-agent: Andibot
Operator: Andi

anthropic-ai

Safe
User-agent: anthropic-ai
Operator: Anthropic

AWS Bedrock Bot

Safe
User-agent: bedrockbot
Operator: Amazon

Azure AI Search

Safe
User-agent: AzureAI-SearchBot
Operator: Microsoft

Big Sur AI

Safe
User-agent: Big Sur AI
Operator: Big Sur AI

Brave Leo AI

Safe
User-agent: Bravebot
Operator: Brave

ChatGLM Spider

Caution
User-agent: ChatGLM-Spider
Operator: Zhipu AI

ChatGPT Agent

Safe
User-agent: ChatGPT Agent
Operator: OpenAI

ChatGPT Operator

Safe
User-agent: ChatGPT Operator
Operator: OpenAI

ChatGPT-User

Safe
User-agent: ChatGPT-User
Operator: OpenAI

ChatGPT-User/2.0

Aggressive
User-agent: ChatGPT-User
Operator: OpenAI

Claude-Code

Safe
User-agent: Claude-Code
Operator: Anthropic

Claude-SearchBot

Safe
User-agent: Claude-SearchBot
Operator: Anthropic

Claude-User

Safe
User-agent: Claude-User
Operator: Anthropic

Claude-Web

Safe
User-agent: Claude-Web
Operator: Anthropic

ClaudeBot

Safe
User-agent: ClaudeBot
Operator: Anthropic

Cloudflare AutoRAG

Safe
User-agent: Cloudflare-AutoRAG
Operator: Cloudflare

cohere-ai

Safe
User-agent: cohere-ai
Operator: Cohere

cohere-training-data-crawler

Caution
User-agent: cohere-training-data-crawler
Operator: Cohere

DeepSeekBot

Aggressive
User-agent: DeepSeekBot
Operator: DeepSeek

Devin (Cognition)

Safe
User-agent: Devin
Operator: Cognition AI

DigitalOceanGenAICrawler

Caution
User-agent: DigitalOceanGenAICrawler
Operator: DigitalOcean

DuckAssistBot

Safe
User-agent: DuckAssistBot
Operator: DuckDuckGo

Exa AI Search

Safe
User-agent: ExaBot
Operator: Exa

Firecrawl Agent

Caution
User-agent: FirecrawlAgent
Operator: Firecrawl

Gemini CLI

Safe
User-agent: Google-Gemini-CLI
Operator: Google

Gemini URL Context

Safe
User-agent: GoogleAgent-URLContext
Operator: Google

Google Firebase AI

Safe
User-agent: Google-Firebase
Operator: Google

Google-Extended

Safe
User-agent: Google-Extended
Operator: Google

GPTBot

Safe
User-agent: GPTBot
Operator: OpenAI

Grok

Safe
User-agent: Grok
Operator: Unknown

GrokBot

Safe
User-agent: GrokBot
Operator: Unknown

iAsk AI

Safe
User-agent: iAskBot
Operator: iAsk

img2dataset

Safe
User-agent: img2dataset
Operator: Open Source Community

Kagi Assistant

Safe
User-agent: kagi-fetcher
Operator: Kagi

Kimi (Moonshot AI)

Safe
User-agent: Kimi-User
Operator: Moonshot AI

Klaviyo AI

Safe
User-agent: KlaviyoAIBot
Operator: Klaviyo

LINER Bot

Safe
User-agent: LINER Bot
Operator: LINER

Linkup Search

Safe
User-agent: LinkupBot
Operator: Linkup

Manus AI Agent

Safe
User-agent: Manus-User
Operator: Manus

Meta Web Indexer

Caution
User-agent: meta-webindexer
Operator: Meta

Mistral Search Index

Safe
User-agent: MistralAI-Index
Operator: Mistral AI

MistralAI-User

Caution
User-agent: MistralAI-User
Operator: Mistral AI

MistralAI-User/1.0

Caution
User-agent: MistralAI-User
Operator: Mistral AI

MyCentralAIScraperBot

Safe
User-agent: MyCentralAIScraperBot
Operator: Unknown

NotebookLM Fetcher

Safe
User-agent: NotebookLM
Operator: Google

OAI-SearchBot

Safe
User-agent: OAI-SearchBot
Operator: OpenAI

OpenAI Ads Validator

Safe
User-agent: OAI-AdsBot
Operator: OpenAI

OpenAI Operator

Safe
User-agent: OpenAI
Operator: OpenAI

OpenAI Operator Agent

Safe
User-agent: Operator
Operator: OpenAI

PanguBot

Safe
User-agent: PanguBot
Operator: Huawei

Perplexity-User

Caution
User-agent: Perplexity-User
Operator: Perplexity AI

Perplexity-User/1.0

Caution
User-agent: Perplexity-User
Operator: Perplexity AI

PerplexityBot

Safe
User-agent: PerplexityBot
Operator: Perplexity AI

Phind AI

Safe
User-agent: PhindBot
Operator: Phind

Project Mariner

Safe
User-agent: GoogleAgent-Mariner
Operator: Google

QuillBot AI

Caution
User-agent: QuillBot
Operator: QuillBot

quillbot.com

Caution
User-agent: quillbot.com
Operator: QuillBot

SBIntuitionsBot

Safe
User-agent: SBIntuitionsBot
Operator: Unknown

Tavily Search

Safe
User-agent: TavilyBot
Operator: Tavily

Vertex AI Crawler

Safe
User-agent: CloudVertexBot
Operator: Google

WRTN AI

Safe
User-agent: WRTNBot
Operator: WRTN

YouBot

Safe
User-agent: YouBot
Operator: You

FAQ

How many AI & LLM Bots are tracked?
We currently track 34 AI & LLM Bots with user-agent strings, safety ratings, and blocking rules.
Which AI & LLM Bots should I never block?
Never block bots with 'critical' impact rating — there are 0 in this category. Blocking them removes your site from their search results.
Are all AI & LLM Bots safe?
Out of 34 bots in this category: 24 safe, 8 caution, 2 aggressive.
How do I block all AI & LLM Bots at once?
Generate a combined robots.txt block rule for all AI & LLM Bots using our bulk robots.txt generator, or manually list each User-agent + Disallow: / directive.