Web Crawler & Bot Directory

196 bots across 8 categories, with 196 written profiles covering user-agent strings, safety ratings, robots.txt rules and blocking guides.

AI & LLM Bots

AI training crawlers collecting content for LLM datasets. Blocking has zero SEO impact.

ChatGPT Agent ChatGPT Operator ChatGPT-User/2.0 GPTBot OAI-SearchBot

Search engine crawlers that index your content for organic search visibility. Critical to allow.

Amzn-SearchBot Applebot Applebot-Extended AspiegelBot Baiduspider

Commercial SEO platform crawlers (Ahrefs, Semrush etc.). Blocking hides your data from competitors.

AhrefsBot AudigentAdBot AwarioRssBot AwarioSmartBot Brandwatch

Social media crawlers fetching Open Graph metadata for link previews.

FacebookBot LinkedInBot Meta-ExternalAgent Meta-ExternalFetcher Pinterestbot

Google's specialized crawlers for Images, Video, News and other services.

Gemini-Deep-Research Google-InspectionTool GoogleOther GoogleOther-Image GoogleOther-Video

Data aggregation and research crawlers. No SEO impact from blocking.

Amazon Kendra Arquivo-web-crawler BLEXBot Barkrowler Bravest

Cloud provider crawlers used for security scanning, CDN warm-up or threat intelligence.

AliyunSecBot Amazonbot Google-CloudVertexBot

Miscellaneous crawlers not fitting other categories.

TurnitinBot ViennaTinyBot archive.org_bot ia_archiver ia_archiver-web.archive.org