Data Scrapers Directory

Data aggregation and research crawlers. No SEO impact from blocking.

51 total
41 safe
8 caution
2 aggressive

All 51 Data Scrapers

FAQ

How many Data Scrapers are tracked?
We currently track 51 Data Scrapers with user-agent strings, safety ratings, and blocking rules.
Which Data Scrapers should I never block?
Never block bots with 'critical' impact rating - there are 0 in this category. Blocking them removes your site from their search results.
Are all Data Scrapers safe?
Out of 51 bots in this category: 41 safe, 8 caution, 2 aggressive.
How do I block all Data Scrapers at once?
Generate a combined robots.txt block rule for all Data Scrapers using our bulk robots.txt generator, or manually list each User-agent + Disallow: / directive.