Meta

8 crawlers operated by Meta

https://www.facebook.com/

Who Meta is, for site owners

Meta operates 8 crawlers in our directory, filed under Social Bots and Social Media Bots and AI & LLM Bots. Of those 8, 6 rated safe, 2 rated caution for crawl behaviour.

Blocking Meta carries an SEO impact score of 0/10 to 2/10 across its crawlers. 5 of them share the highest blocking impact here: Low - No SEO ranking impact.

Our recommendation for Meta, set by its least reassuring crawler (Meta Muse (muse-), rated caution): Generally safe to allow; provides legitimate crawling value.

Meta at a glance

Operator	Meta
Crawlers in directory	8
Categories	Social Bots, Social Media Bots, AI & LLM Bots
SEO impact if blocked	0/10 to 2/10
Crawl behaviour	6 rated safe, 2 rated caution
Website	www.facebook.com
Official crawler docs	Meta crawler documentation

Crawlers operated by Meta

How to block or allow Meta in robots.txt

These rules cover all 8 crawlers Meta operates. Add them to the robots.txt file at the root of your domain.

Block Meta completely

User-agent: FacebookBot
User-agent: facebookexternalhit
User-agent: Meta-ExternalAds
User-agent: muse-
User-agent: meta-webindexer
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: meta-externaltest
Disallow: /

Allow Meta explicitly

User-agent: FacebookBot
User-agent: facebookexternalhit
User-agent: Meta-ExternalAds
User-agent: muse-
User-agent: meta-webindexer
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: meta-externaltest
Allow: /

robots.txt is a request, not an enforcement mechanism. A crawler that ignores it needs a block at the server or CDN layer. Rules are case-sensitive on the path and matched against the user-agent token, not the full user-agent string.

Frequently asked questions about Meta

What crawlers does Meta run?

Meta operates 8 crawlers in our directory: Facebook Crawler (user-agent FacebookBot), facebookexternalhit (user-agent facebookexternalhit), Meta Ads Crawler (user-agent Meta-ExternalAds), Meta Muse (muse-) (user-agent muse-), Meta Web Indexer (user-agent meta-webindexer), Meta-ExternalAgent (user-agent meta-externalagent), Meta-ExternalFetcher (user-agent meta-externalfetcher), Meta-ExternalTest (user-agent meta-externaltest).

Should I block Meta?

Generally safe to allow; provides legitimate crawling value. That guidance follows this operator's least reassuring crawler, Meta Muse (muse-), rated caution, because allowing an operator allows every crawler it runs. The SEO impact score for blocking is 0/10 to 2/10, where 10 means blocking costs you the most visibility.

How do I block Meta in robots.txt?

Add the following to robots.txt at your domain root: User-agent: FacebookBot then User-agent: facebookexternalhit then User-agent: Meta-ExternalAds then User-agent: muse- then User-agent: meta-webindexer then User-agent: meta-externalagent then User-agent: meta-externalfetcher then User-agent: meta-externaltest then Disallow: /. One Disallow line covers all 8 user-agent tokens.

Does blocking Meta hurt my search rankings?

5 of these share the highest blocking impact here, rated: Low - No SEO ranking impact. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Meta does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.