Google

27 crawlers operated by Google

https://developers.google.com/

Who Google is, for site owners

Google operates 27 crawlers in our directory, filed under SEO Tools and Google Bots and AI & LLM Bots and Cloud Services and Search Engines. Of those 27, 26 rated safe, 1 rated caution for crawl behaviour.

Blocking Google carries an SEO impact score of 0/10 to 10/10 across its crawlers. 11 of them share the highest blocking impact here: Critical - Blocking removes you from search results.

Our recommendation for Google, set by its least reassuring crawler (Google-CloudVertexBot, rated caution): Consider blocking based on your content strategy.

Google at a glance

Operator	Google
Crawlers in directory	27
Categories	SEO Tools, Google Bots, AI & LLM Bots, Cloud Services, Search Engines
SEO impact if blocked	0/10 to 10/10
Crawl behaviour	26 rated safe, 1 rated caution
Website	developers.google.com
Official crawler docs	Google crawler documentation

Crawlers operated by Google

How to block or allow Google in robots.txt

These rules cover all 27 crawlers Google operates. Add them to the robots.txt file at the root of your domain.

Block Google completely

User-agent: Chrome-Lighthouse
User-agent: FeedFetcher-Google
User-agent: Google-Gemini-CLI
User-agent: GoogleAgent-URLContext
User-agent: Gemini-Deep-Research
User-agent: Google-Firebase
User-agent: Google Page Speed Insights
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: Google-InspectionTool
User-agent: Google-NotebookLM
User-agent: Google-Read-Aloud
User-agent: Google-Site-Verification
User-agent: Googlebot
User-agent: Googlebot-Discovery
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GoogleProducer
User-agent: NotebookLM
User-agent: GoogleAgent-Mariner
User-agent: Storebot-Google
User-agent: CloudVertexBot
Disallow: /

Allow Google explicitly

User-agent: Chrome-Lighthouse
User-agent: FeedFetcher-Google
User-agent: Google-Gemini-CLI
User-agent: GoogleAgent-URLContext
User-agent: Gemini-Deep-Research
User-agent: Google-Firebase
User-agent: Google Page Speed Insights
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: Google-InspectionTool
User-agent: Google-NotebookLM
User-agent: Google-Read-Aloud
User-agent: Google-Site-Verification
User-agent: Googlebot
User-agent: Googlebot-Discovery
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GoogleProducer
User-agent: NotebookLM
User-agent: GoogleAgent-Mariner
User-agent: Storebot-Google
User-agent: CloudVertexBot
Allow: /

robots.txt is a request, not an enforcement mechanism. A crawler that ignores it needs a block at the server or CDN layer. Rules are case-sensitive on the path and matched against the user-agent token, not the full user-agent string.

Frequently asked questions about Google

What crawlers does Google run?

Google operates 27 crawlers in our directory: Chrome-Lighthouse (user-agent Chrome-Lighthouse), FeedFetcher-Google (user-agent FeedFetcher-Google), Gemini CLI (user-agent Google-Gemini-CLI), Gemini URL Context (user-agent GoogleAgent-URLContext), Gemini-Deep-Research (user-agent Gemini-Deep-Research), Google Firebase AI (user-agent Google-Firebase), Google Page Speed Insights (user-agent Google Page Speed Insights), Google-Agent (user-agent Google-Agent), Google-CloudVertexBot (user-agent Google-CloudVertexBot), Google-Extended (user-agent Google-Extended), Google-InspectionTool (user-agent Google-InspectionTool), Google-NotebookLM (user-agent Google-NotebookLM), Google-Read-Aloud (user-agent Google-Read-Aloud), Google-Site-Verification (user-agent Google-Site-Verification), Googlebot (user-agent Googlebot), Googlebot-Discovery (user-agent Googlebot-Discovery), Googlebot-Image (user-agent Googlebot-Image), Googlebot-News (user-agent Googlebot-News), Googlebot-Video (user-agent Googlebot-Video), GoogleOther (user-agent GoogleOther), GoogleOther-Image (user-agent GoogleOther-Image), GoogleOther-Video (user-agent GoogleOther-Video), GoogleProducer (user-agent GoogleProducer), NotebookLM Fetcher (user-agent NotebookLM), Project Mariner (user-agent GoogleAgent-Mariner), Storebot-Google (user-agent Storebot-Google), Vertex AI Crawler (user-agent CloudVertexBot).

Should I block Google?

Consider blocking based on your content strategy. That guidance follows this operator's least reassuring crawler, Google-CloudVertexBot, rated caution, because allowing an operator allows every crawler it runs. The SEO impact score for blocking is 0/10 to 10/10, where 10 means blocking costs you the most visibility.

How do I block Google in robots.txt?

Add the following to robots.txt at your domain root: User-agent: Chrome-Lighthouse then User-agent: FeedFetcher-Google then User-agent: Google-Gemini-CLI then User-agent: GoogleAgent-URLContext then User-agent: Gemini-Deep-Research then User-agent: Google-Firebase then User-agent: Google Page Speed Insights then User-agent: Google-Agent then User-agent: Google-CloudVertexBot then User-agent: Google-Extended then User-agent: Google-InspectionTool then User-agent: Google-NotebookLM then User-agent: Google-Read-Aloud then User-agent: Google-Site-Verification then User-agent: Googlebot then User-agent: Googlebot-Discovery then User-agent: Googlebot-Image then User-agent: Googlebot-News then User-agent: Googlebot-Video then User-agent: GoogleOther then User-agent: GoogleOther-Image then User-agent: GoogleOther-Video then User-agent: GoogleProducer then User-agent: NotebookLM then User-agent: GoogleAgent-Mariner then User-agent: Storebot-Google then User-agent: CloudVertexBot then Disallow: /. One Disallow line covers all 27 user-agent tokens.

Does blocking Google hurt my search rankings?

11 of these share the highest blocking impact here, rated: Critical - Blocking removes you from search results. Those are search-indexing crawlers, so blocking them can remove your pages from the corresponding search results. Block them only if you intend to leave that index. The remaining 16 crawlers in the list above do not affect search indexing and can be blocked separately.