Safe AI & LLM Bots

Google-Extended

Operated by Google

Quick Facts

User-Agent:
Google-Extended
Category:
AI & LLM Bots
Operator:
Google
Safety:
Safe
Blocking Impact:
Low - No SEO ranking impact
SEO Impact Score:
2/10
Official docs:
developers.google.com

What is Google-Extended?

A standalone token that allows web publishers to control whether their content is used to train Google's AI models (Gemini, Vertex AI) without affecting Google Search ranking.

A standalone token that allows web publishers to control whether their content is used to train Google's AI models (Gemini, Vertex AI) without affecting Google Search ranking. Google-Extended is an AI data-collection crawler operated by Google. It harvests web content to build or expand training datasets for large language models (LLMs). Unlike search crawlers, Google-Extended does NOT influence your page ranking in any search engine. The user-agent string Google-Extended can be safely blocked via robots.txt, meta tags (noai), or the emerging llms.txt standard without any SEO penalty. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.

What happens if you block Google-Extended?

✅ **No SEO Impact** - Blocking Google-Extended does not affect your rankings in Google, Bing, or any other search engine. Google-Extended is an AI training crawler, not a search indexer. You can freely block it via User-agent: Google-Extended / Disallow: / without any SEO penalty. This is the recommended approach if you want to opt out of Google's LLM training datasets.
Generally safe to allow; provides legitimate crawling value.

How to block Google-Extended with robots.txt

<code>User-agent: Google-Extended</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)
User-agent: Google-Extended Disallow: /
Allow all (robots.txt)
User-agent: Google-Extended Allow: /
Block private only (robots.txt)
User-agent: Google-Extended Disallow: /private/ Disallow: /api/ Disallow: /admin/ Allow: /
Nginx server block
# Nginx: Hard-block Google-Extended if ($http_user_agent ~* "Google\-Extended") { return 403 "Bot blocked"; }
Apache .htaccess
# Apache: Hard-block Google-Extended SetEnvIfNoCase User-Agent "Google\-Extended" bad_bot Order Allow,Deny Allow from all Deny from env=bad_bot
Meta robots tag
<meta name="robots" content="noindex, nofollow">
X-Robots-Tag header
X-Robots-Tag: noindex, nofollow

Is Google-Extended safe to allow?

Yes, Google-Extended is a **safe and legitimate** crawler. It is operated by Google, which publicly documents its crawler at an official URL and follows the Robots Exclusion Protocol (RFC 9309). The user-agent string Google-Extended is verifiable via reverse-DNS lookup on the crawling IP addresses. You can safely allow it unless you have a specific reason to block (e.g., AI training opt-out or SEO tool visibility).
Verify by reverse-DNS lookup: legitimate Google-Extended requests resolve to google's domain.

What does Google-Extended do?

Understanding Google-Extended's purpose helps you decide whether to allow or block it.

  • Collects web content for AI model training
  • Gathers data to improve language understanding
  • Scrapes text, code, and other content for datasets
  • May be used for chatbot and AI assistant development
  • Typically does NOT affect search engine rankings

Frequently Asked Questions

What is the official user-agent string for Google-Extended?
The official user-agent string for Google-Extended is: Google-Extended. This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from Google-Extended by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.
Is Google-Extended safe?
Yes, Google-Extended is a **safe and legitimate** crawler. It is operated by Google, which publicly documents its crawler at an official URL and follows the Robots Exclusion Protocol (RFC 9309). The user-agent string Google-Extended is verifiable via reverse-DNS lookup on the crawling IP addresses. You can safely allow it unless you have a specific reason to block (e.g., AI training opt-out or SEO tool visibility).
Will blocking Google-Extended hurt my SEO?
✅ **No SEO Impact** - Blocking Google-Extended does not affect your rankings in Google, Bing, or any other search engine. Google-Extended is an AI training crawler, not a search indexer. You can freely block it via User-agent: Google-Extended / Disallow: / without any SEO penalty. This is the recommended approach if you want to opt out of Google's LLM training datasets.
How do I block Google-Extended in robots.txt?
Add the following lines to your /robots.txt file: 
User-agent: Google-Extended
Disallow: /
This instructs Google-Extended not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt.
Does Google-Extended respect robots.txt?
Yes - Google-Extended is a well-behaved bot operated by Google. It fetches and parses /robots.txt before crawling any page, following RFC 9309.
How do I verify if Google-Extended is crawling my site?
Search your web server access logs for the string Google-Extended (case-insensitive grep: grep -i "Google-Extended" /var/log/nginx/access.log). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For Google-Extended specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).
What is the crawl frequency of Google-Extended?
Google-Extended crawls at a moderate rate. If you notice excessive traffic in your logs, you can add a Crawl-delay directive: 
User-agent: Google-Extended
Crawl-delay: 10
(10 second delay between requests).
Can I block Google-Extended from specific pages only?
Yes. Instead of a global Disallow: / you can restrict Google-Extended to specific paths: 
User-agent: Google-Extended
Disallow: /private/
Disallow: /staging/
Allow: /
This allows Google-Extended everywhere except the listed paths. Path matching in robots.txt uses prefix matching - Disallow: /private/ blocks /private/page.html but NOT /public/private/.
Does blocking Google-Extended prevent AI training on my content?
Blocking Google-Extended via robots.txt signals to Google that your content should not be used for AI training. However, robots.txt is a **voluntary** protocol - there is no technical enforcement. For stronger protection: 1. Add <meta name="Google-Extended" content="noai, noimageai, noindex"> to your pages. 2. Add a llms.txt file at your domain root (emerging standard). 3. Use Cloudflare WAF or Nginx to return 403 for this user-agent. 4. Consider IP blocklists for Google's known crawler IP ranges.
Is there an alternative to robots.txt to opt out of Google-Extended?
Yes. Several additional opt-out mechanisms exist for AI crawlers: • **Meta tag**: <meta name="Google-Extended" content="noindex"> • **X-Robots-Tag HTTP header**: X-Robots-Tag: noai, noimageai • **llms.txt**: Add a /llms.txt file (similar to robots.txt but for LLMs) • **Server block**: Return 403 or 429 for this user-agent via WAF or Nginx Using multiple layers provides the strongest protection.

Related Bots