Google-Extended
Google-Extended is a robots.txt token, not a crawler. Disallowing it tells Google not to use pages it crawls from your site to train future Gemini models or to ground Gemini’s answers. Google says it doesn’t affect your inclusion or ranking in Google Search.
- Operator
- Purpose
- Model training. Crawlers that collect pages to train models. Blocking them does not affect search or AI answers.
- robots.txt token
Google-Extended- We test with
- None. This is a robots.txt token, not a crawler: the operator’s normal crawler fetches the page and this name only governs how it may be used.
What Google-Extended does
Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI.
developers.google.com
Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.
developers.google.com
AI Overviews and AI Mode in Google Search are part of Search, which Googlebot crawls for; Google-Extended governs Gemini Apps and Vertex AI.
The Google-Extended user agent
None. Google says Google-Extended has no user agent string of its own: Google’s existing crawlers fetch the pages, and the token is only read from robots.txt. You won’t see it in your logs.
How Google-Extended reads robots.txt
User-agent: Google-Extended
Disallow: /
- It is read with Google’s other robots.txt rules, which Google generally caches for up to 24 hours.
- Path rules work. Disallow only the sections you want kept out.
- Don’t block Googlebot to achieve this. That removes you from Google Search.
Google-Extended IP addresses
Google-Extended has no addresses of its own. Google says its common crawlers generally crawl from the ranges in common-crawlers.json; to check one, use the Googlebot verifier.
Can you block Google-Extended at a firewall?
No. No request carries the name, and blocking Google’s crawlers at the firewall would remove you from Google Search. robots.txt is the only control.
Block Google-Extended
Add a group naming it to /robots.txt. Nothing stops fetching; the operator’s crawler still reads the page for its normal purpose, but may not use it this way. Googlebot keeps crawling and indexing your pages for Google Search; Google just won’t use them for Gemini training or grounding.
User-agent: Google-Extended
Disallow: /
Allow Google-Extended when everything else is blocked
A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.
User-agent: Google-Extended
Allow: /
User-agent: *
Disallow: /
Does Google-Extended reach your site?
The checker applies your robots.txt the way Google-Extended does, alongside 29 other bots, and opens the report on Google-Extended.
Questions about Google-Extended
What is Google-Extended?
A robots.txt token Google reads to decide whether content it crawls from your site may be used to train future Gemini models and to ground answers in Gemini Apps and Vertex AI. It isn’t a separate crawler.
Does blocking Google-Extended affect my Google rankings?
No. Google says Google-Extended does not affect a site’s inclusion in Google Search and isn’t used as a ranking signal.
Does Google-Extended stop AI Overviews using my site?
No. AI Overviews are part of Google Search, which Googlebot crawls for. Google-Extended covers Gemini Apps and Vertex AI.
Why don’t I see Google-Extended in my logs?
It has no user agent. Google’s regular crawlers fetch pages and the token is only read from robots.txt.
Also run by Google: Googlebot, GoogleOther.
Other ai training: GPTBot, ClaudeBot, CCBot, Applebot-Extended, GoogleOther, Meta-ExternalAgent, Bytespider.
Build a complete file with the robots.txt generator.