GlossaryAI search
What is Google-Extended?
Also called: Google Extended user agent
Definition
Google-Extended is a robots.txt token that controls whether content Google crawls from your site may be used to train Gemini models and for grounding. It doesn't affect Google Search, including AI Overviews.
Google-Extended, explained
Google-Extended isn't a crawler. Google's documentation says crawling is done with existing Google user agents, and the Google-Extended token is used in a control capacity. You write rules for it in robots.txt like any other bot, but no requests ever arrive with that name in your logs.
What it controls: whether content Google crawls from your site may be used for training future generations of Gemini models, and for grounding, which Google defines as providing content from the Google Search index to the model at prompt time to improve factuality and relevancy. Google lists the products it covers: Gemini Apps, the Vertex AI API for Gemini, and Grounding with Google Search on Vertex AI.
What it doesn't control: Google Search. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. AI Overviews and AI Mode are part of Search, so blocking Google-Extended doesn't remove you from them. To limit what appears in AI Overviews, you'd use snippet controls like nosnippet or max-snippet, which also affect your regular results.
So the decision is narrower than it looks. Blocking Google-Extended keeps your content out of Gemini training and Gemini's grounding. It won't cost you Google rankings, but it can mean Gemini is less likely to use or cite your pages in answers that rely on grounding.
Apple has a similar control token, Applebot-Extended, for its AI training. Both follow the same pattern: the normal crawler keeps crawling for search, and the extra token decides AI use.
Why it matters for founders
Founders often block Google-Extended thinking it keeps them out of AI Overviews, or avoid blocking it fearing SEO damage. Neither is right. Knowing what it does lets you make the trade-off on purpose.
Example
A publisher adds User-agent: Google-Extended / Disallow: /. Its pages keep ranking and appearing in AI Overviews, but Google won't use them for Gemini training or grounding.
Common mistakes
- Expecting it to remove your site from AI Overviews.
- Worrying that blocking it hurts Google rankings. Google says it doesn't.
- Looking for Google-Extended requests in server logs.
- Blocking Googlebot instead, which removes you from Search entirely.
Sources
Checked
Related terms
- AI crawlersAI crawlers are bots run by AI companies that fetch web pages, either to collect training data, to build an index for AI search answers, or to read a page a user asked about. Each has its own robots.txt name.
- robots.txtrobots.txt is a plain-text file at the root of a domain that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing, and well-behaved bots follow it voluntarily.
- AI OverviewsAI Overviews are AI-generated summaries Google shows at the top of some search results, with links to supporting pages. AI Mode is Google's separate, conversational search experience built on the same idea.
- GroundingGrounding is giving an AI model real source content, such as search results or documents, at the moment it answers, so the answer is based on that content instead of only what the model memorized in training.