Quick answer: Google-Extended is a robots.txt product token, not a separate crawler. It lets you control whether content Google crawls from your site may be used to train Gemini models and to ground answers in Gemini apps and Vertex AI. It does not affect your inclusion or ranking in Google Search, and it does not control AI Overviews.
- Operator
- Type
- robots.txt control token
- robots.txt token
Google-Extended- Follows robots.txt
- Yes, it only exists in robots.txt
- Used for
- Controlling whether Google may use your content to train Gemini models and to ground Gemini answers
- User agent
No separate user agent. Crawling is done by Google's existing crawlers.- How to verify
- Nothing to verify in logs: it never appears as a user agent
What is Google-Extended?
Google crawls your site with its normal crawlers, such as Googlebot. Google-Extended is a label you can reference in robots.txt to tell Google how that crawled content may be used for its generative AI products outside of Search.
Because it is only a token, you will never see Google-Extended in your server logs. Anyone telling you they saw "Google-Extended traffic" is looking at something else.
What happens if you block Google-Extended?
Adding User-agent: Google-Extended with Disallow: / tells Google not to use your content for Gemini model training or Gemini grounding. Your pages are still crawled by Googlebot, still ranked in Google Search, and still eligible for AI Overviews and AI Mode, which Google treats as part of Search.
Should you allow Google-Extended?
Decide based on Gemini specifically. If you want Gemini apps to use your content when answering, leave it allowed. If you object to training use, blocking it is low risk for search traffic. Just do not expect it to remove you from AI Overviews; the only controls there are Search-level ones such as nosnippet.
Check whether Google-Extended can reach your site
Enter your domain or a specific page. The checker reads your live robots.txt, picks the group Google-Extended would use, and applies the longest matching rule, the same way major crawlers resolve conflicts.
robots.txt rules for Google-Extended
Google-Extended only exists inside robots.txt, so these rules are the entire control. They do not stop Googlebot from crawling; they tell Google how the crawled content may be used.
User-agent: Google-Extended
Allow: /User-agent: Google-Extended
Disallow: /The rule most people get wrong: once Google-Extended has its own group, it ignores everything under User-agent: *. If your wildcard group disallows paths such as /admin/, repeat those lines inside the Google-Extended group too.
Common Google setup: stay in Google Search and AI Overviews, opt out of Gemini training
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /How to verify real Google-Extended traffic
There is nothing to verify in logs, because no request ever carries this name. To confirm the setting, read your live robots.txt and make sure the Google-Extended group exists and is not overridden by a typo. The checker on this page reads it the same way Google parses groups.
Google-Extended compared with other Google agents
| Agent | Operator | Type | Follows robots.txt |
|---|---|---|---|
| Google-Extended | robots.txt control token | Yes, it only exists in robots.txt | |
| Googlebot | Search engine crawler | Yes |
See all 20 agents side by side in the AI crawler directory.
Google-Extended FAQ
See every AI crawler signal at once
The free AI SEO audit checks robots.txt access for 12 AI crawlers, plus llms.txt, schema, content and metadata, in a few seconds.
Run the free AI SEO audit