Quick answer: Googlebot is Google's main search crawler. AI Overviews and AI Mode are built on Google Search, so a page has to be crawled by Googlebot, indexed, and eligible to show a snippet before it can be used in those AI answers. Blocking Googlebot removes you from Search entirely; it is never a sensible AI opt-out.
- Operator
- Type
- Search engine crawler
- robots.txt token
Googlebot- Follows robots.txt
- Yes
- Used for
- Crawling for Google Search, including AI Overviews and AI Mode
- User agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html), plus a smartphone variant- How to verify
- Reverse DNS must end in googlebot.com, google.com or googleusercontent.com, confirmed by a forward lookup
What is Googlebot?
Every AI robots.txt conversation eventually runs into Google, because Google's AI features do not use a separate crawler. AI Overviews and AI Mode draw from the same index as regular results, so Googlebot access is the entry ticket.
Google gives you two different levers. Google-Extended controls use in Gemini outside of Search. Snippet controls such as nosnippet, max-snippet, and data-nosnippet control how much of your text Search features, including AI Overviews, may show.
What happens if you block Googlebot?
Disallowing Googlebot stops Google from crawling your pages, which over time removes them from Google Search, AI Overviews, and AI Mode together. There is no Googlebot setting that keeps you in blue links but out of AI Overviews. If you want to limit AI use of a page, snippet directives are the tool Google provides.
Should you allow Googlebot?
Always allow Googlebot on public pages you want found. Use Disallow only for areas with no search value, such as internal search results, carts, or admin paths, and use nosnippet or data-nosnippet if specific text should not be quoted.
Check whether Googlebot can reach your site
Enter your domain or a specific page. The checker reads your live robots.txt, picks the group Googlebot would use, and applies the longest matching rule, the same way major crawlers resolve conflicts.
robots.txt rules for Googlebot
Put each group in the robots.txt file at the root of your domain. User agent names are matched without regard to case, and the most specific matching group wins.
User-agent: Googlebot
Allow: /User-agent: Googlebot
Disallow: /User-agent: Googlebot
Disallow: /account/
Disallow: /checkout/
Allow: /The rule most people get wrong: once Googlebot has its own group, it ignores everything under User-agent: *. If your wildcard group disallows paths such as /admin/, repeat those lines inside the Googlebot group too.
Common Google setup: stay in Google Search and AI Overviews, opt out of Gemini training
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /How to verify real Googlebot traffic
Google recommends a reverse DNS lookup on the IP: genuine requests resolve to a hostname ending in googlebot.com, google.com, or googleusercontent.com, and a forward lookup of that hostname returns the same IP. Google also publishes its crawler IP ranges in JSON files for automated checks.
To see how often it visits, count requests by user agent in your access log. On a typical Nginx server:
grep -i "Googlebot" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -cGooglebot compared with other Google agents
| Agent | Operator | Type | Follows robots.txt |
|---|---|---|---|
| Google-Extended | robots.txt control token | Yes, it only exists in robots.txt | |
| Googlebot | Search engine crawler | Yes |
See all 20 agents side by side in the AI crawler directory.
Googlebot FAQ
See every AI crawler signal at once
The free AI SEO audit checks robots.txt access for 12 AI crawlers, plus llms.txt, schema, content and metadata, in a few seconds.
Run the free AI SEO audit