Robots.txt for AI: How to Let AI Crawlers In (Without Wrecking Your SEO)

The short version: Your robots.txt now controls access to two completely separate systems: traditional search engines and AI answer engines. They need separate configuration. Ignoring one means missing out on an entire traffic channel.

What robots.txt actually does

Your robots.txt file sits at the root of your website (think yoursite.com/robots.txt) and acts as a set of instructions for automated crawlers. When a bot visits your site, it checks this file first and follows the rules you have written. Simple concept, massive practical consequences.

The format was created in 1994 via the Robots Exclusion Protocol and has barely changed since. You list a User-agent name, then tell it what it can and cannot access using Allow: and Disallow: directives. If a path is not covered by any rule, crawlers can access it by default.

For years, the only crawlers that really mattered were Googlebot, Bingbot, and a handful of others. Most site owners treated robots.txt as something you set once and forgot about. That was fine in 2018. It is not fine in 2026.

Today, a new category of crawler has arrived: AI training bots and AI retrieval bots. These crawlers power products like ChatGPT, Claude, and Perplexity. Whether your content shows up in their answers depends partly on whether you have given them permission to crawl your site. And a lot of site owners have accidentally locked them out without knowing it.

Key Takeaway

Your robots.txt now controls access to two completely separate systems: traditional search engines and AI answer engines. They need separate configuration. Ignoring one means missing out on an entire traffic channel.

This article walks through every major AI crawler, the mistakes that accidentally block them, how to write a proper robots.txt that gives you control, and how to check that your settings are actually working. If you want to skip ahead and just see your current status, the AI Crawler Checker will show you which bots your current file is blocking.

How AI crawlers read robots.txt

Good news: AI crawlers follow the exact same Robots Exclusion Protocol as Google and Bing. There is no new standard to learn. They visit /robots.txt, read the file, and obey User-agent, Allow, and Disallow directives just like every other well-behaved crawler.

The key difference is the user-agent string. Googlebot identifies itself as Googlebot. OpenAI's training crawler identifies itself as GPTBot. Anthropic's crawler says ClaudeBot. If your robots.txt has a rule for User-agent: * that disallows something, that rule applies to all of them unless you have a more specific rule for that particular user-agent.

Two types of AI crawlers

It helps to understand that AI crawlers fall into two categories:

• Training crawlers collect content to train AI models. They visit your site, store the content, and it may end up in a future model's training data. GPTBot and ClaudeBot are in this group.
• Retrieval crawlers fetch content in real time when a user asks a question. They want to include your content in an answer right now. OAI-SearchBot (for SearchGPT) and PerplexityBot are in this group.

This distinction matters because blocking a training crawler means your content is not in future AI responses. Blocking a retrieval crawler means your content does not show up in current AI search answers, even if your site is excellent. Different goals, different consequences for blocking.

Key Takeaway

AI crawlers follow the same robots.txt rules as traditional search crawlers. The only thing that changes is the user-agent name. A wildcard Disallow affects them all unless you override it with bot-specific rules.

Complete list of AI bot user-agents

Here is every major AI crawler you should know about as of mid-2026. You will need the exact user-agent strings when writing robots.txt rules.

Note that Google-Extended is separate from Googlebot. Blocking Google-Extended only prevents your content from going into Gemini AI training. Googlebot continues crawling your site for regular search results completely unaffected. This is the same pattern you will see across the board: AI-specific user-agents are isolated from the traditional search crawlers.

If you want to quickly audit which of these your current robots.txt is blocking, our free AI Crawler Checker tests all of them automatically.

Common mistakes that accidentally block AI bots

Most sites that are blocking AI crawlers are not doing it intentionally. They ended up that way through a plugin update, a security configuration, or a well-meaning developer who set something overly broad. Here are the most common culprits.

The wildcard Disallow everything rule

This is the most common accident. Someone pastes this into their robots.txt (often while setting up a staging environment) and never removes it:

User-agent: *
Disallow: /

This single rule blocks every crawler on the planet. Googlebot. Bingbot. GPTBot. All of them. If your site is live and has this in robots.txt, you have a serious problem that goes well beyond AI crawlers.

WordPress security plugins

Some WordPress security plugins (Wordfence, iThemes Security, and others) add crawler restrictions to your robots.txt automatically. They often block unfamiliar user-agents as a protective measure. The problem is that AI crawlers are relatively new, so many of these plugins flag them as suspicious. Check your robots.txt after installing or updating any security plugin.

Cloudflare bot fight mode and security rules

Cloudflare's Bot Fight Mode and Super Bot Fight Mode are designed to block automated traffic. Unless you have specifically whitelisted AI crawlers, they may be getting blocked at the network level, before they even reach your robots.txt. This is a separate layer of access control that robots.txt cannot override.

If you use Cloudflare, check your firewall rules and Bot Fight Mode settings. You may need to create exceptions for AI crawler IP ranges, which each company publishes (OpenAI publishes theirs at https://openai.com/gptbot-ranges.txt).

Overly broad third-party disallow rules

Some SEO tools or site migration services generate robots.txt files that block everything except Googlebot and Bingbot. This made sense before AI crawlers existed. Now it means you have explicitly told every AI bot to leave. Common pattern:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

That setup gives Google and Bing access while locking out every other crawler, including all AI bots. If you inherited this pattern, fixing it is straightforward, covered in the next section.

CDN and caching layer rewrites

If your CDN or reverse proxy serves a cached robots.txt, edits to your actual file may not propagate immediately. After making changes, verify by visiting your live /robots.txt URL directly in a browser, not just looking at the source file on your server.

Key Takeaway

Most accidental AI bot blocks come from wildcard Disallow rules, security plugins, or Cloudflare settings. Check your live robots.txt URL in a browser right now and scan it with the AI Crawler Checker to see your actual status.

How to allow specific AI bots

If you want fine-grained control, granting access to individual crawlers is straightforward. Here is the pattern.

Allowing GPTBot while keeping a wildcard block

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Allowing Claude-related crawlers

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

Allowing Perplexity and Google's AI crawler

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

The order of rules in robots.txt matters for some crawlers. Put more specific user-agent rules after the wildcard block so they override it correctly. Most major AI bots do respect this ordering.

If this feels tedious to manage, the Robots.txt Generator lets you check boxes for each AI bot you want to allow and generates the correct file instantly.

How to allow all AI bots at once

If your goal is maximum AI visibility and you want every legitimate crawler to have full access, the simplest approach is a universal allow. This is the configuration most content sites and publishers should start with.

User-agent: *
Allow: /

That is the entire file. Two lines. The wildcard * applies to every crawler, and Allow: / means the entire site is accessible. No explicit Disallow means everything is open by default anyway, so this is technically equivalent to an empty robots.txt, but it is explicit and clear about your intent.

You can still exclude specific paths you do not want crawled. Admin areas, user dashboards, and checkout pages are common things to exclude even when you want AI crawlers to access your content:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /dashboard/
Disallow: /checkout/
Disallow: /account/

This gives all crawlers access to your public content while keeping sensitive areas off-limits. For most content-focused sites, this is the right balance.

The blocking debate: reasons to allow vs. block

Plenty of site owners have real reasons to think carefully about whether to allow AI crawlers. This is not a simple "allow everything" situation for everyone. Here are the genuine arguments on both sides.

Reasons to allow AI crawlers

• AI citation traffic is growing. When ChatGPT or Perplexity cites your site in an answer, users click through. This is a real and growing source of referral traffic. Blocking AI bots means opting out of that channel entirely.
• Brand awareness in AI answers. Even when users do not click, seeing your brand name cited in an AI response builds recognition and trust. Publishers who block AI crawlers become invisible in these conversations.
• AI SEO is still early. Sites that establish a strong presence in AI training data and retrieval systems now are likely to benefit as these platforms grow. Early access creates a compounding advantage over time.
• Retrieval-only bots do not use your content for training. If your concern is about training data, you can block training crawlers (like CCBot) while still allowing retrieval-only crawlers (like OAI-SearchBot and PerplexityBot) that fetch your content to cite in live answers.

Reasons to block AI crawlers

• Copyright and content ownership. Many publishers believe that AI companies are commercially benefiting from their content without compensation. Blocking crawlers is currently the primary mechanism available to opt out of this.
• Scraping without attribution. Some AI products summarize content without linking back to the source, eliminating the traffic benefit and leaving only the cost of being crawled.
• Server load. AI training crawlers can be aggressive and consume significant bandwidth. For small sites or those with limited hosting resources, blocking aggressive crawlers is a practical operational decision.
• Legal uncertainty. The legal status of using web content for AI training is still being decided in courts globally. Some publishers prefer to opt out now rather than be included in training datasets while the legal landscape is unclear.

There is no universally correct answer. The right choice depends on your business model, your content type, and your views on how AI platforms relate to your industry. What matters is making an intentional decision rather than an accidental one.

For a broader look at how AI search works and why this all matters for your site, the What is AI SEO article covers the full picture.

Full example robots.txt for maximum AI access

Here is a production-ready robots.txt that gives all major AI crawlers full access to your public content while blocking common sensitive paths. Copy this, adjust the excluded paths to match your site structure, and you are set.

robots.txt

# ============================================================
# robots.txt: Maximum AI Crawler Access Configuration
# Last updated: 2026-06-26
# ============================================================

# Default: allow all crawlers access to public content
User-agent: *
Allow: /

# Block sensitive paths for all crawlers
Disallow: /admin/
Disallow: /dashboard/
Disallow: /account/
Disallow: /checkout/
Disallow: /cart/
Disallow: /login/
Disallow: /register/
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /private/
Disallow: /staging/
Disallow: /api/private/
Disallow: /?s=
Disallow: /search?

# OpenAI crawlers: explicit allow overrides any CDN rules
# GPTBot: training + SearchGPT
# OAI-SearchBot: real-time SearchGPT retrieval
# ChatGPT-User: ChatGPT browsing plugin
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Anthropic crawlers
# ClaudeBot: primary crawler
# anthropic-ai: legacy user-agent, kept for coverage
User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

# Perplexity: real-time retrieval for search answers
User-agent: PerplexityBot
Allow: /

# Google Extended: Gemini AI training only
# (Googlebot is separate and controlled by Google Search Console)
User-agent: Google-Extended
Allow: /

# Apple Intelligence
User-agent: Applebot-Extended
Allow: /

# Meta AI
User-agent: FacebookBot
Allow: /

# Amazon Alexa / Amazon AI
User-agent: Amazonbot
Allow: /

# ByteDance / TikTok AI
User-agent: Bytespider
Allow: /

# Common Crawl: open dataset used by many academic and
# commercial LLMs. Allow if you want broad LLM coverage.
# Block if you want to limit training data usage.
User-agent: CCBot
Allow: /

# Cohere AI
User-agent: cohere-ai
Allow: /

# Diffbot: knowledge graph construction
User-agent: Diffbot
Allow: /

# Sitemap location: helps all crawlers find your content
Sitemap: https://yoursite.com/sitemap.xml

A few things to note about this configuration. First, even though User-agent: * with Allow: / should cover everything, listing AI bots explicitly is a useful safeguard. Some Cloudflare or CDN-level rules may override the wildcard but not explicit rules. Second, keep the Sitemap line at the bottom pointing to your actual sitemap URL. It helps AI retrieval crawlers discover all your pages efficiently.

You can also generate this file automatically with our Robots.txt Generator, which handles all the user-agent strings for you.

How to verify your settings actually work

Writing the robots.txt is step one. Confirming it is actually being served correctly is step two. There are three ways to do this.

1. Check the live URL directly

Open a browser and navigate to https://yoursite.com/robots.txt. What you see there is what crawlers see. It must match what you intend. If the page is blank, returns a 404, or shows something different from what you wrote, there is a serving issue to fix, possibly a CDN cache problem.

2. Google Search Console robots.txt tester

Google Search Console has a built-in robots.txt tester under Settings. You can paste a URL and see whether Googlebot (or any user-agent you type in) would be allowed or blocked. It also highlights syntax errors in your file. This is the most reliable way to validate your rules work as written.

3. AI Crawler Checker

Our free AI Crawler Checker is the fastest way to see your status across all major AI bots at once. Enter your domain and it fetches your live robots.txt, tests it against every user-agent in the table above, and gives you a clear pass or block status for each one. You can see exactly which AI bots you are currently allowing and which ones are being turned away.

4. Manual crawl simulation

If you want to get technical, you can simulate what a crawler sees by fetching your robots.txt with curl and checking specific paths:

curl -A "GPTBot" https://yoursite.com/robots.txt

This fetches the file with the GPTBot user-agent header. While your server returns the same robots.txt regardless of the requesting user-agent (unless you have user-agent-specific serving set up), it is a useful sanity check to confirm the file is publicly accessible and not returning errors for unknown agents.

Re-crawl timing

After you update your robots.txt, AI crawlers will pick up the changes on their next scheduled crawl. There is no way to force an immediate re-crawl for most AI bots (unlike Google Search Console's request indexing feature). Give it a few days to a couple of weeks for changes to propagate fully. If you want your site audited now for overall AI SEO health beyond just crawler access, the AI SEO Audit covers the full picture.

Frequently Asked Questions

Outline Technologies

We build SEO, GEO, and AI optimization tools and strategies. FreeGPTSEO is our free toolkit for checking and improving AI search visibility.