AI crawler directory

AI crawler directory: 20 bots and how to handle each one

GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and the rest, sorted by what they actually do, with the robots.txt rules for each.

By Abd Shanti Updated September 16, 2026 4 min read

Quick answer: AI companies run different agents for different jobs. Search crawlers and user-triggered fetchers decide whether assistants like ChatGPT, Claude and Perplexity can show and cite your pages. Training crawlers decide whether future models learn from your content. Control tokens like Google-Extended are not crawlers at all. Treat each group as a separate decision.

This directory lists the 20 agents that matter most for AI visibility in 2026, with a page for each covering what it does, what blocking it changes, exact robots.txt rules, how to verify genuine traffic, and a live check against your own site.

Search crawlers

Build the indexes AI assistants search when they answer. Blocking these costs visibility.

OAI-SearchBotYesOpenAIBuilding the index that ChatGPT search uses to show and link websites Claude-SearchBotYes, per Anthropic's documentationAnthropicCrawling to improve the quality of search results shown to Claude users PerplexityBotYesPerplexityFinding and linking websites in Perplexity search results GooglebotYesGoogleCrawling for Google Search, including AI Overviews and AI Mode BingbotYesMicrosoftCrawling for Bing, which grounds Microsoft Copilot and is a search source for other AI products ApplebotYesApplePowering search features in Siri, Spotlight and Safari AmazonbotYesAmazonImproving Amazon products and services, including answers from Alexa DuckAssistBotYesDuckDuckGoFetching pages in real time for DuckDuckGo's AI-assisted answers, which cite sources

User-triggered fetchers

Open a specific page because a person asked. Robots.txt is often not a guaranteed block.

ChatGPT-UserNot guaranteed, fetches are user-initiatedOpenAIOpening a page when a ChatGPT user, a custom GPT, or an action asks for it OAI-AdsBotNo, per OpenAI's documentationOpenAIChecking the landing pages of ads submitted to run in ChatGPT Perplexity-UserGenerally no, fetches are user-initiatedPerplexityVisiting a page live when a Perplexity user's question needs it MistralAI-UserUser-initiated, treat robots.txt as a requestMistral AIFetching pages when Le Chat users ask for information from the web

Training crawlers

Collect content for model training. Blocking them does not remove you from AI search answers.

GPTBotYesOpenAICollecting public web content that may be used to train OpenAI models ClaudeBotYesAnthropicCollecting public web content that may be used to train Anthropic's Claude models Meta-ExternalAgentYesMetaCrawling for uses such as training AI models or improving products by indexing content BytespiderWidely reported as inconsistentByteDanceCollecting web content for ByteDance, including training its AI models CCBotYesCommon CrawlBuilding Common Crawl's free public web archive, widely used to train language models

robots.txt control tokens

Not crawlers at all. Switches that tell a search crawler's operator how the content may be used.

Google-ExtendedYes, it only exists in robots.txtGoogleControlling whether Google may use your content to train Gemini models and to ground Gemini answers Applebot-ExtendedYes, it only exists in robots.txtAppleControlling whether Applebot data may be used to train Apple's foundation models

All 20 agents at a glance

AgentOperatorTypeFollows robots.txt
GPTBotOpenAITraining crawlerYes
OAI-SearchBotOpenAISearch crawlerYes
ChatGPT-UserOpenAIUser-triggered fetcherNot guaranteed, fetches are user-initiated
OAI-AdsBotOpenAIAd landing page checkerNo, per OpenAI's documentation
ClaudeBotAnthropicTraining crawlerYes
Claude-UserAnthropicUser-triggered fetcherYes, per Anthropic's documentation
Claude-SearchBotAnthropicSearch crawlerYes, per Anthropic's documentation
PerplexityBotPerplexitySearch crawlerYes
Perplexity-UserPerplexityUser-triggered fetcherGenerally no, fetches are user-initiated
Google-ExtendedGooglerobots.txt control tokenYes, it only exists in robots.txt
GooglebotGoogleSearch engine crawlerYes
BingbotMicrosoftSearch engine crawlerYes
ApplebotAppleSearch crawlerYes
Applebot-ExtendedApplerobots.txt control tokenYes, it only exists in robots.txt
Meta-ExternalAgentMetaTraining crawlerYes
BytespiderByteDanceTraining crawlerWidely reported as inconsistent
CCBotCommon CrawlOpen dataset crawlerYes
AmazonbotAmazonSearch and AI crawlerYes
DuckAssistBotDuckDuckGoAI answer crawlerYes
MistralAI-UserMistral AIUser-triggered fetcherUser-initiated, treat robots.txt as a request

A visibility-first robots.txt

This setup keeps every search crawler and user fetcher in, which is what most businesses that want AI citations need. Add a Disallow: / group for any training crawler you want to exclude, and remember that a bot with its own group ignores your User-agent: * rules.

robots.txt for AI visibility
User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: Bytespider
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Robots.txt is only one layer. Firewalls and CDN bot protection can block crawlers that robots.txt allows, and no robots.txt line will reveal that. The AI crawler checker reads your file, and the ChatGPT visibility checker also tests whether your page can actually be fetched.

Frequently asked questions

If you want visibility in AI answers, allow the search crawlers and user-triggered fetchers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bingbot, Googlebot and DuckAssistBot. Decide separately on training crawlers such as GPTBot, ClaudeBot and CCBot.
No. GPTBot is for training. ChatGPT search relies on OAI-SearchBot and live fetches by ChatGPT-User, which you control separately.
Because they are robots.txt tokens, not crawlers. Google and Apple crawl with Googlebot and Applebot and use these tokens to decide how the content may be used.
Most documented crawlers do. User-triggered fetchers such as ChatGPT-User and Perplexity-User act for a person and may not, and Bytespider is widely reported as inconsistent. Use firewall rules where you need a guaranteed block.
Abd Shanti

Abd Shanti

Co-founder and strategy lead at Outline Technologies, the team behind FreeGPTSEO and AI Citation Monitor. Abd works on how brands get found and cited by search engines and AI assistants.

Check all of them on your site

The free AI SEO audit tests 12 of these agents against your robots.txt, plus schema, llms.txt and content, in a few seconds.

Run the free AI SEO audit
Last updated: September 16, 2026. Facts checked against each operator's crawler documentation.