OAI-SearchBot: the crawler behind ChatGPT search

Operator

OpenAI

Type

Search crawler

robots.txt token

OAI-SearchBot

Follows robots.txt

Yes

Used for

Building the index that ChatGPT search uses to show and link websites

User agent

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.x; +https://openai.com/searchbot

How to verify

Compare the source IP with the ranges OpenAI publishes at openai.com/searchbot.json

What is OAI-SearchBot?

OAI-SearchBot exists so ChatGPT can answer with current, linked sources. When a question needs fresh information, ChatGPT searches, reads candidate pages, and cites them. The pages it can draw from are the ones its search systems were allowed to crawl, and OAI-SearchBot is the agent doing that crawling for OpenAI.

It is separate from GPTBot on purpose. OpenAI lets you opt out of training while staying in search, or the reverse. That split is the most important thing to understand about OpenAI's crawlers, because blocking the wrong one is the most common reason a site that wants ChatGPT traffic gets none.

What happens if you block OAI-SearchBot?

A Disallow for OAI-SearchBot removes your pages from ChatGPT search answers once OpenAI processes the change, which it says can take about a day. ChatGPT may still surface a bare link to your site for navigational queries, but it will not summarise or cite your content. Your standing in Google and Bing is not affected.

Should you allow OAI-SearchBot?

Almost every site that wants traffic from ChatGPT should allow OAI-SearchBot. It is not a training crawler, so allowing it does not hand your content to model training. The rare reason to block it is a page set you do not want summarised at all, such as gated research or member-only material, and for those a narrower Disallow on the specific folder is better than blocking the whole site.

Check whether OAI-SearchBot can reach your site

Enter your domain or a specific page. The checker reads your live robots.txt, picks the group OAI-SearchBot would use, and applies the longest matching rule, the same way major crawlers resolve conflicts.

robots.txt rules for OAI-SearchBot

Put each group in the robots.txt file at the root of your domain. User agent names are matched without regard to case, and the most specific matching group wins.

User-agent: OAI-SearchBot
Allow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /account/
Disallow: /checkout/
Allow: /

The rule most people get wrong: once OAI-SearchBot has its own group, it ignores everything under User-agent: *. If your wildcard group disallows paths such as /admin/, repeat those lines inside the OAI-SearchBot group too.

Common OpenAI setup: appear in ChatGPT search, opt out of training

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

How to verify real OAI-SearchBot traffic

Check that the requesting IP falls inside the ranges OpenAI lists at openai.com/searchbot.json. Also look at your firewall or CDN: bot protection that challenges unknown automated traffic will quietly block OAI-SearchBot even when robots.txt allows it, and nothing in robots.txt will show that problem.

To see how often it visits, count requests by user agent in your access log. On a typical Nginx server:

grep -i "OAI-SearchBot" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

OAI-SearchBot compared with other OpenAI agents

See all 20 agents side by side in the AI crawler directory.

OAI-SearchBot FAQ

Abd Shanti

Co-founder and strategy lead at Outline Technologies, the team behind FreeGPTSEO and AI Citation Monitor. Abd works on how brands get found and cited by search engines and AI assistants.