GPTBot: what it is, and whether you should block it

Operator

OpenAI

Type

Training crawler

robots.txt token

GPTBot

Follows robots.txt

Yes

Used for

Collecting public web content that may be used to train OpenAI models

User agent

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.x; +https://openai.com/gptbot

How to verify

Compare the source IP with the ranges OpenAI publishes at openai.com/gptbot.json

What is GPTBot?

GPTBot is one of several agents OpenAI runs, and it has the narrowest job: gathering publicly available pages as potential training material. OpenAI says it filters out sources that require paywall access, that are known to gather personally identifiable information, or that violate its policies.

The most common mistake site owners make is treating GPTBot as "the ChatGPT crawler". It is not. When ChatGPT answers a question with live links, those results come from the OAI-SearchBot index and from pages fetched on request by ChatGPT-User. GPTBot only influences what future models learned in training.

What happens if you block GPTBot?

Blocking GPTBot stops new crawls of your site for training purposes from the point OpenAI picks up the change. It does not delete anything a model has already learned, and it does not affect whether ChatGPT search can find, show, or link to your pages, because OpenAI treats the GPTBot and OAI-SearchBot settings independently.

Should you allow GPTBot?

If your goal is visibility, the practical question is whether you want your brand, product names, and explanations to be part of what models know by default. Most businesses that want to be recommended by ChatGPT allow GPTBot. Publishers who license content, or who sell the very information a model could reproduce, often block it while still allowing OAI-SearchBot so they keep search traffic.

Check whether GPTBot can reach your site

Enter your domain or a specific page. The checker reads your live robots.txt, picks the group GPTBot would use, and applies the longest matching rule, the same way major crawlers resolve conflicts.

robots.txt rules for GPTBot

Put each group in the robots.txt file at the root of your domain. User agent names are matched without regard to case, and the most specific matching group wins.

User-agent: GPTBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: GPTBot
Disallow: /account/
Disallow: /checkout/
Allow: /

The rule most people get wrong: once GPTBot has its own group, it ignores everything under User-agent: *. If your wildcard group disallows paths such as /admin/, repeat those lines inside the GPTBot group too.

Common OpenAI setup: appear in ChatGPT search, opt out of training

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Disallow: /

How to verify real GPTBot traffic

User agent strings are trivial to fake, so never trust the name alone. OpenAI publishes the IP ranges GPTBot crawls from in a JSON file at openai.com/gptbot.json. A request that says GPTBot but comes from an address outside those ranges is not OpenAI.

To see how often it visits, count requests by user agent in your access log. On a typical Nginx server:

grep -i "GPTBot" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

GPTBot compared with other OpenAI agents

See all 20 agents side by side in the AI crawler directory.

GPTBot FAQ

Abd Shanti

Co-founder and strategy lead at Outline Technologies, the team behind FreeGPTSEO and AI Citation Monitor. Abd works on how brands get found and cited by search engines and AI assistants.