Cloudflare changed its AI crawler defaults on September 15

What Cloudflare actually changed

Cloudflare announced the change on July 1, 2026 and switched it on September 15. For every domain that joins its network from that date, plus new customers and existing customers on the free plan, two of its three crawler categories are blocked by default on any page that displays ads. Training is blocked. Agent is blocked. Search stays allowed.

Cloudflare says it sits in front of more than 20 percent of web domains, so this is not a niche setting. It is the default behaviour for a large and growing share of the web, and most site owners will never see a notification about it. The block is silent from the inside. Your pages load fine in a browser, your analytics look normal, and the only symptom is that AI assistants slowly stop quoting you.

Two details decide whether it applies to you. The first is whether your domain is new to Cloudflare or on the free plan, because established paid zones keep their current settings until someone changes them. The second is whether the page shows ads, because the default targets ad supported pages rather than the whole site.

Search, Agent and Training explained

Cloudflare splits automated AI traffic into three buckets, and this taxonomy is the part worth learning, because other vendors are starting to use the same language.

The logic is easy to follow. Search crawlers send traffic back, so blocking them costs you visitors. Training crawlers take content and return nothing directly. Agent fetches sit in between, which is exactly why this default is the contested one.

That middle bucket is where the damage happens. When someone asks ChatGPT about a topic your page covers and ChatGPT opens the page to read it, that request is an Agent fetch. Blocking it does not protect you from training. It stops the assistant from reading your page for a person who was already interested in your topic.

Which agents sit in which category

Category names are useful only when you can map them onto the user agents in your logs. Here is that mapping for the agents that matter most.

Read that table twice if you run an ad supported site. The agents that answer a live human question are the ones now switched off by default, while the ones that quietly harvest your pages for training sit in a category that was never the target of this change.

Why robots.txt will not save you

This is the part that catches experienced people. Robots.txt is a file on your server. A crawler asks for it, reads it, and decides how to behave. Every polite AI crawler respects it.

A network block never gets that far. The request arrives at Cloudflare, Cloudflare decides the agent is in a blocked category, and the agent receives a refusal. Your server is never asked. A perfectly written robots.txt that welcomes ChatGPT-User has no effect at all, because nothing ever reads it.

That is why a robots.txt checker alone cannot tell you the truth about your visibility. It reads a file that says yes while the front door says no. The only reliable test is to try to fetch the page the way an outside agent would and see what comes back.

How to check your own site today

Three checks, in the order that finds the problem fastest.

1. Test the page from outsideRun the ChatGPT visibility checker on a page that carries ads. It reads your robots.txt for the OpenAI and Bing agents and then tries to fetch the page itself. If robots.txt says the agent is allowed but the fetch fails or returns a challenge, you are looking at a network level block rather than a robots.txt problem.
2. Read your own logsThe clearest evidence is a change in your access log. Count requests by agent for the two weeks before and after September 15. A category block shows up as one group of agents going to zero while another group keeps crawling normally. On a typical Nginx server, count a single agent like this: grep -c "ChatGPT-User" /var/log/nginx/access.log
3. Open the Cloudflare dashboardLook at the AI crawler controls for the zone and check the state of the Search, Agent and Training categories, and whether the ad page condition applies to you. If your domain joined recently or sits on the free plan, assume the new defaults are active until you confirm otherwise.

The log check is the one people skip and the one that settles the argument. Requests that never arrive leave a hole in the data, and once you know the shape of that hole you stop guessing.

What to allow if you want AI citations

If your goal is to be quoted and linked by AI assistants, the answer is simple: allow Search and allow Agent. Decide on Training separately, because that is a licensing question rather than a visibility one.

Allowing Search alone feels safe and is half a strategy. Search crawlers build the index that lets an assistant know your page exists. Agent fetches are how the assistant reads the page while a specific person is waiting for an answer. Block the second one and you appear in the index as a page nobody can open.

Whatever you choose at the network layer, keep your robots.txt consistent with it. Two layers that disagree produce the confusing situation where one tool reports you as open and another reports you as blocked. Our robots.txt generator writes a file that matches a visibility first policy, and the AI crawler directory explains each agent one by one.

The bigger picture: crawling becomes a paid resource

This default is one step in a longer plan. Cloudflare already offers pay per crawl, which lets a site charge AI companies for access, and it has announced a Monetization Gateway that extends the same idea to APIs, datasets and tool calls, priced through the open x402 payment protocol that uses the HTTP status code 402 Payment Required.

The reasoning behind it is the crawl to click ratio. Crawlers fetch pages far more often than assistants send readers back, and for some operators the gap is enormous. We measured that gap on our own servers, engine by engine, in 14 days of real logs. The result explains why so many publishers are willing to close the door.

For most businesses, though, the calculation is different. If you sell something, an assistant that reads your page and recommends your product is not taking value, it is delivering a customer. The block that protects a publisher can quietly cost a business its best new channel.

The bottom line

Check whether the new defaults apply to your domain, because nobody will tell you. If you want AI visibility, allow the Search and Agent categories and treat Training as a separate decision. Then verify from outside your own network, because the only opinion that matters is what an agent receives when it asks for your page.

Frequently asked questions

Abd Shanti

Co-founder and strategy lead at Outline Technologies, the team behind FreeGPTSEO and AI Citation Monitor. Abd works on how brands get found and cited by search engines and AI assistants.