Cloudflare’s new AI bot defaults land September 15th. Here’s what to check first to protect your store’s organic and AI search visibility.
On July 1st, Cloudflare announced changes to how it classifies and manages AI bot traffic across its network, roughly one in five websites globally.
On September 15th, the new controls take effect. New domains joining Cloudflare will get preset defaults, and existing domains will be migrated automatically onto the new controls. Cloudflare says most sites’ settings will carry over as they are, so for many stores nothing will change on the day.
But “nothing changes” isn’t the same as “nothing to check”. The new controls include an option that blocks Google, and it’s easy to choose it by accident while trying to keep AI training bots out.
You might not configure this yourself. It’s something your dev team, hosting provider, or agency likely needs to check on your behalf.
Here’s what’s changing, and the one action worth asking for directly. Don’t let a setting meant to protect your content from AI training end up blocking Google too.
Search, Agent, and Training will no longer be one thing
Cloudflare used to offer a single “Block AI Bots” switch. Simple, but it lumped together three very different behaviours: a bot indexing your site for search results, a bot fetching a page in real time to answer a shopper’s question, and a bot harvesting your product pages to train a model.
Cloudflare will split these into three categories that every customer can manage separately: Search, Agent, and Training.
Search is the behaviour most likely to send you traffic back. Training takes your content with nothing in return. Agent sits in between, a bot like ChatGPT or Gemini acting on a real shopper’s behalf.
Each category can be set to Allow, Block on pages with ads, or Block, and Training also gets a fourth option, Disallow AI Training. The two Block options are where the risk sits.
The wrong setting can catch Google in the crossfire
From September 15th, new sites onboarding to Cloudflare will be offered one of two presets. Sites that carry ads get Search allowed, Training set to “Disallow AI Training”, and Agent blocked on pages with ads. Sites without ads get all three allowed.
The logic is that an ad page is built for a human to see it, so bots that don’t send traffic back get shut out there.
The bigger issue applies to every Cloudflare customer, new or existing. Googlebot, Bingbot and Applebot are mixed-use crawlers: the same bot crawls for search and for AI training.
Cloudflare will handle these in two very different ways depending on which option you choose:
- “Disallow AI Training” opts your site out of training while letting Googlebot, Bingbot and Applebot keep crawling for search. It also publishes your no-training preference in robots.txt automatically.
- “Block” and “Block on pages with ads” will apply to those mixed-use crawlers too. Choose either and you block Googlebot on the affected pages, search ranking included.
If your site uses the old “Block AI Bots” toggle, Cloudflare will migrate it to “Disallow AI Training”, which keeps search access. The risk comes from someone reviewing the new controls, wanting to keep AI out, and picking “Block” because it sounds like the stronger option.
This is the specific thing to get right. For most retailers, disappearing out of Google search outweighs whatever protection you gain from keeping product content out of a model’s training data. If you want to opt out of training, “Disallow AI Training” does that without the search penalty.
The action item for your dev team or agency isn’t “review the settings”. It’s a direct question with a specific answer. In Cloudflare’s Security settings, confirm that Search is set to Allow, and that Training is set to either Allow or “Disallow AI Training”, never Block or Block on pages with ads.
Why this is a competitive question, not just a technical one
Search and AI-driven discovery are converging into the same channel. The infrastructure decisions made this year will determine whether your products surface when a shopper asks an AI assistant for a recommendation, or whether a competitor’s do instead.
You first need to make sure that no one accidentally trades away your Google visibility while trying to keep AI training bots out.
If you want a clear view of how your site actually performs in organic and AI search against the competition, you can request a free audit or arrange a quick call with our team.
