Cloudflare’s updated AI crawler controls introduce a three-tier classification system that replaces the previous single “Block AI bots” toggle, with new defaults taking effect on 15/09/2026 that carry a direct risk of blocking Googlebot, Applebot, and Bingbot for site owners who do not review their settings in advance. Because Cloudflare applies its strictest applicable rule to multi-purpose crawlers, any site with Training crawlers blocked will also lose access from major search engines that bundle indexing and AI training into a single bot.
- Cloudflare’s new three-tier system separates crawlers into Search, Agent, and Training categories, replacing the all-or-nothing toggle that previously existed.
- Free-tier users who take no action before 15/09/2026 will be automatically switched to new defaults that block Training and Agent crawlers on ad-supported pages.
- Googlebot, Applebot, and Bingbot are classified as multi-purpose crawlers, so blocking the Training category blocks these search engines entirely under Cloudflare’s strictest-rule enforcement.
- Network-level blocks enforced by Cloudflare cannot be bypassed by robots.txt directives or search engine crawl preferences, making misconfiguration harder to recover from than standard crawl controls.
- The “Verified” bot label no longer guarantees access, as verification now depends on the bot’s assigned category rather than identity alone.
What Changed and Why It Matters
Cloudflare has replaced its single “Block AI bots” toggle with a three-tier crawler classification system. The change takes effect on 15/09/2026 and carries a real risk of blocking Googlebot for site owners who do not review their settings before then.
The Three Crawler Categories
Cloudflare now sorts AI crawlers into three groups: Search (crawlers that index content for AI-powered Q&A results), Agent (real-time crawlers triggered by individual user requests), and Training (crawlers that harvest data for model training without sending referral traffic). This replaces the previous all-or-nothing control, giving site owners more granular options.
The Default Settings Risk
From 15/09/2026, Cloudflare’s new defaults will block Training and Agent crawlers on ad-supported pages while permitting Search crawlers. Free-tier users who have not manually adjusted their settings will be switched to these defaults automatically. The critical problem is how Cloudflare handles multi-purpose crawlers. Googlebot, Applebot, and Bingbot all perform both search indexing and AI training functions, so if Training is blocked, these bots are blocked entirely. Cloudflare applies the strictest applicable rule when a crawler serves combined functions.
This enforcement also operates at the network level, which makes it more restrictive than robots.txt crawler directives. Unlike robots.txt, which some crawlers can technically bypass, a Cloudflare network block cannot be circumvented. Site owners relying on organic search traffic should audit their Cloudflare crawler settings well before the September deadline.
Key Confirmed Details of Cloudflare’s AI Bot Policy Changes
The updated defaults apply to all Cloudflare customers, including those on the free tier. New customers and new sites added by existing customers receive the new blocking behavior automatically, without any manual configuration required.
A few specific rollout details are worth keeping in mind:
- Sites that previously had the legacy “Block AI bots” setting enabled will have that rule automatically converted to the new multi-purpose blocking rule on 15/09/2025.
- Cloudflare is testing a new “use” signal in robots.txt with three preference levels: immediate (no storage permitted), reference (indexing allowed, now the default), and full (summarization and reproduction permitted).
- The “Verified” bot label no longer guarantees access. Verification now depends on the bot’s category, and bots that reproduce entire pages cannot receive verified status.
- Cloudflare introduced BotBase, a searchable directory available to Enterprise Bot Management customers that shows bot classifications and detection IDs, which can be used to build more precise security rules.
The shift in how “Verified” status works is particularly relevant for publishers and site owners who assumed verified bots were always safe to allow. Understanding how crawling and indexing signals interact becomes more important as these category-level distinctions start influencing bot access at the infrastructure layer.
Who Is Affected and What the Main Implications Are
Three groups face the highest exposure from Cloudflare’s updated AI crawler controls: ad-supported publishers, free-tier users who have not adjusted settings, and SEO professionals managing client sites on the platform.
For ad-supported publishers, the concern is particularly layered. Training and Agent crawlers will be blocked by default on pages that carry ads. This means a publisher trying to protect content from AI scraping could simultaneously reduce visibility to crawlers that also serve search indexing functions, creating an unintended trade-off between protection and discoverability.
Free-tier users who have not modified their settings before 15/09/2025 will be automatically moved to the new defaults. No manual action is required for the switch to happen, which means passive users are the most likely to be caught off guard.
SEO professionals should treat this as an urgent audit task. Sites currently using the “Block AI bots” setting face immediate risk because that setting will now apply to multi-purpose crawlers, including major search engines that also train AI models. Unlike meta robots tags and crawl directives, a network-level block enforced by Cloudflare cannot be overridden by search engines even if they would normally respect or ignore robots.txt instructions. The result is a harder barrier that could reduce Googlebot’s crawl frequency and, over time, harm search rankings.
- Ad-supported pages: Training and Agent crawlers blocked by default
- Free-tier accounts: Auto-switched to new defaults without user action
- Sites using “Block AI bots”: Now affects search engine crawlers too
- Network-level blocks: Cannot be bypassed via robots.txt
The real danger here is not the policy itself but the assumption that existing settings are still doing what site owners think they are. A toggle that once blocked AI scrapers now potentially blocks Googlebot, and that distinction matters enormously for any site that depends on organic search. Auditing before the deadline is not optional for SEO-conscious operators. (Hyogi Park, MOCOBIN)
Practical Response and Next Steps
With the September 15, 2026 deadline approaching, site owners using Cloudflare need to act now rather than wait for automatic changes to take effect. The default shift toward blocking AI crawlers will apply to accounts that do not configure explicit rules beforehand, which means passive inaction carries real consequences for search engine access.
The first priority is reviewing your current setup. Navigate to Security Settings in the Cloudflare dashboard and locate the AI bot policy controls. If the legacy “Block AI bots” toggle is still enabled, disable it. That toggle does not distinguish between AI training crawlers and multi-purpose bots like Googlebot, so leaving it active risks blocking legitimate search engine crawling.
From there, the recommended approach is to build granular crawler rules rather than relying on broad toggles. For most publishers, the practical goal is allowing crawlers that serve a Search function while blocking those used exclusively for Training. Cloudflare’s dashboard now supports this distinction at the individual bot level. You can review how this fits into the broader AI crawler blocking and allowlist strategy for a fuller picture of the policy landscape.
One additional check worth making is Bot Fight Mode. If enabled, confirm it is not set to challenge verified AI search bots, as that can interfere with crawling even when your other rules appear correct. Completing these steps before September 15 keeps you in control of the configuration rather than inheriting Cloudflare’s new defaults.
Signals To Watch
Several developments over the coming months will clarify how much practical impact Cloudflare’s new default settings actually carry. The most consequential question is whether Google, Apple, and Microsoft will separate their crawlers by function, creating distinct bots for search indexing versus AI training. Cloudflare expects bot operators to make this separation within roughly a year, and compliance from major players would resolve the forced trade-off that currently puts site owners in a difficult position.
Right now, multi-purpose crawlers bundle search and training into a single agent, meaning a site that blocks AI training may inadvertently suppress its own search visibility. Watching how quickly that architecture changes will signal whether the broader ecosystem is moving toward cleaner consent frameworks.
The September 15 deadline for free-tier and ad-supported sites to review their settings is another practical benchmark. The rate at which those sites adjust their configurations before that date will indicate how many may accidentally block legitimate search crawlers through inaction rather than intent. Monitoring Google Search Console crawl reports and coverage data after that date will help site owners detect early drops in crawl frequency or ranking positions tied to misconfigured settings.
Adoption of Cloudflare’s new “use” signal in robots.txt by major bots is also worth tracking, though it remains a preference rather than an enforced restriction. Voluntary uptake by well-known crawlers would strengthen the signal’s practical value considerably.
