Cloudflare Unveils Granular AI Bot Controls for Site Owners

Cloudflare has introduced new AI bot controls to help website owners easily separate search indexing bots from those used for AI model training. Starting 15 September 2026, new sites that show ads will allow only Search bots by default, while blocking Agent and Training bots. Experts warn that search traffic may be affected if search bots are not clearly identified, so site owners may need to review their settings. The update also gives more control over which bots can access a site and may lead to AI companies needing direct licenses for content. Reports suggest that up to one-fifth of Cloudflare sites might be affected by these changes.

Cloudflare has rolled out granular AI bot controls that categorize automated traffic into three types: Search (for search engine indexing), Agent (for bots acting on behalf of users), and Training (for AI model training data collection). This update, available on all account tiers, introduces a significant change for ad-supported sites that will block AI training bots by default. While this move empowers publishers, it also introduces a critical risk, as misconfigured settings could inadvertently block traffic from major search engines.

Understanding Cloudflare's New Bot Categories

Cloudflare's new AI bot controls categorize automated traffic into three types: Search, Agent, and Training. This allows site owners to selectively permit search engine indexing while blocking bots that scrape content for AI model training, offering more precise control over website access and data usage.

The new policy groups crawlers by their intended behavior:

Search: Bots that index content for traditional search engines.
Agent: AI assistants that access content in real-time on a user's behalf.
Training: Bots that scrape content to train AI models.

New Cloudflare domains that display ads will automatically allow Search bots while blocking Agent and Training bots.

The SEO Risk: Multi-Purpose Crawlers and Misconfiguration

The primary concern for site owners is how these settings will affect multi-purpose crawlers. If a site owner blocks Training bots, any crawler that combines search indexing with model training functions will be blocked entirely. SEO expert Pedro Dias warned on X that this could cause bots like Googlebot, Applebot, or Bingbot to lose access if they perform both functions (X post). This makes it crucial for publishers to audit their bot settings to avoid a sudden drop in organic visibility.

New Dashboard Tools for Publisher Control

To manage this new landscape, Cloudflare's dashboard provides tools including 'managed robots.txt' (to create and manage robots.txt entries) and 'blocking AI bots on pages with ads' (to block AI bots only on monetized portions).

It is important to note that robots.txt directives are considered advisory. A crawler that ignores these signals may lose verification status and can be blocked automatically.

Industry Impact and the Push for Content Licensing

Cloudflare's policy shift puts significant pressure on AI providers, particularly those using merged user-agents. In a filing, Cloudflare criticized Google's approach, stating it gives the company "roughly twice the web content access" compared to its competitors. By forcing a distinction between crawler types, the industry may move toward separate, single-purpose bots. This could leave AI companies with incomplete training datasets, potentially forcing them to negotiate direct content licenses with publishers to maintain access.

Action Plan for Site Owners

With automated requests now accounting for a significant portion of all HTML traffic, site owners must take proactive steps. Here is a recommended action plan:

Review Your Settings: Navigate to the Security > Bots panel in your Cloudflare dashboard to understand your current configuration.
Define Your Strategy: Decide if allowing Agent or Training bot access aligns with your site's content and monetization goals. Blocking them can lead to cleaner analytics by restoring the accuracy of metrics like bounce rate.
Test Carefully: Before applying changes site-wide, test new bot rules on non-critical pages to ensure your search rankings remain stable.
Final Audit: Revisit your settings regularly to ensure you are not unintentionally blocking valuable, mixed-use crawlers.