Cloudflare Unveils Granular AI Bot Controls, Sets New Defaults September 15

Cloudflare's new granular AI bot controls give site owners the power to manage Search, Agent, and Training bots separately - a significant evolution from simple blocking. The new system, available to all customers including Free tier users, takes effect with changed defaults starting September 15, 2026.

What the Three Bot Categories Mean for Your Site

Cloudflare's controls classify AI bots by their purpose. Search bots build indexes for referral traffic and are allowed. Agent bots perform real-time user tasks and are blocked on ad pages. Training bots harvest data for models, provide no return traffic, and are also blocked by default.

Cloudflare now classifies AI traffic by behavior rather than identity. Search bots index content to help it appear in search results - these remain allowed by default because they drive referral traffic. Agent bots perform real-time tasks on behalf of users, like summarizing pages in chatbots. Training bots collect data to build or fine-tune AI models, an activity that typically returns zero traffic to publishers.

The critical distinction: starting September 15, 2026, new domains will automatically block Agent and Training bots on ad-supported pages while preserving Search access. This creates what Cloudflare calls an "ad-signal" framework - treating advertisements as indicators that content was intended for human consumption, not automated extraction.

How Multi-Purpose Crawlers Get Handled

A key technical detail affects major tech companies running mixed-use crawlers that combine Search and Training functions. Cloudflare applies the most restrictive applicable rule - so if a bot does both and you've blocked Training, the entire crawler gets blocked even for its Search component. This pushes Google, Apple, and Microsoft to separate their functions or risk losing visibility entirely.

Crawlers can declare intent using the Forwarded header with use=reference (Search) or use=immediate (Agent), though adoption remains uneven across the industry.

Why This Matters for Publisher Revenue

The policy addresses what Cloudflare terms "AI vampirism" - bots that extract content value without generating pageviews, ad impressions, or compensation. With automated bot traffic representing a significant portion of global web activity according to industry reports, traditional ad-based monetization faces structural pressure. As noted by TechCrunch, these controls also create a negotiating wedge, encouraging AI firms to pursue paid licensing deals to offset publisher pageview losses.

Publishers implementing these controls report measurable improvements in unauthorized bot traffic reduction and increased data licensing revenue when combining blocking with monetized access alternatives. Cloudflare announced new AI traffic controls and content monetization initiatives in July 2026 that enable per-access charging across web pages, datasets, and APIs.

What Site Owners Should Do Before September 2026

Site owners have options to customize these settings according to their business needs. Site owners should:

• Review current Security > Bots settings in their Cloudflare dashboard
• Assess which bot behaviors align with their business model
• Consider whether ad-supported pages should maintain different rules than other content
• Evaluate the Pay Per Use model as an alternative to blocking

The controls represent a shift from voluntary standards like robots.txt to enforceable, infrastructure-level decisions about content value exchange.

---

Frequently Asked Questions

What are the three AI bot categories Cloudflare introduced?

Cloudflare now groups AI traffic into Search, Agent, and Training based on what bots actually do on your site. Search crawlers index content for future discovery and expected referral traffic. Agent bots perform real-time tasks delegated by human users, like summarizing or extracting specific information. Training crawlers collect content to build or update AI models, an activity that returns almost no traffic to publishers. You can manage these categories independently rather than blocking all AI traffic uniformly.

When do the new default blocking rules take effect?

The new defaults apply to new domains onboarding to Cloudflare beginning September 15, 2026. At that point, Agent and Training bots will be automatically blocked on pages displaying ads, while Search remains allowed. Existing customers can customize these settings according to their preferences - the change is not retroactive for domains already on Cloudflare. Free tier customers are fully included in this policy, not just paid plans.

How does Cloudflare handle crawlers that do multiple things?

Cloudflare applies the most restrictive rule when a bot combines purposes. If a crawler performs both Search and Training, and you've blocked Training, the entire bot gets blocked - including its Search function. This creates pressure on major tech companies like Google, Microsoft, and Apple to separate their crawling activities rather than bundle them, since publishers blocking Training would otherwise lose search visibility too.

Can I monetize AI bot access instead of blocking it?

Yes. Cloudflare has developed charging models where publishers receive compensation when their content contributes to AI-generated responses regardless of how it was fetched. Cloudflare's new AI traffic controls and content monetization initiatives enable per-access charging on web pages, datasets, APIs, and other content types. Publishers report significant revenue increases from data licensing when combining blocking unauthorized access with monetized alternatives.

How do Cloudflare's controls compare to other CDN providers?

Cloudflare leads in granularity and default-blocking posture. Fastly offers AI Bot Management with edge detection but doesn't block by default without custom rules. AWS WAF includes a CategoryAI rule that blocks AI bots, but requires explicit WAF configuration. Netlify currently lacks dedicated AI features, requiring manual edge functions. Squarespace offers an opt-in toggle that is off by default. Cloudflare's approach represents a notable industry position on AI traffic governance.