IAB Tech Lab Unveils CoMP Framework to License AI Content Crawling

The IAB Tech Lab has introduced a new system called CoMP to help publishers control and get paid when AI bots use their content. CoMP lets publishers set rules and prices in a simple, computer-friendly way before AI crawlers can grab their articles. This change comes as news sites are losing lots of visitors and ad money because AI tools answer people's questions directly. While CoMP could help publishers earn more, it faces challenges like getting everyone to use it and stopping bad actors. In the meantime, publishers are also trying new ways to make money, not just depending on search traffic.

The IAB Tech Lab has unveiled the CoMP Framework to license AI content crawling, a new draft standard giving publishers a machine-readable way to set commercial terms before AI systems crawl their content. This protocol aims to replace the current ad-hoc system of robots.txt files and private deals with a single, interoperable standard for content monetization.

Why Publishers Need a New Monetization Standard

The CoMP framework addresses the steep decline in publisher website traffic and revenue caused by AI-powered search summaries. As AI tools answer queries directly, they reduce referral clicks and ad revenue, creating a monetization gap that CoMP aims to bridge by enabling automated, pre-crawl content licensing.

Publishers are experiencing a structural collapse in traffic as AI summaries and chatbots intercept users. Define Media Group reported a significant drop in organic visits for some sites, while major brands like CNN saw substantial year-over-year declines. According to Pew research, users encountering Google AI summary clicked traditional search result links in 8% of visits (vs 15% without AI summary). AI chatbots show traffic growth >80% YoY and higher engagement, but lower overall volume than search. As noted in the report "The Search Traffic Collapse Is Forcing Publishers to Adapt," this trend threatens the viability of many digital publications.

How the CoMP Licensing Protocol Works

CoMP proposes a standardized technical handshake between publishers and AI crawlers. The framework is built on core objects to automate the process:

Declaration: Publishers use a JSON template to declare permissible AI uses, pricing, and other rights for their content.
Negotiation: AI developers or marketplaces can make offers based on this standardized schema.
Access: An AI crawler must present a cryptographically signed "Grant" to prove it has a valid license before it can access and retrieve the content.
Auditing: The system provides receipts to publishers, creating a transparent and auditable trail of content usage and payments.

Key Adoption Hurdles and Industry Challenges

While promising, the wide-scale adoption of CoMP faces several significant challenges:

Timeline Mismatch: Publishers require revenue solutions immediately, but the process of finalizing and implementing a new industry standard will take months.
Enforcement: The framework relies on good-faith participation, and mechanisms for detecting and penalizing bad-actor crawlers that ignore the protocol are still unclear.
Coordination: Success depends on simultaneous adoption by a critical mass of publishers, AI developers, and ad tech platforms to create a functional marketplace.
Competing Solutions: The standard may face fragmentation from proprietary licensing deals (like the OpenAI-AP agreement) and varying national legislation.

What to Expect After the Comment Period

CoMP Specification v1.0 public comment closed on April 9, 2026; no details on next steps like finalization by working group. While no live deployments are documented yet, stakeholders can track progress and find official documentation on the CoMP initiative page.

In the meantime, publishers are not waiting. Many are actively diversifying their revenue streams through subscriptions, live events, and e-commerce to reduce their dependency on search traffic, a strategy that will remain critical even if CoMP achieves widespread adoption.