Why News Publishers Are Split Between Optimizing for AI Scrapers and Fighting Them

More than three years into the era of generative AI, news publishers have fractured into two camps when it comes to dealing with AI scrapers. One group treats AI crawlers like search engines and optimizes for visibility. The other builds collective licensing infrastructure to bargain for payment. Neither approach has proven yet that it works but previous strategies, such as blocking the robot scrapers, doing nothing or suing the AI companies, have either failed or will take many more years to - maybe - be successful. 

The numbers have been brutal. Reuters Institute data tracking 2,500 sites shows Google search traffic to publishers collapsed 33 percent globally year-over-year from November 2024 to November 2025, with US traffic down 38 percent. Publishers expect a 43% decline within three years. Zero-click searches now account for 58 to 60 percent of all queries, and AI Overviews appear in 13 to 16 percent of searches. Individual defensive tactics - litigation, blocking, licensing - either cost traffic or failed to generate revenue. Publishers now choose between doubling down on individual optimization or pooling resources for collective leverage.

Future plc, the British publisher behind more than 260 sites including TechRadar and Tom's Guide, has become the global leader in AI platform visibility. Research by SEO software company Ahrefs found that TechRadar is the most cited publisher website domain globally on ChatGPT, closely followed by The Sun. Future's response to this visibility wasn't defensive litigation or blocking - the company launched Future Optic in February 2026 to sell Generative Engine Optimization (GEO) as a commercial service to brands.

The strategy treats AI platforms as advertising inventory. Future recently worked with Samsung to promote two new phone models through GEO-optimized sponsored content. According to Future SEO director Simon Glanville, the campaign secured up to a 33 percent uplift in visibility on ChatGPT for the Samsung products. "We see this as a big potential revenue driver," Glanville told Press Gazette. "Everyone we are speaking to wants more visibility on LLMs."

Future's approach contradicts publishers who view AI as an existential threat. The company optimizes content to appear in AI responses while maintaining brand attribution and citation potential. Glanville described ChatGPT as "a visibility channel rather than a traffic source" - monetizing through brand mentions and citations rather than click-through advertising.

Wire services are making similar bets but with a commodity-scale model. Associated Press and Reuters each signed real-time news feed deals with Google's Gemini in January 2026, with AP separately feeding Microsoft's Copilot through the company's new publisher marketplace. AFP provides Mistral's Le Chat chatbot with 2,300 stories per day across six languages. Combined, these three wire services generate an estimated $65 million annually from structured feeds.

The model works because wire services produce commodity news at scale. Future's version optimizes branded content for AI visibility and commercializes the expertise. Both approaches assume AI platforms create monetizable value - whether through direct feeds, advertising revenue, or visibility-as-a-service offerings to brands.

The alternative strategy rejects individual optimization in favor of pooled bargaining power. Microsoft launched its Publisher Content Marketplace on February 3, 2026 with seven pilot partners including AP, Business Insider, Condé Nast, Hearst, People, USA Today, and Vox Media. The platform promises to create a two-sided marketplace where AI companies license publisher content on a pay-per-use basis, with Microsoft's Copilot as the first buyer.

Two weeks after launch, none of the seven participants has disclosed earnings from the marketplace yet. The lack of transparency mirrors outcomes at other collective platforms. ProRata raised $40 million in September 2025 and signed 700+ publishers to its Gist network. But Digiday reported in December that "money paid out to publishers has been minimal at best." TollBit signed 3,000+ publishers and monitors 1.5 billion quarterly bot scrapes, yet The Media Copilot reported in January that the platform had so far generated zero revenue for participants.

The most ambitious collective effort is Really Simple Licensing (RSL), a machine-readable standard published in December  2025 that extends robots.txt with licensing terms. RSL 1.0 allows publishers to specify payment requirements - subscription fees, per-crawl charges, attribution mandates - directly in their robots.txt files or HTTP headers. Backed by more than 50 partners including Arena Group, BuzzFeed, Reddit, Vox Media, Yahoo, and Medium, RSL has secured technical adoption from Cloudflare, Akamai, and Fastly.

But technical standards require AI companies to honor them voluntarily. No major AI company - OpenAI, Google, Anthropic, or Meta - has yet committed to honoring RSL standards. The specification creates a framework for licensing but cannot enforce compliance without legal or technical barriers.

Madhav Chinnappa, former global director for news and publishers at Google, framed collective licensing as inevitable despite doubts about execution. Speaking at Newsrewired in November 2025, Chinnappa advocated for what he called a "NATO for news" - collective licensing modeled after the U.S performance rights organisation ASCAP for music rights. "This is not a good idea," he said. "But it's the least worst one that I can come up with."

Publishers are trying these new strategies after previous ones have produced either traffic losses or unclear returns:

These failures pushed some publishers toward either the individual optimization model or the collective infrastructure approach.

Future's success undermines collective bargaining. If tech publishers can monetize AI traffic through advertising, why would AI companies pay marketplace fees? Steinberg's GEO strategy only works if AI platforms continue sending traffic while other publishers boycott or demand payment. The approach relies on competitor restraint to create arbitrage opportunity.

Collective infrastructure faces the opposite problem: It launched with no disclosed revenue and no AI company commitments. Microsoft's marketplace, ProRata's network, and RSL's technical standard all depend on AI companies voluntarily participating. So far, no one has demonstrated that participation will occur at scale.

The fundamental conflict is a classic prisoner's dilemma: Individual optimization captures value from competitors' restraint. Collective infrastructure only works if enough publishers refuse to optimize and force AI companies to negotiate. Publishers face a choice between taking available traffic now or holding out for negotiated terms later. Publishers face a choice between taking available traffic now or holding out for negotiated terms later.