Cloudflare CEO says block threat pushed Google on AI crawling

Cloudflare’s chief executive, Matthew Prince, said his company told Google it would be blocked if it did not change how it crawls the web. “And in September of this year, we said to them, if you don’t change that policy by September 15th, we’re going to block you across the entire Internet,” Prince said.Web Summit Lisbon 2026, Centre Stage livestream (Day One). He said that, together with rulings in the UK and other regulatory pressure, got Google to move. He spoke with Nick Thompson of The Atlantic on Web Summit’s Centre Stage in Lisbon on 10 November.

What changed on 15 September

Cloudflare’s own announcement that day describes the change. It launched a Disallow AI Training setting that lets a site stay in Google, Apple and Microsoft search while refusing to let the same crawlers train AI models on its pages.Cloudflare: Have it both ways: stay discoverable in search while disallowing AI training (15 September 2026). Cloudflare said the three companies honour the setting or “have committed (in a specified time frame) to honor this setting”. It said Google planned more URL-level transparency tools, and that its Block setting now stops Googlebot entirely, search included.

Prince went further on stage. He said Google, the last of the big providers to hold out, now says it will start compensating creators and is willing to pay for news and research it used to get free. Digiday reported on 14 September that Google is testing a Search Console pilot that pays publishers when their content significantly contributes to answers in Gemini, AI Overviews and AI Mode.Digiday: Google rolls out pay-per-value AI licensing program to publishers. Google confirmed it as “an early-stage learning pilot”.

Meta

Prince said Cloudflare also warned Meta over terms that would let it scrape the websites of businesses that use its platforms. He said Cloudflare told Meta it would become the new villain and could be blocked too, and that “I think that we played an influential role in that position”. In October Meta removed the word train from the crawling clause in its new Commercial Terms and limited the uses to advertising and business products, Nieman Lab reported.Nieman Lab: Legal experts warn Meta’s new terms still grant broad scraping permissions, despite language changes. Meta later delayed the rollout and pledged to consult publishers. A Meta spokesperson said the update did not change how Meta uses web crawlers.

A payments problem

Prince’s case is that the web’s business model is breaking. It ran on advertising for 30 years, and “agents don’t click on ads,” he said. Cloudflare wants AI companies to pay per use. Prince said Cloudflare handles about half a billion web requests a second, that 1% to 10% of them could be paid for, and that it would need to support 10 million financial transactions a second from the first day. He compared that with a Visa holiday peak he put at 80,000 a second.

That is the bottleneck, he said, not supply or demand. “And so we are looking both for people” to partner with, he said, or maybe acquire, or Cloudflare will build the system itself.

Thompson asked whether Cloudflare, which carries a large share of web traffic, is a monopolist itself. Prince said it takes five minutes to sign up for Cloudflare and that “It takes 10 seconds to leave us”. He described the company as a bargaining representative for content creators.

His bigger worry is small businesses. Agents prefer whichever institutions have the most information online, he said, so incumbents win and startups lose, which could bring massive consolidation. Cloudflare said in its founders’ letter in September that automated traffic passed human traffic on the Internet in May 2026.Cloudflare: 2026 Annual Founders’ Letter.