In the recent Big Technology Podcast, Cloudflare CEO Matthew Prince discussed the dramatic rise of AI bot traffic on the internet, which has now surpassed human web traffic. This surge is driven by AI agents crawling websites to gather information, train models, and perform tasks on behalf of users. Prince highlighted that this trend is accelerating faster than anticipated, projecting that in five years, automated traffic could be a thousand times greater than human traffic. This exponential growth poses significant challenges for internet infrastructure, content hosting costs, and the overall economics of the web.
Contrary to the “dead internet theory,” Prince argued that the web is actually becoming healthier and more vibrant due to AI tools empowering non-technical individuals to create content and applications. However, he acknowledged concerns about the quality and value of some AI-generated content but emphasized that the net effect is an explosion of creativity and accessibility. The shift from traditional app stores back to the web, facilitated by AI, could revitalize the internet by enabling more diverse content creation and distribution.
A critical issue discussed was the impact of AI crawling on publishers, especially news organizations. Prince revealed that AI companies like Google, OpenAI, and Anthropic crawl websites at rates vastly exceeding the number of clicks they send back, undermining traditional revenue models based on advertising and subscriptions. To address this, Cloudflare is implementing tools that allow publishers to control AI access to their content, including blocking or requiring compensation from AI crawlers. Starting mid-September, Cloudflare plans to block Google’s AI crawler by default for ad-supported sites unless publishers opt out, aiming to level the playing field among AI companies and protect content creators’ interests.
The conversation also explored the future of small businesses in an increasingly agent-driven web. Prince expressed concern that AI agents, with their infinite patience and data-processing capabilities, might favor established brands with rich online information, making it difficult for new entrants to gain visibility and customers. This could lead to market consolidation favoring large corporations unless deliberate efforts are made to provide small businesses with tools and infrastructure to compete effectively in an AI-mediated commerce environment. He stressed the importance of collaboration among technology providers, payment companies, and platforms to develop solutions that support small business growth.
Finally, Prince acknowledged that while the challenges are significant, there is optimism about creating a new, sustainable business model for the internet that fairly compensates content creators and supports diverse participation. He emphasized the need for innovation in payment systems, content delivery efficiency, and AI interaction models to handle the massive scale of AI traffic. The discussion concluded with a call for ongoing dialogue and collaboration to ensure the future internet remains open, equitable, and vibrant for all stakeholders, including media, small businesses, and end users.
