Cloudflare has provided hard data confirming the “dead internet theory,” revealing that the majority of web traffic is now generated by bots rather than humans. Their measurements show that for every 70,000 pages crawled by Anthropic’s AI crawler, only one visitor is human, compared to Google’s crawler ratio of about five pages per visitor. This shift marks a transition to a bot scraping economy, with bots accounting for over 57% of web page requests as of June, a milestone predicted to occur years later. The traditional model where bots crawl sites to send human readers in return is breaking down as bots increasingly dominate web traffic.
The rise of AI has fragmented web crawling into three distinct roles: training crawlers that copy content to train AI models without compensation, retrieval crawlers that maintain fresh indexes for answer engines, and user fetchers that retrieve pages live when users ask chatbots questions. Companies like OpenAI and Anthropic operate separate bots for each function. Cloudflare’s data shows that over half of AI crawler traffic involves re-downloading unchanged pages, treating websites as free APIs while site owners bear the bandwidth costs. Examples include Read the Docs paying thousands in bandwidth fees due to heavy crawler activity and Wikimedia experiencing significant bandwidth increases from scrapers.
Website operators face escalating costs because crawlers access every link, including those humans never visit, often bypassing caches and generating expensive server loads. Git forges are particularly vulnerable due to on-demand page generation that cannot be cached. Attempts to defend against bots through robots.txt files, user-agent blocking, and IP blocking have largely failed, as scrapers use residential proxies and mimic legitimate browsers. More advanced defenses like proof-of-work challenges and serving fake data have limited success, with bots learning to bypass puzzles and operators resorting to serving endless garbage pages to waste crawler resources.
Legal and commercial landscapes around web scraping remain uncertain. Courts have yet to definitively rule on the legality of scraping public web pages, with some cases dismissing claims against scrapers while others continue. Meanwhile, major websites and AI companies have struck lucrative licensing deals, such as Reddit’s multi-million dollar contracts with Google and OpenAI. Infrastructure providers like Cloudflare and AWS are introducing pay-per-crawl models, requiring bots to pay for access, signaling a shift toward monetizing web crawling. This creates a two-tier web where top-tier sites operate under licensing agreements while smaller sites remain in a costly arms race against bots.
To mitigate the impact of bots, website owners are advised to optimize their URL structures by ending pagination and avoiding infinite calendar states, pre-generating image sizes, and caching CPU-intensive routes. These measures reduce the attack surface for crawlers and help control costs. Additionally, emerging standards like Web Bot Off, which cryptographically signs crawler requests, offer hope for better bot management by allowing sites to verify and selectively allow or block crawlers. Until such solutions become widespread, site owners must monitor their traffic closely to understand how much of it is genuine human engagement versus automated bot activity.
