Cloudflare will now block AI bots from crawling its clients’ websites by default Cloudflare will now block AI bots from crawling its clients' websites by default | MIT Technology Review
@artesia summarise
Cloudflare, a prominent internet infrastructure company, has announced that it will start blocking AI bots from crawling its clients’ websites by default. This change aims to give clients more control over how their content is accessed by AI companies. Additionally, Cloudflare is introducing a “pay-per-crawl” service where clients can charge AI bots for accessing their website data. This move is designed to help content creators manage and potentially monetize the way their content is used by AI models, which often pull large amounts of data from the web without offering proper attribution or compensation to original content sources.
The new policy also allows clients to selectively allow or disallow crawling by specific AI bots, offering more granular control over data access for AI training, fine-tuning, and inference processes. This development may foster better relationships between AI companies and website owners by facilitating negotiations and agreements on acceptable usage terms.
Despite these changes, some experts have raised concerns that a broad ban on AI crawlers might hinder non-commercial uses, such as academic research or web archiving. Cloudflare, however, aims to ensure internet openness by enabling more sustainable agreements between AI companies and web content producers.
