Cloudflare, a leading internet infrastructure provider, is taking a bold step to protect online content by blocking known AI web crawlers by default. Announced recently, this move aims to prevent AI bots from accessing website content without permission or compensation, signaling a shift in how the web community manages AI-driven scraping.
Starting now, new domain owners using Cloudflare will be prompted to decide if they want to allow AI scrapers on their sites. In an innovative move, Cloudflare is also introducing a “Pay Per Crawl” program, enabling publishers to set a fee for AI companies seeking to access their content. AI firms can then review these prices and choose whether to register for access or decline. Currently limited to some major publishers and content creators, this initiative ensures that content creators are compensated and their work is used responsibly.
Cloudflare has been a pioneer in helping website owners control AI crawling since 2023. Initially, it allowed sites to block AI bots respecting robots.txt files, but now, it offers the ability to block all AI crawlers—regardless of whether they follow those guidelines. This setting is now enabled by default for new clients, with Cloudflare identifying scrapers through its comprehensive list of known AI bots. Additionally, a new feature launched in March sends bots into a simulated “AI Labyrinth,” making unauthorized scraping more difficult.
Major media outlets and platforms like The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora are already on board with Cloudflare’s stricter AI crawler policies. As AI chatbots increasingly become a primary way people access information—sometimes replacing traditional search engines—content creators are concerned about the future of original content. Cloudflare’s CEO, Matthew Prince, highlighted this trend, noting that public trust in AI is growing, but it’s impacting the consumption of original material.
To foster responsible AI use, Cloudflare is collaborating with AI companies to verify their crawlers and clarify their intentions—whether for training, inference, or search purposes. Website owners can review this information and decide which AI bots to allow, giving them greater control over their content.
Prince emphasizes the importance of protecting original content, calling it “one of the greatest inventions of the last century.” His message is clear: “AI crawlers have been scraping content without limits, and our goal is to empower creators while still supporting AI innovation.” With these new measures, Cloudflare aims to balance the growth of AI technology with the rights of content creators, shaping a fairer future for the web.
