More than 30 companies, including Universal Music Group, have supported a new step taken by Cloudflare. This internet company is now blocking AI bots from copying content from websites without permission.
From July 1, sites using Cloudflare will automatically stop AI web crawlers. Since Cloudflare supports about 20% of the internet, this can seriously affect the way AI companies gather content to train their models.
Now, website owners can also choose which bots are allowed to access their content. Even more, they can charge a fee per crawl. So, if an AI wants to use that data, it has to pay or make a deal.
Web crawlers are software tools that scan websites. They were first used by search engines, but now they help AI systems collect text, images, videos — often without permission. This causes less traffic for websites because people get answers directly from AI tools, without even clicking links. It’s called the zero-click effect.
Cloudflare’s co-founder, Matthew Prince, says it clearly: If the internet is going to survive in the age of AI, creators need more control and a new system that works for everyone.
The move is backed by many media groups like Associated Press, Time, Fortune, and more. Even Universal Music Group supports it. Their COO, Boyd Muir, says they welcome this step, as it helps stop the unauthorized use of creative work.
Cloudflare already has strong tools to check and block bots. If an AI bot tries to hide what it’s doing, Cloudflare can treat it like a cyber threat and stop it.
Still, some people believe that not all AI projects are commercial. Some are for personal use or research. So there’s a discussion about how to protect content while still allowing fair and ethical AI development.
In short, this move could change how AI companies access online content — and for the first time, give creators a way to earn from it.