Cloudflare has announced a new feature that allows its web-hosting customers to block AI bots scraping their websites to train AI models.
“Customers don’t want AI bots visiting their websites, and especially those that do so dishonestly,” the company said in a blog post.
Go deeper with GlobalData
“We fear that some AI companies intent on circumventing rules to access content will persistently adapt to evade bot detection,” it added.
The company has added a new one-click tool for website hosts to block all AI bots and announced it is free for all customers.
AI vendors such as Google and OpenAI allow website owners to block the bots they use for data scraping by editing their website’s robots.txt, the text file that tells bots which pages they can access on a website.
However, Cloudflare said these blocks rely on the AI bot operator respecting robots.txt and honestly identifying who they are when they visit an Internet property.
How well do you really know your competitors?
Access the most comprehensive Company Profiles on the market, powered by GlobalData. Save hours of research. Gain competitive edge.
Thank you!
Your download email will arrive shortly
Not ready to buy yet? Download a free sample
We are confident about the unique quality of our Company Profiles. However, we want you to make the most beneficial decision for your business, so we offer a free sample that you can download by submitting the below form
By GlobalData“Sadly, we’ve observed bot operators attempt to appear as though they are a real browser by using a spoofed user agent,” the company wrote.
Cloudflare said its global machine learning model has consistently recognised this activity as a bot, even when operators lie about their agents.
