The AI search startup Perplexity has landed itself in the middle of controversy, facing allegations of bypassing restrictions designed to prevent its web crawlers from accessing certain websites. According to a report by Cloudflare, the company has been able to skip these protections by concealing its true identity and sneaking past the restrictions placed on the website to protect its content. As per the claim, Perplexity intentionally bypasses safeguards like robots.txt files that tell bots which parts of a website are not available for access, through masked user agents, and uses a different service provider to avoid being caught.
Perplexity is accused of hiding tracks to bypass website restrictions and scrape protected content
Cloudflare's accusation against Perplexity is followed by an in-depth investigation. The company set up a hidden webpage that had crawler restrictions and was not linked or indexed publicly, which was meant to be a honeypot. Despite these protections, Cloudflare maintains that Perplexity's systems were still able to access the page and even included its content in search results, confirming the unauthorized ways the company tends to collect data.
Cloudflare has asserted that Perplexity's actions are unethical and breach its terms of service. The AI startup has now been removed from the list of verified bots, and Cloudflare even plans to go one step further by tightening its restrictions overall. On the other hand, Perplexity has completely denied the claims against it and said that the investigation did not consider openness and transparency, thus lacking solid evidence. It further states that Cloudflare has exaggerated the findings or even misread the situation.
The ongoing situation is rather grave since Cloudflare has maintained an assertive stance by calling out Perplexity for surpassing digital boundaries. For the AI company, this tends to tarnish the brand image it has been trying to push forward, focusing on being transparent compared to traditional search engines. Either way, the incident highlights one huge issue that could become more prevalent in the future: the war for accessing content and its monetization.
The incident also highlights a much broader debate within the AI ecosystem. It concerns how AI models source their data and the gray practices that tend to erupt as the systems become more powerful and commercialized. Cloudflare CEO Matthew Prince has been vocal about the risks these models pose to content creators and publishers. As a result, the company has introduced the option to charge AI companies for accessing their content and has started automatically blocking AI crawlers.
Follow Wccftech on Google to get more of our news coverage in your feeds.
