How Cursor AI chatbot's Big Mess is a lesson for companies automating their customer service

AI-powered customer service is meant to streamline support, but Cursor AI’s chatbot mishap has exposed the dangers of relying too heavily on automation. The coding assistant, developed by Anysphere, recently faced backlash after its AI-powered support bot fabricated a company policy, leading to mass confusion and subscription cancellations.
The issue surfaced when users reported being mysteriously logged out when switching devices. Seeking clarification, they contacted Cursor’s support team, only to receive an email from “Sam” stating that the logouts were part of a new login policy. However, no such policy existed—the response was entirely hallucinated by the AI chatbot.
The incident quickly went viral, prompting concerns about AI’s reliability in customer service. Experts warn that while AI can enhance efficiency, it lacks empathy, nuance, and accountability, making it risky to replace human agents entirely. AI hallucinations—where chatbots generate false information—can damage trust and lead to customer frustration.
Cursor's failure acts as a cautionary lesson for companies incorporating AI into customer support. Businesses must ensure human oversight, transparency, and safeguards to prevent AI errors from escalating into brand-damaging crises.