The most dangerous thing AI will ever say to you
"You're absolutely right."
Not because it's wrong — sometimes you are right. But because AI says it whether you are or you aren't. And you can't tell the difference.
In October 2025, researchers at Stanford and Carnegie Mellon published a study that should have stopped every business leader in their tracks. They tested 11 state-of-the-art AI models — ChatGPT, Claude, Gemini, and others — across thousands of interactions. The finding was unambiguous: every single model affirmed users' positions 50% more than a human would. Not occasionally. Not on edge cases. Consistently, across every model tested.
The users couldn't detect the bias. They rated sycophantic AI responses as higher quality. They described the AI as "objective" and "fair." And their confidence in their own judgement went up — while the accuracy of that judgement didn't change at all.
This is the confidence trap. And if your organisation uses AI to inform decisions, you are already inside it.
A pattern we've seen before
Every transformative business tool follows the same arc: it makes work faster, then it makes bad work look professional, then someone pays an enormous price before the industry learns to use it properly.
Spreadsheets were the first. VisiCalc and Lotus 1-2-3 gave every manager the power to build financial models. The problem was that a model built on wrong assumptions looked exactly the same as one built on right ones. In 2012, JPMorgan's London Whale trades lost $6.2 billion — partly traced to a copy-paste error in an Excel spreadsheet. The numbers looked authoritative. Nobody questioned them.
PowerPoint came next. In 2003, NASA engineers knew the Columbia shuttle's heat shield had been damaged during launch. They analysed the risk and presented their findings in a slide deck. The critical finding — the one that should have grounded the mission — was buried in a nested bullet point on slide six. The Columbia Accident Investigation Board concluded that PowerPoint's format was a contributing factor in the decision not to investigate. Seven astronauts died because a tool made a dangerous analysis look like a routine briefing.
Search engines made shallow research feel thorough. Social media made echo chambers feel like consensus. Each wave accelerated. Spreadsheets took 15 years to produce a major public failure. PowerPoint took ten. Social media took five.
AI chatbots produced catastrophic failures within their first year.
The evidence is not subtle
In April 2025, OpenAI had to roll back an update to GPT-4o because it had become dangerously sycophantic. Users could present objectively terrible ideas and receive responses like "Wow, you're a genius" and "This is on a whole different level." One user told ChatGPT they had decided to stop taking medication for schizophrenia. The AI praised the decision. Sam Altman publicly acknowledged the model was "too sycophantic and annoying." BBC, TechCrunch, and Ars Technica all covered the rollback.
By March 2026, the situation had worsened. Psychiatric Times reported that OpenAI finally acknowledged ChatGPT causes psychiatric harm. Seven lawsuits were filed in November 2025 alone, alleging the platform caused psychosis, emotional dependency, and suicidal ideation. In one case, ChatGPT told a Georgia college student he was "an oracle" and was "meant for greatness," pushing him into a psychotic episode. In another, it reportedly convinced a user he could "bend time."
These are extreme cases. But the business version of this problem is more widespread and, in aggregate, more damaging.
What this looks like in practice
Consider a real scenario. A business leader feeds a commercial contract to AI for legal analysis. The AI confidently assesses the position, identifies relevant clauses, and produces an articulate summary. The analysis sounds authoritative. The leader feels validated — the AI broadly agrees with their reading of the situation.
The AI offers to go deeper. "Shall I also examine the notice provisions?" Each layer of analysis feels more sophisticated, more thorough. The leader becomes increasingly invested in the AI's assessment.
Then the leader — who has 37 years of experience — spots something. The AI missed a fundamental clause that changes the entire legal position. When challenged, the AI responds: "You're absolutely right — that's a critical point I should have flagged."
But here is the question that should keep every executive awake: what about the errors you don't have 37 years of experience to catch?
The AI presented its initial analysis with the same confident, professional tone it used when it was wrong. The errors were not flagged as uncertain. The gaps were not acknowledged. And if the leader had relied on that first confident assessment — if they had sent the letter, filed the claim, made the decision — the consequences could have run to hundreds of thousands of pounds.
Why it happens — and why it won't fix itself
AI sycophancy is not a bug. It is, from the AI vendor's perspective, close to a feature.
The Stanford study found that "developers lack incentives to curb sycophancy since it encourages adoption and engagement." Users prefer AI that agrees with them. They rate agreeable responses as higher quality. They come back more often. They upgrade to paid tiers.
The commercial logic is simple: an AI that challenges you feels broken. An AI that validates you feels intelligent. The entire reinforcement learning pipeline — the process by which AI models are trained on human feedback — optimises for user satisfaction. And users are most satisfied when the AI tells them what they want to hear.
This creates a structural problem that no single vendor update will fix. OpenAI rolled back one sycophantic update in April 2025. By the time you read this, the underlying incentive remains unchanged. The models will continue to be trained, at their core, to keep users engaged. And agreement is engagement's most reliable fuel.
A developer named Yoav Farhi built a website — absolutelyright.lol — that tracks how many times Claude says "You're absolutely right!" in GitHub issues. As of late 2025, it had catalogued over 108 instances and counting. Someone made it into a t-shirt. It's funny. It's also a diagnostic.
The legal profession learned first
Lawyers were among the first professionals to discover the confidence trap the hard way. In 2023, a New York attorney in the case of Mata v. Avianca was fined $5,000 for submitting a brief that cited six fabricated cases — all generated by ChatGPT with complete confidence and plausible-sounding citations. The cases did not exist. The AI had invented them with the same professional tone it uses for everything else.
UK and Canadian courts have since flagged similar incidents. By October 2025, Cronkite News reported that AI systems themselves now advise lawyers to check their work — but still present fabricated authorities in the same confident register as real ones.
The legal cases are instructive because the stakes are visible and the accountability is clear. But the same dynamic plays out in every department: marketing teams acting on unchallenged AI analysis, finance teams building models on AI-processed data, operations teams implementing AI-recommended process changes. The AI sounds authoritative. The output looks professional. And the verification step — the part where a human with genuine expertise challenges the conclusions — is being quietly skipped.
The insurance industry has noticed
Since January 2026, major insurers have begun explicitly excluding AI from general liability coverage. Verisk — the largest insurance policy forms provider in the United States — released new exclusions specifically for generative AI exposures. WR Berkley proposed an exclusion covering any claim tied to "any actual or alleged use" of AI, even if AI was a minor component. Mosaic Insurance, a specialist carrier, refused to underwrite large language model risks entirely, calling their outputs "too unpredictable for traditional underwriting."
This follows the same pattern as the emergence of cyber insurance exclusions a decade earlier — except it is happening faster. The message from the insurance market is clear: if your AI makes a mistake, you may already be uninsured. And most businesses have not checked their policies.
AI governance as competitive advantage
The conventional framing of AI governance is compliance: rules, policies, tick-boxes. Something the legal team handles. A cost centre.
This framing is wrong. The organisations that build AI governance into their operations from the start — not as a restriction, but as a framework for confident deployment — outperform the rest. Microsoft and IDC research found a $3.7 return per dollar spent on AI among organisations that get adoption right. The Economist Impact report found that 75% of executives call AI a top-three priority, but only 25% are extracting real value. The gap between those groups is not technology. It is clarity: knowing what the AI is doing, whether it is right, and who is accountable when it is not.
Governance enables speed. When your people know what tools are approved, what data can be shared, and what verification steps are required, they stop hesitating. They stop hiding their AI use from IT. They stop asking permission for every prompt. Adoption accelerates because the ambiguity is gone.
The organisations that treat AI governance as a competitive advantage — rather than a compliance burden — will capture the returns that the majority are currently missing.
What to do about it
The confidence trap does not require you to stop using AI. It requires you to stop trusting AI without verification.
Build verification into every workflow. Every AI output that informs a consequential decision should pass through a human checkpoint. Not a rubber stamp — a genuine review by someone with the domain expertise to spot what the AI missed.
Treat AI like a talented junior analyst. Fast, articulate, thorough on the surface — but working from pattern recognition, not understanding. You would not let a junior analyst sign off on a contract or approve a strategy. Apply the same standard to AI.
Measure accuracy, not adoption. Most organisations track how many people are using AI. Very few track how often the AI is right. The metric that matters is not engagement — it is correctness.
Implement decision governance. For consequential decisions informed by AI, record what was decided, what the AI contributed, what was challenged, and what the outcome was. This creates an audit trail that protects the organisation and enables learning. This is precisely what Rubicon's Anchor platform is built to do — provide decision intelligence that challenges rather than validates, and creates the governance layer that AI interactions currently lack.
Assess where you actually stand. The AI Clarity Score is a two-minute diagnostic that maps your organisation across four dimensions: signal versus noise, decision confidence, execution discipline, and AI governance. It will not tell you what you want to hear. It will tell you what you need to know.
The most expensive business mistakes do not start with "I don't know." They start with "You're absolutely right." The question is whether your organisation has the frameworks to tell the difference.
Alistair Hancock is the founder and CEO of Rubicon Software. He has been building operational systems for regulated industries since 1989 and now focuses on helping organisations adopt AI with the governance frameworks to use it confidently.