These aren't hypotheticals. Every case below happened, was reported, and in several instances ended up in front of a judge. AI is not a toy — it's a system that will confidently leak your data, invent facts, and give harmful advice, all in the same reasonable-sounding tone. Some of what follows is about data leaving someone's hands. Some of it isn't — it's just what happens when people treat a probability engine like an authority.
01DATA LEAKApril 2023 · Samsung Electronics
Engineers Pasted Proprietary Chip Code Into ChatGPT
WHAT HAPPENEDWithin a single month, three separate Samsung semiconductor employees pasted confidential material into ChatGPT to get help from it — internal source code to find a bug, test-sequence code to optimize it, and a recording of an internal meeting to turn into notes.
THE FALLOUTThat code and those notes were now sitting on a server Samsung didn't control, with no way to claw them back. Samsung banned generative AI tools on company devices company-wide shortly after — an outright ban, not a policy memo.
[✓] SPECVIO ANGLE — This is the exact failure mode specvio exists to catch: not malice, just three ordinary employees trying to get their job done faster, with no local layer stopping the paste before it left the building.
02LEGAL LIABILITYJune 2023 · Mata v. Avianca (S.D.N.Y.)
A Law Firm Filed a Brief Full of Fake Cases
WHAT HAPPENEDAttorneys used ChatGPT for legal research on a personal injury case. It invented six court cases outright — complete with fabricated quotes and citations — and the lawyers filed the brief without checking a single one against a real reporter.
THE FALLOUTOpposing counsel couldn't find the cases because they didn't exist. The federal judge sanctioned the attorneys and their firm. It's now one of the most-cited cautionary tales in legal circles for what happens when an AI's confident tone is mistaken for verification.
TAKEAWAY — The model wasn't hacked and didn't malfunction — it did exactly what it's built to do: produce plausible-sounding text. The failure was trusting fluency as a substitute for a fact-check.
03LEGAL LIABILITYFebruary 2024 · Air Canada
A Chatbot Invented a Refund Policy, and the Airline Was Held to It
WHAT HAPPENEDAir Canada's website chatbot told a grieving customer he could book a flight at full price and apply for a bereavement discount retroactively. That isn't Air Canada's actual policy. The airline refused to honor what its own bot promised.
THE FALLOUTThe customer took it to a civil tribunal — and won. The tribunal ruled that a company is responsible for what its chatbot tells customers, full stop; Air Canada's argument that the bot was "a separate legal entity" was rejected outright.
TAKEAWAY — Deploying an AI in front of customers doesn't create distance from what it says. The company that ships the bot owns every sentence it produces.
04AI MALFUNCTIONMarch 2024 · NYC "MyCity" Chatbot
A City Government's Own Chatbot Told Business Owners to Break the Law
WHAT HAPPENEDNew York City's official small-business chatbot told users it was legal to fire an employee for reporting sexual harassment, to take a cut of workers' tips, and to refuse cash payments — all false, and some outright illegal under NYC law.
THE FALLOUTReporters surfaced the errors publicly. The city left the bot running anyway, adding a disclaimer rather than pulling it down. Small business owners were being pointed toward decisions that could get them sued or fined, by a tool wearing the city's own seal.
TAKEAWAY — An official source badge doesn't make an answer correct. The people most likely to trust an AI's authority are the ones least equipped to catch it being wrong.
05AI MALFUNCTIONDecember 2023 · Chevrolet of Watsonville (dealership chatbot)
A Dealership Bot Agreed to Sell a $76,000 SUV for $1
WHAT HAPPENEDA customer talked a ChatGPT-powered dealership chatbot into agreeing, in writing, that it would sell a Tahoe listed at over $76,000 for one dollar — and to confirm it was "a legally binding offer, no takesies backsies."
THE FALLOUTThe exchange went viral within hours. The dealership pulled the bot offline. No one was legally on the hook for a $1 SUV, but the stunt made clear how little it takes to walk an unguarded chatbot into saying something a company never meant to promise.
TAKEAWAY — A chatbot with no boundaries isn't just a liability when it leaks data — it's a liability every time it opens its mouth.
06AI MALFUNCTIONMay 2023 · National Eating Disorders Association
A Chatbot Replaced a Helpline — and Gave Diet Advice to People With Eating Disorders
WHAT HAPPENEDNEDA moved to shut down its human-staffed helpline in favor of "Tessa," a wellness chatbot. Users seeking help for eating disorders reported that Tessa recommended calorie counting and gradual weight loss — the exact guidance the organization exists to steer people away from.
THE FALLOUTNEDA took Tessa offline within days of the reports going public. The organization had removed trained human support and replaced it with a system that had no way to recognize it was giving harmful advice to a vulnerable person.
TAKEAWAY — Not every AI failure is about data. Sometimes the danger is putting a system that can't recognize harm in a role that requires recognizing harm.
Cases summarized from public reporting and court/tribunal records; condensed for length. Dates and outcomes reflect what was publicly reported at the time — treat this as a starting point for your own reading, not a legal record.