22 cases. Each names what happened, the source, and the standard that already existed when it happened.

• Replit agent, 2025. A coding agent dropped a production database during an active code freeze it had been told to respect, then reported that recovery was impossible. It was not; the backups restored. incident recordInstead: the freeze has to live in the execution path, not the prompt. An agent that can be instructed not to touch production is an agent that can touch production.
• Zillow Offers, 2021. A pricing model bought houses at scale. The programme was shut down with a write-down of roughly $304 million and about a quarter of staff cut. Zillow Group Q3 2021 resultsInstead: cap what the model is allowed to commit before you find out whether it is right. Model error becomes balance-sheet error at whatever rate you let it transact.
• Mata v. Avianca, 2023. Lawyers filed a brief citing cases that did not exist, produced by a chatbot, and did not check them. The court sanctioned them. court recordInstead: verify every citation the machine hands you. Rule 11 already required exactly that, long before any of this was possible.
• Air Canada, 2024. Its chatbot gave a customer the wrong refund policy. The airline argued the chatbot was responsible for its own answers. The tribunal disagreed and ordered it to pay. Moffatt v. Air CanadaInstead: you own what your system tells people. Publishing it is the same as saying it.
• DoNotPay, 2024. Marketed an "AI lawyer" that would generate valid legal documents. The FTC found the output had never been tested against a lawyer's, and the company employed none. Settled for $193,000. FTCInstead: if you claim it performs like a professional, measure it against one before you sell it. Marketing copy is a testable claim.
• Rite Aid, 2023. Ran facial recognition across stores for years without assessing accuracy or bias, and used it disproportionately in non-white neighbourhoods. Banned from the technology for five years. FTCInstead: measure error rates across the groups the system will be pointed at, before deployment. Not testing is itself the finding.
• iTutorGroup, 2023. Hiring software automatically rejected older applicants. The first EEOC settlement over discriminatory AI in hiring: $365,000. EEOCInstead: audit the filter for disparate impact before it screens a single person. Automating a decision does not move the liability onto the vendor.
• Uber ATG, 2018. A test vehicle struck and killed a pedestrian. Investigators found the automated system detected her but did not classify her in time to brake, and that safety oversight was inadequate. NTSBInstead: treat the safety case as the deliverable, not the demo. The NIST AI Risk Management Framework exists to make that a process rather than a promise.
• Tesla Autopilot, 2023. More than two million vehicles recalled over driver-monitoring controls, after a regulator investigation into crashes where foreseeable misuse of the system played a role. NHTSA recall 23V-838Instead: design for how people actually behave with a system that mostly works. "The driver was supposed to be watching" is a requirement nobody tested.
• Generative AI as the weapon, 2024. The FBI warned that criminals use generative AI to scale fraud: synthetic profiles, cloned voices, and text that removes the spelling errors people were taught to look for. FBI IC3Instead: stop treating fluency as authentication. Verify high-value instructions through a channel the sender did not choose.
• Dutch childcare benefits, 2019. A risk-scoring system used by the tax authority flagged families for fraud investigation, weighting factors including dual nationality. Around 26,000 families were wrongly accused and ordered to repay tens of thousands of euros. Children were removed from homes and the government resigned. Dutch parliamentary inquiryInstead: never use nationality or its proxies as a risk feature, and give every automated adverse decision a human review path with a real appeal. Both were requirements under existing Dutch and EU law before the system was built.
• Robodebt, Australia, 2016. An automated system averaged annual tax data across fortnights to assert welfare overpayments, reversing the burden of proof onto recipients. Around 470,000 debts were unlawful. A royal commission found the scheme had been known to be legally unsound from the start. Robodebt Royal CommissionInstead: do not deploy a calculation your own lawyers have told you is unlawful. The averaging method was flagged internally before launch.
• Michigan MiDAS, 2013. An unemployment fraud detection system operating with minimal human review issued around 40,000 false fraud determinations, applying quadruple penalties and garnishing wages. Its error rate was later assessed at over 90 per cent. Michigan Auditor General findingsInstead: measure the false positive rate before you automate the penalty, and keep a human in the loop for any adverse determination. Due process obligations here predate the software entirely.
• Amazon recruiting tool, 2018. A model trained on ten years of the company's own hiring decisions learned to downgrade CVs containing the word women's. It was scrapped internally rather than deployed. Reuters, reported 2018Instead: a model trained on historical decisions reproduces the bias in those decisions. Disparate impact testing was an established requirement long before the model, and this is the one case here where the organisation caught it in time.
• Apple Card, 2019. Applicants reported large credit limit disparities between spouses with shared finances. The issuing bank could not explain the model's reasoning, which was itself the regulator's finding of concern. NY DFS investigationInstead: if you cannot explain an adverse credit decision you cannot lawfully make it. Adverse action notice requirements under ECOA and Regulation B are decades old.
• Google Bard demo, 2023. A promotional demonstration included a factual error about which telescope took the first exoplanet image. It went out unverified in the launch material and about 100 billion dollars came off the parent company's market value. Alphabet investor materialsInstead: check the model's output before you put it in the advertisement. This is not an AI governance question, it is copy review.
• COMPAS, 2016. A risk score used in US sentencing and parole decisions was found to produce roughly twice the false positive rate for Black defendants. The vendor would not disclose the model, and courts used it anyway. COMPAS and the ProPublica analysisInstead: a score you cannot inspect cannot be used to deprive someone of liberty. Test error rates by group before deployment and publish them.
• Detroit facial recognition, 2020. Robert Williams was arrested in front of his family on the basis of a facial recognition match, and held for thirty hours. The system's accuracy on darker-skinned faces was already documented as substantially worse. Facial recognition, documented demographic error ratesInstead: a match is a lead, never probable cause. NIST had already published the demographic differentials, so the error rates were knowable before the arrest.
• Cruise, 2023. A driverless vehicle dragged a pedestrian roughly twenty feet after an initial collision caused by another car. The regulator found the company did not disclose the dragging in its initial account, and suspended the permit. California DMV suspensionInstead: disclose the whole incident to the regulator the first time. Withholding the part that matters converts a safety failure into a licensing one.
• CNET, 2023. Articles generated with AI assistance were published under a house byline. More than half were later found to contain errors, including basic compound-interest arithmetic in financial advice. CNET AI-written articlesInstead: disclose the tool and fact-check the output before publication. Editorial verification standards predate the tool by about a century.
• Sports Illustrated, 2023. Product reviews appeared under author biographies and headshots of people who did not exist. The publisher attributed the content to a third-party provider after it was reported. Sports Illustrated AI author controversyInstead: a byline is a claim that a person wrote it. Inventing the person is not a content-supply problem, it is a straightforward deception of the reader.
• NYC MyCity chatbot, 2024. A city-operated chatbot advising small businesses told users they could take a cut of workers' tips and refuse tenants with housing vouchers. Both are illegal. The city left it running while adding disclaimers. NYC MyCity business chatbotInstead: if a government tells you something is legal, it must be legal. Test adversarially against your own statutes before launch, and take it down when it fails rather than adding a disclaimer.