11 cases. Each names what happened, the source, and the standard that already existed when it happened.

• Knight Capital, 2012. A deploy reached seven of eight servers. The eighth ran dormant code, which woke up and traded. $460 million in 45 minutes. SEC Admin. Proc. 34-70694Instead: verify what the deploy actually reached, and keep a control that stops trading when behaviour goes out of bounds. Required of brokers since 2010 by the SEC's market access rule, 17 CFR 240.15c3-5.
• AWS S3, 2017. A typo in a routine debugging command removed more capacity than intended and took out a region. post-mortemInstead: make destructive tools refuse to remove more than a safe share of capacity, and cap the blast radius by design. AWS changed the tool to do exactly that, and said so in the same post-mortem.
• GitLab, 2017. An engineer troubleshooting replication ran a directory deletion against the primary database instead of the secondary. Five of five backup and replication methods turned out not to work. Roughly six hours of data was gone permanently. GitLab, 2017 incident recordInstead: restore from a backup you have actually restored from. An untested backup is a belief, not a control, and the standard practice of periodic restore drills predates this by decades.
• Cloudflare, 2019. A single regular expression deployed to the global WAF contained catastrophic backtracking. CPU on every machine in the edge network went to 100 per cent and a large fraction of the web went down for about half an hour. Cloudflare post-mortemInstead: stage global config the way you stage code, and bound the work any single rule can do. Regular-expression denial of service was documented well before 2019 and has its own CWE entry.
• Meta, 2021. A routine capacity audit issued a command that withdrew the BGP routes announcing Facebook's DNS servers. The company vanished from the internet for six hours, including the internal tools and door-badge systems needed to fix it. Meta engineering write-upInstead: keep the recovery path off the system it recovers. Out-of-band management is one of the oldest ideas in operations and the reason it exists is precisely this failure mode.
• CrowdStrike, 2024. A content configuration update for the Falcon sensor contained a mismatch between expected and supplied input fields. It crashed roughly 8.5 million Windows hosts into boot loops, grounding airlines and stopping hospitals, and most needed manual intervention per machine. CrowdStrike root cause analysisInstead: content updates are deploys. Stage them, canary them, and give the client a way to reject a payload it cannot parse rather than fail into a kernel crash.
• Rogers, 2022. A routing configuration change removed a filter, flooding the core with routes and collapsing the network nationally for around nineteen hours. Interac payments, government services and 911 calls went with it. CRTC proceedingInstead: maintain a management network that survives the data network, and rehearse the rollback. A change with national blast radius earns a maintenance window and a tested reversal, not a Friday push.
• Atlassian, 2022. A maintenance script intended to remove a deprecated app was given a list of site identifiers instead of app identifiers, and used the permanent-delete API rather than the recoverable one. Around 400 customers lost their sites, and full restoration took roughly two weeks. Atlassian incident reviewInstead: make destructive operations opt-in and reversible by default, and never let a bulk script reach a permanent-delete path without a dry run whose output a human has read.
• OVHcloud, 2021. A fire destroyed a Strasbourg data centre and damaged a second. Many customers discovered their backups were in the same building, or that the backup service they assumed was running had never been enabled. OVHcloud SBG2 fireInstead: a backup in the same fire compartment as the primary is not a backup. Geographic separation of copies is the oldest rule in disaster recovery and is explicit in every framework that mentions the subject.
• British Airways, 2017. A power supply to a data centre was disconnected and then reconnected in an uncontrolled way, causing physical damage to servers. Around 75,000 passengers were stranded over a bank holiday weekend. BA IT outage, May 2017Instead: test the failover, on the real hardware, on a schedule. A redundancy design that has never been exercised is a hypothesis.
• Delta Air Lines, 2016. A power control module failed and several hundred servers did not switch to backup power as designed. Around 2,300 flights were cancelled over three days. Delta operations disruption, August 2016Instead: inventory which systems are actually on protected power, and verify it by pulling the plug in a controlled test rather than by reading the design document.