Real Outages Behind the Fundamentals¶

By Sameer Alam · Updated 20 October 2026

Most famous outages weren't caused by anything exotic. They came from an empty shell variable, a leap second, a certificate nobody watched, a DNS record that stayed in caches, or a package index that trusted the wrong source. Each one maps to a chapter in this track, and each has a guardrail you can apply this week.

Every incident here comes from a public postmortem, advisory, or court record, linked in its section.

Steam for Linux, 2015: An Empty Variable Deleted Home Directories¶

In January 2015 a user reported that launching Steam had deleted every file their account owned, including a backup drive mounted under /media. Steam's launcher script contained this line (issue #3671):

rm -rf "$STEAMROOT/"*

STEAMROOT was computed with cd and $PWD. When Steam's directory had been moved, the cd failed, STEAMROOT became an empty string, and the quoted part collapsed to /. The unquoted * then expanded to every top-level directory. GNU rm refuses rm -rf /, but that protection (--preserve-root) covers only / itself, never /*.

Guardrail: make the expansion fail instead of collapsing. ${VAR:?} stops the script when the variable is empty or unset, and ShellCheck's SC2115 flags the original pattern. set -u would not have helped, because an empty variable is set.

rm -rf "${STEAMROOT:?}/"*
cd "$target" || exit 1

See it in the Bash argument viewer: choose Steam 2015, then …with the guard.

Bumblebee, 2011: One Space Deleted /usr¶

The installer of Bumblebee, a project for NVIDIA Optimus laptops, shipped with this line (fix commit):

rm -rf /usr /lib/nvidia-current/xorg/xorg

The author meant /usr/lib/nvidia-current/xorg/xorg. One space split a path into two arguments, and rm removed /usr.

Guardrail: for destructive commands, review what the program receives, not what the line looks like. Build paths from a checked prefix (rm -rf -- "${PREFIX:?}/lib/nvidia-current"), and run installers in a disposable VM or container first.

GitLab.com, 2017: A Delete on the Wrong Database Server¶

On 31 January 2017, an engineer fighting replication lag ran a recursive delete of the PostgreSQL data directory, intending it for the secondary. The terminal was connected to the primary. About 300 GB was gone in seconds. Then the backups failed one after another: the scheduled pg_dump had been silently producing nothing because of a PostgreSQL version mismatch, and the failure emails were being rejected. GitLab restored from a six-hour-old snapshot taken by chance, and lost about six hours of data (postmortem).

Guardrail: make the host impossible to miss (hostname and role in the prompt, a red prompt on production), make destructive scripts check where they are running, and alert when a backup is missing, not only when one fails. A backup you've never restored is a hope, not a backup.

[[ "$(hostname -s)" == db-replica-* ]] || { echo "refusing: not a replica" >&2; exit 1; }

Leap Second, 2012: Servers Spun at 100% CPU¶

A leap second was inserted at 23:59:60 UTC on 30 June 2012. On many Linux kernels, the clock change wasn't propagated to the high-resolution timer subsystem, so timers fired immediately and repeatedly. Programs that sleep on futexes, notably Java applications and MySQL, woke in a tight loop and pinned every CPU. Setting the clock (date -s "$(date)") forced the timer update and stopped it (LWN).

Guardrail: time is an input that can jump. Large providers now smear leap seconds over hours instead of inserting them, and in 2022 the General Conference on Weights and Measures voted to stop adding leap seconds by 2035. On your own fleet, sync with chrony from a smearing source, or none at all, but never mix smeared and unsmeared servers.

Cloudflare, 2017: Time Went Backwards¶

At midnight UTC on 1 January 2017, another leap second. Cloudflare's DNS service measured round-trip times as time.Now().Sub(start) and passed the result to a random-number function that panics on negative input. During the leap second the wall clock stepped back, a duration came out negative, and the service crashed for some queries until a fix rolled out about six hours later (postmortem). Go 1.9 later added monotonic clock readings to time.Time, so subtracting two time.Now() values became safe.

Guardrail: measure elapsed time with a monotonic clock, never the wall clock: time.monotonic() in Python, CLOCK_MONOTONIC in C, $SECONDS in Bash for coarse timing. See CLI Tools and subprocess.

xz Utils, 2024: A Backdoor Noticed Because SSH Got Slower¶

On 29 March 2024, Andres Freund, a PostgreSQL developer, reported that SSH logins on a Debian testing machine were using noticeably more CPU and producing Valgrind errors. He traced it to xz Utils 5.6.0 and 5.6.1, whose release tarballs had been backdoored after a years-long effort to gain maintainer trust (CVE-2024-3094, oss-security post). sshd doesn't use xz itself; several distributions patch it to link libsystemd, which pulled in liblzma. The affected versions had reached only testing and rolling-release distributions.

Guardrail: a performance regression nobody can explain is worth chasing. Know your dependency graph (ldd $(command -v sshd)), prefer distribution packages that rebuild from source, and keep production on stable releases rather than the newest upstream tarball.

Dirty Pipe, 2022: Corrupted Logs Revealed a Kernel Bug¶

Max Kellermann at CM4all investigated gzip log archives that failed their CRC checks, month after month. The cause was a Linux kernel bug in pipe buffers, present since 5.8, that let data be written into the page cache of files the process could only read. It became CVE-2022-0847, a local privilege escalation, fixed in 5.16.11, 5.15.25, and 5.10.102 (write-up). It is in CISA's catalog of vulnerabilities exploited in the wild.

Guardrail: "weird corruption" is sometimes the kernel. Installing a kernel update does nothing until you reboot (or live-patch), so track the running kernel: compare uname -r with the newest installed kernel, as in Packages and Patching.

Ubuntu 25.10, 2025: Automatic Updates Silently Stopped¶

Ubuntu 25.10 replaced GNU coreutils with the Rust rewrite, uutils. In October 2025 users found that unattended upgrades had stopped checking for updates. The scheduler compares a stamp file's age using date -r FILE, and the Rust date returned the current time instead of the file's, so the last check always looked fresh. Manual apt runs were unaffected, which is why nobody noticed; the fix shipped in rust-coreutils 0.2.2-0ubuntu2.1 (Phoronix).

Guardrail: monitor the outcome, not the mechanism. Alert when /var/lib/apt/periodic/update-success-stamp is older than two days, and test scripts against both GNU and uutils coreutils before an Ubuntu 26.04 upgrade, which ships the Rust tools by default.

Cloudflare, 2019: One Regular Expression Used Every CPU¶

On 2 July 2019 Cloudflare deployed a new web-application firewall rule. Its regular expression contained .*(?:.*=.*), which causes catastrophic backtracking in a backtracking engine: on some inputs the work grows explosively. CPU on every machine serving HTTP traffic hit 100%, and Cloudflare was down globally for 27 minutes (postmortem). The fix was to move to regex engines that guarantee linear time.

Guardrail: grep, sed, and awk mostly use automata, but grep -P, Python's re, Perl, and PCRE backtrack. Avoid nested quantifiers over the same characters ((.*)*, .*.*=), anchor patterns, and when matching untrusted input use RE2-style engines (Go's regexp, Rust's regex, Python's google-re2). See Text Processing.

Slack, 2021: A DNS Rollback Took 24 Hours¶

On 30 September 2021 Slack enabled DNSSEC on slack.com (its third attempt) and published the DS record in the .com zone. A Route 53 bug with wildcard records produced signed "no such record" answers for some queries, and resolvers that cache such proofs, including Google Public DNS, concluded names didn't exist. Slack pulled the DS record and unsigned its zones, but resolvers that had cached the DS record (which .com lets resolvers keep for 24 hours) kept validating against keys that no longer existed and returned SERVFAIL. Under 1% of users were affected, for about 24 hours (postmortem).

Guardrail: before a DNS change, ask which TTL controls the rollback, including records in the parent zone you don't host. Test with validating resolvers (dig @8.8.8.8, dig +dnssec), and lower TTLs a full old-TTL in advance. See DNS.

Dyn, 2016: One DNS Provider, Many Dark Sites¶

On 21 October 2016, a botnet of compromised cameras and routers (Mirai) flooded Dyn, a managed DNS provider. Sites that used Dyn as their only authoritative DNS, including Twitter, GitHub, Reddit, Spotify, and PayPal, were unreachable for hours, mostly on the US East Coast (KrebsOnSecurity).

Guardrail: authoritative DNS is a single point of failure unless the domain's NS set spans two providers. Longer TTLs also keep cached answers working through a provider outage, a trade-off against how fast you can change records.

Let's Encrypt, 2021: An Expired Root Broke Old Clients¶

The root certificate DST Root CA X3 expired on 30 September 2021 at 14:01:15 UTC. Let's Encrypt's chain had already moved to ISRG Root X1, but clients with outdated trust stores, and OpenSSL 1.0.2, which tried the expired path and gave up, began failing to connect, including many servers and containers calling APIs (Let's Encrypt).

Guardrail: the client side of TLS needs patching too. Keep ca-certificates current in base images, and before a CA changes chains, test from your oldest client: an old base image, an appliance, a Java runtime with its own trust store.

Ericsson, 2018: An Expired Certificate Took Mobile Data Offline¶

On 6 December 2018, software in Ericsson's SGSN-MME, part of the mobile core, failed in about 11 countries. O2 in the UK lost mobile data for most of a day, and SoftBank in Japan for over four hours. Ericsson's initial root cause: an expired certificate in the software versions those operators ran (TelecomTV).

Guardrail: certificates hide in appliances, internal services, client certificates, and vendor software, not only on websites. Inventory them all and alert on expiry. With public certificate lifetimes falling to 47 days by 2029, the certificate lifetime planner shows what manual renewal will cost.

Jenkins, 2013: A Force-Push Rewound 186 Repositories¶

On 10 November 2013, a Jenkins developer started a local Gerrit server whose replication plugin was pointed at GitHub and at a directory of 186 stale clones of the project's repositories. The plugin pushed with force by default, so it rewound branches across the Jenkins organization to old commits. The newer commits still existed on GitHub, just unreferenced; branches were restored from developers' up-to-date clones, with help from GitHub support and a script that found the rest (Jenkins report).

Guardrail: block force-pushes to shared branches on the server, because client discipline doesn't scale. When you must force-push your own branch, use git push --force-with-lease --force-if-includes, which refuses if the remote has commits you haven't seen. See Everyday Workflow.

Toyota 2022 and Uber 2016: Keys in Repositories¶

In October 2022 Toyota disclosed that a contractor had published part of the T-Connect source code to a public GitHub repository in December 2017. It contained an access key for a server holding customer data; for nearly five years, 296,019 email addresses and customer numbers were exposed (TechCrunch).

Uber's 2016 breach began with cloud credentials found in a private GitHub repository. The attackers used them to copy data on 57 million riders and drivers from cloud storage. Uber paid them $100,000 to stay quiet and later agreed to a $148 million settlement with US states (Security Affairs).

Guardrail: private doesn't mean safe, and deleting doesn't mean gone: GitHub keeps commits from deleted forks reachable by hash across the fork network (Truffle Security). Rotate a leaked key first, then clean history. Turn on push protection so the next one never lands. See Repository Hygiene.

PyTorch, 2022: pip Chose the Wrong Index¶

Between 25 and 30 December 2022, anyone who installed PyTorch nightly builds on Linux with pip received a malicious torchtriton package. PyTorch's install instructions used --extra-index-url, and pip treats every index as equal, picking the best version from all of them. An attacker had uploaded a torchtriton to PyPI, and pip installed it. It collected system information and files such as SSH keys and sent them out over DNS (PyTorch). This is dependency confusion.

Guardrail: use one index that proxies PyPI, or a resolver with index priority. uv's default first-index strategy takes a package only from the first index that has it, which blocks this attack. Pin hashes in a lock file. See Project Setup.

Check Yourself¶

Next¶

Pick the track whose incident surprised you most: Linux, Shell Scripting, Networking, Git, or Python. For the same treatment of higher-level tools, see Kubernetes Outages and Ansible Outages.