Preparing for Black Swans

The Unknown Unknowns

A Black Swan is a term coined by Nassim Nicholas Taleb for an event with three defining characteristics:

1. It lies outside the realm of regular expectations — nothing in the past convincingly points to its possibility.
2. It carries an extreme impact.
3. Human nature makes us concoct explanations for it after the fact, making it seem predictable in retrospect.

The term comes from the historical assumption that all swans were white — a belief held with complete confidence until Dutch explorers reached Australia in 1697 and discovered black swans. The lesson: the absence of evidence is not evidence of absence.

Engineering Black Swans include:

• A cloud provider permanently losing your data — not just an outage, but actual data loss. This has happened: GitLab's 2017 database incident deleted 300GB of production data due to a mistyped command during an incident response.
• A core open-source dependency being compromised — the xz backdoor discovered in 2024 was embedded in a widely-used compression library and nearly affected millions of Linux systems. The event-stream npm package attack in 2018 targeted a specific Bitcoin wallet by compromising a transitive dependency.
• A government regulation making your architecture illegal overnight — GDPR enforcement in 2018 forced architectural changes across thousands of companies that had never considered data residency or the right to erasure.
• Your entire engineering team becoming unavailable simultaneously — pandemic, office disaster, or mass resignation triggered by an acquisition.

You cannot predict or prevent a Black Swan with standard risk modeling because, by definition, its probability is unknown or unknowable. Your risk matrix has no row for "impossible but it happened anyway."

How to Prepare — Build for Resilience, Not Prediction:

Since you cannot predict what will happen, you must build systems that can survive anything happening:

• Eliminate single points of failure ruthlessly. Not just in infrastructure — in knowledge, in processes, in vendors. If one person, one region, or one provider going away would cripple you, that is a Black Swan vulnerability.

• Optimize for recovery speed (low RTO and RPO). Assume you will experience a catastrophic failure. The question is not "will it happen?" but "how fast can we rebuild from zero?" RTO (Recovery Time Objective) is the maximum acceptable time to restore service after a failure. RPO (Recovery Point Objective) is the maximum acceptable amount of data loss measured in time — for example, an RPO of 1 hour means you can afford to lose at most 1 hour of data. Both must be tested, not assumed.

• Maintain offline, immutable backups. If ransomware encrypts your production database and your online backups (because they are on the same network), you need an air-gapped copy. The 3-2-1 backup rule is the standard: 3 copies of data, on 2 different media types, with 1 stored offsite and offline.

• Practice Chaos Engineering. Intentionally inject failures in production or staging to discover how the system behaves under conditions you did not design for. Netflix's Chaos Monkey randomly terminates production instances. If your system cannot handle a random instance termination, it certainly cannot handle a Black Swan. The goal is to find your weaknesses before a real incident does.

• Write rebuild-from-scratch runbooks. Document how to reconstruct your entire system from zero — from DNS records to database schemas to deployment pipelines. If you have never done this, you do not know if it is actually possible.

Key Point: The most important insight from Black Swan theory for engineers is this: the absence of evidence is not evidence of absence. Just because your system has never experienced a catastrophic failure does not mean it is resilient — it may mean you have been lucky. Resilience is proven by testing, not by uptime streaks.