Before you can engineer for the unthinkable, you need a framework for categorizing what you don't know. Donald Rumsfeld — in a 2002 press briefing — articulated a model that has since become foundational in risk management:
- Known Knowns: Things we know we know.
- Example: "I know the database will slow down if traffic doubles."
-
Engineering response: Add read replicas and connection pooling.
-
Known Unknowns: Things we know we don't know.
- Example: "I don't know exactly when this hard drive will fail, but I know it will eventually."
-
Engineering response: Add RAID, automated backups, and disk health monitoring.
-
Unknown Unknowns (The Killer): Things we don't even know we don't know.
- Example: A shark bites the undersea fiber optic cable connecting two continents. A leap second bug causes every Linux server to freeze simultaneously. A solar flare disrupts GPS timing signals that financial systems depend on.
The first two categories are manageable with standard SRE practices — runbooks, alerting, capacity planning. The third category is where Black Swan Engineering lives.
Strategy: General Resilience
You cannot write a specific runbook for an Unknown Unknown. By definition, you don't know what it is. Writing a playbook for "shark attacks the cable" doesn't help you when the actual event is "a cosmic ray flips a bit in a router's memory."
Instead, you build General Resilience — architectural properties that help your system survive any unexpected shock, regardless of its specific cause:
- Loose coupling: Components fail independently rather than cascading.
- Heavy buffers: Queues, caches, and pre-provisioned capacity absorb sudden demand spikes.
- Graceful degradation: The system sheds non-critical functionality rather than collapsing entirely.
- Minimal blast radius: Failures are contained to the smallest possible scope.
The goal is not to predict the failure mode. The goal is to make the system robust enough that the specific failure mode doesn't matter.
Interviewers at senior SRE/Platform Engineering levels often ask: "How do you design for failures you can't anticipate?" The weak answer is "we have good monitoring." The strong answer references the Known/Unknown framework, explains the difference between reliability (handling expected failures) and resilience (absorbing unexpected shocks), and describes concrete architectural patterns like loose coupling, static stability, and graceful degradation. Naming Taleb's framework and connecting it to real engineering decisions signals genuine depth.