A single component failing is rarely a Black Swan. What turns a manageable incident into a catastrophe is cascading failure — when one component's failure triggers failures in others, which trigger more failures, until the entire system collapses.
How Cascades Happen
Consider a microservices architecture where Service A calls Service B, which calls Service C:
User → Service A → Service B → Service C (database)
Service C slows down due to a disk issue. Service B's requests to C start timing out after 30 seconds. Service B's thread pool fills up waiting for C. Service A's requests to B now also time out. Service A's thread pool fills up. The entire system is now unresponsive — not because of a catastrophic failure, but because of a slow disk in one downstream service.
This is called a cascading timeout failure, and it is one of the most common causes of large-scale outages.
Defense Patterns
Circuit Breakers: A circuit breaker monitors calls to a downstream service. If the failure rate exceeds a threshold (e.g., 50% of calls fail in 10 seconds), the circuit "opens" and subsequent calls fail immediately with a cached or default response — rather than waiting for a timeout. This prevents thread pool exhaustion upstream.
Normal state: Request → Service B → Service C ✓
Circuit open: Request → Circuit Breaker → Returns cached/default response immediately
(does NOT call Service C)
After recovery: Circuit "closes" after a test request succeeds
Bulkheads: Borrowed from ship design — a ship is divided into watertight compartments so a breach in one doesn't sink the whole vessel. In software, bulkheads mean separate thread pools or resource limits for different downstream dependencies. Service B's calls to Service C use a dedicated thread pool of 10 threads. Even if all 10 are blocked, Service B's calls to Service D (a different dependency) use a separate pool and remain unaffected.
Timeouts Everywhere: Every network call must have an explicit timeout. The default in most HTTP clients is "wait forever." In a distributed system, "wait forever" means "cascade forever." Set aggressive timeouts (hundreds of milliseconds for internal services, a few seconds for external ones) and handle the timeout as a first-class failure case.
Shed Load: When a service is overwhelmed, it should actively reject excess requests with a 503 Service Unavailable rather than accepting them and slowing down for everyone. A system that rejects 20% of requests at the edge is far better than one that serves 100% of requests at 10x normal latency (which causes cascades upstream).