Prevention Over Detection
Causal reasoning isn't just for incidents—it's a design tool. When you understand why systems fail (not just that they fail), you can design systems that are resistant to those failure modes before they occur in production.
Consider a common pattern: a service experiences cascading failure because a downstream dependency becomes slow. The pattern-matching response is "add retries." The causal-reasoning response asks a deeper question:
- Why does a slow dependency cause cascading failure? Because the calling service holds connections open while waiting, exhausting its connection pool.
- Why does connection pool exhaustion cascade? Because all request threads are blocked, so the service can't handle any requests—even those that don't need the slow dependency.
- What's the mechanism of the cascade? Upstream services retry against the now-unresponsive service, multiplying load on an already-struggling system.
This causal chain reveals that "add retries" actually worsens the problem (more retries = more load on the struggling service). The correct solutions address different points in the causal chain:
| Causal Link | Solution | Pattern |
|---|---|---|
| Slow dependency holds connections open | Set aggressive timeouts | Timeout pattern |
| All threads blocked by one dependency | Isolate dependency calls into separate thread pools | Bulkhead pattern |
| Upstream retries multiply load | Implement exponential backoff with jitter | Backoff pattern |
| Service accepts requests it can't fulfill | Shed load when approaching capacity | Circuit breaker pattern |
Each solution targets a specific mechanism in the failure chain. This is causal reasoning applied to architecture. Notice that without the causal chain, you'd implement retries and make the system worse—a common mistake that has caused real production outages at scale.
The Pre-Mortem Technique
A pre-mortem inverts the Five Whys. Instead of asking "why did this fail?" after an incident, you ask "how could this fail?" before shipping. The process:
- Assume the system has already failed catastrophically six months from now.
- Work backward: "What sequence of events could have caused this?"
- For each sequence, ask: "What is the mechanism? Is it plausible given our architecture?"
- Prioritize by likelihood and impact. Build mitigations for the top risks before launch.
This technique is valuable because it forces causal reasoning before you're under the stress of a live incident, when System 1 dominates and System 2 is impaired. It also surfaces disagreements about system behavior that are better resolved in a design review than during a 3 AM outage.
A pre-mortem is most effective when it includes people with different perspectives: the engineer who built the service (knows the code), the SRE who will operate it (knows the failure modes), and a skeptic who wasn't involved in the design (asks the "obvious" questions that familiarity blinds the team to).
When asked "How do you ensure reliability in a new system?", most candidates describe monitoring and alerting. Stronger candidates describe pre-mortems and failure mode analysis. Say: "Before we launch, I run a pre-mortem: I assume the system has failed and work backward through the causal chain to identify the most likely failure modes. Then I design mitigations that break the chain at the earliest possible point." This demonstrates that you think about reliability proactively, not reactively—a key distinction between mid-level and senior engineers.
Key Point: The highest-leverage application of causal reasoning is in system design, not incident response. Every architectural decision—timeouts, retries, circuit breakers, bulkheads, queue depths—should be justified by a specific causal chain: "If X happens, it causes Y through mechanism Z, and this design choice breaks that chain at step N." Engineers who can articulate these chains build more resilient systems, make better trade-off decisions, and can defend their architectural choices under scrutiny. The ability to reason causally about failure modes is what distinguishes a reliability engineer from someone who simply reacts to alerts.