A Systematic Way to Find What Can Break
FMEA (Failure Mode and Effects Analysis) is a structured methodology originally developed for aerospace and manufacturing that systematically walks through every component of a system and asks three questions:
- How can this component fail? (the failure mode)
- What happens to the system when it does? (the effect)
- How would we detect it? (the detection capability)
FMEA is more rigorous than brainstorming risks in a meeting. It forces you to examine every component individually, which prevents the common failure of only thinking about the "obvious" risks. More importantly, it introduces a third dimension — detectability — that simple likelihood × impact scoring ignores entirely.
The FMEA Process for a Software System:
For each component in your architecture (load balancer, API gateway, database, cache, message queue, third-party API, DNS, etc.), fill out this table:
| Component | Failure Mode | Effect on System | Severity (1–10) | Likelihood (1–10) | Detection (1–10) | RPN | Recommended Action |
|---|---|---|---|---|---|---|---|
| Redis cache | Process crashes | All requests hit DB directly; latency spikes 10× | 6 | 4 | 2 (monitoring alerts immediately) | 48 | Add cache-aside pattern with graceful degradation |
| Redis cache | Memory full, evictions | Cache hit rate drops; gradual performance degradation | 4 | 5 | 5 (slow to notice without metrics) | 100 | Add memory usage alerting at 70% threshold |
| Payment API | Timeout (>5s) | Checkout hangs; users abandon carts | 8 | 3 | 3 (users complain before alerts fire) | 72 | Implement circuit breaker with 2s timeout |
| Payment API | Returns incorrect amount | Customer overcharged; legal/compliance risk | 10 | 1 | 8 (hard to detect automatically) | 80 | Add reconciliation job comparing charges to orders |
| DNS provider | Total outage | Entire site unreachable globally | 10 | 1 | 1 (immediately obvious) | 10 | Add secondary DNS provider |
The RPN (Risk Priority Number):
RPN = Severity × Likelihood × Detection
The Detection score is what makes FMEA unique compared to simple risk matrices. A high detection score (meaning the failure is hard to detect) dramatically increases the RPN because silent failures compound. A database slowly corrupting data over weeks is far more dangerous than a database that crashes loudly and immediately — even if both have the same severity and likelihood scores.
Detection scoring (inverted — higher means harder to detect):
| Score | Meaning |
|---|---|
| 1 | Failure is immediately obvious (alerts fire, health check fails, users see errors) |
| 3 | Detected within minutes by monitoring |
| 5 | Detected within hours, possibly by users reporting issues |
| 7 | Detected within days, likely during manual review |
| 10 | Virtually undetectable until catastrophic consequence materializes |
Why the detection dimension changes everything:
Consider two failure modes for a payment service:
- Failure A: Service crashes completely. Severity=8, Likelihood=2, Detection=1. RPN = 16.
- Failure B: Service silently rounds all transaction amounts down by $0.01. Severity=8, Likelihood=2, Detection=9. RPN = 144.
Failure A is nine times less risky by RPN despite having identical severity and likelihood. The crash is immediately visible — you fix it in minutes. The rounding bug could run for months before a financial audit catches it, accumulating regulatory exposure the entire time.
Key Point: The detection dimension is where most engineering teams have blind spots. Teams invest heavily in preventing failures (redundancy, retries, circuit breakers) but underinvest in detecting failures that slip through. FMEA forces you to ask: "If this broke right now, how long would it take us to notice?" That question alone will surface risks your standard monitoring setup is completely blind to.