The Five Whys (Done Right)
The "Five Whys" technique, originally developed by Sakichi Toyoda at Toyota, is one of the most widely used root cause analysis methods. It's also one of the most widely misused. When done poorly, it becomes a blame game that terminates at "human error." When done correctly, it's a structured causal chain analysis that reveals systemic weaknesses.
Bad Five Whys (Pattern Matching disguised as analysis):
1. Why did the site go down? → The deploy broke it.
2. Why did the deploy break it? → The developer made a mistake.
3. Why did they make a mistake? → They weren't careful enough.
4. Why weren't they careful? → They were rushing.
5. Why were they rushing? → Management pressure.
This chain is useless. It produces no actionable improvement. "Tell management to stop pressuring people" is not an engineering solution. Notice how each answer is a pattern match—it jumps to the most obvious narrative without examining evidence. It also terminates at a person rather than a system, which means the same failure will recur the moment a different person makes the same mistake under the same conditions.
Good Five Whys (Causal Reasoning with mechanism at every step):
1. Why did the site go down? → Database connections exhausted. (Evidence: connection pool metrics hit max, application logs show "connection timeout" errors.)
2. Why were connections exhausted? → A new code path opens connections but doesn't close them on error. (Evidence: code review shows missing finally block in exception handler.)
3. Why wasn't the connection leak caught before production? → No integration tests exercise the error-handling path with a real connection pool. (Evidence: test suite review shows only happy-path tests for this code path.)
4. Why don't integration tests use a real connection pool? → The test environment uses an in-memory database with unlimited connections. (Evidence: test infrastructure config.)
5. Why is the test environment different from production? → Infrastructure-as-Code templates were never applied to the test environment. (Evidence: Terraform state comparison shows divergence.)
This chain produces four concrete action items: fix the leak, add integration tests for error paths, align test infrastructure with production, and create a process to keep them in sync. Each action item addresses a specific link in the causal chain.
The Key Difference
Notice what makes the good version work: every "Why" answer includes evidence. "The deploy broke it" is an assertion. "Database connections exhausted, as shown by connection pool metrics hitting maximum" is a fact. The good Five Whys is a chain of verified facts, not a chain of assumptions. If you cannot provide evidence for a step, that step is a hypothesis—mark it as such and go find the evidence before continuing.
A secondary rule: stop when you reach a systemic fix, not when you reach a person. If your chain terminates at "a developer made a mistake," ask one more why: "Why was it possible for this mistake to reach production?" The answer will point to a process, tool, or system that can be improved.
The Hypothesis Log
During incidents, your brain generates hypotheses faster than you can test them. Without discipline, you'll chase three theories simultaneously, lose track of what you've already ruled out, and repeat work. The solution is a Hypothesis Log—a written record maintained in your incident channel (Slack, Teams, or wherever your team communicates during incidents).
Here's the format:
TIMESTAMP: 14:32 UTC
HYPOTHESIS: Memory leak in user-service causing OOM kills
EVIDENCE FOR: Memory usage climbing steadily over 2 hours (Grafana dashboard),
3 pods restarted by OOM killer (kubectl events)
EVIDENCE AGAINST: Other pods running identical code version not affected,
heap dumps show normal object distribution
MECHANISM: If a code-level leak exists, it should affect all pods equally
since they run the same binary with the same traffic distribution
STATUS: WEAKENED - investigating pod-specific factors (traffic routing, node placement)
---
TIMESTAMP: 14:47 UTC
HYPOTHESIS: Pod us-east-1a-user-svc-3 is on a node with a noisy neighbor
EVIDENCE FOR: Node-level CPU steal time is 34% (normal is <5%),
only pods on this specific node show memory pressure
EVIDENCE AGAINST: None yet
MECHANISM: High CPU steal → GC cannot run efficiently → heap grows → OOM
STATUS: INVESTIGATING - checking other workloads on node i-0a1b2c3d4e
This log does three things:
- Forces System 2 thinking: Writing down "evidence against" requires you to actively look for disconfirming data, which is the hardest and most valuable part of debugging.
- Prevents circular investigation: When a new responder joins the incident, they can read the log instead of re-suggesting already-rejected theories.
- Creates post-incident gold: The hypothesis log becomes the backbone of your post-incident review, showing exactly how the team reasoned through the problem.
If asked about your incident response process, mention the hypothesis log. Say: "I maintain a written log of hypotheses with evidence for and against each one. This prevents the team from going in circles and gives us a clear audit trail for the post-incident review." Then add: "The most important field is 'evidence against'—I actively look for data that would disprove my theory, because confirmation bias is the biggest trap in incident response." This demonstrates structured thinking and self-awareness about cognitive biases, which is exactly what interviewers are looking for in senior SRE and platform engineering roles.
Key Point: The hypothesis log's most important field is "EVIDENCE AGAINST." Confirmation bias—the tendency to seek evidence that supports your theory while ignoring evidence that contradicts it—is the most dangerous cognitive trap in incident response. It feels productive to keep finding evidence that confirms your hypothesis. It feels uncomfortable to look for evidence that breaks it. That discomfort is exactly where the most valuable debugging happens. Actively seeking disconfirming evidence is what transforms pattern matching into causal reasoning, and it is the single habit most correlated with fast, accurate incident resolution.