The Root Cause
After mitigation, the immediate pressure is off. Customers are no longer impacted. Now the team can take a breath and do the careful work of understanding why the incident happened and ensuring it doesn't happen again.
Resolution is the process of identifying the root cause — the underlying condition that made the incident possible — and implementing a permanent fix. This is distinct from mitigation, which only addressed the symptom.
Resolution typically involves:
1. Root Cause Analysis (RCA): Tracing the chain of events back to the originating failure. Tools like distributed tracing (Jaeger, Zipkin), log aggregation (Splunk, Loki), and metrics (Prometheus) are essential here.
2. Permanent Fix: Writing and testing code, configuration, or process changes that eliminate the root cause.
3. Verification: Confirming in production that the fix works and the system is stable.
4. Post-Mortem: Documenting the incident, the timeline, the root cause, and the action items to prevent recurrence. (Post-mortems are covered in depth in a separate lesson.)
Metric: Time to Resolve (TTR) / Mean Time to Resolve (MTTR)
TTR is the total elapsed time from the start of the incident until the permanent fix is implemented and verified.
TTR = Time of Permanent Fix − Time of Incident Start
When averaged across multiple incidents, this becomes MTTR (Mean Time to Resolve) — one of the four DORA (DevOps Research and Assessment) metrics used to benchmark engineering team performance.
A critical disambiguation: Some organizations define MTTR as ending at mitigation (service restored), while others define it as ending at resolution (root cause fixed). These are very different numbers. A team that rolls back in 10 minutes but takes 3 days to deploy the permanent fix has a TTM of 10 minutes and a TTR of 3 days. If you only track one number and call it "MTTR," you're hiding important information.
Best practice: Track TTM and TTR separately. TTM tells you how fast you protect customers. TTR tells you how fast you eliminate the underlying risk.
Interviewers frequently ask: "What is MTTR and how do you improve it?" A strong answer distinguishes between MTTR-to-mitigate and MTTR-to-resolve, explains that improving TTM requires operational investments (rollback tooling, feature flags, runbooks) while improving TTR requires engineering investments (better testing, observability, and code quality), and notes that MTTR is a lagging indicator — you can't improve it without first measuring it consistently.