Firefighting First
Once an engineer acknowledges the incident, their first instinct might be to find and fix the bug. Resist this. The first job is not to fix the root cause — it's to restore service to customers as fast as possible. This is called Mitigation.
Mitigation is a temporary workaround that stops the bleeding. It doesn't fix the underlying problem, but it makes the system usable again. Think of it like a paramedic stabilizing a patient at the scene: the goal is to stop the immediate harm, not to perform surgery on the roadside.
Common mitigation strategies:
| Situation | Mitigation Action |
|---|---|
| Bad code deployment | Roll back to the previous known-good version |
| Database overwhelmed by traffic | Enable aggressive caching or add read replicas |
| Third-party API failing | Switch to a fallback provider or return cached responses |
| Runaway process consuming CPU | Kill the process and restart the service |
| Feature causing errors | Disable the feature via a feature flag |
Metric: Time to Mitigate (TTM)
TTM is the elapsed time from acknowledgement until the service is restored and customers are no longer impacted.
TTM = Time of Service Restoration − Time of Acknowledgement
- Goal: This is the most customer-facing metric. A short TTM means that even when things break, they don't stay broken for long. Customers experience a brief disruption rather than a prolonged outage.
- How to improve TTM: Invest in fast rollback mechanisms (one-command deploys and rollbacks), feature flags (instant kill switches for bad features), runbooks (step-by-step guides so engineers don't have to think from scratch under pressure), and chaos engineering (practice recovering from failures before they happen in production).
The distinction between mitigation and resolution is one of the most important concepts in incident management. Conflating them leads to two failure modes: (1) engineers spending hours debugging root cause while customers are still impacted, or (2) declaring an incident "resolved" after a rollback without ever fixing the underlying bug, guaranteeing a recurrence.