"If This, Then That"
A runbook must be inextricably linked to a specific alert. The moment an engineer lands on the page, they should know exactly why they are there and what to do next.
- Bad Runbook: A generic page titled "How the Database Works."
- Good Runbook: A page titled "Alert:
DatabaseCPUHigh— CPU > 90% for 5 minutes."
The title alone tells the engineer: this is the right document, and here is the threshold that triggered it.
Essential Sections
Every runbook needs these five standard sections, in this order:
- Trigger: What alert sends you here? Include the exact alert name and threshold. Example: "
HighErrorRateonLoadBalancer— error rate > 5% for 3 minutes." - Impact: Why do we care, and how bad is it? Example: "Users cannot complete checkout. Revenue impact: ~$2,000/minute. Severity: SEV2."
- Triage (Validation): Quick checks to confirm it is a real incident and not a false alarm. Example: "Open the [Checkout Dashboard]. If the error rate graph is flat, this is a false positive — silence the alert and close the incident."
- Mitigation (The Fix): Step-by-step instructions to restore service. Numbered. Imperative. Specific. Example: "1. SSH to the bastion host. 2. Run
kubectl rollout restart deployment/checkout-api. 3. Wait 90 seconds. 4. Verify error rate drops below 1% on the dashboard." - Escalation: If mitigation fails, who is the expert to call, and how? Example: "If errors persist after Step 4, page the Payments Team via PagerDuty rotation
payments-oncall."
Cognitive Style: Write in Commands
The single most important stylistic rule for runbooks is to write in imperative commands — the same way you would speak to someone in an emergency.
| ❌ Don't Write | ✅ Write Instead |
|---|---|
| "The user might consider checking the logs if they have time." | "Check the logs. Run: tail -100 /var/log/app/error.log" |
| "It could be worth restarting the service." | "Restart the service. Run: systemctl restart nginx" |
| "You should investigate disk usage." | "Check disk usage. Run: df -h /" |
Every sentence that requires the engineer to interpret, decide, or infer is a sentence that costs time and introduces error at 3 AM. Remove all of them.
Interviewers for SRE and Platform Engineering roles often ask: "How do you measure runbook quality?" A strong answer goes beyond "it should be clear" and addresses three dimensions: specificity (does it reference exact commands and thresholds?), freshness (when was it last verified against a real or simulated incident?), and linkage (is every alert in your monitoring system pointing to a runbook URL?). Candidates who can articulate all three stand out from those who only discuss writing style.