What Is a Runbook?
A Runbook (also called a Playbook) is a specific type of operational document that describes how to handle a known situation — either a routine task or an incident. It is not a design document or a tutorial. It is a decision tree and checklist for someone who needs to act, not learn.
The defining standard for a runbook is the 3 AM Test: "Could a junior engineer, woken up at 3 AM by a PagerDuty alert, follow this runbook to resolution without calling anyone for help?"
If the answer is no, the runbook is incomplete.
The Six Essential Sections
1. Title and Alert Reference
Name the runbook after the alert or situation that triggers it. If your monitoring system fires an alert called PaymentService_HighErrorRate, name the runbook exactly that. Engineers searching at 3 AM will find it instantly.
2. Triggers
When should this runbook be used? List the exact alert names, thresholds, or conditions.
"Use this runbook when the
PaymentService_HighErrorRatealert fires (error rate > 5% for 5 minutes)."
3. Prerequisites
What access, tools, or permissions does the operator need before starting?
- VPN connected to
prod-vpnkubectlconfigured withprod-clustercontext- PagerDuty
adminrole (to silence alerts)
4. Investigation
How does the operator confirm this is a real problem and not a false alarm? Provide specific dashboards, queries, or commands.
"Check the [Payment Error Rate dashboard]. If the graph shows a spike followed by recovery, this may be a transient issue — monitor for 5 minutes before proceeding."
5. Remediation (The Checklist)
Step-by-step commands to resolve the issue. This is the core of the runbook. Every step must be a specific, copy-pasteable command with expected output.
| Bad | Good |
|---|---|
| "Restart the server." | "ssh deploy@payment-prod-01 'sudo systemctl restart payment-api'" |
| "Check the logs." | "kubectl logs -n payments -l app=payment-api --tail=100 \| grep ERROR" |
| "Scale up if needed." | "kubectl scale deployment payment-api -n payments --replicas=6" |
6. Escalation
If the remediation steps fail, who is the next person to contact? Include their name, role, and how to reach them (phone, Slack handle). Never leave an operator at a dead end.
Narrative vs. Checklist
Runbooks should be checklists, not essays. Do not explain the history of the payment service or the architectural decisions behind the database schema. That belongs in a design document. A runbook reader needs to act, not understand. Use numbered lists, code blocks, and bold text for warnings. Prose paragraphs are the enemy of the 3 AM reader.
A common SRE interview question is: "Walk me through how you would improve your team's incident response process." Mentioning runbooks is expected. What separates strong candidates is the next level: (1) runbooks are linked directly from alerts (the PagerDuty alert body contains the runbook URL), (2) runbooks are tested in game days, and (3) runbooks include a "Post-Incident" section where the responder records what actually happened — turning each incident into a runbook improvement.