The Problem with Threshold Alerts
If you set an alert to fire every time your error rate exceeds your SLO threshold for a single minute, you will generate enormous alert noise. A brief network blip that lasts 30 seconds and affects 0.001% of your monthly budget should not page an engineer at 3 AM. But a sustained degradation that will exhaust your entire monthly budget in 6 hours absolutely should.
The solution is to alert not on the instantaneous error rate, but on the Error Budget Burn Rate.
Defining Burn Rate
The burn rate is a multiplier that tells you how fast you are consuming your error budget relative to the rate that would exactly exhaust it by the end of the window.
$$\text{Burn Rate} = \frac{\text{Current Error Rate}}{\text{SLO Error Rate}}$$
For a 99.9% SLO, the SLO error rate is 0.1%. If your current error rate is 1%, your burn rate is:
$$\text{Burn Rate} = \frac{1\%}{0.1\%} = 10$$
A burn rate of 1 means you will exhaust your budget in exactly 30 days — right on schedule. This is acceptable.
A burn rate of 10 means you are consuming budget 10× faster than allowed. At this rate, you will exhaust your entire monthly budget in 3 days (30 days ÷ 10). This is a serious incident.
Multi-Window Alerting
A single burn rate alert can miss two important failure modes:
1. Fast-burning disasters: A major outage that burns through budget in hours. You need to know immediately.
2. Slow-burning issues: A subtle bug that burns through budget over days. No single minute looks alarming, but the cumulative effect is severe.
The Google SRE Workbook recommends a multi-window, multi-burn-rate alerting strategy:
| Alert Severity | Short Window | Long Window | Burn Rate | Budget Consumed if Unresolved |
|---|---|---|---|---|
| Page (critical) | 5 min | 1 hour | > 14.4× | 2% in 1 hour |
| Page (high) | 30 min | 6 hours | > 6× | 5% in 6 hours |
| Ticket (medium) | 2 hours | 24 hours | > 3× | 10% in 24 hours |
| No alert | — | — | ≤ 1× | On track |
The two-window approach (short + long) reduces false positives: the alert only fires if the burn rate is elevated in both windows, filtering out brief spikes.
Interviewers at companies with mature SRE practices often ask: "How do you avoid alert fatigue with SLO-based alerting?" The answer is burn rate alerting with multi-window evaluation. Explain that alerting on instantaneous error rate generates noise from transient spikes, while burn rate alerting focuses on impact to the error budget. A 10-second spike at 50% error rate that consumes 0.001% of your monthly budget is not worth a page. A sustained 5% error rate that will exhaust your budget in 2 days absolutely is. The multi-window approach (e.g., 5-minute AND 1-hour windows both elevated) further reduces false positives.