An alert is a request. It is asking a human being to stop sleeping, stop their dinner, or stop their weekend — and pay attention. That request should only be made when a human is genuinely needed and genuinely able to do something.
The Three Questions for Every Alert
Before any alert is allowed to page an on-call engineer, it should pass three tests:
-
Is it actionable? Is there a specific action the engineer can take in response to this alert? If the answer is "check and see if it clears," the alert should not page. It can log, it can create a ticket, but it should not wake someone up.
-
Is it urgent? Does this condition require a human response within minutes, or can it wait until business hours? A disk at 75% capacity is not a 3 AM problem. A payment processor returning 500s is.
-
Is it symptomatic? Alerts should fire on customer-visible symptoms, not internal system states. "Error rate > 1% on checkout API" is a symptom. "CPU > 80%" is a system state that may or may not be causing any customer impact. Prefer symptom-based alerts.
Alert Tuning Patterns
Threshold Adjustment: If an alert fires frequently but rarely requires action, the threshold is wrong. Raise it until the alert only fires when action is genuinely needed.
Duration Windows: Instead of alerting on "CPU > 80%," alert on "CPU > 80% sustained for 15 minutes." Transient spikes are normal. Sustained degradation is a problem. Adding a duration window eliminates most flapping alerts.
Automated Remediation: If an alert has a known, scripted fix (restart a service, clear a cache, run a cleanup job), automate the fix. Only page a human if the automated fix fails or if the condition recurs within a defined window.
Alert Consolidation: If five alerts fire simultaneously during an incident, the engineer receives five pages. This is noise. Use alert grouping or inhibition rules (available in tools like Alertmanager) to suppress downstream alerts when a root-cause alert is already firing.
Key Point: The goal of alert tuning is not to reduce the number of alerts you see — it is to increase the percentage of alerts that are worth seeing. A system that pages 20 times per week with 100% actionability is healthier than one that pages 5 times per week with 40% actionability, because the latter is training engineers to ignore the pager.