Curing Alert Fatigue
Alert fatigue is one of the most dangerous failure modes in on-call engineering. When an on-call engineer receives too many low-signal alerts, they begin to ignore them — and eventually miss the one that matters. Studies of major outages consistently find that the warning signals were present but buried in noise.
Think of it like a car alarm in a parking lot. The first time you hear it, you look up. After the tenth false alarm that week, you tune it out completely. The night someone actually breaks in, nobody reacts.
The Root Cause of Alert Storms:
In a microservices architecture, a single failure propagates. If your database becomes slow, every service that depends on it degrades. You might receive simultaneous alarms for:
RDS-CPU-High— the database is under loadAPI-Latency-High— the API is slow waiting for the databaseSQS-Queue-Depth-High— jobs are backing up because the API is slowLambda-Error-Rate-High— Lambda functions are timing out waiting for the API
These are four alarms for one root cause. Your on-call engineer gets four pages, has to deduplicate them mentally, and starts their incident response already stressed and confused.
Composite Alarms: Boolean Logic for Alarms
A Composite Alarm does not monitor a metric directly. Instead, it monitors the state of other alarms and evaluates a boolean expression. The composite alarm only enters ALARM state when its expression evaluates to true.
Supported operators: AND, OR, NOT, and parentheses for grouping.
Example 1: Correlated Incident Detection
Only page if both the database CPU is high AND the API latency is elevated — confirming the database is the cause of the API degradation. The alarm rule expression is ALARM("RDS-CPU-High") AND ALARM("API-Latency-High"); creating the composite alarm is a single call naming that rule and the SNS topic to notify.
Example 2: Maintenance Window Suppression
Suppress the CPU alarm during nightly backups using a "suppressor alarm" — a separate alarm that enters ALARM state when the backup job is running. The rule ALARM("CPU-High") AND NOT ALARM("BackupJob-Active") means the composite alarm only fires if CPU is high AND the backup job is NOT active.
The BackupJob-Active alarm is a separate alarm that you put into ALARM state programmatically at the start of your backup window (using set-alarm-state) and reset afterward. This gives you a clean, auditable suppression mechanism without modifying the original CPU alarm.
Composite Alarm Behavior:
- Composite alarms can reference other composite alarms, enabling hierarchical alarm trees.
- They do not support metric math or direct metric evaluation — they only evaluate the state of child alarms.
- Child alarms still evaluate independently and can still trigger their own actions. The composite alarm adds an additional layer of logic on top.
- There is no additional charge for composite alarms beyond the standard alarm pricing ($0.50/alarm/month in most regions).
Interviewers frequently ask: "How would you design an alerting system that reduces false positives without missing real incidents?" The answer should mention Composite Alarms with correlated signals (AND logic for confirmation) and suppressor alarms (NOT logic for known maintenance). Contrast this with simply raising alarm thresholds — which reduces false positives but also reduces sensitivity to real incidents. Composite Alarms let you keep individual alarms sensitive while requiring corroboration before paging.