The Alert Storm
Consider what happens when a production database cluster loses its primary node unexpectedly.
Within seconds:
- The database monitoring system fires: "Primary DB node is unreachable."
- All 40 microservices that depend on the database begin failing their health checks.
- Each microservice's alert rule fires: "Service X is unhealthy."
- The load balancer detects upstream failures and fires its own alert.
- The SLO burn rate alert fires because error rates are spiking.
The on-call engineer's phone receives 43 simultaneous pages for a single root cause. This is called an alert storm, and it is one of the most disorienting experiences in on-call work. The engineer must triage 43 notifications to understand that they all point to one problem, while the clock is ticking.
Grouping: Consolidating Related Alerts
Modern alerting systems solve this with grouping (sometimes called deduplication). Instead of forwarding every alert the moment it fires, the system buffers incoming alerts for a short window (typically 30 seconds to 2 minutes) and groups alerts that share common labels.
In Prometheus Alertmanager, this is configured in the routing tree: a group_by list naming which labels define a group (e.g. alert name, cluster, service), a group_wait controlling how long to wait before sending the first notification for a new group, a group_interval controlling how long to wait before sending updates to an existing group, and a repeat_interval controlling how often to re-notify while the alert keeps firing.
With this configuration, all 43 alerts from the database incident — if they share the cluster=production label — are consolidated into a single notification: "43 alerts firing in production cluster, related to database connectivity." The engineer gets one page, not 43.
Key Point: Grouping introduces a deliberate delay before the first notification (group_wait). This is a trade-off: you reduce noise at the cost of a slightly slower first page. For critical alerts, keep group_wait short (15–30 seconds). For warning-level alerts, a longer wait (2–5 minutes) is acceptable and further reduces noise.