From Metric to Page
Understanding how an alert rule is structured helps you write better alerts and debug misfiring ones. In Prometheus, alert rules are written in PromQL (Prometheus Query Language) and live in rule files loaded by the Prometheus server.
A complete, production-quality alert rule, grouped under a named rule group, gives the alert a name (CheckoutHighErrorRate), a PromQL expression computing the ratio of 5xx responses to total requests for the checkout service and comparing it against a 1% threshold, a for duration requiring that condition to hold continuously before firing, routing/silencing labels (severity and owning team), and human-readable annotations — a summary, a templated description that interpolates the actual observed value, and a link to a runbook.
Breaking down each field:
| Field | Purpose |
|---|---|
alert |
The alert name. Used for grouping and routing. |
expr |
The PromQL expression. When this evaluates to true, the alert is "pending." |
for |
How long the condition must be true before the alert fires. Filters out momentary spikes. |
labels |
Key-value pairs used for routing and silencing. |
annotations |
Human-readable context. Not used for routing. Shown in notifications. |
The for Duration: Your Noise Filter
The for field is one of the most important and most misunderstood parts of an alert rule. Without it, a single anomalous data point — a one-second CPU spike, a single failed request — can trigger a page.
With for: 5m, the condition must be continuously true for 5 minutes before the alert transitions from pending to firing and a notification is sent. This eliminates the vast majority of transient false positives.
The trade-off: a longer for duration means slower detection. For critical user-facing alerts, for: 2m to for: 5m is a common range. For capacity planning warnings, for: 30m or longer is appropriate.
Interviewers sometimes ask: "Your alert has for: 5m. The outage started at 14:00. When did the alert fire?" The answer is 14:05 at the earliest — only if the condition was continuously true from 14:00 onward. If the condition briefly dipped below the threshold at 14:03, the pending timer resets. This distinction matters when doing post-incident timeline analysis.