The Problem with "Everyone Gets Everything"
A disk filling up over three days is not the same urgency as the checkout flow returning 500 errors. Routing both to the same PagerDuty escalation policy — waking someone up at 3 AM for the disk warning — is a fast path to alert fatigue and engineer burnout.
Routing is the process of directing each alert to the correct destination, at the correct urgency level, based on the alert's labels.
How Routing Works in Practice
The routing system reads labels attached to each alert and matches them against a routing tree — a hierarchy of rules evaluated top to bottom. The first matching rule wins.
A realistic Alertmanager routing configuration sets a low-priority Slack channel as the catch-all receiver, then defines nested routes evaluated in order: critical alerts labeled team=checkout page the checkout on-call via PagerDuty, critical alerts labeled team=platform page the platform on-call via a separate PagerDuty service, and anything else at warning severity posts to a Slack channel with no page.
The labels on the alert determine which route it takes:
| Alert Labels | Destination | Urgency |
|---|---|---|
severity=critical, team=checkout |
PagerDuty → checkout on-call | Immediate page |
severity=critical, team=platform |
PagerDuty → platform on-call | Immediate page |
severity=warning |
Slack #alerts-warnings |
Business hours |
severity=info |
Default Slack channel | Informational |
Ownership Labels Are Critical
The routing system is only as good as the labels on your alerts. Every alert rule should declare which team owns it. This prevents the "who owns this?" confusion during an incident and ensures the right expert is paged — not just whoever is on-call for the general queue.
A common interview question is: "How do you handle an alert that could belong to multiple teams?" The best answer is to define a primary owner in the alert label and use a secondary escalation path. For example, a shared infrastructure alert might page the platform team first, with an escalation to the application team if unacknowledged after 15 minutes. This avoids the bystander effect where everyone assumes someone else will respond.