The Problem: Alerts That Are Correct But Unhelpful
Sometimes an alert fires and is technically accurate, but acting on it right now is impossible or counterproductive. Two mechanisms handle this: silences for planned situations, and inhibition for automatic suppression during cascading failures.
Silences: Planned Muting
A silence is a manually created rule that mutes matching alerts for a defined time window. You create it before or during a known event.
Common use cases:
- Planned maintenance: "We are taking the payment service offline for a schema migration at 2 AM. Mute all team=payments alerts for 90 minutes."
- Active incident: "We know the system is broken. We are fixing it. Stop paging us every 5 minutes with the same alert."
- Canary deployments: "We are intentionally sending 5% of traffic to an experimental build. Mute error rate alerts for version=canary for 30 minutes."
In Alertmanager, a silence is created via the UI or API and matches alerts using label selectors. It expires automatically at the configured end time, so you cannot accidentally leave a silence running forever without noticing.
Inhibition: Automatic Suppression During Cascading Failures
Inhibition is a rule that says: "If alert A is firing, automatically suppress alert B." It handles the cascading failure scenario without requiring human intervention.
A concrete example: an inhibit rule whose source_match is a critical DatacenterDown alert, whose target_match is any warning-severity alert, and whose equal list requires the datacenter label to match between the two. This rule reads: "If a DatacenterDown critical alert is firing for datacenter=us-east-1, suppress all warning-level alerts that also carry datacenter=us-east-1."
Without inhibition: A datacenter outage generates one critical alert plus potentially thousands of warning alerts for every server, service, and endpoint inside that datacenter.
With inhibition: The engineer receives one critical alert. The thousands of downstream warnings are suppressed automatically because they are all consequences of the same root cause.
Key Point: Silences and inhibition serve different purposes and should not be confused. Silences are a human decision applied to a known situation. Inhibition is a pre-configured automatic rule that fires based on alert state. Over-relying on silences is a warning sign — if you are silencing the same alert repeatedly, the alert itself is probably wrong and should be deleted or redesigned.