Symptom-Based vs. Cause-Based Alerting

The Wrong Way: Cause-Based Alerting

Early monitoring systems were built around infrastructure health. The mental model was: if the machines are healthy, the application must be healthy. So alerts were wired to machine-level signals:

• "Alert me if CPU usage exceeds 90%."
• "Alert me if Server A stops responding to pings."
• "Alert me if disk usage exceeds 80%."

This approach has two fundamental problems.

Problem 1 — False positives: A server can run at 95% CPU during a scheduled batch job and serve every user request perfectly. The alert fires, the on-call engineer wakes up, investigates, finds nothing wrong, and goes back to sleep. Repeat this enough times and the engineer stops investigating.

Problem 2 — Missed failures: A Kubernetes cluster can lose an entire node and automatically reschedule every workload onto healthy nodes within seconds. Users experience nothing. But your cause-based alert fires anyway, because a server went down. Meanwhile, a subtle memory leak is slowly degrading response times for 20% of users — and no alert fires because no single machine metric crossed a threshold.

The Right Way: Symptom-Based Alerting

Modern SRE practice, codified in Google's Site Reliability Engineering book, says: alert on symptoms, debug with causes.

A symptom is something a user directly experiences:

• "The homepage error rate is above 1%."
• "Checkout latency at the 99th percentile exceeds 2 seconds."
• "The image upload success rate has dropped below 99%."

These signals are always meaningful. If users are seeing errors, something is broken — full stop. You don't need to guess whether the metric matters.

The cause-level metrics (CPU, memory, disk I/O, network throughput) are still collected and still valuable. You use them on dashboards to diagnose the problem after the symptom alert fires. The workflow becomes:

1. Symptom alert fires: "Error rate on /checkout is 4%."
2. Engineer opens the dashboard and looks at cause metrics to find the root cause.
3. Engineer fixes the root cause.

Key Point: The framework that makes symptom-based alerting rigorous is the Four Golden Signals from Google SRE: Latency (how long requests take), Traffic (how much demand the system is under), Errors (the rate of failed requests), and Saturation (how full the system is). Alerting on latency and errors covers the vast majority of user-visible symptoms. Traffic and saturation are primarily diagnostic signals.