Why Static Thresholds Fail
You cannot set a static alert for "Alert me if Orders < 100/min."
- At 2 PM on a Monday: 100 orders/min might be a disaster (normally 800).
- At 3 AM on a Sunday: 100 orders/min might be impossibly high (normally 15).
- On Black Friday: Every threshold you've ever set is wrong.
Business metrics are cyclical. They follow daily patterns (lunch peaks, evening lulls), weekly patterns (weekday vs. weekend), and seasonal patterns (holiday spikes, end-of-quarter surges). This property is called seasonality — the predictable, repeating variation in a time series.
Static thresholds applied to seasonal data produce two failure modes: false positives (alerts at 3 AM when low volume is normal) and false negatives (no alert during a holiday when a 20% drop is masked by the overall high volume).
Dynamic Alerting with Anomaly Detection
To monitor business metrics reliably, you need Anomaly Detection — algorithms that calculate an "Expected Range" based on historical patterns rather than a fixed number.
Common approaches:
| Algorithm | How It Works | Best For |
|---|---|---|
| Holt-Winters (Triple Exponential Smoothing) | Models level, trend, and seasonality separately | Strong daily/weekly cycles |
| SARIMA | Statistical model for seasonal time series | Complex multi-season patterns |
| Proprietary ML (Datadog, Dynatrace) | Trained on your specific metric history | Ease of use, managed service |
| Percentile Bands | Alert when value falls outside historical Nth percentile for that time window | Simple, interpretable |
The Logic in Practice:
"Based on the last 4 Mondays, we expect orders between 520 and 640 at 9:00 AM. Current value is 310. This is 4.2 standard deviations below the expected baseline. Alert."
This catches subtle degradations — a 15% drop in orders — that would never trigger a static threshold but represent a real business problem.
Configuring Anomaly Detection in Datadog (Example)
A Datadog monitor using its anomalies() function wraps the completed-orders rate metric, names an algorithm variant (basic, agile, or robust), a deviation threshold (e.g. 2 standard deviations), a direction (alerting only on drops, not spikes), an evaluation window, and a seasonality setting.
The seasonality='weekly' parameter tells the algorithm to compare current values against the same time window from previous weeks — automatically accounting for the fact that Tuesday at 10 AM looks different from Saturday at 10 AM.
Tuning Sensitivity
Anomaly detection requires calibration. Too sensitive and you get alert fatigue. Too loose and you miss real incidents. A practical starting point:
- 2σ deviation for business-critical metrics (checkout, payments)
- 3σ deviation for supporting metrics (search, recommendations)
- Minimum duration of 5–10 minutes before alerting, to filter transient spikes