The Silent Failure
An incident begins when the system breaks — not when someone notices. Imagine a bad code deployment at 10:00 AM that introduces a memory leak. By 10:15 AM, servers are crashing and users are seeing errors. But if your monitoring doesn't catch it until 10:30 AM, you've had a 30-minute silent failure window. Every minute of that window is a minute of customer pain you didn't know about.
Detection happens through one of three mechanisms:
- Automated Monitoring: A tool like Datadog, Prometheus, or Grafana detects an anomaly — high error rate, latency spike, disk full — and fires an alert automatically. This is the gold standard.
- Customer Reports: Support tickets or social media posts start appearing. This is the worst-case detection path: it means your monitoring missed something your users noticed first.
- Manual Observation: An engineer happens to check a dashboard and spots something wrong. This is unreliable and doesn't scale.
Metric: Time to Detect (TTD)
TTD is the elapsed time between the actual start of the issue and the moment the organization first becomes aware of it.
TTD = Time of First Awareness − Time of Incident Start
- Goal: Minimize TTD by improving monitoring coverage, reducing alert thresholds, and ensuring every critical user journey has a synthetic monitor (a robot that continuously tests the flow end-to-end).
- Warning sign: If your TTD is consistently measured in hours, or if customers are reporting issues before your alerts fire, your observability stack has gaps. A useful rule of thumb: your monitoring should detect a problem before 1% of your users do.
The most dangerous incidents are the ones with long TTDs — the system degrades slowly and silently until the damage is severe. Gradual memory leaks, slow disk fill, and creeping database table bloat are classic examples.