Histograms and Quantiles: The Hard Math

Why Averages Lie

Imagine 1,000 requests arrive in one second. 990 complete in 10ms. 10 complete in 10 seconds. The average latency is about 109ms. Your dashboard shows "109ms average" and looks fine. But 1% of your users just waited 10 seconds. They are not fine.

This is why SRE teams track percentiles (also called quantiles):
- P50 (median): Half of requests are faster than this.
- P95: 95% of requests are faster than this. 5% are slower.
- P99: 99% of requests are faster than this. The "tail latency."
- P99.9: The worst 0.1% — critical for high-volume services where 0.1% still means thousands of users.

How Prometheus Histograms Work

When you instrument code with a Prometheus Histogram, it automatically creates three metric families:

1. _bucket — Counters for each configured latency bucket. le="0.1" counts requests completing in ≤100ms. le="0.5" counts requests completing in ≤500ms. And so on.
2. _count — Total number of observations (all requests).
3. _sum — Sum of all observed values (total latency across all requests).

The buckets are cumulative: if a request takes 80ms, it increments the le="0.1", le="0.5", le="1.0", and all higher buckets simultaneously.

histogram_quantile()

This function estimates a percentile by interpolating across the bucket boundaries. Computing p99 latency means calling histogram_quantile(0.99, ...) on the rate-of-buckets, aggregated by le.

Step-by-step breakdown:

1. rate(..._bucket[5m]) — For each bucket boundary (le value), compute how fast that bucket is filling up (observations per second). This converts cumulative counters into a rate distribution.
2. sum(...) by (le) — Aggregate across all pods. You MUST keep le — it is the label that identifies which bucket boundary each series represents. Without it, the function cannot reconstruct the distribution.
3. histogram_quantile(0.99, ...) — Given the rate-weighted bucket distribution, estimate the value below which 99% of observations fall.

Common mistake: Forgetting by (le) in the sum(). This collapses all bucket boundaries into a single series, destroying the distribution information. The result will be a nonsensical number.