Sometimes, a simple average isn't enough. If 99 requests take 10ms and one takes 10 seconds, the average latency looks acceptable — but one user is having a terrible experience. To capture this reality, we need to measure the distribution of values, not just their average.
3. Histogram: The Bucket Counter
A Histogram counts observations (like request durations or response sizes) and places them into configurable "buckets."
- How it works: You define bucket boundaries like "less than 0.1s", "less than 0.5s", "less than 1s", "less than 5s". Every time a request finishes, the system increments the counter for every bucket whose boundary the duration falls under. A request taking 0.3s increments the
le="0.5",le="1", andle="5"buckets — but notle="0.1". - What it exposes: Three things — the bucket counters (
_bucket), the sum of all observed values (_sum), and the total count of observations (_count). - Use Case: Measuring request latency or response sizes to calculate percentiles (e.g., "99% of requests finish in under 200ms") on the monitoring server side using Prometheus's
histogram_quantile()function against the bucket data. - Pros: Very cheap for the application to collect. Can be aggregated across many instances easily — you can combine histograms from 50 application pods into one fleet-wide percentile.
4. Summary: The Client-Side Calculator
A Summary works similarly to a histogram but calculates quantiles (percentiles) on the application client side before sending the data to the monitoring system.
- Use Case: When you need very accurate, pre-calculated percentiles from a single instance and the exact bucket boundaries are hard to predict in advance.
- What it exposes: Pre-calculated quantiles (e.g.,
{quantile="0.99"}), the sum, and the count. - Cons: More expensive for the application to calculate. The data cannot be aggregated across multiple application instances — you cannot mathematically average two pre-calculated 99th percentiles and get a meaningful result. For most modern cloud-native systems with multiple replicas, Histograms are strongly preferred over Summaries.
Interviewers frequently ask: "What's the difference between a Histogram and a Summary?" The key answer is where the percentile is calculated and whether the data can be aggregated. Histograms calculate percentiles at query time on the server, so you can combine data from 50 pods. Summaries calculate percentiles on the client, so you cannot meaningfully aggregate them across instances. In a Kubernetes environment with horizontal scaling, always reach for a Histogram.