The Formula
Kingman's approximation (also called the VUT equation) gives the average time a request spends waiting in the queue (not including service time) for a single-server system:
$$W_q \approx \left( \frac{\rho}{1 - \rho} \right) \times \left( \frac{C_a^2 + C_s^2}{2} \right) \times \bar{S}$$
Breaking this down:
- $\frac{\rho}{1 - \rho}$: The utilization factor — this is the hockey stick term. It grows without bound as $\rho \to 1$.
- $\frac{C_a^2 + C_s^2}{2}$: The variability factor — $C_a$ is the coefficient of variation of inter-arrival times; $C_s$ is the coefficient of variation of service times. Higher variability means longer queues even at the same utilization.
- $\bar{S}$: The mean service time per request.
The Hockey Stick in Practice
Focus on the utilization factor alone. Here's what happens as you push utilization higher:
| Utilization ($\rho$) | $\frac{\rho}{1-\rho}$ | Relative Wait |
|---|---|---|
| 50% | 1.0 | 1× baseline |
| 70% | 2.3 | 2.3× baseline |
| 80% | 4.0 | 4× baseline |
| 90% | 9.0 | 9× baseline |
| 95% | 19.0 | 19× baseline |
| 99% | 99.0 | 99× baseline |
Going from 80% to 90% utilization doesn't make your system 10% slower — it makes it more than twice as slow. This is the hockey stick: the curve is nearly flat up to ~70%, then bends sharply upward and approaches a vertical asymptote at 100%.
The Variability Insight
The variability factor is often overlooked but critically important. Even at moderate utilization, high variability in request processing times causes disproportionately long queues. This is why:
- A single slow database query (high $C_s$) can cause a queue to build even when average CPU is low.
- Bursty traffic patterns (high $C_a$) are more damaging than smooth traffic at the same average rate.
- Reducing p99 latency outliers (tail latency) is often more impactful than reducing average latency.
A classic SRE interview question: "Your service is at 70% CPU and latency is fine. You optimize the code and reduce average CPU to 50%. But latency gets worse. How?" The answer lies in Kingman's variability term. If your optimization introduced more variance in processing time (e.g., a cache that sometimes hits and sometimes misses, creating bimodal service times), $C_s$ increases. Even with lower average utilization, higher variability drives up queue wait time. This is a counterintuitive result that separates senior engineers from mid-level ones.
The Practical Rule
Design for 70–80% peak utilization. This gives you enough headroom that normal traffic variance won't push you into the danger zone, while keeping hardware costs reasonable. The remaining 20–30% is not waste — it is your stability margin.