Defining Utilization ($\rho$)
Utilization ($\rho$, "rho") is the fraction of time a server is busy processing work. It is defined as:
$$\rho = \frac{\lambda}{\mu}$$
- $\lambda$ (Lambda): Arrival rate (requests per second arriving at the server).
- $\mu$ (Mu): Service rate (maximum requests per second the server can handle).
If a server can handle a maximum of 1,000 requests/sec ($\mu = 1000$) and is currently receiving 500 requests/sec ($\lambda = 500$):
$$\rho = \frac{500}{1000} = 0.5 \quad \text{(50% utilized)}$$
For a system to be stable — meaning the queue doesn't grow without bound — you need $\rho < 1$. When $\rho \geq 1$, arrivals permanently outpace service, and the queue grows to infinity. This is the mathematical definition of a system that is overloaded.
The Utilization Trap
A common management instinct is to push utilization toward 100%: "We're paying for this CPU, let's use every cycle." Queueing theory proves this is a trap. As utilization approaches 1, wait times don't grow linearly — they grow explosively. We'll quantify exactly how explosive in the next section.
The practical implication: you must leave headroom. A system running at 95% utilization on average has almost no capacity to absorb traffic spikes. A 10% burst in arrivals pushes it past $\rho = 1$, queues explode, and latency skyrockets. This is why SRE teams set autoscaling triggers at 60–70% CPU, not 90%.