Why Simple Division Fails
If a server can handle 100 requests per second, and you expect 400 requests per second, you need 4 servers — right?
This math is seductive but wrong. It assumes requests arrive in a perfectly uniform stream: one every 10 milliseconds, evenly spaced, like clockwork. Real traffic is nothing like this. Users arrive in bursts — a marketing email goes out, everyone clicks the link at the same moment. A cron job fires on every server simultaneously. A retry storm hits after a brief outage. Traffic is stochastic (random and bursty), not uniform.
When traffic is bursty and your servers are running at 100% of their theoretical capacity, any burst — even a small one — causes requests to queue. Queued requests wait. Waiting requests time out. Timed-out requests get retried. Retries add more load. The system collapses.
Erlang C: From Telephone Operators to Microservices
The Erlang C formula was developed in the early 20th century by Danish mathematician Agner Krarup Erlang to solve a practical problem: how many telephone operators does an exchange need so that callers don't get a busy signal? The math translates directly to modern distributed systems.
Erlang C calculates the probability that an arriving request will have to wait in a queue given:
- λ (lambda): The arrival rate of requests (requests per second)
- μ (mu): The service rate of a single server (requests per second it can handle)
- c: The number of servers
The key output is P(wait) — the probability that a new request must queue rather than being served immediately. As utilization (λ / (c × μ)) approaches 1.0 (100%), P(wait) approaches 1.0 (every request queues), and average wait time approaches infinity.
The Hockey Stick Effect
The most important practical lesson from Erlang C is that the relationship between utilization and wait time is non-linear. It looks like a hockey stick:
- At 50% utilization: Queuing probability is near zero. Requests are served almost immediately.
- At 70% utilization: Queuing probability is low. Occasional brief waits.
- At 80% utilization: Queuing probability rises noticeably. Latency begins to climb.
- At 90% utilization: Queuing probability is high. Latency is significantly elevated.
- At 95%+ utilization: The system is effectively saturated. Latency explodes. Small additional bursts cause cascading failures.
Applying This in Practice
The practical rule derived from Erlang C is: never plan to run servers above 70% utilization under normal load. The 30% headroom absorbs random traffic variance without causing queuing.
The capacity formula becomes:
Required capacity = Peak expected load / 0.70
Number of servers = Required capacity / Capacity per server (rounded up)
For a payment service expecting 400 TPS peak, where each server handles 100 TPS:
Required capacity = 400 / 0.70 = 571 TPS
Servers needed = 571 / 100 = 5.71 → 6 servers
The naive answer was 4 servers. The correct answer is 6 servers — 50% more than intuition suggested. This is not over-engineering; it is the minimum safe configuration.
A common SRE interview question is: "Your service has 10 servers each handling 1,000 RPS. Traffic is 7,000 RPS. Are you okay?" The trap is that 7,000 / 10,000 = 70% utilization sounds fine. The correct answer is: "It depends on traffic distribution. If traffic is bursty, 70% average utilization can mean frequent spikes to 90%+ on individual servers, causing latency degradation. I'd want to see the P99 latency trend and per-server utilization histograms, not just the average." This demonstrates you understand the difference between average and peak utilization.
Key Point: The Erlang C model assumes requests are independent and randomly distributed (a Poisson process). Real-world traffic often has correlations — retry storms, synchronized cron jobs, viral events — that make the actual queuing behavior worse than the formula predicts. Treat the 70% utilization target as a floor, not a ceiling. For latency-sensitive services (payments, authentication), targeting 50-60% utilization is prudent.