The Death Spiral
To understand why load shedding is necessary, you need to understand the difference between throughput and goodput.
- Throughput is the total number of requests arriving at your server per second — including requests that will fail, time out, or be retried.
- Goodput is the number of requests that are successfully completed and returned to a waiting client — the useful work your server actually does.
In a healthy system, as throughput increases, goodput increases linearly. But once you cross the system's saturation point — the load at which all CPU, threads, and I/O are fully utilized — the relationship breaks down catastrophically.
The Three Phases of Overload
Phase 1 — Saturation: Throughput keeps rising, but goodput flattens. The CPU is spending increasing time on context switching between threads rather than doing application work. Response latency begins to climb.
Phase 2 — The Cliff: As queues fill, latency spikes. Clients start timing out and issuing retries. Now the server is processing duplicate requests — the original and the retry — for work that will only benefit one client. Effective goodput drops even as CPU stays pegged at 100%.
Phase 3 — Collapse: Goodput falls to near zero. The server is running at 100% CPU but producing almost no useful output. Every thread is busy processing stale, timed-out, or duplicate requests. The system is in a metastable failure state — it cannot recover on its own because the load that caused the collapse is now amplified by retries.
Load shedding prevents Phase 3 by artificially capping throughput at the saturation point. By rejecting excess requests with an immediate 503, the server keeps its queues short, its latency low, and its goodput high.
Interviewers at senior SRE and platform engineering levels frequently ask: "What is the difference between throughput and goodput, and why does goodput collapse under overload?" The answer they want is the retry amplification loop: clients time out and retry, which increases load, which increases latency, which causes more timeouts and retries. This is a positive feedback loop that prevents self-recovery. Load shedding breaks the loop by giving clients a fast, definitive failure signal, which stops retries from amplifying the problem.