The Titanic Problem
Imagine your web service is a ship. Autoscaling is like calling for backup ships to help carry the passengers. It works, but it takes time — often 3 to 10 minutes — for those ships to arrive.
If a massive wave of traffic hits you right now, your ship will sink before help arrives. Your CPU hits 100%, memory fills up, and the server crashes. When it reboots, the traffic is still there, so it crashes again immediately. This is called a crash loop, and it is one of the most dangerous failure modes in distributed systems.
Load Shedding
Load Shedding is the act of deliberately throwing cargo overboard to keep the ship afloat.
It is the conscious decision to reject a percentage of incoming requests immediately so that the remaining requests can be served successfully. The core insight is counterintuitive: it is better to serve 50% of users perfectly than to attempt to serve 100% of users and deliver errors and timeouts to all of them. A 503 response returned in 5ms costs almost nothing. A request that hangs for 30 seconds before timing out consumes threads, memory, and CPU the entire time.
Load shedding is not a failure — it is a deliberate, engineered safety valve. Systems that implement it gracefully are far more resilient than systems that try to absorb every request until they collapse.
Why Not Just Queue Everything?
A common instinct is to buffer excess traffic in a queue and process it when capacity frees up. This fails for two reasons. First, queues consume memory; a large enough queue will exhaust heap space and crash the process. Second, most client requests have an implicit or explicit timeout. A request that sits in a queue for 10 seconds before being processed will be returned to a client that gave up 9 seconds ago — wasting CPU on work whose result will never be used.