Where to Reject Traffic
Load shedding can happen at multiple layers of your stack. The right choice depends on what information is available at each layer.
At the Load Balancer (L4/L7): Tools like NGINX, HAProxy, and Envoy can reject connections or return 503 responses based on upstream health signals. This is efficient because the rejection happens before the request consumes application resources. The limitation is that the load balancer may not have fine-grained knowledge of individual server health or request priority.
At the Application (Middleware): The most accurate location. The application knows its own CPU load, queue depth, and request priority. This is where priority-based shedding is implemented. The cost is that the request has already consumed a TCP connection and been parsed by the HTTP server before being rejected.
At the Client (Rate Limiting + Circuit Breakers): Clients can be designed to stop sending requests when they detect that the server is overloaded. This is the circuit breaker pattern — a complementary technique that prevents load from reaching the server in the first place.
CoDEL: Controlled Delay
A widely used algorithm for queue management is CoDEL (Controlled Delay), originally developed for network packet scheduling and later adapted for application-level queues.
CoDEL monitors the time a request spends waiting in the queue, not just the length of the queue. The key insight is that queue length is a lagging indicator — by the time the queue is long, damage is already done. Queue wait time is a leading indicator of client-perceived latency.
The CoDEL rule: If the request at the front of the queue has been waiting longer than a target interval (commonly 5–100ms depending on your SLA), drop it.
Why this works: If a request waited 200ms in the queue before being processed, the client's timeout may have already fired. Processing the request now wastes CPU on work whose result will be discarded. Dropping it immediately frees that CPU for fresh requests that clients are still waiting for.
CPU-Based Admission Control
The simplest and most common form of admission control uses CPU utilization as the signal:
if (currentCPU > 0.90) {
reject(request, 503)
}
This creates a negative feedback loop that stabilizes the system. As traffic rises, rejections increase, which reduces the number of requests being processed, which lowers CPU, which allows more requests through. The system naturally hovers near the configured threshold.
The threshold should be set below the point of collapse — typically 80–90% — to leave headroom for garbage collection, health checks, and other background work that cannot be shed.
A common interview question is: "Why is CPU-based load shedding a negative feedback loop, and why does that matter?" The answer: when CPU rises above the threshold, more requests are rejected, which reduces active work, which lowers CPU. This is self-correcting. Without load shedding, the system has a positive feedback loop: high CPU → high latency → client retries → more load → higher CPU. Negative feedback loops stabilize systems; positive feedback loops destroy them.
Token Bucket and Leaky Bucket
Two classical algorithms for rate limiting that complement load shedding:
- Token Bucket: A bucket holds up to N tokens. Each request consumes one token. Tokens are replenished at a fixed rate (e.g., 1000/second). If the bucket is empty, the request is rejected. This allows short bursts up to N requests while enforcing a long-term average rate.
- Leaky Bucket: Requests enter a fixed-size queue and are processed at a constant rate. Excess requests that overflow the queue are dropped. This smooths bursty traffic into a steady stream but cannot absorb bursts the way a token bucket can.
Load shedding and rate limiting are complementary: rate limiting protects against sustained overload from a single source, while load shedding protects against aggregate overload from all sources combined.