Not All Requests Are Equal
If you must drop 20% of your traffic, which 20% should it be? Dropping requests randomly is wasteful and unfair. You might drop a POST /checkout request (high business value, user actively waiting) while serving a GET /recommendations request (low value, result will be cached anyway).
The solution is to classify every request into a priority tier before it enters the processing pipeline, and shed lower-priority tiers first.
Implementing Tiers
Priority is typically assigned at the edge — by the API gateway, a reverse proxy, or application middleware — based on URL path, HTTP method, or a custom header like X-Request-Priority.
A common three-tier model:
| Tier | Label | Examples | Shed First? |
|---|---|---|---|
| P0 | Critical | POST /payment, POST /login, health checks |
Never (or last) |
| P1 | Important | GET /product, POST /cart, GET /search |
Second |
| P2 | Background | Analytics events, recommendation generation, email digest triggers | First |
The Shedding Logic
When the system detects overload (for example, CPU > 80% or queue depth > 50 requests):
- Level 1 Shedding: Immediately reject all P2 traffic with
503 Service Unavailable. Return aRetry-Afterheader so well-behaved clients back off. - Level 2 Shedding: If still overloaded, reject P1 traffic. Users browsing the catalog will see errors, but existing sessions are protected.
- Level 3 Shedding: Process only P0. Even during a catastrophic spike, users can complete payments and log in.
This tiered approach ensures that even during a severe incident, the highest-value user actions succeed. Losing a recommendation widget is acceptable. Losing a payment is not.
Communicating Rejection to Clients
A shed request should always return:
- HTTP 503 (Service Unavailable) — not 500, which implies a bug.
- Retry-After: 30 header — tells the client when to try again.
- X-Shed-Reason: load header (optional) — helps with debugging and observability.
Clients that respect Retry-After will back off, reducing load further. This is why well-designed client retry logic with exponential backoff and jitter is the essential complement to server-side load shedding.