Mitigation: Breaking the Loop

You cannot fix a metastable failure by waiting. You cannot fix it by adding capacity alone. You must actively break the feedback loop. Here are the primary tools.

1. Circuit Breakers (Stop the Retries)

A Circuit Breaker wraps calls to a downstream service. When the error rate exceeds a threshold, the circuit "opens" and immediately rejects requests without forwarding them to the overloaded backend. This removes load from the struggling service and gives it time to drain its queue and recover.

The circuit has three states:
- Closed: Normal operation. Requests pass through.
- Open: Failure threshold exceeded. Requests are rejected immediately with a fallback response.
- Half-Open: After a cooldown period, a small number of probe requests are allowed through. If they succeed, the circuit closes. If they fail, it opens again.

Libraries like Netflix Hystrix (now in maintenance) and Resilience4j implement this pattern for JVM services. Go's sony/gobreaker and Python's pybreaker are common alternatives.

2. Exponential Backoff with Jitter

If you must retry, do not retry immediately, and do not retry on a fixed schedule.

• Bad (fixed): Retry at T+1s, T+2s, T+3s — all clients synchronized.
• Bad (exponential, no jitter): Retry at T+1s, T+2s, T+4s — still synchronized across clients.
• Good (exponential + jitter): Retry at T+1.2s, T+4.5s, T+9.1s — each client randomized.

The jitter desynchronizes clients, spreading retries across time instead of concentrating them into synchronized waves. AWS's SDKs use this pattern by default. The formula is typically: sleep = min(cap, base * 2^attempt) * random(0.5, 1.0).

3. Load Shedding (The Nuclear Option)

If the request queue is full, drop requests. Do not queue them. Do not process them slowly. Return HTTP 503 immediately.

This is counterintuitive but correct. It is better to serve 50% of users perfectly and reject 50% with a clear error, than to serve 0% of users because the system is dead trying to process everything. Load shedding gives the system headroom to process the requests it does accept, which allows it to drain the backlog and eventually recover.

Shedding can be based on queue depth, CPU utilization, latency percentiles, or explicit capacity limits. Google's SRE practices formalize this as Utilization Signals — services shed load when their own utilization exceeds a safe threshold.

4. Request Coalescing (Singleflight)

For Cache Stampedes: if 1,000 concurrent requests ask for the same missing cache key, let one request go to the database. Make the other 999 wait for that single result, then fan it out to all waiters simultaneously.

In Go, the golang.org/x/sync/singleflight package implements this directly:

var g singleflight.Group

func getData(key string) (interface{}, error) {
    result, err, _ := g.Do(key, func() (interface{}, error) {
        // Only ONE goroutine executes this, even if 1000 call getData concurrently
        return fetchFromDatabase(key)
    })
    return result, err
}

The third return value (shared bool) tells you whether the result was shared from a concurrent call — useful for metrics.

5. Admission Control (Prevention Over Cure)

The most robust mitigation is preventing the system from entering the vulnerable state in the first place. Admission Control limits the number of concurrent requests the system will accept, regardless of how many arrive. Requests beyond the limit are rejected immediately at the edge — before they consume any backend resources.

This is different from load shedding in that it operates on concurrency (how many requests are in-flight simultaneously) rather than on queue depth or CPU. The token bucket and leaky bucket algorithms are standard implementations.