The "Thundering Herd" Problem
Imagine a database gets briefly overloaded and starts rejecting connections for two seconds. Your application has 1,000 active users. All 1,000 requests fail at the same instant. If your code retries immediately, you have just sent another 1,000 requests to a database that is already struggling to recover. You have turned a two-second blip into a sustained outage — by your own hand. This is called a thundering herd: a surge of retry traffic that prevents a service from ever recovering.
Solution 1: Exponential Backoff
The first fix is to wait longer between each retry attempt, doubling the wait time each time:
| Attempt | Wait Before Retry |
|---|---|
| 1 | 1 second |
| 2 | 2 seconds |
| 3 | 4 seconds |
| 4 | 8 seconds |
| 5 | 16 seconds (or a configured cap) |
This gives the failing service progressively more breathing room. But there is a subtle flaw.
The Hidden Problem: Synchronized Retries
If 1,000 users all fail at exactly 12:00:00.000 and all wait exactly 1 second, they all retry at exactly 12:00:01.000. The thundering herd has not been eliminated — it has been delayed by one second. The traffic spikes are still perfectly synchronized.
Solution 2: Adding Jitter
Jitter means adding randomness to the wait time. The most effective variant is called Full Jitter:
sleep = random_between(0, min(cap, base * 2 ^ attempt))
Instead of every client waiting exactly 2 seconds, one client waits 0.3 seconds, another waits 1.7 seconds, another waits 0.9 seconds. The retry traffic is now spread across the entire window, smoothing the spike into a gentle, manageable trickle that the recovering service can handle.
Choosing a Cap
Without a maximum cap, exponential backoff can produce absurdly long waits (attempt 10 = 1,024 seconds). Always set a max_delay (commonly 30–60 seconds for most APIs). The formula becomes:
sleep = random_between(0, min(max_delay, base * 2 ^ attempt))
Interviewers frequently ask: "Why not just use exponential backoff without jitter?" The answer is that pure exponential backoff only solves the rate problem — it does not solve the synchronization problem. If all clients started failing at the same time, they will retry at the same time regardless of how long the backoff is. Jitter is what breaks the synchronization. AWS published a blog post on this exact topic and recommends Full Jitter as the default strategy for high-scale systems.
Key Point — When Not to Retry
Retries are only appropriate for transient failures — errors that might resolve on their own (network timeouts, 503 Service Unavailable, 429 Too Many Requests). Never retry on permanent failures like 400 Bad Request (your request is malformed — retrying will always fail) or 404 Not Found (the resource does not exist). A production retry implementation must inspect the error type before deciding to retry.