The "Thundering Herd" Problem
Imagine a critical shared service — say, an authentication service — goes down for 30 seconds. Thousands of dependent clients start failing simultaneously. If they all immediately retry every second, the moment the authentication service tries to restart, it gets hit with thousands of simultaneous requests. This massive, synchronized spike of traffic will immediately crash it again before it has a chance to warm up. This is called the Thundering Herd problem, and naive retry logic is one of its most common causes.
Exponential Backoff
To solve this, we don't retry immediately. We wait. If it fails again, we wait longer. The wait time grows exponentially with each failed attempt:
- Attempt 1 fails → Wait 1 second, then retry.
- Attempt 2 fails → Wait 2 seconds, then retry.
- Attempt 3 fails → Wait 4 seconds, then retry.
- Attempt 4 fails → Wait 8 seconds, then retry.
- After N attempts → Give up and return an error.
This gives the failing service breathing room to recover. It also caps the total number of retries so a client does not wait forever. A common pattern is to set a maximum backoff (e.g., never wait more than 32 seconds) and a maximum retry count (e.g., no more than 5 attempts).
Adding Jitter
Even with exponential backoff, if 1,000 clients all started their first attempt at the exact same moment (because they all received the same event or the service went down at the same instant), their subsequent retries will still happen in synchronized waves. All 1,000 clients wait 1 second, then all 1,000 retry simultaneously. All 1,000 wait 2 seconds, then all 1,000 retry simultaneously again.
Jitter adds a small amount of randomness to the wait time. Instead of waiting exactly 2 seconds, each client waits somewhere between 1 and 3 seconds (or between 0 and 2 seconds — there are several jitter strategies). This desynchronizes the clients, spreading their retries across time and smoothing the traffic spike into a manageable ramp.
The most effective strategy, validated by AWS in their architecture blog, is "Full Jitter": sleep = random_between(0, min(cap, base * 2^attempt)). This produces the most even distribution of retry attempts across time.
Key Point: Jitter is not optional in high-scale systems. Without it, exponential backoff alone can still produce synchronized retry storms that overwhelm a recovering service. The AWS SDK, Google Cloud client libraries, and most production-grade HTTP clients implement full jitter by default — but if you are writing a custom retry loop, you must add it explicitly.