Everyone Rushing the Door
Imagine a large concert hall. The doors open, and 10,000 people try to squeeze through one small entrance at the exact same second. The entrance jams, people fall, and nobody gets in efficiently. The solution is obvious: stagger the entry — let people in gradually.
In distributed systems, this is the Thundering Herd problem. It occurs when a large number of clients or processes all wake up and simultaneously attempt to access a shared resource. The shared resource — a database, a cache, a recovering service — gets hit with a load spike it cannot handle, often causing it to fail, which triggers more retries, which causes more failures.
Common Scenarios
1. Cache Stampede (Cache Expiry)
A popular item in a cache (like Redis) expires at exactly 10:00:00 AM. At 10:00:01 AM, thousands of concurrent user requests realize the cache is empty and all simultaneously hit the backend database to fetch the data. The database receives a spike it was never designed to handle and falls over. This specific variant is called a cache stampede or dog-pile effect.
2. Retry Storms After Service Recovery
A critical service crashes. Thousands of dependent services start failing and enter a retry loop. When the critical service finally restarts, all waiting clients retry their connection at the exact same moment, immediately overwhelming the just-recovered service and crashing it again. The service can never stay up long enough to actually recover.
3. Synchronized Cron Jobs
Hundreds of servers all run a cron job at 0 * * * * (top of every hour). Each job hits the same database to pull configuration. The database sees a massive spike every hour on the hour, and is idle in between.
The Solution: Jitter
The fix is to break the synchronization. Instead of retrying at fixed intervals (e.g., exactly every 5 seconds) or expiring cache entries at a single fixed time, clients should add Jitter — a small, random amount of delay.
Naive retry (bad):
retry_delay = 5 seconds # All clients retry at the same time
Exponential backoff with jitter (good):
retry_delay = random_between(0, min(cap, base × 2^attempt))
# Example: attempt 3, base=1s, cap=30s
# delay = random_between(0, min(30, 1 × 8)) = random_between(0, 8)
# Each client gets a different delay, spreading the load
This spreads requests out over time, smoothing the spike into a manageable wave. AWS, Google, and Netflix all document exponential backoff with full jitter as the standard retry strategy for their SDKs.
Cache jitter works the same way: instead of setting all cache TTLs to exactly 3600 seconds, set them to 3600 + random_between(0, 300). Now cache entries expire at different times, and the database never sees a synchronized stampede.