"Are we there yet?"
The most common sustaining effect is the Retry Storm. It works like this:
- Trigger: A database slows down for 1 second due to a lock contention or a GC pause.
- Impact: Frontend requests time out after 500ms and return errors to clients.
- Reaction: Clients — whether human users refreshing the page or automated SDKs with built-in retry logic — immediately retry the request.
- Amplification: Now the database must handle the new incoming requests plus all the retried requests from the previous wave.
- Feedback Loop: Load doubles. The database gets slower. More requests time out. More retries are issued. Repeat.
Even after the original database lag is resolved, the accumulated backlog of retries keeps the system crushed. Goodput — the rate of useful, successful work — drops to near zero. Throughput — total requests being processed — is at maximum. The system is busy doing nothing productive.
The Role of Synchronized Timeouts
Retry storms are dramatically worse when all clients share the same timeout value. If every client times out at exactly 500ms, every client retries at exactly 501ms. This creates a perfectly synchronized thundering herd. The database sees silence for 500ms, then an instantaneous wall of traffic — the worst possible access pattern.