The Empty Cache Nightmare
Your system relies on a cache (Redis, Memcached) to serve 99% of reads. Only 1% of requests reach the database directly.
- Trigger: A standard deployment flushes the cache, or a popular key's TTL expires.
- Impact: Suddenly, 10,000 concurrent requests miss the cache simultaneously.
- Reaction: All 10,000 requests fall through to the database at the same millisecond to fetch the missing data.
- Amplification: The database, designed to handle 1% of read traffic, immediately saturates under 100x its expected load.
- Feedback Loop: Because the database is overloaded, requests time out before the data can be fetched. The cache never gets repopulated. The next wave of 10,000 requests also misses and hits the database.
The system is trapped: the cache is empty because the database is too busy to fill it, and the database is too busy because the cache is empty.
Why Deployments Are Dangerous
Cache stampedes are especially common after deployments that flush caches as part of the release process. The system was healthy before the deploy. The deploy itself is the trigger. Engineers often misdiagnose this as a bug in the new code rather than a metastable failure caused by cache invalidation.
A common interview scenario: "Your service is healthy, you deploy a new version, and within 30 seconds your database is at 100% CPU and the service is down. What happened?" The answer is almost always a cache stampede triggered by cache invalidation during deployment. The follow-up question is: "How do you prevent it?" — which leads to request coalescing, probabilistic early expiration, and staggered cache warming.