The Fallacy of Reliability
Think about ordering food through a delivery app. The app talks to a restaurant service, a payment service, a mapping service, and a notification service — all at once. If any one of those services has a bad moment, what happens to your order? If the code was written optimistically — assuming every call succeeds — a single hiccup in the mapping service could freeze your entire order screen indefinitely.
In distributed systems (like microservices architectures, where an application is split into many small, independently deployed services), failure is not an edge case — it is a certainty. Network packets drop. Databases stall under load. Third-party APIs return 503 Service Unavailable. Hardware fails. Cloud providers have outages.
Resiliency Engineering is the discipline of writing code that treats failure as a first-class citizen. Instead of hoping every call succeeds, resilient code anticipates failure and handles it gracefully — without human intervention, without cascading to other services, and without leaving users staring at a spinner forever.
This lesson covers the three most critical resiliency patterns every backend engineer and SRE must know:
| Pattern | Problem It Solves | Core Mechanism |
|---|---|---|
| Exponential Backoff with Jitter | Thundering herd on retry storms | Randomized, increasing wait times |
| Circuit Breaker | Wasted resources on a dead service | Fail-fast state machine |
| Bulkhead | One failure consuming all resources | Isolated resource pools |
These patterns are not mutually exclusive. Production systems layer all three together. A single outbound HTTP call might be wrapped in a bulkhead (to limit concurrency), a circuit breaker (to stop calling a dead service), and a retry loop with backoff (to handle transient failures).