"Too Big to Fail?"
Many engineers implicitly believe that an entire AWS or Google Cloud region — like us-east-1 — is too big to fail. After all, it contains multiple Availability Zones (AZs) separated by tens of miles, with independent power, networking, and cooling. Surely a failure in one AZ won't take down the others?
History says otherwise. Regions fail. Here are real categories of causes:
- Cascading software bugs: A bad configuration push to the regional control plane replicates across all AZs simultaneously. The AZs are physically separate but logically coupled through shared management software.
- Extreme weather: Hurricanes, ice storms, and flooding can affect power infrastructure across an entire metropolitan area, overwhelming redundancy.
- Dependency failures: A region-wide IAM or DNS failure means nothing in that region can authenticate or resolve names — even if every compute instance is physically running.
The us-east-1 outage in December 2021 is a well-documented example: a configuration change to AWS's network caused cascading failures that affected services across the entire region for hours, including the AWS console itself (making it harder to diagnose and respond).
The Multi-Region Imperative
For a Black Swan event like a region failure, Multi-AZ is not enough. You need Multi-Region architecture.
There are three primary strategies:
| Strategy | Description | RTO | Cost |
|---|---|---|---|
| Passive (Backup/Restore) | Data replicated to a second region via S3 Cross-Region Replication or database snapshots. Application is not running there. | Hours | Low |
| Warm Standby | A scaled-down version of the application runs in a second region, ready to scale up. | Minutes | Medium |
| Active-Active | Full traffic runs in two or more regions simultaneously. Failover is instant. | Seconds | High |
The Hard Truth: Active-Active multi-region is expensive and architecturally complex. It requires solving distributed data consistency problems (how do you handle writes to two databases simultaneously?), global load balancing, and latency-aware routing. Most companies consciously accept a 24-hour RTO for a region failure rather than paying the engineering and infrastructure cost of Active-Active. This is a valid business decision — but it must be made explicitly, not discovered during an outage.