Static Stability is the primary architectural defense against Control Plane failures. The name comes from aviation: a statically stable aircraft, when disturbed, naturally returns to level flight without pilot input. A statically stable system, when its control plane disappears, continues operating on its current configuration without needing to call home.
"The Plane Flies Without the Tower"
An air traffic control tower going dark doesn't cause planes already in the air to fall. They keep flying on their last known heading until they can re-establish contact. Your systems should work the same way.
Unstable Design (Control Plane Dependent):
User Request → Web Server → Calls AWS API to validate permissions → Serves Response
If the AWS API fails, every user request fails. The Data Plane (web server) is coupled to the Control Plane (AWS API).
Statically Stable Design:
User Request → Web Server → Checks local permission cache (TTL: 1 hour) → Serves Response
↑
Background job refreshes cache from AWS API
If the AWS API fails, the cache serves stale-but-valid permissions for up to one hour. The Data Plane survives a Control Plane outage for the duration of the cache TTL.
Pre-Scaling for Stability
Auto Scaling is a Control Plane feature. When traffic spikes, your Auto Scaling Group calls the EC2 API to launch new instances. If the EC2 API is down during a traffic spike — a plausible Black Swan scenario — your system cannot scale up, and it collapses under load.
The Radical Strategy: During high-risk periods (Black Friday, major product launches, known high-traffic events), disable auto-scaling entirely. Provision for peak load plus a 30% buffer, and leave it running.
Trade-offs:
| Approach | Cost | Control Plane Dependency | Risk |
|---|---|---|---|
| Auto Scaling | Low (pay for what you use) | High (scaling requires API calls) | Fails if Control Plane is down during traffic spike |
| Pre-scaled Static | High (pay for idle capacity) | None (instances already running) | Survives Control Plane outage; wastes money if traffic doesn't spike |
This is not a permanent architecture — it's a deliberate, time-bounded trade of efficiency for resilience during your highest-risk windows.
Applying Static Stability Broadly
The principle extends beyond EC2:
- DNS: Cache DNS responses aggressively. If your DNS provider has an outage, cached records keep traffic flowing.
- Secrets: Cache secrets from AWS Secrets Manager locally. Don't fetch them on every request.
- Configuration: Bake configuration into AMIs or container images at build time rather than fetching from a config service at startup. If the config service is down, new instances still start correctly.
- Service Discovery: Cache service endpoint addresses. Don't rely on real-time service registry lookups for every inter-service call.
The pattern is always the same: fetch eagerly, cache locally, serve from cache, refresh in the background.
A common senior SRE interview question is: "Your Auto Scaling Group isn't launching new instances during a traffic spike. Walk me through your diagnosis." Most candidates jump to IAM permissions or launch template errors. The Black Swan answer adds: "I'd also check whether the EC2 or Auto Scaling control plane itself is experiencing an outage — this is a known failure mode where the Data Plane is healthy but the Control Plane is unavailable. The mitigation is pre-scaling before high-risk events and designing for static stability so the existing fleet can absorb load without needing to scale."