Not All Resources Are Equal
The biggest mistake teams make is applying a single remediation strategy to all infrastructure. A one-size-fits-all approach either leaves security gaps (if you're too lenient) or breaks incident response (if you're too aggressive). The right model segments resources by risk and operational characteristics.
Use this framework to classify your resources:
| Resource Type | Example | Recommended Strategy |
|---|---|---|
| Security-critical, stateless | Security Group rules, IAM policies, Network ACLs | Automated revert (event-driven or scheduled, short interval) |
| Application config, low blast radius | Lambda environment variables, S3 bucket tags | Automated revert (scheduled, medium interval) |
| Stateful, incident-adjacent | RDS instances, Auto Scaling Groups, ECS task counts | Human-in-the-loop only |
| Shared/platform infrastructure | VPCs, subnets, DNS zones | Human-in-the-loop with mandatory code review |
| Ephemeral/disposable | Dev/test environments | Automated revert or scheduled destroy-and-recreate |
The Break-Glass Pattern
Any automated remediation system must have a documented escape hatch for emergencies. The break-glass pattern is a controlled mechanism for temporarily suppressing automated remediation on a specific resource during an incident.
Implementation options:
- A special tag on the resource (e.g., terraform-remediation: suspended) that the remediation pipeline checks before applying.
- A feature flag in your pipeline configuration that an on-call engineer can flip.
- A time-bounded suppression rule in EventBridge that expires automatically after a defined window (e.g., 2 hours).
The break-glass action must be logged, audited, and automatically expire. An indefinite suppression is just drift with extra steps.