Remediation Workflows: Manual vs. Automated

1. Manual Remediation (Human-in-the-Loop)

Manual remediation treats drift as an alert rather than an automatic correction. A scheduled process detects the drift and notifies a human, who then decides what to do. This is the most common approach for traditional infrastructure teams and for any environment where manual emergency changes are a realistic operational pattern.

A typical workflow looks like this:

1. A nightly cron job (or CI pipeline schedule) runs terraform plan -detailed-exitcode.
2. Exit code 2 means changes are present — drift exists. The job posts a formatted diff to a Slack channel or opens a Jira ticket.
3. An on-call engineer reviews the diff the next morning.
4. They investigate the change: who made it, when, and why (using CloudTrail or equivalent audit logs).
5. They make a judgment call: revert with terraform apply, or update the code to adopt the change.

The same -detailed-exitcode contract from earlier drives the automation here too: exit 2 triggers the alert with a formatted diff of exactly which resources changed, exit 0 means nothing to report, and exit 1 signals the plan itself failed and needs investigation.

Pros:
- Safe. A human reviews every change before it is applied.
- Preserves emergency hotfixes. If an SRE manually scaled up a database at 2 AM to survive a traffic spike, the automation won't undo it before they've had a chance to document it.
- Provides an audit trail of human decisions, not just automated actions.

Cons:
- Slow. A security group with port 22 open to 0.0.0.0/0 might sit that way for 23 hours before anyone sees the alert.
- Alert fatigue. If drift is common and notifications are noisy, engineers start ignoring them.
- Dependent on human availability. Alerts sent at 11 PM on a Friday may not be actioned until Monday.

2. Automated Remediation (Self-Healing Infrastructure)

Automated remediation removes the human from the loop entirely. The system detects drift and immediately corrects it without waiting for approval. This model is common in GitOps environments — particularly Kubernetes with tools like ArgoCD — and in highly regulated environments where any deviation from the declared state is treated as a security violation.

In a Terraform context, automated remediation typically looks like a scheduled pipeline that runs on a short interval (every 30 minutes, say) and applies with auto-approval, no human review step at all. Every 30 minutes, any manual change made in the AWS Console is silently overwritten. The code is the absolute source of truth.

Pros:
- Guarantees infrastructure always matches code. Zero tolerance for drift.
- Security holes are closed automatically, without depending on human response time.
- Reduces operational toil — no one needs to manually run terraform apply after a drift alert.

Cons:
- Dangerous during incidents. If an SRE is manually scaling up a resource to survive a traffic spike, the automation may scale it back down mid-incident, actively worsening the outage.
- Masks root causes. If drift keeps recurring, auto-remediation hides the symptom without fixing the underlying problem (e.g., a team that doesn't know how to use Terraform).
- Requires extremely high confidence in your Terraform code. A bug in the code will be automatically and repeatedly applied to production.