Event-Driven IaC: The Immune System

From Polling to Pushing

Scheduled drift detection has an inherent latency problem. If you check for drift every 24 hours, a security hole can exist for up to 23 hours and 59 minutes before anyone knows. Checking every minute is technically possible but expensive and noisy. The fundamental issue is that polling — asking "has anything changed?" on a timer — is the wrong model for time-sensitive security events.

Event-driven IaC flips this model. Instead of asking periodically, the system listens for change events and reacts the moment a change occurs. Detection latency drops from hours to seconds.

The Architecture

The event-driven remediation pipeline has five stages:

1. The Event: A user modifies a Security Group in the AWS Console, adding an inbound rule.
2. The Signal: AWS CloudTrail records the API call (AuthorizeSecurityGroupIngress) within seconds, including who made the call, from which IP, and at what time.
3. The Router: Amazon EventBridge monitors CloudTrail events and matches this specific event pattern against a rule you've defined.
4. The Trigger: EventBridge invokes a Lambda function, sends a webhook to GitHub Actions, or calls Terraform Cloud's API.
5. The Action: The triggered system runs a terraform plan (to assess the change) or a full terraform apply (to immediately revert it).

The routing rule that ties this together matches on the specific event source and the specific set of API call names that represent a security-relevant change — any of the security group ingress/egress authorize-or-revoke calls, for instance — so the pipeline only fires on the changes that actually matter, not on every CloudTrail event flowing through the account.

This acts like a digital immune system. The moment a foreign object (an unauthorized manual change) appears in the environment, the system detects it and responds — without waiting for a scheduled check.

Trade-offs and Failure Modes

Event-driven remediation is powerful but introduces its own complexity:

• False positives: Not every Security Group change is unauthorized. If your auto-scaling system legitimately modifies security groups, your event-driven pipeline will trigger on those too. You need filtering logic to distinguish authorized automation from unauthorized manual changes.
• Race conditions: If a legitimate emergency change and an automated revert happen simultaneously, the outcome depends on timing. The last write wins, which may not be the right write.
• Event delivery guarantees: EventBridge delivers events at least once, not exactly once. Your remediation logic must be idempotent — running it twice should produce the same result as running it once.
• Scope creep: It's tempting to wire up event-driven remediation for every resource type. In practice, start with high-risk, high-value resources (security groups, IAM policies, network ACLs) and expand carefully.