State Locking: Preventing Disaster

The Race Condition

The terraform.tfstate file is the source of truth for your infrastructure. It maps every resource in your HCL code to a real-world resource ID (e.g., aws_instance.web → i-0abc123def456). If this file is corrupted or inconsistent, Terraform loses track of what it manages — and recovery is painful.

The race condition is straightforward: if two engineers run terraform apply simultaneously, both processes read the same state file, calculate their own plan, and then both try to write back a modified state file. The second write overwrites the first. Resources created by the first apply are now invisible to the state file. Terraform thinks they don't exist and will try to create them again — or worse, delete them.

Remote State and Locking

The solution has two parts: move the state file to a shared, durable location (remote state), and add a mutual exclusion mechanism (locking).

The standard AWS pattern uses S3 for storage and DynamoDB for locking, configured once as a backend block naming the bucket, the state file's key path, and the DynamoDB table to use for locks.

How the lock works step by step:

1. Engineer A runs terraform apply.
2. Terraform writes a lock record to the DynamoDB table with a unique LockID, the user's identity, and a timestamp.
3. Engineer B runs terraform apply at the same time.
4. Terraform tries to write a lock record — DynamoDB rejects it because a record already exists.
5. Engineer B sees: Error: Error locking state: Error acquiring the state lock.
6. Engineer A's apply completes. Terraform deletes the lock record.
7. Engineer B can now retry.

The DynamoDB table requires only a single attribute: a partition key named LockID of type String. The table does not need any special throughput settings — Terraform's locking pattern is extremely low-volume.

Senior nuance: If a Terraform process is killed mid-apply (e.g., a CI runner times out), the lock record remains in DynamoDB. No subsequent apply can run until the lock is manually released with terraform force-unlock <LOCK_ID>. This is intentional — Terraform refuses to guess whether the interrupted apply left infrastructure in a consistent state. Always investigate before force-unlocking in production.