Simulation Is Not Enough
Static analysis and plan checks are powerful, but they are simulations. They can tell you that your code looks correct and that your plan intends to do the right thing. They cannot tell you whether your new EC2 instance can actually reach your RDS database over the network, whether your load balancer health checks pass, or whether your Lambda function has the IAM permissions it needs to write to S3.
To know for sure, you have to build it.
Terratest is an open-source Go library from Gruntwork that automates the deploy-verify-destroy cycle. You write tests in Go — a statically typed, compiled language well-suited for infrastructure automation — and Terratest provides helper functions for common operations: running Terraform commands, making HTTP requests, connecting via SSH, querying AWS APIs, and more.
A Terratest integration test follows a consistent three-phase pattern:
Phase 1 — Spin Up: The test calls terraform.InitAndApply(), which runs terraform init and terraform apply against a real cloud account. This creates actual resources: real EC2 instances, real VPCs, real load balancers.
Phase 2 — Verify: The test uses Go's standard library and Terratest helpers to assert that the infrastructure works correctly. This might mean making an HTTP GET request to a load balancer URL and asserting the response code is 200, or SSH-ing into a server and checking that a specific process is running.
Phase 3 — Tear Down: The test calls terraform.Destroy() to delete all created resources. Critically, this is scheduled with Go's defer keyword, which guarantees cleanup runs even if the test panics or an assertion fails. Without this, failed tests leave "zombie" resources running in your cloud account, accumulating cost indefinitely.
A realistic Terratest for a module that creates an HTTPS-enabled web server follows exactly that three-phase shape in code: it configures the Terraform options for the module under test, immediately schedules terraform.Destroy() via Go's defer — before anything is even created — then calls terraform.InitAndApply(), reads the load balancer URL from the module's outputs, and retries an HTTP GET against that URL for up to several minutes (giving the server time to boot) until it gets back a 200 with the expected body.
Why Go? Terratest uses Go because Go compiles to a single binary, has excellent concurrency support (useful for running many tests in parallel), and has a rich standard library for networking, SSH, and HTTP. You don't need to be a Go expert to write Terratest tests — the patterns are repetitive and the Terratest library handles most of the complexity.
Cost and time considerations: Integration tests are expensive. A test that deploys a VPC, EC2 instance, RDS database, and load balancer might cost a few dollars and take 10–20 minutes to run. This means you don't run integration tests on every commit. A common strategy is:
- On every PR: Run static analysis and plan checks (seconds).
- Nightly or on merge to main: Run integration tests against a dedicated sandbox account.
- Before a major release: Run the full integration test suite.
A common interview question is: "What happens if a Terratest test fails halfway through — do you end up with orphaned resources?" The answer is: not if you use defer terraform.Destroy(). The defer keyword in Go guarantees the function runs when the enclosing function exits, regardless of how it exits — normal return, panic, or failed assertion. This is the idiomatic Terratest pattern and interviewers expect you to know it.
Key point: Integration tests are not a replacement for static analysis and plan checks — they are the final layer of the pyramid. The goal is to run cheap, fast checks as often as possible and reserve expensive integration tests for the scenarios where only real infrastructure can give you confidence: network connectivity, IAM permission chains, service health checks, and cross-resource dependencies.