Free — no signup required

Cost Control: Stopping the Bleeding

3 min read

The "Zombie Environment" Risk

Ephemeral environments are only cheap if they are actually ephemeral. The most common failure mode is zombie environments — environments that were created for a PR, never cleaned up (because the cleanup job failed, or the PR was closed via a method the pipeline didn't catch), and are now silently consuming compute and storage resources. A team with 20 active developers can easily accumulate 50–100 zombie environments over a few months, adding thousands of dollars per month to the cloud bill.

Control Strategy 1: Time-To-Live (TTL)

Implement a hard policy: all preview environments are automatically deleted after a fixed period (e.g., 24 or 48 hours), regardless of PR status. A scheduled job (a Kubernetes CronJob or a CI pipeline on a timer) scans all namespaces with the label type=preview, checks their age, and deletes any that exceed the TTL.

Developers who need the environment back can comment /redeploy on the PR to trigger a fresh deployment. This is a minor inconvenience that saves significant money.

In practice, this is a Kubernetes CronJob running hourly under a scoped service account, listing every namespace labeled as a preview environment, filtering to those older than the TTL, and deleting each one.

Control Strategy 2: Scale to Zero (Sleep Mode)

Rather than deleting an environment after inactivity, you can scale it down to zero replicas. KEDA (Kubernetes Event-Driven Autoscaling) and Knative both support scaling a deployment to zero pods when there is no incoming traffic, and scaling back up to one pod when a request arrives.

The user experience: if no one has visited pr-123.app.com for 30 minutes, the pod is terminated. When someone clicks the link, the request is held by the Ingress for 5–10 seconds while a new pod starts, then served normally. This reduces compute costs by approximately 80–90% for environments that are only accessed intermittently.

This is the preferred strategy for long-running PRs (e.g., a feature branch that takes two weeks to review) where you want the environment to remain available on demand without paying for idle compute.

Control Strategy 3: Spot/Preemptible Instances

Since preview environments are non-production, they can tolerate interruption. Run the nodes that host preview environments on AWS Spot Instances, Google Cloud Preemptible VMs, or Azure Spot VMs. These are spare cloud capacity sold at a 60–90% discount, with the caveat that the cloud provider can reclaim them with 2 minutes' notice.

For a preview environment, a 2-minute interruption means the developer refreshes their browser and the pod reschedules on a new node within 60 seconds. This is a minor inconvenience, not a production incident. The cost savings are substantial enough that most platform teams dedicate a separate Spot node pool exclusively to preview workloads.

Interview Tip

A common interview question is: "How do you prevent ephemeral environments from becoming a runaway cost center?" The answer has three layers: (1) automated cleanup on PR close events as the primary defense, (2) TTL-based reaping as a safety net for missed cleanup events, and (3) scale-to-zero for long-lived environments to eliminate idle compute costs. Mentioning all three layers — and explaining why you need all three — signals that you have thought about failure modes, not just the happy path.

Tagging and Visibility

Every preview environment resource should be tagged with metadata that enables cost attribution and auditing — labels identifying it as a preview resource, the PR number, branch name, owning developer, creation timestamp, and team.

These labels feed into cloud cost dashboards (AWS Cost Explorer, GCP Billing) so you can see exactly which team, which developer, and which PR is responsible for each dollar spent. Without this visibility, cost optimization is guesswork.

This is one of 18 chapters

Every chapter is free — Kubernetes, Terraform, SRE, distributed systems, and more — with fast daily review built in.

Sign up free