Avoiding Cloud Bankruptcy
The most dangerous failure mode of ephemeral environments is not a broken deployment — it is a forgotten one. If you spin up 50 environments a week and the cleanup step fails silently, you will accumulate idle Kubernetes workloads consuming CPU, memory, and potentially cloud resources. Your cloud bill will grow until someone notices.
Robust lifecycle management requires three layers of cleanup:
Layer 1: Event-Driven Cleanup (Primary)
Your CI/CD pipeline must listen for Pull Request lifecycle events and act on them immediately:
| PR Event | Pipeline Action |
|---|---|
opened |
Create namespace, install Helm chart |
synchronize (new commit pushed) |
Run helm upgrade to update the environment |
closed or merged |
Run helm uninstall, then kubectl delete namespace |
The closed event fires for both merged and abandoned PRs, so a single handler covers both cases.
Layer 2: The Safety Net (Cron Job)
Event-driven cleanup is reliable but not perfect. A webhook can fail. A pipeline can time out. A namespace can be left orphaned. The safety net is a scheduled job (daily or nightly) that:
- Lists all namespaces matching the pattern
ns-pr-* - Extracts the PR number from each namespace name
- Queries the GitHub API to check if that PR is still open
- Deletes any namespace whose PR is closed or does not exist
Layer 3: TTL-Based Expiry
As a final backstop, label every namespace with a creation timestamp and enforce a maximum age (e.g., 72 hours). Any namespace older than the TTL is deleted regardless of PR state. This catches edge cases where the PR number cannot be resolved or the GitHub API is unavailable. A cleanup job then compares this label against the current time and deletes stale namespaces.
A common interview question is: "What happens if your cleanup webhook fails and environments accumulate?" The answer demonstrates operational maturity: you need defense in depth — event-driven cleanup as the primary mechanism, a scheduled reconciliation job as the secondary, and TTL-based expiry as the final backstop. Relying on a single cleanup trigger is a single point of failure.