Retention Policies: Hoarding vs. Cleaning

The Cost of Success

A high-velocity CI/CD pipeline is a double-edged sword. If your team ships 50 pull requests a day and each PR triggers a build that produces a 500MB Docker image, you're generating 25GB of artifacts per day — 9TB per year — from feature branches alone. Most of those images will never be deployed to production. They exist only to validate that the PR's code compiles and passes tests.

Storage is cheap, but 9TB is not free. More importantly, unmanaged artifact sprawl creates real operational problems:

• Security exposure: Old images with known CVEs sit in your registry indefinitely, creating audit findings.
• Confusion: Developers can't tell which image tags are safe to use.
• Slow tooling: Registries with millions of unindexed artifacts become slow to query.

However, you can't just delete everything old. Some artifacts must be kept.

The Two Forces: Compliance vs. Cost

A well-designed retention policy explicitly encodes the tension between two opposing requirements:

Force 1 — Compliance (Keep It):

Regulatory frameworks (SOC 2, PCI-DSS, HIPAA) and internal audit requirements often mandate that you can reproduce any artifact that was ever deployed to production. If a security incident occurs in 2027 involving software deployed in 2024, you need to be able to rebuild that exact binary from the exact same inputs.

• Rule example: "Any artifact tagged as a production release must be retained for 7 years."
• Implementation: Use artifact promotion. When an image passes all staging gates and is deployed to production, it is copied (promoted) from the dev-images repository to a release-images repository. The release-images repository has no automated cleanup rules and requires explicit approval to delete anything.

Force 2 — Cost (Delete It):

CI builds for feature branches and pull requests become irrelevant the moment the branch is merged or closed. Keeping them wastes money and creates noise.

• Rule example: "Any image in the dev-images repository tagged with pr-* or branch-* is deleted after 14 days."
• Implementation: Use automated lifecycle policies. AWS ECR Lifecycle Policies, for example, let you write rules in JSON that run nightly and delete images matching specific tag patterns or age thresholds.

Implementing ECR Lifecycle Policies

AWS ECR Lifecycle Policies are JSON documents attached to a repository. They evaluate rules in priority order and expire images that match.

A practical policy for a repository that holds both feature-branch builds and release tags declares three rules, evaluated in priority order: the first protects release-* tagged images by keeping only more than a set count of the most recent ones (implicitly preserving everything else); the second cleans up the untagged images that Docker creates when you push a new image with an existing tag — the old image loses its tag and becomes "untagged" — once they're older than a day; the third expires ephemeral branch/PR/dev-tagged images once they're older than 14 days.

Senior-level nuance: Lifecycle policies operate on the repository, not on individual images. If you store both production and development images in the same repository, your rules become complex and error-prone. The cleaner architecture is separate repositories per environment tier (myapp-dev, myapp-staging, myapp-release), each with its own simple lifecycle policy. This also makes IAM easier: CI runners get write access to myapp-dev, and only the promotion pipeline gets write access to myapp-release.