Caching: Speed by Default

How Dagger's Cache Works

Slow CI pipelines are one of the biggest productivity drains in software engineering. A pipeline that takes 20 minutes to run means a developer waits 20 minutes to find out if their change is correct. Multiply that by 50 engineers running 10 pipelines a day, and you have a significant engineering cost.

Dagger's caching is built on BuildKit's content-addressable cache. Every operation in the DAG is identified by a hash of its inputs: the base image, the source files, the command being run, and the environment variables. If the inputs haven't changed, the output is retrieved from cache instantly — no re-execution.

Here is what this looks like in practice: a cold-cache run pulling the base image, downloading dependencies, compiling, and testing might take around 98 seconds end to end. Change one line in an application file and re-run, and the base image pull and dependency download are served from cache — only the compile and test steps, whose inputs actually changed, re-execute, cutting the run to roughly 51 seconds. Change only a README and re-run, and every step hits cache since nothing that actually affects the build changed — the run finishes in under a second.

The cache is not just local. Dagger supports remote cache backends — you can configure it to push cache layers to a registry (like AWS ECR or Docker Hub) and pull them on CI runners. This means a CI runner that has never seen your code before can still benefit from the cache populated by your laptop or a previous CI run.

Cache Invalidation and Ordering

A critical insight for getting maximum cache benefit: the order of operations in your Dagger pipeline matters. Operations that change frequently (like copying your source code) should come after operations that change rarely (like installing dependencies). This is the same principle as optimizing a Dockerfile layer order.

# GOOD: Dependencies cached separately from source code
container = (
    client.container()
    .from_("golang:1.22")
    .with_workdir("/src")
    .with_file("/src/go.mod", source_dir.file("go.mod"))
    .with_file("/src/go.sum", source_dir.file("go.sum"))
    .with_exec(["go", "mod", "download"])   # ← cached until go.mod/go.sum change
    .with_directory("/src", source_dir)      # ← invalidates cache when source changes
    .with_exec(["go", "build", "./..."])
)

# BAD: Copying source first invalidates the dependency cache on every change
container = (
    client.container()
    .from_("golang:1.22")
    .with_directory("/src", source_dir)      # ← any source change invalidates everything below
    .with_exec(["go", "mod", "download"])   # ← re-runs even if go.mod didn't change
    .with_exec(["go", "build", "./..."])
)

Senior engineers understand that cache invalidation strategy is as important as the pipeline logic itself. A poorly ordered pipeline with perfect logic can still be 10x slower than it needs to be.