How Dagger Works Under the Hood
Dagger is a client-server system. Understanding this split is key to understanding why it works the way it does.
The SDK (Client side): You write your pipeline in a standard programming language using the Dagger SDK. For example, a Python script ci/pipeline.py that defines functions for building, testing, and publishing your application. This is just regular Python code — you can import libraries, write loops, define classes, and write unit tests for it.
The API layer: When you execute your script, the Dagger SDK does not run your commands directly on the host machine. Instead, it translates your code into a DAG — a Directed Acyclic Graph — of operations. A DAG is a dependency graph where each node is a task and edges represent "this must complete before that." Dagger uses this graph to understand which operations can run in parallel and which depend on each other.
The Engine (Server side): The DAG is sent to the Dagger Engine, which is itself a container running on your machine (or on the CI runner). The engine is built on top of BuildKit — the same technology that powers docker build. It executes each operation inside isolated containers, manages layer caching, and handles pulling images from registries.
Because the engine is just a container, it runs identically everywhere Docker runs: your MacBook, a GitHub Actions ubuntu-latest runner, a GitLab shared runner, or a bare EC2 instance. The pipeline logic is defined in your code. The execution environment is defined by the engine. Neither depends on what software happens to be installed on the host.
The Dagger Module System
Beyond the core SDK, Dagger has a module system that lets you package and share pipeline components. A Dagger module is a self-contained unit of pipeline logic — for example, a "Trivy security scanner" module or a "Helm deploy" module — that anyone can install and use in their own pipeline with a single install command, referencing the module's source and version.
This is the equivalent of pip install or npm install for your CI/CD logic. Instead of copy-pasting a "scan for vulnerabilities" step from one pipeline to another, you install a versioned, tested module. The Daggerverse is the public registry of community-contributed modules.