The "Waiting" Game
When a developer says "the build is slow," they are reporting a symptom, not a cause. Before you can fix anything, you must decompose Total Build Time into its two fundamentally different components. Confusing them leads to expensive, ineffective solutions.
Total Build Time = Queue Time + Execution Time
1. Queue Time (Pending Time)
- Definition: The elapsed time between the moment a job is triggered (e.g., a Git push fires a webhook) and the moment a runner picks it up and begins executing.
- What it looks like: The job appears in the CI UI with a "Pending" or "Queued" status. No work is happening. The developer is just waiting.
- Root cause: Resource starvation. You have more jobs triggered than you have available runners to execute them. This is a pure infrastructure capacity problem.
- Correct fix: Increase runner capacity — either by adding more runner instances (horizontal scaling) or by using autoscaling groups that spin up runners on demand and terminate them when idle.
- Wrong fix: Optimizing your build scripts. A faster script still waits in the same queue.
2. Execution Time (Run Time)
- Definition: The elapsed time from when a runner picks up the job to when the job completes (success or failure).
- What it looks like: The job is actively running. Logs are streaming. CPU and memory are being consumed.
- Root cause: Slow operations within the job itself — large Docker image pulls, uncached dependency downloads, slow test suites, underpowered runner hardware (CPU throttling, memory pressure causing disk swapping).
- Correct fix: Optimize the job — implement dependency caching, parallelize the test suite, use smaller base images, or upgrade runner CPU/RAM specs.
- Wrong fix: Adding more runners. More runners reduce queue time, but they do not make any individual job run faster.
The Percentile Trap
Average execution time is a dangerous metric on its own. If your average build time is 8 minutes but your P95 (95th percentile) is 35 minutes, then 1 in 20 builds is catastrophically slow — and the average hides it completely.
Always track:
| Metric | What it tells you |
|---|---|
| P50 (Median) | The typical developer experience |
| P95 | The worst experience for 1 in 20 builds |
| P99 | Outliers — often caused by flaky external dependencies |
| Max | The absolute worst case — useful for SLA analysis |
A healthy CI system has a tight distribution: P95 should be no more than 2x the P50. A wide distribution (P95 is 5x P50) signals flakiness — some builds are hitting a condition that most do not.
A common interview question is: "Your CI builds are slow. Walk me through how you'd diagnose it." The trap is jumping straight to solutions. The correct first step is to separate queue time from execution time by looking at metrics. Then, if execution time is the problem, use traces to identify the slow span. Only then do you propose a fix. This structured approach — measure, locate, fix — is what distinguishes a senior engineer's answer.
Key Point: Queue time and execution time require completely different organizational responses. Queue time is an infrastructure procurement and autoscaling problem — it often requires budget approval and cloud configuration changes. Execution time is a developer productivity problem — it requires code changes, caching strategies, and test architecture decisions. Conflating them in a single "build time" metric causes the wrong team to be assigned the wrong problem.