From Raw Data to Actionable Panels
Collecting traces and metrics is only half the work. The other half is surfacing the right information at the right time. A well-designed CI observability dashboard answers the questions engineers actually ask: "Is CI healthy right now?", "What regressed this week?", and "Which pipeline is burning the most developer time?"
A production-grade CI dashboard in Grafana (or a similar tool like Honeycomb) should have the following panels:
Pipeline Health Overview
- Success rate by pipeline (last 24h, 7d, 30d)
- Total job volume over time (detects traffic spikes)
- Active runner count vs. queued job count (detects capacity pressure in real time)
Latency Panels
- P50, P95, P99 queue time per pipeline
- P50, P95, P99 execution time per pipeline
- Execution time trend over 30 days (detects slow regressions — a build that grew from 8 to 24 minutes over 3 months)
Failure Analysis
- Top 10 failing jobs by failure count (not just failure rate — a job that runs 1,000 times and fails 5% of the time causes more pain than one that runs 10 times and fails 50%)
- Failure breakdown by error type (timeout vs. assertion failure vs. infrastructure error)
- Flakiness rate: jobs that fail then pass on retry without a code change
Cost and Efficiency
- Runner utilization rate (idle time vs. active time — high idle time means over-provisioned runners)
- Cache hit rate per pipeline
- Compute-minutes consumed per pipeline per day (identifies the most expensive pipelines)
Alerting on CI Health
Dashboards are for humans who are already looking. Alerts are for catching problems before anyone notices. Set up alerts on:
- Queue time P95 > 5 minutes: Runners are saturated. Trigger autoscaling or page the platform team.
- Pipeline success rate < 80% over 1 hour: Something systemic is broken — not a flaky test, but an infrastructure or configuration issue.
- Execution time P95 increases > 50% week-over-week: A recent change introduced a significant regression in build performance.
- Cache hit rate drops below 50%: The cache configuration is broken — likely a lockfile path change or a cache key misconfiguration.
Key Point: The most insidious CI performance regressions are slow ones. A build that grows from 8 minutes to 24 minutes over three months never triggers an alert because no single day's change is dramatic. This is why a 30-day execution time trend panel is essential — it catches the boiling-frog pattern that point-in-time metrics miss entirely.