Probabilistic Failure Analysis: Is It Me or the Test?

The Diagnosis Problem

When a build fails, you face an immediate question: did I break something, or is this a known flaky test acting up? Getting this wrong in either direction is costly. If you assume flake and it's a real bug, you ship broken code. If you assume real failure and it's flake, you waste an hour debugging a ghost.

How Probabilistic Analysis Works

Advanced CI platforms and internal tooling at companies like Google, Spotify, and Netflix use probabilistic failure analysis to automate this diagnosis.

The system maintains a historical record of every test result: pass/fail, which commit triggered it, which files were changed, and what time it ran. From this history, it computes a flakiness score for each test.

Bayesian Inference in Practice:

• Scenario A: test_checkout_flow has passed 2,000 times in a row over 6 months. Your PR modifies checkout_service.py. The test fails. Posterior probability: Real Failure. Alert the developer.

• Scenario B: test_checkout_flow has a 25% failure rate over the past 2 weeks. Your PR modifies README.md. The test fails. Posterior probability: Flake. Alert the platform team, not the developer.

• Scenario C: test_checkout_flow fails, but only on Tuesday mornings between 2am and 4am UTC. Posterior probability: Infrastructure flake (perhaps a scheduled job contends for database connections). Alert the infrastructure team.

Building a Simple Flakiness Tracker

You don't need a sophisticated ML system to get started. A simple database of test results, queried for the failure rate of each test over a recent window and filtered to the band between "occasionally fails" and "consistently fails" (consistently-failing tests are bugs, not flake), provides enormous value — it surfaces your flakiest tests, ranked by severity, so you can prioritize which ones to fix first.

Key Point: The most important signal in probabilistic analysis is the correlation between code changes and test failures. A test that fails only when its covered code changes is almost certainly a real failure. A test that fails regardless of what changed is almost certainly flaky. Building this correlation into your CI system — even manually at first — transforms failure triage from guesswork into data-driven decision-making.