"It Passed on My Machine!"
You push your code. The CI pipeline fails. You groan, hit the "Re-run" button, and grab a coffee. When you come back, it's green.
You just encountered a Flaky Test.
A test is "flaky" if it produces different results — Pass or Fail — given the exact same code and the exact same environment. Think of it like a smoke detector that sometimes goes off when you make toast, and sometimes doesn't. You can't trust it to tell you when there's a real fire.
- Stable Test: Deterministic. Fails only when the code is broken. Every run with the same code produces the same result.
- Flaky Test: Non-deterministic. Fails due to race conditions, network timeouts, shared state, or timing assumptions baked into the test itself.
The Cost of Flake
Flakiness is not a minor inconvenience — it is a systemic threat to your engineering culture.
- Loss of Trust: If tests fail randomly, developers stop treating failures as signals. "Oh, that's just the login test being weird again." Eventually, they ignore a real bug, and it ships to production. This is the most dangerous outcome: a test suite that exists but provides no safety guarantee.
- Slow Velocity: If you must re-run a 30-minute test suite three times to get a green build, you've wasted 90 minutes of developer time and compute budget — per engineer, per day.
- Alert Fatigue Debt: Teams begin adding retries everywhere, masking the underlying problem. The suite grows more expensive and less trustworthy simultaneously.
Google's internal research found that roughly 1 in 7 tests in their codebase exhibited flakiness at some point. At that scale, even a 1% flake rate across 50,000 tests means 500 tests that lie to you on every run.
Interviewers often ask: "How do you handle flaky tests in your CI pipeline?" A weak answer is "we retry them." A strong answer distinguishes between retries as a short-term unblocking tactic and quarantine + root-cause analysis as the sustainable strategy. Mention that retries mask bugs and erode trust, and that the goal is to eliminate flakiness, not tolerate it.
Key Point: Flakiness is a reliability problem, not a testing problem. A flaky test is a test that has encoded an assumption about timing, ordering, or environment that the real world doesn't always satisfy. Fixing flakiness means fixing that assumption — not hiding the symptom.