Strategy 1: The Band-Aid (Retries)
Most CI systems allow you to auto-retry a failed test — in GitHub Actions, this typically means wrapping the test command in a retry action configured with a timeout and a maximum attempt count.
- Mechanism: If
test_loginfails, the runner immediately runs it again up to 3 times. If any attempt passes, the build is marked Green. - Pros: Unblocks the team immediately. Handles genuine one-off glitches (like a transient network blip to a real external service that you haven't mocked yet).
- Cons: Hides bugs. If the test fails 10% of the time due to a race condition, auto-retrying effectively deletes that bug report. You are shipping code that crashes 10% of the time in production while your test suite tells you everything is fine.
Retries are acceptable as a temporary, time-boxed measure — never as a permanent policy. Every retried test should generate a ticket to investigate the root cause.
Strategy 2: The Cure (Quarantine)
A more robust approach is Test Quarantine. The principle: a flaky test should not block deployments, but it should not be invisible either.
- Detection: A test is flagged as flaky when it fails on the main branch but passes on retry, or when it fails on a PR that touches no related code. This can be automated using flakiness tracking tools or a simple database of test results over time.
- Exclusion: The main CI pipeline skips quarantined tests using a tag or marker — in pytest, this means decorating the flaky test with a
quarantinemarker and running the main suite with a filter that excludes anything carrying that marker. The build stays green and fast. - Repair Pipeline: Quarantined tests run in a separate scheduled pipeline (e.g., nightly). A developer is assigned ownership. The test is only promoted back to the main suite after it passes a stability threshold — for example, 50 consecutive passes with no failures.
- Visibility: Quarantined tests must be tracked in a dashboard or ticket system. A test that sits in quarantine for 6 months is a test that should be deleted.
Choosing Between Them
| Situation | Recommended Action |
|---|---|
| Occasional network blip to an external service | Short-term retry (max 2 attempts) + ticket to mock the dependency |
| Test fails 5–30% of the time | Quarantine immediately, investigate root cause |
| Test fails consistently | It's a real bug. Fix the code. |
| Test hasn't been fixed in quarantine for 30+ days | Delete it or escalate ownership |
Key Point: The quarantine strategy works only if it has teeth. If quarantined tests are never fixed, the quarantine list becomes a graveyard of ignored coverage. Enforce a maximum quarantine duration (e.g., 30 days) after which the test is either fixed or deleted. Deleted tests must be tracked so the coverage gap is acknowledged and eventually filled.