Flaky Test Management

Best Practice: Flaky Test Management

A flaky test passes and fails on the same code without any change, usually due to timing, shared state, ordering, network calls, or randomness. Flaky tests are corrosive: they erode trust in CI, train developers to ignore failures, and hide real regressions behind noise. Google has reported that a large share of its test failures come from flaky tests, which is why systematic management matters. The goal is a reliable signal. That means detecting flakiness automatically, isolating offenders so they do not block delivery, and fixing the root cause rather than masking it with blanket retries.

Step-by-Step Implementation Guidance

1. Detect flakiness by tracking pass/fail history per test and flagging nondeterministic results.
2. Quarantine confirmed flaky tests so they run but do not block merges.
3. File a tracked ticket for each quarantined test with an owner and deadline.
4. Diagnose the root cause: timing, test order dependence, shared fixtures, or external calls.
5. Fix deterministically by controlling clocks, isolating state, and stubbing external services.
6. Use bounded retries only as a temporary safety net, never as a permanent fix.
7. Return repaired tests to the blocking suite and monitor for recurrence.

Common Mistakes Teams Make When Ignoring This Practice

• Adding unconditional retries that hide real intermittent bugs.
• Disabling flaky tests permanently without a tracking ticket.
• Letting tests share mutable global state or depend on execution order.
• Relying on real network or time in unit tests.
• Normalizing red builds until genuine failures go unnoticed.

Tools and Techniques That Support This Practice

• Test history and flake dashboards in CI platforms.
• Quarantine/skip annotations with linked tracking issues.
• Deterministic clocks and fake timers.
• Network virtualization: WireMock, MSW, Testcontainers.
• Test isolation and randomized order checks to surface hidden dependencies.

How This Practice Applies to Different Migration Types

• Cloud Migration: Remove dependence on real network latency so tests stay stable across environments.
• Database Migration: Reset and isolate test data per run to avoid order-dependent flakiness.
• SaaS Migration: Stub third-party services so suite reliability does not hinge on external uptime.
• Codebase Migration: Stabilize the suite first so flaky noise does not mask regressions during the rewrite.

Checklist

• Flaky tests are detected automatically from pass/fail history.
• Confirmed flaky tests are quarantined, not silently deleted.
• Each quarantined test has an owner and a tracked ticket.
• Root causes are diagnosed before fixes are applied.
• External calls and time are controlled deterministically.
• Retries are bounded and temporary.
• Repaired tests return to the blocking suite and are monitored.