Flaky Test Rate Benchmark

Flaky test rate benchmarks measure how many tests in a suite produce nondeterministic results, passing on one run and failing on the next without any code change. Flaky tests are corrosive: they cause false CI failures, train engineers to ignore red builds, mask real regressions, and waste time on reruns. Quantifying flakiness is the first step to controlling it.

At scale, even a small per-test flake probability makes whole pipelines unreliable, because the chance that at least one test in a large suite flakes grows quickly.

What It Measures

Core metrics include the flaky rate (share of tests or runs that are flaky), flaky test count, rerun pass rate (how often a failing test passes on retry, a flakiness signal), and CI failure attribution (fraction of build failures caused by flakiness rather than real defects). The number of tests quarantined or skipped due to flakiness is also tracked.

Methodology

Flakiness is detected by observing test outcomes across many runs of unchanged code. The common technique reruns failing tests automatically; a test that fails then passes without code change is flagged as flaky. Continuous tracking records per-test pass/fail history across builds, computing how often each test changes outcome on identical commits. Some systems run the suite repeatedly on a schedule against a fixed commit to surface intermittent failures. The benchmark aggregates per-test flakiness into a suite-level rate, attributes CI failures to flaky versus genuine causes, and tracks the quarantine list over time. Categorizing root causes (timing, concurrency, test order, external dependencies) guides remediation.

How to Interpret Results

A low and falling flaky rate indicates a trustworthy suite; a high or rising rate predicts ignored failures and hidden regressions. CI failure attribution is especially telling: if most red builds are flakiness, developers stop trusting the signal entirely. Rerun pass rate quantifies how much retry logic is masking instability rather than fixing it. Watch the quarantine list, since quarantining controls noise but, left unmanaged, becomes a graveyard of disabled tests that erodes coverage. Prioritize fixing flaky tests on critical paths and those that fail most often, and treat the trend as the key health indicator.

Limitations

Flakiness detection relies on reruns and history, which add cost and can themselves mask real failures by retrying until green. Distinguishing a genuinely flaky test from a real intermittent bug is hard, and aggressive rerunning can hide true defects. The metric depends on rerun and tracking policy, so rates are not comparable across organizations. It measures test reliability, not test value or coverage. Quarantining to lower the visible rate trades reliability for reduced coverage. Best used as a trend with root-cause categorization, paired with a policy to fix or remove flaky tests rather than hide them.