Flaky Pipeline

A flaky pipeline is a CI/CD pipeline that passes or fails nondeterministically, with no change to the code under test. The same commit goes red, then green on a rerun. The flakiness usually comes from timing dependencies, shared mutable state, network calls, race conditions, or environment instability rather than genuine defects.

A flaky pipeline is corrosive because it destroys the one thing a pipeline must have: trustworthy signal.

Why It Happens

Flakiness creeps in through tests that depend on real time, real networks, or specific execution order. Shared test environments introduce contention and leftover state. Resource-constrained CI runners cause timeouts under load. Each individual cause seems minor, and the easy fix, clicking rerun, hides the problem while letting it accumulate. Over time the pipeline becomes a coin flip.

Why It Hurts

When builds fail randomly, engineers learn to ignore red and just rerun. That habit is catastrophic: real failures get dismissed as flakes, and genuine bugs ship. Reruns waste enormous compute and human time waiting. The pipeline's purpose, fast trustworthy feedback, is defeated. Teams lose confidence in their tests and stop writing them, accelerating the decline in quality.

Warning Signs

• The standard response to a failure is to rerun until green.
• Failures are intermittent and not reproducible locally.
• Tests fail on timeouts under load.
• Test outcomes depend on execution order.

Better Alternatives

Make tests deterministic: control time with clocks you inject, replace real network calls with stubs, and remove inter-test dependencies through proper isolation. Build hermetically so runs do not depend on external state. Track flaky tests systematically, quarantine them out of the blocking path, and fix or delete them. Treat a flake as a defect, not a nuisance.

How to Refactor Out of It

Instrument the pipeline to record which tests fail intermittently and how often. Quarantine the worst offenders so they stop blocking merges, but keep them visible so they cannot be forgotten. Diagnose root causes, shared state, timing, order dependence, and fix them one by one. Re-admit tests to the blocking suite only once proven stable, and enforce a zero-tolerance policy for new flakes.