CI Build Time (p95) Benchmark

CI build time benchmarks measure how long a continuous-integration pipeline takes to give developers feedback on a change. Build time sits directly on the inner loop of development: slow pipelines lengthen lead time, encourage large risky batches, and tempt developers to skip or bypass checks. Focusing on the p95 build time captures the painful tail rather than the rosy average.

A pipeline that is usually fast but occasionally very slow still hurts, because developers plan around the worst case they regularly hit.

What It Measures

The headline metric is p95 build time: the duration under which 95% of pipeline runs complete. Supporting metrics include median build time, queue time (waiting for a runner before work starts), per-stage duration (checkout, dependency install, compile, test, package), and cache hit rate, which strongly influences total time. Together they locate where time is spent and how variable it is.

Methodology

The benchmark collects timing data from the CI system across many pipeline runs, recording total duration and the duration of each stage, plus the time spent queued waiting for an available runner. Percentiles are computed over a representative window of real builds rather than a single run, because build time varies with cache state, concurrency, and changed files. Cache hit rate is measured for dependency and build caches, since a cold cache can multiply build time. Distinguishing queue time from execution time is essential: long queues indicate insufficient runner capacity, while long execution indicates inefficient steps. Tracking these over time reveals regressions and the effect of optimizations like parallelization or test splitting.

How to Interpret Results

Use p95, not the average, because the slow tail is what developers feel and route their behavior around. Separate queue time from execution time: high queue time is solved by adding runner capacity, while high execution time is solved by optimizing steps, caching, parallelizing, or splitting tests. A low cache hit rate is a common and easily fixed cause of slow builds. Read stage durations to find the bottleneck, usually tests or dependency installation. Benchmark against your own trend and against a target feedback time (many teams aim to keep core CI under ten minutes), and watch for creeping regressions as the codebase grows.

Limitations

Build time depends heavily on infrastructure, runner specs, and concurrency, so numbers do not transfer across environments. p95 can be dominated by a few pathological runs or by queueing unrelated to pipeline efficiency. Aggressive optimization can trade thoroughness for speed, for example by skipping tests, which harms quality. Caching improves time but can mask real dependency or build issues. The metric measures feedback latency, not correctness or coverage, and should be balanced against pipeline thoroughness rather than minimized at all costs.