MLPerf Training

MLPerf Training is a benchmark suite maintained by MLCommons that measures the time required to train representative machine-learning models to a defined target accuracy. It is the most widely cited standard for comparing AI training systems, from a single accelerator to clusters of thousands of GPUs or TPUs.

The suite covers a rotating set of workloads spanning image classification (ResNet on ImageNet), object detection, large language model pretraining (GPT-3 and Llama-class reference models), recommendation (DLRM), graph neural networks, and speech. Each task fixes the model architecture, dataset, optimizer, and a target quality threshold, so the only variable is how fast a system reaches that quality.

What It Measures

The headline metric is time-to-train: wall-clock seconds to reach the target metric (for example 75.9% top-1 accuracy for ResNet-50). Because the quality bar is fixed, faster numbers reflect genuine end-to-end system performance — accelerator throughput, interconnect bandwidth, data pipeline efficiency, and software stack maturity all contribute. Results are reported in Closed (fixed hyperparameters and model) and Open (algorithmic freedom) divisions, plus Available, Preview, and Research system categories.

Methodology

Submitters run the reference implementation or an equivalent, log every run with the MLPerf logging library, and submit multiple runs. The benchmark uses the median (or a trimmed mean) of several runs to reduce variance from random seeds. Results undergo peer review by other submitters before publication. Scaling submissions report results at many node counts so buyers can read strong-scaling behavior.

How to Interpret Results

Compare systems only within the same division, category, and round, since workloads and rules change between rounds. A 2x lower time-to-train at the same scale indicates roughly 2x more usable training performance. Watch scaling efficiency: doubling accelerators rarely halves time because communication and data loading impose overhead. Normalize by accelerator count, power, or cost to judge efficiency rather than raw speed. Closed-division numbers are the fairest apples-to-apples comparison.

Limitations

MLPerf Training rewards systems tuned specifically for these workloads, which may not match a team's real models. Large submissions require expensive clusters, so the top results reflect vendor engineering effort more than out-of-the-box experience. The fixed accuracy target ignores convergence behavior beyond that point, and energy reporting is optional, limiting sustainability comparisons. Results age quickly as the suite and hardware evolve.