MLPerf Inference

MLPerf Inference, from MLCommons, measures how quickly and efficiently a system runs trained models in production-like conditions. While MLPerf Training measures learning speed, MLPerf Inference measures serving speed — the metric that matters most for deployed applications, recommendation engines, chatbots, and edge devices.

The suite includes vision (ResNet, RetinaNet), language (BERT, GPT-J, Llama 2 70B, Mixtral), recommendation (DLRM), and speech models. It defines four scenarios that mirror real deployment patterns: Offline (maximize batch throughput), Server (handle a Poisson stream within a latency bound), Single-Stream (lowest latency for one request), and Multi-Stream (concurrent streams within a deadline).

What It Measures

Metrics depend on scenario. Offline reports throughput in samples per second. Server reports the maximum queries per second sustainable while meeting a strict latency target such as a 99th-percentile bound. Single-Stream and Multi-Stream report latency directly. A separate power category reports energy per query, enabling performance-per-watt comparisons critical for data centers and edge hardware.

Methodology

Submitters use the LoadGen harness, which generates traffic patterns and enforces latency constraints so results are comparable across vendors. Accuracy must stay within a tolerance of the reference model — typically 99% or 99.9% — preventing systems from cheating with overly aggressive quantization. Divisions (Closed and Open) and categories (Datacenter and Edge) separate comparable systems. Results are peer-reviewed before release.

How to Interpret Results

Match the scenario to your use case: Server numbers suit online APIs, Offline suits batch pipelines, Single-Stream suits interactive edge devices. Always check the accuracy target — a 99.9% result is harder than 99%. Compare within the same round and category. Power-category results reveal efficiency that raw throughput hides; a chip with lower peak throughput but far better energy per query may win on total cost of ownership.

Limitations

LoadGen workloads are synthetic traffic, not your exact request mix. Vendor submissions are heavily optimized, so real deployments rarely match published peaks. Quantization and batching tricks valid under the rules may not suit accuracy-sensitive applications. LLM workloads in the suite lag the frontier models teams actually deploy, and edge results vary widely with thermal and power envelopes not fully captured by the benchmark.