TPCx-AI

TPCx-AI is an express benchmark from the Transaction Processing Performance Council that measures end-to-end machine-learning and data-science pipelines. Unlike benchmarks that time a single trained model, TPCx-AI evaluates the whole lifecycle — data generation, ingestion and preprocessing, model training, and model serving — to reflect how analytics platforms run AI workloads in practice.

What It Measures

The primary metric is AIUCpm@SF (AI use-case throughput per minute at a scale factor), with price/performance and energy variants. The benchmark defines ten representative use cases drawn from retail and finance, spanning classification, regression, clustering, time-series forecasting, recommendation, computer vision, and natural-language tasks. For each, it measures both the training phase and the serving (inference) phase, while also requiring that models meet a minimum quality threshold so speed cannot be bought by sacrificing accuracy.

Methodology

TPCx-AI ships as a runnable kit that generates a synthetic but realistic data set at a chosen scale factor and provides reference implementations of the ten use cases using common ML frameworks. A compliant run executes the pipeline end to end: load and preprocess data, train each model, validate that it meets the accuracy/quality gate, then run a serving throughput test. The power phase runs use cases serially and the throughput phase runs them concurrently, combined into the AIUCpm metric. Because it covers the entire pipeline, hardware accelerators, data I/O, and framework efficiency all influence the score. Official results are audited and published with full disclosure.

How to Interpret Results

Report the scale factor and examine the balance between training and serving, since platforms differ sharply in each. The accuracy gate matters: confirm that reported throughput came from models meeting the quality threshold. Price/performance and energy figures are important for capacity planning, especially with GPUs. Because the kit standardizes implementations, the benchmark compares platforms and hardware rather than novel algorithms; do not read it as a measure of state-of-the-art model quality.

Limitations

The fixed reference implementations limit how much algorithmic innovation the benchmark can reward, and the ten use cases may not match a given organization's AI workload. Full runs are operationally heavy, so published results are limited. The synthetic data and capped accuracy targets are deliberately modest. Use TPCx-AI to compare the end-to-end ML pipeline performance and efficiency of platforms and hardware, not to evaluate cutting-edge model accuracy. By forcing the full pipeline and an accuracy gate into one score, it captures a dimension of machine-learning performance that single-model timing benchmarks miss entirely.