SPECjbb 2015

SPECjbb 2015, from the Standard Performance Evaluation Corporation, benchmarks Java application server performance. It models the business logic of a worldwide supermarket company — point-of-sale transactions, online purchases, supplier management, and data-mining queries — exercising the JVM, garbage collector, and underlying hardware under realistic enterprise load.

It replaced the older SPECjbb2005 and is designed for both single-system and multi-JVM, multi-host topologies. Because it stresses the full Java runtime rather than microbenchmarks, it is widely used to compare server CPUs, JVMs, and garbage-collection strategies for transaction-heavy backends.

What It Measures

SPECjbb reports two primary metrics. max-jOPS is the maximum throughput in jBB operations per second the system sustains. critical-jOPS is the throughput achievable while meeting a set of strict response-time service-level agreements at several percentiles (for example 10ms, 25ms, 50ms, 75ms, 100ms). The pairing rewards systems that are both fast and predictable, since a high peak throughput is worthless if tail latency violates SLAs.

Methodology

The benchmark drives an injection rate that ramps until the system saturates, recording throughput and response-time distributions throughout. It runs in fixed configurations: a single combined JVM, a multi-JVM setup on one host, or a distributed configuration across hosts. Submitters must publish full hardware, OS, and JVM tuning details. Results are reviewed before publication on SPEC's site.

How to Interpret Results

Use max-jOPS to gauge raw capacity and critical-jOPS to gauge usable capacity under SLA pressure. A large gap between the two signals latency problems, often from garbage-collection pauses — relevant when comparing collectors like G1, ZGC, or Shenandoah. Compare only configurations of the same topology and SPECjbb version. Normalize by core count, socket count, or power to judge efficiency rather than absolute size.

Limitations

The supermarket model is a synthetic workload and may not reflect a specific application's allocation and threading patterns. Results are highly sensitive to JVM flags and heap tuning, so published numbers represent expert-tuned configurations. The benchmark does not exercise I/O, networking, or database back ends heavily, so it isolates CPU and JVM behavior rather than full-stack performance.