BenchBase

BenchBase is an open-source, multi-DBMS benchmarking framework written in Java, developed by the Carnegie Mellon Database Group as the successor to OLTPBench. It provides a single harness for running a large library of standardized workloads against any database accessible over JDBC, making it a research-grade tool for fair, repeatable comparisons across engines.

What It Measures

BenchBase measures throughput (requests per second) and detailed latency distributions, including percentiles, for each transaction type. It supports rate-limited and unlimited modes, and can report goodput (successful transactions) under controlled arrival rates. Because it bundles many workloads, it can characterize OLTP, web, analytical, and microservice-style patterns within one tool.

Methodology

The framework ships with around twenty workloads, including TPC-C, TPC-H, YCSB, TATP (telecom), Wikipedia, Twitter-like (Voter), SmallBank, SEATS (airline reservations), and AuctionMark. Each workload defines a schema, a data loader, and a transaction mix with configurable weights. You provide an XML configuration specifying the JDBC target, scale factor, terminal (client thread) count, target rate, and the per-transaction weights. BenchBase loads data, runs a warm-up, then executes the timed workload and emits throughput and latency results plus raw samples for analysis. Its consistent harness across workloads and engines is its key advantage for comparative studies.

How to Interpret Results

Report the workload, scale factor, terminal count, target rate, and isolation level. Prefer rate-limited runs that measure latency at a fixed offered load over peak-throughput runs, since tail latency is usually what matters. Because BenchBase uses JDBC and a generic harness, it favors portability over native-client tuning, so absolute numbers may trail vendor-specific tools — its strength is relative comparison under identical conditions. Inspect per-transaction latency, since a healthy aggregate can hide a slow transaction type.

Limitations

The JDBC abstraction and Java client can become the bottleneck at very high throughput, understating fast engines. Some bundled workloads are simplified academic models rather than production patterns. As a self-run framework, its results are configuration-dependent and unaudited. Use BenchBase when you need a consistent, scriptable way to compare multiple databases or workloads side by side, and document the full configuration. Its research pedigree and uniform harness make BenchBase especially valuable for academic studies and for teams that need reproducible, apples-to-apples comparisons across several engines or workloads at once. Teams typically pick the two or three bundled workloads closest to their application, run them rate-limited across the candidate engines, and compare tail latency at a shared offered load.