In-Memory Store Benchmark (memtier/redis-benchmark)

In-memory data store benchmarking measures the performance of caches and key-value stores such as Redis, Memcached, KeyDB, and Dragonfly, where data lives in RAM and operations complete in microseconds to low milliseconds. Because these systems are often on the hot path of an application, the benchmark focuses on extremely high operation rates and tight tail latency. The standard tools are redis-benchmark (bundled with Redis) and memtier_benchmark (a more flexible, multi-protocol load generator).

What It Measures

The core metrics are operations per second (throughput across GET, SET, and other commands), latency percentiles especially p99 and p999 in the sub-millisecond range, the hit rate when used as a cache, and how throughput scales with concurrency and pipelining. The defining concern is sustaining millions of operations per second while keeping tail latency low and predictable.

Methodology

memtier_benchmark generates load with configurable client counts, threads, a GET/SET ratio, key and value sizes, key-access patterns (random, sequential, or Gaussian to simulate hot keys), and pipeline depth (how many requests are sent before awaiting replies). It populates the store, then drives the operation mix while recording throughput and a latency histogram. redis-benchmark is simpler, cycling through individual commands at a chosen concurrency and pipeline depth. Pipelining dramatically increases throughput by amortizing network round trips, so it is a key variable. Tests also vary the data-structure commands exercised (strings, hashes, sorted sets) and whether persistence (AOF/RDB) is enabled, since durability settings affect performance.

How to Interpret Results

Always report value size, GET/SET ratio, client/thread count, pipeline depth, and persistence settings — pipelining especially can change throughput by an order of magnitude, so a number without it stated is uninformative. Read p99/p999 latency rather than averages, since cache value comes from predictable tail behavior. Use a realistic key-access distribution (hot keys via Gaussian) to expose contention; uniform access understates real hotspots. Network latency between client and server can dominate, so colocate appropriately and note the setup.

Limitations

Default tests use uniform synthetic keys and simple commands, missing real access skew, large pipelines, and complex data-structure usage. Results are heavily influenced by network, CPU, and persistence configuration, so they are relative rather than absolute. Single-node tests do not reflect cluster behavior, replication, or failover. Use these tools to characterize in-memory store throughput and tail latency under realistic value sizes and access patterns, and validate cluster setups separately. Given how often an in-memory store sits on an application's critical path, accurate sub-millisecond benchmarking with realistic value sizes and pipelining is essential before trusting it with production traffic.