ANN-Benchmarks

ANN-Benchmarks is the widely used open-source framework for evaluating approximate nearest neighbor (ANN) search algorithms — the core of vector databases and similarity search. As embeddings and vector search became central to retrieval-augmented generation and semantic search, ANN-Benchmarks became the reference for comparing index libraries such as HNSW, FAISS, Annoy, ScaNN, and the vector engines built on them. Its signature output is the recall-versus-throughput trade-off curve.

What It Measures

The benchmark measures the fundamental tension in approximate search: recall (typically recall@10, the fraction of true nearest neighbors found) against queries per second (QPS) at each recall level. It also reports index build time and index size (memory footprint). Because exact search is slow at scale, every ANN method trades some accuracy for speed; the benchmark quantifies exactly how much.

Methodology

ANN-Benchmarks runs each algorithm over standard public data sets — such as SIFT, GloVe, Deep1B subsets, and others — that come with precomputed ground-truth nearest neighbors. For each algorithm it sweeps the index parameters (for example, HNSW's M and efSearch) to trace a Pareto frontier: for every achievable recall, the highest QPS that method reaches. Runs are containerized for reproducibility and use a single configuration of CPU resources to keep comparisons fair. The output is a recall/QPS plot per data set where curves further toward the upper right (high recall and high throughput) are better, plus tables of build time and memory. The harness is extensible, so vector-database vendors add their engines.

How to Interpret Results

Never compare a single QPS or recall number in isolation — read the whole curve, because an algorithm that wins at recall 0.9 may lose at recall 0.99. Pick the recall level your application actually needs, then compare QPS there. Weigh build time and memory: a fast-querying index that takes hours to build or needs huge RAM may be impractical. Results are data-set-specific (dimensionality and distribution matter), so choose a benchmark data set resembling your embeddings. Note that most runs are single-machine and CPU-based.

Limitations

The benchmark uses static data sets and offline queries, so it does not capture index updates, deletes, filtered search, or distributed scaling — all important in production vector databases. It focuses on the algorithm, not the full database (durability, concurrency, metadata filtering). Hardware and configuration strongly affect results. Use ANN-Benchmarks to choose an index algorithm and understand the recall/throughput trade-off, complemented by end-to-end tests of the actual vector database.