BigANN Benchmark

The BigANN Benchmark (the Billion-Scale Approximate Nearest Neighbor challenge) extends vector-search benchmarking to data sets of a billion or more vectors. Introduced as a NeurIPS competition, it addresses a gap left by smaller benchmarks: how ANN algorithms behave at the scale where memory, disk, and hardware constraints dominate. It is the reference for evaluating vector indexes intended for very large corpora, including disk-based and accelerator-assisted methods.

What It Measures

BigANN measures recall (such as recall@10) against throughput (QPS) at billion-vector scale, under explicit resource constraints — bounded memory, fixed hardware, or a build-time budget depending on the track. It also reports index build time and memory/disk footprint. The central challenge is achieving high recall and throughput when the data far exceeds what fits comfortably in RAM, forcing algorithms to use compression, disk, or specialized hardware.

Methodology

The benchmark provides several billion-scale public data sets (such as BIGANN/SIFT1B, Deep1B, Microsoft Turing-ANNS, and text-to-image sets) with ground-truth neighbors for a query set. Competitors implement an index that must build within a time limit and run within a fixed hardware envelope. There are multiple tracks: a standard track on a defined machine, a track that caps memory to force disk-based or compressed indexes, and tracks that allow custom hardware or accelerators. Each submission is evaluated by sweeping search parameters to find the best QPS at a required recall threshold, then ranked. Reproducibility is enforced through standardized environments and public code.

How to Interpret Results

Always read results within their track and constraints — a result from an unconstrained-hardware track is not comparable to a memory-limited one. Focus on QPS at the recall level your application requires, and weigh it against the memory or disk budget, since billion-scale deployment cost is dominated by footprint. Build time matters when data is frequently re-indexed. Check which data set was used, because text-to-image and SIFT data have very different difficulty. The constrained tracks best reflect realistic production trade-offs.

Limitations

The benchmark uses static data sets and offline batch queries, so it does not measure incremental updates, deletes, filtered or hybrid search, or distributed serving. It emphasizes the index algorithm rather than full database features and operability. Hardware tracks make some results hard to compare. Use BigANN to evaluate large-scale vector index algorithms and their memory/throughput trade-offs, complementing it with end-to-end production tests of the actual system. As production retrieval systems grow into the billions of vectors, BigANN's constrained tracks have become an important reference for the cost-efficiency frontier of large-scale similarity search.