BEIR (Benchmarking Information Retrieval)

BEIR (Benchmarking-IR) is a benchmark for information retrieval that emphasizes zero-shot generalization. Instead of measuring a model trained and tested on one dataset, BEIR asks whether a single retriever can find relevant documents across many domains it never trained on. This mirrors real deployments, where a search system must handle questions, claims, and queries it has never seen.

What It Measures

BEIR bundles datasets covering fact checking, question answering, bio-medical search, news, scientific citation, duplicate detection, and entity retrieval, among others. Query styles range from natural questions to short keyword queries and full claims. The headline metric is nDCG@10, which rewards placing relevant documents near the top of the ranked list. Recall@100 and MAP are also reported to capture deeper retrieval quality.

Methodology

Each task provides a corpus, a set of queries, and relevance judgments. A retrieval model encodes the corpus once and ranks documents for each query, with no fine-tuning on the target dataset. BEIR supports lexical baselines such as BM25, dense bi-encoders, sparse learned retrievers, and reranking pipelines. Because all systems run the same datasets and judgments, scores are directly comparable. The benchmark deliberately includes corpora of very different sizes and writing styles to stress generalization.

How to Interpret Results

Average nDCG@10 across tasks summarizes generalization, but the spread across tasks is revealing. A dense model may beat BM25 on average yet lose on long-tail or keyword-heavy datasets, which is why hybrid lexical-plus-dense systems are common in production. Compare candidates on the specific BEIR tasks closest to your domain. Watch index size and query latency too: strong rerankers boost nDCG but add cost per query.

Limitations

Relevance judgments in some BEIR datasets are sparse, so a genuinely relevant document not labeled as such is scored as a miss, penalizing models that surface novel results. Several included corpora are older web or academic text and may not reflect current language. Because BEIR is widely used, models are sometimes tuned to resemble its datasets, weakening the zero-shot claim. Use it to shortlist retrievers, then validate on your own labeled queries.