MTEB (Massive Text Embedding Benchmark)

MTEB, the Massive Text Embedding Benchmark, is the de facto standard for comparing text embedding models. An embedding maps text to a dense vector so that semantic similarity becomes geometric closeness. MTEB tests whether those vectors are useful across many downstream tasks rather than a single one, which is important because embeddings are reused for search, clustering, deduplication, and retrieval-augmented generation.

What It Measures

MTEB aggregates dozens of datasets grouped into task types: retrieval, reranking, clustering, pair classification, classification, semantic textual similarity (STS), and summarization. Each task type has its own primary metric. Retrieval uses nDCG@10, STS uses Spearman correlation, clustering uses V-measure, and classification uses accuracy or F1. A model receives a per-task score and an overall average that ranks it on the public leaderboard.

Methodology

A model under test produces a fixed-size vector for each input text using a standard pooling strategy. The benchmark harness then runs each task with frozen embeddings: no fine-tuning is allowed, so the test measures the general quality of the representations. Retrieval tasks encode a corpus and a set of queries, then rank documents by cosine similarity. Classification tasks train a lightweight logistic-regression head on top of the frozen vectors. Results are reproducible because the datasets, splits, and evaluation code are fixed and open source.

How to Interpret Results

Read the overall average for a quick ranking, but always inspect the task breakdown. A model can top the average yet trail on retrieval, which is what most production search and RAG systems actually need. Compare scores at equal embedding dimension and sequence length, since larger vectors and longer context windows raise cost and latency. Also weigh model size and inference speed: a small model close to the leader is often the better engineering choice.

Limitations

MTEB is dominated by English and by academic datasets, so high scores may not transfer to domain-specific corpora such as legal or medical text. Leaderboard pressure encourages training on data similar to the benchmark, inflating numbers. The frozen-embedding protocol ignores gains from task-specific fine-tuning. Finally, a single average hides the trade-off between general-purpose quality and specialization, so teams should re-evaluate shortlisted models on their own data before committing.