MMLU (Massive Multitask Language Understanding)

MMLU (Massive Multitask Language Understanding) is one of the most widely cited knowledge benchmarks for large language models. Introduced in 2020, it measures whether a model can recall and apply factual knowledge across a broad range of academic and professional domains rather than within a single narrow task.

The benchmark is popular because it correlates reasonably well with general capability and is easy to score: every question is multiple choice with a single correct answer. For several years it was the default headline number labs cited when announcing a new model.

What It Measures

MMLU contains roughly 16,000 multiple-choice questions spread across 57 subjects. Tasks range from elementary mathematics and US history to professional law, clinical medicine, abstract algebra, and machine learning. Each question has four answer options. The headline number is the average accuracy over all subjects.

Because subjects span elementary to graduate and professional difficulty, MMLU probes both breadth of knowledge and the ability to handle specialized terminology.

Methodology

Models are typically evaluated in a few-shot setting, classically five examples per subject prepended to the prompt, though zero-shot evaluation is now common for instruction-tuned models. The model selects one of A, B, C, or D. Scoring compares the chosen letter to the gold answer; partial credit does not exist.

Evaluation harnesses differ in prompt formatting, answer extraction, and whether they read answer log-probabilities or parse generated text. These differences can shift reported scores by several points, so cross-paper comparisons should be treated cautiously.

How to Interpret Results

Random guessing yields 25 percent. Strong frontier models now score in the high 80s to low 90s, approaching the estimated human expert ceiling around 89 percent. Above roughly 88 percent the benchmark saturates: remaining errors are often ambiguous or mislabeled questions rather than genuine knowledge gaps.

Look at per-subject breakdowns rather than just the average. A model may be excellent at humanities but weak at formal logic or college mathematics, and the macro average hides this. For tasks needing harder reasoning, MMLU-Pro or GPQA are better discriminators.

Limitations

MMLU is multiple choice, so it rewards recognition over generation and is vulnerable to answer-position bias and lucky guessing. Some questions contain errors or outdated answers. The dataset is old enough that contamination is a real concern: many models have likely seen the questions during pretraining, inflating scores. Finally, because top models cluster near the ceiling, MMLU no longer separates the best systems well, which is why MMLU-Pro was created.

Practical Use

In practice MMLU is best used as a coarse capability filter and as one input among many, not as a sole ranking. When selecting a model for a knowledge-heavy application, weight the subjects that match your domain rather than the global average, and pair MMLU with harder reasoning tests and a task-specific evaluation on your own data. Because the benchmark is saturated and contaminated, treat a one-point difference as meaningless and look for consistent advantages across several independent benchmarks before drawing conclusions.