MMLU-Pro

MMLU-Pro is a 2024 redesign of the original MMLU benchmark, built specifically because frontier models had pushed MMLU scores near its ceiling and could no longer be distinguished by it. It raises difficulty and reduces the value of guessing.

What It Measures

MMLU-Pro measures applied reasoning across academic and professional domains, not just factual recall. It contains around 12,000 questions consolidated into 14 broad disciplines such as mathematics, physics, chemistry, law, engineering, health, and economics. Many questions require multi-step reasoning rather than a single lookup.

The key structural change is the answer set: each question offers ten options instead of four. This drops the random-guess baseline from 25 percent to 10 percent and reduces the chance of accidentally correct answers.

Methodology

The authors filtered the original MMLU to remove trivial, noisy, or ambiguous items, then added harder questions drawn from STEM textbooks and other exam sources. Distractor options were expanded and made more plausible so that elimination strategies are less effective.

Models are usually evaluated with chain-of-thought prompting because the questions reward explicit reasoning. The benchmark is also notably less sensitive to prompt wording than MMLU, which the authors highlight as a stability improvement. Answers are extracted from generated text and matched against the gold option.

How to Interpret Results

Scores on MMLU-Pro are substantially lower than on MMLU for the same model, often 15 to 25 points lower, which is the point: the headroom restores discrimination among top systems. Reasoning-optimized models that use long chains of thought tend to gain the most here relative to their MMLU scores.

Compare models on the same evaluation harness and prompting style. Because chain-of-thought matters, a model's MMLU-Pro number reflects both knowledge and reasoning effort, so token budget and decoding settings influence results.

Limitations

MMLU-Pro remains multiple choice, so it still does not test open-ended generation. Its heavier STEM weighting means it is not a balanced measure of all knowledge. Like any public benchmark, contamination over time will erode its value. Finally, its reliance on chain-of-thought makes results sensitive to inference configuration, complicating fair comparison.

Practical Use

MMLU-Pro is most valuable when you need to distinguish strong models that have saturated MMLU, especially for STEM and professional workloads. Report the evaluation harness, prompting style, and decoding settings alongside the score, since chain-of-thought makes results sensitive to inference configuration. Use it together with GPQA and domain-specific evaluations rather than alone, and remember that its STEM weighting means it is not a balanced proxy for every kind of knowledge or reasoning your application may require.