MATH (Competition Mathematics)

The MATH benchmark measures advanced mathematical problem solving using competition-level problems. It is far harder than GSM8K and was designed to test genuine mathematical reasoning rather than simple arithmetic.

What It Measures

MATH contains 12,500 problems drawn from high school mathematics competitions such as AMC and AIME. Problems span seven subjects, including algebra, geometry, number theory, counting and probability, precalculus, and intermediate algebra, and are tagged with difficulty levels from 1 to 5.

The benchmark measures whether a model can produce a complete, correct derivation for problems that often require creative steps, case analysis, or knowledge of specific techniques. Each problem has a well-defined final answer, frequently expressed in LaTeX.

Methodology

Evaluation typically uses chain-of-thought prompting so the model shows its work. Scoring is by exact match on the final boxed answer after normalization to handle equivalent LaTeX forms, fractions, and ordering. Because answers can be expressions rather than integers, robust answer parsing is important and a source of scoring noise.

Reasoning models often draw multiple samples and use self-consistency or verifier models. A widely used subset, MATH-500, provides a smaller, stable test split for faster comparison.

How to Interpret Results

For years MATH was a strong differentiator, with early models scoring under 10 percent. Modern reasoning-focused models now exceed 90 percent on MATH-500, so it is approaching saturation at the top. Per-difficulty breakdowns are informative: level-5 accuracy separates strong reasoners far better than the overall average.

Because results depend on prompting and answer normalization, compare models within the same harness. Large gaps between a model's plain and reasoning-mode scores reveal how much it benefits from extended deliberation.

Limitations

Exact-match scoring can mismark correct answers written in unexpected forms and gives no credit for sound reasoning with a final slip. The benchmark only checks final answers, not proof validity, so a model can guess or reach the right number through flawed logic. Contamination is a serious concern given competition problems circulate widely online. As top models saturate it, olympiad-level sets such as AIME and harder proof benchmarks have become the new frontier.

Practical Use

MATH and its MATH-500 subset remain useful for comparing mid-tier and specialized math models, even as the very top saturates. Because answer normalization drives scoring noise, use a well-tested grader and report which subset and harness you used. For frontier comparison, supplement MATH with AIME and proof-oriented benchmarks. Treat MATH as evidence of derivation skill at competition level, not as proof a model can produce rigorous proofs, which it does not check.