AIME (Competition Math Benchmark)

AIME refers to the use of American Invitational Mathematics Examination problems as a benchmark for advanced mathematical reasoning. As MATH and GSM8K saturated, AIME became a leading frontier test because its problems are genuinely olympiad-level and resist guessing.

What It Measures

The AIME is a prestigious 15-question exam for top high-school competitors in the United States. Every answer is an integer from 0 to 999, which makes automatic grading clean while keeping the problems extremely hard. Benchmark sets typically use recent AIME contests, such as AIME 2024 and 2025, each contributing a small number of problems.

The benchmark measures deep, multi-step mathematical reasoning: setting up the right approach, executing long derivations without error, and arriving at the exact integer answer. It strongly differentiates reasoning-optimized models.

Methodology

Models solve each problem, usually with extended chain-of-thought, and the final integer is checked for exact match. Because answers are integers in a fixed range, scoring is unambiguous and free of the LaTeX-normalization headaches that affect MATH.

Since the problem count is tiny, evaluations almost always sample multiple solutions per problem and report pass@1 averaged over many samples, or consensus@k (majority vote). Reporting the number of samples and the specific contest year is essential for fair comparison.

How to Interpret Results

Due to the small problem set, single runs are very noisy; trust averaged pass@1 over many samples rather than one-shot numbers, and note the contest year because difficulty varies. Reasoning models that spend many tokens deliberating tend to dominate here, so scores reflect inference effort as well as capability.

Using the most recent contest reduces contamination, since problems released after a model's cutoff cannot have been memorized. A strong AIME score is currently one of the better signals of elite mathematical reasoning.

Limitations

The small number of problems makes scores high-variance and sensitive to sampling settings. Heavy reliance on chain-of-thought means results depend on token budget and decoding, conflating capability with compute. Older contests are likely contaminated. Integer-answer matching gives no credit for correct methods with a final slip and cannot detect a right answer reached by flawed reasoning.

Practical Use

Because the problem set is tiny, report averaged pass@1 over many samples or consensus@k rather than single runs, and always name the contest year. Use the most recent contests to limit contamination, and disclose token budgets, since reasoning models trade compute for accuracy here. AIME is one of the best current signals of elite mathematical reasoning, but treat it as one data point among several; combine it with GPQA and broad benchmarks before concluding a model is generally strong.