GSM8K (Grade School Math 8K)
GSM8K tests multi-step grade-school arithmetic reasoning with single numeric answers and is a classic setting for chain-of-thought evaluation. Frontier models have largely saturated it, shifting hard-math evaluation to MATH and AIME.
GSM8K (Grade School Math 8K) is a benchmark of grade-school math word problems created to measure multi-step arithmetic reasoning in language models. Despite the elementary subject matter, it was for years a strong discriminator because solving the problems requires chaining several reasoning steps without arithmetic slips.
What It Measures
GSM8K contains about 8,500 problems written by human authors, split into roughly 7,500 training and 1,000 test items. Each problem is a short word problem solvable in two to eight steps using basic arithmetic: addition, subtraction, multiplication, and division. The target is a single integer or simple numeric answer.
The benchmark measures whether a model can parse a natural-language scenario, identify the relevant quantities, and execute a correct sequence of operations. It is a test of reasoning reliability more than of mathematical sophistication.
Methodology
Each training example includes a natural-language solution showing intermediate steps, which makes GSM8K a canonical setting for chain-of-thought prompting. Models are scored by exact match on the final numeric answer; the reasoning trace is not graded directly.
Common evaluation variants include zero-shot chain-of-thought, few-shot with worked examples, and self-consistency, where multiple samples are drawn and the majority answer is taken. Self-consistency typically raises accuracy several points. A harder adversarial variant, GSM-Symbolic, perturbs numbers and phrasing to test robustness.
How to Interpret Results
Early large models scored well under 50 percent. Modern frontier and reasoning models exceed 95 percent, so GSM8K is now largely saturated for grade-school arithmetic. A high score confirms basic numeric reasoning competence but no longer differentiates strong models.
Watch for the gap between direct-answer and chain-of-thought accuracy: a large gap indicates the model relies heavily on explicit step-by-step reasoning. For harder math, MATH or AIME are the appropriate successors.
Limitations
Because it is saturated, GSM8K no longer separates top models. Exact-match scoring can penalize correct reasoning with formatting differences and can credit lucky guesses. The problems are short and templated, so high scores may not generalize to longer or trickier reasoning. Contamination is likely given the dataset's age, and studies using perturbed variants show some apparent gains were brittle pattern matching rather than robust reasoning.
Practical Use
Today GSM8K functions mainly as a regression and sanity check: a capable general model should clear it easily, so a low score signals a basic reasoning or formatting problem rather than a frontier limitation. For applications that involve real numeric reasoning, validate on harder sets like MATH or AIME and on your own representative problems. When reporting GSM8K, note whether you used chain-of-thought, self-consistency, and how many samples, since these choices materially shift the headline number.