MGSM (Multilingual Grade School Math)

MGSM, Multilingual Grade School Math, measures mathematical reasoning across languages. It translates a fixed set of grade-school math word problems into many languages, including high-resource ones like Spanish and Chinese and lower-resource ones like Swahili and Bengali. The goal is to separate genuine reasoning ability from English-only fluency.

What It Measures

Each problem is a short word problem that needs several arithmetic steps to solve. The model must read the problem in the target language, reason about quantities and relationships, and produce the correct final number. Scores are reported per language and as an average, exposing how much a model's reasoning degrades when it leaves English.

Methodology

Problems are drawn from a common pool and human-translated to keep the underlying math identical across languages, so differences in score reflect language handling rather than problem difficulty. Evaluation typically uses chain-of-thought prompting, where the model is asked to show its working before giving the answer, and the final numeric answer is extracted and compared by exact match. A key research question MGSM probes is whether reasoning in English (translate, then solve) beats reasoning natively in the source language.

How to Interpret Results

Compare the average across languages and the spread between the strongest and weakest. A small spread indicates robust multilingual reasoning; a large gap suggests the model leans heavily on English. Check low-resource languages specifically, since they reveal whether multilingual training is broad or shallow. Because MGSM uses chain-of-thought, results also reflect the model's ability to follow a reasoning format consistently in each language.

Limitations

MGSM is small and focused on simple arithmetic, so it does not capture advanced mathematics or domain reasoning. Saturation is a concern: leading models now score very high on high-resource languages, compressing differences. Translation quality and answer-extraction heuristics can introduce noise, especially for languages with different numeral or formatting conventions. Treat MGSM as a multilingual reasoning sanity check rather than a comprehensive math evaluation.