ARC (AI2 Reasoning Challenge)

ARC (AI2 Reasoning Challenge), from the Allen Institute for AI, is a multiple-choice question-answering benchmark based on grade-school science. It was designed to require reasoning and knowledge composition rather than simple retrieval, addressing the weakness of earlier QA datasets that surface-matching could solve.

What It Measures

ARC contains about 7,800 natural science questions written for students in grades three through nine. It is split into an Easy set and a Challenge set. The Challenge set specifically collects questions that both a retrieval-based and a word-association baseline answered incorrectly, so they demand genuine reasoning.

The benchmark measures scientific reasoning and the ability to combine facts, such as understanding causation, properties of materials, or simple experimental logic, to select the right answer among typically four options.

Methodology

Each question is multiple choice. Models are scored by accuracy, often using answer-likelihood comparison or parsed choices, in zero-shot or few-shot settings. The Challenge set is the meaningful number to report because the Easy set is now trivial for capable models.

ARC also ships with a large supporting corpus of science text, originally intended to help retrieval-augmented systems, though modern LLMs usually answer from parametric knowledge alone. The ARC-Challenge accuracy is the standard headline figure.

How to Interpret Results

Report ARC-Challenge, not the combined or Easy score, since Easy is saturated. For years the Challenge set was difficult, but strong models now score in the mid-90s, so it too is approaching saturation among frontier systems.

ARC remains useful as a clean, well-curated check of basic scientific reasoning, especially for smaller or specialized models, and as part of broader benchmark suites. Treat very high scores as confirmation of competence rather than as a meaningful ranking signal.

Limitations

The main limitation is saturation: top models leave little headroom, so ARC no longer separates them. Multiple-choice format permits guessing and answer-position effects. As an established, widely distributed dataset, contamination likely inflates current scores. Finally, grade-school science is a narrow slice of reasoning, so high ARC accuracy does not indicate strength on advanced scientific or quantitative tasks, where GPQA is more appropriate.

Practical Use

Always report ARC-Challenge rather than the Easy or combined score, and treat very high results as confirmation of competence rather than a ranking signal among strong models. ARC is well curated, so it remains a clean inclusion in evaluation suites and a fair test for smaller or specialized models. For advanced scientific reasoning, move to GPQA, which preserves headroom that ARC has largely lost. Note the scoring method, since likelihood selection and parsed choices can differ.