AGIEval

AGIEval is a benchmark that evaluates language models using real human standardized exams. Its premise is that tests designed to assess human aptitude provide a meaningful, human-centric yardstick for general intelligence in models.

What It Measures

AGIEval draws questions from high-stakes official exams, including college entrance tests such as the SAT and the Chinese Gaokao, law school admission tests, lawyer qualification exams, math competitions, and civil-service exams. It spans both English and Chinese.

The benchmark measures general reasoning, knowledge application, and problem solving as humans are actually evaluated, covering verbal, quantitative, logical, and analytical skills. Because the exams are official, performance can be compared against real human score distributions.

Methodology

The dataset reuses authentic exam questions, mostly multiple choice with some open-ended math items. Models are evaluated zero-shot and few-shot, and chain-of-thought prompting is commonly applied to the reasoning-heavy and math sections.

Scoring is accuracy, by option match or answer matching, reported per exam and as an aggregate. Some sections, especially competition math, are graded by exact match on the final answer. The bilingual design allows separate English and Chinese analysis.

How to Interpret Results

Use per-exam breakdowns: AGIEval's value is showing where a model stands relative to human test-takers on specific assessments, such as legal reasoning versus quantitative aptitude. The human baseline lets you judge whether a model performs at, above, or below typical human levels on each exam.

Because sections differ in format and difficulty, the aggregate score is a rough summary; the per-exam and per-language views are more informative. Hold prompting style constant when comparing models.

Limitations

Using real, published exams makes contamination especially likely, since these tests circulate widely online and may appear in training data, inflating scores. The multiple-choice majority permits guessing. Coverage is shaped by which exams were available, skewing toward admissions and legal content. Translation and formatting of original exam items can introduce noise, and exam-style aptitude is only a proxy for general capability.

Practical Use

Lean on AGIEval's per-exam and per-language breakdowns rather than the aggregate, since sections differ widely in format and difficulty and the human baseline is exam-specific. Because it reuses widely published exams, treat results cautiously as a likely upper bound inflated by contamination, and prefer recent or private exam material when you need a clean read. Hold the prompting style constant across models, and use AGIEval as a human-relatable supplement to contamination-resistant benchmarks rather than a primary ranking source.