GPQA (Graduate-Level Google-Proof Q&A)

GPQA (Graduate-Level Google-Proof Q&A) is a benchmark of very hard science questions written by domain experts. It targets the upper end of difficulty, where even skilled non-experts with internet access struggle, making it a strong discriminator for frontier reasoning models.

What It Measures

GPQA contains 448 multiple-choice questions in biology, physics, and chemistry, authored and validated by PhD-level experts. The questions are deliberately Google-proof: finding the answer by searching is hard because it requires understanding specialized concepts, not locating a fact.

The benchmark measures deep, graduate-level scientific reasoning. A frequently cited subset, GPQA Diamond, contains the highest-quality, most difficult questions where expert agreement is strong, and is the usual headline.

Methodology

Each question was written by an expert, then validated by other experts in the same field and by non-experts who were allowed extensive web search. Items were kept only if domain experts answered them correctly while skilled non-experts mostly failed even with search, calibrating real difficulty.

Evaluation is multiple choice, scored by accuracy. Reasoning models typically use chain-of-thought and may sample multiple times. Because the set is small, scores carry meaningful variance, so multiple runs and the Diamond subset are commonly reported.

How to Interpret Results

Expert humans in their own field score around 65 to 75 percent; skilled non-experts with web access score far lower. A model exceeding the non-expert baseline shows real reasoning rather than retrieval, and top reasoning models now match or surpass expert-level accuracy on Diamond.

GPQA is one of the better current discriminators among elite models, so gains here are meaningful. Because of small size, treat differences of a few points cautiously and prefer averaged, multi-run results.

Limitations

The small question count makes scores noisy and sensitive to run-to-run variance. Coverage is limited to three science fields, so it does not measure general capability. Multiple-choice format still permits guessing. As frontier models approach expert accuracy, even GPQA is beginning to lose headroom, and its popularity raises future contamination risk despite its Google-proof design.

Practical Use

Because GPQA is small, report averaged accuracy over multiple runs and prefer the Diamond subset, and treat differences of a few points as noise. It is currently one of the better discriminators among elite reasoning models, so meaningful gains here are informative. Disclose sampling and prompting settings, since chain-of-thought and multiple-sample strategies drive results. For a fuller capability picture, combine GPQA with broad knowledge benchmarks and task-specific evaluation, as three science fields cannot represent general competence.