SimpleQA

SimpleQA is a benchmark built to measure factual accuracy on short, closed questions. Each question has a single, indisputable answer that was correct at creation time, such as a specific date, name, or number. The benchmark targets a persistent weakness in language models: confidently stating false facts, also called hallucination.

What It Measures

The core question is whether a model knows a fact and whether it knows when it does not. SimpleQA grades each response into one of three buckets: correct, incorrect, or not attempted. The not-attempted category captures appropriate abstention, where the model says it is unsure rather than guessing. From these, headline rates and an F-score that balances correctness against attempt rate are computed, rewarding models that are both accurate and well-calibrated about their uncertainty.

Methodology

Questions are written to be hard enough that strong models still miss many, and each is verified against authoritative sources with a single accepted answer. Responses are graded by an automated judge that classifies them as correct, incorrect, or not attempted, using the reference answer. Because answers are short and unambiguous, grading is far more reliable than for open-ended generation. The design deliberately stresses calibration: a model that always guesses can raise its correct rate but also raises its incorrect rate.

How to Interpret Results

Do not read correct rate alone. A high correct rate paired with a high incorrect rate indicates a model that guesses aggressively and will mislead users. Compare the incorrect rate and the not-attempted rate together to judge calibration: a trustworthy model abstains when uncertain instead of fabricating. The F-score gives a single calibration-aware number. For applications where wrong answers are costly, prioritize low incorrect rate over raw coverage.

Limitations

SimpleQA covers only short, single-answer facts, so it ignores reasoning, synthesis, and long-form factuality. Facts can drift over time, so an answer correct at creation may become outdated, requiring dataset maintenance. The benchmark skews toward obscure trivia to avoid saturation, which may not reflect the questions real users ask. It also does not test retrieval-augmented setups, where external sources change the factuality picture entirely.