DROP (Discrete Reasoning Over Paragraphs)

DROP (Discrete Reasoning Over Paragraphs) is a reading-comprehension benchmark that demands more than extracting a span of text. To answer, a model must perform discrete reasoning operations over the content of a passage, combining comprehension with computation.

What It Measures

DROP contains about 96,000 questions over Wikipedia paragraphs, many drawn from sports summaries and historical text. Answering requires operations such as addition and subtraction, counting, sorting, comparison, and selecting multiple spans, often chaining several together.

The benchmark measures whether a model can read a passage, locate the relevant facts, and then manipulate them numerically or logically, for example computing how many years separated two events or which of several entities had the highest value.

Methodology

Questions were crowd-authored adversarially against a strong reading-comprehension model so that simple span extraction would fail. Answers may be numbers, dates, or one or more text spans.

Scoring uses exact match and a numeracy-aware F1 that handles numbers and multi-span answers, tolerating minor formatting differences. Models are evaluated zero-shot or few-shot and may use chain-of-thought to perform the intermediate calculations before producing the final answer.

How to Interpret Results

Report F1 as the primary metric since it credits partially correct multi-span answers and handles numeric equivalence; exact match is stricter and noisier. A model strong on plain reading comprehension but weak on DROP signals a gap in combining retrieval with arithmetic or logical operations.

Frontier models with chain-of-thought now score very high, approaching human performance, so DROP is largely saturated at the top but remains a useful diagnostic for numerical reading reasoning in smaller models.

Limitations

Numeracy-aware scoring still mismarks some valid answers due to formatting or alternative phrasings, and exact match is harsher still. The passages skew toward sports and history, narrowing the reasoning styles tested. The adversarial authoring occasionally yields ambiguous questions. As an older, popular dataset, contamination likely inflates current scores, and saturation limits its ability to separate top models.

Practical Use

Report F1 as the primary DROP metric, since it credits partially correct multi-span answers and numeric equivalence that exact match penalizes. DROP is a useful diagnostic for whether a model can combine reading with arithmetic and logic, especially in smaller models where saturation has not set in. Because passages skew toward sports and history, supplement it with domain-relevant comprehension tests for your use case, and disclose whether chain-of-thought was used, since it strongly affects the intermediate calculations.