DocVQA (Document Visual Question Answering)

DocVQA, Document Visual Question Answering, tests whether a model can answer natural-language questions about images of documents. Documents combine printed and handwritten text, tables, forms, and figures arranged in complex layouts, so success requires reading text, understanding spatial structure, and reasoning over it together.

What It Measures

Given a scanned or photographed document and a question such as "What is the invoice total?" or "Who signed the form?", the model must return the answer, usually a short span found in or derived from the document. The benchmark stresses optical character recognition, layout comprehension (associating a value with its label or column), and lightweight reasoning. Variants extend the task to multi-page documents and to collections, where the model must also locate the relevant page or file.

Methodology

DocVQA's primary metric is ANLS, Average Normalized Levenshtein Similarity. Rather than requiring an exact string match, ANLS measures edit-distance similarity between the predicted and ground-truth answer, awarding near-full credit for minor OCR or formatting differences and zero below a similarity threshold. This is important because document answers often hinge on exact tokens that OCR can slightly garble. Models are evaluated on held-out document sets, and scores are sometimes broken down by document type such as forms, tables, or figures.

How to Interpret Results

ANLS rewards getting the answer almost exactly right, so a high ANLS indicates reliable extraction, not just plausible guessing. Examine document-type breakdowns: table and form questions test layout grounding hardest, while free-text questions lean on reading and reasoning. For production document automation, prioritize models strong on the document types you actually process. Watch performance on long or multi-page documents, where retrieval of the relevant region becomes the bottleneck.

Limitations

ANLS tolerance can occasionally credit a near-miss that is functionally wrong, for example a transposed digit in a financial figure. The benchmark's documents skew toward certain languages and formats, so results may not transfer to other scripts or to highly degraded scans. Short-span answers under-test multi-step reasoning and synthesis across a document. As multimodal models improve, simpler DocVQA splits saturate, pushing evaluation toward harder multi-page and infographic variants.