VQAv2 (Visual Question Answering)

VQAv2, the second version of the Visual Question Answering benchmark, tests multimodal understanding: given an image and a free-form question about it, a model must produce a correct natural-language answer. It probes whether a model genuinely combines vision and language rather than guessing from text priors alone.

What It Measures

Questions range from yes/no and counting to open-ended descriptions, for example "What color is the bus?" or "How many people are sitting?" Answers are short phrases. VQAv2's defining design fixes a flaw in the original VQA: for many questions it pairs two similar images that yield different answers, so a model cannot succeed by memorizing that "What sport..." is usually "tennis." This balancing forces the model to actually look at the image.

Methodology

Each question was answered by multiple human annotators, and a predicted answer is scored by how many annotators gave it: full credit if at least a few humans agreed, partial credit otherwise. This soft accuracy tolerates the natural variation in how people phrase short answers. Results are commonly reported overall and split by answer type: yes/no, number, and other. Models are evaluated on held-out splits to prevent memorization.

How to Interpret Results

Look at the answer-type breakdown. Yes/no accuracy is typically highest and least informative; number and other categories are harder and more revealing of true visual reasoning. Counting in particular remains difficult and is a good discriminator. Because of the balanced-pair design, strong VQAv2 scores indicate the model uses visual evidence rather than language bias. Still, compare against unimodal baselines: a model that scores well using text alone signals residual bias in the evaluation.

Limitations

The soft, annotator-vote accuracy can reward common phrasings and penalize correct but unusual answers. Many questions are simple and the benchmark has largely saturated for strong multimodal models, reducing its discriminative power. It emphasizes short factual answers over complex reasoning or explanation. Residual language priors persist despite balancing, so high scores do not fully guarantee deep visual grounding, and newer multimodal benchmarks now target harder reasoning.