MMBench (Multimodal Benchmark)

MMBench is a multimodal benchmark for vision-language models that aims to be systematic and robust. Rather than a single score, it decomposes multimodal competence into many fine-grained abilities and tests each with multiple-choice questions, while adding a procedure to guard against models guessing or exploiting answer position.

What It Measures

MMBench organizes questions into a hierarchy of abilities, including perception skills such as object localization, attribute recognition, and OCR, and reasoning skills such as relation reasoning, attribute comparison, and logical inference over an image. Each question is multiple choice with one correct option. Reporting per-ability accuracy produces a profile that shows where a model is strong or weak, which a single aggregate score would hide.

Methodology

A distinctive feature is CircularEval. For each multiple-choice question, the benchmark presents it several times with the answer options cyclically shuffled, and counts the model correct only if it answers correctly every time. This penalizes lucky guesses and position bias, where a model favors a particular option slot. MMBench also uses an LLM to map free-form model outputs to one of the choices, so models that do not emit clean option letters are still scored fairly. Both English and Chinese versions exist to test multilingual multimodal ability.

How to Interpret Results

Use CircularEval accuracy as the trustworthy number; plain accuracy overstates ability because of guessing. Read the per-ability breakdown to match a model to a use case, for example prioritizing OCR scores for document tasks or relation reasoning for scene understanding. A model with high perception but weak reasoning scores can describe images but struggles to infer from them. Compare English and Chinese splits if multilingual deployment matters.

Limitations

Multiple-choice format constrains the kinds of competence measured and can be easier than open-ended generation. The LLM-based choice extraction introduces a small dependency on the extractor's reliability. Ability categories are a useful but imperfect taxonomy, and some questions span several skills. As a widely used benchmark it is exposed to contamination and targeted tuning, so strong MMBench results should be confirmed with open-ended and task-specific evaluation.