MMMU (Massive Multi-discipline Multimodal Understanding)

MMMU (Massive Multi-discipline Multimodal Understanding) evaluates multimodal models on college-level problems that mix text with images. It was created to test expert-level reasoning over visual material, going well beyond simple image captioning or object recognition.

What It Measures

MMMU contains around 11,500 questions spanning six broad disciplines, including art and design, business, science, health and medicine, humanities, and engineering, across roughly 30 subjects. Questions incorporate over 30 image types: diagrams, charts, chemical structures, medical scans, sheet music, maps, and more.

The benchmark measures whether a model can perceive specialized visual content and reason about it at a university level, combining domain knowledge with image interpretation rather than treating the picture as decoration.

Methodology

Questions are collected from textbooks, exams, and course materials, and include both multiple-choice and open-ended formats interleaved with one or more images. Models must read the figures and the text together to answer. Scoring is accuracy, by option match for multiple choice and by answer matching for open-ended items.

Evaluation is typically zero-shot for instruction-tuned multimodal models. A harder, refined version, MMMU-Pro, adds more candidate options and a vision-only setting where the question text is embedded in the image, reducing shortcuts.

How to Interpret Results

MMMU is a leading measure of multimodal capability, so per-discipline breakdowns reveal where a model's visual reasoning is strong or weak, for example charts versus medical imagery. Expert humans score in the high 80s; strong multimodal models have climbed into the 70s and beyond, narrowing but not closing the gap.

Use MMMU-Pro when comparing top models, since the original is becoming easier and the vision-only setting better isolates genuine image understanding from text shortcuts.

Limitations

Many questions can be partially answered from text alone, so the original benchmark sometimes overstates visual reasoning, a gap MMMU-Pro targets. Coverage skews toward exam-style content, which may not reflect practical multimodal tasks. Image quality and rendering can affect results. Multiple-choice items allow guessing, and as a public dataset MMMU faces growing contamination risk.

Practical Use

For comparing top multimodal models, prefer MMMU-Pro, whose extra options and vision-only setting reduce the text-only shortcuts that inflate original MMMU scores. Read per-discipline results to see where visual reasoning is strong, such as charts versus medical imagery, rather than relying on the aggregate. Validate on your own image types before deployment, since exam-style figures may not match production content, and ensure your serving pipeline renders images at sufficient resolution, which materially affects measured accuracy.