MT-Bench

MT-Bench is a benchmark for evaluating conversational quality in chat models, introduced alongside the LLM-as-a-judge methodology. It focuses on open-ended, multi-turn dialogue, which automatic metrics like accuracy cannot capture.

What It Measures

MT-Bench consists of 80 carefully designed multi-turn questions across eight categories: writing, roleplay, reasoning, math, coding, extraction, STEM knowledge, and humanities. Each item has a first question and a follow-up, so the model is tested on whether it maintains context across two turns.

The benchmark measures overall helpfulness and response quality in realistic chat scenarios, including instruction following, coherence, and the ability to handle a follow-up that depends on the prior answer.

Methodology

MT-Bench popularized using a strong model, originally GPT-4, as an automatic judge. The judge reads the question and the model's response and assigns a score from 1 to 10, optionally with reasoning. It supports single-answer grading and pairwise comparison between two models.

The headline metric is the average judge score, often broken out by first and second turn. Pairwise mode yields win rates. The authors validated that judge rankings correlate well with human preferences, though they also documented judge biases.

How to Interpret Results

MT-Bench scores summarize subjective response quality, so use category and per-turn breakdowns to see where a model excels or drops, for example strong writing but weak second-turn reasoning. Because the judge is an LLM, hold the judge model and prompt constant when comparing systems.

MT-Bench is best treated as a quick, automatable proxy for chat quality rather than an absolute measure. For human-grounded rankings, Chatbot Arena is the complementary standard.

Limitations

LLM judges exhibit known biases: preferring longer answers, favoring the first response in a pair, and rating their own family of models higher. With only 80 questions, coverage is thin and scores are noisy. The 1-10 scale compresses at the top, where strong models cluster. Results are sensitive to the judge model and prompt, and as a popular benchmark it is exposed to contamination and tuning to its style.

Practical Use

MT-Bench is a convenient automatable proxy for chat quality, useful for rapid iteration during model development, but it should not stand in for human-grounded ranking. Fix the judge model and prompt across comparisons and be alert to length and position biases that can favor verbose or first-listed answers. Use Chatbot Arena for preference rankings grounded in real users, and read MT-Bench's per-category and per-turn breakdowns rather than the single average to locate specific conversational weaknesses.