MBPP (Mostly Basic Python Problems)
MBPP evaluates entry-level Python program synthesis from short descriptions using execution-based pass@k scoring. It is largely saturated and, like HumanEval, best treated as a basic competence check rather than a real-world engineering measure.
MBPP (Mostly Basic Python Problems) is a code-generation benchmark from Google designed to test entry-level Python programming ability. It complements HumanEval with a larger and slightly easier problem set, and the two are often reported together.
What It Measures
MBPP contains roughly 1,000 crowd-sourced problems aimed at beginner programmers. Each task gives a short natural-language description, such as writing a function to find the shared elements of two lists, plus a small number of test cases the solution must pass.
The benchmark measures basic program synthesis: turning a brief, plain-English request into a correct, self-contained Python function. The problems cover common idioms, simple algorithms, string and list manipulation, and elementary math.
Methodology
Models generate a function that is then executed against the provided test assertions; passing all of them counts as success. Like HumanEval, MBPP uses execution-based scoring and pass@k metrics, with pass@1 the most common headline figure.
A manually verified subset, often called sanitized MBPP, removes ambiguous or poorly specified problems and is preferred for clean comparison. Evaluation can be few-shot, prepending example problem-solution pairs, or zero-shot for instruction-tuned models. A small number of provided tests means the verification is lighter than ideal.
How to Interpret Results
MBPP scores track HumanEval closely and are usually a few points higher because the tasks are simpler. Strong code models now exceed 85 to 90 percent pass@1, so MBPP is largely saturated as a standalone measure.
Report whether you used the full or sanitized set and whether evaluation was zero-shot or few-shot, since these choices move scores noticeably. Treat MBPP and HumanEval together as a basic competence check, not as evidence of real-world engineering skill.
Limitations
The few test cases per problem allow some incorrect solutions to pass, overstating correctness. Problems are short, single-function, and Python-only, so they do not reflect multi-file projects or other languages. The dataset is widely available and likely memorized by recent models. Because both MBPP and HumanEval are saturated, repository-scale benchmarks such as SWE-bench better reflect practical coding ability.
Practical Use
MBPP pairs naturally with HumanEval as a basic Python competence check, and the two together give a slightly broader view than either alone. State whether you used the full or sanitized set and the shot count, since these move scores. For production model selection, do not stop at MBPP: add execution-based multilingual benchmarks and a repository-scale test, and validate on tasks drawn from your own codebase, which exercise context and tooling that MBPP's single-function format never touches.