HumanEval

HumanEval is a foundational code-generation benchmark released by OpenAI alongside Codex in 2021. It shifted code evaluation away from surface text similarity toward functional correctness verified by running the generated code.

What It Measures

HumanEval contains 164 hand-written Python programming problems. Each problem provides a function signature, a docstring describing the desired behavior, and a set of hidden unit tests. The model must produce a function body that passes all tests.

The benchmark measures whether a model can translate a natural-language specification into working code. Tasks include string manipulation, simple algorithms, math utilities, and basic data processing, at roughly interview or coursework difficulty.

Methodology

HumanEval introduced the pass@k metric. For each problem the model generates k samples, and the problem counts as solved if any sample passes all unit tests. pass@1 estimates the chance a single attempt works; pass@10 and pass@100 measure success given more attempts. To reduce variance, pass@k is computed with an unbiased estimator from a larger sample pool.

Generated code is executed in a sandbox against the hidden tests. Because execution is the judge, correctness is objective, though sandboxing and timeout handling can affect results.

How to Interpret Results

pass@1 is the headline figure and the most practically meaningful, reflecting first-attempt reliability. Frontier and code-specialized models now exceed 90 percent pass@1, so the original HumanEval is largely saturated. The gap between pass@1 and pass@100 shows how much sampling helps; a wide gap means the model often knows a solution but is inconsistent.

For a fuller picture, pair HumanEval with HumanEval+, which adds far more tests to catch solutions that pass the sparse original tests but are actually buggy.

Limitations

The original test suites are thin, so some incorrect solutions pass; HumanEval+ exists to address this. The 164 problems are Python-only and short, so scores do not reflect multi-file, real-world engineering. The benchmark is old and widely memorized, making contamination likely. High HumanEval scores therefore confirm basic code-writing ability but say little about a model's performance on large codebases, where SWE-bench is more representative.

Practical Use

Use HumanEval as a quick smoke test of basic Python generation, not as a hiring decision for a coding model. Always report HumanEval+ alongside it to catch solutions that pass the sparse original tests but are buggy, and pair both with multilingual and repository-scale benchmarks for a realistic picture. Because the 164 problems are old and widely memorized, a high pass@1 today is table stakes rather than a differentiator, and real engineering ability is better judged by SWE-bench Verified.