SWE-bench Verified

SWE-bench Verified is a curated, human-validated subset of SWE-bench created to fix reliability problems in the original benchmark. It has become the de facto standard for reporting agentic coding performance because its tasks are confirmed to be solvable and fairly tested.

What It Measures

The set contains 500 task instances filtered by professional software engineers from the original SWE-bench. Each retained issue has a clear, sufficient problem description and a test suite that genuinely validates a correct fix without being impossible or underspecified.

Like its parent, it measures end-to-end software engineering: understanding a real GitHub issue, navigating a large Python repository, and producing a patch that passes the relevant tests. The verification step means a failure is more likely to reflect a real model limitation than a broken task.

Methodology

Human annotators reviewed candidate instances and removed those with vague requirements, missing context, or tests that were unfair, flaky, or checked behavior unrelated to the issue. The result is a cleaner 500-task slate.

Evaluation is identical in spirit to SWE-bench: an agent explores the repo, emits a diff, and the harness applies it and runs the tests. The metric is resolved rate, the fraction of issues whose tests all pass after patching. Reports should still note the agent scaffold and tool configuration, which materially affect outcomes.

How to Interpret Results

Because the tasks are validated, scores here are more trustworthy than on the full set, and Verified is now the headline number labs cite for coding agents. Frontier models combined with strong scaffolds resolve a large majority of the 500 tasks.

Still compare only across similar harnesses, and treat the resolved rate as a measure of practical bug-fixing reliability. A model scoring well on Verified is demonstrating genuine multi-file engineering competence rather than puzzle solving.

Limitations

With only 500 tasks all from Python open-source projects, coverage is narrow and may not generalize to other languages or proprietary codebases. Passing tests still does not prove a patch is ideal or side-effect-free. Contamination remains possible for older issues. And because performance depends on the agent framework, the benchmark measures a model-plus-scaffold system as much as the model alone.

Practical Use

SWE-bench Verified is the number to cite when comparing coding agents, but it still measures a model-plus-scaffold system, so reproducibility requires publishing the harness configuration. Because all tasks are Python open-source issues, validate separately on your own languages and proprietary code before trusting the result for production. Use trend over time within a fixed harness to judge genuine model improvement, and discount single-point gains that coincide with scaffold changes rather than model changes.