SWE-bench

SWE-bench is a benchmark for evaluating language models on realistic software engineering tasks. Instead of self-contained puzzles, it uses actual GitHub issues and asks the model to fix them within a full repository, making it one of the most practically relevant coding benchmarks.

What It Measures

The original SWE-bench contains over 2,000 task instances collected from popular open-source Python projects such as Django, scikit-learn, and SymPy. Each instance pairs a real issue or feature request with the repository state at the time and a set of tests that the correct fix must satisfy.

The model receives the issue description and the codebase and must generate a patch. Success means producing changes that make the previously failing tests pass without breaking existing ones. This measures issue comprehension, code navigation across many files, and the ability to make targeted edits.

Methodology

The model, usually embedded in an agent scaffold with file-reading, search, and shell tools, explores the repository and produces a unified diff. The harness applies the patch and runs the project's test suite, including the specific tests tied to the issue.

The headline metric is the percentage of issues resolved, meaning all relevant tests pass after the patch. Because results depend heavily on the surrounding agent, reports should specify the scaffold, tool access, and model. Subsets include SWE-bench Lite for cheaper runs and SWE-bench Verified for human-validated solvability.

How to Interpret Results

Resolved rate is what matters: it reflects end-to-end ability to fix real bugs. Scores rose dramatically as agentic tooling matured, from single digits to well over half of issues on the full set for the best systems. Because the agent scaffold contributes substantially, compare model results only under comparable harnesses.

Use SWE-bench as evidence of practical engineering capability rather than raw code-writing skill; it stresses retrieval and multi-file reasoning far more than HumanEval.

Limitations

Some original instances are underspecified or have flawed tests, which motivated SWE-bench Verified. Patches that pass tests are not guaranteed correct, since tests can be incomplete. The benchmark is Python-only and drawn from public repositories, raising contamination risk for issues predating a model's training cutoff. Results are sensitive to scaffold quality, making clean model-to-model comparison difficult.

Practical Use

When evaluating coding agents, treat the resolved rate as a system-level metric and always disclose the scaffold, tool access, and model versions, because the agent harness contributes a large share of performance. Prefer SWE-bench Verified for clean comparison, and use the full set mainly for scale. For your own context, build a private set of issues from your repositories so you measure ability on code the model has not seen, which is the surest defense against contamination-inflated public scores.