LiveCodeBench

LiveCodeBench is a coding benchmark designed to resist contamination by continuously collecting fresh problems over time. It addresses a core weakness of static code benchmarks like HumanEval: once problems are public, models may have memorized them.

What It Measures

LiveCodeBench gathers problems from competitive-programming platforms such as LeetCode, AtCoder, and Codeforces, each tagged with a release date. It evaluates several coding-related abilities, including code generation from a problem statement, self-repair given failing tests, test-output prediction, and code execution reasoning.

By recording when each problem was published, the benchmark can evaluate a model only on problems released after its training cutoff, giving a cleaner estimate of true generalization rather than memorization.

Methodology

Problems are scored by execution: the model's solution is run against hidden test cases, and pass@1 and pass@k are computed as in other execution-based benchmarks. The defining feature is the time window: evaluators select problems from a date range, typically after the model's cutoff, to avoid contamination.

The holistic design lets researchers report not just generation accuracy but also self-repair success and execution-reasoning accuracy, separating distinct skills. The problem pool grows continuously, so the benchmark refreshes rather than going stale.

How to Interpret Results

Always note the evaluation time window. A model's score on post-cutoff problems is the trustworthy figure; a large drop from pre-cutoff to post-cutoff problems is a red flag for contamination on static benchmarks. Compare models on overlapping windows for fairness.

pass@1 on recent problems is the headline for generation, but the self-repair and execution-reasoning sub-scores give a richer view of practical coding ability. Difficulty tiers from the source platforms also help interpret where a model breaks down.

Limitations

Competitive-programming problems emphasize algorithmic puzzles, which differ from real software engineering, so high scores do not imply repository-scale competence the way SWE-bench does. Difficulty and topic mix shift as new problems arrive, complicating longitudinal comparison. Determining exact training cutoffs is imprecise, weakening the contamination guarantee. Execution-based scoring depends on test coverage and sandbox reliability.

Practical Use

Always report the evaluation time window and compare models on overlapping post-cutoff ranges, since the contamination-resistance of LiveCodeBench depends entirely on this discipline. A large drop from pre-cutoff to post-cutoff problems on other benchmarks is a contamination warning. Use the self-repair and execution-reasoning sub-scores, not just generation pass@1, for a richer view. Remember that algorithmic puzzles differ from real software engineering, so pair it with SWE-bench Verified for repository-scale competence.