CRUXEval (Code Reasoning, Understanding, and Execution)
CRUXEval tests code execution reasoning by asking models to predict outputs from inputs and inputs from outputs on short Python functions. It exposes whether code models truly understand programs or only pattern-match generation.
CRUXEval (Code Reasoning, Understanding, and eXecution) tests a skill distinct from writing code: understanding what code does. Many code models can generate plausible programs yet fail to predict how a given program behaves. CRUXEval isolates this execution-reasoning ability with simple functions and concrete examples.
What It Measures
The benchmark has two tasks built on the same functions. Output prediction gives the model a function and an input and asks for the resulting output. Input prediction gives the function and a desired output and asks for an input that produces it. Both require the model to mentally simulate execution. Together they probe forward reasoning (tracing computation) and backward reasoning (inverting it), which underlie debugging, test generation, and program comprehension.
Methodology
CRUXEval uses hundreds of short, self-contained Python functions with no external dependencies, chosen so a person could trace them by hand. Each comes with an input-output pair. Predictions are graded by execution: a predicted input is checked by running the function and comparing the result, and a predicted output is checked directly against the true output. Metrics are pass@1 and pass@5 for each task, and chain-of-thought prompting is often evaluated separately to see how much explicit step-by-step tracing helps.
How to Interpret Results
Compare output and input prediction: input prediction is usually harder because multiple inputs may be valid and the model must reason backward. A large gap between a model's code-generation score and its CRUXEval score reveals shallow understanding masked by pattern-matched generation. Chain-of-thought typically lifts scores, so report whether numbers use it. Strong CRUXEval performance correlates with better debugging and reasoning over unfamiliar code.
Limitations
Functions are deliberately short and Python-only, so results do not capture reasoning over large or concurrent codebases. Because inputs can be non-unique, input-prediction grading accepts any valid input, which is correct but loosens the task. The benchmark can saturate as models improve, and as a public dataset it faces contamination risk. It measures execution reasoning specifically, not overall coding competence, and should be paired with generation benchmarks.