Terminal-Bench

Terminal-Bench evaluates AI agents on their ability to accomplish real tasks in a command-line environment. It targets a practical and demanding setting: operating a shell to install software, manipulate files, run builds, debug, and administer systems.

What It Measures

The benchmark provides a collection of tasks, each defined by an instruction, a sandboxed environment (typically a Docker container), and an automated test that verifies whether the task was completed correctly. Tasks include things like compiling a project, fixing a broken configuration, processing data with command-line tools, training a small model, or recovering from a system problem.

It measures end-to-end agentic competence in a terminal: understanding the goal, issuing correct commands, interpreting their output, handling errors, and iterating until the environment reaches the desired state.

Methodology

An agent connects to the sandbox and issues shell commands over multiple steps, observing outputs and adjusting. The environment is isolated and reproducible so results are consistent. After the agent finishes, an automated verification script inspects the environment, for example checking files, process output, or test results, to decide success or failure.

The headline metric is task success rate, the fraction of tasks completed correctly. Because agents drive the interaction, performance reflects both the model and its scaffold, so reports should specify the agent harness used.

How to Interpret Results

Success rate measures practical autonomy in a realistic developer and operations setting, complementing code-writing benchmarks with execution and environment interaction. A model strong on code generation but weak on Terminal-Bench reveals difficulty with iterative, stateful tool use and error recovery.

Compare only across comparable harnesses and task versions, since the agent framework and task set strongly affect outcomes. Per-category breakdowns, such as build tasks versus debugging, help locate specific weaknesses.

Limitations

Results depend heavily on the agent scaffold, so the benchmark measures a system rather than a model in isolation. Automated verification can miss alternative valid solutions or be fooled by superficially correct states. Task coverage, while diverse, cannot represent every real-world command-line scenario. Long, multi-step tasks introduce variance, and contamination is possible as tasks become public.

Practical Use

Treat Terminal-Bench results as system-level, disclosing the agent harness and task version, because the scaffold strongly shapes outcomes. It complements code-generation benchmarks by testing iterative, stateful tool use and error recovery, so a gap between strong code generation and weak Terminal-Bench performance is diagnostic. Use per-category breakdowns to locate weaknesses such as debugging versus building, and validate on tasks resembling your own operations workloads before trusting an agent with real command-line autonomy.