tau-bench (Tool-Agent-User Benchmark)
tau-bench tests tool-using agents in simulated customer-service dialogues that demand correct API calls and strict policy compliance, with pass^k measuring reliability across repeated attempts. It captures realistic agentic competence but is limited to a few domains and depends on the user simulator and scaffold.
tau-bench (the Tool-Agent-User benchmark) evaluates language-model agents in realistic, interactive task settings. It was created to test whether agents can hold a multi-turn conversation with a user, call tools correctly, and follow domain rules, which static benchmarks do not capture.
What It Measures
tau-bench places an agent in simulated customer-service scenarios in domains such as retail and airline support. The agent must converse with a simulated user, query and update a backend database through provided API tools, and comply with detailed domain policies, for example refund eligibility or cancellation rules.
The benchmark measures end-to-end agentic competence: gathering information across turns, choosing and parameterizing the right tool calls, respecting business rules, and reaching the correct final state. It tests both correctness and rule-following under interaction.
Methodology
A simulated user, driven by a language model following a scenario script, interacts with the agent. The agent has access to a set of tools and a written policy document. After the dialogue, the benchmark checks the resulting database state and outputs against the ground-truth target to determine success.
A distinctive metric is pass-hat-k (pass^k), which runs each task multiple times and measures the probability that the agent succeeds on all k attempts, capturing reliability and consistency rather than best-of-k luck. Standard task success rate is also reported.
How to Interpret Results
Task success rate shows whether the agent can complete realistic tasks at all, while pass^k reveals consistency: a model can have a decent single-run success rate yet collapse under pass^k if it is unreliable. For production agents, the pass^k drop-off is often more telling than the headline success rate.
Because tasks require policy compliance, failures often stem from rule violations or wrong tool arguments rather than ignorance, which is diagnostic for agent design. Compare models under the same tool set, policies, and user simulator.
Limitations
The simulated user is itself a model and may behave unrealistically, adding noise and potential bias. Coverage is limited to a few customer-service domains, so results may not generalize to other agentic tasks. Success checks based on final state can miss partially correct behavior or credit lucky outcomes. Results depend heavily on the agent scaffold and the tools provided, making the benchmark a measure of the whole system rather than the model alone.
Practical Use
For production agents, watch the pass^k curve as closely as the single-run success rate, since reliability across repeated attempts is what matters in deployment. Inspect failure modes: rule violations and malformed tool arguments are more actionable than knowledge gaps and point directly at scaffold and prompt fixes. Compare models under identical tools, policies, and user simulators, and validate on your own domain, since tau-bench covers only a few customer-service settings that may not match your application.