BBH (BIG-bench Hard)

BBH (BIG-bench Hard) is a focused subset of BIG-bench containing the tasks that language models found hardest. It was introduced to provide a compact, reasoning-heavy benchmark and became the standard way to report BIG-bench-style performance.

What It Measures

BBH comprises 23 tasks (27 by some counts including subtasks) selected because models at the time scored below the average human rater on them. They emphasize multi-step reasoning: logical deduction, tracking objects through a sequence of operations, date and temporal reasoning, navigation, word and symbol manipulation, and disambiguation.

The benchmark measures whether a model can carry out structured, multi-hop reasoning rather than recall a fact. Many tasks are deliberately algorithmic, so guessing rarely helps.

Methodology

The defining finding behind BBH is that chain-of-thought prompting dramatically improves performance on these tasks, often flipping below-human scores to above-human. Evaluation therefore usually reports both direct-answer and chain-of-thought accuracy, with the latter as the headline.

Most tasks are multiple choice or have short canonical answers, scored by exact match or option selection. A small number of few-shot exemplars, sometimes with worked reasoning, are commonly prepended. Scores are reported per task and as an average across the suite.

How to Interpret Results

The gap between direct and chain-of-thought accuracy is itself informative: a large gap shows the task genuinely requires deliberate reasoning and that the model can exploit it. Aggregate BBH accuracy is a reasonable proxy for general reasoning skill and is far cheaper to compute than full BIG-bench.

Strong reasoning models now score very high on BBH, so it is partially saturated, but per-task breakdowns still reveal specific weaknesses such as multi-step tracking or formal logic.

Limitations

With only 23 tasks, BBH is narrower than full BIG-bench and can be gamed by overfitting to those specific formats. Its heavy reliance on chain-of-thought means results depend on prompting and decoding choices. As frontier models saturate many tasks, it loses discriminative power at the top, prompting use of harder reasoning sets. Contamination is also a concern given the dataset's popularity.

Practical Use

BBH is a cost-effective stand-in for general reasoning and is widely reported, so it is a reasonable inclusion in any evaluation suite. Report both direct and chain-of-thought accuracy and the prompting setup, because the gap between them is part of the signal. As frontier models saturate many of its tasks, lean on per-task breakdowns to find residual weaknesses, and add harder reasoning benchmarks such as GPQA and AIME when you need to separate elite systems.