HellaSwag

HellaSwag is a commonsense reasoning benchmark built around sentence completion. Its name reflects its adversarial design: the wrong answers are generated to look plausible to machines while remaining obviously wrong to humans, creating a gap that probes genuine understanding.

What It Measures

Each HellaSwag item presents a short context, often describing an everyday activity drawn from sources like ActivityNet captions or WikiHow, followed by four possible continuations. The model must choose the most sensible ending.

The benchmark measures everyday commonsense and physical plausibility: knowing how ordinary situations typically unfold. Humans score around 95 percent, but the adversarial distractors were specifically constructed to fool earlier models, exposing shallow pattern matching.

Methodology

Wrong endings were produced by a generator and filtered through adversarial filtering, which repeatedly keeps the machine-generated continuations that current models found hardest to reject. This raises the difficulty for models without confusing humans.

Evaluation is multiple choice. Models are scored by accuracy, typically by comparing the likelihood the model assigns to each candidate completion and selecting the highest, or by parsing a generated choice. It is usually run zero-shot or few-shot.

How to Interpret Results

When HellaSwag launched, models scored near chance while humans were near ceiling, making it a strong test. Today large models exceed 95 percent, matching or beating humans, so the benchmark is saturated and no longer discriminates among frontier systems.

A high score now mainly confirms basic commonsense competence. It remains useful for evaluating smaller or new architectures and for sanity-checking that a model has not regressed on everyday reasoning, but it should not be used to rank top models.

Limitations

Saturation is the chief limitation: the human-machine gap that gave HellaSwag its value has closed for strong models. Because answers can be chosen by completion likelihood, results can be sensitive to scoring method and tokenization. The adversarial construction occasionally yields ambiguous or noisy items. As an old, widely used dataset, it is also prone to contamination, further inflating modern scores.

Practical Use

HellaSwag is now best used as a lightweight regression check and as a component of broad evaluation harnesses for smaller or newly trained models. Because it is saturated for frontier systems, do not use it to rank them; a near-ceiling score simply confirms intact everyday commonsense. When you do report it, fix the scoring method, since likelihood-based selection and generated-choice parsing can yield slightly different numbers, and be aware that contamination likely inflates modern results.