MuSR (Multistep Soft Reasoning)

MuSR, Multistep Soft Reasoning, tests reasoning over long, realistic narratives rather than short formal puzzles. Each problem is a paragraphs-long story that embeds the clues needed to answer a question, forcing a model to track entities, integrate commonsense, and chain many inference steps across natural prose.

What It Measures

MuSR spans three domains. Murder mysteries require deducing a culprit from motive, means, and opportunity scattered through a story. Object-placement tasks require tracking where items end up as characters move and act. Team-allocation tasks require assigning people to roles under stated constraints. All three demand soft reasoning, meaning inferences grounded in everyday plausibility rather than rigid logic, applied over multiple linked steps.

Methodology

Narratives are generated with a neurosymbolic procedure: a reasoning tree is constructed first, then expanded into a fluent story so that the gold answer is provably supported by the text. This lets the benchmark scale difficulty while guaranteeing solvability. Models read the full narrative and answer a multiple-choice question; accuracy is measured by exact match against the correct option. Chain-of-thought prompting is common, and results are reported per domain and overall.

How to Interpret Results

MuSR is calibrated so that humans score well but many models do not, which makes it useful for distinguishing strong reasoners even when other benchmarks saturate. Look at per-domain scores: object placement stresses state tracking, mysteries stress evidence integration, and team allocation stresses constraint satisfaction, so the profile shows where a model is weak. Because the narratives are long, scores also reflect long-context comprehension, not reasoning alone.

Limitations

The synthetic generation process can produce stylistic artifacts that a model might exploit without true reasoning. Multiple-choice format allows lucky guesses and lets models eliminate options rather than solve from scratch. The three domains are narrow, so high MuSR scores do not guarantee broad reasoning. As with any popular benchmark, contamination and targeted tuning can inflate results over time.