RULER (Long-Context Benchmark)

RULER is a benchmark for evaluating how well language models actually use long context. It was built because advertised context windows are misleading: a model may accept 128K tokens yet fail to reason over them, and simple needle-in-a-haystack retrieval is too easy to reveal this.

What It Measures

RULER measures a model's effective context length, the length at which it can still perform reliably, across several task types harder than basic retrieval. These include multi-needle retrieval (finding several items), variable tracking (following references through a long text), aggregation (collecting and summarizing scattered information), and multi-hop question answering over the context.

By synthesizing tasks at controllable lengths, RULER distinguishes models that merely accept long inputs from those that genuinely reason over them.

Methodology

Tasks are generated synthetically, so the input length, number of distractors, and difficulty are all configurable, and the benchmark can scale to very long contexts without needing natural documents. Each model is evaluated at increasing context lengths, and accuracy is recorded per length.

The effective context length is defined as the longest length at which the model still exceeds a performance threshold, often calibrated to a reference. Because tasks are synthetic and parameterized, RULER avoids contamination better than fixed-document benchmarks and gives a clean accuracy-versus-length curve.

How to Interpret Results

Focus on the accuracy-versus-length curve and the effective context length, not the model's nominal window. A model may claim 1M tokens but degrade sharply past 64K, which RULER exposes. The gap between simple retrieval and the harder aggregation or multi-hop tasks shows whether the model reasons over long context or only locates isolated facts.

Compare models at matched lengths and task types. A high effective context length on the harder tasks is the meaningful signal for long-document workloads.

Limitations

RULER's tasks are synthetic and may not perfectly reflect natural long-document reasoning such as reading a long contract or codebase. The effective-length threshold is a chosen cutoff, so different thresholds yield different headline numbers. Results depend on prompt formatting and how the long input is presented. Synthetic patterns could be partly learnable, and the benchmark measures structured tasks rather than open-ended long-form generation quality.

Practical Use

When choosing a model for long documents, judge it by its RULER effective context length on the harder aggregation and multi-hop tasks, not by its advertised window. Compare models at matched lengths and task types, and report the threshold used, since it defines the headline number. Because RULER's tasks are synthetic, confirm the result with a small evaluation on your own long inputs, such as contracts or codebases, whose natural structure differs from RULER's controlled patterns.