FRAMES (Factuality, Retrieval, And reasoning MEasurement Set)

FRAMES (Factuality, Retrieval, And reasoning MEasurement Set) evaluates retrieval-augmented generation (RAG) as a whole rather than its parts. Many benchmarks test retrieval or reasoning in isolation; FRAMES asks whether a system can find the right evidence across multiple documents and then reason over it to produce a correct, grounded answer.

What It Measures

FRAMES questions are designed to require evidence from several documents and to need multi-hop reasoning, where the answer follows only after combining facts. Categories include numerical reasoning, temporal reasoning, tabular reasoning, and constraint satisfaction. The benchmark measures final answer accuracy, but its value is in stressing the full pipeline: a model must retrieve all needed sources and chain them correctly, not just recall one fact.

Methodology

Each item pairs a complex question with a set of relevant Wikipedia articles and a gold answer. FRAMES supports multiple evaluation modes. In an oracle mode the system is given the correct documents, isolating reasoning quality. In a retrieval mode the system must find documents itself, testing the full RAG stack. Answers are graded for correctness against the reference, often with an automated judge because answers can be phrased many ways. Comparing oracle and retrieval scores separates reasoning failures from retrieval failures.

How to Interpret Results

The oracle-versus-retrieval gap is the most informative signal. A high oracle score but low retrieval score means the model reasons well but the retriever is the bottleneck, so investment should go into search. A low oracle score points to reasoning limits that better retrieval cannot fix. Within categories, weakness on temporal or numerical questions flags specific reasoning gaps. Multi-hop accuracy shows whether a system genuinely combines evidence or only answers single-source questions.

Limitations

FRAMES is built on Wikipedia, so it does not reflect noisy, contradictory, or proprietary corpora common in enterprise RAG. Automated answer grading can mismark valid paraphrases or partially correct answers. The fixed document sets simplify retrieval compared with web-scale search. As a relatively new benchmark it is still evolving, and like all public sets it risks contamination as models train on similar data.