RewardBench

RewardBench evaluates reward models, the components that score and rank candidate responses during reinforcement learning from human feedback (RLHF) and during inference-time selection. A reward model that mis-ranks responses will steer training and selection toward worse outputs, so measuring its judgment directly is valuable. The same benchmark also applies to LLMs used as judges.

What It Measures

RewardBench is built from prompts each paired with a chosen (better) and a rejected (worse) response. The reward model must assign a higher score to the chosen response. Accuracy is the fraction of pairs ranked correctly. Categories span general chat, harder chat, reasoning and code, and safety, including cases where the correct preference is to refuse a harmful request or to decline an over-cautious refusal. This category structure shows where a reward model's preferences align with or diverge from intended behavior.

Methodology

Each test item is a curated preference pair where the better response is clearly justified, often drawn from human annotations or constructed to isolate a specific quality such as correctness or safety. A reward model scores both responses; the item is correct if the chosen response scores higher. For generative judges, the model is prompted to pick the better response and its choice is compared to the gold label. Per-category and overall accuracy are reported, with safety and reasoning categories weighted heavily because errors there are most consequential.

How to Interpret Results

Look at category accuracy, not just the average. A reward model strong on chat but weak on reasoning will reward fluent but incorrect answers, degrading any model trained against it. Safety category scores indicate whether the reward signal encourages appropriate refusals. High preference agreement across categories signals a well-rounded judge suitable for both RLHF and response selection. Because reward models drive alignment, small category weaknesses can compound during training.

Limitations

Preference pairs encode the annotators' notion of better, which may not match a given application's values. Curated pairs can be easier than messy real-world comparisons, inflating accuracy. The benchmark measures pairwise ranking, not calibration of absolute scores, which also matters for RLHF. As reward modeling research advances, categories evolve, and contamination remains a risk for any public preference dataset.