IFEval (Instruction-Following Eval)

IFEval, short for Instruction-Following Eval, tests a narrow but critical skill: does a model do exactly what the prompt tells it to do? Many evaluations grade the substance of an answer, but real applications also depend on the model honoring constraints such as length, format, language, and keyword inclusion. IFEval isolates that behavior with instructions that can be checked by a program, removing the need for a human or LLM judge.

What It Measures

IFEval uses verifiable instructions, for example "respond in at least 300 words," "write in all lowercase," "include the word 'data' three times," "answer in JSON," or "do not use commas." Each prompt may carry several such constraints. Scores are reported at two granularities. Prompt-level accuracy counts a prompt as passed only if every instruction in it is satisfied. Instruction-level accuracy counts each constraint independently. Both are reported in strict and loose variants, where loose allows minor formatting leniency.

Methodology

Because every instruction is objectively checkable, IFEval runs deterministic Python verifiers over each model response. There is no model-as-judge and no human rating, so results are cheap to reproduce and free of grader bias. The strict score applies the verifier literally; the loose score normalizes the output first, for example stripping markdown wrappers, to avoid penalizing trivial presentation differences.

How to Interpret Results

Report all four numbers. A model with high content quality but low prompt-level strict accuracy will frustrate users who need reliable formatting for downstream parsing. The gap between strict and loose scores shows how much failure is cosmetic versus substantive. The gap between instruction-level and prompt-level scores shows whether a model handles single constraints but breaks when several stack up. For agent and tooling pipelines, prompt-level strict accuracy is the number that matters most.

Limitations

IFEval covers only instructions that are machine-verifiable, so it ignores nuanced or open-ended directions that dominate real conversations. It does not assess whether the content is correct, helpful, or safe, only whether constraints were met. Models can also be tuned specifically to pass common constraint types. Treat IFEval as a reliability check on formatting and constraint compliance, complementary to quality and reasoning benchmarks.