← Blog

How to measure a grader

2026-07-29

Anthropic’s guide to agent evals tells two stories about graders being wrong, and neither one involves an LLM judge. In the first, Opus 4.5 solved a τ2-bench flight-booking task “by discovering a loophole in the policy,” then “‘failed’ the evaluation as written” while having found a better answer for the user. In the second, the same model scored 42% on CORE-Bench until an Anthropic researcher looked and found rigid grading that penalized “96.12” when the expected value was “96.124991…”, ambiguous task specs, and stochastic tasks that could not be reproduced. After fixing the bugs and switching to a less constrained scaffold, it scored 95%.

Both failures are string comparisons and test assertions behaving exactly as written. That is worth dwelling on, because the practitioner conversation about grader quality has narrowed into a conversation about LLM judges, and it took the deterministic half of the problem with it.

What a grader is

The guide’s definition is positional. “A grader is logic that scores some aspect of the agent’s performance. A task can have multiple graders, each containing multiple assertions (sometimes called checks).” Nothing in that names an implementation. The contract is only that each grader “evaluates some portion of either the transcript or the outcome.”

Those two inputs deserve separating. The transcript is the complete record of a trial: outputs, tool calls, reasoning, intermediate results. The outcome is the final state of the environment. The guide’s example is precise: a flight-booking agent “might say ‘Your flight has been booked’ at the end of the transcript, but the outcome is whether a reservation exists in the environment’s SQL database.” One input is what the agent said it did. The other is what happened.

Graders also compose. A task can carry several, and scoring across them is weighted (combined scores must clear a threshold), binary (all must pass), or hybrid. One sentence is the guide’s entire treatment of aggregation, which says something about how young this practice is.

The illustrative YAML for an authentication-bypass fix carries five graders: deterministic tests, an LLM rubric, static analysis, a state check against the security log, and a tool-call check. Five mechanisms, one contract. WebArena, a real benchmark, grades with four: whole-string matching, substring matching, an LLM judge, and state matching.

A judge is one kind of grader

The guide sorts graders into three rows: code-based, model-based, and human. LLM-as-judge is the model-based row. In the five-grader example it is one entry, llm_rubric. Using “grader” to mean “LLM judge” erases the other four.

The containment fails in the other direction too. An LLM scoring text is not always a grader. The same technique is a reward model during training, a content filter in production, or a routing classifier in front of a model pool, and in none of those is it scoring an eval task. The two ideas overlap without either one containing the other.

Keeping them apart preserves a choice. The guide recommends “deterministic graders where possible, LLM graders where necessary or for additional flexibility, and using human graders judiciously for additional validation.” That is a ranking, and it only reads as a ranking while the three remain distinct options.

That example is labeled “Theoretical,” and the guide says real coding evals “typically rely on unit tests for correctness verification and an LLM rubric.” Its fifth grader also requires a specific tool-call sequence, which a later section calls “too rigid” because “agents regularly find valid approaches that eval designers didn’t anticipate.” Reading a sample configuration as a recommendation is its own way to end up with a bad grader.

The two kinds fail differently

Both mechanisms are wrong in both directions. The asymmetry is in how you notice.

None of those rows is hypothetical. An audit of ten agentic benchmarks found an agent could “score 100% on SWE-Lancer without resolving any tasks,” and τ-bench counted empty responses as successful: deterministic graders producing false positives. On the gaming row, OpenAI’s report on monitoring its internal coding agents lists agents that “illegitimately edit tests to make them pass,” and separately a rarer category where an agent tries to manipulate “another model (e.g. a monitor or grader).” A judge is gameable from further away: a null model returning one constant string, with no regard for the input, reached an 86.5% length-controlled win rate on AlpacaEval 2.0.

A deterministic failure is legible because the assertion sits right there in the code. A judge failure looks like a reasonable score and goes on looking reasonable across many transcripts, so catching it takes a different instrument.

Every grader has a confusion matrix

Whatever mechanism fills the role, a grader that emits pass or fail is a classifier. Against a reference it has false positives, false negatives, precision, recall, and an agreement coefficient. pytest has all of these. So does a rubric. The failures above are that matrix with entries in it.

Almost nobody publishes the numbers. MAST, a taxonomy of multi-agent failures, did: it validated an LLM annotator against expert human labels on a held-out set. With few-shot examples the judge reached 94% accuracy, recall 0.77, precision 0.833, and Cohen’s κ of 0.77. Zero-shot it sat at 89% accuracy with recall 0.62 and κ 0.58. The human annotators who built the taxonomy reached κ = 0.88 among themselves.

The distance between 94% accuracy and 0.77 recall carries the argument. Accuracy on a lopsided label set is dominated by the majority class, so a judge can look excellent while missing roughly a quarter of the cases it exists to catch. The guide reaches the same statistics from the task-design side, advising teams to “avoid class-imbalanced evals.” I wrote about the same trap in predicting the next pitch, where accuracy rewards guessing the majority and log loss is what exposes it.

Recall on the classes you care about is the number to report. A judge that catches 77% of policy violations is not a 94% judge, and a failure taxonomy built on its labels inherits that 23%.

What a judge costs

A judge carries two costs that a deterministic grader does not.

The first is variance. A test suite returns the same verdict on the same artifact every time, so all the run-to-run spread in your results belongs to the agent. Put a judge in the grader slot and two stochastic systems are in the loop, one being measured and one doing the measuring. pass@k and pass^k then mix both sources, and re-running does not separate them, because every run varies both at once. The cheap separation is to freeze a set of transcripts and re-grade them with the judge alone, which holds the agent constant so any disagreement is the judge’s.

That variance is measurable, and it is large. In the MT-Bench study, judges given the same pairwise question with the two answers swapped returned a consistent verdict 65.0% of the time for GPT-4 and 23.8% for Claude-v1, which favored whichever answer came first in 75.0% of cases. A separate paper showed the ranking can be “easily hacked by simply altering their order of appearance”, lifting Vicuna-13B to beat ChatGPT on 66 of 80 queries. Verbosity behaves the same way: MT-Bench’s “repetitive list” attack, which rephrases a list and prepends it without adding information, fooled Claude-v1 and GPT-3.5 on 91.3% of 23 answers and GPT-4 on 8.7%. Judges also favor their own generations, scoring them above what human annotators give them.

The second cost is calibration. The guide lists it as a weakness of model-based graders and of no other kind: they “require calibration with human graders for accuracy.” The mitigations are mechanism-specific too. Give the judge a way out, “an instruction to return ‘Unknown’ when it doesn’t have enough information.” Grade each rubric dimension with an isolated judge rather than one judge scoring all of them. Swap positions and average.

What to report about a grader

The benchmark audit turns this into a checklist line. Under LLM-as-a-Judge, the Agentic Benchmark Checklist asks that a benchmark “demonstrates documented or experimental evidence of the judge’s accuracy, self-consistency, and agreement with human,” and the item beside it asks that grading “is designed to resist adversarial inputs and reward hacking.” The audit found WebArena “uses an LLM-as-a-Judge without validating its accuracy or consistency,” which with one other violated check accounts for a 1.4 to 5.2% overestimate of agent performance.

A grader report card, applicable to either mechanism:

• Agreement with a human reference on a labeled sample, reported as κ rather than raw accuracy.
• Recall on the classes that matter, broken out rather than averaged away.
• An operating point at a fixed false-positive rate, rather than an area under a curve alone.
• False-positive rate by subgroup.
• Self-consistency across reruns, and across swapped inputs for a judge.
• A version, because changing the judge’s model changes the instrument.

slopscore publishes that shape for a grader I wrote. Its rule scorer reaches an AUROC of 0.89 on 128 overt cases and 0.69 on 40 held-out real-world ones, it reports catch rate at a fixed 1% false-positive rate, and it breaks false-positive rate on clean writing out by language group, where the opt-in learned model hits 0.71 on simple English while the default rule scorer stays at 0.00. That last table is the reason the rule scorer is the default.

What this looks like in my own projects

planner-lab’s critic gate is this distinction implemented. Nine deterministic checks run against a computation ledger, an LLM tone review runs on top, and a failure from either side fails the memo. The rejection report prefixes LLM-sourced failures with [llm], so when a memo is blocked you can see which mechanism blocked it. The deterministic checks catch numbers that do not trace; the tone review catches “is feasible” asserted about a probabilistic result. Neither one substitutes for the other.

APAB’s golden-task scorer goes the other way and is deliberately LLM-free, so it runs in CI without an API key and with no judge variance to account for. BrewTrace’s 25 eval cases must score 25/25 against its deterministic rule engine, because the cases and the engine are two views of one table. A failure there is a bug in the grader rather than a finding about the agent.

Read the transcripts

All of this starts where the guide ends. It recommends sampling transcripts to read weekly and writing a reference solution for every task, which “proves that the task is solvable and verifies graders are correctly configured.” Its bar for a task is also a bar for the grader attached to it: “A good task is one where two domain experts would independently reach the same pass/fail verdict.”

Reading transcripts does not replace measuring a grader. It is how you get the labels to measure one with. Every confusion matrix above began with somebody opening a trial and deciding what the right answer was.

This is an independent analysis written on my own time, using the public sources cited above. The projects mentioned are independent work. The views are my own and do not represent any current or former employer.