← Blog

How to build an eval harness for LLM agents

2026-07-12

A passing demo run tells you almost nothing about an agent. The same task, model, and prompt can produce different tool sequences and different outcomes across runs. When τ-bench measured top function-calling agents on customer service tasks, GPT-4o passed fewer than half on a single attempt, and its probability of passing the same task eight times in a row fell below 25% in the retail domain. An agent that demos well and an agent you can ship are separated by exactly the machinery this post describes.

I wrote earlier about how agentic systems change what engineers review: the review surfaces grow from the code to the artifact, the trace, the decision environment, and measured behavior across scenarios. This post expands the last surface. An eval harness is the instrument that turns stochastic behavior into something you can compare across candidates, regress in CI, and make release decisions with. Two definitions up front, because the words collide: the agent harness is the runtime that lets the model plan and call tools; the eval harness is the machinery that feeds it tasks, captures what it did, and grades the result. The thing under evaluation is the whole system (model, prompts, tools, orchestration), so a score belongs to a configuration, never to a model alone.

A task is a versioned spec

The unit of evaluation is a task, and a task is more than a prompt. It pins down the starting environment, what the agent may use, what it must never do, and how the result gets graded:

id: auth-empty-password-001
version: 3
instruction: >
  Fix the authentication bug that lets an empty password pass
  credential validation. Preserve the public API.
environment:
  image: agent-eval/python-auth@sha256:...
  repository_commit: 82c41d9
  network: disabled
budgets: { max_tool_calls: 100, max_wall_seconds: 900 }
invariants:
  - tests_directory_unchanged
  - public_api_unchanged
graders:
  - hidden_functional_tests
  - invariant_checks

Two rules keep task suites trustworthy. First, every requirement a grader enforces must be inferable from the instruction; hidden tests may stay hidden, but an agent should never fail because the grader silently expected a filename the task never mentioned. Second, write a known-good reference solution for each task and confirm it passes every grader. Anthropic’s guide to agent evals recommends both practices, and its sizing advice is worth taking literally: “20-50 simple tasks drawn from real failures is a great start.” Task quality beats task count.

Isolate every trial

Each trial starts from a clean snapshot: pinned repository commit, pinned container image, fresh working directory, controlled network, explicit resource limits. Anything shared between trials is a threat to the measurement; Anthropic notes that “shared state can also artificially inflate performance,” as when one trial’s leftover files help the next.

The graders need isolation of a different kind. Run them where the agent cannot reach: freeze the agent’s final filesystem or database state, hand a read-only copy to a separate grading environment, and keep hidden tests and policy checks out of the agent’s workspace entirely. This is a live concern rather than a theoretical one. OpenAI’s account of monitoring its internal coding agents documents models that optimized for the evaluation signal instead of the task, editing tests so they always pass or disabling checks to hide failures, and (rarely) attempting prompt injection against a monitor or grader. An agent with write access to its own graders is an agent whose scores mean nothing.

Grade outcomes first

The grading stack has a strict order of preference.

Deterministic end-state checks come first, wherever the result can be verified mechanically: hidden functional tests, build and type checks, a database compared against an annotated goal state. τ-bench’s design makes the case plainly; it grades by comparing “the database state at the end of a conversation with the annotated goal state,” so an agent claiming success while the refund never happened scores zero.

Invariant gates come second and are absolute. No secrets exposed, no approval bypassed, no tests edited, no irreversible action without authorization. A run that passes every functional test while violating one of these fails, full stop. Severity ordering belongs in the gate structure, since a composite score lets a cheap, fast, unsafe candidate look good.

Trace checks come third, for diagnosis and for the few behaviors where the path itself is policy (a destructive action verified before execution, an escalation that must happen). Everywhere else, allow multiple valid paths: Anthropic’s guidance is to grade what the agent produced rather than the route it took, because agents find legitimate solutions the evaluator never anticipated. One more subtlety from the same guide: test both directions. A suite that only checks whether the agent escalates when it should will happily reward an agent that escalates constantly.

LLM judges come last, only for dimensions that resist mechanical verification (explanation quality, whether uncertainty was communicated), scored per dimension against explicit criteria rather than as one holistic number, and calibrated against human expert labels. A judge is a measurement device with its own error rates; if you have not measured its false-negative rate on serious violations, you do not know what its scores mean.

Measure reliability, and price it

Run each task multiple times and pick the metric that matches the product. pass@k (at least one of k attempts succeeds) fits workflows where generating alternatives is acceptable; pass^k (all k succeed) fits anything users invoke expecting it to work every time. τ-bench introduced pass^k precisely because single-attempt scores hid how inconsistent agents are, and Anthropic’s guide draws the same product-based distinction: pass@k where one success matters, pass^k where consistency is essential.

Report cost and latency next to correctness, since an agent that is right slightly more often at triple the token spend is a real tradeoff and should be visible as one. And compare candidates in pairs on identical tasks, fixtures, and budgets; run-to-run variance can reverse a two-point difference, so confidence intervals beat point estimates.

Keep the benchmark honest

The harness can be wrong in every way the agent can. A 2025 study of agentic benchmarks found that task ambiguity, weak graders, and environment bugs cause “under- or overestimation of agents’ performance by up to 100% in relative terms,” and its examples are humbling: τ-bench itself counted empty responses as successful, and SWE-bench Verified shipped tests too weak to catch wrong patches. The paper’s Agentic Benchmark Checklist is a good audit template. The cheap version: routinely read full traces from a sample of passes as well as failures, and treat a suspiciously cheap success as a bug report against the harness.

Suites also need a lifecycle. Capability tasks (hard, low pass rate, for comparing architectures) graduate into the regression suite once they become reliably solvable, and the regression suite runs continuously to catch drift. Feed real production failures back in as new tasks, and keep a holdout set you never tune prompts against, or the team ends up optimizing the benchmark instead of the product. When something fails, categorize it: wrong plan, wrong tool arguments, retrieval miss, premature stop, hallucinated completion, grader bug. A score says something changed; the taxonomy says what to fix.

What I run in my own projects

BrewTrace, a local tool-calling agent, carries a 25-case eval where the cases and the deterministic rule engine are two views of the same table, so the no-LLM pipeline must score 25/25 and any miss means a bug in one of them. The live agent scored 20/25 with qwen3-8B in the tutorial baseline and 22/25 in a matrix run days later; that two-case swing on identical tasks is the repeated-trials argument in miniature. Because every eval run is traced with eval.case_id and eval.passed, the baseline failures resolved into three named buckets (a schema mapping that inverted a direction, a second-choice adjustment, guesses where the rules ask for more data), each suggesting a different fix. A sweep across five local models then made the reliability case plainly: pass rates ran from 88% to 0% among models that all advertise tool support, failing in four different ways. One ignores the tools, one calls them and discards their answers, and one reasons its way to an answer it never states. Each failure mode is one eval.passed=false query away.

APAB, the phased-array design agent, runs golden-task evals that score a run against its own audit bundle: expected tools called in a sensible order, run completed within a call budget, output metrics inside thresholds. The scorer is deliberately LLM-free, so it runs in CI without an API key and without nondeterminism.

planner-lab, the financial planning agent, shows the runtime complement: its critic gate is effectively an eval that runs on every single request, nine deterministic checks that refuse to emit any memo whose numbers fail to trace. Offline evals tell you how often the system is right; a runtime gate bounds what happens when it is wrong.

Build on a framework, own the spec

You do not need to build all of this from scratch. Inspect AI, the open-source framework from the UK AI Security Institute, covers execution: datasets, solvers, and scorers as composable pieces, sandboxed runs in Docker or Kubernetes, and support for 20-plus model providers. Platform tracing tools cover observability, and OpenTelemetry works as the interchange format (the GenAI semantic conventions now have a dedicated repository, though they are still evolving; the BrewTrace post covers living with that churn).

What you should own, in a portable format independent of any framework: the task definitions, the fixtures, the graders, the run manifests, and the historical results. Those are the measurement; the framework is just the runner. The minimum viable harness is small: 20-50 versioned tasks from real failures, a fresh container per trial, structured logs of every model and tool call, deterministic outcome and invariant graders, repeated trials on the critical subset, and a readable report. Dashboards, judge panels, and synthetic task generation can all come later.

The harness is the spec

A product spec says what the agent is supposed to do. The eval harness demonstrates whether it does it, repeatedly, within constraints, at a known cost. For a system whose behavior emerges from a model, prompts, and tools rather than from explicit control flow, that demonstration is the closest thing to a specification you can execute, and it is the difference between shipping an agent and shipping a demo.

This is an independent analysis written on my own time, using the public sources cited above. The projects mentioned are independent work. The views are my own and do not represent any current or former employer.