← Blog

Agentic systems change what engineers review

2026-07-11

Engineers meeting coding agents for the first time tend to ask the same question: if the model chooses which tools to call and what to do next, who writes the requirements, and who reviews the code? The short answer is that humans still do both. The longer answer is that the question assumes the thing under review is still a body of source code, and in an agentic system part of the behavior no longer lives there.

In a conventional application, the decision logic is encoded explicitly, and a reviewer can read the branches and predict what the program will do:

if invoice.total != sum(line.amount for line in invoice.lines):
    reject(invoice)
else:
    post(invoice)

An agent runs a different shape of program:

while not done:
    action = model.decide(state, tools)
    state = execute(action)

The model selects the next action from the current state, so the sequence of tool calls can differ between two runs that start from similar requests. Behavior emerges from the model, the instructions, the tool definitions, and the accumulated state as much as from any code path. That does not make the system unreviewable. It moves the review.

Workflows and agents

The word “agent” gets used loosely, and one distinction is worth keeping. Anthropic’s engineering guidance separates workflows, “systems where LLMs and tools are orchestrated through predefined code paths,” from agents, where models “dynamically direct their own processes and tool usage.” The same guidance recommends the simplest architecture that works: many problems are solved by a single model call with good retrieval, most of the rest by a workflow, and only the remainder by an agent.

Production systems are usually hybrids. Deterministic code handles the parts that can be specified in advance; the model handles the parts where the correct path depends on context that cannot be enumerated ahead of time.

The engineer builds the harness

An agent has no inherent access to anything. Every capability it holds (a terminal, a repository, a database, a browser) is a tool an engineer wired in, with credentials, permissions, sandboxes, timeouts, and approval gates the engineer chose. A model can propose deleting a table; whether the system can actually delete one is decided by the harness around it.

This surrounding machinery has a measurable effect on outcomes, so it deserves review as part of the product. The SWE-agent paper showed that “interface design affects the performance of language model agents”: a purpose-built agent-computer interface for editing code, navigating repositories, and running tests made the same model substantially better at fixing real issues. OpenAI’s account of running Codex agent-first inside one of its own codebases describes where the human effort went in practice: prioritizing work, turning feedback into acceptance criteria, validating outcomes, and adding the missing tools, guardrails, and documentation whenever the agents struggled.

Requirements become behavioral contracts

A requirement for an agent is less a procedure than a contract. Four parts do most of the work: the objective, the constraints the agent must never violate, the acceptance criteria that count as evidence of success, and the escalation policy that says when to stop and involve a person.

A weak requirement prescribes nothing and permits everything:

Fix this GitHub issue.

A strong one defines the envelope without dictating commands:

Reproduce the defect, find the cause, and make the smallest reasonable fix. Do not add dependencies or change public interfaces without approval. Run the affected tests, open a draft pull request, note any remaining uncertainty, and do not merge.

The second version leaves the agent free to choose its path while pinning down what it may touch, what done means, and where its authority ends. Writing that envelope well is the real specification work.

Four review surfaces

Review does not shrink in this model; it spreads across four surfaces.

The artifact comes first. A diff produced by an agent gets the same review a human’s diff gets: correctness, security, test coverage, maintainability. Nothing about the author changes the bar.

The trajectory is second. The closest thing an agent has to a stack trace is the record of what it did: which files it read, which tools it called with what arguments, what failed, what it retried. The path does not need to repeat from run to run, but it should be explainable and inside policy, and that requires recording it. Tracing has become a first-class feature of agent platforms (OpenAI’s Agents SDK ships it built in), and OpenTelemetry fills the same role for custom stacks.

The decision environment is third, and it is the least familiar. When an agent misbehaves, the defect is often upstream of any generated line: an ambiguous tool description, an over-broad permission, a missing stop condition, misleading context in the prompt. Reviewing these is closer to reviewing a machine learning system than a CRUD application.

Behavior across scenarios is fourth. Because several paths can be valid, one passing run proves little. The agent layer needs evals: how often does it complete the task, does it hold the required invariants, does it ask when the request is ambiguous, does it recover from tool failures, does it stop when it should. The NIST AI Risk Management Framework frames this as lifecycle work (govern, map, measure, manage) rather than a gate before release.

Test invariants, not paths

The deterministic pieces inside an agentic system still get ordinary tests. A parser, a pricing function, or an authorization check should behave identically on every run and be tested that way. The agent layer is different: several tool sequences may be acceptable, so its tests pin outcomes and invariants instead of steps. The bug is reproducible before the fix and gone after; the test suite passes; restricted files are untouched; no dependency arrived unapproved; the agent did not merge its own pull request. Which files it searched along the way is left free, the same way a process standard for people constrains results without scripting keystrokes.

Oversight proportional to risk

Human-in-the-loop is a design space, and both edges of it fail. Requiring approval for every read and every reversible command trains people to click approve without looking. Letting an agent take irreversible, externally visible actions unreviewed is the opposite failure.

Anthropic’s study of deployed agent sessions found both instincts in real usage: experienced users run far more of their sessions in full auto-approve (over 40% of sessions after roughly 750 sessions of experience, about double the rate of new users) while also interrupting the agent more often (around 9% of turns, versus 5% for new users). The authors read this as oversight shifting form: seeing what the agent is doing and being able to step in matters more than approving each action. The workable rule is proportionality. Reversible, low-impact actions proceed on their own; sensitive reads get monitoring; irreversible or externally visible actions wait for a person.

Two working examples

I build this argument into my own projects, in two different domains.

planner-lab is the strict version. The model drafts prose and reviews tone; deterministic calculators produce every number into a ledger with citable ids; and a critic gate rejects any memo whose figures cannot be traced back to that ledger. Its review surfaces are exactly the four above: the memo, the audit sidecar recording every computation, the tool and check definitions, and checked-in case studies that exercise the gate, including one where the correct outcome is refusing to produce a memo.

APAB is the observability-heavy version, for phased-array design. The RF physics lives in typed tools served over the Model Context Protocol; the agent sequences them; and every session writes OpenTelemetry traces, a provenance manifest, and an auditable run bundle, with a golden-task eval harness scoring runs against expected tool usage and output thresholds.

Different domains, same design: the model gets the judgment, the tools get the math, and the system records enough that all four surfaces can actually be reviewed.

The envelope is the job

The question that opens this post has a compact answer. People still write the requirements, as objectives and constraints rather than procedures. Engineers still review, but the unit of review has grown from the code to the decision environment around the model: what it can see, what it can do, what it must never do, how its behavior is measured, and when a person takes over. An agent is worth deploying when the decisions delegated to it are bounded, observable, testable, and correct often enough for the risk of the application.

This is an independent analysis written on my own time, using the public sources cited above. The views are my own and do not represent any current or former employer.