← Blog

Auditing tool-call traces in MCP agents

2026-08-01

A refund agent fails in production. The visible error is a payment call that returned a 422, so that is where the on-call engineer starts, and the payment tool gets a better error message and a retry cap. The failure rate does not move.

The trace says something else happened. Four calls earlier the agent resolved a customer name to the wrong account, passed that customer id into an order lookup, got back an order that belonged to someone else, and then asked the payment service to refund it. The 422 was correct. Entity resolution was the defect, and every step after it was the system faithfully executing a bad premise.

An outcome grader catches this run. It compares the final database state against the goal state, sees no refund, and scores zero, which is the right score. What it cannot tell you is which of the four layers between the model and the payment API produced the wrong answer, and that is the question you have to answer before you can fix anything.

Where the trace sits in the eval stack

Outcome verification stays the primary signal. Anthropic’s guide to agent evals draws the line cleanly: a transcript is “the complete record of a trial, including outputs, tool calls, reasoning, intermediate results,” while the outcome is “the final state in the environment at the end of the trial.” A flight-booking agent that says “Your flight has been booked” has produced a transcript claim; whether a reservation exists in the database is the outcome. Grade the outcome.

The same guide warns against the obvious next move. Checking that an agent followed a specific sequence of tool calls is “too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn’t anticipate.”

I have written about how to build an eval harness around that ordering, and about bucketing failures so a declining score becomes a work queue. This post is about the layer underneath both: what an agent’s tool calls have to record for either to work, and what changes when those calls go over the Model Context Protocol.

Trace the protocol boundary

A single MCP tool call crosses four boundaries:

model decision  →  client serialization  →  server validation  →  downstream API

A trace that records search_customer returned an error has thrown away the only information that distinguishes those four. A model that invented a customer id, a client that dropped an optional field during serialization, a server whose schema validation rejected a valid input, and a database that timed out all arrive at the same flat log line. Each one has a different owner and a different fix, and picking the wrong one costs an afternoon.

The fix is span nesting that mirrors the boundaries:

invoke_agent
└─ chat                       # the model turn that proposed the call
   └─ execute_tool <name>     # gen_ai.tool.name, gen_ai.tool.call.id
      └─ mcp.client.request   # what the client actually put on the wire
         └─ mcp.server.handle # validation result, handler entry
            └─ http POST /v1  # the downstream call, with its own status

The OpenTelemetry GenAI conventions define the top of that stack. invoke_agent and execute_tool are real operation names, gen_ai.tool.name is required on the tool span, and gen_ai.tool.call.id is recommended when available, which is what lets you join a model’s proposed call to the execution that followed it. The bottom of the stack is yours to instrument. These conventions are still Development status and they moved out of the main semantic-conventions repository into their own; I covered what that churn costs you separately.

Two kinds of MCP error

Most trace schemas have one error field. MCP has two error channels, and they mean different things.

The tools specification separates protocol errors, which “indicate issues with the request structure itself that models are less likely to be able to fix” (unknown tool, malformed request, server error), from tool execution errors, which “contain actionable feedback that language models can use to self-correct and retry with adjusted parameters” and are “reported in tool results with isError: true”. The normative guidance follows from that: clients “SHOULD provide tool execution errors to language models to enable self-correction” and “MAY provide protocol errors, though these are less likely to result in successful recovery.”

Those two channels answer different debugging questions. A JSON-RPC error means the integration is broken and no amount of model improvement will help. An isError result means the model asked for something the business logic refused, and whether the model recovered is a genuine behavioral measurement. A trace that records tool_failed=true for both cannot compute either number.

One more field earns its place next to the error: whether a side effect landed before the failure. An execution error after a partial write is the case where an automatic retry does damage, and it is invisible unless the server reports it and the trace keeps it.

Record enough to reproduce the run

A trace you cannot interpret six weeks later is a log file. The interpretation depends on what the tool contract was at the time, and that contract moves.

This is also where advice about MCP has gone stale fastest. Revision 2026-07-28 removed the handshake: the versioning spec now states that “there is no negotiation handshake. Every request carries its protocol version, and the server accepts or rejects each request independently.” Protocol-level sessions went with it, so Mcp-Session-Id is gone from the Streamable HTTP transport. If you are on 2025-11-25 or earlier you are still tracing initialize and a session id, and the spec now labels that path legacy.

What replaces it is per-request and easier to trace, not harder. The debugging guide points at both halves: call server/discover “to see which protocol versions the server supports,” and check that “every request must carry io.modelcontextprotocol/protocolVersion and io.modelcontextprotocol/clientCapabilities” in its _meta. A version mismatch surfaces as UnsupportedProtocolVersionError (-32022) with the supported versions in its data field.

So the run-level fields worth stamping on every trace are the negotiated protocol version, the client and server identities, the capabilities each side declared, and a hash of the tool schemas in effect. The schema hash is the one teams skip and later want. When a server renames a parameter or tightens an enum, every trace recorded before that change becomes ambiguous: you can no longer tell whether the model got the argument wrong or the argument was right under the old contract. A hash turns that into a lookup.

Cross-server data flow

The moment a second MCP server enters the picture, one server’s output becomes another server’s input, and the trace becomes a security artifact rather than a debugging convenience.

MCP’s client best practices are blunt about the trust boundary: “Tool results from one server are untrusted input to another. The broker should apply the same input-review policy to brokered calls as to direct ones; output truncation alone does not prevent exfiltration.” The same page keeps credentials out of the flow entirely, since “API keys and tokens are held by the host” and never reach generated code.

What that asks of a trace is one specific edge:

source_server → returned field → transformation → destination_server argument

Record that and you can answer, after the fact, which server saw which customer record, whether a value that originated in a filesystem server ended up in an argument to an outbound email tool, and whether any of it needed to travel at all. Without it, “the agent leaked a document” is an unanswerable question. Authorization has its own failure mode here too, and the spec’s security best practices cover the confused-deputy case in the OAuth proxy sense.

What the Inspector can and cannot tell you

The MCP Inspector is the right first tool and the wrong last one. It lists tools, shows their schemas, runs them with custom inputs, displays results, surfaces server notifications and logs, and its own best-practices section tells you to check capability negotiation early and to test edge cases: “Invalid inputs; Missing prompt arguments; Concurrent operations.”

That is a component test. It establishes that the tool behaves when a person invokes it on purpose with inputs they chose. It establishes nothing about whether the agent reaches for that tool at the right moment, fills its arguments from context that actually supports them, notices an empty result set, or stops calling it once it has what it needs.

Both layers need their own tests, and the gap between them is where a healthy-looking server fronts a failing agent. A tool that passes every Inspector check can still be described badly enough that the model picks its neighbour, and no amount of server testing will surface that.

Three tiers of trace content

Traces of tool calls contain user prompts, database rows, file contents, and arguments that were never meant to leave the process. The two ecosystems you are most likely to be running disagree on what to do about that by default, which is worth knowing before you assume either one.

The OpenTelemetry convention is opt-in. Instrumentations “SHOULD NOT capture [instructions, user messages, and model outputs] by default, but SHOULD provide an option for users to opt in,” and gen_ai.tool.call.arguments and gen_ai.tool.call.result carry requirement level Opt-In with an explicit note that they may contain sensitive information. The OpenAI Agents SDK is the opposite polarity: it captures LLM inputs, outputs, and function call data by default, and gives you RunConfig.trace_include_sensitive_data and OPENAI_AGENTS_TRACE_INCLUDE_SENSITIVE_DATA to turn it off. Run both in one pipeline and you are running two default postures at once.

Picking a posture deliberately means three tiers rather than one switch:

• Always on. Tool and server names, call ids, parent ids, timing, status, error class, token counts, argument schema validity, content hashes, and a summary of what state changed. This tier answers most debugging questions and carries almost no payload risk.
• Redacted structure. Allowlisted fields, masked identifiers, truncated results, normalized error messages, and secure references to full payloads held elsewhere.
• Full fidelity, restricted. Short retention, explicit access control, its own audit log, and no production secrets.

One rule cuts across all three. MCP’s authorization guidance states it directly: “Never log Authorization headers, tokens, codes, or secrets. Scrub query strings and headers. Redact sensitive fields in structured logs.”

Should a reviewer agent see the call first?

The natural extension of trace auditing is to stop doing it after the fact and put a reviewer in front of the call. It works, selectively, and the reason to be careful about it is now measurable.

Reinforced Agent moves evaluation into the execution loop: “a specialized reviewer agent evaluates provisional tool calls prior to execution, shifting the paradigm from post-hoc recovery to proactive evaluation and error mitigation.” The part worth copying is not the architecture, it is the pair of metrics. The paper scores a reviewer on “helpfulness,” the percentage of base agent errors the feedback corrects, and “harmfulness,” the percentage of correct responses the feedback degrades. A reviewer will do both. One that reports only the first number is not reporting enough to justify itself.

What I would actually deploy, in order:

1. Deterministic gates first, for every rule that can be written down. Scope checks, allowlists, confirmation requirements, and idempotency keys are cheaper, faster, and more predictable than any model reviewer, and they never degrade a correct call.
2. Shadow mode next. Run the reviewer without letting it block anything, and measure harmfulness on calls that were already correct. This is the number that decides whether it ships.
3. Blocking review only where the action is irreversible or externally visible. Outbound email, deletes, payments, permission changes, publishing.
4. Post-hoc review for the rest, sampled rather than exhaustive.

Adding a reviewer because more agents sounds safer buys latency, cost, and one more model whose error rates you now have to measure.

What this looks like in my own projects

APAB is the worked example, because its RF physics tools are served over MCP and the audit trail was the point of the 0.3.0 release. Spans nest as apab.session → turn → apab.tool.<name>, with tool arguments captured under a redaction-aware policy rather than wholesale. The run bundle is where the boundary separation pays off: audit.json holds a per-tool-call record tagged with trace_id and span_id, trace.jsonl holds the spans, and manifest.json holds the config hash and dependency versions, all sharing one trace id. A plot, the calls that produced it, and the timing behind them are one join away from each other.

The MCP layer is deliberately framework-independent, and a Strands agent drives the same tools over stdio in one of the examples. That is section two made concrete: because the tool layer and the agent loop are separate processes, a defect on one side cannot hide inside the other. The golden-task evals then score a run against its own bundle, checking that expected tools were called in a sensible order within a call budget. The scorer is LLM-free, so it runs in CI with no API key.

BrewTrace covers the query-driven half. Every eval case runs inside its own span tagged eval.case_id and eval.passed, so a failing case is a Jaeger query rather than a decrement in an aggregate. Typing eval.passed=false pulls up the complete trace of what the model did instead. That is how a sweep across five local models, whose pass rates ran from 88% to 0%, resolved into four distinct failure modes: one model ignores the tools, one calls them and discards their answers, one reasons its way to an answer it never states. A single aggregate number would have hidden all three. It also carries the naming rule worth repeating anywhere you add domain attributes: custom fields get their own namespace (brew.*, eval.*) and never extend gen_ai.*, because a future revision of the spec may define the name you just took.

Read the trace when something breaks

Trace auditing does three jobs that outcome grading cannot. It locates the first causal error, which is usually several steps upstream of the visible failure. It measures the behaviors that live in the path itself, such as whether the agent recovered from an isError result or repeated the same call four times. And it produces the record that answers what an agent accessed, transmitted, and changed, which is a question you will eventually be asked by someone who is not debugging.

None of that makes the path the standard. An agent that reaches the right end state by a route you did not anticipate has succeeded, and an eval that fails it has a bug. Grade the outcome, gate the handful of invariants that genuinely have to hold, and instrument deeply enough that when the score moves you can say which layer moved it.

This is an independent analysis written on my own time, using the public sources cited above. The projects mentioned are independent work. The views are my own and do not represent any current or former employer.