← Blog

Data provenance for agentic AI workflows

2026-08-17

Three agent submissions to my phased-array benchmark were scored on 2026-08-10, and all three passed. The sidelobe requirement was measured by taking the highest pattern sample outside a fixed 8 degree radius of the beam target. For that geometry the 8 degree ring sits inside the main lobe, so the metric was scoring the main lobe’s own skirt and saturating once designs got good enough to reach past it.

I fixed the metric later. Under the current evaluator, the third submission achieves a peak sidelobe level of -17.59 dB. Its stored result.json says -17.02 dB, because that is what the evaluator said at the time, and the run bundles are committed rather than rewritten.

Both numbers being on record is the system working. The problem is the next question. Which of the figures, tables and claims I have since published rest on the pre-fix metric, and which do not? Every bundle points backward to its own inputs in detail. Not one of them points forward. Answering that question meant reading files by hand, and reading files by hand does not scale past the point where it stops being embarrassing and starts being wrong.

That gap is what provenance is for, and agentic systems widen it.

What provenance means

The W3C PROV data model, a Recommendation since April 2013, gives the vocabulary. An entity is “a physical, digital, conceptual, or other kind of thing with some fixed aspects”. An activity is “something that occurs over a period of time and acts upon or with entities”. An agent is “something that bears some form of responsibility for an activity taking place, for the existence of an entity, or for another agent’s activity”. Entities are generated by activities (wasGeneratedBy), activities use entities (used), entities derive from other entities (wasDerivedFrom), and both attribute back to agents (wasAttributedTo).

That is three node types and a handful of edges. Its value is that it forces you to name the edges rather than infer them from timestamps.

Five words get used interchangeably here and answer different questions.

Lineage is the dependency graph. Provenance is that graph carrying enough evidence to be checked, and traversable in both directions. Most systems that claim provenance have lineage and a backward traversal.

An agent picks its own inputs

The reason this changes with agents is narrow and worth stating plainly. In a conventional pipeline you can write the input list down in advance, because the code names its inputs. When a model chooses tools at runtime, the list of things a result depends on is not knowable until the run is over. So it has to be recorded while it happens, by machinery the agent does not control.

My benchmark harness instruments physics-model calls by generating a sitecustomize.py and prepending it to PYTHONPATH, so any python the agent shells out to gets counted whether or not the agent cooperates. The line in the README is the whole argument for doing it that way: that record “cannot be reconstructed after a run”.

Inputs also include the environment. My harness passes agents an explicit allowlist of ten variables, because a probe of the shell it was launched from found 32 of 41 variables reaching the child process, including a live SSH_AUTH_SOCK. An agent under test was separately observed running find / -iname "*array_pattern*", looking for the scoring code. Neither of those inputs appears in any task definition.

Four things a run has to pin

Here is what my run manifests carry, which is a reasonable minimum for any agent workflow whose outputs someone will later cite.

The specification, by content. task_sha256 is a hash of the whole task file, so a run pins the exact spec it was scored against rather than a filename that later changed underneath it. The same applies to MCP server configuration, which is hashed both as written and as the harness actually rewrote it, so the question of what the agent received stays answerable.

Both environments. The harness records the library versions it scored with and, separately, probes the interpreter the agent actually reaches, because the harness virtualenv is not on the agent’s PATH. They disagree. Across the 17 committed bundles, 14 were scored with numpy 2.5.2 against an agent designing with 2.5.1, and 8 of those were scored with phased-array-systems 0.10.1 against an agent using 0.11.0. Both versions satisfy the declared constraint >=0.10,<1.0, so no dependency check would have caught it. Only comparing the two halves does.

Recording both halves is only useful if something reads them against each other, which until this week nothing did. The comparison is fiddlier than a dict diff, because the agent’s probe reports import names and the harness reports distribution names, and the interesting pair is not mechanical: the package phased_array is installed by the distribution phased-array-modeling.

What the agent actually did. The tool-call transcript, the per-call physics log tagged by fidelity tier, and the spans the MCP servers emitted.

The verdict and the code that produced it. This is the one my record was missing. task_sha256 pinned what was asked and nothing pinned what did the asking, which is exactly why the sidelobe correction is hard to bound. Manifests now carry the harness git SHA, a dirty flag, the evaluator dispatched to, and a hash of that evaluator’s source.

A version field alone would not have worked. Every bundle in my history reports its aedl version as 0.0.1, read from installed package metadata that a stale editable install froze there while the source moved to 0.1.0.dev0. The value is present, well formed, and wrong in a way nothing flags. Content hashes exist because of that failure mode.

A trace covers one execution

I have written before about what a tool call has to record across the MCP boundary and about the state of the OpenTelemetry GenAI conventions. Both are about the inside of one run. Here is what that leaves open, taken from a bundle I published.

server-trace.jsonl in one t3-001 run holds 8 spans for 8 MCP tool calls. All 8 carry a distinct trace_id and parent_span_id: null. Every tool call opened its own root trace. In the same bundle, transcript.jsonl records the same 8 calls, identifying one of them as toolu_01KTzrTbkoyMTVGRmJcgg6bK. Neither record carries the other’s identifier.

transcript.jsonl        server-trace.jsonl
  toolu_01KTz...          trace_id 7005be14..., parent null
  toolu_...               trace_id 8d2aaf32..., parent null
  toolu_...               trace_id b2592354..., parent null
        \                        /
         two accounts of the same eight calls,
         joinable only by tool name and count

The cause was mundane. The MCP server honors an inbound W3C traceparent and my harness never sent one, so I was collecting spans that belonged to nothing. Forwarding TRACEPARENT into each server’s environment and recording the trace id in the manifest closes it: the spans now share the run’s trace and the bundle joins to them.

The residue is worth naming, because it is the general shape of the problem. The variable is read once at server start, so the spans are siblings under a run-level root and still cannot be attributed to the individual agent turn that caused them. That needs per-request metadata on every tool call. Tracing gives you ordering inside a process. Provenance needs identifiers that outlive it.

Identify data by content

This is the layer that ML people usually have and agent builders usually skip. A path like s3://bucket/training/latest/ is a pointer, and pointers move. Capture the object version id, the table snapshot, the commit, or a content hash of the bytes.

Two tools show the two available shapes. DVC “enables data versioning through codification”: you commit small metafiles to Git describing what to track, and the data itself lives in a cache synchronized to a remote. lakeFS puts “version control over the data lake, using Git-like semantics”, where a commit is “an immutable checkpoint containing a complete snapshot of a repository”, and its own docs sell that for tracking “the transformation of data from raw datasets to the final version used in experiments”.

For the graph that connects them, OpenLineage is the portable vocabulary: a Job is “a process that consumes or produces Datasets”, a Run is “an instance of a Job that represents one of its occurrences in time”, and facets carry everything else. It is an LF AI and Data Foundation project, so the vocabulary outlives whichever tool you happen to run.

One caution on what to store. A provenance record wants identifiers, versions, hashes and metadata rather than copies of the payload, both for size and because a metadata store that accumulates credentials, customer data and proprietary prompts becomes its own problem. My server spans record args_hash and the list of argument keys without the values. W&B’s reference artifacts take the same approach for external data, storing checksums and URIs while the files stay where they are.

Evidence and its authority

The layer above lineage is the one I think separates serious agent systems from demos: recording what a claim rests on, and how much that source was entitled to settle the question.

Two mechanisms in my own stack do this concretely. The harness reads each run’s transcript for filesystem calls touching the withheld reference solutions and writes an integrity field of clean, suspect or unknown. The code describes it as “a flag for review, not a verdict”, which is the right posture: it marks evidence as possibly contaminated and leaves the judgement to a person. Separately, the RF tool server refuses to compare simulated against measured data unless the measured file carries a provenance sidecar naming the instrument, date, calibration state, uncertainty, operator, and whether the data is synthetic. A tool that declines to proceed without provenance is worth more than a dashboard that displays it.

The honest limit sits in the same repository. For the three earliest bundles, my notes record that one attempt ran that filesystem-wide find and another re-ran the evaluator source, that re-deriving a metric is permitted while reading the worked solution is not, and that “the run format used here keeps no tool-call log that could settle which happened”. Those three runs can never be certified either way. The record you kept is the record you have.

Nothing standardizes this layer yet. PROV predates LLMs. The OpenTelemetry GenAI conventions identify which data source was queried rather than which passage supports which sentence. C2PA covers asset origin and AI disclosure. The rest is research, and a 2026 survey of evidence tracing and execution provenance in LLM agents is a reasonable map of it. If you build claim-to-evidence edges today, you are designing your own schema. My planner-lab does a narrow version of this by failing any memo containing a number that cannot be traced to a computation.

What replay can and cannot give you

Provenance is often sold as reproducibility, and with agents those come apart.

My evaluators are deliberately deterministic. The registry docstring requires that the same specification and the same submission yield the same result, seeds are fixed, held-out failure envelopes are enumerated rather than sampled, and the binding worst case is recorded alongside the seed that produced it. Replay that side and you get identical numbers.

The agent side does not behave. Six attempts at the same X-band radar task, same task hash, same model, same 30 minute timeout, same harness:

Two orders of magnitude in model evaluations between the cheapest pass and the timeout. The bundles in my repository are committed rather than regenerated for exactly this reason: a nondeterministic agent cannot reproduce them, so the artifacts are the evidence.

So the useful goal is reconstructability. Provenance should let you state precisely which inputs, versions, prompts and configuration the agent observed, so that when two runs disagree you can attribute the difference to a cause rather than shrug at it. Identical token sequences are neither achievable nor the thing you needed.

What to learn, and in what order

Deploying every tool below is a reliable way to produce metadata nobody reads. The durable part is the vocabulary, because it survives changing your mind about products.

SLSA is worth ten minutes even if you never adopt it, because its definition is the clearest one-liner in the field: provenance is “verifiable information that can be used to track an artifact back, through all the moving parts in a complex supply chain, to where it came from”. Note “verifiable”. A dashboard showing boxes is not that.

Of my own stack, I run none of these products. The harness writes its own manifest. That is a defensible choice at one-person scale and stops being one the moment someone else needs to query across runs, which is the point at which the vocabularies start earning their keep.

The forward query

Here is the test I would apply to any agent system whose outputs get cited.

Pick a claim it produced. Can you walk backward through machine-readable records to the evidence underneath, naming versions at every step? Then pick an input that turned out to be wrong. Can you walk forward to everything that depended on it?

My bundles now answer the first question. They still do not answer the second, which is why the sidelobe correction cost me an afternoon of reading files. The shape of the fix is already written in my own planning notes as an experiment store holding (requirement, hypothesis, design, simulation, result, decision) tuples with provenance and cost. That tuple is a PROV graph with the names changed: requirements and results are entities, simulations are activities, agents are agents, and the edges between them are the ones that make invalidation a query instead of an afternoon.

If the answer to both questions is yes, you have provenance. If the answer is that you kept the agent logs, you have logs.

The AEDL and APAB numbers above were re-read from the committed run bundles and source while writing: the 17 manifests in runs/, server-trace.jsonl and transcript.jsonl from 20260812T070655Z_t3-001_claude_8a38b7ac, and the t3-002 attempt table in runs/README.md. Costs are API-equivalent estimates for subscription-covered runs. Sources for the external tools are linked inline and were checked on 2026-08-17.

AEDL and the phased-array tools are independent projects I build on my own time. The views are my own and do not represent any current or former employer.