← Blog

An agent found the free lunch in my phased-array benchmark

2026-08-11

I wanted to know whether an AI coding agent could do real phased-array design, so I wrote a task I thought was hard and a scorer that decides pass or fail from physics rather than from anything the agent says about itself.

The task: a 16 by 16 planar array at 10 GHz, half-wavelength spacing, steered to 30 degrees off broadside. Each element has a 2-bit phase shifter, so it can be set only to 0, 90, 180 or 270 degrees. Thirteen elements are dead, and the scorer zeroes them regardless of what the agent submits. Hold the peak sidelobe below -9 dB.

The reasoning that made this look hard goes like this. With four phase states per element you cannot place the phase you actually want. The residual between the phase you wanted and the phase you got is an error spread across the aperture, and that error radiates. Round the ideal steering phases onto the nearest available state and the sidelobes come out at -7.2 dB, nearly two decibels above the limit, with the beam pulled 3 degrees off target. So the task demands something better than rounding.

That reasoning is wrong, and an agent showed me why in about fifteen minutes.

The run that passed with impossible numbers

I gave the task to Claude Code running Sonnet 5. It passed on the first attempt: peak sidelobe -12.32 dB, directivity 28.39 dBi.

Those are exactly the figures for an ideal array with continuous, unquantized phase control. Reaching them with 2-bit shifters should be about as possible as hitting an arbitrary voltage with a 2-bit DAC.

So I looked at what it submitted. Unit amplitude everywhere, all phases on the four allowed states, thirteen dead elements left alone. Fully compliant. Then I subtracted the ideal steering phases from the submitted ones, element by element, expecting a spread of rounding errors.

Every one of the 256 differences was exactly 45 degrees.

A symmetry that costs nothing

Here is what I had missed. The array is centered on the origin, so with 16 elements the x-positions are half-integer multiples of the spacing: -7.5d, -6.5d, and so on. Steering to 30 degrees with half-wavelength spacing gives a phase increment of exactly 90 degrees per element of spacing. Multiply a half-integer by 90 and every ideal phase lands on 45, 135, 225 or 315 degrees.

Every element, without exception, sat exactly halfway between two available phase states. That is the worst case for rounding, which is why direct rounding scored so badly and why the task looked hard.

The part I had not thought through is that the offset is uniform. Adding a constant to every element phase leaves the radiation pattern untouched, because it only moves the phase reference, like deciding where to start a stopwatch. That costs nothing. Add 45 degrees to all 256 elements and every ideal phase lands precisely on an available state. The quantization error does not get small. It becomes zero, and the array radiates as if the phase shifters were continuous.

The agent found the one-line move that turned a deliberately worst-case geometry into an exactly solvable one. My own reference solution, which used dithered quantization to spread the error into a diffuse floor, was strictly worse: 26.42 dBi against the agent’s 28.39.

The general statement is a physics one, and it is not specific to my task:

When the phase ramp is commensurate with the quantization grid, the coherent worst case is always removable by a global phase offset.

Commensurate means the per-element phase increment and the quantization step share a common measure, which is what happens at the round steering angles and round element spacings anyone reaches for when writing a test case. 30 degrees and half-wavelength spacing are the two most natural choices available, and together they destroy the problem.

The design lesson generalizes past phased arrays. Before trusting a threshold, enumerate the transformations your physics leaves invariant: a global phase, a uniform scaling, a relabelling, a change of origin. Then check that none of them converts the naive approach into a passing one. A benchmark task is a claim that a certain thing is hard, and a free symmetry is the cheapest way for that claim to be false.

The repair, and what it cost

I retargeted the task to (27 degrees, 10 degrees). Off the cardinal cut and away from a round angle, no constant offset helps: sweeping all possible global rotations, the best one still leaves 43 degrees of residual error against a 90 degree grid. Then I raised the requirement from -9 dB to -14 dB, because at the new geometry the numbers had moved.

The middle row measures the repaired exploit, which now buys nothing. A regression test sweeps global phase offsets and asserts that the best of them still leaves more than 20 degrees of quantization error, and separately that it still fails the sidelobe requirement end to end. This class of degeneracy cannot come back silently.

The second bug, which the check agreed with

After that I got suspicious about the scorer itself, and I was right to be, though not for the reason I first landed on.

My first theory was discretization. The evaluator computes the pattern on a grid of 361 by 721 angles and reports the largest sidelobe it finds, and the true peak of a continuous function almost never lands exactly on a sample. So I wrote a second measurement that refines each candidate peak continuously instead of reading it off the grid, compared the two, and found the optimized designs reading about a quarter of a decibel better than they actually achieved. It looked like a clean result: the grid flatters exactly the designs the task exists to discriminate.

It was wrong. The refinement I wrote to check the metric was constrained to stay outside the same 8 degree radius the metric used, so when it climbed uphill it stopped on that boundary and reported roughly what the metric reported. A check that shares an assumption with the thing it checks will agree with it. That is how a wrong number survives being verified.

Removing the constraint showed the real defect. The task scored the highest pattern sample outside a fixed 8 degree radius of the target. For a 16 by 16 array steered to 27 degrees, 8 degrees sits inside the main lobe. A point at exactly 8.00 degrees still climbs uphill to the main peak. So for any design whose real sidelobes fall below its own main-lobe skirt, the metric stopped reporting sidelobes and started reporting the skirt.

Two of the five sat on the boundary, both reporting a value at exactly 8.00 degrees out. Note the direction: the metric understated those designs. The reference really achieves -16.98 dB and was being scored -16.72.

A quarter of a decibel of pessimism is the small part. The metric saturated. Once a design pushed its sidelobes below its own skirt level, further suppression stopped registering at all, which is the one thing a benchmark has to be able to see. There was also a symptom I had blamed on sampling: refining the evaluation grid moved the reported value around by 0.23 dB without converging, because the reported value was whichever sample happened to land nearest the hottest point on an 8 degree circle, and that changes with grid density.

The fix is to detect local maxima and drop the main peak, which is what a sidelobe is. Under that metric the 361 by 721 reading agrees with unconstrained continuous refinement to within 0.01 dB on every design measured, and it returns the same value at every grid density I tried. Grid sampling was never the problem.

The part I find hardest to write down is that I had fixed this exact bug the day before, in a different codebase. A source audit of my own phased-array-systems found that its compute_sidelobe_level excluded the main beam out to one half-power beamwidth, and the first null of a tapered aperture sits at roughly 1.3 to 1.8 times the beamwidth. A 32 by 32 array with a Taylor -35 dB taper reported -14.03 dB where the true peak sidelobe is -35.24 dB. Same defect, different code, written at a different time for a different purpose, and I did not recognize it when it appeared again a day later.

Where AEDL sits in the tool stack

I maintain a few open-source phased-array tools, layered the way a real design flows. EdgeFEM is the full-wave finite-element solver for when a unit cell needs real electromagnetics. phased-array-modeling computes radiation patterns with the impairments that matter: mutual coupling, phase quantization, tapering, element failures. phased-array-systems runs the requirements-driven trade studies on top. APAB is an LLM agent that drives that chain over the Model Context Protocol.

AEDL is the layer that asks whether any of that agent work produces designs that survive contact with physics. It owns almost no physics of its own. A task is a YAML file with a geometry, hardware constraints and a set of requirements, each expressed as a metric and a bound. An agent gets the task and a submission contract, works in an isolated directory, and leaves a weight vector. The evaluator computes the pattern, enforces the hardware constraints itself rather than trusting the submission, zeroes the dead elements, and scores each requirement.

The rule the whole thing rests on is that agents never grade themselves. An agent may legitimately re-derive the scoring metric from the public libraries. The task asks whether it can design a compliant array, and working out how that array will be measured is part of doing so. Reading the worked solution is off limits.

What this does not show

One task. Three attempts, all of which passed, at -15.96, -16.52 and -17.59 dB against a -14 dB bar. One model family. None of the three beat the reference’s -16.98 dB, though the third came within 0.6 dB. The dollar figures I record ($1.83, $1.06, $0.93) are API-equivalent estimates rather than money spent, since those runs were subscription-covered.

The attempts also cannot be certified uncontaminated. They predate a fix that strips read permission from the reference solutions while a run is in progress. During one attempt the agent ran find / -iname "*array_pattern*", hunting the filesystem for the scoring code, and another attempt’s transcript records that it re-ran the evaluator source to check its work. Re-deriving the metric is permitted. What those two facts establish is that the agent had this repository in reach during a scored attempt, and the run format I used keeps no tool-call log that could show whether it also opened the reference directory. Separately, none of the three recorded a call to an instrumented physics entry point, which is in tension with the claim to have re-run the evaluator, since the evaluator calls into the pattern library. That is unresolved.

The physics that decides pass or fail comes from a library I wrote myself and have not validated against an independent solver, and that is the first thing I would want checked if someone showed me this work. Earning the word benchmark takes more tasks than one and at least one result that fails.

What this looks like in my own projects

Both defects were invisible to analysis and obvious to measurement. I had reasoned carefully about the first one and reached the wrong conclusion, because my reasoning was about quantization error and the flaw was about symmetry. I found the second only because the first had made me distrust my own setup, and then I got its mechanism and its direction backwards on the first try.

That is the same failure I keep writing about from other angles. In how to measure a grader the point is that a scorer is a measurement instrument with its own error rate, and reporting a pass rate without characterizing the grader tells you very little. Here the grader had a 0.26 dB bias and a saturation ceiling, and the saturation was the part that mattered. How to build an eval harness for LLM agents has the recipe AEDL follows: a versioned task spec, isolated trials, outcome-first grading, and reliability measured across repeated runs. What I would add after this week is a sequencing note. I wrote the harness before the rest of the task set, on the argument that a difficulty threshold stays a guess until an agent has attempted it, and three attempts on one task is enough to notice a degenerate task even though it is nowhere near enough for a pass rate. It paid for itself on the first run by invalidating the task I was most confident in. Failure taxonomies is about grouping this sort of thing before trying to fix it, and the grouping here is now explicit: a main-lobe exclusion that does not reach the first null is apparently my default mistake, so any new metric that separates a main beam from its sidelobes gets checked against local-maximum detection before I trust it.

The agent-side lesson lands where APAB 0.3.0 put its effort, on provenance and evals rather than on the agent itself, and where agentic systems change what engineers review lands: the durable artifacts are the tools and the audit trail. The phased-array-systems 0.10.0 changelog carries the sibling fix, and model-based engineering for phased arrays has the longer story of how these tools came to exist.

Try it

The tasks are the point, so clone the repository rather than installing from an index.

git clone https://github.com/jman4162/aedl-electromagnetic-design-agent
cd aedl-electromagnetic-design-agent
pip install -e ".[dev]"

aedl list
aedl run --task t2-001 --agent claude --model sonnet --attempts 3

Every run writes a bundle with a provenance manifest, per-requirement scoring, the agent’s workspace, and a log of every physics call tagged by fidelity tier, even when the agent crashes or submits nothing. scripts/verify_sidelobe_metric.py reproduces the comparison table above, and docs/roadmap.md records the calibration log and why each decision was made.

What I would say with confidence is narrower than where I started, and more useful. An agent given a physics task will find the free lunch in it if there is one, which makes it a good instrument for discovering that your evaluation is broken. Better to learn that before publishing a leaderboard.

AEDL is pre-release and deliberately narrow: one calibrated task, one evaluator, and a harness. The numbers here are reproducible from the reference solution and the tests in that repository, and the caveats above limit what they support.

AEDL and the phased-array tools are independent projects I build on my own time. The views are my own and do not represent any current or former employer.