What agents do when they lose

An agent that fails to repair a vulnerability can fail in six ways. It can submit a patch that builds and leaves the vulnerability triggerable. It can submit a patch that does not build. It can submit a patch that builds and then never finishes running. It can decline to submit. It can run out of time. Or the attempt can be lost before the verifier sees it at all. A pass rate collapses all six into one number, and the six do not mean the same thing.

The board, and what it was selected for

These figures come from the expert board, recorded 2026-05-08: 100 samples, 16 agent configurations, 1,581 scored attempts. One rule chose the samples: every frontier agent already failed the sample under both its own harness and a second one. Rates on this board are conditioned on frontier failure and are not comparable to a randomly drawn set. Read every rate below as conditional on a sample that already defeated the frontier once.

It is a hard board and it stays hard. 14 of the 100 samples were repaired by nothing we ran, and 86% still separate one agent from another. What happens to the attempts that do land is the companion note, most attempts score exactly zero.

The mix

Repairs: 465. Patches that built and still triggered: 1,075, or 68% of scored attempts, the largest bucket on the board by a wide margin. The rest is small, and the behaviour is in the rest: 12 attempts broke the build, 16 declined to submit, 11 ran out of time, and 2 produced a patch that built and then never terminated. A further 15 attempts never reached the verifier and are excluded from every rate.

The rubric puts broken output below no output

The board carries two columns. Pass rate is repairs over scored attempts. The signed score subtracts the attempts that broke the build or never terminated, and leaves declining and running out of time at zero, which is also what a patch that built and missed is worth.

So the order the rubric asserts runs: a repair is worth one, a wrong but buildable patch is worth nothing, declining is worth nothing, and a patch that breaks the build is worth less than nothing. claude running claude-opus-4-7 carries 35.9% by pass rate and 33.7% signed, a 2.2-point penalty from 2 attempts. kiro running claude-opus-4.6 declined 8 times and its two columns are identical.

Whether that order is right is a judgement about who pays downstream. A patch that does not compile costs a reviewer a minute. A patch that compiles, reads well, and hangs in production costs considerably more. The rubric prices them the same, and prices both below a patch that quietly does nothing.

The attempts that never terminated

Both are the same sample, attempted by two different agent configurations. Both built. Both ran past the verifier's replay budget. Both scored below zero.

Neither is an agent gaming a reward. Both agents diagnosed the same root cause in the same decoding routine, and each wrote a one-line fix at a different call site. Both wrote the fix a maintainer would write: a size field was carried forward across files in a way that let it outrun the buffer it described, and both patches restored it to the buffer's real size. The out-of-bounds read genuinely had that cause. Two independent diagnoses converging is strong evidence the diagnosis was right.

What neither agent saw is that the oversized value was also the bound on the decode loop. Remove the overrun and the loop stops terminating on this trigger. The memory-safety defect was sitting on top of a non-termination defect and hiding it.

We read both trajectories for any sentence about termination, looping, or runtime. Neither contains one. Every confidence statement in both is about memory safety. Neither agent ever built or ran the code: one spent its shell calls entirely on version control, the other started a build, got one command away from configuring it, and returned to reading source. The failure is not a reward being gamed. It is a fix whose correctness is dynamic, asserted from reading alone.

Behaviour and infrastructure are different facts

Two of the small buckets sit next to each other in a table and mean opposite things.

Declining to submit is a choice made with a working environment. Of the 16 declines on the board, 50% belong to one configuration: kiro running claude-opus-4.6, which declined 8 times against 3 for the next agent down. The same model runs under 3 other harnesses here and declined 3 times across all of them, 2 of the 3 never once. The model is part of it. The pairing is most of it.

Attempts lost before scoring are not a choice at all. kiro running claude-opus-4.6 lost 11 of its 100 attempts before the verifier saw them, leaving 89 scored and a correspondingly wider interval. That is a fact about a provider, not about a model. Only the first of the two is behaviour, and a board that reports one column cannot tell you which you are reading.

What this does not show

We cannot tell from the published record whether the declines are a model refusing the task or a harness failing to capture a submission the model made. The concentration in one pairing fits both readings, and separating them needs harness logs we do not publish.

The non-terminating attempts are one sample. One sample supports no rate, so we make no claim about how often a correct-looking repair fails a liveness check. We also did not check whether the maintainer's own fix for that sample terminates under the same replay. If it does not, the check is measuring the sample rather than the patch, and that cuts against us.

Every figure here is conditioned on frontier failure. These are the ways agents lose on samples chosen because agents lose on them. What agents do on ordinary work is a different measurement, and this board does not contain it. The full outcome mix sits on every row of the board. The gap between the two columns is the part of a run that a single rate does not carry.