The samples nobody repaired

A leaderboard answers which model is ahead. It does not answer the question a security team actually has, which is what remains unfixed after you have tried everything available. For that you stop ranking models and start taking their union.

The union does not close

Counting a sample as covered if any model repaired it in any attempt, 103 of 128 samples are reachable — 80.5%. The remaining 25, or 19.5% of the corpus, were never repaired by anything we ran.

That number is not a scoring artefact. These samples had a verifier, a build, and a known fix. Every model in the run reached them and none produced a repair the verifier accepted.

And it closes fast, then stops

The returns from adding models collapse almost immediately. The single best model covers 62.5% of the corpus. Adding the model that contributes the most new repairs on top of it reaches 75%. Every remaining contributor together adds the rest, and after 6 models nothing further is gained at all.

For a buyer this is the difference between a portfolio strategy and a procurement one: running a second model is worth real coverage, running a sixth is worth almost none. The full step-by-step table is on the economics page.

What this does not show

We do not know why the residue resists repair; that requires reading the trajectories sample by sample, which we have not published. Coverage here is also generous by construction — a single success in a single attempt counts, which flatters the pooled number relative to what any one team would achieve in one run. And this is one corpus: another corpus with a different weakness mix would have a different residue. It would not, we think, have none.