A quarter of attempted repairs never compiled

Our verifier separates three outcomes: the patch built and the vulnerability no longer triggers; the patch built and it still triggers; the patch did not build at all. Most benchmarks collapse the last two into "fail". That collapse hides the finding below.

The size of it

27.3% of scored attempts in this run produced a patch that did not compile. Not a fix that missed the vulnerability — a diff the project could not build. Split by harness, the range is wide: opencode attempts fail to build 36.2% of the time, claude attempts 18.3%.

That ordering tracks the harness gap reported in the harness is part of the model. It suggests a plain mechanism for part of that gap: harnesses that let a model compile and iterate before submitting convert would-be build failures into scored attempts, and some of those land.

Where it concentrates

Build failure is not spread evenly across weakness classes. It is heaviest in Access of Resource Using Incompatible Type ('Type Confusion') (CWE-843), where 65.5% of attempts never compiled against a repair rate of 34.5%. Repairing that class means changing types or lifetimes that the rest of the program depends on, so a half-correct edit does not merely miss — it stops the project building.

This is why we publish the outcome mix on every board row and weakness class rather than a single pass rate. Two models at the same score can be failing in ways that need completely different fixes.

What this does not show

We do not claim the build failures would otherwise have been correct repairs; most probably would not. We also cannot attribute build discipline cleanly to the harness, because harness and model are not fully crossed in this run — some pairings do not exist. And a build failure in our verifier is our environment's build, not the project's CI: a patch rejected here might build elsewhere under different flags. That is a limit of our instrument, and it cuts against us.