Skip to main content
XOR

Which agents repair real vulnerabilities?

Every agent runs the same corpus and is graded by the same verifier: a repair passes only if the vulnerability no longer triggers and the software still works. Scores are pass rates over scored attempts, with 95% intervals clustered by codebase. How we test.

The weakness class matters more than the model: agents repair Out-of-bounds Read at 57.2% and Use of Uninitialized Resource at 22.2% — the point estimates differ 2.6×. Their intervals overlap at the edges, so read this as a large gap whose exact size this corpus does not pin down.

By weakness class

WeaknessRepair rateRate95% intervalAttempts
CWE-125Out-of-bounds Read57.2% (95% interval 43.9–66.4)57.2%43.9–66.41101
CWE-787Out-of-bounds Write55.5% (95% interval 40.3–73.0)55.5%40.3–73.0247
CWE-416Use After Free31.7% (95% interval 13.2–57.5)31.7%13.2–57.5186
CWE-908Use of Uninitialized Resource22.2% (95% interval 8.0–50.6)22.2%8.0–50.6126
CWE-415Double Free53.5% (95% interval 0.0–80.0)53.5%0.0–80.043
CWE-1284Improper Validation of Specified Quantity in Input44.8% (95% interval 0.0–86.7)44.8%0.0–86.729
CWE-843Access of Resource Using Incompatible Type ('Type Confusion')34.5% (95% interval 0.0–66.7)34.5%0.0–66.729
CWE-476NULL Pointer Dereference24.1% (95% interval 14.3–33.3)24.1%14.3–33.329
CWE-47540.0% — no interval: too few codebases or attempts to generalise40.0%—151 codebase
CWE-56226.7% — no interval: too few codebases or attempts to generalise26.7%—151 codebase

A dash means no interval: the class comes from a single codebase, or carries too few attempts for a bootstrap to say anything. The point estimate stands; there is nothing to generalise from. Rows sharing a rank are tied — their intervals overlap.

By model

RankAgentModelOutcomesRate95% intervalAttempts
1=codexgpt-5.279 repaired · 12 not repaired · 35 did not build62.7%51.9–71.5126
1=cursoropus-4.680 repaired · 24 not repaired · 24 did not build62.5%50.7–71.9128
1=claudeclaude-opus-4-677 repaired · 28 not repaired · 20 did not build61.6%48.9–71.3125
1=gemini31gemini-3.1-pro-preview64 repaired · 18 not repaired · 27 did not build58.7%44.0–71.2109
1=opencodegemini-3.1-pro-preview67 repaired · 25 not repaired · 30 did not build54.9%42.9–66.7122
1=cursorgpt-5.263 repaired · 34 not repaired · 25 did not build51.6%38.1–63.6122
1=opencodegpt-5.263 repaired · 11 not repaired · 48 did not build51.6%39.5–63.1122
1=cursorgpt-5.3-codex64 repaired · 40 not repaired · 23 did not build50.4%36.2–61.2127
1=codexgpt-5.2-codex63 repaired · 27 not repaired · 38 did not build49.2%35.9–59.8128
1=opencodeclaude-opus-4-658 repaired · 15 not repaired · 49 did not build47.5%32.0–59.7122
1=claudeclaude-opus-4-558 repaired · 43 not repaired · 26 did not build45.7%33.1–55.5127
1=cursorcomposer-1.557 repaired · 39 not repaired · 30 did not build45.2%32.5–58.0126
1=geminigemini-3-pro-preview55 repaired · 36 not repaired · 37 did not build43.0%32.8–52.1128
2=opencodegpt-5.2-codex48 repaired · 32 not repaired · 47 did not build37.8%25.0–48.7127
2=opencodeclaude-opus-4-546 repaired · 29 not repaired · 50 did not build36.8%25.0–48.5125

Bars show repaired / not repaired / did not build. Intervals overlap across most of this table: treat close ranks as a tie.

What this does not measure

These results score the repair stage only. Whether an agent can locate a weakness on its own, or trigger it, is measured in the environment estate but is not published here — those stages read as finding and exploit, not as a zero. A permanent held-out slice never appears in any published number, so a model cannot be tuned against this page.

Explore every run →What a repair costs →

Corpus cve-bench-136 · run February 2026 · 1,864 scored evaluations · /benchmarks.json