Cyber frontier eval: GPT-6 Astra and Claude Fable 5.1 repair only a third of the flaws in realworld conditions
On Unpatched, attack success looks higher than repair success under partial acceptance, but requiring a full fix flips that outcome.
Attack runs produce an exploit, input that demonstrates a weakness. Triggered means it exposed the weakness. Exploited means the grader, the code assessing the result, accepted it.
Repair runs produce a patch. Any credit means partial or full acceptance. Fixed means full acceptance.
- 63.5%of completed exploit runs triggered the flawFig 1
- 0.0%of completed exploit runs had the exploit acceptedFig 1
- 21.2% vs 11.8%of completed patch runs earned any credit, and were fully fixedFig 1
What we posed
Unpatched tests unfixed vulnerabilities. We asked systems to write an exploit, a patch, or both in one run.
The table scrolls sideways.
| System | Triggered | Exploited | Any credit | Fixed |
|---|---|---|---|---|
| Claude Code Opus 5 max | 95.0% | 0.0% | 19.4% | 12.9% |
| OpenCode Kimi K3 xhigh | 85.0% | 0.0% | 7.1% | 3.6% |
| OpenCode Qwen 3.8 Max xhigh | 84.2% | 0.0% | 12.5% | 4.2% |
| OpenCode GLM-5.3 xhigh | 78.9% | 0.0% | 20.0% | 8.0% |
| Codex GPT Daybreak Blue xhigh | 54.5% | 0.0% | 14.8% | 11.1% |
| Codex GPT-5.6 Sol xhigh | 40.0% | 0.0% | 29.2% | 16.7% |
| OpenCode Gemini 3.8 Flash xhigh | 25.0% | 0.0% | 27.6% | 6.9% |
Pooled rates combine completed runs across systems
| Any trigger | Any credit | Full exploit | Full fix |
|---|---|---|---|
| 63.5% | 21.2% | 0.0% | 11.8% |
Break first, then fix
The same system met the same weakness twice. Asked to break then fix it, it repaired only the path its own exploit used.
“The pre-fix exploit reached the sentinel and exited 0.”
Its exploit stopped working, but the original weakness remained. The grader gave no credit.
Asked only to repair, the system found a different malformed input, widened the repair, and earned full credit.
Different grading criteria prevent a direct comparison.
No run has been graded on the same scale in both roles.
What else could explain it
- An exploit needs one input that fires. A patch must hold against everything the grader tries.
- Pooling gives systems different weights because they completed different numbers of runs in each role.
What this does not show
Cost and tokens were not measured. No interval is drawn, so a few points between two rows is not a separation. The break-then-fix case is two transcripts, and nothing here counts how often that shape occurs.
We read every rate from the Unpatched leaderboards. Each completed run with a grade counts once.
September 2026
Blocking every attack earned zero when the repair also blocked legitimate users
When one system stopped every planted attack by locking down the service entirely, our legitimate-use check caught the lockout and the grader awarded zero.
August 2026
We graded GLM-5.3-Flash before it had a name
In an earlier evaluation, five of nine systems repaired the vulnerability but then rejected every legitimate message, while every test still passed. The system that kept traffic working was the one we graded before it had a public name.