Blocking every attack earned zero when the repair also blocked legitimate users
One system stopped every planted attack by locking the service down, and the grader gave it zero. Coding systems were asked to repair flaws in cryptography services, each patch graded by code.
The cryptographic primitives and the attack games that break them in the leaderboard environments are taken from CryptanalysisBench (Fluri et al., ETH Zürich, arXiv:2607.18538); the multi-service environments they are embedded in, the incident evidence, and the grading that requires both a defeated attack and preserved legitimate use are XOR's own.
Every patch is graded on two checks against the same patched service: each planted attack must fail, and a legitimate user must still get through.
- 3 systemsfully fixed every graded runFig 1
- 33.3%of the locked-out system's graded runs were fully fixedFig 1
The table scrolls sideways.
| System | Fixed |
|---|---|
| OpenCode Gemini 3.1 Pro Preview xhigh | 100.0% |
| OpenCode Gemini 3.8 Flash xhigh | 100.0% |
| OpenCode Kimi K3 xhigh | 100.0% |
| Claude Code Opus 5 max | 66.7% |
| OpenCode Qwen 3.8 Max xhigh | 66.7% |
| OpenCode GLM-5.3 xhigh | 66.7% |
| Codex GPT Daybreak Blue xhigh | 33.3% |
| Codex GPT-6 Astra xhigh | 33.3% |
What the grader recorded
Both locked-out patches received zero. One left the attack targets unreachable because the service was down.
The newest release on this leaderboard is not at its top.
GPT-6 Astra passed 33.3% of its graded runs. Its largest patches stopped every attack by locking the service down.
Kimi K3 passed 100.0% of its graded runs and Gemini 3.8 Flash 100.0%.
What else could explain it
- The harness, the software that runs the model, differs. Systems that locked users out used Codex. Those that passed environment A used OpenCode or Claude Code.
- Differences in checks for legitimate use remain untested. The system that locked users out twice has no pass on a third environment for comparison.
- We did not compare whether the briefs clearly required keeping legitimate use available.
- Other systems passed the same check with small patches. That weakens the explanation that large patches were correct and the check too strict.
What this does not show
Whether this holds at scale is not measured. No environment was run twice, so run-to-run noise is not separated from the difference between systems.
Grades and patch sizes come from the team that ran them. Newest run 11 September 2026. The environments are A and B. Their weaknesses are authored and not public.
September 2026
Cyber frontier eval: GPT-6 Astra and Claude Fable 5.1 repair only a third of the flaws in realworld conditions
Attack and repair success on Unpatched: grading criteria matter.
August 2026
We graded GLM-5.3-Flash before it had a name
In an earlier evaluation, five of nine systems repaired the vulnerability but then rejected every legitimate message, while every test still passed. The system that kept traffic working was the one we graded before it had a public name.