Unpatched vulnerabilities
Vulnerabilities XOR researchers discovered and disclosed in a confidential inference stack.
The environments and their graders are withheld; the leaderboards show aggregate results.
Exploit and repair
With limited vulnerability information
Average task score (%). Exploit and repair use different checks.
| System | Exploit scoreAverage exploit task score, shown as a percentage. Each system has 9 or 10 matched comparisons. | Repair scoreAverage repair task score, shown as a percentage. Each system has 9 or 10 matched comparisons. |
|---|---|---|
| Claude Code / Opus 5 | 60.0% | 21.7% |
| Codex / GPT-5.6 Sol | 13.3% | 23.3% |
| Codex / Daybreak | 13.3% | 20.0% |
| OpenCode / Gemini 3.8 Flash | 4.4% | 21.7% |
| OpenCode / Kimi K3 | 51.9% | 13.0% |
| OpenCode / Qwen3.8 Max 0902 | 53.3% | 0.0% |
| OpenCode / GLM-5.3 | 59.3% | 31.5% |
Method and source
Expertise-selected software environments include investigated and constructed cases. The collection is not a representative population.
Repeated scores are averaged within each role and cell; complete environment–information cells have equal weight.
Attack and repair are matched within environment, information condition and system. The role comparison excludes combined attack and repair.
Claude Code / Opus 5: 10 matched comparisons; Codex / GPT-5.6 Sol: 10 matched comparisons; Codex / Daybreak: 10 matched comparisons; OpenCode / Gemini 3.8 Flash: 10 matched comparisons; OpenCode / Kimi K3: 9 matched comparisons; OpenCode / Qwen3.8 Max 0902: 10 matched comparisons; OpenCode / GLM-5.3: 9 matched comparisons.
Percentages are mean partial-credit task scores. Attack scores reflect trigger checks.
Does AI Help Cyber Attackers or Defenders? — Tables II and IIIExploit
Shallow exploit: the test input triggered the flaw, full included. Full exploit: the exploit passed the task’s acceptance check.
the test input triggered the flaw; includes full exploits | the exploit passed the task’s acceptance check | |
|---|---|---|
| Claude Code Opus 5 max | 95.0% | 0.0% |
| OpenCode Kimi K3 xhigh | 85.0% | 0.0% |
| OpenCode Qwen 3.8 Max xhigh | 84.2% | 0.0% |
| OpenCode GLM-5.3 xhigh | 78.9% | 0.0% |
| Codex GPT-5.6 Sol xhigh | 40.0% | 0.0% |
| OpenCode Gemini 3.8 Flash xhigh | 25.0% | 0.0% |
How we measured this
- The full-exploit column reports grader acceptance. It does not establish that every attack failed.
- What is measured. Trigger checks and accepted exploit submissions.
- Shallow exploit. the test input triggered the flaw; includes full exploits
- Full exploit. the exploit passed the task’s acceptance check
- Support. Rates count graded runs; repeated runs count separately.
- Graded runs per system. Claude Code Opus 5 max: 20 graded runs; OpenCode Kimi K3 xhigh: 20 graded runs; OpenCode Qwen 3.8 Max xhigh: 19 graded runs; OpenCode GLM-5.3 xhigh: 19 graded runs; Codex GPT-5.6 Sol xhigh: 20 graded runs; OpenCode Gemini 3.8 Flash xhigh: 28 graded runs.
- Order. Sort by any column. No rank is assigned.
- Contamination. No public fix existed when these vulnerabilities were evaluated; the grader tests whether the vulnerability still triggers.
- Get in touch
Repair
Shallow patch: a nonzero score on the task’s checks, full included. Full patch: 100% on those checks.
the repair earned a nonzero score on the task’s checks; includes full patches | the repair earned 100% on the task’s checks | |
|---|---|---|
| Codex GPT-6 Astra xhigh | 41.2% | 35.3% |
| Claude Code Fable 5.1 high | 42.9% | 28.6% |
| Codex GPT-5.6 Sol xhigh | 29.2% | 16.7% |
| Claude Code Opus 5 max | 19.4% | 12.9% |
| OpenCode GLM-5.3 xhigh | 20.0% | 8.0% |
| OpenCode Gemini 3.8 Flash xhigh | 27.6% | 6.9% |
| OpenCode Qwen 3.8 Max xhigh | 12.5% | 4.2% |
| OpenCode Kimi K3 xhigh | 7.1% | 3.6% |
How we measured this
- What is measured. Two readings of every graded patch attempt: whether the patch earned full credit, and whether it earned any credit at all.
- Shallow patch. the repair earned a nonzero score on the task’s checks; includes full patches
- Full patch. the repair earned 100% on the task’s checks
- Support. Rates count graded runs; repeated runs count separately.
- Graded runs per system. Codex GPT-6 Astra xhigh: 17 graded runs; Claude Code Fable 5.1 high: 7 graded runs; Codex GPT-5.6 Sol xhigh: 24 graded runs; Claude Code Opus 5 max: 31 graded runs; OpenCode GLM-5.3 xhigh: 25 graded runs; OpenCode Gemini 3.8 Flash xhigh: 29 graded runs; OpenCode Qwen 3.8 Max xhigh: 24 graded runs; OpenCode Kimi K3 xhigh: 28 graded runs.
- Order. Sort by any column. No rank is assigned.
- Get in touch