Benchmarks

Where we evaluate, and why

What we are testing

Inference runs on hardware a lab must trust. That trust rests on an attestation chain, the proof a machine gives of what it is running. A vulnerability in that chain is a vulnerability in the trust anchor.

Discovery and remediation

XOR evaluates coding agents on vulnerability discovery and remediation end to end.

Why we can be precise

XOR built the private confidential inference stack these runs use. Each run stays isolated on it, which is why we can say exactly what a patch had to hold against.

Unpatched vulnerabilities

Exploit and repair

With limited vulnerability information

5 October 2026

Average task score (%). Exploit and repair use different checks.

68 matched comparisons · Published order.
System	Exploit score	Repair score
Claude Code / Opus 5	60.0%	21.7%
Codex / GPT-5.6 Sol	13.3%	23.3%
Codex / Daybreak	13.3%	20.0%
OpenCode / Gemini 3.8 Flash	4.4%	21.7%
OpenCode / Kimi K3	51.9%	13.0%
OpenCode / Qwen3.8 Max 0902	53.3%	0.0%
OpenCode / GLM-5.3	59.3%	31.5%

Method and source

Expertise-selected software environments include investigated and constructed cases. The collection is not a representative population.

Repeated scores are averaged within each role and cell; complete environment–information cells have equal weight.

Attack and repair are matched within environment, information condition and system. The role comparison excludes combined attack and repair.

Claude Code / Opus 5: 10 matched comparisons; Codex / GPT-5.6 Sol: 10 matched comparisons; Codex / Daybreak: 10 matched comparisons; OpenCode / Gemini 3.8 Flash: 10 matched comparisons; OpenCode / Kimi K3: 9 matched comparisons; OpenCode / Qwen3.8 Max 0902: 10 matched comparisons; OpenCode / GLM-5.3: 9 matched comparisons.

Percentages are mean partial-credit task scores. Attack scores reflect trigger checks.

Does AI Help Cyber Attackers or Defenders? — Tables II and III

View Unpatched

Unpatched vulnerabilities

Exploit

Shallow exploit: the test input triggered the flaw, full included. Full exploit: the exploit passed the task’s acceptance check.

Ordered by Full exploit, highest first. Ties sort by Shallow exploit, then by name. Unranked.

Claude Code Opus 5 max	95.0%	0.0%
OpenCode Kimi K3 xhigh	85.0%	0.0%
OpenCode Qwen 3.8 Max xhigh	84.2%	0.0%
OpenCode GLM-5.3 xhigh	78.9%	0.0%
Codex GPT-5.6 Sol xhigh	40.0%	0.0%

In order of pass rate; too few runs to put one above another.

Unpatched leaderboardShowing 5 of 6 systems

Unpatched vulnerabilities

Repair

Shallow patch: a nonzero score on the task’s checks, full included. Full patch: 100% on those checks.

Ordered by Full patch, highest first. Ties sort by Shallow patch, then by name. Unranked. Two readings of every graded patch attempt: whether the patch earned full credit, and whether it earned any credit at all.

Codex GPT-6 Astra xhigh	41.2%	35.3%
Claude Code Fable 5.1 high	42.9%	28.6%
Codex GPT-5.6 Sol xhigh	29.2%	16.7%
Claude Code Opus 5 max	19.4%	12.9%
OpenCode GLM-5.3 xhigh	20.0%	8.0%

In order of pass rate; too few runs to put one above another.

Unpatched leaderboardShowing 5 of 8 systems

Memory safety

Exploit versus repair

Exploit and repair rates on the same memory-safety vulnerabilities with public fixes.

← Higher exploit rateHigher repair rate →

OpenCode GPT-5.3 Codex: +6.7 points, -0.3 to +13.8, not separated
OpenCode GPT-5.5: -11.0 points, -16.5 to -6.2, higher exploit rate
Claude Code Opus 4.7: -22.5 points, -30.8 to -13.9, higher exploit rate

Repair rate minus exploit rate, in percentage points, on the vulnerabilities every system saw. Thin line: 95% CI. † The 95% CI includes zero.

Exploit versus repair results

AUTOMATED VULNERABILITY REPAIR

Vulnerability repair

Memory-safety vulnerabilities in low-level software with public fixes: systems patch them and exploit them.

On these leaderboards, exploit measures attacker performance while patch measures defender performance.

RankSystemPass rate

Codex GPT-5.5

86.3%80.5–90.6

OpenCode GPT-5.5

85.3%78.3–90.8

OpenCode GPT-5.3 Codex

84.0%78.0–88.6

Ranks use confidence intervals; a shared rank does not establish equal performance.

Vulnerability repair leaderboardTop 3 of 18 systems measured

Compare systems

Choose any measured leaderboard to compare up to five systems side by side.

Compare systems