Cyber AI Evals
XOR is a security lab that evaluates and improves the cyber capabilities of coding systems.
See the resultsWhat we measure
A coding system meets a vulnerability with no public fix, and is set to attack it, repair it, or both. Code decides the outcome, and every result is published.
Benchmarks
Unpatched
0.0%Full exploit rateThe flaw fired at 63.5%; none cleared the bar for the full result.
Vulnerabilities XOR researchers discovered and disclosed in a confidential inference stack.
Unpatched benchmark →Attack versus defend
Memory-safety vulnerabilities in low-level software with public fixes: defend rate minus attack rate.
- OpenCode GPT-5.3 Codex: +6.7 points, -0.3 to +13.8, not separated
- OpenCode GPT-5.5: -11.0 points, -16.5 to -6.2, better as the attacker
- Claude Code Opus 4.7: -22.5 points, -30.8 to -13.9, better as the attacker
[RESEARCH]
Two attestation challenges for confidential AI inference
How a CPU-side TEE and AI accelerators compose Evidence, and how attestation can avoid disclosing confidential workload contents.
Cyber frontier eval: GPT-6 Astra and Claude Fable 5.1 repair only a third of the flaws in real-world conditions
Nine coding systems met flaws with no public fix. Two of them repaired about a third of what they were given, and no full exploit was accepted from any system on any run.
GPT-6 Astra stopped every attack by taking the service offline, and earned zero
Security fixes get rolled back for a dull reason more often than a dramatic one: they break the people who were supposed to keep working. The system under repair here is the broker between an engineer and production credentials, attacked in chained stages where each one opens the next. A repair has to stop every stage and still serve a legitimate caller. GPT-6 Astra shut the path the attacks used, shut its own operators out with it, and earned nothing.
Method
Systems attack or defend, and code decides the outcome. Get in touch.