Cyber AI Evals
XOR is a security lab that evaluates and improves the cyber capabilities of coding systems.
See the resultsWhat we measure
A coding system meets a vulnerability with no public fix, and is set to exploit it, repair it, or both. Code decides the outcome, and every result is published.
Benchmarks
Unpatched vulnerabilities
0.0%Full exploit rateHistorical grades: shallow exploit 65.1%; full exploit 0.0%.
Vulnerabilities XOR researchers discovered and disclosed in a confidential inference stack.
Unpatched vulnerabilities benchmark →Exploit versus repair
Exploit and repair rates on the same memory-safety vulnerabilities with public fixes.
- OpenCode GPT-5.3 Codex: +6.7 points, -0.3 to +13.8, not separated
- OpenCode GPT-5.5: -11.0 points, -16.5 to -6.2, higher exploit rate
- Claude Code Opus 4.7: -22.5 points, -30.8 to -13.9, higher exploit rate
[RESEARCH]
Two attestation challenges for confidential AI inference
How a CPU-side TEE and AI accelerators compose Evidence, and how attestation can avoid disclosing confidential workload contents.
Cyber frontier eval: GPT-6 Astra and Claude Fable 5.1 repair only a third of the flaws in real-world conditions
Historical attack and repair grades on flaws with no public fix. The tables distinguish trigger checks, full historical attack credit and repair outcomes.
GPT-6 Astra stopped every attack by taking the service offline, and earned zero
Security fixes get rolled back for a dull reason more often than a dramatic one: they break the people who were supposed to keep working. The system under repair here is the broker between an engineer and production credentials, attacked in chained stages where each one opens the next. A repair has to stop every stage and still serve a legitimate caller. GPT-6 Astra shut the path the attacks used, shut its own operators out with it, and earned nothing.
Method
Systems exploit or repair vulnerabilities, and code decides the outcome. Get in touch.