Cyber AI Evals
XOR is a security lab that evaluates and improves the cyber capabilities of coding systems.
See the resultsWhat we measure
Code grades exploit and repair outcomes; benchmark pages publish aggregate results. Unpatched uses vulnerabilities without public fixes at evaluation time; the memory safety comparison uses vulnerabilities with public fixes.
Benchmarks
Unpatched vulnerabilities
Exploit and repair evaluations on vulnerabilities without public fixes at evaluation time.
Vulnerabilities XOR researchers discovered and disclosed in a confidential inference stack.
Explore exploit and repair results →Exploit versus repair
Exploit and repair rates on the same memory-safety vulnerabilities with public fixes.
- OpenCode GPT-5.3 Codex: +6.7 points, -0.3 to +13.8, not separated
- OpenCode GPT-5.5: -11.0 points, -16.5 to -6.2, higher exploit rate
- Claude Code Opus 4.7: -22.5 points, -30.8 to -13.9, higher exploit rate
Research
Two attestation challenges for confidential AI inference
How a CPU-side TEE and AI accelerators compose Evidence, and how attestation can avoid disclosing confidential workload contents.
Cyber frontier eval: GPT-6 Astra and Claude Fable 5.1 repair only a third of the flaws in real-world conditions
Coding systems worked on exploits and repairs on flaws without public fixes. The tables report trigger checks, accepted submissions and repair scores.
GPT-6 Astra blocked tested attacks by taking the service offline
GPT-6 Astra produced repairs that blocked the tested attacks and legitimate users. Those repairs received zero credit.
Method
Systems exploit or repair vulnerabilities, and code decides the outcome. Get in touch.