Benchmark Results
50.7% pass rate. $4.16 per fix. Real data from 1,224 evaluations.
9 agents benchmarked on 136 real vulnerabilities
OutcomePick the right agent before you deploy. See which ones produce fixes that pass.
Mechanism1,224 evaluations. Pass rates, cost per fix, and difficulty scores for every agent.
ProofBest pass rate: 50.7%. Cheapest verified fix: $4.16.
Agent Rankings
- Pass rate (primary metric)
- Cost per verified fix
- Difficulty-weighted performance
- Trend over time
Cost Economics
Fix a CVE with an agent for $4.16–$87. Incident response costs thousands. Scale pre-production agent fixes instead.
Difficulty Scoring
Vulnerabilities range from trivial (syntax errors) to hard (architectural refactors). Difficulty score helps you understand agent capability on different threat classes.
Current test dataset
136 real bugs tested, 1,224 test runs across 9 agent configurations. Growing to 6,138+ vulnerabilities across 250+ open source projects.
How it works
- Reproduce each bug with a known way to trigger it and a known-good fix.
- Run each agent in an isolated environment.
- Apply the agent's fix and check if the bug is gone.
- Record pass/fail results and categorize failures.
- Adjust scores for bug difficulty so results are fair.
What's in the report
- Agent leaderboard by pass rate and cost
- Failure categories and why fixes fail
- Guide for choosing the right agent and model
- Fix examples and test results
Why this matters
Engineering leaders need proof before scaling AI fixes to hundreds of developers. Security leaders need audit-ready evidence. XOR provides independent, tested results that both teams can trust.
Complete Benchmark Results
Enter your email below to access the full benchmark report, including agent configurations, patch examples, failure analysis, and methodology.
Unlock full results
Enter your email to access the full methodology, per-sample analysis, and patch examples.
FAQ
Which agent has the highest pass rate?
Claude Opus 4.6 at 50.7% on the CVE benchmark dataset. See the full rankings with cost breakdowns.
How much does it cost to fix a vulnerability?
Costs range from $4.16 to $87 per verified fix, depending on agent and model. Pre-production fixing via agents is 100x cheaper than incident response.
Are these costs real or estimates?
Real. Calculated from 1,224 verified fixes at actual API costs (no rounding, no statistical assumptions).
Do pass rates change?
Yes. As new models ship, benchmarks update. We re-run tests regularly so rankings stay current. Data updated as of today.
Patch verification
XOR writes a verifier for each vulnerability, then tests agent-generated patches against it. If the fix passes, it ships. If not, the failure feeds back into the agent harness.
Automated vulnerability patching
AI agents generate fixes for known CVEs. XOR verifies each fix and feeds outcomes back into the agent harness so future patches improve.
Benchmark Results
50.7% pass rate. $4.16 per fix. Real data from 1,224 evaluations.
Agent Cost Economics
Fix vulnerabilities for $4.16–$87 with agents. 100x cheaper than incident response. Real cost data.
Agent Configurations
9 agent-model configurations evaluated on real CVEs. Compare Claude Code, Codex, Gemini CLI, Cursor, and OpenCode.
Benchmark Methodology
How CVE-Agent-Bench evaluates 9 coding agents on 136 real vulnerabilities. Deterministic, reproducible, open methodology.
Agent Environment Security
AI agents run with real permissions. XOR verifies tool configurations, sandbox boundaries, and credential exposure.
Security Economics for Agentic Patching
Security economics for agentic patching. ROI models backed by verified pass/fail data and business-impact triage.
Automated Vulnerability Patching and PR Review
Automated code review, fix generation, GitHub Actions hardening, safety checks, and learning feedback. One-click install on any GitHub repository.
Continuous Learning from Verified Agent Runs
A signed record of every agent run. See what the agent did, verify it independently, and feed the data back so agents improve.
Signed Compliance Evidence for AI Agents
A tamper-proof record of every AI agent action. Produces evidence for SOC 2, EU AI Act, PCI DSS, and more. Built on open standards so auditors verify independently.
Compliance Evidence and Standards Alignment
How XOR signed audit trails produce evidence for SOC 2, EU AI Act, PCI DSS, NIST, and other compliance frameworks.
See which agents produce fixes that work
136 CVEs. 9 agents. 1,224 evaluations. Agents learn from every run.