How we measure

A pass rate is only as honest as the grader behind it.

The two roles

Neither role discovers vulnerabilities; the vulnerability is given.

The attacker writes an input, with Any trigger measuring triggered weaknesses and Full exploit measuring credited exploitation. The defender writes a patch. Any credit includes partial fixes; Full fix measures complete fixes.

The cryptography flaw-report leaderboard is the exception: it asks the system to report a planted flaw and stay silent on a copy where none was planted.

What a patch must prove

The table scrolls sideways.

Ways a patch can look right and be wrong
Problem	What it looks like
Lockout	Stops every attack by disabling the service or the operation it protects.
One-trigger fix	Stops the exact input we replay, but not near variants of it.
Regression	Closes the vulnerability and breaks other behaviour at the same time.
Does not build	The patch applies, but the software no longer compiles or starts.
Symptom not cause	Suppresses the failure we observe without removing what produced it.
Stub	Passes by emptying or deleting the capability instead of repairing it.
A fix a reviewer would reject	A human reviewer would reject the change even though it's correct.

What an exploit must prove

The whole test rests on the pair: the exploit must fire against the vulnerable code, and the reference fix must stop it.

The ‘exploited’ column counts only runs the grader credited; the ‘triggered’ column counts runs that reached the vulnerability’s trigger.

The table scrolls sideways.

Ways an exploit can look right and be wrong
Problem	What it looks like
A stub that prints success	The output declares the attack worked without attacking anything.
A guessed or self-chosen answer	The system prints a value it guessed, replayed or chose itself. It doesn't attack the target.
The wrong failure	The input makes the software fail, but not through the vulnerability the question was about.
Harness-dependent	The harness around the system makes the attack work. It doesn't work against the target.
No output	The run ends with nothing to grade.

The two verifier shapes

Protocol and cryptography. In these environments, the grader requires a patch to reject non-conforming input and still accept valid input. A patch that rejects everything fails.

Contamination, stated per benchmark

Unpatched's defects have never been published, so there's no public repair or working attack for them.

We author and plant the cryptography flaws. Some underlying cryptographic primitives and attack games come from a published research benchmark, credited in the note.

No public fix exists for the leaderboard environments. A third cryptography environment built on a public vulnerability was left off every board.

The terms we use

System: A harness, the requested model at its reasoning effort, the CLI version, the tool policy and the stopping behaviour. A change in any of these is a different system.
Run: One system on one vulnerability. Every retained run counts as its own cell; a system re-run on a cell has both runs counted.
Attacker and defender: Each system plays a role, attacker or defender. The attacker produces a triggering input for a known vulnerability; the defender produces a source patch for it. Neither discovers vulnerabilities. The cryptography flaw-report board is the one exception.
Grader: Every published rate is graded by code, not an evaluation model. One grader build per leaderboard version, and the same build scores every system on it.
Leaderboard: A snapshot of vulnerabilities and one role, ordered by its grader-accepted column. Leaderboards are never pooled and never compared with each other; a rate belongs to one leaderboard.
Score: The share of graded runs that passed the named column.
95% CI: The range a score is likely to sit in, drawn as the thin line beside it. A score uses our repository-block bootstrap; a paired difference uses the Tango interval with an exact McNemar test and Holm adjustment across systems.
Graded: A run with a grade from the grader. Every graded run is a cell in its leaderboard. A run that used its whole time budget without a result is a graded failure.
Contamination: Whether a system could have read the repair before the run, answered per benchmark above. A remembered repair earns a pass it did not work for.
Withheld: The prompts and graders behind a published leaderboard stay private. Research notes name public defects and public libraries; the Unpatched benchmark reports its unpublished defects by pass rate only, until fixes ship upstream.

XOR (2026). XOR benchmarks: coding systems on vulnerabilities. Data version 052fee2d42becd04. https://www.xor.tech/benchmarks.

BibTeX

@misc{xor_benchmarks_2026,
  author = {XOR},
  title = {XOR benchmarks: coding systems on vulnerabilities},
  year = {2026},
  note = {data version 052fee2d42becd04},
  howpublished = {\url{https://www.xor.tech/benchmarks}}
}