DPO Training Pairs

Direct Preference Optimization pairs for security patching

Each CVE-Agent-Bench evaluation produces a preference signal: pass fixes the bug, test-fail compiles but does not fix, build-fail breaks the build. This is Direct Preference Optimization (DPO) training data. 1,920 labeled examples across 15 agent configurations provide ground truth for post-training security patching models.

Preference pairs tell a model which outcomes are desirable. A pass outcome paired with a build-fail outcome teaches the model that generating compilable code is better than generating broken code. A pass paired with a test-fail teaches that fixing the bug is better than compiling. These distinctions matter because the strongest training signals come from outcomes that differ only in one dimension.

Three tiers of preference signals

Outcome Distribution

Agent-level breakdown of pass, fail, and build outcomes across all evaluations. The distribution shows how frequently each outcome type appears in the dataset. Pass outcomes (fixes that work) are the foundation for preference signals. Build-fail outcomes provide negative examples to teach the model what not to generate. Test-fail outcomes sit in the middle, representing patches that compile but miss the bug entirely.

Understanding outcome distribution helps you assess data quality for training. If most evaluations land on one outcome type, you have unbalanced training data. A healthy distribution spreads across all three outcome types, giving the model diverse learning signals.

Preference Pair Distribution

Gold, Silver, and Bronze DPO pair counts from the evaluation dataset. Each pair type has different signal strength for training. Gold pairs (pass vs build-fail) provide the clearest learning signal because they differ in the most impactful dimension: does the patch work at all? Silver pairs (pass vs test-fail) teach the model to distinguish between compilable code and correct code.

The ratio of Gold to Silver to Bronze pairs reflects the underlying data. More Gold pairs mean stronger overall training signal. Bronze pairs are rare because they only occur when you have multiple agents all failing on the same sample in different ways.

Reward Signal Comparison

Ternary (+1/0/-1) vs five-level reward shaping distributions. The ternary signal is simple: pass is good, anything else is not. The five-level signal adds nuance by splitting pass into "surgical fix" and "complex fix", and by distinguishing build-fail from infrastructure errors.

More reward levels let you train on finer-grained distinctions, but they also require more data to distinguish between reward buckets. The five-level approach is strictly more informative if you have enough evaluations per reward level. For datasets with 1,920 total evaluations spread across 15 agents, the ternary signal may be more stable.

Reward scoring for RLHF

Use a ternary or five-level reward function to train your model. The ternary signal applies to all 1,920 evaluations across all agents and CVE samples.

Curriculum ordering by difficulty

Start training on easy samples and progress to hard ones. CVE-Agent-Bench assigns empirical difficulty scores (0.0 to 1.0) to each sample based on pass rate across all 15 agents. Use these to order your training curriculum.

Pair generation formula

To generate preference pairs from raw evaluations, cross all outcomes for each sample:

for each CVE sample:
passing_agents = agents with outcome=="pass"
failing_agents = agents with outcome=="test-fail" or "build-fail"

# Gold pairs
for pass_eval in passing_agents:
  for build_eval in build_failing_agents:
    pair = (pass_eval, build_eval, reward=pass_reward - build_reward)

# Silver pairs
for pass_eval in passing_agents:
  for test_eval in test_failing_agents:
    pair = (pass_eval, test_eval, reward=pass_reward - test_reward)

# Bronze pairs
for test_eval in test_failing_agents:
  for build_eval in build_failing_agents:
    pair = (test_eval, build_eval, reward=test_reward - build_reward)

The formula generates |passing_agents| × |build_failing_agents| gold pairs per sample. For samples where all agents fail, only bronze pairs exist. For samples where all agents pass, no pairs exist (no learning signal).

Patch composition and semantic categories

Passing patches in CVE-Agent-Bench fall into semantic categories. Use this to understand what your model is learning:

• Logic fix. Change an operator, swap variables, or restructure control flow. 71 evaluations.
• Guard check. Add a conditional to check preconditions before an unsafe operation. 155 evaluations.
• Bounds check. Add size validation before buffer access. 164 evaluations.
• Allocation fix. Fix memory allocation (size calculation, leak, use-after-free). 89 evaluations.
• Null check. Validate pointer before dereference. 92 evaluations.

Training on semantically diverse examples teaches your model to recognize different fix patterns, not memorize specific code changes.

Data access and format

Request access to the full dataset at /contact. You will receive a JSON file with all 1,920 evaluations. Each record includes:

• sample_id, agent_model, outcome (pass | fail | build | infra)
• patch (full text), patch_bytes (size in characters)
• difficulty (0.0-1.0), semantic_category (logic_fix, guard_check, etc.)
• cost_usd (API cost for this evaluation)

Pair generation is deterministic. The same input data produces the same preference signal across all implementations.

See also