gemini-3-pro-preview vs gemini-3.1-pro-preview
Both models ran the same corpus under the same verifier — gemini and gemini31 respectively. The harness differs where the names differ, and that is part of what separates them.
Their confidence intervals overlap. On this corpus the difference of 15.7 points is not resolved — treat these two as tied.
Where they differ
| Weakness | gemini-3-pro-preview | gemini-3.1-pro-preview | Difference |
|---|---|---|---|
| CWE-125Out-of-bounds Read | 53.3% | 66.2% | -12.9 |
| CWE-787Out-of-bounds Write | 47.1% | 62.5% | -15.4 |
| CWE-416Use After Free | 23.1% | 33.3% | -10.2 |
| CWE-908Use of Uninitialized Resource | 11.1% | 0.0% | +11.1 |
| CWE-415Double Free | 33.3% | 100.0% | -66.7 |
| CWE-843Access of Resource Using Incompatible Type ('Type Confusion') | 0.0% | 50.0% | -50.0 |
| CWE-476NULL Pointer Dereference | 0.0% | 50.0% | -50.0 |
| CWE-1284Improper Validation of Specified Quantity in Input | 50.0% | 100.0% | -50.0 |
| CWE-475 | 0.0% | 100.0% | -100.0 |
| CWE-562 | 0.0% | 0.0% | 0.0 |
Per-class samples are small; a difference here is a hint about where to look, not a result. The corpus-level intervals above are the honest comparison.
Corpus cve-bench-136 · run February 2026 · How we test
See the benchmark →