We graded GLM-5.3-Flash before it had a name
In an earlier evaluation, five of nine systems repaired the vulnerability but then rejected every legitimate message, while every test still passed. The system that kept traffic working was the one we graded before it had a public name.
Most of the patches the systems produced ended up blocking legitimate traffic.
GLM-5.3-Flash was an anonymous listing at the time, and its fix broke none of the traffic.
- 9systems on one briefFig 1
- Five of ninerejected valid trafficFig 1
- 1expression separates the two groupsFig 2
- 0score of the only system that stopped every attackFig 1
On 20 August 2026, GLM-5.3-Flash appeared under the name ox-alpha. OpenRouter said its provider had chosen to remain anonymous during this preview
.
The brief
The brief asked each system to fix the weakness while keeping the build and existing tests passing.
Then we replayed traffic through the patches.
The table scrolls sideways.
| system | valid traffic | attacks stopped |
|---|---|---|
| kimi-k3 | all rejected | 100.0% |
| gpt-5.6-sol · codex | all rejected | 66.7% |
| gpt-5.6-sol · opencode | all rejected | 66.7% |
| daybreak-blue · opencode | all rejected | 66.7% |
| qwen3.8-max | all rejected | 66.7% |
| daybreak-blue · codex | none rejected | 66.7% |
| GLM-5.3-Flash | none rejected | 66.7% |
| grok-4.6 | none rejected | 66.7% |
| Other system | none rejected | 33.3% |
Five of nine closed one gap by opening another that points the other way.
One expression apart
Every patch answered the same question in a single expression.
Five systems wrote the check one way and rejected valid messages. The four that did not wrote it another way and let the same messages through.
Other explanations
Patch size did not separate the groups. One system wrote more than three that broke traffic, yet broke nothing.
Nor did the harness, the software that runs the model. One model failed under both harnesses it used.
The scope of the check also matched. A patch from each group examined the whole message.
One model tested only malformed input in both runs. It never checked whether valid input still worked.
What GLM-5.3-Flash did instead
GLM-5.3-Flash applied the same check differently. Its patch kept legitimate traffic working.
qwen3.8-max identified the weakness in its own patch twice, ten steps apart. It shipped anyway, calling strictness the safe direction.
One model wrote that strictness is the safe direction and rejected every legitimate message. The other rejected none of the legitimate messages.
Then it had a name
On 26 August, Z.ai identified ox-alpha as GLM-5.3-Flash.
We also tested GLM-5.3, the larger sibling. Triage asked whether the reported weakness existed.
The table scrolls sideways.
| measure | GLM-5.3 | GLM-5.3-Flash |
|---|---|---|
| triaged correctly | 33.3% | 100.0% |
| produced a building patch | 33.3% | 100.0% |
| defeated the seed exploit | 0.0% | 100.0% |
The larger model usually judged the reported weakness absent and produced no patch. Its patches did not defeat the supplied exploit.
Every run of GLM-5.3-Flash produced a patch that defeated that exploit.
The tests stayed green
The test suite, the implementation's existing tests, passed on every patch. Those tests never exercised the traffic that broke.
The inversion
kimi-k3 was the only model to stop every attack. It scored zero, because its patch broke legitimate traffic.
The grader, the code assessing the patch, credited repairs that kept legitimate traffic working.
What this does not show
One defect family, one point in time. We did not mount a fresh attack against any finished patch, so nothing here says whether the four that preserved traffic close every variant of the vulnerability. How common messages of this shape are in production traffic, we did not measure.
This earlier evaluation appears on no current leaderboard. We read the systems' transcripts and patches before we knew which model ox-alpha was.
September 2026
Cyber frontier eval: GPT-6 Astra and Claude Fable 5.1 repair only a third of the flaws in realworld conditions
Attack and repair success on Unpatched: grading criteria matter.
September 2026
Blocking every attack earned zero when the repair also blocked legitimate users
When one system stopped every planted attack by locking down the service entirely, our legitimate-use check caught the lockout and the grader awarded zero.