We graded GLM-5.3-Flash before it had a name
Four of seven systems produced patches that rejected valid messages while passing the existing tests. The patch that stopped every tested attack received zero credit because it also blocked legitimate traffic.
Most of the patches the systems produced ended up blocking legitimate traffic.
GLM-5.3-Flash was an anonymous listing at the time, and its fix broke none of the traffic.
- 7systems on one briefFig 1
- Four of sevenrejected valid trafficFig 1
- 1expression separates the two groupsOne expression apart
- 0score of the patch that stopped every tested attackFig 1
On 20 August 2026, GLM-5.3-Flash appeared under the name ox-alpha. OpenRouter said its provider had chosen to remain anonymous during this preview
.
The brief
The brief asked each system to fix the weakness while keeping the build and existing tests passing.
Then we replayed traffic through the patches.
The table scrolls sideways.
| system | valid traffic | attacks stopped |
|---|---|---|
| kimi-k3 | all rejected | 100.0% |
| gpt-5.6-sol · codex | all rejected | 66.7% |
| gpt-5.6-sol · opencode | all rejected | 66.7% |
| qwen3.8-max | all rejected | 66.7% |
| GLM-5.3-Flash | none rejected | 66.7% |
| grok-4.6 | none rejected | 66.7% |
| a system we do not publish | none rejected | 33.3% |
Four of seven patches rejected valid messages.
One expression apart
Every patch answered the same question in a single expression.
Four systems wrote the check one way and rejected valid messages. The three that did not wrote it another way and let the same messages through. That one expression decided every outcome in the table above.
Other explanations
Patch size did not separate the groups. One system wrote a larger patch than three of the systems that broke traffic, and broke nothing.
Nor did the harness, the software that runs the model. One model failed under both harnesses it used.
The scope of the check also matched. A patch from each group examined the whole message.
One model tested only malformed input in both runs. It never checked whether valid input still worked.
How GLM-5.3-Flash repaired
GLM-5.3-Flash applied the same check differently. Its patch kept legitimate traffic working.
qwen3.8-max identified the weakness in its own patch twice, ten steps apart. It shipped anyway, calling strictness the safe direction.
One model wrote that strictness is the safe direction and rejected every legitimate message. The other rejected none of the legitimate messages.
Then it had a name
On 26 August, Z.ai identified ox-alpha as GLM-5.3-Flash.
We also tested GLM-5.3, the larger sibling. Triage asked whether the reported weakness existed.
The table scrolls sideways.
| measure | GLM-5.3 | GLM-5.3-Flash |
|---|---|---|
| triaged correctly | 33.3% | 100.0% |
| produced a building patch | 33.3% | 100.0% |
| defeated the seed exploit | 0.0% | 100.0% |
The larger model usually judged the reported weakness absent and produced no patch. Its patches did not defeat the supplied exploit.
Every run of GLM-5.3-Flash produced a patch that defeated that exploit.
The tests stayed green
The test suite, the implementation's existing tests, passed on every patch. Those tests never exercised the traffic that broke.
The inversion
kimi-k3 was the only model to stop every tested attack. It scored zero, because its patch broke legitimate traffic.
The grader, the code assessing the patch, credited repairs that kept legitimate traffic working.
What this does not show
One defect family, one point in time. We did not mount a fresh attack against any finished patch, so nothing here says whether the three that preserved traffic close every variant of the vulnerability. How common messages of this shape are in production traffic, we did not measure.
This earlier evaluation appears on no current leaderboard. We read the systems' transcripts and patches before we knew which model ox-alpha was.
September 2026
Two attestation challenges for confidential AI inference
How a CPU-side TEE and AI accelerators compose Evidence, and how attestation can avoid disclosing confidential workload contents.
September 2026
Cyber frontier eval: GPT-6 Astra and Claude Fable 5.1 repair only a third of the flaws in real-world conditions
Coding systems worked on exploits and repairs on flaws without public fixes. The tables report trigger checks, accepted submissions and repair scores.