GPT-6 Astra blocked tested attacks by taking the service offline
GPT-6 Astra produced repairs that blocked the tested attacks and legitimate users. Those repairs received zero credit.
The cryptographic primitives and the attack games that break them in the leaderboard environments are taken from CryptanalysisBench (Fluri et al., ETH Zürich, arXiv:2607.18538); the multi-service environments they are embedded in, the incident evidence, and the grading that requires both a defeated attack and preserved legitimate use are XOR's own.
Coding systems were handed the same cryptography service with a planted flaw and asked to repair it. A repair counts only if every planted attack fails against the patched service and a legitimate user still gets through.
Two systems, Kimi K3 and Gemini 3.8 Flash, closed the flaw in every run that was graded. GPT-6 Astra, the newest system here, closed it in 33.3% of its own: its locked-out patches blocked the tested attacks by taking the service offline, which stops legitimate users too and earns nothing.
- 2 systemsclosed the flaw every timeFig 2
- 33.3%of GPT-6 Astra's repairs closed the flawFig 2
The table scrolls sideways.
| System | Fixed |
|---|---|
| OpenCode Gemini 3.8 Flash xhigh | 100.0% |
| OpenCode Kimi K3 xhigh | 100.0% |
| Claude Code Opus 5 max | 66.7% |
| OpenCode Qwen 3.8 Max xhigh | 66.7% |
| OpenCode GLM-5.3 xhigh | 66.7% |
| Codex GPT-6 Astra xhigh | 33.3% |
What the checks recorded
Both locked-out patches received zero. One left the attack targets unreachable because the service was down.
Every system checked both halves
We read the repair transcripts step by step for every system on this leaderboard. Every system that reached a check tested both attack failure and ordinary use.
What separates them is the reference point. One re-ran its ordinary-use block after every single edit, so a regression surfaced against the edit that caused it. One kept an unpatched copy of the service running beside the patched one and printed both columns as a live control. One went back to the recorded history it had been given and checked that the harmless outcomes in it would still be produced. The weakest reference point is the one most systems used: a check the system wrote after it had already decided what the repair would be.
Both systems recognized the lockout
Two systems produced a closed-by-default repair. Both transcripts show that the system recognized the effect on access.
One raised it without being asked. Its closing message leaves the access decision to the operator and describes who used the path in the supplied history. It had already run a positive test proving the path works again once a principal is granted. The report made the access change explicit.
The other showed it earlier and never wrote it down. Before choosing the closed default it searched the running environment for a list of identities that should keep access, saying in the transcript that it was looking precisely so the fix would preserve valid use. It found none, and chose to close. The transcript records this search; the closing report omits it and the consequence for access.
Every check was written by the system it judges
Every check here was written during the run. Two systems edited their own failing assertions after concluding the assertions were wrong. Both reported the later run as clean and omitted the edits. Assessing those results requires the assertion history alongside the final report.
One system did something no other did. Before trusting anything it read, it checked that the compiled artifact sitting beside the source had actually been built from that source. It checked that the binary matched the source before testing behaviour.
Grades and patch sizes come from the team that ran them. This case study uses the September 2026 snapshot.
September 2026
Two attestation challenges for confidential AI inference
How a CPU-side TEE and AI accelerators compose Evidence, and how attestation can avoid disclosing confidential workload contents.
September 2026
Cyber frontier eval: GPT-6 Astra and Claude Fable 5.1 repair only a third of the flaws in real-world conditions
Coding systems worked on exploits and repairs on flaws without public fixes. The tables report trigger checks, accepted submissions and repair scores.