Half the steps ran on a different model

An agent labelled with one frontier model spent 59 of its 128 steps on a smaller, cheaper one. The configuration does not say so. The result does not say so. It is visible only step by step, inside the trajectory.

The run is a security repair task on sigstore-java. The harness is Claude Code. The configured model is claude-opus-5, reasoning effort set to max. At step 13 the main chain stops working and dispatches a sub-agent.

STEP 13

“Report: a ranked list of concrete candidate flaws, each with file:line, the exact code, why it is wrong, and how an attacker would exploit it. Be concrete and quote code. Do NOT write any files.”

Steps 15 through 73 answer that. Every one of them carries claude-haiku-4-5 on the per-step model field, flagged as a sidechain. At step 74 the configured model resumes and builds the trigger and the fix.

configuredclaude-opus-5 · effort max(unchanged for every row below)

step	model on the step	what happens
04	claude-opus-5	reads RekorVerifier.java
13	claude-opus-5	dispatches a sub-agent
15	claude-haiku-4-5	delegate begins
16–42	claude-haiku-4-5	delegate searches
43	claude-haiku-4-5	delegate names lines 167–168
73	claude-haiku-4-5	delegate returns
74	claude-opus-5	main chain resumes, writes the fix

FIG 1The run label and the model setting are constant down the whole table. The column that moves is the one nothing in the configuration mentions.

The delegate's report is emphatic about what it found.

STEP 43

“Now I see the main vulnerability clearly! At lines 167-168, the verifyDigest() call returns a boolean but the return value is not checked.”

The code is the kind of defect that survives review because it looks like handling. A signature check is wrapped in a try block that catches four exception types. The boolean the verifier hands back is discarded. An authentic checkpoint and a forged one leave the method by the same door.

the shape of the defect

try {    // returns a boolean the caller never reads    verifiers.verifyDigest(digest, signature);} catch (NoSuchAlgorithmException       | InvalidKeyException       | SignatureException       | IOException e) {    throw new RekorVerificationException("...", e);}// falls through identically for a valid and an invalid signature

FIG 2The failure is silent. Nothing is thrown, nothing is logged, and the caller cannot tell the two cases apart.

What the delegate actually added

The obvious reading is that the cheap model found the bug and the expensive one took the credit. That reading is wrong, and the trajectory says so.

STEP 04

The configured model issues a Read on RekorVerifier.java as the sixth tool call of the run. The observation returns the file with lines 167 and 168 rendered in it.

STEP 13

Its dispatch prompt scopes the delegate to the transparency-log verification path and leads the list with “missing or ignored return values of signature-verification calls”.

THE MAIN CHAIN PICKED THE FILE AND THE WEAKNESS CLASS BEFORE IT DELEGATED.

FIG 3Nine steps before the dispatch, the configured model had already opened the file and had lines 167 and 168 in its context. What the delegate supplied was the line binding.

The configured model's own later write-up goes past the delegate entirely, naming the checkpoint parser that never retains the bytes the log actually signs. The word Checkpoints appears zero times in the delegate's report.

There is a reason this is easy to get backwards. The configured model emitted zero characters of prose at every step from 3 to 13. Its entire output was tool calls. Read the assistant text alone and you would conclude the expensive model contributed nothing and the cheap one did the work. The transcript produces that error, and the transcript is what most people read.

Which model you get is not something you set

Across the runs of this harness we examined, whether the sub-agent runs on the configured model or on the cheaper one tracks the harness build string recorded in the trajectory. The run label does not change. The model setting does not change.

The delegate model appears in one place: the per-step model field, on steps flagged as a sidechain.

If you benchmark an agent, or buy one, the model on the label may not be the model that did the work. A harness upgrade can move that between two releases without changing anything you configured, and without changing anything you would think to record. Two runs that agree on every field you control can still have spent their compute differently.

The practical form of this is short. When you record a run, record the per-step model, not the configured one. When you compare two runs, compare their harness versions before you attribute a difference to the model. When a vendor quotes you a number, ask which model produced which part of it.

What this does not show

A small number of runs of one harness on one vendor's models, and the build version travels with the batch and the date, so a per-batch setting that moves with the build is not excluded. The correlation is a hypothesis about the harness, not a reading of its release notes. The configured model emitted no prose before it delegated, so what can be shown is that it opened the vulnerable file and named the weakness class, not that it had already settled on the defect.

Read from one agent trajectory, step by step, on the per-step model field. The harness is Claude Code, a public product. The library is sigstore-java. We name no measurement we did not take.

← All research