The verdict that survived its own correction

8/8 in 2.1 minutesThe re-test

The task was a word search game: three files, a 10x10 grid, eight words hidden horizontally, vertically or diagonally, click-and-drag to select, correct words struck through in the list, a New Game button. Eight scored criteria. The eighth is the one that matters: ask for one follow-up fix (make found words green) and check it lands without breaking the drag highlighting that already worked. That regression check is what had killed every previous run.

Run through little-coder, a small harness that gives the model real write and read tools instead of asking it to print code into a chat window, against qwen3-coder-30b served by Ollama, the result was 8 out of 8 in 2.1 minutes. The follow-up fix took another 30 seconds. All of it verified in a real browser with simulated drag events, not taken from the model's own account of what it had done.

The same benchmark against the smaller qwen3.5-9b scored 7.5 out of 8 in 44.1 minutes. It lost half a point on reverse-placed and diagonal words: no reversed-string check, an off-by-one in the highlight loop. It still passed the regression test, the first local setup here to manage that.

Two honest marks sit against the 8 out of 8. One transient failure died with "Stream ended without finish_reason" and an identical retry worked. On criterion 8, one green pixel was never confirmed, because the preview tab had collapsed to a zero-width viewport and inline styles had stopped applying: impossible in a real browser, so it was scored as a harness fault rather than a code fault. That was a judgment call, made and written down as one at the time.

unaudited for two weeksWhat this project wrote down that evening

The June verdict was a harness problem, not a model problem.

It's a clean line. It reads as an insight. It went into the project's working memory as a headline finding, was quoted straight into the active constraints file, and shaped everything scoped afterward: if the harness is the variable that matters, you invest in harnesses and stop shopping for models.

The investment was right. The claim wasn't, and nobody checked it for two weeks, because a conclusion that's useful and turning out well doesn't feel like it needs auditing.

one row of six use casesWhat the original verdict actually said

The June work had been handed over as a document. Reading it properly, two weeks late, the verdict is one row in a table of six use cases:

The scope is in the verdict line. Not "local models cannot do iterative coding." Local models cannot do iterative coding in a chat UI, which is a claim about an interface.

Below it, five structural failure modes are listed. Stateless regression, where a fix in turn N gets silently undone in turn N+1: observed, in that same word search task, as a drag highlighter fixed and then broken. Overconfident reporting. Silent context truncation. Effective context not matching advertised. No cross-session persistence.

And then, closing that section:

Why Claude Code avoids 1–2: tool-mediated verification per step, persistent conversation context — architecture, not raw model capability.

June had already worked out that the constraint was architectural. It's written down, in the document that was handed over, in bold, five weeks before the July re-test that "discovered" it.

July didn't overturn June. July supplied the harness June had described, and measured what happened.

the half that didn't holdWhere June was too optimistic

Of the five failure modes, June expected a harness to handle the first two: stateless regression and overconfident reporting, both covered by tool-mediated verification and persistent context.

The first half held. The second didn't.

Overconfident reporting is a model claiming completion when a feature doesn't actually work, and a harness only catches that if its tests are adversarial. This project has watched a weak test launder a false claim into a green, twice. A redaction check passed with assert "84210.55" not in out while a partially redacted amount, 842[AMOUNT], leaked straight through it. A loader passed nineteen of nineteen with a leaked loop variable, masked by a fixture that only ever exercised one file.

Real file tools, persistent context, tests that ran green: both greens were wrong.

So June wasn't simply more careful than its own summary. On this one point it was optimistic, and the correction only turned up once this project built the harness June had described and ran it long enough to get burned.

four variables, one real failureWhat actually went wrong

Four things actually changed between June and July, worth stating because it means the re-test was never a controlled experiment. The model changed. The harness changed. The runtime flags changed (July ran with flash attention and a quantised key-value cache, June predated both). And the prompt changed, because June's prompt text was never preserved and had to be rebuilt from scratch.

That last one isn't carelessness. The June raw evaluation file survives, and it's 809 bytes: a findings summary whose entire task description is one line. There was no prompt left to preserve by the time anyone went looking. Reconstruction was the only option on the table.

But the four variables are a side issue. The real failure was compression.

What travelled from June into this project was the phrase "iterative coding not viable." The scope, any local chat UI, didn't make the trip. Neither did the mechanism: architecture rather than raw model capability.

Then someone re-tested the compressed form, found it wanting, and corrected it into a claim broader and less accurate than the original. A precise, well-reasoned, correctly scoped verdict went in one end and came out the other as a slogan, and the slogan is what the project ended up arguing with.

The handover document's own header says it plainly:

Nothing here needs re-deriving — it was all tested hands-on.

This project re-derived it anyway, and ended up with a worse version.

one failed lookupThe part that only appeared while writing this

Writing this piece meant going back to the June sources, and the first attempt to find them failed. The active constraints file listed the handover document under a path in the other project, the one that had written it. It wasn't there. Searching that project turned up nothing.

The conclusion drawn from that was that both June primaries had been deleted. That conclusion went into a correction to this project's own files, into an early draft of this piece, and into a request asking another project to recover them from git history.

The document was sitting in this project's own repository root. It had been there since the first commit. It had been written elsewhere and moved across at handover, exactly as its own header says. The raw evaluation file had moved into a shared knowledge vault during a migration weeks earlier, and was just as findable. The other project located both files in minutes, and mentioned, almost as an aside, that it has no git history at all, so the recovery being asked for could never have worked anyway.

One failed lookup at one stale path became "gone from disk," and a fair amount was built on top of that before anyone checked the obvious place.

That's the same mistake this piece is about, made while writing about it.

An absence, asserted from a single negative result. A conclusion, drawn from a summary (the path in the constraints file) rather than from the thing itself.

two rules, both mechanicalWhat changed

The finding was rewritten to what the evidence actually carries. The harness thesis is June's, not this project's, and it was right. What this project added is confirmation and a measurement. The broad version, "harness, not model," is now marked as over-stated, because it took this project's own later evidence, not June's, to find the one place a harness alone wasn't enough.

The practical guidance didn't change at all, which is the uncomfortable part. Invest in the harness. Verify with real tool output. Keep the local tier for bounded work. All still correct, all still what this project does. A claim can be over-stated and still point the right way, and that combination is exactly what stops anyone from checking it again.

This isn't a new failure shape for this operation, either. A decision that outlives the reasons behind it is the same mistake as a routing choice nobody re-checks. A conclusion asserted from one failed lookup is the same mistake as a result scored green once and never looked at twice. Both of those have already been written about here. This is just the first time the mistake showed up in the writing itself, not in the thing being written about.

Two rules came out of this, both narrow and mechanical. Before re-testing an inherited verdict, read the primary document, not your own summary of it: an active-constraints file strips scope and exceptions by design, because stripping detail is what it's for. And an absence needs more than one failed lookup, especially when the path you searched came from the same compressed source you're already re-deriving from.

What travels between sessions, projects and people is the conclusion. The conditions that produced it stay behind. A conclusion without its conditions is a rumour with a number attached.