The benchmark that disproved its own result

5 fixtures, 30 runsThe setup

Two models already pulled, no new downloads needed.

The general side was chosen after checking its chat template directly through the runtime's own API: a 1,723-character real ChatML template. That check mattered because another model on this box ships a template that's just a passthrough, which silently degrades everything sent through it, and picking that one by accident would have produced a large, fake, extremely publishable result.

Five fixtures, one per task type: JSON repair, YAML frontmatter, CSV cleanup, Docker log extraction, markdown tables.

These models are stochastic, and one run is an anecdote.

So every fixture ran three times, thirty runs in total.

Every fixture gets an objective gate that is full-equality or parse-and-compare, never substring matching. That distinction isn't pedantry. A substring check asking "does the output contain the right value" passes an output that contains the right value and three wrong ones too. This project has had weak tests pass broken local-model work before, twice, so the checkers are written to be hostile.

Two deliberate choices worth naming. Thinking mode off, because on this box's qwen models it either hides the answer inside a thinking block or gets rejected outright by the server. Default temperature rather than the low temperature used for code generation, because the claim is about how these models behave in ordinary use, and tuning the sampling toward precision would have quietly answered a different question.

datetime, not stringThe grader had a bug, and it found it before any model ran

The checkers are the ground truth in an evaluation like this. If the grader's wrong, every number downstream is decoration. So structured-text-eval.py has a --selftest mode: known-correct answers fed to each checker, with no model involved, before a single request goes out.

It failed on the YAML fixture, for a reason that would never have been noticed later.

The fixture asks a model to add one key to a YAML frontmatter block without disturbing anything else, and the checker verified the untouched keys still matched, including a date field, 2026-07-01. YAML's parser doesn't return that as a string. It returns a datetime.date object, because unquoted ISO dates are a date type in YAML. Comparing that object to the string "2026-07-01" is always false.

The checker would have failed a perfectly correct answer, on every trial, for both models. The fix is the str() in the line that guards those untouched keys, still sitting there today:

if fm.get("title") != "Weekly Report" or str(fm.get("date")) != "2026-07-01" \
        or fm.get("tags") != ["ops", "weekly"]:

What matters is when it was caught: before the run, by a check written specifically to distrust the grading logic, rather than after, when a mysterious YAML failure would have needed explaining and might have been written up as a finding instead.

The grading logic needs the same adversarial discipline as the thing it grades.

It's code, and nobody reviews it, because it lives in the part of the experiment everyone treats as infrastructure.

We have written about checks like that before. That check could only ever come back green, whatever it was pointed at. This one had the same defect running the other way, and would only ever have come back red.

15/15 vs 12/15The first result confirmed the article

Thirty runs, and the coding model came out ahead.

That fixture was extracting every ERROR line from a Docker log into structured JSON. The coding model scored 3 out of 3 there, the general model 0 out of 3.

Look at the shape of that. It isn't noisy. It isn't one unlucky trial. One model succeeded every time and the other failed every time, on the fixture most about following a structural rule, which is exactly the mechanism the article named. A clean, consistent, mechanism-matching split, in the predicted direction.

That's what a real finding looks like. It's also what this one looked like, and it was a bug in the fixture itself.

15/15 vs 15/15What the raw output actually said

The harness records every output verbatim rather than just pass or fail, so the failures were readable, not just countable.

The general model hadn't mangled the JSON. It hadn't missed lines, invented fields, or drifted from the schema. Across all three trials it had produced valid, well-formed, correctly structured output, and put the log-level word inside the message field:

"message": "ERROR Failed to connect to cache: timeout after 5s"

The coding model had stripped it. The expected answer stripped it. So the checker failed the general model, correctly, against a specification the prompt never actually stated.

The prompt said to extract each ERROR line into {timestamp, service, message}. It never said whether the word "ERROR" belongs in the message. One model read it one way, the other read it the other way, and both readings are defensible: the line begins with the level, and whether that level is part of the message or metadata about it is a genuine ambiguity, not a comprehension failure.

Reword the prompt, adding one clause: "the message field must contain ONLY the text after the ERROR level word, not the word ERROR itself." Rerun all thirty.

The gap didn't shrink. It vanished.

the confirming runThe uncomfortable part

The result that ended up under scrutiny was the one that confirmed the hypothesis.

That's backwards from how it usually goes. A disconfirming result gets picked apart, because it's annoying and someone wants it to be a bug. A confirming result gets written up, because it agrees with what you already believed and there's nothing left to explain.

Here, the confirming result was the buggy one. The rigour that caught it wasn't scepticism about the article. Its claim was taken seriously enough to test properly. It was a checklist habit: read the raw output behind a pass rate before reporting it, regardless of which way the result points. Without that habit, the write-up would have been "tested it, and the article is right," with a clean table underneath and no reason for anyone to look again.

The null result is also weaker than it sounds, worth stating plainly. It doesn't show the article is wrong. The article's actual comparison was a 3B coding model against a much larger general one, a full generation apart and roughly three times the size difference. Both models tested here are modern and well above the size where basic structure-following stops being in question. The claim may be entirely real at the article's scale and simply invisible at this one. That wasn't tested, because the relevant general model doesn't fit this box's memory budget. So the finding is narrow: on these two models, on these five task types, with unambiguous instructions, there's no difference. Nothing about standing practice changed.

overwritten in placeOne more thing that did not survive

Preparing this write-up meant going back to the raw output file. The committed file holds thirty results, all passing. The run that produced the twelve-out-of-fifteen split, the one this entire piece is about, no longer exists: the corrected fixtures were rerun into the same output path and overwrote it.

The specific output quoted above survives because it was copied into the results log the same day, by the session that had it on screen. That's a good contemporaneous record. It isn't the artefact.

Worth noticing which run it was. What didn't survive was the disconfirming evidence: the thing that was wrong, the run that failed, the state before the correction. Nobody deleted anything on purpose. The rerun simply wrote to the same filename, because the interesting output was assumed to be the final one.

The final output is the one you'll never need. It's the one that agrees with the conclusion you already wrote down. The run worth keeping is the one that made you look twice.