A proof that proves nothing

One Terminal-Bench 2.0 task asks the agent to finish a Coq proof that addition is commutative. I handed its grader a file that declares n + m = m + n as an axiom and then "proves" it by citing that axiom. The grader gave it full marks.

That is one of 16 wrong solutions I found accepted across 8 of the 26 Terminal-Bench 2.0 tasks I audited. All 16 pass under the benchmark's own harness, and all 8 tasks are now reported upstream with test fixes (#92–#99).

Why this matters: a benchmark score is only as good as its verifier. A verifier that accepts wrong work inflates scores, and if the same tasks are used as RL environments, it rewards the wrong behavior. You can't see these gaps by reading the tests or by looking at the reference solution, because both are correct. You see them when you grade solutions that are almost right.

How I got here: a benchmark the cheapest models solved

I started out building KernelMind, a benchmark for debugging networked systems, the kind of work I did on network operating systems. It has 20 tasks in a small Wi-Fi control plane and in GoBGP (95k lines of Go). Many transplant the causal chain of a real production incident into new code, so the buggy code was never public.

I spent most of my effort on the graders. Each one is validated three ways: the untouched starter fails, alternative correct fixes pass, and at least three plausible wrong fixes each fail the specific check aimed at them. If a wrong fix passed, the grader was fixed before the task shipped.

Then I tried to make the tasks hard, and couldn't. Over 8 rounds and 122 attempts (about $1.10), the cheapest current models, Claude Haiku 5.5 and GPT-6 Luna, solved every design I tried:

• bigger codebases and code no model has seen
• evidence with no textual clues, and decoys
• two interacting faults behind a live reproduction
• a regression introduced by an earlier fix, with git history to dig through

The trajectories showed why. These models search for every term in the evidence at once, re-run the reproduction after every change, and read the commit message before reverting anything. My working hypothesis is that they fail when information is missing, not when it is spread out or disguised. I haven't tested that yet.

So the difficulty study was a negative result. But the part I trusted most, grading against wrong fixes, looked useful on its own. Did other benchmarks do it?

The idea: grade solutions that are almost right

A verifier is a classifier with two kinds of error: it can accept wrong work (lenient) or reject correct work (strict). To measure either, you need solutions whose correct verdict you already know. The reference solution only tells you the grader says yes to one right answer.

So KernelMind audit writes three kinds of candidate for each task and runs the task's own tests on each, in a fresh container:

Near misses come in six categories taken from KernelMind's wrong-fix controls: partial fix, missed edge case, symptom-only fix, over-broad change, hardcoded output, wrong format. The model sees the instruction and the reference solution but never the tests, so near misses are plausible mistakes, not test-gaming.

A passing near miss is only a suspect. Models often claim a flaw their code doesn't have. Every suspect was checked by hand against the instruction's exact words, and where possible with a differential probe: an input the instruction decides, on which the reference and the candidate disagree. Confirmed findings were then rerun under Harbor, Terminal-Bench's own harness, with the wrong solution swapped in for the reference.

What it found: 8 of 26 Terminal-Bench 2.0 tasks accept wrong work

I audited 30 tasks (a 5-task pilot plus 25 sampled with a fixed seed under rules written before any results) and graded 26; four needed hardware or parts of Harbor's runtime I didn't reproduce, or had a broken reference. Eight tasks, 31%, accepted at least one solution that is wrong by the instruction's own words. Harbor gave all 21 reproduction trials (8 references, 13 wrong solutions) reward 1.0.

Severity varies. The axiom proof, the recoverable secret and the hardcoded deploy are serious: an agent that does them has not done the task. The SPARQL boundary cases are minor. Fifteen of the 16 wrong solutions came from the auditor (12 near misses, 3 mutants); I found the Admitted. variant by hand while testing the fix for the axiom case.

Each issue includes a test change. I validated 7 of the 8 locally: the reference still passes and every wrong solution now fails. Before filing, I searched the tracker; none of the eight had been reported.

Five ways verifiers go wrong

The eight findings share five causes. None is exotic, and each has a cheap fix.

1. A stated requirement is never tested. "DO NOT MODIFY", the version pins, "fix the WAL file", "commit messages remain untouched". If the instruction says it and no test checks it, it's optional.
2. The one fixed input lacks the deciding case. The COBOL test data has no invalid sale, the SPARQL graph has no non-EU country and no department with exactly 10 students, and no log line is dated Feb 29.
3. The test input is the instruction's own example. The webserver test pushes exactly the hello world the instruction shows, so writing that string passes.
4. Negative samples are confounded. In regex-log, every line with an invalid date also has an invalid IP. Those samples test the IP rule, so a broken date rule never gets caught.
5. The test checks the wrong object, or text instead of meaning. The leak test lists dangling commit headers, not the blob that holds the secret. The proof test looks for the string Qed. instead of asking Coq which assumptions the proof uses (Print Assumptions does that).

All five are invisible if the only solutions ever graded are the reference and a few agent runs. They show up as soon as you grade solutions that are wrong in one specific way.

How good is the auditor?

It catches gross verifier flaws every time and misses most subtle ones, so 31% is a floor, not an estimate.

To measure recall, I planted flaws in copies of 11 KernelMind tasks, whose graders I know are sound: I disabled the tests that catch one known wrong fix, audited the weakened task, and regraded the same candidates against the original tests.

With four near misses per task, the model rarely makes the one mistake a subtle gap lets through. More near misses, or one per instruction sentence, is the obvious next experiment.

Precision depends on the target. On Terminal-Bench, 12 of 18 model-written lenient suspects were real. On KernelMind's validated graders, only 3 of 18 deserved attention (two small real gaps in my own tests and one open question); the rest were equivalent mutants and near misses whose claimed flaw wasn't in their code. The auditor ranks where to look; a person still decides.

Strictness is a blind spot. None of the 9 failing alternatives turned out to be a correct solution the verifier wrongly rejects, while other people reported strictness and crash issues on two tasks I audited (#84, #85). Writing a correct alternative is much harder for a model than writing a near miss.

Cost: under $0.25 of model calls for everything ($0.16 for the Terminal-Bench candidates, $0.05 for the planted-flaw study). The real cost is container time, roughly 10–15 minutes per task on a laptop.

Takeaways for benchmark builders

Grade wrong answers before you trust a grader. Concretely:

• For every sentence in the instruction that states a requirement, point to the test that fails without it. If there isn't one, write it or cut the sentence.
• Never test with the instruction's own example. Use a second input the agent hasn't seen.
• Give each negative sample exactly one defect, so it tests the rule you meant.
• Check meaning, not text: ask the tool (Coq, git, the database) rather than grepping its output.
• Before shipping a task, run a handful of near misses through its tests. It costs cents.

The same lesson held for my own benchmark. KernelMind's graders survived the audit because each had been checked against wrong fixes from the start, and the audit still found two small gaps in them.

Links

• Issues filed on Terminal-Bench 2.0: #92, #93, #94, #95, #96, #97, #98, #99
• Terminal-Bench 2.0, audited at commit 2fd12b8
• Code, full technical report and audit log: the KernelMind repository (link to add when it goes public)

Thanks to the Terminal-Bench maintainers for an open, well-documented benchmark; making it easy to audit is what made this possible.

Update (Nov 7): A strong model read each task’s tests and wrote solutions designed to pass while violating the instruction, and every one was run against the real tests. Reading alone, it found 12 of the 14 gaps above, for about $2 across 26 tasks. It also found four more tasks, including a pipeline-parallel training step that never runs the backward pass and a terminal that ignores Ctrl-C. All four are confirmed under Harbor and filed with fixes (#100–#103), bringing the total to 12 of 26 tasks. One task’s own reference solution turned out to be wrong too. These gaps aren’t hard to see if you ask a model to construct a wrong solution that passes, and then run it. Related work, Terminal Wrench (Zhong et al., 2026), found 13 of 89 Terminal-Bench 2.0 tasks hackable by prompting models to hack; only 2 overlap with mine.