stone, 2026-10-11, at the terminal: "I don't think the tests on this glossary are turning up the sort of clarity that I expect. Can you try running some individual definitions by me, 1, 2 or 3 at a time, once you have tested them?"
Each test hands a reader only a few definitions and the questions they must settle, with no code and no tools (build.py, run.sh). The readers are Haiku and Sonnet in the battery's sandbox, Qwen 7B five times, and Gemma 31B twice. A batch goes to stone only once every reader answers right. Each revision answers what the last test's readers missed.
| Test | Definitions | What its readers missed |
|---|---|---|
| 1 | OCX message, run, started and ended | the objective, mentioned only in passing, read as part of every message (Qwen, 4 of 5); "refused" said "would not", not "cannot" (Sonnet) |
| 2 | the same, with a run's fields listed | "a run" and "the run message" read as two things (Sonnet); a returning node starting the run again (Qwen, 3 of 5); one started per post (Qwen) |
| 3 | the same, with a JSON example of each | several ended per session, or a started before refused (Qwen) |
| 4 | the same, with the only orders a run's messages come in | nothing: Qwen, the only reader rerun, answered 3, 4 and 8 right in 5 of 5 |
| 5 | Runner, Subscription, Limits, with batch 1 as context | a board test for every check-in: Haiku looked for a board's own runner; Qwen chose a one-off command in 5 of 5, since command listed "a test on a target board" |
| 6 | the same, with a rule (once: command; every check-in: ci) and a ci example; multi-line posts, refused writes and paths settled |
Qwen named the test in objective, not command, in 5 of 5 |
| 7 | the same, saying command is what runs and objective what passes |
the same, 5 of 5; batch 1's example run, a command run "checking each check-in on dev", was itself wrong |
| 8 | batch 1's example made a one-off build | Qwen still named the test in objective in 4 of 5; every other answer right |
| 9 | Sender, Verdict, Reading verdicts, with batches 1 and 2 as context | a verdict on one's own check-in counted (Qwen, 4 of 5); "checked by the node" read as "sent by the node" (Qwen, 4 of 5) |
| 10 | the same, the sender rules as sentences, not a table | Qwen copied the ci example into a reviewer's verdict (4 of 5) |
| 11 | the same, with a reviewer's and a ci run's example side by side |
"anyone may write" read as "anyone's counts" (Qwen, 5 of 5) |
| 12 | the same, saying a verdict on one's own check-in never counts | nothing: every reader answered every question right, Qwen in 5 of 5 |
| 13 | Node, Tools, Versions, with batches 1 to 3 as context | Qwen: a build host learns of runs from the hook, not by syncing (3 of 5); a verdict written with fossil tag add does not count (4 of 5); an unknown key outside limits refuses the run (2 of 5) |
| 14 | the same, each of those stated outright | Qwen: two repositories share a server-code if they are clones (4 of 5); the wrong reason a node acts at once (3 of 5) |
| 15 | the same, with create (write a run) apart from run (perform one by hand), a run's manual, and run --trial (stone, 2026-10-11) |
Haiku, Sonnet and Gemma answered every question right; Qwen got the manual and trial questions right in 5 of 5, and still missed questions 3, 4 or 6 in 4 of 5 |
Batch 3 is defs-12.md. Batch 4 is defs-15.md, with
context-15.md adding manual to the run.
Batch 1 is defs-4.md. Batch 2 is defs-8.md, with context-8.md, batch 1 as amended:
- its parser takes a bare fence and ignores other text (stone, 2026-10-11);
- its example run is a one-off build.
Batch 5: codex's review
stone, 2026-10-12, at the terminal, passing on codex's review of c28028fbd556: "settle attribution and crash recovery first. Then add a compact set of worked failure scenarios … The reader tests demonstrate comprehension of the stated rules; they do not yet demonstrate that those rules form a consistent, implementable protocol."
So batch 5 adds a second kind of test. A fresh Opus, in the same sandbox and handed only the protocol, is asked for every contradiction, every behavior two careful implementers could build differently, and every rule that cannot be built as written (prompt-16r.txt, prompt-17r.txt). The comprehension tests now hand readers the whole protocol, as it stood, which each test's prompt file holds word for word, and ask the failure cases as questions.
| Test | What it held | What its readers found or missed |
|---|---|---|
| 16 | the protocol and its tools, revised for codex's findings 2 to 6 | Haiku, Sonnet and Gemma answered every case right (Gemma's first run stopped at question 7, out of tokens, so run.sh now allows 8000). Qwen, in 4 or 5 of 5: a branch read at its newest, every manual start refused, a non-agent session reading its thread, contradictory limits enforced |
| 16r | the same, with the failure cases | Opus: 32 findings. The worst: run --trial open to any caller, reports anyone could forge, nodes on clones contradicting the build-host example |
| 17 | the protocol, its tools and its verdicts, revised for both | Sonnet and Gemma (twice) answered all 16 right. Haiku gave three posts as one turn, and let an agent's trial run. Qwen, at 7,500 tokens of protocol, missed most of what it had answered in test 16 |
| 17r | the same, with the failure cases | Opus: 26 findings. The worst: a build host's clone passes everything it pushes as its node user, a forged ended could block a run where the text said "no ended", and a stop arriving while a run was prepared could be overtaken |
| 18 | the protocol, its tools and its verdicts, with stone's option (a): a node on a clone asks the server who sent what | Haiku, Sonnet and Gemma answered every question but one right; all three, and Qwen, had a run written straight into a build host's repository acted on, which the text never ruled out |
| 18r | the same, with its limits and the failure cases | Opus: 27 findings. The worst: ci verdicts could never count, stops written with OCX's stop went as the run's user, and a build host could not count its own unsynced started |
| 19r | all six pages, as committed in e7b7d62cae | Opus: 25 findings. The worst: nothing named node users, so a run could take one over; a build host took a pulled forgery for its own report; branch could not hold while the program wrote its own clone; tools on a build host sent a person's writes as the node user |
Qwen 7B follows the protocol a few definitions at a time (batches 1 to 4), but not whole: what a smaller model is handed should be the sections its task needs. Test 17's two Haiku misses and the second review's findings are answered in the check-in after them. stone chose how a clone learns who wrote what it pulls: it asks the server (option (a), HISTORY.md decision 37). Test 18's shared miss, and the third review's findings, are answered in e7b7d62cae; the fourth review's, in the check-in after it. The fourth was the one more review stone asked for before building v1 (the maintainer at the terminal, 2026-10-12, asked whether to review once more first: "yes, please").
Batch 6: the protocol as tables
stone, 2026-10-13, at the terminal, of the draft docs/protocol-v1.md: "I'll be interested in how the battery sees it." Each reader was handed the whole draft (defs-20.md, defs-21.md) and twelve questions (questions-20.md), the last asking for every place two builders could differ.
| Test | What it held | What its readers missed or found |
|---|---|---|
| 20 | the draft at a38dbeafca | Haiku and Sonnet: every question right. Gemma: right but for the shell's own $$ (question 4), and one run posted nothing on exit 0, which found a gap: is an empty STDERR "kept"? Qwen, 5 of 5: directory setting $$tree$$ (5), the wrong reason for the run-user refusal (8), the node's own files (9). Readers' gaps: "having kept anything"; the first line's values; "the server's order" undefined here; when values are checked; manual in a list of strings; a path in both lists; FIFO for waiting deliveries; note's form |
| 21 | the same, those gaps closed | Haiku and Sonnet: every question right; Gemma: every question right but the shell's own $$, a matter of the shell, not the protocol. Qwen still misses questions 5, 8 and 9 in most runs |
Qwen 7B, as in batch 5, follows the protocol a few definitions at a time but not whole.