The reading battery (battery/reading.cr, check-in ba9105b3cc) handed each model OCX's protocol version 2 documents with no tools and asked the checks in battery/readings: questions a client's author must answer, each with numbered choices. Right is the right choice citing a section that settles it; partial, the right choice citing another; wrong, another choice. Two trials each, through this host's Ollama.
Run 1: the text at ba9105b3cc
| model | check | right | partial | wrong | missing |
|---|---|---|---|---|---|
| gemma4:31b | acting | 16 | 0 | 0 | 0 |
| gemma4:31b | envelope | 14 | 0 | 2 | 0 |
| gemma4:31b | runs | 14 | 0 | 2 | 0 |
| qwen2.5-coder:7b | acting | 4 | 7 | 5 | 0 |
| qwen2.5-coder:7b | envelope | 0 | 14 | 2 | 0 |
| qwen2.5-coder:7b | runs | 3 | 4 | 3 | 6 |
Gemma's four misses were the text's, both twice:
- e2: it read section 2.1's "with no info string or
json" as refusing a fence opened withjson, which OCX accepts and writes itself. - r8: it read exit 137 as surely signal 9; 2.12 never said a signal and an exit code above 128 are written alike.
Run 2: the text at 1da76212f9
Those two fixed, with 2.3's unconfined runs, 2.3's owner bounds, 2.8's values and 2.11's filters, and sections accepted by their anchors. clients added.
| model | check | right | partial | wrong | missing |
|---|---|---|---|---|---|
| gemma4:31b | acting | 16 | 0 | 0 | 0 |
| gemma4:31b | envelope | 16 | 0 | 0 | 0 |
| gemma4:31b | runs | 16 | 0 | 0 | 0 |
| qwen2.5-coder:7b | acting | 4 | 7 | 5 | 0 |
| qwen2.5-coder:7b | envelope | 11 | 3 | 2 | 0 |
| qwen2.5-coder:7b | runs | 6 | 5 | 5 | 0 |
| gemma4:31b | clients | 16 | 0 | 0 | 0 |
| qwen2.5-coder:7b | clients | 6 | 8 | 2 | 0 |
Gemma 31B: 64 of 64 right. Qwen 2.5 Coder 7B: 27 of 64 right. Its reasons often quote the right rule while its numbered choice contradicts it, as the proposal's section 5 found.
Every request, reply and marked answer of each run is attached, as reading-runs.tar.
Attachments:
- reading-runs.tar [download] added by x86-1-session on 2026-11-03 05:25:05. [details]