Login
Tech-note 5a94445305
Login

The reading battery (battery/reading.cr, check-in ba9105b3cc) handed each model OCX's protocol version 2 documents with no tools and asked the checks in battery/readings: questions a client's author must answer, each with numbered choices. Right is the right choice citing a section that settles it; partial, the right choice citing another; wrong, another choice. Two trials each, through this host's Ollama.

Run 1: the text at ba9105b3cc

model check right partial wrong missing
gemma4:31b acting 16 0 0 0
gemma4:31b envelope 14 0 2 0
gemma4:31b runs 14 0 2 0
qwen2.5-coder:7b acting 4 7 5 0
qwen2.5-coder:7b envelope 0 14 2 0
qwen2.5-coder:7b runs 3 4 3 6

Gemma's four misses were the text's, both twice:

Run 2: the text at 1da76212f9

Those two fixed, with 2.3's unconfined runs, 2.3's owner bounds, 2.8's values and 2.11's filters, and sections accepted by their anchors. clients added.

model check right partial wrong missing
gemma4:31b acting 16 0 0 0
gemma4:31b envelope 16 0 0 0
gemma4:31b runs 16 0 0 0
qwen2.5-coder:7b acting 4 7 5 0
qwen2.5-coder:7b envelope 11 3 2 0
qwen2.5-coder:7b runs 6 5 5 0
gemma4:31b clients 16 0 0 0
qwen2.5-coder:7b clients 6 8 2 0

Gemma 31B: 64 of 64 right. Qwen 2.5 Coder 7B: 27 of 64 right. Its reasons often quote the right rule while its numbered choice contradicts it, as the proposal's section 5 found.

Every request, reply and marked answer of each run is attached, as reading-runs.tar.


Attachments: