2026-10-15 and 16, OCX at 4121f101b8 (branch agent-tools), battery/main.cr --audit. The maintainer, 2026-10-15: "For all of the local models, I'd like to also judge them on their ability to rate a code change, specifically looking at code check-ins in the history, against the 4 virtues of clarity, interchangability, drift and parsimony. Ultimately, I want to put local models in to perform the audits, but I need an analysis of how well they can do the task."

• The virtues: README.md's four, as written: Clarity; Loose Coupling, "can be replaced", the maintainer's interchangeability; Demonstrability, which guards against drift; Parsimony.
• What each model saw: a check-in's comment and diff (files it added shown whole) and README's virtues section; it answered through Ollama's structured output: a score from 1 to 5 and a reason for each virtue, a verdict and a finding.
• Items: eight seeded pairs (battery/audits/seeded.json), each a change clean and with one planted fault against one virtue; and sixteen check-ins of the history with a verdict recorded on them, eight fail and eight pass. Three trials each, ten local models.
• Fault identified: judged afterwards, for each rating of a faulty change, by a Claude judge per model, each "yes" then challenged by a skeptic who overturned what did not name the planted fault itself. The judgments are attached.
• On the GPU: Ollama held qwen3-coder-next (51.6 GiB) with 42.7 GiB on the GPU, the rest on the CPU; deepseek-r1:70b's 199 seconds a rating fits its running largely on the CPU too.

gemma4:31b's "faulty failed" counts all 24 of its faulty ratings; one of its clean ratings was unusable, so its paired columns are of 23.

Analysis

Ten models each rated 32 check-ins three times: 8 seeded pairs (a change clean and with one planted fault) and 16 history check-ins with recorded verdicts (8 fail, 8 pass). Counts are of 24 (8 items × 3 trials) unless shown; gemma4:31b gave one unusable answer. Fault identified counts ratings verified as naming the planted fault.

What it shows

Four models can partly audit. The top four name the fault in 12 to 16 of 24, too close to rank. All catch the bypassed wrapper every time; vague names, the overbuilt framework and ENV["HOME"] mostly. Duplicate hits prove little; the diff comment admits it. No rating caught the swallowed error, one caught "Demonstrates: it works", and one caught the comment saying newest where the code takes the oldest.

Read the finding, not the verdict. The gemma4 models fail 21 faulty ratings but name the fault in 15–16. qwen2.5-coder:32b and qwen3-coder-next often name the fault, then pass the change.

Six cannot audit. qwen3-coder:30b and openbeagle-coder-codex:30b pass nearly everything; their only fails are the wrapper pair. hunyuan:7b and deepseek-coder-v2:lite fail clean and faulty items equally often. qwen2.5-coder:7b's reasons are often bare file:line references. deepseek-r1:70b names 8, at 199 seconds a rating.

False alarms. The top four fail 2–3 of 24 clean ratings: one in eight to twelve clean check-ins sent to a person needlessly. The gemma4 models' three are all one clean change, failed for reading a file: a habit.

History. Recorded passes agree because the models pass nearly everything. Of 240 ratings of recorded fails, 7 failed the check-in, none for the recorded fault.

Few trials. Each fault is one change rated three times, so per-fault figures are 0–3 of 3. codex-crystal-gemma4 and openbeagle-coder-codex:30b repeat themselves nearly word for word, so their trials are nearly one observation.

What diff-only rating cannot catch

Four recorded faults need more than the hunk: defects in untouched files (99287f70fe, 102d703bdb), a citation needing a lookup (094cacaf84), and failures seen only when code or CI runs (877097e6b9, 1d590ec03e). A local auditor would need the whole changed files and the code they call, cited artifacts fetched by hash, and the output of the gate and each Demonstrates command. That fixes input only: the other four sat wholly in the hunk, and every model missed them.

Recommendation

Try gemma4:31b first, qwen2.5-coder:32b second (three times faster), both read by finding. codex-crystal-gemma4 adds little over gemma4. None caught a recorded fault, so none replaces the current audit; use them as a pre-check a person reads.

Next:

1. Make "names the planted fault" the headline number.
2. Rerun the recorded fails with whole files, fetched citations and gate output.
3. Plant the missed faults: swallowed error, missing demonstration, contradicting comment, unchecked name joined into a path, open before type check. More pairs, not more trials.
4. Add a clean pair with a plain File.read to measure the gemma4 habit.

The records (every exact request and reply), the ratings, the items and the verified judgments are attached as one tar.