Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
| Comment: | The battery measures auditing: local models rate check-ins against the README's four virtues; ok and next_step stay, by the data Intent: The maintainer, 2026-10-15: "For all of the local models, I'd like to also judge them on their ability to rate a code change, specifically looking at code check-ins in the history, against the 4 virtues of clarity, interchangability, drift and parsimony. Ultimately, I want to put local models in to perform the audits, but I need an analysis of how well they can do the task." And of the proposal's question 4: "what does the data show works best?" Expression: battery/audit.cr, audit_score.cr, audit_run.cr: --audit REPOSITORY rates each item with each model through Ollama's structured output (a score and reason per virtue, a verdict, a finding), the README's virtues section as written; items: eight seeded pairs in battery/audits/seeded.json, each change clean and with one known fault against one virtue, and check-ins of the history with a verdict recorded, failed and passed alike; measured: faulty rated lower, failed and named, clean failed, agreement with the recorded verdict, verdicts steady over trials. Place#diff_of (added files whole), #comment_of, #verdicts (audited and sym-audited). Ollama::Request takes a format. The stand-in Ollama moves to spec/support. HISTORY.md decisions 68 (ok and next_step stay: 28 against 25 of 73 trials for the 7B model, the larger alike) and 69. Demonstrates: spec/battery/audit_spec.cr (a stand-in failing every change, measured; the virtues section), spec/ocx/fossil_spec.cr (diff, comment, verdicts); crystal run ci.cr green: 353 examples, 0 failures; coverage 100.0%. |
|---|---|
| Downloads: | Tarball | ZIP archive |
| Timelines: | family | ancestors | descendants | both | agent-tools |
| Files: | files | file ages | folders |
| SHA3-256: |
4121f101b8c20c8e995f4bbc60753dc5 |
| User & Date: | opus-ocx2 2026-10-15 14:31:35.183 |
Context
|
2026-10-16
| ||
| 00:01 | The audit battery names a fault only by words specific to it, and counts the same words in the clean twin; node says why it exists Intent: The audit run (audit-1): the measure "fault named" matched words such as name, path and test that appear in almost any review, crediting models that failed nothing (qwen3-coder:30b named 18/24 while failing 3/24). And the maintainer, 2026-10-15, of node's summary: "Why does it say \"host other than the server's\"?" Expression: battery/audits/seeded.json: each pair's keywords specific to its planted fault (wp, /var/lib, rescue nil, registry...); Audit::Measure#named_clean, the same words in the clean twin's rating, beside named; the table's separator follows its columns. node's summary says the hook acts on what is pushed to the server, nothing is pushed to a build host, and a clone cannot tell who sent what. Demonstrates: spec/battery/audit_spec.cr; crystal run ci.cr green: 353 examples, 0 failures; coverage 100.0%. check-in: 29d5660c16 user: opus-ocx2 tags: agent-tools | |
|
2026-10-15
| ||
| 14:31 | The battery measures auditing: local models rate check-ins against the README's four virtues; ok and next_step stay, by the data Intent: The maintainer, 2026-10-15: "For all of the local models, I'd like to also judge them on their ability to rate a code change, specifically looking at code check-ins in the history, against the 4 virtues of clarity, interchangability, drift and parsimony. Ultimately, I want to put local models in to perform the audits, but I need an analysis of how well they can do the task." And of the proposal's question 4: "what does the data show works best?" Expression: battery/audit.cr, audit_score.cr, audit_run.cr: --audit REPOSITORY rates each item with each model through Ollama's structured output (a score and reason per virtue, a verdict, a finding), the README's virtues section as written; items: eight seeded pairs in battery/audits/seeded.json, each change clean and with one known fault against one virtue, and check-ins of the history with a verdict recorded, failed and passed alike; measured: faulty rated lower, failed and named, clean failed, agreement with the recorded verdict, verdicts steady over trials. Place#diff_of (added files whole), #comment_of, #verdicts (audited and sym-audited). Ollama::Request takes a format. The stand-in Ollama moves to spec/support. HISTORY.md decisions 68 (ok and next_step stay: 28 against 25 of 73 trials for the 7B model, the larger alike) and 69. Demonstrates: spec/battery/audit_spec.cr (a stand-in failing every change, measured; the virtues section), spec/ocx/fossil_spec.cr (diff, comment, verdicts); crystal run ci.cr green: 353 examples, 0 failures; coverage 100.0%. check-in: 4121f101b8 user: opus-ocx2 tags: agent-tools | |
| 13:38 | Text of more than one line is laid out in a fenced block, its lines exactly, nothing added before them Intent: Battery run 3: gemma4:31b failed add-function in all 6 trials, its old never found: the block layout of read's text indented each line of the file by four spaces beneath text: |, and the model copied those spaces into old. The same layout likely cost qwen2.5-coder:7b many of its edits. Expression: Text.line puts text of more than one line in a fenced block, longer than any run of backticks it holds, the text exactly between its fences. Demonstrates: spec/ocx/elements_spec.cr (a fenced block, a longer fence), spec/battery/battery_spec.cr; crystal run ci.cr green: 351 examples, 0 failures; coverage 100.0%. check-in: a1faf72aee user: opus-ocx2 tags: agent-tools | |
Changes
Changes to HISTORY.md.
Added battery/audit.cr.
Added battery/audit_run.cr.
Added battery/audit_score.cr.
Added battery/audits/seeded.json.
Changes to battery/main.cr.
Changes to battery/ollama.cr.
Changes to battery/run.cr.
Changes to docs/survey/functionality.json.
Added spec/battery/audit_spec.cr.
Changes to spec/battery/battery_spec.cr.
Changes to spec/ocx/fossil_spec.cr.
Added spec/support/stand_in_ollama.cr.
Changes to src/ocx/fossil/scratch.cr.