2026-10-15, OCX at f6cb19bbff (branch agent-tools), the same tasks and settings as run 1, after its four interface fixes: an argument a tool does not take is refused naming those it does; an empty replaces is none; old empty adds to a file's end; old and new are text, not a diff. qwen2.5-coder:7b 5 trials of each task and variant; the two larger models 1, to check nothing they did broke.
Passed
| model | steer | plain |
|---|---|---|
| qwen2.5-coder:7b | 11/25 | 10/25 |
| qwen3-coder:30b | 5/5 | 5/5 |
| gemma4:31b | 5/5 | 5/5 |
qwen2.5-coder:7b by task:
| task | steer | plain |
|---|---|---|
| add-section | 4/5 | 0/5 |
| fix-bug | 1/5 | 1/5 |
| rename | 1/5 | 2/5 |
| share-binary | 5/5 | 5/5 |
| wiki-status | 0/5 | 2/5 |
What it says
- The 7B model passed 42% of its trials, from 33% in run 1; the larger models passed every trial in both runs, so these tasks do not tell them apart.
okandnext_step(steer against plain): 11 against 10 of 25, within a trial of each other. They stay, as costing a larger model nothing; more trials, and harder tasks for the larger models, are needed to decide.- The 7B model's failures are now mostly its own: edits it does not copy exactly, and stopping before it saves.
The records, every exact request, tool answer and reply of each trial, are attached.