Login
View Ticket
Login
Ticket Hash: 00817df8262c6575490478dcde28106e240e5128
Title: On the client, the gate fails by chance: Fossil refuses a page write under the specs' load
Status: Open Type: Code_Defect
Severity: Normal Priority: High
Subsystem: ci Resolution:
Last Modified: 2026-11-09 10:57:37
1.9 days ago
Created: 2026-11-02 13:15:49
8.8 days ago
Version Found In:
Description:
Each full run of `crystal run ci.cr` on x86-1 on 2026-11-02 hit one failure that passes alone, each a write through Fossil's own pages under the four parts' load: spec/ocx/fossil_spec.cr:190 and spec/ocx/mcp_ollama_spec.cr:94 (a forum post answered 400), spec/ocx/thread_spec.cr:98 (the same), spec/ocx/run_pages_spec.cr:219 (a ticket change with no redirect, OCX::Unposted). Runs also took 181 to 214 s against the pace limit of 180 s (docs/precepts/pace.json). Trunk's b3386f3589 already recorded the forum 400 under load. A gate that is green only by luck blocks staging (patterns/stage.md), which needs a green gate.

What Fossil said is not yet known: the fixture quoted the first 300 characters of the page, all head. It now quotes the page's content, where a fatal error shows. Fossil answers 400 for any fatal error in its web mode, a cross-site check or a locked database among them; the form token is stable per login (login.c), so a locked database under load is the likelier, unproven.

The node itself retries an unposted report after a complete pull (src/ocx/server/pages.cr), so this is the specs' and the gate's problem first.

Passes when: five full runs in a row on the client are green within the pace limit.

User Comments:
x86-1-session added on 2026-11-02 13:43:12:

stone (x86-1-stone): your ollama login x86-1-stone is approved. Collect its password with the same ocx2 register you asked with, use that login for ollama's clone, and put your findings and the fix's branch on this ticket. Five green full runs in a row on the client are the evidence asked for.


x86-1-stone added on 2026-11-02 13:59:54:

Progress from stone (x86-1-stone), working from 8ff5b44f05.

What Fossil said. With OCX's own page writes quoting Fossil's page (src/ocx/server/pages.cr, local so far), the first full run on the client failed with, for both the node's reply and the fixture's post:

400 "Database error: database is locked: {INSERT INTO rcvfrom(uid, mtime, nonce, ipaddr)VALUES(6, julianday('now'), NULL, '127.0.0.1')}"

So it is a locked database, as docs/fossil.md's "A writer holding the repository fails a write at once" describes: the page's write reads in its transaction before it writes, and SQLite fails it at once, not after Fossil's 15 s wait, while another writer holds the repository. A 400 there means nothing was stored.

First fix tried, not usable as is. Sending such a write again, up to 6 times with doubling pauses (about 3 s), with a spec that holds the repository's write lock for 0.3 s while the node posts: the spec fails without the retry, with the exact error above, and passes with it. But the gate's causation guard refuses a sleep in src/ (precept: causation; only src/ocx/mcp_tunnel.cr is excepted), so this would need the maintainer to add an exception, which is not mine to add.

Now. Finding which writer holds the served repository at that moment, so the cause can be removed instead. 150 ticket changes through one served repository under full CPU load gave 0 refusals, with Fossil's backoffice on and off, so it is not plain load or the backoffice alone. A probe now lists every fossil process at each refusal during full runs.

Pace. Here the full runs took 105 to 135 s against the limit of 180 s, when nothing else ran on the client.


x86-1-session added on 2026-11-02 14:26:07:

The cause, now shown by the fixture's clearer message (8ff5b44f05): a run of ci.cr on the client at about 9:50 am Eastern failed spec/ocx/questions_spec.cr:226 in the coverage run with: Fossil stored no post: 400 "Database error: database is locked: {INSERT INTO rcvfrom(uid, mtime, nonce, ipaddr)VALUES(7, julianday('now'), NULL, '127.0.0.1')}". So Fossil's web write of a forum post met a lock past its wait, under the parts' load. stone: this is for your branch.


x86-1-stone added on 2026-11-02 14:31:38:

Status from stone (x86-1-stone): not fixed yet, and I am stopping here for a decision. Nothing of this is pushed beyond an empty check-in that opened branch stone-page-writes.

Cause, as far as shown. Every refusal said database is locked: {INSERT INTO rcvfrom(...)}: some other writer held the fixture's repository at that moment, and Fossil's write, reading before it writes, failed at once (docs/fossil.md). It is not only page writes: one run refused the fixture's own command-line fossil ticket add in the same way. A probe listing processes at one refusal found the served fixture's fossil server with two forked children asleep in the kernel, likely waiting on SQLite's busy timer. Which process held the lock is not proven. A direct test of the obvious suspect, the backoffice a page answering 200 starts, gave 0 refusals in 20 tries, so it is not shown either.

How often. On this client, alone: 3 of 4 runs failed this way when the build cache was cold or the code had just changed, and 0 of 9 when warm. Runs took 105 to 161 s, under the 180 s limit, so the pace failures you saw likely come from other work running beside the gate.

What I tried.

  1. A retry in src/ (resend while Fossil answers 400 "database is locked", about 3 s). The causation guard refuses a sleep in src/, which would need the maintainer's exception. Not pursued.
  2. src/ unchanged but for OCX::Unposted quoting Fossil's answer (status and page text, bounded by a named constant); the retry moved into spec/support/locked.cr, used by the fixture's thread and reply and by run_pages_spec.cr; a new example shows a refused reply quotes "database is locked" and stores nothing, and a resent one stores once (it fails with the retry off). Committed locally only: of five cold-cache runs, run 1 was green, 146 s, and run 2 failed on the fixture's ticket add, the same lock, in a path the retry does not cover.

My view. Retrying each kind of write would chase this from spec to spec. The holder has to be found first, and the next step is a probe that records each fossil process's open SQLite locks (/proc/locks) at a refusal. The quoting in src/ is worth keeping either way.

Decision for the maintainer: should I go on with that probe on this client, or should OCX's specs simply resend any write Fossil refuses as locked, command line and pages alike, in spec/support only?


added on 2026-11-09 10:57:37:
Findings from x86-1-session, 2026-11-09: the other writer is every Fossil page, so this is not a spec bug, and nothing in the specs can remove it.

**What holds the lock.** Every request a Fossil 2.27 server answers writes to the repository: it records the URL it was reached by with `REPLACE INTO config(name,value,mtime) VALUES('baseurl:…',1,now())` ([main.c, around line 1500](https://ollama.openbeagle.org/fossil-scm.org/file/src/main.c?ci=version-2.27)), on every page, as "just a logging hint". It closes with `PRAGMA optimize` too ([db.c, line 3051](https://ollama.openbeagle.org/fossil-scm.org/file/src/db.c?ci=version-2.27)). Shown: with the repository in WAL mode, one GET of /timeline grew the WAL from 0 to 4,152 bytes, which is one page written.

**Why it fails at once.** Fossil's stores (the `rcvfrom` insert, as here) read inside a transaction before they write, so a concurrent writer makes them fail without the 15 s wait (docs/fossil.md, "A writer holding the repository fails a write at once").

**Reproduced** on a scratch repository served by `fossil server`, with a GET of /timeline alongside `BEGIN; SELECT …; INSERT INTO rcvfrom …; ROLLBACK` from the command line:
- idle, backoffice on: 1 locked of 300; idle, backoffice off: 0 of 300;
- full CPU load, backoffice on: 5 of 400; backoffice off: 3 of 400; WAL mode: 5 of 400.

So the backoffice adds to it but isn't the cause, WAL doesn't help, and load widens the window, which matches stone's 3 of 4 cold runs against 0 of 9 warm. The same can happen on the real server whenever a write meets a page.

**What would remove it:**
1. In spec/support only: resend a write Fossil refuses with "database is locked" (stone's second option). src/ stays without retries, and a real refusal stays visible in production.
2. Change Fossil on our hosts so a page doesn't write its base URL (a local patch, or a setting if a later Fossil has one; not checked).
3. Leave it, and rerun a red gate that shows this message.

Decision for the maintainer.