| Ticket Hash: | 2be3582b5deeda4d549ba610f599eb40ac385048 | ||
| Title: | Fossil chat: a chat row missing from chatfts1 makes every signed-in chat-poll answer 400 once it ages out | ||
| Status: | Open | Type: | Code_Defect |
| Severity: | Severe | Priority: | High |
| Subsystem: | fossil | Resolution: | |
| Last Modified: |
2026-11-06 04:16:24 5.1 days ago |
Created: |
2026-11-06 04:11:40 5.1 days ago |
| Version Found In: | |||
| Description: | ||||
|
Found Nov 5–6 by x86-1, all times Detroit. Every signed-in /chat-poll on stone, bootstrap-qcow2, glint and crystal-robots answered 400 from 5:13 PM Nov 5, first from the Stone app on glint, then from x86-1's daemon, which ended on it (ticket [8986816c0b](/tktview/8986816c0b)). Reproduced below on Fossil 2.27 [99675884a9] with a throwaway repository. Nothing has been reported upstream.
**1. What breaks it.** `chatfts1` is an external-content FTS5 index over `chat` (src/chat.c, chat_rebuild_index). It is filled once with 'rebuild' when it is created. After that only TEMP triggers keep it in step, `chat_ai` and `chat_ad`, which chat_create_tables sets up in each connection that goes through Fossil's chat pages. A `chat` row written without those triggers never enters the index. When that row, or any row below it, is older than chat-keep-days, chat_purge runs `DELETE FROM chat WHERE mtime<julianday('now')-:mxage AND msgid<N`. `chat_ad` then asks FTS5 to delete an entry it never held, and SQLite answers SQLITE_CORRUPT_VTAB (11/267). The purge runs on every signed-in chat-poll with name<=0, and on every chat-send. Each one fails, so chat is down for every signed-in client until the index is rebuilt. Tested on a throwaway repository, purging after each change: - a row Fossil wrote, aged: 200, purged - a row inserted with no triggers, aged: **400, SQLITE_CORRUPT_VTAB**, nothing purged - an indexed row whose xmsg was changed, or set to NULL, or mdel set: 200, purged Still unknown: how the four server repositories came to hold unindexed rows. Fossil 2.27 sets up the triggers before each of its own chat inserts. So a writer outside Fossil 2.27's chat pages would do it, such as a Fossil older than chat search, or direct SQL, once `chatfts1` already existed. ocx2 holds no chat SQL. The repositories' history before Nov 5 wasn't checked. Signed-out polls are untouched, because Fossil refuses those before chat_create_tables. **2. Reproduce** (Fossil 2.27 [99675884a9], a throwaway repository only, never a served one): ``` #!/bin/sh # Reproduces the chat-poll 400 on a throwaway repository. Never point it at a served one. set -e export USER=tester D=$(mktemp -d); F=$D/t.fossil; B=http://127.0.0.1:8100 fossil init $F >/dev/null fossil settings chat-keep-days 1 -R $F fossil settings chat-keep-count 0 -R $F fossil server --port 8100 --localauth $F >/dev/null 2>&1 & P=$!; sleep 1 for m in one two three; do # through Fossil: indexed by the chat_ai trigger curl -s -o /dev/null -H "Referer: $B/chat" -H "Origin: $B" -F msg=$m $B/chat-send done # one row Fossil's index never saw: written without the TEMP triggers fossil sql -R $F "INSERT INTO chat(msgid,mtime,lmtime,xfrom,xmsg) VALUES(0,julianday('now'),'x','u','never indexed')" # age the oldest rows past chat-keep-days, so the next poll purges fossil sql -R $F "UPDATE chat SET mtime=julianday('now','-30 days') WHERE msgid<3" curl -s -H "Referer: $B/chat" -o $D/p.html -w "chat-poll: %{http_code}\n" "$B/chat-poll?name=-1&raw=1" sed 's/<[^>]*>//g' $D/p.html | grep -o 'SQL error ([^)]*)[^[]*' kill $P; rm -rf $D ``` Output: `chat-poll: 400` and `SQL error (11,267: database disk image is malformed) while running []`. Without the `INSERT` line it answers 200 and purges. **3. What the server answers.** `HTTP/1.0 400 Bad Request`, a Fossil page saying `Database error: SQL error (11,267: database disk image is malformed) while running []`. /var/log/fossil.err.log on the server: ``` warning: SQLITE_CORRUPT(267): statement aborts at 9: database disk image is malformed; /* TRIGGER chat_ad */ DELETE FROM chat WHERE mtime<julianday('now')-:mxage AND msgid<91 while in /chat-poll ``` The first one was at 5:13:44 PM Nov 5 (glint, Stone/71). The same error came at 9:30 PM for stone (msgid<144), bootstrap-qcow2 (<63), glint (<93) and crystal-robots (<12). x86-1's daemon logs it as "stopped: the server answered 400 to /chat-poll". That stopped it on release 0397ade7223b at 8:33 and 8:59 PM, and on 11f8fa74834d at 9:10 PM. 11f8fa74834d waits out its first 420-second chat wait before polling again, so it lasted 7 minutes and 0397ade7223b lasted seconds. The release made no difference. **4. Repair.** Read from Fossil's current source, so 2.27's pages may differ. Any of these rebuilds `chatfts1` from the `chat` table, so it holds every row again and the purge can delete them: - the repository's /setup_chat, "Rebuild Full-Text Search Index" (rbldchatidx → chat_rebuild_index(1): drop `chatfts1`, create it, 'rebuild') - /srchsetup, "Rebuild The Chat FTS Index", shown only when the main search index exists - /srchsetup, "Delete The Full-Text Index": zFtsDrop drops `chatfts1` with the main index. The next chat-send or signed-in chat-poll goes through chat_create_tables → chat_rebuild_index(0), which creates it and 'rebuild's it. Search stays off. - `fossil chat reindex -R REPO.fossil` on the server A stopgap that leaves the index wrong: raise chat-keep-days so nothing old enough to purge exists. The maintainer repaired the four from the web UI on the night of Nov 5. After that, x86-1's signed-in chat-poll answered 200 on all 11 projects, and its daemon has run clean since 10:27 PM. **Open:** - How did the unindexed rows get in? Check a repaired repository's oldest rows against its history of Fossil versions. - Whether Fossil should repair the index itself, rather than fail every poll. That is the maintainer's call to take upstream. - OCX's own part: one project's chat-poll failure should not end the whole daemon. That belongs on [8986816c0b](/tktview/8986816c0b). | ||||
| User Comments: | ||||
|
x86-1 added on 2026-11-06 04:13:47:
Correction to section 4: Fossil 2.27 has no chat rebuild button or command. "Rebuild Full-Text Search Index" on /setup_chat, "Rebuild The Chat FTS Index" on /srchsetup, and The maintainer raised chat-keep-days on the four repositories instead. That only postpones the failure: the unindexed rows are still there, and the 400s come back when they age past the new limit. Repairs tested on 2.27, on the throwaway repository from section 2. Before each: chat-poll 400. After each: chat-poll 200, the purge completed, and
Either one needs a shell on the server, with a backup of the repository file first. After it, chat-keep-days can go back to its old value. Untested, web UI only. When search is off, /srchsetup on 2.27 shows only "Create A Full-Text Index". Pressing it, then "Delete The Full-Text Index", should drop added on 2026-11-06 04:16:24: **Already reported and fixed upstream; no Fossil forum post is needed.** Checked Nov 6 on fossil-scm.org. - **Report:** Richard Hipp, "Bug fix in /chat full-text-index", Dec 11 2025 8:17 AM: https://fossil-scm.org/forum/forumpost/7dcfc776a0. Fossil's own chatroom failed with the same SQLITE_CORRUPT(267). He traced it to chat rows missing from the contentless FTS index, then a DELETE of them. This is the same cause as section 1 above. - **Fixes, all on Dec 11 2025:** https://fossil-scm.org/home/info/3fe805ac adds `fossil chat purge`. https://fossil-scm.org/home/info/43af043b adds `fossil chat reindex` and rebuild buttons on /srchsetup and /setup_chat. https://fossil-scm.org/home/info/d5504f9d adds a warning on /setup_chat when chat rows are missing from the index. The same work makes chat_create_tables callable from outside chat.c. His post says the cause of the missing entries is "sorted, I think", and that he is monitoring it. - **Releases:** all of it is in Fossil 2.28, released Mar 11 2026 7:31 AM (https://fossil-scm.org/home/info/version-2.28). It is not in 2.27, which is what the server runs. The source diff of src/chat.c between version-2.27 and version-2.28 confirms this. Trunk (2.29) still purges through the same chat_ad trigger, so a missing row still fails the purge. The repair is the rebuild, and 2.28 offers it from the web UI. - **Possible origin of our unindexed rows:** a Nov 19 2025 forum reply suggests writing chat from other programs with `fossil sql -R REPO "insert into chat ..."`. That bypasses the TEMP chat_ai trigger. The search result is titled "Informing from external app using chat, ticket, ...". It is unchecked whether anything on our server did this. So the open question is ours, not upstream's: upgrade the server to Fossil 2.28, or repair the four repositories on 2.27 by shell (the corrections post above). Raising chat-keep-days only postpones the 400s. | ||||