Login
Transition runbook
Login
Help

Transition runbook: serve, confine, the receipt reader, and run-work in the daemon

What this switches:

scripts/README.md says what each sandbox holds. The receipt reader and the confined daemon were tried on a scratch deployment (tech-note bae6999eb9).

One account. The server runs in a user namespace where only root is mapped, so Fossil, its pages, the receipt reader and the daemon all run as that root. No ocx account is made. The sandboxes keep them apart, not accounts: the kernel refuses a nested sandbox's reads into Fossil's processes, which was tried on this host on Oct 29. The one thing an account would add: the daemon can still signal Fossil's processes.

What it does not switch yet: the page's other paths still run the release's program, with its direct access. That is the plan's temporary exception E6, which ends in phase 2 below. The maintainer chooses when this runs (decision D9).

Admin: root on the server host, inside the namespace Fossil runs in. Steps were rehearsed on scratch repositories on Oct 27 unless marked not rehearsed. If a check fails with no fix given, stop.

Shell

D=/var/lib/fossil; OLD=/var/lib/ocx/release2; ROOT=/var/lib/ocx/agents-daemon; U=https://ollama.openbeagle.org; SU=ec2-user
LX=/usr/local/libexec/ocx; EXT=/var/www/cgi-bin
NODES="bootstrap-qcow2:bq2-node cleave:cleave-node crystal-robots:crobots-node glint:glint-node interactor:interactor-node mechanical-turk:node-mt ollama:ollama-node stone:stone-node swk-ai:swk-ai-node"
ls $D/stone.fossil $OLD/current/ocx2 /root/.local/bin/claude
for r in $D/*.fossil; do fossil user capabilities $SU -R $r | grep -q s || echo "$r: no setup"; done

Check:

Before the switch

0. The staged release. 

REL=$(readlink $OLD/staged/next); cat $OLD/staged/$REL/gate.txt
Check: REL is 89362e0607d5 (staged 3:03 PM EDT Oct 29, 642 examples, 0 failures) and the line starts green; if not, stop.

1. Forgejo (stone). Change ocx/stone's pull address from https://ollama.openbeagle.org/ollama/ext/git/stone.git to https://ollama.openbeagle.org/stone/ext/git/stone.git; both served the same HEAD on Oct 27. Check: the next pull succeeds within 15 minutes, and GitHub is unchanged. If not, restore the old address.

2. swk-ai program (admin). Deferred by the maintainer on Oct 29: "I don't want to mess with the isolation of swk-ai quite yet until everything is running nice." The program and its extension link stay as they are. Later: copy the program to a fixed path outside any run workspace, repoint its extension link, and check that /swk-ai/home still reaches the login page and a maintainer's view loads.

A. The maintainer's programs. The maintainer reviewed and merged scripts/ into agent-tools (e1c71c1371). In a checkout of 89362e0607 on agent-tools, or of any later one on that line: 

shards build serve confine receipt-reader sync-fossil
install -d -o root -g root -m 0755 $LX
install -o root -g root -m 0755 bin/serve bin/confine bin/receipt-reader bin/sync-fossil $LX/
install -o root -g root -m 0644 scripts/hosts.json $LX/hosts.json
Check: ls -l $LX shows the four programs and hosts.json, owned by root. confine refuses a hosts.json that is not owned by root or that the group or others can write.

B. The daemon's homes and the reader's socket directory. Copy, don't move, so rollback needs nothing. hosts.json binds each /home/ocx/X at /root/X inside the daemon, so runs' patterns and r-senders.json are unchanged. 

install -d -m 0755 /home/ocx
for p in .codex .claude .claude.json .local; do [ -e /root/$p ] && cp -a /root/$p /home/ocx/; done
install -d -o root -g root -m 0700 /run/ocx
Check: ls -a /home/ocx lists the four copies. serve does not make /run/ocx; if /run is emptied at boot, make it again before each start, or the receipt reader does not start.

3. Hand off, then stop. Over Remote Control, stone asks glint, crystal-robots and swk-ai to write their handoff notes. Then the admin runs: 

while read n r; do echo "== $n"; $OLD/current/ocx2 --repository $D/$n.fossil --user $SU stop $r </dev/null | grep -E '^(ok|stopped)|ending'; done <<'EOF'
glint f62d622a55
crystal-robots 3c42ee4b9a
swk-ai 9d9c6722f2
glint b2d65da166
crystal-robots 546a05a4dd
crystal-robots 8ed286a74e
swk-ai 8430486bbf
interactor f960dfe6a8
mechanical-turk 35003d31bf
mechanical-turk 79d2e85b2c
cleave 57c73e54e3
cleave 18ec7f20e8
bootstrap-qcow2 4bf43c40a0
bootstrap-qcow2 1033437e64
bootstrap-qcow2 51c04f65dc
EOF
Rows, as of Oct 29: the handed-off sessions, the replaced and duplicate ones, idle interactor, the Slowpen tunnel, mechanical-turk's session, cleave's waiting edit, cleave's session, and aarch64-1's waiting runs. cleave's session was started on Oct 9 by an older form, ocx2 run --session; if it does not end, kill -TERM its ocx2 run process, and its sandbox ends with it. As a link, $U/NAME/ext/ocx2-cgi/stop?run=RUN shows the dry run, and one press stops.

Check: 

pgrep -af 'ocx2-hook .*--watch'
for p in $NODES; do echo "== ${p%%:*}"; $OLD/current/ocx2 --repository $D/${p%%:*}.fossil --user $SU status | grep -A2 '^  runs:'; done
pgrep prints nothing; every repository shows runs: none, though bootstrap-qcow2 may still list aarch64-1's runs. Stop any other run that remains the same way; if it lingers, kill -TERM its watcher (it ended within 8 s in the rehearsal). Sessions are down from here to step 8.

The switch (admin)

Run each accept and migrate with --dry-run first.

4. Agent root. 

mkdir -p $ROOT/staged && cp -a $OLD/staged/$REL $ROOT/staged/ && ln -s $REL $ROOT/staged/next
$OLD/staged/next/ocx2 --repository $D/ollama.fossil --user $SU accept --root $ROOT
ls -l $ROOT/current/ocx2
Check: current: $REL, and the program is listed.

5. Accept on the old root. 

$OLD/current/ocx2 --repository $D/ollama.fossil --user $SU accept --root $OLD
Check: current: $REL, previous: aa650c80c410. Nothing exports until migrate runs (27 s in the rehearsal), so go straight on.

6. Migrate. 

mig() { $ROOT/current/ocx2 migrate $D $U $ROOT timeline session --shares 'crystal-robots glint ollama swk-ai' --run-users 'swk-ai:codex' "$@"; }
mig --dry-run | grep '^  - ' | sort | uniq -c
Check: the dry run lists 31 add_hook, 31 set_chat_timeline_user, 10 delete_hooks, 9 make_run_user_filed and 10 set_password (the Oct 27 counts; a repository added since adds one to each of the first two). 
mig > /root/migrate.txt; head -1 /root/migrate.txt
for r in $D/*.fossil; do echo "$(basename $r) export=$(fossil hook list -R $r | grep -c 'git export --if-mirrored') old=$(fossil hook list -R $r | grep -c ocx2-hook) chat=$(fossil setting chat-timeline-user -R $r | awk '{print $NF}')"; done
fossil git status -R $D/stone.fossil | grep Status
Check: ok: true. Every repository shows export=1 old=0 chat=timeline, and stone's export is up-to-date.

7. Restart with serve. Stop the running start-fossil the maintainer's way. Then remove the page's old link, so that serve links confine in its place, and start $LX/serve the maintainer's way. 

readlink $EXT/ocx2-cgi        # note it for rollback: $OLD/current/ocx2
rm $EXT/ocx2-cgi
serve prints linked /var/www/cgi-bin/ocx2-cgi to /usr/local/libexec/ocx/confine, then starts confine daemon and confine receipt-reader, then Fossil. Not rehearsed on this host; serve was run on scratch.

Check the receipt reader: these two lines name the same user. 

H1=$(fossil sql -R $D/ollama.fossil "SELECT b.uuid FROM blob b JOIN forumpost f ON f.fpid=b.rid ORDER BY f.fmtime DESC LIMIT 1" | tr -d "'")
curl -s -d "json=on&artifacts=$H1" $U/ollama/ext/ocx2-cgi/receipt | grep -o '"sender":"[^"]*"'
fossil sql -R $D/ollama.fossil "SELECT u.login FROM blob b JOIN rcvfrom r USING(rcvid) JOIN user u USING(uid) WHERE b.uuid='$H1'"
A 503 receipt service not answering means the reader is not running: read serve's output (a missing /run/ocx is the likely cause).

Check the page: signed in, the OCX page and console load as before.

Check the daemon: each node shows ok: true, a recent when: and runs: none. Rerun until all nine answer (clone time is unmeasured). 

for p in $NODES; do n=${p%%:*}; echo "== $n"; $ROOT/current/ocx2 --repository $ROOT/node/clones/$n.fossil status --nodes $ROOT/node/nodes.json | grep -E '^ok:|^  runs:|in_care:|^    when:'; done
A run showing in_care: false was missed in step 3: stop it, then kill -TERM its old watcher.

Check the daemon's sandbox: entering its namespaces, the repositories are not there, and Fossil's are refused. 

DP=$(pgrep -n -f "$ROOT/current/ocx2 daemon"); FP=$(pgrep -o -f 'fossil server')
nsenter -t $DP -U -m --preserve-credentials head -c1 $D/ollama.fossil
nsenter -t $DP -U -m --preserve-credentials head -c1 /proc/$FP/root$D/ollama.fossil
The first says "No such file or directory", the second "Permission denied". Both were rehearsed on this host on Oct 29, with a daemon-style sandbox. If either reads a byte, the boundary does not hold: roll back.

After the daemon starts

8. Successor sessions. Signed in, stone opens each link, reads the dry run and presses once.

Check: each run reaches started and answers a post. If a session stalls on Claude's trust question, the admin runs $ROOT/current/ocx2 attach RUN, answers it, and leaves with Ctrl-] (not rehearsed).

9. swk-ai. Time one test message's answer (about 5 s before). Confirm the page ignores timeline lines and the next greeting is answered. swk-ai's single greeting daemon is live since 42877cf10c.

10. A run inside the confined daemon. The first session run in step 8 reaching started shows that bubblewrap still starts an agent sandbox inside the daemon's sandbox, as it did on scratch.

Each later release. The ocx2 daemon exits when a release is accepted on ROOT, and serve starts it only once. After each accept, start it again in the background as serve does, with $LX/confine daemon. Stopping serve does not stop the two programs it started, so before starting serve again, stop them as in rollback step 2.

Rollback

History, 2026-10-31: the switch is done and working, and migrate has been replaced by project (the maintainer: "should not we be done with migrate and remove it?"). These steps are kept as a record; they need a release from before that change.

  1. While the daemon runs, stop each run it started (step 7's loop lists them); wait for each ended. With the daemon down a stop does nothing, so start serve first if needed. 
    $OLD/current/ocx2 --repository $D/NAME.fossil --user $SU stop RUN
  2. Stop serve the maintainer's way; this ends Fossil but not the daemon or the receipt reader. Stop those two, put the page's link back, and start the old start-fossil the maintainer's way: 
    pkill -TERM -f "$ROOT/current/ocx2 daemon"; pkill -TERM -f "$LX/receipt-reader"
ln -sfn $OLD/current/ocx2 $EXT/ocx2-cgi
    Check: pgrep -af "$ROOT/current/ocx2 daemon|$LX/receipt-reader" prints nothing. The copies in /home/ocx can stay.
  3. Use rollback; accept refuses. 
    $OLD/current/ocx2 --repository $D/ollama.fossil --user $SU rollback --root $OLD
    Check: current: aa650c80c410.
  4. Remove the node users' capabilities, which the old hook refuses, then reinstall the old hook. 
    for p in $NODES; do n=${p%%:*}; u=${p#*:}; fossil user capabilities $u "" -R $D/$n.fossil; $OLD/current/ocx2 --repository $D/$n.fossil --user $SU hook --login $u --root $OLD | grep -E 'outcome|stopped'; done
$OLD/current/ocx2 --repository $D/jkembedded-audio-board.fossil --user $SU hook --root $OLD | grep outcome
    Check: every line shows outcome: installed.
  5. Check: swk-ai answers at once again.

Steps 1, 3, 4 and 5 were rehearsed; step 2 replaces the rehearsed systemd stop. Once rolled back, the switch can run again.

Phase 2: the rest of the boundary (outline)

These steps are written in full once their check-ins are staged and accepted (the plan, section 10). The receipt reader, confine and the daemon's sandbox are already in the switch above. In order:

  1. The daemon's socket for the web page, and its web clones. migrate adds the web user and web.json. The daemon restarts and clones each web repository. Check: node/web.ready.
  2. confine, as ocx2-cgi, sends every path to the daemon, not only /receipt, so the passthrough (E6) goes.
    • Checks: the plan's A1, A2 and A4 to A9.
    • Rollback: the previous build of confine, which the maintainer reviews like every change in scripts/.
  3. Writes through Fossil's own form, as D2 decides.
  4. The git program under its own policy, reading only the mirrors.
  5. The v1 pieces (D7): retire the v1 /ext programs, the wake hook and ocxd, after opus-stone has moved, and withdraw ocx2-cgi-next (D4).
  6. swk-ai reads and sends through its own socket, as its own user holding R (its session's change), and its scheduler becomes a service.
  7. The end of E6. No program but Fossil and the receipt reader opens a served repository. Check: A1, A3 and A12 again.