Login
OCX protocol v1: failures, worked through
Login

Each case is a failure or a race, and what protocol v1 says comes of it, artifact by artifact, with the section that decides it. Together they pin down what the rules might otherwise leave to a reader, as codex's review of c28028fbd5 asked (stone, 2026-10-12). The specs name each case they show by its number, so a case keeps its number.

Crashes

  1. The watcher goes before taken. It may have prepared workspace steps but has not written taken or launched. A later daemon round hands the run again; its next watcher prepares it and writes taken before started. If taken was written, even without started, the next watcher taking the free run lock ends what the old lock names and posts lost, never launching again (10).

  2. The watcher goes after taken, before posting started, launching or while running. A later daemon round hands it again. Its watcher takes the run lock, ends the old lock's processes, writes ending with ended, lost, posts it and marks done. Nothing launches twice. With ending already present it retries that exact text; with done it writes nothing (10).

  3. A session's watcher goes while its program runs on, as a sandbox whose ends_with_node is false may. The daemon's next complete pull hands the run to its next watcher. Finding taken and the run lock free, it takes the lock, ends what the old lock names and posts lost, subject to cases 34 and 36 (nodes).

  4. A manual run is waiting when the node goes down. When it starts again, the run has no started, and its manual is true: it goes on waiting for a person (nodes).

Starting

  1. The checkout fails. A run's step {"step": "checkout", "of": "release-9"} names a branch the node's repository does not have, perhaps not yet synced. The node writes ended, refused, with a note naming the step and why, and no started. To try again, send a new run (11, row 10).
  2. The program cannot be launched. command names make, which the sandbox does not have. started is written, the sandbox exits non-zero, and the run ends exited with that code (10).
  3. A branch moves after the run starts. A run's checkout step names dev. The node resolves dev to a check-in, writes started with it in full as the value checkout, and launches. A check-in landing on dev a minute later changes nothing: the process works on the check-in its started names (5).
  4. Two people start one manual run at once, each using run on the agent machine. Both may hand it to watchers. Only one takes its run lock and writes taken, so only one program launches. A later call finds it in care, started or ended and is refused (tools).

Stopping

  1. Stopped before it starts. stone sends a run with manual true, then a stop for it. The node writes ended, stopped, and no started: the thread holds run, stop, ended. A later run tool on it is refused, since it has an ended (10).
  2. Two stops. While a run runs, two admins each send a stop. The node acts on the first it received: it ends the program and writes ended, stopped. The second does nothing, and nothing answers it.
  3. The program exits as a stop arrives. Whichever the node meets first decides: ended, exited, the stop then doing nothing; or ended, stopped. Never both, since a run gets one ended.

Delivery

  1. A busy session. Three posts from users in its hears arrive while its program is busy. They wait in its standard input, one JSON object to a line, in the order received, and it reads them when it is ready (8.1).
  2. A missed signal. A chat wake arrives during a pull, or an approval adds no chat event. The next chat answer, even empty, causes a whole pull. Receipt numbers find every new artifact; each watcher reads its thread through the new completion, so the missed wake loses no post (9). A chat answer that is Fossil's error item, which it gives only to a login it no longer signs in, as once an admin saves that user on the Users page in any repository of its login group, makes the daemon sign in again, once, from the node's file, and ask again; only a second such answer ends it, saying what Fossil said. Before, the first ended it, as the client's did on 2026-11-02 for all its projects.

Refusing

  1. A key the node does not know. A run asks {"limits": {"gpu": 1}}, or {"process": {"stdin": "file"}}, a key a later addition might define. The node refuses it, ended, refused, naming the key, before any started; it never runs it as if the key were not there (11, row 2; 15).
  2. A program not on the host. A run's command names /opt/gemini/bin/gemini, which the node's host does not have. started is written, the process cannot be launched, and the run ends exited with a code other than 0, its kept log saying why (8.2, 12).
  3. A forged run. A user holding only check-in (i) pushes a forum post whose U card says stone, holding a run for lab-1, a build host's node. lab-1 pulls it and asks the server who sent it: the server received it from that user, not from stone, so lab-1 answers it ended, refused, and never decides it again (3, senders).

  4. A forged report. Another user posts an ended whose user card names the node. The server's receipt names the actual sender, so it counts for nothing and cannot settle the run, even with no state file (3, 10).

  5. A sandboxed agent asks to start a run. MCP withholds run, which starts manual runs only on the agent machine. It refuses before any program runs (tools).

Across hosts

  1. A run for a build host, sent from the browser. stone posts a run for lab-1 in the server's forum. lab-1 pulls it, and asks the server with OCX's receipt: stone sent it, and holds setup there. lab-1 starts it, and its started and ended reach the server as received from lab-1 (senders).

  2. The server cannot be reached. A failed pull publishes no completion, and the daemon decides nothing from it. A failed pull, or a failed receipt request while the daemon decides runs, ends it, logging the failure. On the server, the next request to OCX's web page starts it again if no daemon holds its lock (scripts/confine.cr); on another agent machine nothing starts it again until someone does, as by restarting ocx-node's container. It pulls first; existing watchers run on, giving nothing from a failed receipt read until another completion (nodes).

  3. A run written on the build host itself. A run written straight into the node's clone has no server receipt, so the daemon ignores it. The daemon pulls, never pushes runs. If someone later pushes it as the node user, a daemon restart reads it and the watcher refuses it. Send runs to the server as their actual sender (senders).

  4. A program on a build host speaks. A session on lab-1 posts its progress with OCX's post, which writes it in the run's own clone. It reaches the server as received from the run's user, so a session elsewhere whose hears names that user is given it, and the server never takes it for lab-1's.

  5. A build host goes down around a launch. Its watcher writes taken before posting started and launches only after Fossil redirects to the post it stored. State survives reboot, the lock does not: the next watcher finding taken and the lock free takes the lock, ends what the old lock names and posts lost, never relaunching. If the state file was lost, a started confirmed by the server still prevents launch. An ended the server's receipt page says the node user sent settles a run without a state file, however old the report (10). For a run naming a clone, only an ended whose clone starts with the run's counts (protocol v1, section 3).

  6. A post whose sender is not known yet. A failed receipt read gives nothing from that pull; at the next completion number the watcher reads the same pull again. An artifact the server says it has not received, such as a post written straight into the clone, is given to no session, while those after it are given. If it reaches the server later it is given then, in the server's order; nothing already given is repeated (9).

  7. A run names a node user as its user. An admin sends a node a run whose user is lab-2, the node user of a build host, named as the node of runs an admin sent. The node refuses it, so it never lets a program write lab-2's reports (11, row 4).

  8. Two runs for one user. Two runs for the same node both act as builder. The node starts the first; the second waits, unstarted, until the first ends and a later daemon round reconsiders it (nodes).

  9. The node user of a build host passes a check-in. lab-1's host is taken over, and writes an audited pass on a check-in as lab-1. lab-1 is a node user, so its audited verdict never counts, and its ci verdict counts only on the check-in a build sent to lab-1 took, against that build (verdicts).

A workspace removed

  1. The process leaves a child running. A run whose directory step has "remove_on": "saved" runs sh -c '(sleep 30; echo late > late.txt) &', and sh exits 0 at once. The run's sandbox ends with the node, so the watcher, made the parent of what its process left, waits until the sandbox's init has ended, and every process of the sandbox with it, before it reads the workspace; late.txt is never written, and ended, exited 0, says workspace removed (5).
  2. The run ends with work the server lacks. The same run instead writes made.txt in its checkout, or commits it without a push. The watcher removes nothing: ended says workspace kept: EXTRA made.txt, or kept: 2 artifacts not sent, and the workspace stays on the node (5). On a node that does not run as root, a directory the process left that the node's account may not read cannot be emptied: ended says kept: what could not be removed: sub, never removed while anything is left.

A run not read

  1. The program never reads its run. A run's command is sh -c 'exit 3', and its objective is longer than the pipe of the program's standard input holds. The program exits before reading its first line, so the node never finishes writing it. What a program reads is its own to decide: that is no failure, and ended says exited 3. A session's program, at a terminal or not, ends the same way, and a session each of whose deliveries starts a process waits for its next delivery (8.1, 10).
  2. The program leaves its terminal full. A session whose terminal is true runs sh -c 'sleep 1; exit 3', and is given a post of 100 000 bytes while it sleeps. Its run's line waits on the terminal, read by no one, so the terminal fills and the node waits to type the rest. The program exits, and the node stops typing: ended says exited 3. A stop that comes meanwhile ends it stopped, and a TERM with no stop, lost, as for any process (8.1, 10).

The watcher's own failures

  1. The program takes a kept log's place. A run's program makes a directory named stderr.log in its workspace, prints oops on its STDERR and exits 3. The node cannot write its kept log there, so that file is not kept and the directory stays; the kept logs are posted where the run is, and ended says exited 3 (8.2).
  2. The watcher fails after started. A step of the node's own fails while the run is in its care, as a write on a full disk. The node ends the run's process, if it still runs, as a stop does, and once it has ended writes ended, lost, its note naming the failure, as it writes budget or stopped; a workspace to be removed it keeps, unread, ended saying kept: what it holds was not read, as its watcher failed. The failure, whole, goes to the run's log, and the user's next run is reconsidered at a later daemon round. An unknown post outcome waits for another completion; ending holds the exact report for the next watcher of the run to post, as in case 2 (10, 12).

What a program leaves running

  1. An unconfined program leaves a child running. On a node that runs processes unconfined, a run with a wall of 6s runs sh -c 'sleep 40 & exit 3', and sh exits at once, its child holding its STDOUT. The program started in a session of its own, so its child is in that session: in the program's process group, or in one of its own, where a shell's job control (set -m, or any shell at a terminal), setpgid, Python's os.setpgrp or GNU timeout puts it. Once the program has ended, the node ends every group of the session that holds what it left, as a sandbox ends with its init (case 28), and writes ended only once each process has ended: exited 3 at once, the child gone, whether it held the program's output or sent its own elsewhere, so no next run of the same user starts under it (case 26). At a terminal the same, a child that ignores the hang-up too. A stop sends TERM to the program's group and to each group of the session holding what it left, and a job the program still waits for ends with KILL once the program has ended; past the wall, timeout ends the program, and the node the rest. A watcher that has gone, as when it is killed, ends none of it: the next watcher finding taken and the run lock free takes the lock, ends what the old lock names and posts lost. It ends every group of the program's session, whether the program still runs or has ended, as at a terminal, whose hang-up ends it as the watcher goes (nodes). What remains: a process that starts a session of its own, as a daemon does with setsid, leaves the program's session, and the node does not end it; only a sandbox holds it (7, 10).

A writer outside OCX

  1. A writer outside OCX holds the server repository. The watcher posts through Fossil's pages. If Fossil does not redirect to the post it stored, it waits for another completed pull, checks whether the same text was received from its node user, and retries only if absent. Other runs can decide while it waits; it holds only its run lock. Verdicts and shares still use the clone and named remote. Tool writes on repository files retain Fossil's bounded lock waits (nodes).

What a sandbox leaves running

  1. A sandbox that does not end with the node is stopped. A run whose sandbox has ends_with_node false runs sleep 40, and is stopped. The stop ends bubblewrap's outer process, which without --die-with-parent leaves the sandbox's init, and the program under it, holding the program's output. The node, made the parent of that init, ends it once the outer process has ended, and the kernel every process of the sandbox with it: ended says stopped at once; past a wall of 2s, budget at once; and once the program exits by itself, nothing it left in the sandbox runs on. ends_with_node false says only that the sandbox runs on when the node's process that watches it ends, as when that process is killed, until the next watcher finds taken and the run lock free, takes the lock, ends what the old lock names and posts lost. It ends the sandbox with the run's process groups: while bubblewrap's outer process runs, the sandbox's init as that process's child, in a session of its own (own_session) and so in no group of the run's; once the program's end has ended that process, with the group the init is in, the watcher's or timeout's (nodes). What remains: a sandbox in a session of its own whose outer process has ended too, at its program's end or its wall, before the next watcher reads the old lock: nothing its run's lock names leads to its init, and it runs on (6, 10).

Daemon and completion failures

  1. Chat fails or a pull hangs. A failed chat request ends the daemon; a pull past ten minutes is killed, publishes no completion and ends it too. It is started again as in case 20, and it pulls before waiting again.
  2. The daemon restarts while watchers run. Detached watchers retain their run locks. The restarted daemon reads everything once and leaves those runs in care. It settles done and confirmed ended runs; waiting runs remain candidates at every complete pull (10).
  3. A post's answer is lost. At the next completion the watcher confirms sender, text and reply target before retrying, so a stored report is not posted twice. A failed receipt read waits for another completion. A stop confirmed while started was unknown prevents launch and ends the run stopped (nodes).

  4. The agent machine reboots. Run state and completion files survive; locks do not. The daemon pulls whole first. The next watcher takes the free run lock: with taken, it checks the old lock's boot, ends what it names from that boot and posts lost; with ending, it retries the exact report; with done, it does nothing. A malformed state counts as taken; a forged user card proves no report's sender (10).

  5. A verdict is requested without i. The watcher refuses the run before launch. A share without y is reported as not shared, as before (11).