Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
| Comment: | An import pipes git's stream straight into Fossil, never a file, each step's largest file within a third of the space free; from the web page it goes on detached, answering at once, a TERM stopping it as a failure, its outcome logged where status shows it with the imports running and those that ended unreleased; two imports of one name never meet, and taking a directory never waits on a fossil command
Intent: the maintainer, 2026-10-21: "My `import` of https://github.com/beagleboard/beagleboard-org keeps failing. The post times out, so I don't know what happened." The tool's purpose, slice 39's: "comfort that i can import projects and reliably serve development back to git with ocx2". Reproduced on this host, the same code at the command line: the upstream, 3.0 GB packed with 527,641 commits, stopped after 10 minutes at import_branches, "git in the import's sandbox stopped by its files limit (SIGXFSZ)": the whole fast-export stream was written to a file, past the 8G files limit; and the page ran the whole import inside its request, so the browser gave up and the person never learned the outcome. An adversarial review of the change as first built (uncommitted on 22d4decadd) found seven defects, each fixed here: 1. a detached import that TERM or KILL ended logged no outcome, and status then showed nothing of it; 2. the 64 GB limit on each file alone could not stop a runaway before the disk filled; 3. the web page waited with no limit on the new repository's turn, which fossil git export held for two minutes in the proof; 4. a failed import left NAME.fossil-ocx.lock beside the served repository; 5. a detached import refused at check_import or read_served_setup left its directory, its job.json with any token in the upstream's URL; 6. Fossil's import took its turn from a checkout above its directory, failing at a .fslckout that did not read; 7. HISTORY.md and docs/interfaces.md gave the detached import's argument as its job, not its directory. Expression: src/ocx/commands/import.cr: ImportBranches runs git's fast-export and `fossil import --git --no-vacuum` side by side, each in a sandbox of its own, through an unnamed pipe, each end handed to its task, which closes it in this process as the task starts (src/ocx/launch.cr, src/ocx/tasks.cr), so either task's end ends the other's; fossil's failure is kept as a value, so git's is always received, and the import says both, git's first, leaving out git's SIGPIPE when fossil failed (Git::Failed keeps the signal, src/ocx/git.cr). Each task's limits (ImportCommand.limits): 48 hours of wall clock and as much CPU time, and a largest file of FILES, 24 GB, about twice the 10.6 GB repository the proof made, or a third of the space free where it works as it starts, when that is less, read by statvfs (Tasks.free, src/ocx/tasks.cr), so the new repository and its journal, or the clone, leave a third free (finding 2). TakeWork takes the directory first and answers whether it took it now; ImportCommand.run releases it only then, an import the web page started releasing it itself once its outcome is logged. From the web page (StartImport) the release's program starts detached, by setsid (src/ocx/session.cr, answering the process), as `ocx2 --import-detached WORK` (src/ocx/entry.cr), WORK's job.json, 0600, holding the served repository, the user and the input. src/ocx/imports.cr, new: Imported, with unmade! and discard (moved from ImportCommand.discard); the directory `.NAME.import` named `PID @TICKS`; Imports.take, Imports.release and Imports.standing hold the imports' lock, an flock on the served repository's directory itself (Imports.holding), exclusive to take or release, shared to read, never a fossil command, so no taking waits on one and no lock file is left (findings 3 and 4); release removes the directory only while it names the calling process; running reads a process as importing only while /proc shows it at that number, with that start, not ended; standing gives the imports running, `left`, each directory whose process ended without releasing it, as KILL leaves one (finding 1), and the log's last outcome. src/ocx/imports_detached.cr, new, split from imports.cr to keep both within 400 lines: the Job, Outcome and Detached elements, StartImport, Imports.start, and Imports.detached, which traps TERM for an Ending: Ending#ask stops the import's log (Log#stop, src/ocx/log.cr: each operation after is refused, "OP was not begun: SIGTERM asked the import to end") and sends TERM to each child not yet waited for (Sandbox.signal_children, src/ocx/sandbox.cr), so the step running fails, its sandbox with it, under GNU timeout too, and the import stops as at any failure, discarding what it made (finding 1); a failure no operation refused, as a File::Error opening a repository's lock, which Entry.identified, before the answer, does not take for a refusal, is logged as the outcome too, its backtrace on standard error; then, however it stopped, refused at its first checks too, it releases its directory and job (finding 5). src/ocx/command.cr: a tool's answer takes the log its caller holds. src/ocx/fossil/sync.cr: import_git names the new repository by `on:`, as clone does, so its turn is that file's, never a checkout's (finding 6); src/ocx/fossil.cr: a sandboxed fossil reads a pipe's end, and its failure says how it ended. src/ocx/commands/status.cr: ReadImports, for whoever may import, in a release root. src/ocx/fossil/direct.cr: the export's state seeded in one transaction that waits on a reader (33 minutes for 399,783 check-ins before, 9 s for 527,641 now). src/ocx/hook.cr: Hook.last reads the last line of a log of its form; src/ocx/care.cr: Node.stat_of public. HISTORY.md decision 120, the next free number; USAGE.md's import section and row; docs/interfaces.md's import row (finding 7); docs/slices-git.md; docs/linux.md; docs/backlog.md, two items: an old check-in of so long a history reads in minutes, and an upstream's URL shows on git clone's command line while the clone runs; docs/fossil.md, over 400 lines with this change's points, its "Git export and import" moved whole to the new docs/fossil-git.md, named in knowledge-map.json; docs/survey/functionality.json; docs/design/modules.json and the modules.md and knowledge-map.md generated. The specs' git helpers moved unchanged to spec/support/upstreams.cr, and the detached examples to spec/ocx/imports_detached_spec.cr. Every file within precept C1: src/ocx/commands/import.cr 391 lines, src/ocx/hook.cr 400, USAGE.md 400. Review: Gemma (gemma4:31b) read the change ten times (scratchpad gemma-import-large-land-1.txt to -10.txt, requests and replies in gemma-reviews/import-large-1 and import-large-land-2 to -10). Held: the third reading's, that "left" was said of imports KILL ended, or that logged no outcome, while it is any directory whose process ended without releasing it, its outcome logged or not: status's help, USAGE.md, HISTORY.md and the module docs now say so. The fourth's, that an exception no operation turned into a refusal escaped Imports.detached unlogged: as a File::Error opening the served repository's lock, raised in Entry.identified before the answer, whose capabilities rescue Fossil::Failed alone; it is now logged as the outcome, and a spec holds it. Misread, each a clarity finding, the code changed and asked again: the first's, that a failed start left the directory (take's comment now says a failing block leaves none, and only the named process releases it); that ImportBranches left git's result unreceived when fossil failed (the call now says fossil's failure is a value, git's always received); that release missed a zombie (its comment now says the importing process releases its own directory); that USAGE.md put the outcome in NAME.log (reworded). The third's and fourth's, that the started process writes `importer`, so a taking or status could look between: the line naming it now says this taking writes it before letting go of the lock that every taking and standing hold; not raised again. The fifth's, that a process dead at once, or one that took its number, reads as running: running's checks are now three lines, each named; not raised again. The sixth's, that `answer` is undefined after the rescue: Crystal makes it nil, as the spec of the jammed lock runs, and it is now declared nil first; not raised again. The seventh's, that the backtrace goes nowhere: the doc now says standard error is the same output file. Wrong: the second's, a directory KILL left lingering until the next import of its name: by design, status lists it as left, and OCX keeps no sweep or timer (decision 39; causation precept); its job.json, 0600 in a 0700 directory, holds no more than the clone's and the export's git configuration already keep. The third's, that a raise skips the TERM reset (ensure runs), and that KILL bypasses discard (no code runs on KILL; status lists the directory). The fourth's, that git's signal might not compare (Ended.signal is the name, "PIPE", as Signal::PIPE.to_s). The fifth's, a job.json corrupted by hand: the import then logs nothing and its directory stays, which status lists as left. The seventh's, that status should report a KILL'd import's outcome: `left` is that report. The eighth reading weighed several and concluded it found nothing. The ninth and tenth, after the specs' fixes below, the first's zombie again: that release, called by the importing process itself, may find it ended in /proc and leave its directory; a process calling release runs and has not ended, so running reads it whenever the directory names it, as a comment there now says, and the check is what keeps a late release from removing a directory taken since for another process, while take clears one whose process has ended; the tenth also had take refuse a left directory, which it clears. Demonstrates: spec/ocx/import_spec.cr, "imports a stream larger than its tasks' files limit, git's stream piped into Fossil and never written to a file": as before, now in a directory below one holding an empty .fslckout, the directory then holding its turn's made-ocx.lock; with import_git's `on:` taken out it fails ".fslckout could not be read: no such table: vvar" (tried); ImportCommand.limits gives 10 GiB of 30 GiB free and FILES of 300 GiB, and Tasks.free reads the space free and refuses a directory not there. "takes an import's directory for one process at a time, under the imports' lock alone, and anew once that one has ended, which status shows": a take while another holder keeps the new repository's turn, as a fossil command of the import does (the take as first built took that turn itself, so it waited as long as the holder held it: the review's run, 7.7 s for a holder of 8 s), release by a process not named leaving the directory, standing showing the running sleep, a process named with its start that has ended and is not yet waited for read as not running, then once it ended `left` [work], cleared by the next take, and no lock file left. "makes a served repository whose export keeps the upstream's commits..." and "leaves nothing of an import that fails..." pass as before, the latter through Imported#discard. spec/ocx/imports_detached_spec.cr, "starts an import from the web page detached, answering at once, refuses another of its name meanwhile, and logs its outcome, which status shows": as before, plus the stand-in program's directory shown as left once it ended unreleased; the failed import leaves no lost.db-ocx.lock and nothing left; a detached import refused at check_import, its repository made meanwhile, releases the directory taken for it; one whose served repository's lock is a directory, so that opening it raises a File::Error no operation refuses, logs "File::Error: ..." as its outcome, its backtrace on standard error, and releases its directory. "ends a detached import that TERM asks to end as at a failure: its outcome logged, its sandbox ended, its directory released": git's clone of an upstream that accepts and never answers, a TERM sent to this process once the upstream has its connection, which the import's trap hears: exit 1, the outcome "git in the import's sandbox ended by SIGTERM...", the directory gone; and one whose Ending was asked before its first operation logs "check_import was not begun: SIGTERM asked the import to end". spec/ocx/fossil_spec.cr: a sandboxed fossil's failure says it exited 1, and only a sandbox reads a pipe. spec/support/upstreams.cr: the helpers, moved. spec/guards/size_spec.cr, spec/guards/derivation_spec.cr, spec/guards/documents_spec.cr, spec/guards/delineation_spec.cr and spec/guards/survey_spec.cr hold the documents, docs/fossil-git.md among them. These hold src/ocx/imports.cr, src/ocx/imports_detached.cr, src/ocx/commands/import.cr, src/ocx/commands/status.cr, src/ocx/log.cr, src/ocx/command.cr, src/ocx/sandbox.cr, src/ocx/tasks.cr, src/ocx/launch.cr, src/ocx/session.cr, src/ocx/entry.cr, src/ocx/git.cr, src/ocx/fossil.cr, src/ocx/fossil/sync.cr, src/ocx/fossil/direct.cr, src/ocx/hook.cr and src/ocx/care.cr. The proof, on this host, of the change as first built, before these fixes: beagleboard-org posted as setup to a scratch served repository's web page, niced, answered in 48 ms; 14 h 13 min later the import log said ok and status showed it: 527,641 check-ins, main branch master, in the login group, a 10.6 GB repository, its export's refs the upstream's own commits, fossil git status up-to-date; the clone 7 minutes, git's fast-export 10 h 5 min and 11 minutes of CPU for a 50.6 GB stream, Fossil's import 14 h 4 min and 13 h 17 min of CPU. The fixes were not run on that upstream again. The gate's first run failed its coverage ratchet, src/ocx/imports.cr missing 1 statement and src/ocx/imports_detached.cr 3: running's check of an ended process, the TERM trap's block, and the rescue of a failure no operation refused, whose first spec, a served repository gone, reached a refusal instead, as Entry.identified rescues Fossil::Failed; the specs above now run each. crystal run ci.cr green: 506 examples, 0 failures. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QVpmCsBwVF3d4zhMHHDAxs |
|---|---|
| Downloads: | Tarball | ZIP archive |
| Timelines: | family | ancestors | descendants | both | agent-tools |
| Files: | files | file ages | folders |
| SHA3-256: |
905756db67305b6c66adc58bc8b521e1 |
| User & Date: | opus-ocx2 2026-10-23 10:55:22.846 |
|
2026-10-23
| ||
| 11:57 |
A watcher that decides its run waits hands that run on itself once it has let go of its locks, so a run from a user holding R alone that the run it waited for read in the node's care, its watcher still holding it, is not left waiting
Intent: an adversarial review of 3869da228e, which has a node take one run from a user holding R alone at a time, found a run left waiting, unstarted, with no watcher to hand it on, shown by a spec. Two such runs, X and Y, of different process.user, are handed to watchers together; X's watcher takes the node's lock of such runs and X runs; Y's watcher decides Y waits, then, still holding Y's run lock, takes the node's lock again to write its answer, which Crystal's blocking flock takes by trying again every tenth of a second with no queue, so X's watcher, X ending meanwhile, can take it first. X's following then read Y in the node's care and handed on nothing, and Y's own following left Y out, so Y waited until the node's deliberate check, since a round decides only the runs that arrived (HISTORY.md decision 39). Expression: src/ocx/watcher.cr: Watched gains waited, false unless the watcher decided its run waits; decided returns waited: decision.act.wait? when the run is not started, and says a run that waits is written no answer; watched passes watched.waited to following; following takes the watcher's own run among those it may hand on when waited, `(waited || run.said.artifact_hash != hash)`, so of X's and Y's followings, each run under the node's lock once its own watcher has let go of its locks, the second reads both free and hands Y on when Y is to start; should both hand it on, Y's run lock takes one watcher, the other writing nothing. Its comments say so: following's, reflowed, that only Y's lock kept it in the node's care; watched's, that the locks are let go as the watcher's process dies, killed too. spec/support/held_watcher.cr, new: a seam that reopens OCX::Watcher and, through previous_def, holds the next watcher of the run held_run once decided has returned, before watched lets go of its locks, until told to go on, so the spec puts Y's watcher where the review found it, in one process, with nothing timed. spec/ocx/watcher_spec.cr: the setup of the example of runs from a user holding R alone moved, unchanged, into the helpers build and alone, which the new example shares. HISTORY.md decision 121, amending decision 108 and naming the review; GLOSSARY.md's Watcher: it hands on its own run when it found that run waits. Within precept C1 (File Boundedness): src/ocx/watcher.cr 400 lines, spec/ocx/watcher_spec.cr 399. Review: Gemma (gemma4:31b) read the change nine times (scratchpad gemma-alone-race-1.txt to -9.txt, requests and replies in gemma-reviews/alone-race-1 to -9), the first three given the full intent, the rest a shorter one asking for every interleaving of the two watchers' steps. The first three found nothing. Held: none. Misread, each a clarity finding, the code changed and asked again: the fourth's, that a killed watcher's lock stays held: watched's comment now says its locks are let go as its process dies, and the ninth reading said so itself. The sixth's, that the watcher a waiting run is handed to finds it still in the node's care, a waiting run being kept so: following's comment now says only the run's lock kept it in care, decided's that a run that waits is written no answer, and the new example now runs the watcher it is handed to, which ends it exited 0; the seventh and ninth still spoke in passing of a waiting run kept in care, which nothing keeps: a run that waits is written nothing, and its care is read from its run's lock alone (Node.care). The seventh's, that two watchers handed one run both start it: following's comment now says its run's lock takes one watcher, the other writing nothing, as the example of a second watcher leaving the first's lock as it is shows; not raised again. Wrong: the fourth's, fifth's, eighth's and ninth's, that a watcher killed after deciding its run waits, before its following, strands the run: the kernel lets go of a dead process's locks, so the following of the run it waited for, in each order given there after the death, reads the run free and hands it on; only a death after that following read it in care leaves it, which the node's deliberate check mends, as it does any watcher's death between its steps (decision 39; docs/protocol-v1-failures.md, case 3), with no sweep or timer (the causation precept), and an ensure, as the fourth proposed, does not run on KILL. The seventh's, as a defect: a run handed on twice is by design, as decision 108 had it. Demonstrates: spec/ocx/watcher_spec.cr, "hands on a run from a user holding R alone that waited by its own watcher when the run it waited for ended while that watcher held it": X, of builder, runs until the file go is in its workspace; Y's watcher, of coder, held by spec/support/held_watcher.cr once it decided Y waits, Y reads in the node's care; go is written and X's watcher returns, its next_runs empty, its following having read Y in care; Y's watcher goes on and returns next_runs [Y]; the watcher Y is handed to ends it exited 0. With following's test reverted to `run.said.artifact_hash != hash` the example fails, "Expected: [Y] got: []" (tried). "takes one run from a user holding R alone at a time..." passes as before through the helpers. src/ocx/watcher.cr holds the change; spec/guards/size_spec.cr holds src/ocx/watcher.cr at 400 lines and spec/ocx/watcher_spec.cr at 399; spec/guards/documents_spec.cr renders HISTORY.md and GLOSSARY.md with their links. crystal run ci.cr green: 507 examples, 0 failures. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QVpmCsBwVF3d4zhMHHDAxs check-in: 488aecd89f user: opus-ocx2 tags: agent-tools | |
| 10:55 |
An import pipes git's stream straight into Fossil, never a file, each step's largest file within a third of the space free; from the web page it goes on detached, answering at once, a TERM stopping it as a failure, its outcome logged where status shows it with the imports running and those that ended unreleased; two imports of one name never meet, and taking a directory never waits on a fossil command
Intent: the maintainer, 2026-10-21: "My `import` of https://github.com/beagleboard/beagleboard-org keeps failing. The post times out, so I don't know what happened." The tool's purpose, slice 39's: "comfort that i can import projects and reliably serve development back to git with ocx2". Reproduced on this host, the same code at the command line: the upstream, 3.0 GB packed with 527,641 commits, stopped after 10 minutes at import_branches, "git in the import's sandbox stopped by its files limit (SIGXFSZ)": the whole fast-export stream was written to a file, past the 8G files limit; and the page ran the whole import inside its request, so the browser gave up and the person never learned the outcome. An adversarial review of the change as first built (uncommitted on 22d4decadd) found seven defects, each fixed here: 1. a detached import that TERM or KILL ended logged no outcome, and status then showed nothing of it; 2. the 64 GB limit on each file alone could not stop a runaway before the disk filled; 3. the web page waited with no limit on the new repository's turn, which fossil git export held for two minutes in the proof; 4. a failed import left NAME.fossil-ocx.lock beside the served repository; 5. a detached import refused at check_import or read_served_setup left its directory, its job.json with any token in the upstream's URL; 6. Fossil's import took its turn from a checkout above its directory, failing at a .fslckout that did not read; 7. HISTORY.md and docs/interfaces.md gave the detached import's argument as its job, not its directory. Expression: src/ocx/commands/import.cr: ImportBranches runs git's fast-export and `fossil import --git --no-vacuum` side by side, each in a sandbox of its own, through an unnamed pipe, each end handed to its task, which closes it in this process as the task starts (src/ocx/launch.cr, src/ocx/tasks.cr), so either task's end ends the other's; fossil's failure is kept as a value, so git's is always received, and the import says both, git's first, leaving out git's SIGPIPE when fossil failed (Git::Failed keeps the signal, src/ocx/git.cr). Each task's limits (ImportCommand.limits): 48 hours of wall clock and as much CPU time, and a largest file of FILES, 24 GB, about twice the 10.6 GB repository the proof made, or a third of the space free where it works as it starts, when that is less, read by statvfs (Tasks.free, src/ocx/tasks.cr), so the new repository and its journal, or the clone, leave a third free (finding 2). TakeWork takes the directory first and answers whether it took it now; ImportCommand.run releases it only then, an import the web page started releasing it itself once its outcome is logged. From the web page (StartImport) the release's program starts detached, by setsid (src/ocx/session.cr, answering the process), as `ocx2 --import-detached WORK` (src/ocx/entry.cr), WORK's job.json, 0600, holding the served repository, the user and the input. src/ocx/imports.cr, new: Imported, with unmade! and discard (moved from ImportCommand.discard); the directory `.NAME.import` named `PID @TICKS`; Imports.take, Imports.release and Imports.standing hold the imports' lock, an flock on the served repository's directory itself (Imports.holding), exclusive to take or release, shared to read, never a fossil command, so no taking waits on one and no lock file is left (findings 3 and 4); release removes the directory only while it names the calling process; running reads a process as importing only while /proc shows it at that number, with that start, not ended; standing gives the imports running, `left`, each directory whose process ended without releasing it, as KILL leaves one (finding 1), and the log's last outcome. src/ocx/imports_detached.cr, new, split from imports.cr to keep both within 400 lines: the Job, Outcome and Detached elements, StartImport, Imports.start, and Imports.detached, which traps TERM for an Ending: Ending#ask stops the import's log (Log#stop, src/ocx/log.cr: each operation after is refused, "OP was not begun: SIGTERM asked the import to end") and sends TERM to each child not yet waited for (Sandbox.signal_children, src/ocx/sandbox.cr), so the step running fails, its sandbox with it, under GNU timeout too, and the import stops as at any failure, discarding what it made (finding 1); a failure no operation refused, as a File::Error opening a repository's lock, which Entry.identified, before the answer, does not take for a refusal, is logged as the outcome too, its backtrace on standard error; then, however it stopped, refused at its first checks too, it releases its directory and job (finding 5). src/ocx/command.cr: a tool's answer takes the log its caller holds. src/ocx/fossil/sync.cr: import_git names the new repository by `on:`, as clone does, so its turn is that file's, never a checkout's (finding 6); src/ocx/fossil.cr: a sandboxed fossil reads a pipe's end, and its failure says how it ended. src/ocx/commands/status.cr: ReadImports, for whoever may import, in a release root. src/ocx/fossil/direct.cr: the export's state seeded in one transaction that waits on a reader (33 minutes for 399,783 check-ins before, 9 s for 527,641 now). src/ocx/hook.cr: Hook.last reads the last line of a log of its form; src/ocx/care.cr: Node.stat_of public. HISTORY.md decision 120, the next free number; USAGE.md's import section and row; docs/interfaces.md's import row (finding 7); docs/slices-git.md; docs/linux.md; docs/backlog.md, two items: an old check-in of so long a history reads in minutes, and an upstream's URL shows on git clone's command line while the clone runs; docs/fossil.md, over 400 lines with this change's points, its "Git export and import" moved whole to the new docs/fossil-git.md, named in knowledge-map.json; docs/survey/functionality.json; docs/design/modules.json and the modules.md and knowledge-map.md generated. The specs' git helpers moved unchanged to spec/support/upstreams.cr, and the detached examples to spec/ocx/imports_detached_spec.cr. Every file within precept C1: src/ocx/commands/import.cr 391 lines, src/ocx/hook.cr 400, USAGE.md 400. Review: Gemma (gemma4:31b) read the change ten times (scratchpad gemma-import-large-land-1.txt to -10.txt, requests and replies in gemma-reviews/import-large-1 and import-large-land-2 to -10). Held: the third reading's, that "left" was said of imports KILL ended, or that logged no outcome, while it is any directory whose process ended without releasing it, its outcome logged or not: status's help, USAGE.md, HISTORY.md and the module docs now say so. The fourth's, that an exception no operation turned into a refusal escaped Imports.detached unlogged: as a File::Error opening the served repository's lock, raised in Entry.identified before the answer, whose capabilities rescue Fossil::Failed alone; it is now logged as the outcome, and a spec holds it. Misread, each a clarity finding, the code changed and asked again: the first's, that a failed start left the directory (take's comment now says a failing block leaves none, and only the named process releases it); that ImportBranches left git's result unreceived when fossil failed (the call now says fossil's failure is a value, git's always received); that release missed a zombie (its comment now says the importing process releases its own directory); that USAGE.md put the outcome in NAME.log (reworded). The third's and fourth's, that the started process writes `importer`, so a taking or status could look between: the line naming it now says this taking writes it before letting go of the lock that every taking and standing hold; not raised again. The fifth's, that a process dead at once, or one that took its number, reads as running: running's checks are now three lines, each named; not raised again. The sixth's, that `answer` is undefined after the rescue: Crystal makes it nil, as the spec of the jammed lock runs, and it is now declared nil first; not raised again. The seventh's, that the backtrace goes nowhere: the doc now says standard error is the same output file. Wrong: the second's, a directory KILL left lingering until the next import of its name: by design, status lists it as left, and OCX keeps no sweep or timer (decision 39; causation precept); its job.json, 0600 in a 0700 directory, holds no more than the clone's and the export's git configuration already keep. The third's, that a raise skips the TERM reset (ensure runs), and that KILL bypasses discard (no code runs on KILL; status lists the directory). The fourth's, that git's signal might not compare (Ended.signal is the name, "PIPE", as Signal::PIPE.to_s). The fifth's, a job.json corrupted by hand: the import then logs nothing and its directory stays, which status lists as left. The seventh's, that status should report a KILL'd import's outcome: `left` is that report. The eighth reading weighed several and concluded it found nothing. The ninth and tenth, after the specs' fixes below, the first's zombie again: that release, called by the importing process itself, may find it ended in /proc and leave its directory; a process calling release runs and has not ended, so running reads it whenever the directory names it, as a comment there now says, and the check is what keeps a late release from removing a directory taken since for another process, while take clears one whose process has ended; the tenth also had take refuse a left directory, which it clears. Demonstrates: spec/ocx/import_spec.cr, "imports a stream larger than its tasks' files limit, git's stream piped into Fossil and never written to a file": as before, now in a directory below one holding an empty .fslckout, the directory then holding its turn's made-ocx.lock; with import_git's `on:` taken out it fails ".fslckout could not be read: no such table: vvar" (tried); ImportCommand.limits gives 10 GiB of 30 GiB free and FILES of 300 GiB, and Tasks.free reads the space free and refuses a directory not there. "takes an import's directory for one process at a time, under the imports' lock alone, and anew once that one has ended, which status shows": a take while another holder keeps the new repository's turn, as a fossil command of the import does (the take as first built took that turn itself, so it waited as long as the holder held it: the review's run, 7.7 s for a holder of 8 s), release by a process not named leaving the directory, standing showing the running sleep, a process named with its start that has ended and is not yet waited for read as not running, then once it ended `left` [work], cleared by the next take, and no lock file left. "makes a served repository whose export keeps the upstream's commits..." and "leaves nothing of an import that fails..." pass as before, the latter through Imported#discard. spec/ocx/imports_detached_spec.cr, "starts an import from the web page detached, answering at once, refuses another of its name meanwhile, and logs its outcome, which status shows": as before, plus the stand-in program's directory shown as left once it ended unreleased; the failed import leaves no lost.db-ocx.lock and nothing left; a detached import refused at check_import, its repository made meanwhile, releases the directory taken for it; one whose served repository's lock is a directory, so that opening it raises a File::Error no operation refuses, logs "File::Error: ..." as its outcome, its backtrace on standard error, and releases its directory. "ends a detached import that TERM asks to end as at a failure: its outcome logged, its sandbox ended, its directory released": git's clone of an upstream that accepts and never answers, a TERM sent to this process once the upstream has its connection, which the import's trap hears: exit 1, the outcome "git in the import's sandbox ended by SIGTERM...", the directory gone; and one whose Ending was asked before its first operation logs "check_import was not begun: SIGTERM asked the import to end". spec/ocx/fossil_spec.cr: a sandboxed fossil's failure says it exited 1, and only a sandbox reads a pipe. spec/support/upstreams.cr: the helpers, moved. spec/guards/size_spec.cr, spec/guards/derivation_spec.cr, spec/guards/documents_spec.cr, spec/guards/delineation_spec.cr and spec/guards/survey_spec.cr hold the documents, docs/fossil-git.md among them. These hold src/ocx/imports.cr, src/ocx/imports_detached.cr, src/ocx/commands/import.cr, src/ocx/commands/status.cr, src/ocx/log.cr, src/ocx/command.cr, src/ocx/sandbox.cr, src/ocx/tasks.cr, src/ocx/launch.cr, src/ocx/session.cr, src/ocx/entry.cr, src/ocx/git.cr, src/ocx/fossil.cr, src/ocx/fossil/sync.cr, src/ocx/fossil/direct.cr, src/ocx/hook.cr and src/ocx/care.cr. The proof, on this host, of the change as first built, before these fixes: beagleboard-org posted as setup to a scratch served repository's web page, niced, answered in 48 ms; 14 h 13 min later the import log said ok and status showed it: 527,641 check-ins, main branch master, in the login group, a 10.6 GB repository, its export's refs the upstream's own commits, fossil git status up-to-date; the clone 7 minutes, git's fast-export 10 h 5 min and 11 minutes of CPU for a 50.6 GB stream, Fossil's import 14 h 4 min and 13 h 17 min of CPU. The fixes were not run on that upstream again. The gate's first run failed its coverage ratchet, src/ocx/imports.cr missing 1 statement and src/ocx/imports_detached.cr 3: running's check of an ended process, the TERM trap's block, and the rescue of a failure no operation refused, whose first spec, a served repository gone, reached a refusal instead, as Entry.identified rescues Fossil::Failed; the specs above now run each. crystal run ci.cr green: 506 examples, 0 failures. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QVpmCsBwVF3d4zhMHHDAxs check-in: 905756db67 user: opus-ocx2 tags: agent-tools | |
| 09:28 |
The node's check, finding a watcher gone, ends the process group GNU timeout leads in the watcher's session for a walled sandbox, as before aeabbd54e0; and the comments and docs/fossil.md say the repository and a checkout's own database hold a command's start as Fossil's settings do
Intent: protocol v1's promise that a run's ended comes only once its program and all it left have ended (docs/protocol-v1.md, sections 7 and 10; HISTORY.md decision 118), and comments that say what Fossil does. Two review findings, each confirmed. 1. An adversarial review of aeabbd54e0 (scratchpad adv-sg6: gtimeout.c, endleft_new.cr, endleft_old.cr, endleft_fix.cr, out-endleft.txt): aeabbd54e0 changed Node.end_left's test from the launched process leading its own group to its leading its own session, dropping a process that leads its group but not its session. GNU coreutils timeout, the server host's, calls setpgid(0, 0), so a walled run in a sandbox, started as timeout -s KILL N bwrap ... in the watcher's session, leads its own group, not its session: when its watcher died the check killed nothing and answered the run lost while timeout, bubblewrap and the sandbox ran on to the wall, and the user's next run, or a second run from a user holding R alone, could start beside it (the probe: 1 s after the check, timeout and the sandbox's sleep still ran; with the fix, gone; under BusyBox timeout, which becomes bubblewrap in the watcher's group, bubblewrap, the watcher's own child, ends with it by --die-with-parent, its sandbox with it, either way). 2. A review of 31c712a101 (scratchpad rv31: hold.sh, batch.sh, m/): the comments of Fossil::LET_GO, Fossil::TRIES and Place#hosted, and docs/fossil.md, blamed a slow start of a command, or of OCX's wait, on Fossil's settings alone, while the repository and a checkout's own database each hold a command's start the same way. Expression: src/ocx/care.cr: Node.end_left, past the watcher's group, when the launched process started when the run's lock says, ends every process group of the session it leads, or else the group it leads, by its own number: session == launched ? groups_of(launched) : ([launched] if group == launched). Its comment now names the three places the node's own launcher can be in (Watcher.launched): bubblewrap, or BusyBox timeout become bubblewrap, in the watcher's group; setsid, for an unconfined run or one at a terminal, leading a session; GNU timeout leading a group in the watcher's session. docs/protocol-v1-fossil.md, section 5, gets back "or its own group, for one that leads one, as GNU timeout does"; docs/design/modules.json says so of care.cr, and docs/design/modules.md is generated from it. src/ocx/fossil.cr, comments alone: LET_GO says a holder keeping new readers out of the repository, as a writer does at its COMMIT while a reader holds it, fails the wait at its start, in a checkout too, after twice BUSY, its last line "not a valid repository", and one keeping them out of a checkout's own database fails the wait there after three times BUSY; TRIES and Place#hosted give each wait's start up to twice BUSY for the repository or the settings, three times for a checkout's own database. Tried again with Fossil 2.27 (scratchpad gt-trial: trial.sh, trial2.sh, trial3.sh, out-repo.txt, out-repo2.txt, out-co2.txt, out-set.txt), a reader holding the database and a writer of it waiting at its COMMIT: the repository so held, a wait by -R, fossil info, and in its checkout fossil add, fossil changes and the wait there each failed after 30.03 s, last line "not a valid repository", "SQLITE_BUSY(5): database is locked" before it, so Place#hosted still reads it as a holder met; the checkout's own database, its fossil add, fossil changes and wait after 45.08 s, "database is locked" at an ATTACH, a wait by -R at once; the settings, each after 30.04 s, in the checkout too. docs/fossil.md, at 400 lines, folds these into its point "Every fossil command reads what it opens as it starts", and two of its points, on a checkout's own database and on fossil commit --branch, are re-wrapped to width 76, their words unchanged, so it stays at 400 lines. HISTORY.md decision 119, the next free number, names both reviews and amends decision 118. Review: Gemma (gemma4:31b) read the change four times (scratchpad gemma-gnu-timeout-1.txt to -4.txt). Held: none. Misread, each a clarity finding, the code changed and asked again: in the first reading, that a launched process in a group it does not lead, other than the watcher's, is missed, reading the launched process as any program: end_left's comment now says it is the node's own launcher and names the three places one can be in; in the second, that kill(-launched) may miss the group when the group's number differs from the process's, inside a branch taken only when they are equal: the comment over the groups to end now says a group or a session a process leads takes its number from it. Wrong, in the third reading: the first finding again, by a wrapper between the watcher and the launched process that leads its group, proposing to end whatever group the launched process is in; no such wrapper exists, as the launched process is the watcher's own child (Sandbox.start), starting in the watcher's group, and none of the node's launchers, bubblewrap, BusyBox or GNU timeout and setsid, joins a group it does not lead but the watcher's, so the instructed test stands. The fourth reading, of the change as landed, the BusyBox example's reads moved before the watcher's end (below), raised the same and concluded it found nothing that contradicts the intent or fails at run time. Demonstrates: spec/ocx/care_spec.cr, "ends a gone watcher's walled run, its timeout, bubblewrap and sandbox, in the group GNU timeout leads in the watcher's session, only when it started when its lock says": a stand-in for GNU timeout's handling of processes, from its timeout.c, built with cc (pending, saying so, where cc is not on the PATH), as timeout -s KILL 300 bwrap ... sleep 300 under a watcher leading its own session, killed once the sandbox's sleep runs; the stand-in leads its own group in the watcher's session; a lock naming another start ends nothing, the right one ends timeout, bubblewrap, the sandbox's init and its sleep. Against aeabbd54e0's src/ocx/care.cr it fails, "waited 30 s for the walled run ended". "leaves nothing of a gone watcher's walled run under BusyBox timeout, which becomes bubblewrap in the watcher's group, ending with the watcher": the same with BusyBox timeout (pending where busybox is not on the PATH), bubblewrap in the watcher's group, every process of the run ended, on aeabbd54e0 as now. As first built it read bubblewrap's /proc/PID/stat after the watcher's end, and the gate's first run failed there, No such process: bubblewrap, the watcher's own child, has --die-with-parent and ends with it, tried by hand too; the reads now come before the watcher's end. The gate's first run also failed spec/guards/delineation_spec.cr, HISTORY.md naming docs/fossil.md without a link, now a link. These hold src/ocx/care.cr; src/ocx/fossil.cr changes in comments alone, held by the trials named above. spec/guards/size_spec.cr (docs/fossil.md at 400 lines, spec/ocx/care_spec.cr at 241), spec/guards/derivation_spec.cr (docs/design/modules.md as modules.json generates it), spec/guards/documents_spec.cr (each edited document's links) and spec/guards/decisions_spec.cr. crystal run ci.cr green: 502 examples, 0 failures. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QVpmCsBwVF3d4zhMHHDAxs check-in: e06e736748 user: opus-ocx2 tags: agent-tools | |