Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUH8KP8UEDm35iV7mCqbNY Leaf check-in: f88fb54c03 user: x86-1-bq2 tags: seed-ocx-links

F5 on 2a8699a249. In the seed every https clone stopped with "Empty enumerable" (max_by on an empty credential list). Past that, three protocol faults: the want list put "done" before the flush-pkt, so GitHub answered with an empty body; the reader waited for a flush after NAK, but the pack follows it directly; and the pack went to stdout, where the fetch capability expects the helper to store objects itself. New functest case git-https-reaches-network fails on the current seed and checks this offline. Checked in the x86_64 seed with a helper built by the seed's crystal: git clone --bare of jadonk/tt-jadonk-tiny-tapeout, git clone --mirror of crystal-sqlite3 (27 tags, fsck clean) and a refetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUH8KP8UEDm35iV7mCqbNY check-in: 2e1e7bda1b user: x86-1-bq2 tags: seed-ocx-links

Approved by the maintainer on the BQ2 thread, forum post 8278faa17d.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUH8KP8UEDm35iV7mCqbNY check-in: 5b29e9a751 user: x86-1-bq2 tags: seed-ocx-links

Intent: Glint's request (https://ollama.openbeagle.org/bootstrap-qcow2/forumpost/80272781df, updated at 9e46d03c1c): one x86 host builds Glint's aarch64 UEFI image and its Cortex-M0+ target (Thumb, which LLVM builds from its ARM backend) with BQ2's clang and Crystal. Today clang on x86-1 refuses aarch64 triples and Crystal stops with "LLVM was built without ARM target".

Approval: the maintainer, Thursday, November 5: "Sure, we can target a more complete llvm, but please try to keep the size down." Relayed by the cross-session tracker from ocx-console post ebb20e11ad. So only these two backends are added; aarch64 is unchanged (AArch64;ARM).

Change: llvm_targets_to_build gives "X86;ARM;AArch64" on x86_64. Crystal is built against this LLVM in the same seed build, so it gains the same targets.

Checks on x86-1: crystal tool format --check is clean; crystal spec: 210 examples, 0 failures, 5 pending; the targets spec now expects X86;ARM;AArch64 on x86_64. Not yet shown on a rebuilt seed; an x86 seed build from this check-in follows, with its size against the published x86_64 seed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUH8KP8UEDm35iV7mCqbNY check-in: 0b99a242bf user: x86-1-bq2 tags: seed-ocx-links

Intent: the maintainer asked, through x86-1-bq2 (forum post 8f419fcc88), that run 0de68bf272 fix the 3 specs that fail on aarch64.

Expression: - Three examples in spec/sysroot_builder_spec.cr compared against x86_64-bq2-linux-musl paths, but built their plan with SysrootBuilder.new, whose architecture defaults to the host's. On aarch64-1 they failed with the aarch64 triple: - LLVM's two stages: the builtins archive path; - the system crystal's CRYSTAL_CONFIG_TARGET; - the final musl loader path. - Each now passes architecture: "x86_64", as the ldd and soname example already pins aarch64. No source changes.

Demonstrates: - crystal tool format --check is clean. - crystal spec on aarch64-1 at this check-in: 210 examples, 0 failures, 5 pending (before: 3 failures).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUH8KKecLJryWiTGbAfUrh check-in: d9dbd800cf user: aarch64-1-bootstrap-qcow2-session tags: seed-ocx-links

Fixes F6 on ticket 2a8699a249. Plan step 2, which the maintainer approved.

- Every request with a body (GET, -L, -o, POST) failed with "HTTP::Client::Response#body_io cannot be nil": HTTP::Client.exec without a block reads the body into a String. The request now uses the block form and streams the body to stdout or the -o file. - New flags, the ones consumers use (curl -sS -H, -sI, -X POST -H, -sSfL): -s, -S, -f (exit 22 and write no body on HTTP 400 or above), and bundled short flags such as -sSfL. An option's value, such as -d -sS, is not split. - -I now prints the status line, and headers as "Key: value" instead of "Key: [\"value\"]". - Errors print as "curl: ..." unless -s is given without -S.

Checks on x86-1: - spec/curl_command_spec.cr (new) runs against a local HTTP server: GET to -o, -L through a redirect with bundled -sSfL, POST -d, -f on a 404 (exit 22, no file) and without -f (body written), and the flag splitting. - crystal tool format --check is clean; crystal spec: 210 examples, 0 failures, 5 pending. - Built bq2, before and after, against real HTTPS: GET https://www.example.com/ (before: body_io error; after: the page), -sI (before: invalid option; after: HTTP/1.1 200 OK and headers), -sSfL of the shards v0.18.0 tarball from GitHub (after: sha256 46a830afd929..., the same as the aarch64 session's wget download), -sSf of a missing page (after: exit 22).

Still not supported, with no consumer seen: -m -w -O -T -u -k -A --retry --data-binary --version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUH8KP8UEDm35iV7mCqbNY check-in: 8927b666d0 user: x86-1-bq2 tags: seed-ocx-links

Intent: run 0de68bf272, step 4. The BQ2 seed rebuilt from itself was not byte-identical to its parent (forum, 4:58 PM, Wednesday, November 4, Detroit; F9 on ticket 2a8699a249). The maintainer said "ok" to the proposed fix.

Expression: - Cause: rootfs-from-sysroot rebuilds musl in the tree that sysroot-from-seed already built, and musl's Makefile does not track the compiler. The libc.so, libc.a and CRT objects it installs are the ones built by the seed's toolchain. In step 3's tree, lib/libc.so dates from sysroot-from-seed. In an Alpine-built seed, libc.so carries "GCC: (Alpine 15.2.0)", and every linked ELF file differs from the BQ2-built generation's. - Fix: src/sysroot_builder.cr's clean_build_for returns true for musl in rootfs-from-sysroot. The autotools path then runs make distclean before configure.

Demonstrates: - crystal tool format --check is clean. - crystal spec on aarch64-1: 205 examples, 3 failures, 5 pending. The 3 failures are the existing x86_64-triple specs at spec/sysroot_builder_spec.cr:301, :390 and :574, which also fail without this change on an aarch64 host. The new example in spec/sysroot_builder_spec.cr checks that musl is a clean build in rootfs-from-sysroot and not in sysroot-from-seed. - Not yet shown on a rebuilt seed; run 0de68bf272 rebuilds A from Alpine and B from A at this check-in and compares them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUH8KKecLJryWiTGbAfUrh check-in: 8d4e36c6ec user: aarch64-1-bootstrap-qcow2-session tags: seed-ocx-links

Intent: run 0de68bf272 (native aarch64 seed check), step 4, which could not continue. Announced in that run's thread before the change, as its objective requires.

Expression: - Fault: the BQ2-from-BQ2 aarch64 build stopped at the start of system-from-sysroot: m4's configure found "no acceptable grep". autoconf's grep probe checks with diff, and the new rootfs's busybox diff segfaults (exit 139). - Cause 1, a miscompile: clang on aarch64 loads ptr_to_globals before SET_PTR_TO_GLOBALS stores it, because BusyBox declares it const. diff_main+0x64 then stores G.opt_U_context through NULL (address 0x4). Found with an LD_PRELOAD SIGSEGV handler and llvm-objdump. - Fix 1: src/step_runner.cr's busybox strategy also sets CONFIG_EXTRA_CFLAGS="-DBB_GLOBAL_CONST=''", libbb.h's own remedy, for every compiler. disable_kconfig_symbols becomes set_kconfig_symbols, and BUSYBOX_DISABLED_SYMBOLS becomes the one table BUSYBOX_KCONFIG. - Cause 2, a stale binary: rootfs-from-sysroot set STRIP=/bin/true. BusyBox's Makefile makes busybox by stripping busybox_unstripped, so that rule wrote nothing, and make install shipped the busybox left in the same tree by sysroot-from-seed, built by the seed's compiler. Seeds built from Alpine therefore ship Alpine gcc's busybox, which hid cause 1. - Fix 2: src/sysroot_builder.cr gives rootfs-from-sysroot SKIP_STRIP=y, so the Makefile copies busybox_unstripped to busybox. finalize-rootfs still strips debug info.

Demonstrates: - crystal tool format --check is clean. - crystal spec on aarch64-1: 204 examples, 3 failures, 5 pending. The same 3 fail without this change (spec/sysroot_builder_spec.cr:290, :379, :563 expect x86_64 triples, and the builder defaults to the host arch). spec/step_runner_spec.cr covers set_kconfig_symbols and the BB_GLOBAL_CONST value. - Control on aarch64-1: step 4's busybox tree, rebuilt by the BQ2 seed's clang with SKIP_STRIP=y, segfaults in diff as-is; with only the flag added, diff returns 0 for identical files and 1 for different ones. - Not yet shown on a rebuilt seed; run 0de68bf272 redoes steps 3 and 4 at this check-in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUH8KKecLJryWiTGbAfUrh check-in: 2469a5375c user: aarch64-1-bootstrap-qcow2-session tags: seed-ocx-links

Intent: ticket d78258f03b, reported by cleave's x86-1 session. The maintainer asked, through that session, for a ticket and a fix. Also F1 on log ticket 2a8699a249.

Expression: - Cause: BusyBox 1.36.1's wget runs openssl s_client ... -verify_hostname NAME for https. LibreSSL 3.8.2's s_client rejects that option, and wget hides s_client's stderr, so every https URL failed with "Connection reset by peer". - LibreSSL's s_client cannot check a hostname at all: wrong.host.badssl.com passes with -verify_return_error, and no release up to 4.1.0 adds the option. Dropping the option would therefore have accepted any valid certificate for the wrong host. - New bq2 applet tls-connect [--insecure] HOST:PORT NAME (src/tls_connect_command.cr). It connects with Crystal's TLS client, which checks the chain and the name, then pipes stdin and stdout over the connection. It strips the brackets wget keeps around IPv6 addresses. - patches/busybox-1_36_1/wget-use-bq2-tls-connect.patch makes wget run tls-connect, passing --insecure for --no-check-certificate, and leaves stderr open so the real error shows. It also removes is_ip_address(), which nothing uses any more. The seed's /usr/bin/tls-connect link comes from the bq2 applet list.

Demonstrates: - crystal tool format --check is clean. crystal spec gives 203 examples, 0 failures, 5 pending; the new pending case needs network. - Built by hand in this run's root filesystem (x86_64): BusyBox 1.36.1 with both patches and defconfig minus STATIC_LIBGCC and MODPROBE_SMALL. Its wget, with tls-connect on PATH: - fetches https://www.example.com/ and https://1.1.1.1/, both exit 0; - fetches a GitHub tarball through a redirect, byte-identical to upstream (sha256 ea549484...); - refuses wrong.host, expired and self-signed .badssl.com with "certificate verify failed", exit 1; - with --no-check-certificate, fetches wrong.host, exit 0. - PatchApplier applies both patches to pristine source, and the resulting wget.c is identical to the one tested. - Not yet shown on a rebuilt seed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUH8KP8UEDm35iV7mCqbNY check-in: ee75b652a0 user: x86-1-bq2 tags: seed-ocx-links

Intent: plan step 1 of ticket 2a8699a249 (what OCX's client needs beyond the 1.0.0 seed). The maintainer, 2026-11-02: "go ahead with plan step 1". Fixes F4 and N5.

Expression: - system-from-sysroot links /usr/lib/libunwind.so.1, libc++.so.1 and libc++abi.so.1 to the triple directory. Today they resolve only through /etc/ld-musl-ARCH.path, which OCX's sandbox does not bind, so ocx2, confine, serve and receipt-reader fail to load. Only the sonames are linked: that is what the loader looks up, and clang already finds the triple directory for -l. - It also links /usr/bin/ar -> llvm-ar and /usr/bin/ranlib -> llvm-ranlib, the names make and autotools default to. - sysroot-functest gains ar-archive and runtime-libs. The contract row "cc, c++, ld" becomes "cc, c++, ld, ar, ranlib"; the crystal row also cites runtime-libs. FUNCTEST.md matches.

Demonstrates: crystal tool format --check is clean; crystal spec gives 198 examples, 0 failures, 4 pending, with the seed-links spec asserting all five links on aarch64. The two new cases' commands pass in this run's rootfs, whose overlay has equivalent links. Not yet shown on a rebuilt seed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUH8KP8UEDm35iV7mCqbNY check-in: 0cb221436e user: x86-1-bq2 tags: seed-ocx-links

Intent: CLAUDE.md loads into every session; its repository layout was derivable from the tree and already stale (~8k lines, ~16 specs vs 10.6k and 20), and the in-process LLVM plan was stated twice and only matters when working on it. Expression: drop the layout section and the CI summary line; move strategy 1 detail and "LLVM integration in Crystal" into .claude/skills/inproc-llvm/SKILL.md, leaving a one-line pointer in strategy 1. Demonstrates: CLAUDE.md 16.2k -> 13.1k chars; no other guidance changed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx Leaf check-in: 6d67be9e92 user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the x86_64 layer build from the 1.0.0 seed stopped at tools/lto/LTO.exports, which LLVM generates with Python (_Python3_EXECUTABLE-NOTFOUND); the seed has no Python. Expression: -DLLVM_TOOL_LTO_BUILD=OFF and -DLLVM_TOOL_REMARKS_SHLIB_BUILD=OFF, as the base LLVM already sets; the toolchain-only install ships neither library. Demonstrates: crystal spec passes; the layer build resumed with these flags gets past LTO.exports.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 8ee1fb8d7a user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the layer phase runs inside the seed namespace, but its LLVM step listed the patches by their host paths, so the x86_64 layer build from the 1.0.0 seed failed opening the first patch. Expression: map the patch paths to the namespace workspace, as patches_for does for regular phases; the layer spec checks every patch path is under /bq2-rootfs/workspace/. Demonstrates: crystal spec, 199 examples, 0 failures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: eb19984bee user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-26, approving "Prune openssl/ocspcheck and git's imap-send, daemon, http-backend and scalar from the base; turn them into layers only if someone asks."

Expression: src/sysroot_builder.cr adds these to ROOTFS_PRUNE: /usr/bin/openssl, /usr/bin/ocspcheck, /usr/libexec/git-core/{git-imap-send,git-daemon,git-http-backend,scalar} and /usr/bin/scalar. Fossil and Crystal link only LibreSSL's libraries (Crystal's specs `require "openssl"`; nothing in its source or specs runs the program), git stays a client, and the image's rebuild uses none of them.

Demonstrates: spec/sysroot_builder_spec.cr ("prunes unused files ...") checks openssl and git-daemon among the prunes. crystal spec gives 199 examples, 0 failures, 4 pending.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 67ce2be513 user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-26: "if it isn't needed for rebuild, it should be an opt-in able to build from the seed." Earlier, on cleave's needs: "keeping it as another phase build should be fine", and "can we keep it as an optional build target to keep a set of tested rules to get there?"

Expression: - src/sysroot_builder.cr: - LAYERS = {"bsp-tools" => "/opt/bsp"} and SysrootBuilder.new(layer:), with `sysroot-builder --layer=NAME`. A layer plan has host-setup (the seed plus only the layer's sources: llvm-project, and bootstrap-qcow2 for its patches) and one layer-<name> phase in the seed's namespace. That phase builds with the seed's own toolchain into the layer's prefix, then packages the prefix as bq2-layer-<name>.tar.gz with entries from opt/..., so it unpacks onto /. A layer needs a BQ2 seed, and an unknown name raises. - bsp-tools (bsp_tools_llvm_flags) is cleave's recipe from tools/build-beagley-ai-bsp.sh as one CMake build: clang only, the native backend plus ARM, LLVM_INSTALL_TOOLCHAIN_ONLY (clang with llvm-objdump/readobj/size among LLVM's toolchain tools), statically linked, CLANG_DEFAULT_* set, Python and tests off. It adds an llvm-readelf -> llvm-readobj link. - The base build goes back to the native backend only (aarch64: AArch64), and llvm-objdump/readobj/size go back on the disabled tool list. The image does not need either to rebuild itself. - host_setup_steps takes the package list. - src/step_runner.cr: the tarball strategy takes TARBALL_BASE, the directory entry names are relative to (default: the source root). - CLAUDE.md: item 4 is native-only again, and anything the rebuild does not need is an opt-in layer.

Demonstrates: spec/sysroot_builder_spec.cr. New: "plans the opt-in bsp-tools layer on a BQ2 seed". Updated: "builds only the native LLVM backend", and the shipped LLVM turns objdump/readobj/size off. crystal spec gives 199 examples, 0 failures, 4 pending. No layer has been built for real yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 5f8e509333 user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-25: "if the verification passed, you are clear to merge to trunk and also tag as 1.0.0."

Expression: brings trunk (d539ac7c21) to 0219f453ae, adding: - cleave's seed needs: - cc/c++ links; - llvm-objdump, llvm-readelf and llvm-size; - the ARM backend on aarch64; - full busybox modutils; - pkg-config and ldd links. - The floating development leaves merged on 2026-10-20: - g14 (dbab4b7013): sysroot-runner fails cleanly on unexpected exceptions, with bdwgc instrumentation; - g12 (e9dd87ad0a): bit-identical seed (SOURCE_DATE_EPOCH, -ffile-prefix-map, build state left out); - the sysroot-functest harness (cd8783522f). - Crystal 1.21 support: sysroot-namespace enters the user namespace by fork+exec. - The BQ2/OCX tool contract (FUNCTEST.md). - Seed fixes: - one exclude list for both tarball writers; - PAX for long names; - the Crystal cache kept in /tmp; - busybox Kconfig set in Crystal; - hard links stored once; - /etc/passwd and /etc/group; - USER set in the namespace. - Three named toolchains drive every phase env, with one after-LLVM switch rule. The system LLVM is built once, and --single-stage-llvm is opt-in. - Ship less: - no libclang; - unused archives and tools pruned; - debug info stripped once per file; - git and bq2 installed as symlinked multi-call binaries. - Version 1.0.0.

Demonstrates: the merged tree is identical to 0219f453ae. crystal spec gives 198 examples, 0 failures, 4 pending. Verification on x86_64 is recorded in 0219f453ae: - seeds B and C byte-identical at 0f09d5412f; - seeds F and G byte-identical (101,680,340 bytes) at bd4cc17010; - in G: Crystal self-hosts, its spec suites pass apart from 2 sandbox-only std_spec cases, and Fossil, git and the bq2 applets pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx Leaf check-in: c86ec69825 user: run-58530641 tags: 1.0.0, origin/codex-development-20260223, origin/codex/bookmark-state, origin/codex/fix-sysroot-runner-overwriting-issue-m0vpua, trunk

Intent: the maintainer, 2026-10-25: "if the verification passed, you are clear to merge to trunk and also tag as 1.0.0."

Expression: src/bootstrap_qcow2.cr sets VERSION to "1.0.0". VERSION is used in the image's os-release and the rootfs tarball name. Nothing else changes from bd4cc17010.

Demonstrates: the 1.0 acceptance, from REPRODUCE.md and milestone 5bec0b87, on x86_64 (8-core host), recorded in repro-x86/: - Alpine -> seed A -> seed B -> seed C at 0f09d5412f: B and C byte-identical (1,271,618,658 bytes, sha256 420a70c4...25af2ca3). - At bd4cc17010, the ship-less image: seed F (from B) and seed G (from F) byte-identical (101,680,340 bytes, sha256 420c891d04f05e5ab81b18f709cc9df1be0a32652a0ff7e4cd9fea3495a4a853). The cut-down image rebuilds itself exactly. - Inside G's rootfs, using only the image's tools: - `make crystal` rebuilds Crystal's compiler, which then runs `crystal eval "puts 6 * 7"` (42); - with that compiler, primitives_spec, compiler_spec and interpreter_spec pass; std_spec passes 17,817 of 17,819, and both failures come from the sandbox (no /sys/devices/system/cpu for the cpu_count shell check; uid 0 is the only one mapped, so the unprivileged-chroot test stays root); - Fossil round trip and Fossil -> git export pass; the git round trip passes; the bq2 applets (pkg-config, curl, git-remote-https) pass. crystal spec gives 198 examples, 0 failures, 4 pending.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 0219f453ae user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-25: "shipping/building less should be the biggest target", "a smallest size target is a high priority", "what is required for the image to rebuild itself is required", and on git's ~140 hard-linked commands: "that's fine, as long as the space isn't wasted. probably want same mechanism for bq2 and busybox."

Measured on a copy of gen E's x86_64 rootfs (2c43dec7b8), each step repacked by the hard-link-aware TarWriter: - as built: 1,271.6 MB compressed, 2.98 GB unpacked; - hard links stored once (f6ddbdc3cb): 260.5 MB; - without the static LLVM/clang/lld archives, libclang, c-index-test and man/doc/info: 161.8 MB; - debug info stripped (77 unique ELF files, once per inode): 117.0 MB compressed, 0.42 GB unpacked. In that image, clang links libclang-cpp.so and libLLVM.so; crystal and ld.lld link libLLVM.so; and `llvm-config --shared-mode` is "shared" (-lLLVM-18). Nothing links the .a archives.

Expression: - src/sysroot_builder.cr: - llvm_configure_flags adds CLANG_TOOL_LIBCLANG_BUILD=OFF and CLANG_TOOL_C_INDEX_TEST_BUILD=OFF, so libclang and its test driver are not built. - finalize-rootfs gains one remove-tree step per ROOTFS_PRUNE glob (the LLVM/clang/lld .a archives, llvm-tblgen, clang-tblgen, ctest, cpack, /usr/share/{man,doc,info}), then a strip-debug step over ROOTFS_STRIP_DIRS, both before rootfs-tarball. Everything the image needs to rebuild itself stays: cmake, make, headers, llvm-config, libLLVM.so, libclang-cpp.so. - git builds with INSTALL_SYMLINKS=YesPlease, so its dashed commands are symlinks to git. - system-from-sysroot links every bq2 applet into /usr/bin (bq2_applet_links, from CLI.symlink_names, the list `bq2 --install` uses). That covers pkg-config, curl and git-remote-https, which gives git an https transport without libcurl. - src/step_runner.cr: - "remove-tree" takes a glob, and no match is fine. - New "strip-debug" strategy, StepRunner.strip_debug. It runs `llvm-strip --strip-debug` once per inode over ELF files and ar archives, picked by their magic bytes. The stripped copy replaces the first name by rename, which is safe even for libLLVM.so while llvm-strip has it mapped, and the other names are linked back, so hard links survive. (Stripping each name in place had turned git's 144 links back into copies: 0.52 -> 0.99 GB.) - src/cli.cr: CLI.symlink_names is shared by `bq2 --install` and the builder. - spec/spec_helper.cr loads every command main.cr loads (curl, pkg-config, git-remote-https), so specs see bq2's full applet list.

Demonstrates: - spec/step_runner_spec.cr: ".strip_debug strips each file once and keeps its hard links" (a real `cc -g` binary and a second name; .debug_info gone, same inode, still executable) and "remove-tree removes every match of a glob". - spec/sysroot_builder_spec.cr: "prunes unused files and strips debug info before packaging the rootfs", and "links cc, c++, pkg-config and ldd" now also checks curl and git-remote-https. - The plan diff against 2c43dec7b8 changes only those steps and flags, plus the rootfs user-database steps from 0c10abdae9. - crystal spec gives 198 examples, 0 failures, 4 pending; the format check passes. A real seed build and the image acceptance checks are next.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: bd4cc17010 user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-25: "getting to a crystal compiler that passes all its own tests should illustrate the minimum to ship (also with fossil and git and bq2)". The first acceptance run inside gen E's rootfs (2c43dec7b8, x86_64, image tools only, via bin/sysroot-namespace) failed three checks for one reason. The seed has no user database, and USER was unset: - std_spec stopped compiling at spec/std/system/group_spec.cr:8, `{{ `id -gn` }}`, with "id: unknown ID 0"; - `fossil commit` said "Cannot figure out who you are"; - the Fossil -> git export failed because it depends on that commit.

Expression: - src/sysroot_builder.cr: rootfs-from-sysroot also writes /etc/passwd and /etc/group (ROOTFS_PASSWD, ROOTFS_GROUP), with root and nobody at 65534, Linux's default overflow uid/gid. No passwords. - src/sysroot_namespace.cr: reset_environment sets USER and LOGNAME to root, the name of the uid the caller is mapped to, next to HOME and PATH.

Demonstrates: - spec/sysroot_builder_spec.cr ("gives the rootfs a user database with root and nobody"). The rootfs-profile and plan-shape specs now find steps by path, not by prepare-rootfs index. - crystal spec gives 195 examples, 0 failures, 4 pending. - With the two files in gen E's rootfs and this check-in's sysroot-namespace: - Fossil round trip (init, open, add, commit, clone, timeline) passes; Fossil -> git export passes; - std_spec runs 17,819 examples with 2 failures, both from the sandbox: System.cpu_count's shell check falls back to `nproc --all`, which reads 1 here because this sandbox has no /sys/devices/system/cpu, and Process.chroot "raises when unprivileged" stays root, because only uid 0 is mapped and setuid(65534) fails. - The first run already passed primitives_spec, compiler_spec (19 min), interpreter_spec, the git round trip and the bq2 applets. No check reported a missing command.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 0c10abdae9 user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-25: "shipping/building less should be the biggest target." The x86_64 seed B (2c43dec7b8's chain, 1,271,618,658 bytes) unpacks to 2.98 GB in 12,957 files, and 2.07 GB of that is usr/libexec/git-core. git installs its built-in commands as hard links to one 14.6 MB binary, and TarWriter wrote every link as a full copy. The 0.4.0 seed (541,647,749 bytes) was written by GNU tar, which keeps hard links. TarWriter has written every seed since the plan's rootfs-tarball step and sysroot-tarball both moved to it.

Expression: src/tar_writer.cr writes the first name it reaches for a multiply-linked file (lstat st_nlink > 1, keyed by device and inode) in full, and every later name as a hard-link entry (type '1') pointing to that first path. Long link names still get a PAX linkpath record. The walk is sorted, so the same name is always the one written in full and archives stay reproducible. Bootstrap::Tarball already extracts type '1' entries as hard links.

Demonstrates: spec/sysroot_runner_spec.cr ("leaves build scratch and bookkeeping out, keeps long names and stores hard links once"). Two names for one 64 KiB random file extract as the same inode, and the archive is under 100 KB, so it holds one copy. crystal spec passes; the format check passes. The size on a real seed will be measured after the running gen E, whose tarball step uses the previously built bq2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: f6ddbdc3cb user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-25, agreeing to revert it. Measured on this 8-core x86_64 host: gen D (55daf72c7a, seed B -> D, trimmed stage1) took 41.5 min for stage1, against 42.4 min in gen B (0f09d5412f), and 2 h 52 min overall, the same as B. The time goes into the LLVM and clang libraries that the clang driver itself needs, not into the tools the trim dropped. A 19-name skip list for about 2% is configuration bloat.

Expression: src/sysroot_builder.cr drops STAGE1_SKIPPED_CLANG_TOOLS and STAGE1_SKIPPED_LLVM_TOOLS and restores stage1's flags to those of 0f09d5412f (libLLVM.so on, all targets, all tools). spec/sysroot_builder_spec.cr drops "builds only what stage2 needs in stage1". `--single-stage-llvm` stays.

Demonstrates: all six default plans (aarch64 and x86_64 with Alpine, published BQ2 and local BQ2 seeds) are byte-identical to 0f09d5412f's. Gen D's seed matched seed B file for file (17,087 entries), except usr/bin/bq2, which embeds its source check-in. crystal spec gives 194 examples, 0 failures, 4 pending; the format check passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 2c43dec7b8 user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-24: "ultimately, i want you to look at legitimate ways to speed it up (no games)". On skipping stage1 for BQ2 seeds: "the problem is that seed->seed might opt into newer llvm versions, so keep that as an option."

Measured on this 8-core x86_64 host, generation B of 0f09d5412f (BQ2 seed -> BQ2 seed, 2 h 52 min): LLVM stage1 42.4 min, stage2 42.3 min, system 42.5 min, out of 172 min.

Expression (src/sysroot_builder.cr): - llvm_stage1_flags: stage1's only use is to compile stage2, so it builds just what that needs: - clang (the driver), lld, ar/nm/ranlib/objcopy/strip, llvm-config, compiler-rt and the C++ runtimes; - the native backend only (AArch64 on aarch64; stage2 still builds AArch64;ARM); - no libLLVM.so (LLVM_BUILD_LLVM_DYLIB was ON in stage1); - none of clang's other tools. STAGE1_SKIPPED_CLANG_TOOLS turns off libclang, libclang-cpp (clang-shlib), clang-repl (which pulls in the JIT), clang-scan-deps, c-index-test, clang-format and the offload, refactoring and arch tools. Names are from LLVM 18's clang/tools/CMakeLists.txt. - no llvm-objdump/readobj/size (STAGE1_SKIPPED_LLVM_TOOLS). Stage2 and the system build are unchanged, so what ships is unchanged. - SysrootBuilder.new(single_stage_llvm:) and `sysroot-builder --single-stage-llvm`: with a BQ2 seed, sysroot-from-seed builds LLVM once, as `llvm-project` in build-sysroot-from-seed. It uses the seed's /usr/bin/clang and /usr libc++, and the runtimes get --sysroot=/. Valid only when the seed already has DEFAULT_LLVM_VER; the default stays two stages, so a seed -> seed build to a newer LLVM is unaffected. With the Alpine seed it raises, because Alpine's clang has no libc++. llvm_stage2_flags takes the compiler's sysroot separately from its prefix.

Plan diff (against 0f09d5412f, all six plans): only the sysroot-from-seed llvm-project-stage1 configure flags change. They gain LLVM_BUILD_LLVM_DYLIB=OFF, 19 CLANG_TOOL_*_BUILD=OFF and LLVM_TOOL_LLVM_{OBJDUMP,READOBJ,SIZE}_BUILD=OFF, and on aarch64 LLVM_TARGETS_TO_BUILD=AArch64. Without --single-stage-llvm nothing else moves.

Demonstrates: spec/sysroot_builder_spec.cr. New: "builds only what stage2 needs in stage1" and "builds the sysroot LLVM once with a BQ2 seed's clang only when asked". The specs for objdump/readelf/size and for the ARM backend now check the shipped (system) LLVM, not stage1. crystal spec gives 195 examples, 0 failures, 4 pending; the format check passes. Not yet built for real; that comes after the running generation C (B -> C) finishes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 55daf72c7a user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-24: "not being able to properly plan the settings for the toolchain being used and instead changing settings in a way that isn't clear on the toolchain state actually in use at that phase has caused rippling changes in a way that bloats the configuration data COMPLETELY unnecessarily", and "building llvm twice per seed dominates the build time."

Expression (src/sysroot_builder.cr): - New `Toolchain` enum (Seed, Sysroot, Usr) and `toolchain_env`, the only place that says which compiler, C++ runtime and binutils each one is: - Seed: Alpine's /usr/bin/clang, or a BQ2 seed's Usr toolchain; - Sysroot: /opt/sysroot clang with /opt/sysroot's libc++; - Usr: the prefix-free /usr clang. PATH and LD_LIBRARY_PATH are set by each phase, since they describe where it runs. SOURCE_DATE_EPOCH and CRYSTAL_CACHE_DIR come from `build_env`. sysroot_phase_env, rootfs_phase_env, bq2_phase_env, ToolchainKind and the post_llvm_* bookkeeping are removed. - One rule, `toolchain_after_llvm`, used by both phases that build LLVM: every package after llvm-project switches to the toolchain that LLVM build just installed. Before, only system-from-sysroot did this. In sysroot-from-seed, crystal and shards had their own CC/CXX overrides, while bdwgc, pcre2, gmp, libiconv, libxml2, libyaml and libffi kept the seed's compiler. With a BQ2 seed, PATH already made the bare `clang` resolve to /opt/sysroot's after LLVM, but with the seed's /usr libc++ flags. Those per-package CC/CXX overrides and a duplicate LD_LIBRARY_PATH are removed. - llvm_stage_steps: only sysroot-from-seed builds LLVM in two stages. It starts on the seed's compiler, so stage1 makes /opt/sysroot's clang, and stage2 rebuilds LLVM with it. system-from-sysroot already has that clang and builds LLVM once, as the step `llvm-project` in build-system-from-sysroot. Its stage1 was a static LLVM without runtimes that stage2 overwrote without using: one of four LLVM builds per generation, about 22 minutes on this 8-core x86_64 host. The stage1 "disable runtimes" option, used only there, is removed.

Plan diff (all six plans: aarch64 and x86_64, each with the Alpine, published BQ2 and local BQ2 seeds, effective env per step, against the plans from f4973b340f). Only: - system-from-sysroot: llvm-project-stage1 and -stage2 become llvm-project, with configure flags, env and patches identical to the old stage2 apart from its build dir. - sysroot-from-seed: the packages after LLVM build with /opt/sysroot's clang, libc++ flags and llvm-ar/nm/ranlib/strip. crystal and shards get the toolchain's CC/CXX in place of their hand-written ones. - sysroot-from-seed: PATH also lists /usr/sbin:/sbin. - system-from-sysroot: the packages after LLVM keep /opt/sysroot/bin at the end of PATH; /usr/bin still comes first.

Demonstrates: spec/sysroot_builder_spec.cr. Updated: "builds LLVM in two stages from the seed and once, with the sysroot toolchain, for the system" and the per-phase build dirs. New: "switches every package after llvm-project to the toolchain that LLVM build installed", for both seeds. crystal spec gives 193 examples, 0 failures, 4 pending; the format check passes. A real x86_64 build of this plan is next. The build running now, of f4973b340f, is the timing baseline.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 0f09d5412f user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-22: "can we start with seed size reductions (not including unnecessary files like sources), specific fixes ...". The first aarch64-1 seed build of this branch, which cleave ran as tickets e62c55b33e and 8236bdab85 at 2377e2ecc7, found these: - sysroot-tarball wrote 5,208,299,902 bytes against 0.4.0's 541,647,749, because it packed /workspace (all sources and build trees), /tmp and /var/lib. - It first died with "Path too long for tar header even with PAX" on a Crystal cache that had leaked into <rootfs>$HOME/.cache/crystal. - busybox depmod still ignored -b. - There was no pkg-config. - crystal --version reported "Default target: aarch64-bq2-linux-gnu".

Expression: - src/sysroot_workspace.cr: SEED_EXCLUDES lists what no seed contains: proc, sys, dev, run, tmp, workspace, work, var/lib, .bq2-rootfs and .pivot_root. These are the paths the plan's rootfs-tarball step already excluded, which gave 0.4.0 its size, plus the pivot_root leftover. - src/step_runner.cr: the "tarball" strategy calls TarWriter.write_gz with SEED_EXCLUDES and an epoch mtime, in place of shelling out to `tar` (resolves its TODO). The plan's rootfs tarball is now byte-reproducible like sysroot-tarball's. - src/sysroot_runner.cr: sysroot-tarball uses the same SEED_EXCLUDES, not its own partial list. - src/tar_writer.cr: removes assert_paths_fit and LongPathError. They refused any basename over 99 bytes, although write_entry already emits a PAX path/linkpath record for those, which has no length limit. The check also walked the excluded trees. - src/sysroot_builder.cr: - CRYSTAL_CACHE_DIR=/tmp/crystal_cache is set in the sysroot, rootfs and bq2 phase envs, replacing the two per-package copies on the crystal package. Before, the bootstrap-qcow2 and shards builds used $HOME/.cache/crystal, and the host $HOME appeared inside the rootfs. - system-from-sysroot links /usr/bin/pkg-config -> bq2 (its pkg-config applet) and /usr/bin/ldd -> /lib/ld-musl-ARCH.so.1. Crystal runs `ldd --version` to pick gnu or musl and assumes gnu when there is no ldd. musl's loader run as ldd prints "musl libc ..." (checked on x86_64 musl 1.2.5). - src/step_runner.cr: the busybox strategy turns off STATIC_LIBGCC and MODPROBE_SMALL in Crystal (StepRunner.disable_kconfig_symbols) after `make defconfig`, in every phase. Before, it ran tools/kconfig/config only when BQ2_KCONFIG_CONFIG_TOOL named an existing file and otherwise just warned and skipped. The seed shipped the small modutils, whose depmod chdirs to the absolute /lib/modules. tools/kconfig/config and the env var are removed.

Demonstrates: - spec/sysroot_runner_spec.cr ("leaves build scratch and bookkeeping out and keeps names longer than 99 bytes"), spec/step_runner_spec.cr (".disable_kconfig_symbols", the busybox strategy) and spec/sysroot_builder_spec.cr ("links cc, c++, pkg-config and ldd in the seed"). - crystal spec gives 192 examples, 0 failures, 4 pending; the format check passes; shards build succeeds. - By hand: busybox 1.36.1 defconfig, then disable_kconfig_symbols, `make silentoldconfig` and a rebuild in a tree already built with MODPROBE_SMALL: `depmod -b /tmp/stx 6.12.0` wrote modules.dep.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: f4973b340f user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-21: "if you want to leave the line and say it is explicitly not provided and `crystal` is provided *instead*, I would have accepted that wording, but, instead, it looked like you were going to implement `python3` *IN* `crystal`."

Expression: restores the python3 row as "not provided; crystal instead". The bash and jq rows get the same wording: "not provided; busybox ash (/bin/sh) instead" and "not provided; crystal eval instead". None of them reads as the expected tool being reimplemented. FUNCTEST.md says that such rows name a different tool to use, not a reimplementation of the expected one. The case comments for sh-posix and crystal-json are updated to match.

Demonstrates: spec/sysroot_functest_spec.cr. crystal spec gives 190 examples, 0 failures, 4 pending, and the format check passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 6367d820dd user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-21: "Never include \"python3\" for anything."

Expression: removes the python3 row, which offered Crystal as its replacement, from the contract in FUNCTEST.md and src/sysroot_functest.cr. Also removes the python3 mentions from the doc's intro, the class comment and the crystal-hello case comment. crystal-hello still backs the "crystal, shards" row.

Demonstrates: spec/sysroot_functest_spec.cr ("uses every corpus case for some BQ2 row" still passes). crystal spec gives 190 examples, 0 failures, 4 pending, and the format check passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: fcea4fa2b3 user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-21: "The suggested split in the document, with tools not targeting humans clearly belonging to OCX, holds."

Expression: the contract states the rule, and the rows follow it. A tool that also serves people at the shell is BQ2's. A tool that only an agent uses is OCX's, even when a shell command could do the job. New OCX rows: reading, writing and editing files for the agent; searching code for the agent (what LLMs use rg for; people use grep -rn); and fetching web pages for the agent (people use bq2 curl or wget). The rg row moves from BQ2 to that OCX search row. FUNCTEST.md, src/sysroot_functest.cr and its doc comment are updated to match.

Demonstrates: spec/sysroot_functest_spec.cr ("keeps the agent-only tools, runtime and commit path with OCX"). crystal spec gives 190 examples, 0 failures, 4 pending, and the format check passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: c1e7b077b2 user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-21: "The FUNCTEST.md reads a little misguided. BQ2 should *NOT* try to reproduce OCX agent functionality. It *SHOULD* be a contract between BQ2 and OCX for *ONE* of them to provide the tools LLMs expect (or a suitable replacement)." Also: "The OCX design has changed a lot since that doc was written as well."

Expression: - FUNCTEST.md is rewritten as the contract. Each tool LLMs expect has exactly one provider. BQ2 rows give the binary or the replacement that ships in the rootfs: busybox ash for bash, `grep -rn` for rg, Crystal for python3 and jq. OCX rows give only the capability: the agent runtime and Node, test-gated commit and push, forum and ticket I/O, and runs on nodes. How OCX does these is left to OCX, so OCX's design can change without touching BQ2. Removed: the "three ocx surfaces" (native, mcp-server, agent-tool), the dual-provider rows, and the canned-response agent-loop plan. - src/sysroot_functest.cr: Plane/Disposition/matrix/--matrix become Provider/ContractRow/contract/--contract. Each BQ2 row names the corpus cases that check it. New cases: sh-posix, find-name, diff-patch, make-target, crystal-json and vcs-present. tar-roundtrip is now tar-gzip-roundtrip. - spec/sysroot_functest_spec.cr: checks that each expected tool appears once, that BQ2 rows cite existing cases and OCX rows cite none, that every case backs some row, and that the agent runtime and commit path stay with OCX.

Demonstrates: spec/sysroot_functest_spec.cr. crystal spec gives 190 examples, 0 failures, 4 pending, and the format check passes. `bin/sysroot-functest --sysroot alpine-minirootfs-3.23.2-x86_64` passes sh-posix, grep-exact, sed-substitute, find-name and tar-gzip-roundtrip. The other cases exit 127 there because that rootfs lacks patch, make, cc, crystal, git and fossil. The diff-patch command prints "c" with busybox 1.36.1 built from BQ2's defconfig.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 78206835eb user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-20: "Crystal 1.21 breaks the userns ... Can you build/install Crystal 1.21 and check that bin/sysroot-namespace works?" With bq2 built by Crystal 1.21.1 (built here from source with LLVM 21, x86_64 Alpine), `bin/sysroot-namespace --rootfs ROOTFS` died with "unshare failed: EINVAL". The g13 fork+exec rewrite (ticket d3a1b086ab) fixed only sysroot-runner. sysroot-namespace still called unshare(CLONE_NEWUSER) in place, which the kernel refuses for a multi-threaded process, and Crystal 1.21 is multi-threaded from the start.

Expression: - src/sysroot_namespace.cr: SysrootNamespace.reexec_in_user_namespace now takes the bq2 subcommand argv plus the uid/gid to map, not a sysroot-runner descriptor. It stays the single user-namespace entry. - src/sysroot_namespace.cr: run_namespace re-execs `sysroot-namespace` with its original arguments when it lacks CAP_SYS_ADMIN, and returns the child's exit code. CLI.parse consumes args, so they are copied first. The child holds CAP_SYS_ADMIN and enters the rootfs in place, as before. - src/sysroot_runner.cr: passes ["sysroot-runner", "--resume", descriptor] to the generalized entry. - docs/design/userns-namespace-entry.md: notes that sysroot-namespace uses the same entry. - spec/sysroot_namespace_spec.cr: the "maps uid/gid without pre-existing CAP_SYS_ADMIN" integration spec used to call the in-place unshare from `crystal eval`, so it failed on 1.21 with EINVAL. It now drives reexec_in_user_namespace and checks, in the re-exec'd child, for a new user namespace, CAP_SYS_ADMIN, and an in-place mount-namespace unshare.

Demonstrates: spec/sysroot_namespace_spec.cr ("enters a user namespace and maps uid/gid without pre-existing CAP_SYS_ADMIN"). crystal spec gives 188 examples, 0 failures, 4 pending under both Crystal 1.21.1 and 1.18.2, and the format check passes. By hand, with both compilers: `bin/sysroot-namespace --rootfs alpine-minirootfs-3.23.2-x86_64 [--bind $PWD:/work] -- /bin/sh -c 'id; ...; exit 7'` runs as uid 0 with /proc, /dev and /tmp mounted, sees the bind, and returns 7.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 26de05849a user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-20: "There are a number of floating leaves and Crystal 1.21 breaks the userns. Can you merge the different development branches over the last month?"

Expression: merges three open leaves that trunk d539ac7c21 does not contain: - dbab4b7013 (session ...-g14, which includes 761197aeee): sysroot-runner fails cleanly on unexpected step exceptions, logs a workspace watch for the bdwgc Makefile.in loss, and makes the report directory writable under the mapped uid (tickets d3a1b086ab, 7f813f3da7). - e9dd87ad0a (session ...-g12, with edc2126fe3): bit-identical seed. Adds a fixed SOURCE_DATE_EPOCH to every phase env and -ffile-prefix-map=/workspace=., drops the workspace and build state from the seed, and excludes sysroot-build-reports. - cd8783522f (session a89ee18a...): the sysroot-functest functional-parity harness (FUNCTEST.md, src/sysroot_functest.cr). Not merged: 12bcda9205 (g11, with 26c6d077b8 and ddf00a8501). Its probe/re-exec userns entry is the family that the g13 fork+exec rewrite on trunk retired. The Aug 25-26 qcow2->AMI leaves (296e93831e and its parents) belong to deferred ticket 184ea2b80e and are older than a month. All merges applied cleanly, with no conflicts.

Demonstrates: crystal tool format --check, crystal spec (188 examples, 0 failures, 4 pending) and shards build with Crystal 1.18.2. spec/sysroot_functest_spec.cr and spec/sysroot_runner_spec.cr come in with the merges.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 62cf6fbc8d user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-19: "The reason the standing direction is to only build for the host target is to try to optimize build time. This is a valid need and I approve changing CLAUDE.md to enable the arm32 target on arm64 hosts." On aarch64-1, cleave builds a second LLVM 18.1.7 (about 10 minutes on 48 cores) only to get the ARM backend. It needs ARM for the TI K3 R5 SPL (tiboot3.bin) and the Linux arm64 compat vDSO.

Expression: in src/sysroot_builder.cr, llvm_targets_to_build returns "AArch64;ARM" for aarch64/arm64. x86_64 still builds X86 alone. Item 4 of the CLAUDE.md build-time list is now "Minimal LLVM targets", with the ARM exception and the reason for it.

Demonstrates: spec/sysroot_builder_spec.cr ("builds the AArch64 and ARM LLVM backends on aarch64, X86 alone on x86_64"). crystal tool format --check, crystal spec (173 examples, 0 failures, 4 pending) and shards build all pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: 2377e2ecc7 user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Intent: the maintainer, 2026-10-19: "help cleave integrate any features it needs into this project, bootstrap-qcow2 (BQ2), without adding new packages". Cleave's jobs on aarch64-1 build its tools, a BeagleY-AI boot chain and a Linux kernel inside the BQ2 0.4.0 seed (cleave tickets 650aed1f3f, 33d4fd6ade and the BSP runs after them). Its tools/build-beagley-ai-bsp.sh patches the seed before every build to work around these gaps.

Expression: three gaps that trunk still had. None adds a package. - src/sysroot_builder.cr: system-from-sysroot links /usr/bin/cc -> clang and /usr/bin/c++ -> clang++. Crystal links with $CC, which defaults to cc, so `shards build` in the seed failed until cleave set CC=clang. - src/sysroot_builder.cr: LLVM_OBJDUMP, LLVM_READOBJ (which also installs llvm-readelf) and LLVM_SIZE move from the disabled tool list to the enabled one. Linux `make LLVM=1` and U-Boot call them by name, and the seed had none of them. - src/step_runner.cr: busybox's Kconfig step also disables MODPROBE_SMALL. That defconfig default builds a depmod that ignores -b, so `make modules_install` could not write modules.dep into a staging tree. The full depmod honors -b. Checked by building busybox 1.36.1 with the patched config and running `busybox depmod -b STAGE 6.12.0`, which wrote modules.dep. Trunk had already fixed the seed's other two gaps after 0.4.0: /etc/ld-musl-ARCH.path listing /usr/lib/TRIPLE, and clang defaulting to compiler-rt, libunwind and libc++. Cleave needs a seed newer than 0.4.0 to get those fixes.

Demonstrates: spec/sysroot_builder_spec.cr ("makes the seed clang default to compiler-rt and builds llvm-objcopy/strip", "links cc and c++ to clang in the seed") and spec/step_runner_spec.cr ("busybox strategy turns off MODPROBE_SMALL so depmod honors -b"). crystal tool format --check, crystal spec (172 examples, 0 failures, 4 pending) and shards build all pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCfptovmnPWyqTgXoZKNdx check-in: c138545e79 user: run-58530641 tags: agent/run-58530641-4d9e2e9d

Based on session/7d18ba12...-userns-probe-uses-crystal-system-process tip 761197aeee (the branch the maintainer is testing on slotcar), merged into this session branch rather than starting from trunk.

Two independent problems from the maintainer's first full sysroot-runner run since the userns rewrite (ticket d3a1b086ab):

1. bdwgc's Makefile.in goes missing between the sysroot-from-seed phase's bdwgc step (14:18 in his log) and the system-from-sysroot phase's re-entry into the same on-disk workspace directory (15:28) -- both phases build gc-8.2.6 in what is physically one shared directory (seed-rootfs/bq2-rootfs/workspace, mounted at different paths per namespace), so something between those two points deletes or makes unreachable a file phase 1 left in place.

Added SysrootRunner#log_workspace_watch, called at phase entry and after every step, that logs presence/inode/size/mtime for a small watch list (gc-8.2.6/Makefile.in) -- or, when the file can't be found, the ownership of the nearest existing ancestor via the existing SysrootNamespace.ownership_diagnostic, to tell a genuine deletion apart from a path that became unreachable under the single-entry uid mapping (both look identical to File.exists?). This will name the exact step boundary and cause on the maintainer's next run.

I could not reproduce the loss locally: a fresh extraction of the real gc-8.2.6 tarball, patched, configured, built, and installed (phase 1), then patched again (idempotent, as PatchApplier reports), distcleaned, and reconfigured (phase 2) -- the same sequence step_runner.cr runs -- left Makefile.in intact throughout, including through `make distclean` (which I confirmed does not touch Makefile.in for this package). That rules out the clean_build/distclean logic as the cause in isolation. I don't have the means to reproduce the real userns/mount/pivot_root environment here (this container runs as root, bypassing the user namespace entirely) to test further.

One-line verdict for the ticket: inconclusive from the evidence available to me. The userns rewrite's own diff does not touch build/patch/extraction code (confirmed by re-reading it), and my local repro of that unchanged code does not reproduce the loss either -- but this container's pre-rewrite seed-rootfs tree (sysroot-build-state.json mtime 2026-08-31, both sysroot-from-seed and system-from-sysroot show bdwgc completed with Makefile.in intact on disk today) ran fully privileged (no user namespace at all), so it isn't a clean before/after comparison of the same mechanism. The instrumentation above is what will actually answer this on the next real run.

2. The resumed runner could not create /var/lib/sysroot-build-reports under its single-entry uid mapping. Root cause, not just the g13 fallback: SysrootBuildState#report_dir now resolves under the workspace instead of var/lib. The workspace is where source extraction and every build step's output already land successfully for the entire run (proven by the two full LLVM builds that got this far), unlike var/lib, whose ownership is less certain by the time a later phase's install steps have touched it. The workspace is already excluded from the shipped rootfs tarball, so this changes nothing about built images.

With report_dir reliably writable, SysrootRunner#write_report_or_fallback's second fallback location (write_report_or_fallback) is now dead for its original purpose and is removed; write_report just writes directly and logs+swallows on failure (still never crashes the runner, still surfaces ownership_diagnostic on failure). SysrootBuildState#save's own fallback-to-tempdir is untouched -- it covers state_path under var/lib, which this change does not move.

Also folded in: SysrootBuildState#save's fallback log line now includes SysrootNamespace.ownership_diagnostic, matching write_report, so a state write failure shows the same "who actually owns this" detail a report write failure does.

Updated README.md, AGENTS.md, and the two iteration skill docs that documented the old var/lib/sysroot-build-reports path.

Testing: - crystal tool format --check src/ spec/: clean. - shards build: clean, zero warnings. - crystal spec: 172/0/0 (4 pending, env-gated; +1 vs the branch this merges in, which was 171/0/0). - New/changed specs: sysroot_build_state_spec.cr's report_dir path assertion now points at workspace_path; sysroot_runner_spec.cr's existing "report/state directory unwritable" spec is split into two -- one confirming state alone falls back when only var/lib is blocked (report succeeds normally, no longer sharing that failure mode), and a new one confirming write_report's direct (fallback-free) failure path still doesn't crash the runner when the workspace itself is blocked.

Not gated yet -- reporting the branch per the coordinator's routing; holding for review before this goes to trunk-ops. Leaf check-in: dbab4b7013 user: agent-claude tags: session/7d18ba12b6b41d47857a4ea962a1e564eb440cb8f8e13be2fdbbbee32424662b-g14

The real 1.21 sysroot build hit a genuine build error (PatchApplyError from File.read failing on a patch path) that propagated straight past run_runner's only rescue clause (AltCommandExit) to Crystal's default top-level handler, which dumps a raw "Unhandled exception" stack trace and exits. run_steps already logs the specific failure, writes a report, and records it in build state before re-raising by design (to mark the run failed) -- the missing piece was converting that re-raise into a clean, leveled log line at the CLI boundary instead of letting it crash out uncaught. run_runner now has a second rescue ex that logs via Log.error(exception: ex) and returns exit code 1.

This is intentionally generic: it isn't specific to PatchApplyError, since run_steps's existing rescue ex funnels every non-AltCommandExit step failure through the same re-raise point (checksum mismatches, command failures, missing files, etc.), so a single top-level catch covers all of them uniformly.

Also extracted SysrootNamespace.ownership_diagnostic(path), generalizing the ownership/permissions diagnostic StepRunner already had for zero-output command failures, and wired it into SysrootBuildState#save's and write_report_or_fallback's existing write-failure log lines. Both already fall back to a world-writable temp path and log clearly when a rootfs location isn't owned by the namespace's single mapped uid (this is the documented, working-as-designed behavior from the prior 4638def1 fix, not a new bug) -- the diagnostic just answers "owned by whom" on the spot instead of requiring another round trip to find out.

Investigated the missing-patch-file report itself: the llvm-project cmake-guard-cxx-compiler-id.patch file is (and has long been, since January) checked into the tree, and bootstrap_repo_dir builds its path correctly, so this isn't a path bug in bq2. The plan bakes an absolute path to a self-emitted bootstrap-qcow2-CHECKIN source snapshot (SysrootBuilder#prepare_bootstrap_source!/emit_bootstrap_source_tarball) at plan-write time, and the source-extraction step is skipped on resume when its directory already exists (by design, to avoid re-extracting a full LLVM tree every run) -- the most likely explanation is a stale extracted snapshot left over from an earlier point in this multi-day validation cycle. Not reproducible from here without the maintainer's actual on-disk bootstrap-qcow2-CHECKIN state; flagged for them to check directly (fossil finfo the checkin the plan was written against, and whether the workspace's extracted source tree matches it) rather than guessed at further.

Testing: crystal tool format --check clean; shards build clean, zero warnings; crystal spec 171/0/0 (4 pending, env-gated). New regression: sysroot_runner_spec.cr "fails cleanly instead of crashing with a raw backtrace when a step raises an unexpected exception" drives the real CLI entry point (SysrootRunner.run with --workdir and --no-report, command "sysroot-runner"), not the in-process run_plan double, with a step whose patch path doesn't exist, and asserts it returns exit code 1 instead of raising -- reproducing the exact crash class from the maintainer's log. Ran the built binary's sysroot-namespace-check afterward to confirm the unrelated fork+exec userns path (from the earlier fix in this ticket) is untouched: unshare(CLONE_NEWUSER) probe still reports ok.

Regressions checked: none. The write-fallback/report behavior from 4638def1 is unchanged (still exercised and passing in sysroot_runner_spec.cr's existing "does not crash when the report/state directory cannot be written..." spec); this change only adds a rescue around what was previously an unhandled propagation path, plus generalizes an existing diagnostic helper -- no behavior changes for any run that currently succeeds.

Still open: the maintainer's Crystal 1.21 sysroot-runner run remains the acceptance bar for this ticket. This commit does not change the fork+exec namespace-entry mechanism validated earlier in this ticket; it only makes step failures during that build fail cleanly.

Co-Authored-By: Claude Sonnet 5 (noreply@anthropic.com) Leaf check-in: 761197aeee user: agent-claude tags: agent/20260914T231541Z-userns-probe-uses-crystal-system-process

Integrates session/7d18ba12...-g13 tip 4638def1 (design doc: docs/design/userns-namespace-entry.md, ACCEPTED, maintainer-confirmed 2026-09-10). Replaces the retired ensure_user_namespace!/probe-reexec family (g11) with the runc/bubblewrap idiom: exactly one fork() at user-namespace creation. sysroot-runner forks only when the plan needs a non-host namespace and CAP_SYS_ADMIN is not already held; the single-threaded child does raw libc syscalls only (close_range, unshare(CLONE_NEWUSER), uid/gid map writes, execve of --resume DESCRIPTOR) with all heap data prebuilt in the parent before the fork; the resumed runner re-initializes Crystal inside the new namespace and every later host->seed->bq2 boundary stays in-place.

Also (4638def1): version-gated Process::Status constructor (warning-free on 1.21); SysrootBuildState#save and the failure-report writer never raise, falling back to a world-writable temp path when the mapped uid does not own the rootfs path; ownership/uid-gid diagnostics surface when a failed command captured no output; reproduce_seed_command routed through the version-safe monotonic_now helper.

Gated: format clean, shards build with zero warnings, full suite 171/0/0 (3 pending), integrity 0 errors. Live-verified the real sysroot-runner consumer end-to-end under genuinely dropped CAP_SYS_ADMIN (bounding set + capset): fork -> unshare -> uid/gid map -> execve -> resumed runner entered the seed namespace and completed a real package install inside it, no EINVAL, no crash. 1.18.2 cannot reproduce Crystal 1.21's multi-threaded startup floor, but fork()'s single-threaded-child guarantee is a kernel property independent of runtime version, so this exercises the same code path 1.21 will run. Maintainer-approved (merge 4638def1); the maintainer's own 1.21 sysroot-runner run remains the formal acceptance bar for d3a1b086ab. aarch64/x86_64.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> check-in: d539ac7c21 user: agent-trunk-ops tags: origin/codex-development-20260223, origin/codex/bookmark-state, origin/codex/fix-sysroot-runner-overwriting-issue-m0vpua, trunk

Ticket d3a1b086ab. Fixes the four items from the maintainer's 1.21 validation of 73a6a58a (real slotcar run, relayed via the coordinator): the compile warning is now clean, and the resumed runner no longer dies with an unhandled File::AccessDeniedError when a build step fails and the failure-report/state write can't reach its usual rootfs location.

1. Deprecation warning (sysroot_namespace.cr): `Process::Status.new` is deprecated on 1.21 in favor of the portable `Process::Status.[]`. Added `process_status_for`, a version-gated helper (same {% if %} pattern as the existing monotonic_now helpers) that uses `.[]` when the running Crystal provides it and falls back to `.new` on 1.18. `shards build` is warning-free on both.

2. REAL BUG -- resumed runner crash on write failure: - `SysrootBuildState#save` (sysroot_build_state.cr) now never raises. On a write failure it falls back to `SysrootBuildState.fallback_path` -- a location under the system temp dir (the active namespace's own tmpfs mount once pivoted, world-writable) that every uid can reach regardless of which id ends up owning the rootfs location that failed -- and logs where the state actually landed, to both Log and stderr. This covers mark_success/mark_failure/mark_current_phase/invalidate_progress!, all of which funnel through save. - `SysrootRunner#write_failure_report` (sysroot_runner.cr) uses the same fallback via a new `write_report_or_fallback` helper instead of just giving up on any write error, so a failure report actually gets written somewhere inspectable instead of only logging a WARN and losing it. - Belt-and-suspenders: the `mark_failure` call site in `run_steps` also rescues directly, so even an unexpected failure recording the failure cannot mask or crash out of the original build exception, which always propagates and drives the runner's non-zero exit. - Diagnostics for the "why did it fail" half: a command that fails with zero captured output (the observed zlib `make -j12` symptom: exit=2 in 5ms, no stdout/stderr) previously logged nothing else useful, because `capture_tail` returns nil for both a missing path and empty output -- indistinguishable from the log alone. `StepRunner#ownership_diagnostic` now reports the runtime uid/gid (SysrootNamespace.current_uid/gid, i.e. the mapped-namespace values) alongside the failing cwd's owner uid/gid and mode, so a single-entry uid-mapping mismatch is visible directly in the failure message instead of manifesting as an unexplained empty-output failure. I could not reproduce the zlib failure itself here (Crystal 1.18's single-threaded default hides the multi-thread constraint this whole rewrite is about, and I don't have a genuine unprivileged multi-uid host); this instrumentation is what will show the actual cause on the next real run instead of another silent "exit=2, no output".

Testing: - `crystal tool format --check`: clean (no warnings/changes in tracked source). - `shards build` / `crystal build src/main.cr`: clean, zero warnings (was one Process::Status deprecation warning before this commit). - `crystal spec`: full suite 170/0/0 (4 pending, env-gated; +2 vs 73a6a58a). New specs pinning this fix: - spec/sysroot_build_state_spec.cr "#save ... falls back to a writable location when the target directory cannot be created": replaces the log dir with a plain file (ENOTDIR regardless of privilege level, so it's deterministic even run as root) and asserts save() does not raise and the state lands in the fallback with progress intact. - spec/sysroot_runner_spec.cr "does not crash when the report/state directory cannot be written, and preserves the original failure": same technique at the SysrootRunner level; asserts run_plan still raises the ORIGINAL "Command failed (23)" (not a write error) and that both the state and the failure report land in the shared fallback directory. - Live functional check on the real sysroot-runner path (not the sysroot-namespace proxy), cap-dropped (CAP_SYS_ADMIN removed via capget/capset) to force the unprivileged fork+exec path: confirmed (a) the fork+exec handoff still works after these edits -- child enters the namespace and execs --resume with no unshare EINVAL -- and (b) a real step failure (missing ./configure, not a mock) inside the resumed process goes through the actual write_failure_report + mark_failure code paths, writes the report and state normally, and exits non-zero carrying the original exception message.

sysroot_namespace.cr size is left alone per the coordinator's note (a separate, future unit).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Closed-Leaf check-in: 4638def1a3 user: agent-claude tags: session/7d18ba12b6b41d47857a4ea962a1e564eb440cb8f8e13be2fdbbbee32424662b-g13

Ticket d3a1b086ab. Crystal 1.21's default runtime is multi-threaded, and the kernel refuses unshare(CLONE_NEWUSER) from a multi-threaded process (EINVAL); spike2 proved even GC-markers-off floors at 2 threads, so the in-place approach (and the retired -g11 probe/re-exec-entry family) cannot work. Replace it with the runc/bubblewrap idiom: exactly ONE fork() at user-namespace creation.

sysroot_namespace.cr: - SysrootNamespace.reexec_in_user_namespace prebuilds the exec argv/env and uid/gid-map payloads in the (safe) multi-threaded parent, forks once, and the single-threaded child runs RAW libc only -- close_range(3,~0) via syscall(2), unshare(CLONE_NEWUSER), write uid_map/setgroups=deny/gid_map, execve of sysroot-runner --resume DESCRIPTOR. No malloc/GC/Crystal IO between fork and exec (other threads' locks are frozen in the fork copy); GC.disable guards the pre-exec window. Parent waitpid + propagates the exit code. close_range goes through syscall(2) because this musl links no wrapper. - Retire the in-place cruft: the USERNS_PROBE_ENV re-exec probe, the load-time exec_probe_child_if_requested hook, and the env switch. probe_user_namespace now uses the same fork idiom (child unshare + _exit errno). Per-boundary host->seed->bq2 transitions stay in-place (CLONE_NEWNS + mount + pivot_root): the re-exec'd runner holds CAP_SYS_ADMIN inside the new namespace, so privileged_mount_only? is true and no second fork occurs.

sysroot_runner.cr: - run_runner hands off to reexec_in_user_namespace when the plan needs a non-host namespace and we lack CAP_SYS_ADMIN; on --resume it reconstructs the run from the descriptor (the only channel across the exec boundary) plus the on-disk plan/state and runs in-process. Root and --dry-run keep the direct path.

docs/design/userns-namespace-entry.md: the accepted design -- one fork, the raw child sequence, the blanket close_range (no enumerated fds; bq2 opens no SSE/ epoll/eventfd/DB back-channel), and argv-only state transfer.

reproduce_seed_command.cr: run_phase uses a version-safe monotonic_now helper (Time.instant on newer Crystal, Time.monotonic otherwise) -- nit re-applied on this generation.

Verified on Crystal 1.18: format clean, shards build ok, full suite 168/0/0 (4 pending, env-gated). Functional check on the real sysroot-runner path under a cap-dropped launcher: parent forks, child enters the user namespace + maps, execs --resume, the resumed runner logs the handoff and does the seed NEWNS+mount+pivot in-place, with NO unshare EINVAL; as root it skips the fork. The 1.21 MT gate is the maintainer's (cannot be reproduced on 1.18's single-threaded default). aarch64/x86_64.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> check-in: 73a6a58af7 user: agent-claude tags: session/7d18ba12b6b41d47857a4ea962a1e564eb440cb8f8e13be2fdbbbee32424662b-g13

The -g11 fix only covered the sysroot-namespace CLI (re-exec + exec leaf). The sysroot-runner path calls enter_rootfs IN-PROCESS after host phases already spun ~14 threads (Log + Process.run), so unshare(CLONE_NEWUSER) still hit EINVAL there.

Converge on the real constraint: only unshare(CLONE_NEWUSER) needs a single-threaded process; unshare(CLONE_NEWNS), uid/gid map writes, mount, and pivot_root are all MT-safe. - New SysrootNamespace.ensure_user_namespace!: creates the user namespace and maps uid/gid exactly once, idempotently, and asserts single-threaded. No-op when already privileged (CAP_SYS_ADMIN). - unshare_namespaces now = ensure_user_namespace! (idempotent) + unshare(CLONE_NEWNS) + make private. So per-phase rootfs entry no longer creates a user namespace. - sysroot-runner calls ensure_user_namespace! up front in run_runner, right after arg parsing and BEFORE log_build_provenance/workspace setup spin any worker thread. Every later per-phase seed/bq2 entry is then just a mount-ns transition, safe from the multi-threaded runner. The re-exec CLI entry child still works: its fresh single-threaded process creates the userns inline via unshare_namespaces. - assert_single_threaded_for_userns! kept (now guards only ensure_user_namespace!).

Nit: reproduce_seed_command Time.monotonic (3 sites, deprecated on 1.21) routed through a new version-safe monotonic_now helper (mirrors sysroot_runner). libc/ syscall wrappers retain their docs.kernel.org references.

Verified on this 1.18.2 host by running the ACTUAL sysroot-runner consumer path, not the proxy: (1) default build enters the seed namespace + runs a step inside it; (2) unprivileged (CAP_SYS_ADMIN dropped) run creates the userns up front and enters the seed namespace; (3) -Dpreview_mt build (confirmed multi-threaded, probe EINVAL) enters the seed namespace WHILE multi-threaded via the mount-ns path without crash; (4) -Dpreview_mt unprivileged hits the guard with the clear message. Full suite 166/0 (4 pending), format clean. 1.21 lazy-MT timing (the maintainer env) matches the probe path proven single-threaded there. Session branch -g11. x86_64/aarch64. Leaf check-in: 12bcda9205 user: agent-claude tags: session/7d18ba12b6b41d47857a4ea962a1e564eb440cb8f8e13be2fdbbbee32424662b-g11

First tranche of the BQ2 1.0 functional-parity corpus, following the maintainer's 3-surfaces correction. New sysroot-functest command mirrors reproduce-seed: fast CI-safe --list/--matrix inspection modes plus a --sysroot execute mode that runs each case in the built rootfs via sysroot-namespace and diffs stdout/exit against an exact reference.

Corpus (sysroot binaries): grep-exact, sed-substitute, tar-roundtrip, cc-compile-link, crystal-hello (a user program, not a self-rebuild).

Matrix records the per-item split across the bq2 sysroot and the three ocx surfaces (native tools, mcp-server, agent-tool CLI). Native-agent loop is ocx-only, to be tested against a canned-response stub service so the suite stays reproducible and offline; real-ollama runs remain point checks.

Adds FUNCTEST.md and spec (12 examples). Primary arch aarch64; host-agnostic.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com Leaf check-in: cd8783522f user: agent-claude tags: session/a89ee18a41cb63aa0ab5832aadd29e539dbb7a068287a0ffdb17fd1af132bb54

Completes the var/lib build-bookkeeping exclusion: the empty (on success) sysroot-build-reports dir was the one remaining build-state member shipping in the seed. Re-emit of the existing gen-B/gen-C trees confirmed workspace=0 members and the differing shipped-file set collapsing 191 -> 2 (only bin/busybox + usr/bin/fossil, the SOURCE_DATE_EPOCH targets a rebuild resolves). Seed shrank 5.1 GB -> 1.29 GB. Suite 167/0. Session branch -g12. Leaf check-in: e9dd87ad0a user: agent-claude tags: session/7d18ba12b6b41d47857a4ea962a1e564eb440cb8f8e13be2fdbbbee32424662b-g12

Three parts toward byte-identical seed_b == seed_c (maintainer word 2026-09-03):

1. workspace-drop: the seed tarred the entire bq2-rootfs, so the build workspace (source checkouts + compile intermediates inside the rootfs) shipped by default -- most of the 5 GB and 186 of the 191 B->C diffs. TarWriter.write_gz gained an exclude option; run_tarball excludes the workspace dir. A fresh generation re-extracts its own sources, so nothing needs the previous gen workspace. Also excludes the host-written var/lib build bookkeeping (plan, resume state, overrides, bootstrap-source ledger): run-specific absolute paths, regenerated per build, not deployed content (the 3 var/lib path diffs).

2. SOURCE_DATE_EPOCH (2024-01-01) threaded into every build phase env so packages embedding a build timestamp (busybox banner, __DATE__/__TIME__) go deterministic -- the remaining ~2 shipped-binary diffs (busybox, fossil). A build input in the plan env, not an ENV read in program logic.

3. -ffile-prefix-map=/workspace=. normalizes any build path embedded in binaries (defensive; binaries already build byte-identically at a consistent /workspace).

Suite 167/0 (4 pending), format clean. Zero-diff verification needs a full B->C reproduce run. Session branch -g12. x86_64/aarch64. check-in: edc2126fe3 user: agent-claude tags: session/7d18ba12b6b41d47857a4ea962a1e564eb440cb8f8e13be2fdbbbee32424662b-g12

PRIMARY regression: bin/sysroot-namespace crashed 'unshare failed: EINVAL' from the real enter_rootfs path (unshare_namespaces -> unshare!(CLONE_NEWUSER)). The kernel forbids creating a user namespace from a multi-threaded process, and on Crystal 1.21 the binary runs the default multi-threaded execution-contexts runtime. The probe was already fixed (re-exec runs single-threaded), but the ACTUAL entry ran in the multi-threaded main process.

Fix (maintainer option 2): run_namespace no longer unshares in-process. It builds an EnterRequest (rootfs, binds, home, env, command, alpine-setup) and re-execs bq2 with the hidden USERNS_ENTER_COMMAND subcommand, threading everything through argv JSON -- never an env flag. The re-exec'd child's minimal main->dispatch->unshare path is still single-threaded (Crystal spins worker threads lazily on the first fiber), so unshare(CLONE_NEWUSER) succeeds -- the same technique that makes the probe work. The child then maps uid/gid, mounts, pivots, and execs the target (replacing itself); parent inherits stdio and returns the child's exit code.

Also adds a defensive assert_single_threaded_for_userns! (reads /proc/self/status) before unshare(CLONE_NEWUSER): if a thread was somehow spun first, it fails with an actionable message instead of a cryptic EINVAL.

Hidden subcommand absent from help/symlinks; no env behavior switch; no top-level import side effect; no bare fork. Verified on this 1.18.2 host: the re-exec entry runs enter_rootfs end-to-end (pivot + exec of the target command), the full suite 166/0 (4 pending) incl. the cap-dropped userns integration test, format clean. The 1.21-specific multi-threaded timing cannot be reproduced on 1.18.2 (its -Dpreview_mt spins eagerly, unlike 1.21's lazy contexts) -- final confirmation should run on the maintainer's 1.21 env, where the child's dispatch->unshare matches the working probe. Session branch -g11. x86_64/aarch64. check-in: 26c6d077b8 user: agent-claude tags: session/7d18ba12b6b41d47857a4ea962a1e564eb440cb8f8e13be2fdbbbee32424662b-g11

Removes the two anti-patterns the maintainer flagged in c86183a3's userns re-exec probe, both in the definitions file src/sysroot_namespace.cr: 1. Load-time import side effect: the bare top-level statement 'Bootstrap::SysrootNamespace.exec_probe_child_if_requested' ran on every require (bq2, tools, specs). Deleted. 2. Env behavior switch: BQ2_USERNS_PROBE_CHILD decided whether the process silently became a throwaway probe child. Deleted.

Replaced with a hidden, dispatch-only CLI subcommand (USERNS_PROBE_COMMAND = 'sysroot-namespace-userns-probe'), routed by the CLI dispatcher at the one real main entry (CLI.run), never at file load: - probe_user_namespace(executable = Process.executable_path) re-execs with a single explicit argv (no env); executable is overridable so tests drive the built bq2, not the spec runner. - run_userns_probe_child does the unshare(CLONE_NEWUSER) + _exit(errno). - SysrootNamespace.run routes the subcommand; it is absent from help_entries and CLI.hidden_from_symlinks? now hides any command not advertised in help, so --install creates no bin/ symlink for it. - namespace_check_report takes an optional injected probe; specs inject a known result or point probe_user_namespace at bin/bq2, so no binary is ever hijacked.

1.21 fork-free goal preserved (no bare fork in src/ or spec/). Verified: the built bq2's re-exec probe returns ok (exit 0); the hidden subcommand is absent from help. Full suite 166/0/0 (4 pending), format clean. Forked from trunk 6f9e2fa2. x86_64/aarch64. check-in: ddf00a8501 user: agent-claude tags: session/7d18ba12b6b41d47857a4ea962a1e564eb440cb8f8e13be2fdbbbee32424662b-g11

Integrates session/7d18ba12...-g10 tip 41cda3bd. The B->C reprocheck proved the writer deterministic and the two gen trees content-identical, but archives still differed at the first header's mtime field: every entry's mtime reflected the build wall-clock (~4h gap between gen-b and gen-c). TarWriter.write_gz now takes an optional mtime that forces every entry's timestamp; sysroot-tarball passes Time.unix(0) so seeds are byte-reproducible regardless of build time. Callers wanting real mtimes omit the arg (preserve-mtime spec still passes). Spec: two pinned-mtime emits over changed on-disk mtimes yield identical archives. Merge-result gated on trunk in an isolated clone: format clean, shards build ok, full suite 167/0/0 (3 pending), integrity 0 errors; 1.21 userns fix preserved (0 bare fork). Merge on green per coordinator. aarch64/x86_64. check-in: 6f9e2fa259 user: agent-trunk-ops tags: origin/codex-development-20260223, origin/codex/bookmark-state, origin/codex/fix-sysroot-runner-overwriting-issue-m0vpua, trunk

The B->C reprocheck proved the writer is deterministic (b1==b2 byte-identical) and the two gen trees are content-identical, but the archives still differed. cmp of the decompressed tars located the first difference at byte 143 -- the mtime field of the very first header -- and direct stat confirmed EVERY entry's mtime (files and dirs) reflects the build wall-clock, differing by the ~4h gap between the gen-b and gen-c builds. (Earlier find-based mtime scans were false negatives: busybox find lacks -printf %T@.)

Content was already fully reproducible; only mtimes differed. TarWriter.write_gz now takes an optional mtime that forces every entry's timestamp; sysroot-tarball passes Time.unix(0) so seeds are byte-reproducible regardless of build time. Callers that want real mtimes omit the arg (preserve-mtime spec still passes). Spec added: changing on-disk mtimes between two pinned-mtime emits yields identical archives. Suite 166/0. Session branch -g10. aarch64/x86_64. Closed-Leaf check-in: 41cda3bd15 user: agent-claude tags: session/7d18ba12b6b41d47857a4ea962a1e564eb440cb8f8e13be2fdbbbee32424662b-g10

Integrates session/7d18ba12...-g9 tip fce41575. Closes the last ~16KB gap to bit-identical seeds (B->C parity -> match): - TarWriter.write_gz pins the gzip header modification_time to Time.unix(0) (was the build clock) and #walk sorts Dir.children so member order is deterministic. Spec: write_gz twice is byte-identical. - reproduce-seed gains --from-seed (like-for-like BQ2->BQ2 bit-repro, e.g. seed_b -> seed_c) and --source-checkin (pin in-image source across gens). Merge-result gated on trunk in an isolated clone: format clean, shards build ok, full suite 166/0/0 (3 pending), integrity 0 errors; Crystal 1.21 userns fix preserved (0 bare fork in src/spec). Merge on green per coordinator. aarch64/x86_64. check-in: 9dea46702d user: agent-trunk-ops tags: origin/codex-development-20260223, origin/codex/bookmark-state, origin/codex/fix-sysroot-runner-overwriting-issue-m0vpua, trunk

The B->C bit-reproducibility run (job 0b75032a) came back 'parity', not ':match': seed_b vs seed_c differ by only 16199 bytes / 5.1GB. Root-caused to the tarball writer, not the toolchain -- all 408110 members have identical names, sizes, and metadata, and spot-checked usr/bin/bq2 is byte-identical across the two seeds. Two non-determinism sources remained: - TarWriter#walk emitted members in filesystem readdir order (varies per build); now sorts Dir.children so member order is deterministic. - write_gz let the gzip header modification_time default to Time.utc (the build clock); now pinned to Time.unix(0).

Spec added: write_gz twice -> byte-identical archive. Suite 165/0. This should make a future B->C chain reach :match. Session branch -g9. aarch64/x86_64. Closed-Leaf check-in: fce4157598 user: agent-claude tags: session/7d18ba12b6b41d47857a4ea962a1e564eb440cb8f8e13be2fdbbbee32424662b-g9

Threads a --source-checkin flag through the builder phases (both A->B and --from-seed). A like-for-like seed_b -> seed_c comparison must build gen-C from the SAME bootstrap-qcow2 source as gen-B, otherwise seed_c differs from seed_b merely because the driving binary's source changed (e.g. adding the --from-seed harness code, which compiles into the in-image bq2). Pinning the source check-in isolates determinism as the only variable. Suite 164/0. Session branch -g9. aarch64/x86_64. check-in: 709fffdbfc user: agent-claude tags: session/7d18ba12b6b41d47857a4ea962a1e564eb440cb8f8e13be2fdbbbee32424662b-g9

Integrates session/7d18ba12...-g9 (tip 2850d011). Fixes the reproduce-trunk chain failure where a sandboxed job could not run fossil (repo outside the sandbox), so source-checkin fell back to the release version, emission was skipped, and download-sources crashed with 'Local source not found'. Now: - bootstrap_source_checkin falls back to compile-time BUILD_REVISION when fossil info cannot run, keeping the deterministic artifact name. - source tarball moves to a per-run shared cache (run-root/source-artifacts) so both chain generations share one artifact and it can be pre-staged once. - prepare_bootstrap_source reuses an existing artifact (hash+pin, no fossil); else emits via fossil in a checkout; else warns naming the path to pre-stage. Backed by the g9 8.3h end-to-end chain. Merge-result gated on trunk in an isolated clone: format clean, shards build ok, full suite 165/0/0 (3 pending), integrity 0 errors; the Crystal 1.21 userns fix is preserved (0 bare fork in src/spec). Maintainer-reviewed g9 report. aarch64/x86_64. check-in: 7a5c5bf344 user: agent-trunk-ops tags: origin/codex-development-20260223, origin/codex/bookmark-state, origin/codex/fix-sysroot-runner-overwriting-issue-m0vpua, trunk