The managerial-level map of how this system fits together. Definitions here are the single source of truth — TODO.md, BACKLOG.md, tickets, and notes link to these anchors instead of re-explaining. Renders with diagrams via Fossil at /doc/trunk/docs/architecture.md.

Why this project exists

It started to get codex working with a local ollama model and actually do useful work — codex struggled against a local model, and understanding why drove most of the design. The goal grew into: drive coding/chat work through a Fossil web UI, with the model being local Ollama or a hosted provider, behind one consistent tool layer.

Axioms

The three anti-failure axioms — Clarity, Interchangeability, and Accountability — are defined in README.md ("Why this exists"). That is the source of truth; this is a pointer, not a restatement.

They are not tasks and never "done" — they are how we judge whether the workbench is getting more robust or drifting toward failure. The seams and definitions below exist to serve them: this whole document is Clarity; the Model/Provider/AgentRunner seam and the interop boundaries are Interchangeability made concrete; the verify/report/evidence discipline is Accountability.

Vocabulary

The terms that kept getting conflated. Defined once, here.

• Model — an LLM endpoint: a provider + a model id. Interchangeable behind a Provider.
• Provider — configuration for ONE LLM model endpoint: base URL, API key, model id. A Provider is just "where a model lives," generic of usage: the same Provider can be a runner's primary, a stateless consultant, or the /chat model — usage is never a property of the Provider. It carries no agent type or capability metadata (ProviderConfig is a pure endpoint; old blobs' agent/capability keys are ignored). Who drives it and what they can do are the AgentRunner's, decided per Task — any Provider pairs with any runner.
• Agent / AgentRunner — the loop and strategy that drives a Model through tools to complete a Task or a Chat. It owns a session, a strategy, and a tool set. Implementations are pluggable behind one seam:
  • our native runner (tool-loop) and model-only runner (plain completion, no tools),
  • codex, claude, zeroclaw runners that wrap those external CLIs. Each runner is Model-Provider-agnostic; each knows how to load its own MCP servers, point at a provider, and save/resume its session. A Task names its runner explicitly (runner_id); the worker dispatches on the runner's type, never inferring it from the Provider. Session save/resume is resume-id-first and built for the claude runner (claude -p --resume <resume_id>, see sessions); the native runner's compressed-state resume is still open.
  • Workspace — a runner declares whether it works in an isolated session checkout (workspace: "session", the default — feature work) or ON the live primary tree (workspace: "primary"). The built-in trunk-ops runner is the only primary-tree runner: a Claude brain bound to /root/ollama on trunk, scoped to repository operations (merge / revert / GC), never feature development. runner_configs always MERGES the built-ins with the stored runner blob, so a stored config can override a built-in by id but can never make a wired built-in (claude, trunk-ops) vanish.
  • Consult is a runner strategy, not a Provider. Part of a strategy may be to consult a second model at controlled entry points (EP1 before the first patch, EP2 before failing). The consult is a stateless one-shot: the consulted Provider keeps no session state even within the run — only the primary model carries context forward; the answer is folded back into the primary's context. So a consult policy is just (a generic Provider to ask) + (triggers + token budget) (ConsultPolicy{provider_id, ep1, ep2, budget}), referencing the same Provider list as everything else. There is no special "consult provider" config.
• Capability — what a runner can actually do, derived from its type (AgentRunners.capability_for), never free text: tool-loop (y/n) · shell (y/n) · MCP (y/n) · read-only (y/n).
• Task vs Chat — a Task is agent work on an isolated branch that produces a report/commit; a Chat is an interactive exchange in Fossil's /chat. A chat request that needs code becomes a Task — see Conversation: Fossil /chat is the surface.

AGENT_CODEX is exactly the conflated, model-coined term these definitions retire: it merges the Codex product with "HTTP provider." It is replaced by an explicit AgentRunner (codex) + a Provider (the model endpoint) + derived Capability.

Conversation: Fossil /chat is the surface

The conversation lives in Fossil's own /chat — a single chatroom persisted in the repository database. There is no bespoke conversation store and no files on disk for it. This retires the file-based ThreadStore (a Thread/Turn model we invented) and the legacy SavedChat transcript store; both are being removed. The win is Clarity and using Fossil's native persistence instead of our own.

• FossilChat (src/fossil_chat.cr) — the only conversation store. Reads (recent_messages) and writes (post_message) the native chat table via fossil sqlite3 -R. One stream, no threads — durable threaded sessions are forum-backed. Humans post as their Fossil login; the agent posts directly as a labelled identity (no separate login).
• Chat agent loop — runs inside the long-lived task-worker -d daemon: it watches /chat for new messages, drives the local Model as a tool-loop agent, and posts replies back into /chat.

The agent's authority is split by what it may touch, and the split is enforced structurally by which tools it holds:

• Non-code Fossil actions, done inline — create/update tickets, edit wiki pages, read/search the repo, answer questions. These are native tools the chat agent has.
• Code changes go through a Task — the chat agent has no code-editing tools. The only way it can change code is a task-enqueue tool that kicks off the isolated-checkout Task process; it then reports the task id/link in chat. Code never changes inside the chat turn.

So a request in /chat is answered or acted on inline when it is non-code, and routed to the Task pipeline when it is code — and that boundary holds because the chat agent simply lacks the tools to do otherwise.

Forum-backed sessions

/chat is deliberately ephemeral — one global room, quick exchanges, no memory. A long-lived, resumable session is a different shape, and Fossil's /forum is its natural home: durable, threaded, one topic per thread. This is the concrete form of the AgentRunner session the milestone is about.

A forum thread holds a whole milestone's worth of work, interactively — one sustained collaboration in which a human and a capable agent reason together over time and spawn the many coding Tasks a milestone needs. It is not a wrapper around a single ticket. (The working prototype is exactly the kind of long, interactive, milestone-spanning session you have in a Claude terminal — many decisions, a dozen commits, one line of thought.)

**A forum thread is bound 1:1 to one AgentRunner + one session + one checkout.** One runner per thread is a hard rule — it is what makes the thread a cohesive line of thought (compaction may shorten it, but it stays one thread of reasoning).

The session brain is runner-pluggable — and aspires to be more than one option. The premium, key-free path is the claude runner driven via claude -p --resume <resume_id> on the Pro login (sanctioned; no API key) — sustained Claude-grade reasoning resumable across days, with resume_id holding the CLI's session id. (Spike-verified 2026-06-06: a fact set under --session-id <uuid> was recalled by a separate --resume <uuid> process, headless, on the Pro login; --output-format json returns result + session_id. Pattern: own a uuid → --session-id on turn 1, --resume after.) But the aspiration is explicitly not Claude-only: that other runners approach that functionality — the native local model pushed further with consult runs that lean on Claude/ChatGPT for the higher-order reasoning minimally and on budget (the project's reason for being), plus RAG-style context awareness to compensate for its smaller context window. Claude is one brain; the goal is a capable, cost-efficient local-driven session that consults the pro models only when it must.

Three conversation surfaces, not two

The forum is the durable log, not the real-time channel. Keep them distinct:

• /chat — ephemeral, global, quick. Unchanged (see above).
• The task page — real-time task execution: live model updates, events, the report. Already exists; stays the home of live task data.
• The session Thread UI — the live human↔agent surface within a session, a chat-style page at /ext/agent?session=<root-hash> (with a Sessions index at ?sessions=1). Built and shipped:
  • Transcript + compose — role-tagged bubbles (you / agent / other) with markdown bodies and a plain compose box (Enter sends; no Fossil reply/preview form). The page polls ?session_posts=<root> for new posts; the forum thread is the durable store behind it.
  • Human posts as themselves (cookie-forward) — "Send" forwards the human's existing Fossil login cookie (HTTP_COOKIE) to the forum web endpoint, so the post attributes to them, not the runner (ForumWriter.post_as_human). The runner replies as its own login.
  • Trigger — the daemon polls open bound threads (5s, armed by OLLAMA_CODEX_SESSIONS=1 or the DB flag config['ollama-session-poll']='1'); a human "Send" also drops a kick so the turn starts in ~1s instead of waiting for the poll.
  • Live streaming — the claude runner drives turns with --output-format stream-json; each message (thinking / tool_use / tool_result / assistant text) becomes a events.jsonl row tailed to the page over the per-session live-events SSE channel (session-<root>), so you watch the work tool-by-tool, then the transcript refreshes on turn_finished.
  • Push trigger evaluated & deferred (2026-06-07): a Fossil email/alert pipe trigger would need the whole alert subsystem and is backoffice-latency-bound (~60s) — slower than the poll/kick; the SSE channel above is the real-time path.
• The forum thread — the durable session record under the Thread UI. The runner's final reply lands here (the runner posts summaries of outcomes, answers, decisions, and links to real-time task data); live per-turn activity streams over SSE, not into the forum. It is the audit trail you can re-read months later.

Identity, binding, and storage

• Each AgentRunner configuration gets its own Fossil login. Contributions (commits, forum posts) are then attributable to a specific runner config — who/what changed the code. Posting to the forum requires a login anyway: forum posts are hashed control artifacts with no fossil forum send CLI and no plainly-writable table (unlike chat), so the agent reads forum artifacts directly but writes through the forum web endpoint as its login.
• The binding lives in the Fossil config table, keyed by the thread's root artifact hash: ollama-session:<root-hash> → {runner_id, runner_login, runner_type, branch, checkout, status, resume_id, last_post, workspace}. DB-backed, native, no files. runner_id resolves the AgentRunner (→ provider/model/workspace); resume_id is the runner's own handle (codex/claude); last_post is the cursor of the last forum post handled (so a poll never re-answers history or itself); workspace is session or primary (see checkout lifecycle). For the native runner the "session" is the thread's posts + checkout state.
• resume_id is cache, the forum thread is truth. The claude CLI prunes transcripts after ~30 days (and codex's rollout files can similarly go missing), so a --resume/exec resume against an old resume_id can come back "no conversation/rollout found" for an otherwise-healthy session. That failure is deterministic for that resume_id — no amount of retrying revives a pruned transcript — so SessionAgent#drive_claude_turn/drive_codex_turn treat it specially (StaleResumeError, ticket 7ddc2714ec): clear resume_id and re-prime from the thread immediately, in the same turn, as a fresh --session-id/exec call — never the transient backoff ladder in TaskWorker#handle_turn_error (which would misclassify this as a rate limit and escalate to needs_attention after MAX_SESSION_TURN_RETRIES). Losing the CLI's cache costs one re-prime, not a stuck session.
• Task ↔ thread linkage reuses the existing Task.thread_id field, now carrying the forum-root hash. The forum agent reuses the chat agent tool loop, but session-scoped: enqueue_task targets the session checkout, not a fresh one.

Session checkout lifecycle

Most sessions own one long-lived checkout on a session/<root-hash> branch (workspace: "session"). Unlike a one-shot Task (fresh checkout, discarded after), it persists — but it is still isolated from /root/ollama; only its lifetime differs. The exception is a primary-tree session (workspace: "primary", the trunk-ops runner), which binds to the live /root/ollama on trunk and exists to operate on the repository — so it has no isolated checkout by design.

• Cumulative by design. Tasks run in the session checkout and commit straight to the session branch. Every session carries cumulative-state risk (sometimes you just start a fresh thread); the mitigation is the existing revert tool, not avoidance.
• Two merge directions, both repeatable without closing the thread:
  • session → trunk — accept_and_merge. A session can afford heavier verification than a one-shot task (verify the branch in an isolated clone first), and merging does not end the session: it continues on its branch.
  • trunk → session — pull the latest trunk into the session checkout/branch.
  • **Today these run through the trunk-ops session** — a primary-tree Claude operator driven conversationally from the Thread UI (merge / revert / GC, with hard safety rails: no feature dev, no live shards build mid-merge, serialize merges). Dedicated one-click thread buttons are the remaining polish.
• Review-then-merge is not close-out. A thread may go through many review→merge cycles while staying open.
• Close-out is separate: on thread close, a final merge or abandon → sweep the checkout → mark the binding closed (re-openable). Idle sessions: sweep the checkout but keep the binding so the thread can be resumed. (Close-out automation is the open item; trunk-ops can do it on instruction now.)

One binary, several interfaces

There is a single compiled artifact, bin/ollama-codex, containing all native tools. Everything else is an interface onto that one tool collection — the same binary via argv0/mode dispatch and compatibility symlinks.

box "ext/agent   ext/docs   agent-tool   task-worker   mcp-server" "( interfaces )" fit
arrow
box "bin/ollama-codex" "one binary — all native tools" fit fill 0xeeeeee

• bin/agent-tool — CLI / operator interface (human supervision, scripts, specs). The native chat/session/task agents call these tools in-process (AgentTool.*); they do not shell out to this CLI. The external claude runner reaches the same tools over MCP (it's launched with --mcp-config pointing at bin/mcp-server rooted at its checkout — mcp__ollama-codex__*); only codex still reaches them by running the CLI (its MCP wiring is future work).
• bin/task-worker — automation interface; runs queued model Tasks (the AgentRunner host). Has two modes: a one-shot run that drains the queue and exits (the next run picks up a rebuild automatically), and a persistent -d daemon that stays resident to host the chat agent, the forum-session poll, and the live-events SSE server.
• bin/mcp-server — MCP interface for external assistants (Codex, Claude desktop); exposes the same AgentTool methods as structured tools (including session_discovery / session_bind / session_list).
• ext/agent — authenticated browser interface; the Agent Console plus the session Thread UI (?session=) and Sessions index (?sessions=). (ext/chat is a backward-compatible alias.)
• ext/docs — authenticated browser interface to the embedded API docs. (The docs are embedded at compile time via {{read_file}}, so build-docs must run BEFORE shards build or /ext/docs ships empty.)
• Live-events SSE sidecar — Fossil buffers CGI output, so real-time push can't come from the CGI. The persistent task-worker -d daemon runs its own TLS HTTP::Server on :8443 that tails an events.jsonl and streams each line (HMAC-token auth). It serves both per-task channels (the task page) and per-session channels (session-<root>, the Thread UI live stream).

Deploy discipline. The served CGI binary (/var/www/cgi-bin/ollama-codex) is a symlink to bin/ollama-codex, and the CGI spawns fresh per request — so for the CGI a rebuild is the deploy. The persistent -d daemon is different: it holds its compiled binary in memory and does NOT auto-update. Because its agents call tools in-process (not via the auto-rebuilt CLI), any deploy that changes daemon-run code requires an explicit rebuild then restart (task-worker --stop && task-worker -d) — never assume the daemon picks up a rebuild on its own.

Language-generic vs project-specific content

The same binary drives any Crystal+Fossil project (see the active milestone in TODO.md) — so every piece of data it carries has to declare which side of that line it sits on:

• Language-generic — stays compiled-in. Content that documents the Crystal language itself is identical for every project the binary can drive, so embedding it is a pure speed win.
  • The Crystal stdlib index (agent-docs/stdlib-index.json, built by bin/ollama-codex build-stdlib-index) → EmbeddedIndexes::STDLIB_INDEX_JSON. Powers crystal_stdlib_lookup without a file read.
  • The /ext/docs payload is technically this project's docs, but it is keyed by API surface that lives in the binary itself — so it remains embedded via {{read_file}}. Re-evaluate if a project's ext/docs ever ships independently.
• Project-specific — loaded from the target checkout at runtime. Anything that describes this project's symbols, identities, or process must live in the checkout, not the binary; otherwise pointing the binary at a different repo would still surface this project's content.
  • The agent-docs index (agent-docs/html/index.json) — AgentTool#load_index_type reads it from <root>/agent-docs/html/index.json, where resolve_index_path takes <root> from Config.deployment_checkout? || Dir.current, and parse-caches the root JSON. The MCP server resolves --index=PATH against the target root for the same reason.
  • Per-project runner customisations live in the target repo's Fossil config table (ollama-codex-runners), Provider configs (ollama-codex-providers), and session bindings (ollama-session:<root-hash>) — see the seam.
  • The LOCAL_CODEX_PREPROMPT / CLAUDE_PREPROMPT priming text in task_worker.cr is currently compiled-in but is project-specific in flavour; the open follow-up is to move it under agent-docs/ so each project ships its own.

agent-tool check-project prints both halves of this boundary (project_state_root, project_ephemeral_root, runners_config_key, etc.) so an operator can see what a re-pointed binary will resolve.

The Model / Provider / AgentRunner seam

box "Task / Chat" fit
arrow
box "AgentRunner" "strategy + session + tools" fit
arrow
box "Model" "via Provider" fit

A Task or Chat is handed to an AgentRunner. The runner picks a strategy, opens or loads a session, and drives a Model — reached through a Provider — using its tool set, until it produces a report (Task) or a reply (Chat). Swapping the runner (native / codex / claude / zeroclaw) or the provider (ollama / openai / anthropic / litellm) is a substitution at this one seam — Interchangeability made concrete. Running the same Task through two runners and comparing the evidence is Accountability at work.

The native runner's model call goes through ProviderChat.complete — one completion against the runner's Provider. So any Provider × the native runner holds for sessions and /chat, not just tasks — a native session can run on local Ollama or a remote model (e.g. OpenAI gpt-4o) through the same in-process tool loop. (Earlier, a native session was hardcoded to local Ollama and a remote provider's model name 404'd against it.)

The Provider owns its wire protocol

How to talk to an endpoint is the Provider's concern, not each caller's. A ProviderProtocol (one per api style — OllamaProtocol / OpenAIProtocol / AnthropicProtocol) is the single authority on a provider's wire format: auth headers, endpoint paths, request-body generation, and response parsing (chat text, thinking, usage, embeddings). ProviderConfig#protocol selects it (Ollama by endpoint, the rest by api style), and the façade — config.chat(messages, opts, repository:) and config.embed(input, repository:) — is the one door callers use; a single ProviderHTTP.post is the only HTTP seam. Consumers (HTTPModelProvider, ProviderChat, the tool-loop model call, the connectivity test, VecEmbedder) never branch on api style — they build a request via the protocol, post it, and parse via the protocol. Embeddings are genuinely multi-provider (Ollama /api/embed, OpenAI /v1/embeddings; Anthropic has no native endpoint and raises). This replaced the old shape where every caller pulled raw ProviderConfig fields and reconstructed the interface from data.

Interop boundaries

Every Process.run is a point where we hand a role to an external tool — a seam we could swap, test, or reimplement. bin/agent-tool interop-points monitors them: it classifies each call site by role (scm, crystal_toolchain, agent_runner, sandbox, archive, …) and splits two buckets:

• leaky — a role that has a native interface, called raw → route it through the native tool;
• uncovered — a role with no native interface yet → candidate to make one.

Roles worth naming as first-class interfaces: Scm (fossil), CrystalToolchain (crystal), AgentRunner (the loop), Sandbox (timeout). Note: leaky means candidate, not "must route" — a site is only safe to route once the native tool preserves the call site's behavior.