| Ticket Hash: | 98640d84c4a26a6773a60eb39fe6edc4cc364c6e | ||
| Title: | Ollama container: set OLLAMA_MAX_LOADED_MODELS>=2 + KEEP_ALIVE so embed+summary models co-reside (vec-rebuild thrash) | ||
| Status: | Closed | Type: | task |
| Severity: | Normal | Priority: | medium |
| Subsystem: | runtime | Resolution: | Overcome_By_Events |
| Last Modified: |
2026-11-09 21:53:27 34.3 hours ago |
Created: |
2026-07-06 23:28:23 127.4 days ago |
| Version Found In: | |||
| Description: | ||||
|
Runtime/deployment tuning for the Ollama Docker container. Observed 2026-07-06 during a vec-rebuild --summarize: the embedder (nomic-embed-text, ~0.27G) and the summary model (qwen2.5-coder:7b, ~4.7G) were MUTUALLY EVICTING each other on the GPU — /api/ps mostly showed 0 models, cycling one-at-a-time, and a model with a long keep-alive still got evicted for the other. That is the signature of OLLAMA_MAX_LOADED_MODELS=1. VRAM is NOT the constraint (the host runs 19-51G models on GPU, so >=20G; the two rebuild models total ~5G and co-fit trivially). Cost: each summary call reloads qwen, roughly doubling the summary phase over ~300 summaries.
Context: the assistant/agent runs in a container with API-only reach to Ollama (no docker socket), and Ollama runs in its OWN docker container — so the setting is NOT in host env files; it lives in the ollama container environment (compose environment: / docker run -e). Recommendation (apply on the host, recreate the container — plain restart will not pick up new env; do it when no rebuild is in flight): - OLLAMA_MAX_LOADED_MODELS=2 (keep embedder + summary model both resident) - OLLAMA_KEEP_ALIVE=10m (do not unload between the spaced summary calls) Optionally OLLAMA_NUM_PARALLEL as appropriate. Deliverable: document this in docs/runtime-setup.md (Ollama container tuning for the vec-rebuild --summarize workload, which alternates the embed + chat models) + verify the effect (both models stay resident in ollama ps during a --summarize run; summary phase wall-time drops). Relates to the long-job observability ticket c83c1450a6. | ||||
| User Comments: | ||||
|
added on 2026-11-09 21:53:27:
Closed in OCX's ticket sort, batch 6 (ollama post 8f6c9afbb7): tracked under the semantic-search row of docs/parity-v1.md (0e2d18fbea), held: the maintainer chose Fossil's full-text search first. Reopen this when semantic search is taken up. | ||||