Share one inference engine across every lilbee process on the machine#547
Open
tobocop2 wants to merge 43 commits into
Open
Share one inference engine across every lilbee process on the machine#547tobocop2 wants to merge 43 commits into
tobocop2 wants to merge 43 commits into
Conversation
Two lilbee servers on one data dir silently compete: the second overwrites server.port and spawns a second engine fleet, doubling resident model memory. serve now holds an OS file lock next to server.port for its whole lifetime; a second instance waits up to ten seconds for a dying predecessor to hand off, then exits with a clear error. The kernel releases the lock the moment its holder dies, however it died, so a crashed or killed server leaves no stale state to repair. A signal-terminated server also left no trace in server.log, making killed sessions indistinguishable from crashes when reading a diagnostics bundle. The hard-exit handler now logs the received signal, and the fleet teardown logs whether the fleet was stopped (group count, duration) or left warm for the next launch.
…rvisors A supervisor that manages several data dirs (the Obsidian plugin's shared root) needs one-server-ever across all of them, not just per data dir. It can now pass LILBEE_EXCLUSIVE_SCOPE=<dir>: the server holds an OS lock there for its lifetime and names itself in a sidecar, so a refused start can report, or gracefully take over from, the holder. Refused starts exit with a distinct code (3) so supervisors can tell 'another server owns this' from a crash without parsing output. Replacing a running server no longer requires signalling foreign processes: POST /api/shutdown (token-authed) delivers SIGTERM to the server's own process shortly after the response flushes, so the fleet teardown and shutdown logging behave identically however the stop arrives.
The per-group SIGTERM grace was 10s and groups stop sequentially, so a pathological teardown (four hung llama-servers) could take 40s while a supervisor's stop grace and a successor's lock-acquire grace both assumed seconds. llama-server holds no persistent state, so an early SIGKILL is safe: the per-group grace drops to 2.5s, bounding the whole teardown near 10s, and the lock handoff grace rises to 15s so it always exceeds the worst-case teardown with margin. The budget chain is documented at both constants so the next change keeps it monotonic.
On a SIGTERM-driven shutdown the fleet stops on a dedicated thread while SystemExit unwinds the main thread, so serve()'s finally released the OS locks several seconds before the process actually exited. A successor could acquire them and start its fleet while the predecessor's models still occupied memory; with the eager fleet build, the successor's VRAM plan probe could land inside that window and under-plan permanently. serve now joins the teardown thread before releasing, so a held lock always means the memory is still owned, and the take-over handoff serializes in the kernel regardless of supervisor timing.
Live state files recorded pid/ports but carried an empty launches payload; only detach() persisted the per-role model/ctx/slots contract. A second lilbee that wants to use a running sibling's fleet needs that contract to know what the proxy serves without reverse-engineering it from /running. SwapManager now captures the launch payload at start, carries it forward through adopt, and writes it in every state write, which also lets detach() drop its payload parameter since the manager already holds the current contract.
States the principle behind lilbee's one process boundary (a separate process only where the kernel cannot share the resource through the filesystem), why the engine crosses it and the retrieval index does not, what in-process retrieval buys, and the coordination price deliberately paid for it.
The shared engine needs three primitives no process can corrupt by dying: a build lock so two simultaneous starts never both construct an engine, a per-process user lock the kernel releases on any death, and a last-out probe that try-acquires every peer lock, cleaning up after dead processes in passing and reporting whether any live peer remains. All three are directory-scoped and agnostic to what they front. No consumers yet; the fleet provider wires them in with the acquisition ladder.
Sharing a running engine is only safe between lilbee processes whose bundled engine build matches: a release is tested against the llama.cpp, llama-swap, and gguf-parser versions it ships, and behavior (chat templates, tokenizer fixes, server flags) varies across builds even when the HTTP surface does not. The engine wheel now carries its pinned source versions as a pin string, regenerated by the wheel build from engine-versions.env and asserted in sync by a test; engine_pin() resolves it at runtime, treating a bring-your-own llama-server path as its own identity and older pin-less wheels as their wheel version. Every state write records the pin, so a binder can refuse an engine built from sources its own lilbee never shipped.
bind() points a swap manager at a live engine's proxy with no ownership taken: the binder writes no state, and its shutdown merely drops the binding, leaving the engine serving whoever else uses it. Whether a bind is allowed is decided by the contract module: the engine must serve every configured (role, model) pair and carry the same engine build pin; values the planner derives (ctx, slots) are accepted from the running engine rather than recomputed, since they legitimately vary with GPU occupancy at plan time. No production callers yet: the acquisition ladder wires these in next.
…ditional engine stop Binding exists to skip planning, so the contract's wanted side becomes the configured (role, model) pairs rather than planned launches, whose derived fields vary with GPU occupancy. stop_engine() stops whatever a dir's state files record through the records themselves, never a process handle, so it works on engines this process did not build: the off switch behind lilbee engine stop and the last-user-out path.
…uild Every fleet start now runs the acquisition ladder under a cross-process build lock: bind to the machine slot's running engine when healthy, pin-equal groups cover every configured (role, model) pair; build into the empty slot otherwise; and overflow to the config root's private engine dir when the slot is occupied by an incompatible engine. Two simultaneous starts can never both build, and a departing last user (stop-if-last runs under the same lock) can never race an arriving binder. Engine lifetime moves to kernel-refcounted membership: each provider holds a user lock in every engine dir it uses; releasing last stops the engine unless keep_engine_warm opts into persistence, which now decides only whether the engine outlives lilbee. The idle TTL applies in every mode, so even a persistent engine releases weights when unused. A config change stops the shared engine for every user (peers rediscover) while keeping membership. The warm detach/adopt machinery this replaces is removed from the provider; reloads rebuild into the dir their groups already live in.
A shared engine can be restarted out from under a user: another process changes a model, or the engine dies. That surfaces as a connection-kind provider error; embed, rerank, and non-streaming chat now drop the swap refs and retry once, which routes the call back through the acquisition ladder to rediscover the new proxy ports or rebuild. Membership is already held, so rediscovery is cheap and race-free. One retry only; a second failure surfaces. Mid-stream cuts still surface to the caller as retry errors by design, and the next call recovers. All-replica-dead now takes the same path, so a fleet whose every replica vanished gets one rebuild before the emptiness is reported.
lilbee engine stop becomes the unconditional off switch: it stops whatever the machine slot and this root's private overflow record, whichever process started them, instead of only reaping detached warm fleets (find_detached_state, its last consumer gone, is deleted). The keep_engine_warm and idle-ttl settings copy, config comments, and the usage guide's engine-lifecycle section now describe the shared engine: one engine for every lilbee process, stopped when the last one exits, warm as the opt-in to outlive lilbee, TTL applying in every mode. The MCP server's parent-death handler releases engine membership before its hard exit, since os._exit skips atexit and a dying agent host must not leave the engine unaccounted for.
detach(), adopt(), the detached state flag, keep_detached, sweep_owned, owner-liveness checks, and live-sibling sparing all maintained one idea: that a fleet belongs to the process that spawned it. The ladder made that idea obsolete: engines are machine infrastructure discovered by contract, membership is kernel-refcounted, and the build lock guarantees a single builder per engine dir. Reaping simplifies to one rule that can never disagree with binding: an engine answering on its proxy is in use and spared, whoever started it; anything else is stopped through its record and cleaned. Owner fields are no longer written (legacy files still parse), and the teardown primitives keep direct tests now that the owner-flavored suites are gone.
The engine-lifecycle section now describes what ships: the machine slot, the acquisition ladder (bind by models + engine pin, build into the empty slot, overflow privately on incompatibility), kernel-refcounted membership with last-out stop, warm as the opt-in to outlive lilbee, the ttl applying in every mode, and the one reap rule that can never disagree with binding. The two honest costs (restart blast radius on config changes, the unauthenticated localhost proxy) get their lines, and the stale owner-era comments in the swap manager now describe pid segments as uniqueness rather than ownership.
Four concurrent opencode agents on one box, each on its own project root and per-agent HOME, all pinned to one machine engine slot. Phases: fixtures with per-project knowledge bases, a single-agent baseline, then four-way load, with assertions that make the shared engine load-bearing: servers two through four bind instead of building, exactly one llama-swap/llama-server set serves everyone, VRAM stays flat against the baseline, every seeded task verifies, no server crashes, and the engine stops with an empty membership dir after the last exit. Runs on a GPU pod after the opencode QA bootstrap.
Seeds the engine cache from the prebuilt cu124 wheel (pods never compile llama.cpp), runs the opencode QA bootstrap against this branch, then executes the harness and records the exit code under /root/harness-results.
The first harness run left eight llama-swap processes and 20GB of VRAM behind. Three root causes, each reproduced in isolation on the pod: Membership holds released on a different thread never released at all. filelock instances are thread-local by default, so a hold acquired on the fleet warm-up thread read as unlocked on the teardown path, skipped its own release, and then counted its own still-held lock as a live peer: stop-if-last never fired, for any front. The hold is now created with thread_local=False. An incompatible engine nobody uses used to push every arrival into private overflow forever. A fleet built while only some configured models were installed serves a partial contract; once its builder exited, it poisoned the machine slot. The ladder now replaces an incompatible incumbent that has no live user locks (machine and private dirs alike) instead of overflowing around it; overflow remains for incumbents actively in use. Catalog commands eager-warmed the fleet. model pull built the services container, which warms by default, so a pure download spawned an engine mid-pull, and with only the chat model installed yet, that engine was exactly the partial-contract poisoner above. pull, show, rm, and browse now suppress eager start like model list already did.
Seed the engine cache with executable bits (zipfile extraction drops them; a non-executable gguf-parser silently degrades planning to file-size estimates), advertise the served chat context to opencode so its requests fit the engine's slot size, stop servers through POST /api/shutdown with a TERM fallback, run the post-ingest engine stop from every project root, and reset each project's knowledge base and agent HOME so reruns start clean.
Blast-radius pass over the fix commits: architecture.md still described unconditional overflow on an incompatible slot, the bind helper's docstring implied coexistence the ladder no longer guarantees, and the harness extracted the server token in two places. Also pins that a process probing a dir where it holds membership sees itself as a live user: the hold and the probe construct filelock instances with different arguments for the same path, and a filelock upgrade changing its per-path singleton semantics must fail this test rather than silently delete a live lock.
Agent clients size num_predict from their own prompt estimate, which under-counts against the server tokenizer; run 2 of the 4-agent harness lost an agent the moment it read a whole doc into one turn, because the requested reservation left the un-droppable prompt no room even though the window had ~14k tokens spare. The window fitter now honors the requested reservation while it fits and falls back to the default generation room otherwise; llama-server stops at the context edge anyway, so a greedy request degrades output room instead of erroring. Context overflow still raises when even the clamped reserve cannot fit the system messages, tools, and final turn. Also stops the harness fixture from pointing documents_dir at the kb source dir, which made lilbee add copy the kb into itself.
CI's Linux legs caught what macOS's flock semantics hid: user locks are fcntl-based there, so two lilbee instances in one process (per-instance Lilbee objects under services_scope, exactly what the integration tests run) constructing separate FileLock objects on the same pid-named path either falsely succeed or trip filelock's deadlock detection, failing real ingests. User locks now share one process-wide reentrant instance per path (is_singleton, not thread-local): holds are counted, the lock file is unlinked only when the last in-process hold releases, and the liveness probe treats a path this process holds as a live user instead of reacquiring it. Covered with a genuine subprocess peer (in-process peers short-circuit on the shared instance, so only a real process boundary exercises the kernel-refusal path). Also unwraps the scope-refusal assertion in the serve test: the console hard-wraps at terminal width, and CI's long tmp paths split the directory name across lines.
Windows CI runs the LOCALAPPDATA arm, leaving the XDG default line uncovered there; the split is already excluded on the win32 side, so exclude its mirror too.
The integration suites kept deadlocking despite the singleton fix, and the survivor is a three-way interaction: filelock's deadlock-detection registry is thread-local, so a hold acquired on a pooled worker thread but released on the teardown thread orphans the worker's registry entry; once the released instance is collected and the pool reuses that worker, a fresh instance's infinite blocking acquire false-positives as a deadlock. Reproduced exactly with a pinned worker plus a collection between holds. The detection only applies to infinite blocking acquires, and the pid-named user lock has no legitimate contender (another process cannot share our pid; our own process re-enters the singleton), so a finite timeout both sidesteps the false positive and turns any genuinely pathological hang into a loud Timeout instead of a silent stall.
tobocop2
changed the title
tobocop2
marked this pull request as ready for review
July 18, 2026 19:20
The architecture sections and the new lock/ladder comments carried the reasoning story alongside the invariant; per the development guide, comments state the invariant and stop. Process-boundaries and engine- lifecycle sections in architecture.md cut to roughly half, multi- sentence comment blocks reduced to one or two lines, and the smell- trigger sweep confirms the branch adds none of the flagged patterns.
Three pod scripts that turn the shared-engine claims into measured results: setup provisions a serving lilbee with a warm engine, the sweep drives an OpenAI-compatible load generator against /v1 at rising concurrency (plus direct-to-engine cells so lilbee's overhead is a delta), and the soak loops concurrent chat streams and CLI engine cycles for hours with per-round invariants on VRAM, process counts, and membership locks, killing an engine process every fifth round to prove self-healing.
… dead slots in place Chaos soak findings on an A100 (SIGKILL a random engine process every 5th round, 4 concurrent chat streams + a CLI engine cycle per round): - A SIGKILLed llama-swap proxy surfaces as a raw httpx transport error, not a ProviderError, so _with_rediscover never fired and a long-running serve returned 500 for every chat until restarted while fresh CLI processes recovered fine. Rediscovery now classifies transport errors through is_connection_failure, chat_with_tools gets the same wrapper, and streaming chat primes its first frame inside the wrapper so a dead proxy fails where it can be rediscovered. The primed stream is a small closable wrapper so a caller that truncates before iterating still releases the request slot. - With one swap group dead and a live member holding the machine slot, an arriving process treated the slot as an incompatible incumbent in active use and built a full second engine in the overflow dir, loading duplicate 37GB weights. The ladder now rebuilds the slot in place when the healthy groups are pin-equal and serve only wanted models: that engine is the contract's own, and its members recover through rediscovery.
…benchmarks The chat model is a sparse MoE, so its decode rate is not representative of a dense model of the same VRAM footprint; say so beside the setup and list the coverage gaps (model architectures, prefill-heavy prompts, concurrent embed load, other hardware) rather than leaving them implied.
Two dense model tiers people would actually code with, four concurrent agents chosen to stress different parts of the engine (including retrieval, so embed and chat share load for the first time), prompts written the way a person asks rather than the imperative fixtures the old harness used, and a per-tier storyboard.
Qwen3-Coder-Next as the single-GPU workhorse (purpose-built for agentic coding and already a verified tool-call family here), MiniMax-M2.1 as the frontier and multi-GPU row, and the already-measured Qwen3.6 MoE as a free reference. Drops Llama-3.3-70B, which is neither coding-specialized nor current, and records why Kimi K2.6 is not worth four H200s and a 2-bit quant for this purpose.
Tasks now scale with the model: well-scoped single-subsystem work for Qwen3-Coder-Next, cross-file judgment calls for MiniMax, and real concurrency and design problems for Kimi. The harder tasks are drawn from bugs that actually happened here, so the rubric is the known root cause; runs pin their checkout before the real fix so the answer is not sitting in git history. Adds the opencode theme as a versioned file matching the official rose-pine mapping. The v2 reel used a community theme with function and type swapped and every variable painted rose, which is the highlighting that looked wrong.
Consumer cards wherever it is honest: a 30B coder on a single 4090 or 5090, the 80B on a dual-card desktop or a 64GB Mac, MiniMax on a large-memory workstation, and only Kimi in a warehouse. Records why the 80B needs more than one consumer card today, which is that lilbee has no MoE expert offload at all (bb-cuzq), and notes that a small card will serve four agents on fewer than four slots so the narration says sharing rather than parallel. Replaces the invented tasks. Every prompt is now either a bug that actually happened here, pinned to the commit before its fix so the answer is not in git history and the rubric is the known root cause, or an open issue somebody wants, so a run produces a usable patch rather than throwaway motion.
Tracker IDs belong in commits and PRs, not in repo files where they go stale and mean nothing to a reader without the tracker. Describes the referenced work in prose instead.
tobocop2
force-pushed
the
fix/serve-data-dir-singleton
branch
from
July 19, 2026 03:12
fb64376 to
a6b8923
Compare
…flags Sparse models activate a fraction of their weights per token, so their experts tolerate system memory far better than a dense model's layers would. cpu_moe and n_cpu_moe emit --cpu-moe and --n-cpu-moe, gated on the GGUF declaring routed experts so the flag is never a no-op on a dense model, and the weights-exceed-VRAM refusal now stands down when offload is configured and points sparse models at the setting that helps rather than at n_gpu_layers. Also fixes a pairing that could not load: quantized KV requires flash attention, and nothing checked, so flash_attention=false with the default q8_0 cache emitted --flash-attn off alongside --cache-type-k q8_0. Both the launch and the estimator now fall back to f16 so their sizing agrees.
The self-check and MCP status dumps list the GPU and memory settings someone reads when a model will not load. Omitting the pair that now controls exactly that would send them looking in the wrong place.
The self-check re-derived the chat flash and KV flags in its own expression, so it carried the unloadable quantized-KV-without-flash-attention pairing and knew nothing about expert offload. A user with offload configured would have seen the diagnostic fail on a config the fleet launches fine. Both callers now share the same helpers, which is why the flag decisions became public.
gguf-parser was charging the GPU for expert weights that --cpu-moe keeps in system memory, so the planner budgeted against a footprint that never materializes and handed a sparse model fewer batching slots than its real residency warranted. It accepts --override-tensor with the same pattern syntax, so the estimate now offloads exactly what the launch does, using llama.cpp's own expert regex to keep the two in step.
The fit chip was computed against GPU VRAM alone, so a sparse model larger than the card read as won't-run even though expert offload now serves it. The budget borrows system RAM when offload is configured, and only where the budget really is device memory: Apple unified memory, non-NVIDIA hosts, and CPU-only hosts already report system RAM, so adding it again would invent capacity that does not exist.
…holds A tensor-split chat was hardcoded to one sequence slot, so a model too big for one card served every concurrent agent through a single slot: they queued instead of decoding in the same batch. Tensor-split (weights across cards) and --parallel (sequences across slots) are orthogonal, and the split KV had room for more. The launch now picks the largest slot count whose sequences each keep the full working window, sized against the placed cards' real per-device headroom via the existing split-ctx fit, so it can never exceed what those cards hold and never OOMs; it falls back to one full-context slot on a tight split. The placement estimate stays at one sequence as a conservative card-count floor. On two large cards a big MoE now serves four agents concurrently, each with a full context, instead of one at a time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every lilbee process builds its own model engine. Open the TUI while a coding agent uses
lilbee mcp, or runlilbee servealongside either, and each process spawns a full llama-server fleet: identical weights load twice, and on small machines both fleets degrade or OOM.A second
lilbee serveon one data dir also silently overwritesserver.portand the session token.Solution
One inference engine per machine, shared by every lilbee process running the same model setup.
How a process gets its engine
Each process runs a short ladder under a cross-process build lock.
It binds to the machine slot's running engine when that engine is healthy and serves the configured models. Binders spawn nothing and write nothing; they point at the recorded proxy ports.
It builds into the slot when the slot is empty. An engine that no live process is using is replaced in place, so leftovers never poison the slot.
It overflows to the config root's private dir only when the slot holds a live incompatible engine in active use. That is exactly the case of two genuinely different model setups running at once.
What "compatible" means
The contract is the per-role models plus the bundled engine-build pin. lilbee version is deliberately not part of it: releases sharing an engine pin share an engine, while differing pins never run on a build they were not tested against.
Lifetime
Membership is kernel-refcounted. Each process holds a user lock the OS releases on any death, and the last clean exit stops the engine, leaving the machine clean by default.
keep_engine_warmis the explicit opt-in for the engine to outlive lilbee. The idle TTL applies in every mode, so even a persistent engine releases weights when unused.lilbee engine stopremains the unconditional off switch.What this deletes
The old ownership machinery (detach/adopt, owner-liveness checks, sibling sparing, the ps-scan sweep) is removed outright.
Server groundwork
lilbee servegains per-data-dir and per-installation OS locks, a token-authedPOST /api/shutdown, a bounded fleet-teardown budget, and shutdown logging.Hardening found by QA
Everything below was discovered while load-testing and benchmarking the shared engine, not planned up front. Each item is a capability or a correctness rule that was missing rather than broken by this branch.
Expert offload was missing entirely
A sparse mixture-of-experts model activates a fraction of its weights per token, so its experts tolerate system memory far better than a dense model's layers would.
llama.cpp has supported offloading them since well before the version lilbee pins, and lilbee emitted none of it: no
--cpu-moe, no--n-cpu-moe, no--override-tensor. Every model therefore had to fit VRAM outright, which is why large MoE coders appeared to need a datacenter card.cpu_moeandn_cpu_moenow emit those flags, gated on the GGUF declaring routed experts so the flag is never a silent no-op on a dense model. No engine bump was needed; the pinned build already had this.The estimator did not know about offload either
gguf-parser was sizing models as though every expert were resident, so the planner budgeted against a footprint that never materializes and granted fewer batching slots than the real residency warranted.
gguf-parser accepts
--override-tensorwith the same pattern syntax llama.cpp uses. The estimate now offloads exactly what the launch offloads, sharing llama.cpp's own expert regex so the two cannot drift apart.The catalog called runnable models unrunnable
The fit chip that marks a model "won't run" was computed against GPU VRAM alone. With offload, a model larger than VRAM runs fine, so the catalog was discouraging pulls that now work.
The budget borrows system RAM when offload is configured, and only on hosts whose budget really is device memory. Apple unified memory, non-NVIDIA hosts, and CPU-only hosts already report system RAM, and adding it twice would invent capacity.
Quantized KV could produce a command line that would not load
Quantized KV cache requires flash attention, and nothing checked the pairing. With
flash_attentionoff and the defaultq8_0cache, the launch emitted--flash-attn offalongside--cache-type-k q8_0.Both the launch and the estimator now fall back to f16 so their sizing agrees.
The setup self-check disagreed with the fleet
The self-check built its own llama-server command line and re-derived the chat flags in a hand-written expression, so it carried that same unloadable pairing and knew nothing about offload. A user with offload configured would have watched the diagnostic fail on a configuration the fleet launches fine.
Both callers now share one set of flag decisions instead of two copies.
Smaller fixes from the same passes
Catalog commands (
model pull/show/rm/browse) no longer eager-start the fleet, so a download can never spawn a partial engine mid-pull. A chat request whose output reservation exceeds the served window is clamped instead of rejected, so agent clients that over-reserve keep working. The membership locks survive cross-thread release, multiple lilbee instances in one process, and thread-pool reuse.Acceptance test
A four-agent load harness ships with the branch (
tools/qa/shared-engine): four opencode agents on one GPU box, each in its own project, all served by one engine. It asserts servers bind instead of building, VRAM stays flat against the single-agent baseline, every task completes, and teardown ends with zero engines and zero locks. It passes 18/18.Load and stability results
Measured on an A100 80GB, with the harness and raw data committed under
docs/benchmarks/.The capacity sweep runs 130.9 tok/s per stream at one stream and holds roughly 230 aggregate tok/s from 4 to 16 concurrent streams, saturating rather than collapsing.
Routing through lilbee costs nothing measurable against talking to the engine directly: 130.9 versus 130.8 tok/s per stream, with time-to-first-token inside run-to-run noise.
A chaos soak then ran 600 rounds over 2 hours 24 minutes: 2,400 chat completions across 4 concurrent streams, 600 CLI engine cycles, and a random engine process killed every fifth round. 120 forced kills, zero resource leaks, VRAM flat within a 0.5 GB band, and the membership lock count never left 1.
588 of 600 rounds were fully green. The 12 degraded rounds were the crash-recovery window, where requests landing mid-rebuild currently fail fast rather than waiting the rebuild out.
An earlier run of that same soak is why two of the fixes above exist: on the pre-fix build, killing the llama-swap proxy orphaned its VRAM, built a duplicate engine in the overflow slot, and left the resident server erroring until restart.