Router co-loads same-device-pinned models and OOMs: --models-max fit decision ignores per-device VRAM

[marksverdhei]

Summary

The native model router co-loads two models that pin to the same CUDA device and OOMs at load time, because the admit/fit decision reasons about total VRAM across devices instead of per-device free memory. Surfaced as intermittent HTTP 500s from the titan-llm deployment (2 × RTX 3090) during the 2026-06-04 heierchat incident.

Symptom

titan-llm runs the router with --models-max 2 --models-preset llama-router.ini. Every large preset pins to GPU0 via tensor-split = 100,0 (to leave GPU1 for a co-located ComfyUI). When two big CUDA0-pinned models are requested close together (e.g. devstral-24b ~13.5 GB already resident, then gemma-4-31b-iq4 needing ~15.6 GB weights), the second load fails:

[36701] CUDA0 : NVIDIA GeForce RTX 3090 (24124 MiB, 10552 MiB free)
[36701] W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 999, abort
[36701] E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 15598.97 MiB on device 0: cudaMalloc failed: out of memory

The router then routes to the failed/closed instance → client sees proxy error: Could not establish connection (HTTP 500) / 502 / 000. Intermittent — only when a second CUDA0-pinned model is already resident; if GPU0 is free the same model loads fine (~23 GB on a 24 GB card, by design).

nvidia-smi at load time confirmed GPU0 had ~13.5 GB occupied (devstral) and GPU1 was nearly empty (389 MiB) — usable capacity the router won't touch because of the pin.

Root cause

--models-max 2 keeps two models resident, and the admit decision treats VRAM as one 48 GB pool. It does not account for tensor-split / --device pinning both models onto the same physical GPU0 (24 GB). So it co-admits 14 GB + 16 GB believing 48 GB suffices; they collide on GPU0.

Note: docker/unified-llm/entrypoint.sh deliberately uses --models-max 2 (not 1) to dodge a load-race (max=1 force-kills a model still in LOAD state via unload_lru after stop_timeout), justified as "without changing memory headroom (each load still respects the per-model GPU footprint)." That last clause is the bug — max=2 does overcommit a shared-device pin.

Proposed fix (per @ht-llama.cpp-dev's read)

Land in two places:

• Admit decision — common_fit_params.
• Eviction policy — tools/server/server-models.cpp (pick_any_resident + the LRU walk).

The router already parses which device(s) a model pins to (tensor-split). Add a compute_free_per_device(active_models) helper that subtracts loaded-model footprints from per-device cudaMemGetInfo. The admit check becomes: for each device the candidate targets, candidate.bytes <= free_after_lru_evict[device] — and LRU-evict from that device, not globally. This makes --models-max 2 safe for same-device pins (today it's the OOM trigger) and lets the router correctly use GPU1 when a second model would fit there.

Environment

• ht-llama.cpp b0daec55b (origin/ht); deployed image unified-llm:dflash-65f46f0f8.
• titan: 2 × RTX 3090 (24 GB each). Router flags: --models-max 2 --parallel 1 --cont-batching.
• Diagnosed jointly by snoop-kube (cluster) + ht-llama.cpp-dev.

Related: companion issue on context-checkpoint host-RAM cap.

[marksverdhei]

Proposed fix shape

(Captured from coordination with snoop-kube 2026-06-04.)

1. Surface per-device demand from common_fit_params. Today it does all its work in-process at model-load time, with no router-visible artifact. Add a dry-run entry that resolves the planned tensor split (or other device routing) against the candidate model and returns the per-device byte demand WITHOUT actually allocating anything. Signature shape:

   struct common_fit_plan {
       std::array<size_t, GGML_MAX_N_DEVICES> bytes_per_device;
       size_t total_bytes;
   };
   common_params_fit_status common_fit_params_dry_run(
       const char * model_path,
       const common_params & params,
       common_fit_plan * out_plan);

   Crux of the work — fit_params has lots of inertia and dry-run mode needs to share its sizing logic without the side effects.

2. Track reserved-by-loaded per device in the router. In tools/server/server-models.{cpp,h}, maintain a std::array<size_t, GGML_MAX_N_DEVICES> reserved updated on each load (+= candidate.bytes_per_device) and unload (-= ...). Persist alongside mapping under the existing mutex.

3. Replace unload_lru() with unload_lru_for_devices(targeted, candidate_per_device). Today's unload_lru is a count-based check against --models-max. New shape:

   • Determine the set of devices D the candidate targets (any d where candidate.bytes_per_device[d] > 0).
   • For each d ∈ D, require free_device[d] - reserved[d] >= candidate.bytes_per_device[d]. free_device[d] from ggml_backend_dev_memory.
   • If under-budget for any d, LRU-evict from the subset of loaded models that occupy d (filtered by reserved_d > 0), oldest-first, until the candidate fits OR no more loaded models pin to d.
   • If still doesn't fit after evicting everything on d, return a model_unavailable_error mapped to HTTP 503 (already exists in the codebase).

4. Keep --models-max as a hard upper bound. It's the global concurrency cap; the per-device check is the tighter resource constraint. Both still useful — admit needs to pass BOTH.

Test repro (provided by snoop-kube)

Deterministic OOM repro on titan today: load devstral-24b (CUDA0), then request gemma-4-31b-iq4 (also CUDA0). Without the fix, the second admit succeeds against the 48 GB total pool but cudaMalloc OOM allocating 15598.97 MiB on device 0, 10552 MiB free. Snoop-kube will bake the prototype branch + canary-roll for live verification once it's ready.

Out-of-scope for this issue

• Cluster-side --models-max config band-aid (separate snoop-kube bounce; removed once this lands).
• Checkpoint host-RAM cap (Context checkpoints capped by count not bytes -> host-RAM OOM under long contexts #67 / PR fix(server): per-slot byte cap on context checkpoints (closes #67) #68 — independent fix for the OOM-from-RAM half of the same incident).

[marksverdhei]

Accounting correction (from snoop-kube 2026-06-04)

My original sketch had a subtle double-counting bug. Initial formulation:

free_d = live_cudaMemGetInfo(d) - reserved[d]
where reserved[d] = sum of bytes_per_device[d] across ALL loaded models on d

Wrong: live_cudaMemGetInfo(d) already reflects every loaded model's allocation. Subtracting reserved[d] (= all loaded models on d) double-subtracts — admit would reject loads that actually fit.

Corrected model

free_d = live_cudaMemGetInfo(d) - reserved[d]
where reserved[d] = sum of bytes_per_device[d] for models in LOADING state only
                   (admitted but cudaMalloc not yet visible in cudaMemGetInfo)

The router's existing state machine has the right states already: model transitions UNLOADED → LOADING → LOADED. The accounting follows:

• Admit decision commits → status LOADING → reserved[d] += plan[d]
• Load thread completes (status → LOADED) → reserved[d] -= plan[d] (the bytes are now in cudaMemGetInfo visibility)
• Unload completes → no reserved[] touch (the allocation just falls out of cudaMemGetInfo)

Why this matters on titan specifically

GPU1 is shared with ComfyUI (separate process, CUDA_VISIBLE_DEVICES=1, no router visibility). Only live cudaMemGetInfo captures ComfyUI's allocation. A static "total minus baseline" approach (or static accounting of router-loaded models) would miss it and admit a model onto a GPU ComfyUI is already using — same OOM, different cause.

Keeping live cudaMemGetInfo as the truth-source + reserved[] strictly for the in-flight admit race window:

1. Avoids double-counting (loaded models are in cudaMemGetInfo, not reserved).
2. Respects external GPU users (ComfyUI, anything else) automatically.
3. Handles the admit-to-alloc-visible race correctly without polling.

Implementation note

reserved[d] decrement happens at the LOADING→LOADED transition, which the router observes via the existing status callback. No new polling/timers needed.

[marksverdhei]

Architecture finding during step 2 scoping

Picked up steps 2-4 to start a shadow-mode prototype (compute + log + track reserved[] but don't gate yet). Hit a real design question worth flagging:

compute_admit_plan is bigger than a helper. To call common_fit_params from the router for a candidate model, the router needs to construct that model's llama_model_params + llama_context_params from the model's preset. Today that bootstrap happens only inside the spawned child subprocess via common_init_from_params — there's no router-process path that turns a preset into mparams/cparams.

Two viable approaches:

1. In-router CLI mini-bootstrap — extract the preset → mparams/cparams logic into a shared helper that both the child boot path and the new router admit path can call. Cleanest long-term, but the extraction touches common_init_from_params and several arg.cpp helpers — non-trivial refactor.

2. Out-of-process dry-run subprocess — shell out to tools/fit-params (extended with a --print-per-device-bytes flag) for each admit decision. Adds ~50-200ms latency per admit (subprocess spawn + model header read). Brittle output parsing. Avoids the in-process refactor entirely.

Approach 1 is the right long-term answer; approach 2 is a faster path-to-validation. Worth Markus's input on which to pursue before implementing — the choice affects the size of step 2's diff by an order of magnitude.

Will continue holding step 2 until that decision lands. PR #69 (out_bytes_per_device foundation) is independently mergeable regardless.

[marksverdhei]

Status check: #136 fixed the steady-state, but the concurrent-admit window is still open — don't close this yet

#136 (30023636e) landed the per-device admit + targeted LRU eviction (Approach 1, in-process). server_models::unload_lru() now checks candidate.per_device_bytes[id] > live_free[id] per GPU and evicts only from constrained devices (tools/server/server-models.cpp:856-944). That fixes the common case in this issue's repro.

Still unimplemented: the reserved[] LOADING-state accumulator from the accounting-correction comment above. The admit uses only live ggml_backend_dev_memory() (:882-889), and the under-lock re-check in load() (:988-998) guards the model count, not per-device memory. So the exact titan scenario still races: model A admitted → LOADING (:1004) → child spawned, cudaMalloc not yet visible; model B requested close behind reads stale-high free on the shared GPU0 → co-admits → OOM. --models-max 2 means the count check passes (2 ≤ 2), so only the (racy) memory check stands.

Fully closing #66 needs the LOADING-only reserve: reserved[d] += plan[d] on admit→LOADING, -= plan[d] on LOADING→LOADED, and free[d] := live_free[d] - reserved[d] in the admit test. Detailed mapping + RFC cross-ref in #146 (comment). Recommend leaving this open until that guard lands.

[marksverdhei]

Draft PR #148 implements the LOADING-state reserved[] accounting from the discussion above — reserve on enter-LOADING, release on leave-LOADING, unload_lru subtracts it, LOADING models excluded from eviction. Compiles clean (CPU server build). One policy call left open in the PR (proceed-vs-503 when a device is saturated by in-flight loads). Recommend keeping this issue open until #148 (or the 503 gate) merges.