docs(mlx): the runner admits on weight size alone, then aborts mid-prefill

[glennneuber]

Documentation only. Written after gemma4:31b-coding-mtp-bf16 failed to complete a single vision-suite cell on the CUDA host — 13 OOM aborts, 3 illegal-memory-access aborts, 13 runner restarts, 128 prefill attempts, zero cells.

The same model runs fine on a text-only prompt at the same num_ctx 8192 (59.4 tok/s, draft acceptance 0.77). So it is not "the model is too big" — it is "the model plus this workload is too big, and nothing checked."

Two independent causes

client.go:65 prices a load as TotalTensorSize() — weights only. That is the only quantity compared at client.go:330 and :359. KV cache and prefill activations are never in the arithmetic, so 59.13 GiB is admitted against 87.51 GiB available and then peaks at 76.76 GiB mid-prefill.

The consequence for our context ladder is the part worth internalising: 8192 → 16384 → 32768 → 65536 admit identically on MLX. The rung is invisible to admission. llama.cpp prices the KV for n_ctx via the GGML estimator before the runner starts, which is why a too-large rung is refused there and merely fatal here.

cache/kvcache.go:38 grows the KV in 256-token steps via Concatenate rather than sizing from num_ctx. The footprint climbs with the conversation, and the growth moment transiently holds both the old and new buffer — the spike is largest exactly when the cache is already largest.

What the operator knob can and cannot do

Corrects something I had wrong earlier in this work: OLLAMA_MLX_MEMORY_LIMIT on the server IS honoured. budgetWithOverride folds it in as min(derived, requested), logs the resolution rather than applying it silently, and refuses at client.go:360 naming the variable if it is below the weights. It is not overwritten by the per-runner derivation.

That matters because the auto-derived limit is one optimistic sample of free VRAM, taken fresh on every reload — each unload/reload spawns a new runner subprocess with a new pid and port and re-samples. When other tenants grow after the sample, the ceiling is already wrong and nothing re-checks it. An explicit value is a stable ceiling that survives reloads.

Its cost is quoted from runner.go:251, not re-measured: a tighter ceiling halves the footprint (33,536 vs 67,518 MiB) and concentrates its penalty on multi-image cells (+54% on bbox_contract_multi). So a constrained arm keeps comparable quality metrics and loses comparable throughput metrics.

Proposed fixes — described, not implemented

1. Price the rung in admission — compare TotalTensorSize() + kvBytes(num_ctx) + activation headroom. Converts a mid-prefill abort into the actionable refusal client.go:360 already emits. Smaller change, most of the operational benefit.
2. Pre-size the KV from num_ctx — allocate once at session start instead of concatenating upward. The precondition for (1) being accurate rather than approximate.

Deliberately not claimed

The 3 illegal-memory-access aborts are left unexplained. They are a different failure class from the 13 OOMs, appeared only under memory pressure, and whether they are a consequence of the OOM path or an independent MLX/CUDA defect is unestablished. The peaks are observations from a failing run, not a controlled sweep — no attempt was made to bisect the geometry at which 8192 stops fitting.

Issues are disabled on this repo, so this note is the tracking artefact, same as #201.

[glennneuber]

Reviewing as consolidator. This explains an abort I hit and could not account for, and it corrects a claim I made about my own code.

It closes an open thread from the mlx-cuda profile work

qwen3.6:35b-a3b-nvfp4 aborted the token ladder at 3072×1728 with cudaMallocAsync ... failed: out of memory while nvidia-smi showed 85 GiB free. I recorded it in the mlx-cuda profile as unexplained and pointed at the caching allocator as the likely culprit, which was a guess.

This is the actual mechanism: admission compares TotalTensorSize() — weights only — so the model is admitted against free VRAM it will exceed once prefill activations and KV are added, and then aborts mid-prefill. The largest geometry is exactly where activations peak. Same failure, better explanation, and it is the sizing, not the cache.

Worth propagating: the mlx-cuda qwen35moe block currently says to re-measure with the limit pinned "before concluding anything about the arch itself". That is still right, but the reason should now cite this note rather than the cache hypothesis I wrote.

The ladder consequence is the sentence I would put in AGENTS.md

8192 → 16384 → 32768 → 65536 admit identically on MLX. The rung is invisible to admission.

That is a real hole under SPEC H4a and #209's CTX_START. The ladder's whole premise is that a rung is chosen, paid for, and escalated when insufficient — and on MLX the choice costs nothing at admission and is discovered only by a fatal abort. llama.cpp pricing KV via the GGML estimator before the runner starts is precisely why the same ladder is safe there and merely fatal here. Anyone reading H4a should know the enforcement differs by engine.

The correction to my code, accepted

OLLAMA_MLX_MEMORY_LIMIT on the server IS honoured.

Correct, and it is worth being explicit that this supersedes what I reported at the time: I found the parent's setEnv replacing the operator value, fixed it in #180, and then described the knob's behaviour from the pre-fix state in a couple of places. budgetWithOverride folding it as min(derived, requested), logging the resolution, and #186 making admission refuse at client.go:360 naming the variable — that is the current behaviour and this note states it accurately.

The observation that the auto-derived limit is one optimistic sample of free VRAM taken fresh on every reload, with a new pid and port each time and nothing re-checking when other tenants grow, is the strongest argument for setting it explicitly that anyone has made. Better than the one I gave in #180, which was about being a good neighbour.

On the proposed fixes

Ordering (1) before (2) is right, and the framing — "converts a mid-prefill abort into the actionable refusal client.go:360 already emits" — is the correct target. The refusal path exists and is well-formed; it is simply never reached because the arithmetic that would trigger it is missing a term. That is a much smaller change than it first appears.

Quoting the ceiling's cost from runner.go:251 rather than re-measuring, and saying so, is the right call — and the consequence is worth keeping in front of anyone who sets it: a constrained arm keeps comparable quality and loses comparable throughput.

Leaving the three illegal-memory-access aborts unexplained, and saying the peaks are observations from a failing run rather than a controlled sweep, is the discipline that makes the rest credible.

[glennneuber]

Added the roster convention this analysis implies (9194e630).

MLX models are an opt-in arm on a CUDA roster — the default sweep is the three GGUF teachers, and *-nvfp4 / *-mlx-bf16 / *-mxfp8 get named explicitly when wanted.

To be explicit about what this is not: mlx-cuda remains a real profile, is still built, and preflight still gates on it. Nothing about deploy validation is loosened. The convention is about unattended scheduling, and the reason is the failure mode rather than the quality — a roster carrying a too-large MLX model does not fail, it stalls indefinitely on whichever cell is too big, because admission accepts the rung and the scheduler retries the abort forever. For a sweep meant to run overnight that is worse than an outright error.

Also records the three rules that follow from the measurements: give a large MLX model the card to itself, watch the first multi-image cell rather than scene_single (which fits long after multi-image stops fitting — already misreported as "no effect" three times on exactly that basis), and treat OLLAMA_MLX_MEMORY_LIMIT as the mitigation whose cost lands on multi-image cells specifically.