llm: force f32 cuBLAS compute for qwen25vl runners

[glennneuber]

Fixes the qwen2.5vl poison-image garbage decode from #214 by forcing fp32 accumulation in the model's runner — without patching the llama.cpp payload.

What

When the launcher starts a llama-server runner for arch qwen25vl, applyArchServerEnvs (llm/llama_server.go) sets GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32 in that subprocess's environment. Each runner serves exactly one model, so stock ggml's process-wide cuBLAS compute override becomes model-scoped. An operator-set value always wins (=f16 reproduces stock behavior for A/B; =bf16 is the perf-friendlier alternative — cuBLAS bf16 GEMMs also accumulate fp32). Every other arch launches bit-identical to today.

Why

#214 localized the '?'×31 garbage (quant-independent, CPU-healthy) to the vision tower/merger's f16-weight matmuls running as fp16-accumulate cuBLAS GEMMs on CUDA/HIP (compute_type = src0->type at pin b10488), while the CPU backend always accumulates fp32. Confirmed causally at runtime (comment): the class is alive in the latest upstream release (HHXH on stock 0.33.0) and the knob heals it (HH), with f32/bf16 healing and f16 reproducing. No upstream version bump saves us — and our current build sits inside the 0.32.10–0.32.15 sticky-slot window, where one poison request contaminates the resident runner until reload.

Why this shape

• No fork-carried llama.cpp patch — nothing to re-cut on pin bumps.
• No behavior change for any other model — no vision-baseline drift for gemma4/nemotron/qwen3vl, no preflight re-recording outside the qwen25vl family.
• Go-only — deployable via binary swap; the knob already exists in every shipped CUDA/ROCm payload at the current pin.

The per-op form (compat patch 904: GGML_PREC_F32 post-pass over the clip graph, compile-validated) is shelved on fix/clip-mm-prec-f32 as the upstreamable fix and the fallback if a future pin drops the env knob; llama/README.md's update checklist now carries that hazard.

Scope notes

• Family-wide (qwen25vl: 3b/7b/32b) — the class is latent per-graph, each implementation has its own poison set. qwen2vl shares the clip graph builder but has no measured trigger and stays stock. A 3B-only variant (key on embedding width 2048) is a one-line change if 7B baseline continuity is preferred.
• Perf: vision encode pays f32 GEMMs; text side ~nil for quantized tags (MMQ/MMVQ carry them, f16 decode vectors already accumulate fp32); -fp16 text tags pay an fp32-GEMM prefill cost.
• Details: docs/maxusai/qwen25vl-cublas-f32-env.md; probe shape and decision table: docs/maxusai/clip-mm-prec-f32-validation.md (on the sibling branch).

Validation

• TestApplyArchServerEnvs: forces f32 / operator override wins / qwen2vl and gemma4 untouched — pass.
• Full-tree go build + ./llm/... tests green (golang:1.26 container), gofumpt clean.
• Remaining runtime validation before deploy: the 3-request poison probe on a gated build — no env set → expect HHH; -e GGML_CUDA_CUBLAS_COMPUTE_TYPE=f16 → garbage must return — then the standard preflight.

🤖 Generated with Claude Code

[glennneuber]

Reviewing as consolidator. The causal confirmation is what elevates this above a mitigation, and the shape of the fix is the right trade.

f32/bf16 heal, f16 reproduces, on demand — that is the two-sided test. A one-sided "the knob fixes it" would leave open that something else changed; showing the defect returns when the knob is set back is what establishes the mechanism. And confirming the class is alive on stock 0.33.0 kills the "wait for an upstream bump" option outright, which is the question a reader would otherwise ask first.

The localisation is consistent with what #214 could support and no further: fp16-accumulate cuBLAS GEMMs in the vision tower/merger, against a CPU backend that always accumulates fp32, explaining quant-independence and CPU-health without needing either to be assumed.

Why the env shape is right here

Three properties, and the middle one is the one I care about most:

• no fork-carried llama.cpp patch — nothing to re-cut on pin bumps, which this fork has learned the cost of
• no behaviour change for any other arch — no vision-baseline drift for gemma4/nemotron/qwen3vl, and no preflight re-recording outside the qwen25vl family. Every expect.* row measured this month stays valid, which given how many were corrected in the last week is worth stating explicitly
• Go-only, deployable by binary swap — the knob already exists in every shipped payload at the current pin

Keying on arch and letting an operator value win follows the pattern #180/#186 settled for the memory limit and #212 for the thrash check: derived default, explicit override, never silently overwritten. Three knobs now share that shape, which is worth more than any of them individually.

The perf note is honest and I would keep it visible

Vision encode pays f32 GEMMs; text ~nil for quantized tags; -fp16 text tags pay an fp32-GEMM prefill cost. That last clause is the one that will surprise someone, because it is a text-side cost from a vision-motivated change. A campaign comparing a -fp16 qwen25vl tag across this boundary will see prefill move and should not read it as a payload regression — the SPEC H11 server_version comparability boundary covers it, but only if someone thinks to look.

Two things I would want before deploy

1. The runtime probe is still outstanding and the PR says so — no env → HHH, forced f16 → garbage returns. That is the same two-sided shape as the causal test, run against the gated build rather than the bare knob. Worth landing the result in the PR before merge, as mlx: run with MLX's graph-cache thrashing check off; keep the first panic #212 did with its /proc/<pid>/environ read: "we set it" and "it arrived and did the thing" are different claims.
2. Preflight against the CUDA host afterwards. Not because qwen25vl is in a profile — it is not — but because applyArchServerEnvs is a new code path in the launcher every arch flows through, and the cheapest proof it is inert for gemma4/nemotron is a green run rather than a unit test asserting the map lookup.

Shelving compat patch 904 as the upstreamable form and the fallback if a pin drops the knob — with the hazard recorded in llama/README.md's update checklist — is the right disposition. That checklist is exactly where a future pin bump will look.

[glennneuber]

Dress rehearsal on a gated build: HHH / HXX / HHH — gate heals, override reproduces, bf16 also heals

Built /bin/ollama from this branch via the publish-go Docker target (bigdisk builder) and bind-mounted it over maxusai/ollama:sync-0.32.15 — native payload untouched, since the knob is read by the shipped libggml-cuda. Fresh container per cell, docker_ollama_data store :ro, RTX PRO 6000 (Blackwell), 3-request probe good → poison (02c9d7e1…) → good, num_ctx 8192, temperature 0, num_predict 250. Runner env verified via /proc/<llama-server>/environ in each cell.

Cell	Container env	Runner env (verified)	Pattern	Poison request
A	none — gate default	GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32	HHH	sensible ("wooden shelf with multiple cables…"), stop, prompt_eval 676 ms
B	GGML_CUDA_CUBLAS_COMPUTE_TYPE=f16	…=f16 — gate did not clobber it	HXX	'?'×31, done_reason: null, slot stayed poisoned
C	GGML_CUDA_CUBLAS_COMPUTE_TYPE=bf16	…=bf16	HHH	sensible, stop, prompt_eval 734 ms

• A validates the shipped default end-to-end: the launcher injects f32 into the qwen25vl runner and the poison image serves correctly on the current fork build.
• B is the causality check both ways: the operator value wins exactly as coded, and stock behavior returns in full — garbage plus the 0.32.15-window sticky slot (HXX).
• C answers "could the gate value be bf16": yes — the knob's parser accepts f32/fp32, f16/fp16, bf16, auto; bf16 heals (cuBLAS bf16 GEMMs accumulate fp32 — 16-bit inputs, fp32 sums) at timing parity with f32 here (676 vs 734 ms warm poison encode, n=1 each). Caveat: not validated on Turing — sm_75 predates native bf16 tensor cores — so f32 stays the everywhere-safe gate default, with bf16 operator-selectable per container (cell C doubles as proof the override path carries it).

Also demonstrated in passing: the Go-only binary swap deploy path works — a branch-built /bin/ollama over the stock image is a complete runtime test of this PR.

Remaining before deploy: standard preflight on a fully built image.

🤖 Generated with Claude Code

[glennneuber]

Turing check: gate heals on the 2080 Ti too, and bf16 is safe (but earns nothing) on sm_75

Same rig as the dress rehearsal above — branch-built /bin/ollama over maxusai/ollama:sync-0.32.15, fresh container per cell, store :ro, 3-request probe good → poison (02c9d7e1…) → good — moved to the RTX 2080 Ti (Turing, sm_75), runner env verified per cell:

Cell	Container env	Runner env	Pattern	Poison request
A-T	none — gate default	GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32	HHH	sensible ("wooden shelf with multiple cables…"), stop, warm encode 1059 ms
B-T	…=f16	…=f16	HXH	'?'×31, done_reason: null — and the slot recovers next request
C-T	…=bf16	…=bf16	HHH	sensible, stop, warm encode 1223 ms

• A-T closes the gate's hardware coverage: the shipped default heals on both failing GPU generations (Blackwell HHH above, Turing HHH here).
• B-T reproduces stock garbage under the operator override — and refines the sticky-slot picture: on the same 0.32.15 build, Blackwell poisons sticky (HXX, cell B above) while Turing recovers (HXH), matching the original knob-matrix split. So slot persistence is GPU/FA-flavored within the 0.32.10–0.32.15 retention window, not purely a version property.
• C-T resolves the bf16 caveat from the previous comment: cuBLAS accepts the bf16 path on sm_75 — no error, no crash, heals correctly — but at 1223 vs 1059 ms it buys nothing over f32 without native bf16 tensor cores (n=1 each, contention possible; read as parity). With bf16 at parity on Blackwell too (676 vs 734 ms above), f32 stays the gate default everywhere — closest to CPU numerics, no measured cost — and bf16 remains a per-container operator choice that now has measured support on every CUDA generation in this estate.

🤖 Generated with Claude Code

[glennneuber]

Gate validated against the shareable synthetic trigger — evidence now rests on public artifacts only

Following #214's synthetic-repro update (make_poison_repro_image.py, the 56 px checkerboard at 1350×1800 measuring 1.06× the fp16 ceiling — stronger than either corpus trigger), two cells on the built gated image maxusai/ollama:pr215-gate-rc0 (sync-0.32.15 natives + this branch's Go binary), RTX 2080 Ti, fresh container per cell, 3-request probe good → checkerboard → good:

Cell	Container env	Runner env	Pattern	Checkerboard request
S1	none — gate default	GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32	HHH	"black and white checkered pattern…", stop
S2	…=f16	…=f16	HXX	'?'×31, done_reason: null — and the slot stays poisoned

Two observations beyond the pass:

• The gate heals the strongest known trigger, not just the corpus images — consistent with docs/maxusai: document the qwen2.5vl-3b poison-image serving bug #214's measured mechanism (f32 accumulation moves the ceiling from 6.5×10⁴ to ~3.4×10³⁸; the margin question disappears).
• S2 refines the slot-stickiness picture again: with the corpus poison Turing recovered (HXH, earlier comment), but the synthetic trigger — larger image, ~3k-token visual sequence — leaves the slot sticky (HXX) on Turing too. Stickiness tracks trigger strength/sequence footprint as well as GPU, which fits the prefix-cache-retention theory.

#214's mechanism comment lands on exactly this PR's shape as the fix's final form: "the multimodal matmuls of this family cannot safely run fp16 accumulation… GGML_CUDA_CUBLAS_COMPUTE_TYPE=bf16 (perf-neutral) or f32 should be the default." This PR ships f32 per-runner for qwen25vl, operator-overridable either way.

🤖 Generated with Claude Code

[glennneuber]

Preflight now asserts this fix on every future build: poison_probe added (60d3b79)

The defect class gets a permanent canary in the pre-deploy harness rather than living only in this PR's comment trail:

• Run-once, profile-keyed check ([poison.cuda-dynres-903]): renders docs/maxusai: document the qwen2.5vl-3b poison-image serving bug #214's synthetic 1.06×-fp16-ceiling checkerboard with the harness's stdlib PNG writer (verified pixel-identical to make_poison_repro_image.py's PIL output), sends it to qwen2.5vl:3b-q4_K_M on a fresh slot, requires a healthy decode, then a text-only follow-up on the same slot to require no poisoning residue. Profiles without an entry SKIP; it runs even under --arch-narrowed runs since it loads its own 3B.
• The classifier pins the measured fingerprint — done_reason: null or a single repeated glyph — and deliberately not short/empty responses, which the num_predict trap produces for unrelated reasons. Fingerprint table + verdict wiring covered in test_verdicts.py (50 tests OK, 7 new).
• Validated both directions on live builds: PASS inside a full harness run against maxusai/ollama:pr215-gate-rc0 (1.06x-ceiling trigger decodes healthily (52 chars, done_reason='stop'); slot clean after; run VERDICT PASS) — and FAIL with the exact recorded fingerprint (done_reason=None head='?'×31) against ungated sync-0.32.15, with the diagnosis pointing at the gate docs. The canary demonstrably catches the defect it exists for.

With this, reverting the gate — or a future pin/env change silently re-opening the fp16-accumulate path — fails preflight instead of waiting for a poison image in production.

🤖 Generated with Claude Code

[glennneuber]

Preflight complete: full CUDA matrix PASS on the gated image — deploy-ready

Image under test: maxusai/ollama:pr215-gate-rc0 — sync-0.32.15's bit-identical native payload + this branch's version-stamped Go binary (0.32.14-dynres-111-gda676e30). The harness vouches for that identity itself: version gate matched profile cuda-dynres-903 and payload_pin confirmed 9d77fa172 (b10488). Host 10.8.0.6 (Blackwell), store :ro, port :11438.

Run	Checks	Verdict
No-GPU gates	test_verdicts.py 50/50 (incl. 7 new poison-probe tests); TestImageTokensForSize OK	PASS
gemma4 + qwen35, full (pinned incl.)	13 PASS, 2 SKIP (no pinned expectation recorded for these arches — the honest recorded state)	PASS
nemotron_h_omni, full (pinned incl.)	10 PASS, 0 SKIP, endpoint_exclusive clean	PASS
poison_probe (harness-integrated, this branch)	PASS on this image; FAIL with the '?'×31 fingerprint on ungated sync-0.32.15	PASS

Reading of the result: token ladders 5/5 per arch, text baselines, payload proofs, think+format, and pinned budgets all match recorded expectations exactly — the gate provably changes nothing outside qwen25vl — while the poison probe asserts the qwen25vl fix itself, in both directions.

That closes the last pre-deploy item from the rehearsal comments. Evidence chain for this PR, end to end: mechanism measured (#214), kill-switch on stock latest, gated-build rehearsal on Blackwell + Turing, synthetic-trigger validation, and now a passing preflight with a permanent regression canary.

🤖 Generated with Claude Code