qwen2.5-vl: fp16-accumulate cuBLAS vision GEMMs overflow on specific images — deterministic '?'×31 garbage; one-line env workaround; ≥fp32 accumulation must be the default

[glennneuber]

Consolidated, upstream-ready report of the qwen2.5-vl fp16-accumulate overflow class — filed here as the record; written so it can be carried to ollama/ollama / ggml-org/llama.cpp verbatim.

Summary

Specific, ordinary-looking images make qwen2.5vl:3b return '?'×31 with done_reason: null, deterministically, in every quant (q4_K_M / q8_0 / fp16), on stock ollama on CUDA — reproduced on stock ollama/ollama:0.32.9, 0.32.15, and 0.33.0 (current latest). The same request on CPU, or with fp32/bf16 cuBLAS accumulation, decodes correctly. In the 0.32.10–0.32.15 window the first poison request additionally poisons the resident runner — every subsequent request returns garbage until reload.

One-minute repro, no private data

docs/maxusai/make_poison_repro_image.py (on main since #214) writes a 56 px black/white checkerboard at 1350×1800. Send it to stock qwen2.5vl:3b-q4_K_M via /api/chat or /api/generate (temperature 0, num_ctx 8192), fully GPU-resident:

serving	result
default (fp16 cuBLAS accumulate)	'?'×31, done_reason: null — every call
GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32 (or bf16)	healthy description of the checkerboard

Mechanism (measured, not inferred)

Activation hooks over the HF Qwen2.5-VL-3B-Instruct vision tower (bf16, so true magnitudes survive; script docs/maxusai/measure_qwen25vl3b_vision_activations.py):

• The tower's final block (blocks.31.mlp) carries massive-activation outliers at 0.57× the fp16 ceiling (65,504) for ordinary images — every input walks the cliff edge.
• Trigger images push those channels to 0.77–1.07×. The stored tensors still fit in fp16; the partial sums inside the next fp16-accumulate GEMM transiently exceed the ceiling before cancellation → inf → NaN cascade → degenerate decode.
• The synthetic checkerboard measures 69,120 (1.06×). Image size is a trigger axis: upscaling the same content 1008→1800 px raises peaks ~1.6×, and longer visual sequences lengthen GEMM reductions so partial sums overflow even at 0.77× stored maxima. 1800–2048 px OCR-style pipelines sit in the worst part of the space.

This one mechanism accounts for the full evidence matrix: deterministic per-image failure; per-implementation disjoint trigger sets (fp16 overflow depends on summation order — ollama's removed Go engine and clip.cpp each have their own poison sets, measured); the 7B's relative immunity (lower final-block outliers); fine-tuned checkpoints shifting trigger sets (typhoon-ocr1.5-3b / upstream ollama#17687 never reproduced on our estate across 0.7.1→0.33.0 including its own 1800 px pipeline — same class, different triggers); and the immunity of CPU, f32, and bf16 accumulation.

What was ruled out

Fresh container per cell, three-request H/X probes: --flash-attn on and off, KV cache f32 and bf16, GGML_CUDA_FORCE_MMQ=1 — every CUDA config that serves at all garbles. Architecture-independent within CUDA (RTX 2080 Ti / Turing and RTX PRO 6000 / Blackwell alike). The overflow lives in the f16-weight vision tower/merger matmuls: ggml_cuda_mul_mat_cublas uses compute_type = src0->type, i.e. CUBLAS_COMPUTE_16F; the small-batch mmf/mmvf kernels already accumulate fp32, so cuBLAS is the only fp16-accumulate matmul stage.

Separable second defect: slot poisoning

Sticky-until-reload occurs only in 0.32.10–0.32.15 (0.30.0, 0.32.9 and 0.33.0 recover next request) and its severity also tracks trigger strength/sequence length — consistent with prefix-cache retention of NaN-contaminated state, not with the garbage generation itself. A runner-level recycle-on-degenerate-decode guard would bound it.

Fix

fp16 accumulation is structurally unsafe for this family's multimodal matmuls — the fine→poison margin is ~40% on a tower that idles at 0.57× the ceiling. ≥fp32 accumulation for the vision tower/merger GEMMs must be the default. Two working forms in this repo:

1. Merged (llm: force f32 cuBLAS compute for qwen25vl runners #215): the launcher injects GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32 into qwen25vl runners (one runner = one model, so the process-wide knob is model-scoped; operator env wins). Validated: three-way A/B causal on both GPU generations, synthetic-trigger cells, full preflight PASS with zero drift for other arches, plus a permanent poison_probe preflight canary.
2. Shelved upstreamable form (branch fix/clip-mm-prec-f32): clip-graph post-pass setting GGML_PREC_F32 on every f16-weight mul_mat (compat patch 904) — the shape llama.cpp itself could take; also note clip.cpp's non-FA KQ F32 override sits commented out upstream.

bf16 compute also heals at parity (cuBLAS bf16 GEMMs accumulate fp32) and is validated on sm_75 despite no native bf16 tensor cores.

Prior art

• Qwen2.5-VL 3B crashes with HTTP 500 on specific PNG image ollama/ollama#14170 — 3B + one specific PNG, crash-era presentation of the same class
• Typhoon OCR 1.5 3B produces deterministic degenerate output ("@" repeated) on RTX 50-series (Blackwell) GPU via Ollama — works correctly on CPU and on other vision models on the same GPU ollama/ollama#17687 — qwen2.5-vl-derived 3B, '@'×31 on GPU / correct on CPU: the same fingerprint; per-checkpoint trigger set
• Eval bug: [Bug] Qwen2.5-VL-7B-Instruct produces garbled output on llama.cpp b9010 (qwen2vl) ggml-org/llama.cpp#23608 — the 7B garbled-output episode

Full record

Diagnosis and evidence trail: #214 (doc docs/maxusai/qwen25vl-3b-poison-image-garbage-decode.md + comment thread). Fix, validation and regression canary: #215 (docs/maxusai/qwen25vl-cublas-f32-env.md, docs/maxusai/clip-mm-prec-f32-validation.md, docs/maxusai/vision-suite/preflight/).

🤖 Generated with Claude Code

[glennneuber]

Filed upstream: ollama/ollama#18070

The fix from #215 is now submitted to upstream ollama, carrying the full evidence chain from this report: the one-minute synthetic reproducer (checkerboard PNG linked and rendered inline, md5 afc8ff7e84ee8958878b44675565d5b0), the measured mechanism, the release matrix, the blast-radius table across every vision model in the estate, and prior art (huggingface/transformers#33294, ggml-org/llama.cpp#20081).

Two results landed after this issue was written and are worth recording here:

1. The failing node is now named. Instrumenting the clip graph with a non-finite eval callback identifies ffn_down-31 (MUL_MAT, src0 v.blk.31.ffn_down.weight f16 [3420,1280]) as the first node to go non-finite — 3 inf out of 15,728,640 elements. With f32 accumulation the same run reports zero non-finite nodes. A minimal llama.cpp fix is therefore a single ggml_mul_mat_set_prec(GGML_PREC_F32) in clip_graph::build_ffn, validated end to end and carried as llama/compat/905-fix-clip-ffn-down-prec-f32.patch.

2. typhoon-ocr1.5-3b IS affected — this overturns the earlier finding. The PR #214 sweep concluded it did not reproduce on this estate; that sweep never tried ead2a6c7 or the synthetic checkerboard, and both garble it (2/7 triggers versus the base model's 7/7). It is the model in upstream ollama#17687, which therefore now has a reproducer needing no private data — and ollama resolves it to arch qwen25vl, so the shipped gate heals it.

Full investigation, including the four hypotheses that were falsified along the way: #224.