docs/maxusai: document the qwen2.5vl-3b poison-image serving bug

[glennneuber]

Bug REPORT only (issues are disabled on this repo) — no fix attempted.

One corpus image garbages qwen2.5vl:3b in every quant (q4_K_M / q8_0 / fp16) and permanently poisons the resident runner until ollama stop. Reproduced fresh-instance on the vsuite build (sync-0.32.15), on 0.32.14-rc0-dynres, and — decisively — on STOCK ollama/ollama:0.32.15 with the same registry blob: this is an upstream ollama bug, not a fork regression, and should be filed upstream too. The 7B processed the same image cleanly across a 2,496-request run; a pixel-identical lossless re-save of the image still triggers it (pixel content, not container). Suspect: the 3B (2048-embed) mmproj/merger translation path emitting NaN/garbage projector output.

Full minimal repro, the H/X evidence matrix across builds and quants, server launch flags, and downstream impact (experiment 00017.8's 3B ladder rungs blocked) are in the added doc: docs/maxusai/qwen25vl-3b-poison-image-garbage-decode.md.

🤖 Generated with Claude Code

[glennneuber]

Reviewing as consolidator. The isolation here is the strongest part and it is what makes this filable upstream rather than an internal curiosity.

Four variables eliminated, each by a distinct test rather than by argument:

• not a fork regression — reproduced on stock ollama/ollama:0.32.15 with the same registry blob
• not a build — vsuite sync-0.32.15 and 0.32.14-rc0-dynres both
• not a quantisation — q4_K_M / q8_0 / fp16 all
• not the container format — a pixel-identical lossless re-save still triggers it, so it is pixel content

And the 7B processing the same image cleanly across a 2,496-request run is the control that turns "this model family is fragile" into "this is the 3B's path". That is a much narrower claim and a much more useful one.

The persistence is the part I would put first

permanently poisons the resident runner until ollama stop

That is the detail that changes how everyone else should treat this. A one-off garbage response is a bad cell; a runner that stays poisoned means every subsequent request in that process is suspect, including ones that look fine. Any campaign that touched this image and kept serving has cells after it that cannot be trusted, and nothing in a score file would show it.

It is also the same hazard class the suite's cold-server-per-cell rule exists for — the upstream cross-request leakage (ollama#17459 / ollama#17475) that RESTART_CMD was introduced to keep out of benchmark numbers. Worth cross-referencing in the doc, because an operator who reads "poisons the runner" and knows about RESTART_CMD will correctly conclude that a campaign run with cold restarts is bounded to one cell, while a run without them is not bounded at all.

One thing worth adding while the repro is fresh

The suspect — "the 3B (2048-embed) mmproj/merger translation path emitting NaN/garbage projector output" — is testable one step short of a fix: if the projector output is NaN, that is visible in the runner at debug/trace level without touching the model. Confirming or eliminating NaN would sharpen the upstream report considerably, since "garbage decode" and "NaN in the projector" invite very different investigations from a maintainer.

Not asking for it here — this is explicitly a report, and stopping at the boundary of what was measured is right.

Filing

Since issues are disabled here, this doc is the artifact, same as #201 and #210. Worth noting in the file whether the upstream issue has been opened and its number once it is, so the two do not drift — a local bug report that outlives its upstream counterpart is how a fixed defect stays documented as open.

[glennneuber]

Upstream prior art found (2026-08-26 search):

• Qwen2.5-VL 3B crashes with HTTP 500 on specific PNG image ollama/ollama#14170 (open since Feb, 1 comment): Qwen2.5-VL 3B + one specific PNG → HTTP 500 + runner crash-loop; same-dimension sibling image fine. Same class as this report — in v0.15.6 it crashed, by 0.32.15 it degrades to poisoned garbage instead.
• Typhoon OCR 1.5 3B produces deterministic degenerate output ("@" repeated) on RTX 50-series (Blackwell) GPU via Ollama — works correctly on CPU and on other vision models on the same GPU ollama/ollama#17687 (open, Aug 12): qwen2.5-VL-derived 3B (Typhoon OCR 1.5) emits a single repeated character '@' × 31 on GPU, correct on CPU. Our garbage is '?' × 31 — identical degenerate signature at identical length.
• Eval bug: [Bug] Qwen2.5-VL-7B-Instruct produces garbled output on llama.cpp b9010 (qwen2vl) ggml-org/llama.cpp#23608 (closed/fixed May): the 7B garbled-output episode — engine precedent, 3B case never covered.

Neither upstream issue has the quant-independence, pixel-content-trigger, poison-until-reload, or 7B-immunity evidence from this report — an upstream filing should consolidate all three.

🤖 Generated with Claude Code

[glennneuber]

Architecture A/B (same build, store shared ro via OLLAMA_MODELS, container pinned per GPU):

GPU	arch	3b-q4_K_M poison-first	7b-q4_K_M same image
RTX PRO 6000	Blackwell	X ('?'×31, slot poisoned)	H
RTX 2080 Ti	Turing	X ('?'×31, fully on-GPU 2.9/2.9 GB, slot poisoned)	H

Architecture-independent — the fault tracks the 3B model/projector, not the GPU generation. Upstream ollama#17687's GPU-vs-CPU split is either a different trigger or misattributed; a CPU-inference run of this repro remains the one untested cell.

🤖 Generated with Claude Code

[glennneuber]

CPU cell run (decisive), and one retraction:

Runtime	3b-q4_K_M poison image	good after
CPU only (NVIDIA_VISIBLE_DEVICES=void, size_vram 0.0 verified)	H — sensible labels, 54s	H
CUDA, RTX PRO 6000 (Blackwell)	X ('?'×31, slot poisoned)	X
CUDA, RTX 2080 Ti (Turing)	X ('?'×31, slot poisoned)	X

The bug is CUDA-path-specific and architecture-independent within CUDA — same blob, same image: CPU answers correctly, every CUDA GPU garbages and poisons the slot. Retracting my previous comment's aside: ollama#17687's GPU-vs-CPU split is consistent with this bug, not misattributed. Refined suspect: a CUDA kernel in the qwen2.5vl-3B (2048-embed) vision/projector path producing NaN/garbage for specific inputs.

🤖 Generated with Claude Code

[glennneuber]

Runtime knob matrix: the garbage survives FA-off, f32/bf16 KV, and MMQ — it is generated inside the shared vision-encoder compute

Follow-up to the CPU-healthy result. We swept every runtime precision/kernel lever the build exposes, fresh container per config (same store, read-only), 3-request probe good → poison → good-after (H = healthy, X = garbage):

Config	GPU	Pattern	Note
control (--flash-attn auto)	RTX 2080 Ti (Turing)	HXH	slot recovers on Turing
OLLAMA_FLASH_ATTENTION=1	Turing	HXH
OLLAMA_KV_CACHE_TYPE=f32	Turing	HXH	full-precision KV: no change
OLLAMA_KV_CACHE_TYPE=bf16	Turing	HXH	accepted (--cache-type-k/v bf16): no change
GGML_CUDA_FORCE_MMQ=1	Turing	HXH	MMQ never touches the f16 vision weights anyway
OLLAMA_FLASH_ATTENTION=0	Turing	XXX	separate defect: runner dies with unexpected EOF on every image request (-b 256 and -b 1024 alike, three relaunches per probe)
OLLAMA_FLASH_ATTENTION=0 (native -b 512)	RTX PRO 6000 (Blackwell)	HXH	FA off does NOT heal the poison request
OLLAMA_FLASH_ATTENTION=0 + num_batch=1024	Blackwell	HXH	same with the batch pinned

Three conclusions:

1. The garbage is produced in vision-encoder compute shared by the FA and non-FA attention paths. Not the FA-path F16 casts alone (the pinned llama.cpp b10488 clip.cpp already sets GGML_PREC_F32 on vision flash-attn — and disabling FA entirely still garbles), not KV precision, not the quantized-matmul kernel choice. The one fp16 stage common to every failing CUDA config and absent from the healthy CPU path is the f16-weight matmul family of the vision tower / merger (fp16-accumulate GEMM on CUDA vs fp32 accumulate on CPU). Consistent with the well-known Qwen2.5-VL fp16 activation-overflow behavior (fp16 fine-tuning garbage vs bf16 clean). Note b10488 clip.cpp carries the non-FA vision-attention F32 override commented out (// F32 may not needed for vision encoders?) — FA-off still garbling means attention is not the (only) overflow site, so restoring that line alone is unlikely to be sufficient.
2. The permanent slot poisoning is separable from the garbage and is FA-linked on Blackwell: FA auto/on → poisoned until reload (XX); FA off → the very next request is healthy (HXH). Turing recovers even with FA on.
3. Turing + --flash-attn off cannot serve this model at all (crash-loop on any image request) — worth its own look, but orthogonal to the poison image.

Workaround status: none reachable via environment. Remaining fix candidates are build/model-side: force F32 precision on the clip graph's mm ops, or ship the vision-tower/mmproj tensors as f32/bf16 in the GGUF so the CUDA GEMMs leave the fp16-accumulate path.

Probe harness: 3-request H/X pattern scripts (fp16_knob_ab.sh / fp16_knob_ab2.sh, containerized per config) — available on request; identical request shape to the original repro (num_ctx 8192, temperature 0, num_predict 250).

[glennneuber]

Known-good implementation found: ollama/ollama:0.7.1 serves the poison image perfectly on the SAME blob and SAME GPUs — the regression window is the CGO-engine removal

Tested ollama/ollama:0.7.1 (first qwen2.5vl-era release) against the same read-only store and the identical blob (e9758e589d44…):

Runtime	Pattern	GPU residency
0.7.1, RTX 2080 Ti (Turing)	HHH	size_vram=6.89GB/6.89GB, 37/37 layers
0.7.1, RTX PRO 6000 (Blackwell)	HHH	same — its CUDA v12 payload compiles sm_120 (ARCHS=…,1200), so this is real Blackwell CUDA, not a CPU fallback

0.7.1 even describes the poison image sensibly and consistently on both GPUs ("Worn and Damaged Wood: the wooden shelf…"). It serves qwen2.5vl via ollama runner --ollama-engine — the Go engine's own vision implementation — not the llama-server/mtmd-clip path current builds use. Bonus: its dynamic-resolution preprocessing is pixel-exact for our smart-resized inputs (980 measured image tokens vs (H/28)*(W/28)=972 expected +8 markers), so it is a like-for-like healthy server, not a downscaling one.

Bisect boundary

The healthy Go implementation and the broken clip-path routing swap in a single commit:

9db4bdba — runner: Remove CGO engines, use llama-server exclusively for GGML models (ollama#16031), 2026-05-29

It deletes model/models/qwen25vl/ (the Go vision encoder that handles this image correctly) and routes the arch to llama-server + the compat mmproj translation. Release mapping: last release without it = v0.24.0 (no stable 0.25–0.29 exists; numbering jumps to 0.30), first release with it = v0.30.0. We are pulling ollama/ollama:0.24.0 and :0.30.0 for the release-level A/B:

• 0.24.0 H + 0.30.0 X → the regression is the engine switch itself (the clip.cpp graph / mmproj translation numerics), with a working reference implementation sitting in git history at 9db4bdba~1.
• 0.24.0 X → the Go era broke earlier (ggml kernel drift between 0.7.1 and 0.24.0); bisect that window instead.
• 0.30.0 H → introduced after the switch (0.30 → 0.32 ggml/clip changes).

Combined with the knob matrix (previous comment): the overflow lives in vision-encoder compute shared by the FA and non-FA clip paths, while the deleted Go implementation of the same math on the same hardware is numerically fine — strongly suggesting the clip.cpp graph's precision choices (f16-weight matmuls without GGML_PREC_F32, including the commented-out KQ override) rather than a GGML-kernel-level defect.

[glennneuber]

Correction to my previous comment: 0.7.1 is NOT a clean implementation — it has its own poison image; the trigger sets are disjoint

Running the full production val fold through ollama/ollama:0.7.1 falsified the "known-good implementation" claim (kept in place above per our retraction discipline; the PR doc now carries this correction in the Root-cause localization section, commit 5932aabe).

Image (md5, shape)	clip.cpp path (0.32.x)	Go engine (0.7.1)
02c9d7e1… (756×1008) — original poison	X (?×31, sticky)	H — sensible answer
04431b0d… (1288×616) — newly found	H — sensible answer	X (!×31, sticky)
ordinary corpus images	H	H

Key facts:

• 0.7.1 processed 8 fold rows healthily including the original poison image, then 04431b0d… returned done_reason: null + !×31 and poisoned the slot (our harness's consecutive-unhealthy tripwire caught it at 10).
• On a freshly reloaded slot, 04431b0d… garbles request Gemma 4 visual token budgets (image_min_tokens / image_max_tokens) — on last Go-runner base #1 → image-specific trigger, not state decay.
• Cross-probe on a fresh 0.32.15 container: 04431b0d… → H, original poison → X. Disjoint sets, identical signature (only the degenerate glyph differs: ! vs ?).

Interpretation: the overflow class is latent in the shared CUDA fp16 vision compute of both implementations; each graph's op order decides which activation patterns cross the fp16 cliff, giving each engine its own poison set. This strengthens the f16-weight-matmul localization and narrows the fix guidance: a clip.cpp-only precision patch would merely move the trigger set — the durable fix keeps the vision tower/merger GEMMs out of fp16 accumulation (F32 prec on the mm ops, or f32/bf16 vision tensors in the GGUF) in whichever engine serves.

The 0.24.0/0.30.0 release A/B is reframed accordingly (per-image trigger boundary, not class introduction). First 0.24.0 probe attempt timed out on its first request (>300 s on the Turing card) — re-probing with a longer timeout and both trigger images; results to follow.

[glennneuber]

Release matrix complete + operational workaround (options.num_gpu = layers-1)

Full matrix now measured with BOTH trigger images, fresh container per cell, residency verified — table and analysis promoted into the PR doc (commit ee70371b). Headlines:

• The clip-path trigger has been there since the very first llama-server release: 0.30.0 with the mmproj actually on GPU garbles 02c9d7e1… exactly like 0.32.15. Earlier "0.30.0 healthy" was a fit-check artifact — on a crowded card ollama emits --no-mmproj-offload and the vision path silently runs on CPU.
• 0.24.0 (Go engine, fully GPU-resident) passes BOTH known triggers — the Go path changed between 0.7.1 (which has its own poison, 04431b0d…) and 0.24.0. Its own unknown trigger set is not excluded; per-implementation sets are the rule here.
• mmproj offload is memory-conditional (shouldDisableMMProjOffload: cpu-only / partial-text-offload / projector+1 GiB fit) — poison exposure varies with free VRAM at load time, which explains intermittent reports of this model working.
• Workaround verified on 0.32.15: request options.num_gpu one below the model's layer count (36 for the 3B) → partial-text-offload → --no-mmproj-offload → vision runs on CPU fp32, LM stays on GPU: HHHH on both triggers. Poison-proof by construction, at the cost of CPU vision encode + one CPU text layer.
• Persistence half: 0.30.0 (launches without --context-shift --keep 4) recovers after a poison request; 0.32.x (with them) stays poisoned until reload — the prefix cache appears to retain the NaN state. Suggest treating the sticky slot as its own fix item.

[glennneuber]

Root cause CONFIRMED: fp16 cuBLAS accumulation — and there is a runtime kill-switch

Glenn asked whether GGML_CUDA_CUBLAS_COMPUTE_TYPE had been tried. It exists in the pinned llama.cpp (b10488, runtime getenv in ggml-cuda.cu) and it settles the whole investigation (doc updated, commit 4f3d015c):

GGML_CUDA_CUBLAS_COMPUTE_TYPE (sync-0.32.15, GPU0, fully GPU-resident)	02c9d7e1…	slot after
unset (auto → CUBLAS_COMPUTE_16F for f16 weights)	X (?×31)	sticky
f32	H	clean
bf16	H	clean
f16 (explicit positive control)	X (?×31)	sticky

One env var flips the failure on and off at full GPU residency — the poison class is fp16 accumulation in the cuBLAS GEMM path, exactly as the elimination matrix predicted. bf16 is likely perf-neutral (tensor cores, fp32 accumulate); f32 is the conservative choice. Upstream fix guidance sharpens to: run the multimodal (vision tower/merger) matmuls with ≥fp32 accumulation by default — the per-image "poison" behavior, the per-implementation trigger sets, and the fine-tuned-checkpoint severity (typhoon/ollama#17687) are all just different activation magnitudes meeting the same 16-bit accumulator.

Also correcting the earlier knob matrix: the GGML_CUDA_FORCE_MMQ=1 row was an inert env — neither GGML_CUDA_FORCE_MMQ nor GGML_CUDA_FORCE_CUBLAS exists in b10488's runtime env set — so that row duplicated the control and proved nothing about MMQ.

[glennneuber]

Latest-release status: the class is ALIVE in 0.33.0 — and the kill-switch works there too. Typhoon (ollama#17687) closure.

New cells on stock images (fresh container per cell, all fully GPU-resident on Blackwell):

Cell	Pattern	Reading
0.33.0 (latest), qwen 02c9d7e1…	HHXH	the poison class ships in the current release
0.33.0 + GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32	HH	the runtime kill-switch works on latest upstream
0.32.9 (the ollama#17687 reporter's version), qwen triggers	HHXH	our trigger reproduces on their version
0.32.9 + 0.33.0 + current stack, typhoon photos AND a synthetic dense-text document page	all H	typhoon does not reproduce on this estate at all

Two refinements:

1. Slot-persistence timeline: sticky-until-reload occurs only on 0.32.15 (stock and fork alike); 0.30.0, 0.32.9, and 0.33.0 all recover on the next request. The sticky half of the defect appeared in the 0.32.10–0.32.15 window and is gone by 0.33.0 — supporting the prefix-cache/--context-shift --keep retention theory and confirming garbage-generation and slot-poisoning are separable defects.
2. Typhoon OCR 1.5 3B produces deterministic degenerate output ("@" repeated) on RTX 50-series (Blackwell) GPU via Ollama — works correctly on CPU and on other vision models on the same GPU ollama/ollama#17687 (typhoon-ocr1.5-3b): not reproducible here in any configuration — every release from 0.7.1 to 0.33.0, photo inputs, both of our qwen trigger images, and an invoice-style synthetic document page, always full-GPU. Their 100%-failure symptom is environment- or image-specific on their side; the corpus trigger images in this PR remain the canonical repro for the class. The shared 31-glyph degenerate-decode fingerprint still marks it as the same family.

Upstream-ready summary: deterministic single-image repro on the CURRENT release, mechanism pinned to fp16 cuBLAS accumulation with a three-way runtime A/B (f32/bf16 heal, f16 reproduces), a one-line env workaround, and a concrete fix direction (≥fp32 accumulation for the multimodal matmuls by default).

[glennneuber]

Why a poison image poisons — measured: final-block massive activations at 0.77–0.85× the fp16 ceiling

We hooked every stage of the HF Qwen2.5-VL-3B-Instruct vision tower in bf16 (fp32-like range, so true magnitudes survive) and ran a known-good corpus image against both trigger images, on the same smart-resized pixels the serving path consumes. Script committed in this PR: docs/maxusai/measure_qwen25vl3b_vision_activations.py (commit 6567dd84).

stage (max abs activation)	good image	02c9d7e1…	04431b0d…
blocks.31.mlp (final block)	37,376	55,808	50,688
mid-tower plateau (blocks 17–27)	~13,000	~12,200	~13,800
relative to fp16 max finite (65,504)	0.57×	0.85×	0.77×

The 3B vision tower carries a massive-activation outlier structure in its final block that sits above half of fp16's ceiling for ordinary images — every input walks the cliff edge. A "poison" image is one whose patch statistics resonate with those outlier channels and push them ~50% higher: the stored tensors still fit in fp16, but the partial sums inside the next fp16-accumulate GEMM (block-31's MLP projection / the merger consuming it) transiently exceed 65,504 before cancellation, overflow to inf, and cascade to NaN → the 31-glyph degenerate decode.

This one mechanism accounts for every behavior in this PR's matrix: deterministic per-image failure; rare-but-inevitable triggers across a large corpus (the fine→poison margin is only ~40%); per-implementation trigger sets (different engines tile GEMMs differently, and fp16 overflow depends on summation order, not just final magnitudes); fine-tuned checkpoints failing harder (typhoon-ocr / fp16 bitsandbytes training shift the outlier scales so every input crosses); the 7B's relative immunity (lower final-block outliers against the same ceiling); and the immunity of CPU, f32, and bf16 accumulation (the ceiling moves from 6.5×10⁴ to ~3.4×10³⁸).

Fix guidance, final form: the multimodal matmuls of this family cannot safely run fp16 accumulation — the margin is structurally too thin. GGML_CUDA_CUBLAS_COMPUTE_TYPE=bf16 (perf-neutral) or f32 should be the default for the vision tower/merger, per the three-way A/B earlier in this thread.

[glennneuber]

Shareable synthetic repro — no client data needed — plus: image SIZE is a trigger axis

Two updates, one of which corrects my earlier claim in this thread.

1. A three-line generator reproduces the bug on stock everything

docs/maxusai/make_poison_repro_image.py (committed with the exact PNG we tested, ab34a271) writes a 56 px black/white checkerboard at 1350×1800. On stock ollama/ollama:0.32.9, stock qwen2.5vl:3b-q4_K_M, fully GPU-resident:

serving	result
default (fp16 cuBLAS accumulate)	?×31, done_reason: null — deterministic, every call
GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32	healthy, describes the checkerboard

That is the whole bug — trigger, mechanism, and fix — reproducible by anyone in under a minute with no corpus image and no fine-tuned checkpoint.

2. Correction: synthetic triggers DO transfer — I tested the wrong kind

Earlier I reported that a gradient-ascent image reaching 2.53× the fp16 ceiling decoded cleanly, and concluded synthetic triggers don't transfer. The right conclusion was narrower: adversarially-optimized noise doesn't transfer (it is brittle to weight quantization), while ordinary high-contrast patterns at a large working size do.

3. Image size is a first-class trigger axis

Max |activation| at blocks.31.mlp (bf16 HF tower; fp16 ceiling 65,504) vs serving outcome on 0.32.9:

image	size	max abs	vs fp16	serving
corpus poison 04431b0d	1288×616	50,688	0.77×	H (X on 0.7.1)
same, upscaled → 1800	1800×860	66,048	1.01×	X
same, tiled 2×2 → 1800	1800×860	70,144	1.07×	X
ordinary corpus image	1008×756	37,376	0.57×	H
same, upscaled → 1800	1800×1350	59,392	0.91×	H
synthetic checkerboard p56	1350×1800	69,120	1.06×	X
synthetic checkers p14/p28, rings p5	1350×1800	50,432–65,280	0.77–1.00×	X

Two compounding mechanisms: upscaling the same content raises the activation peaks (~1.6× from 1008→1800 px), and longer visual sequences lengthen the GEMM reductions so partial sums overflow even when stored maxima stay under the ceiling (synthetic rings garble at 0.77×, while a natural image at 0.95× does not).

Consequence for clients: the larger the image you send, the more exposed you are. Any olmOCR-style pipeline that renders pages at 1800–2048 px sits in the worst part of this space — including typhoon_ocr's ocr_document(target_image_dim=1800) from upstream ollama#17687.

4. Cross-model status

scb10x/typhoon-ocr1.5-3b stayed healthy on every image above, across 0.7.1/0.24.0/0.30.0/0.32.9/0.32.15/0.33.0, photos and documents, including its own package's 1800 px pipeline reproduced from source. Its fine-tuned vision weights carry different activation scales, so its trigger set is disjoint from the base model's — the same per-implementation pattern we measured between ollama's Go and clip engines. Pinning their exact trigger would need that checkpoint's HF weights and the same measurement sweep.