llama/compat: env-gated clip node meter (new 8xx diagnostics band) - #228
llama/compat: env-gated clip node meter (new 8xx diagnostics band)#228glennneuber wants to merge 1 commit into
Conversation
…and) 801-clip-node-stats-meter.patch adds a per-node meter to the clip graph. With OLLAMA_CLIP_NODE_STATS=<name substring> (or *) each matching node's output is scanned after evaluation and its max |value|, counts above 32k/49k/60k and non-finite counts are logged. Inert otherwise: without the variable the eval callback is never registered, so a normal build and a normal run are unchanged. Why carry it. The qwen2.5-vl fp16-accumulate fault (ollama#18070) is invisible from the product surface -- the observable is garbled text, the cause is a handful of elements out of millions overflowing at v.blk.31.ffn_down. Diagnosing that without an instrument degenerates into sampling, and sampling cannot separate "no fault" from "not found yet". Reconstructing the meter per incident cost hours; in-tree it costs a build. It covers every mtmd model and any node, not just this one, because the eval callback already sees the whole graph. New 8xx band, documented in llama/compat/README.md: diagnostics that are inert unless enabled. Distinct from 0xx (compatibility shims, leave when published models do) and 9xx (fork-carried upstream defect fixes, leave when the pin moves past them) -- 8xx is not waiting on anything and is not expected to leave. On a pin bump an 8xx failure means regenerate against the new anchor, not delete. 41 inserted lines at one anchor, the eval-callback site in the clip_ctx constructor. Verified from a clean fetch of b10488: full band incl. 801 applies and the mtmd target builds. Runtime behaviour was validated earlier with identical code -- calibrated against a known trigger (headroom 1.201) versus confirmed non-triggers (0.72-0.75). Usage, calibration discipline and the trigger corpus: docs/maxusai/vision-suite/synthetic-triggers/README.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Reviewing as consolidator. The band is justified and the instrument earns its keep; I checked the CI coverage question because a new band is exactly where it goes missing. CI does cover it, by construction rather than by luckI traced both patch jobs:
So the new band inherits coverage automatically. That is worth stating in the README section, because #137 landed The band itself
Why I think carrying it is right
That is the sentence, and this month has three instances of it: a blind sweep of 297 generated images finding nothing while the meter explained why in an hour; The generality argument holds too — the eval callback already sees every node, so this is not a qwen2.5-vl tool. Node-by-node backend diffing is the shape of Two small thingsInert-by-default is the property to keep pinned. Without the variable the callback is never registered — worth one line in the README stating that a future 8xx patch must hold the same property, since the band's whole justification is that carrying it costs nothing when unused.
|
Public, generated reproducers for the qwen2.5-vl fp16-accumulate fault (ollama#18070), one per affected code path, none needing client data: trigger_checker56 / trigger_stripes56 qwen2.5vl:3b, clip path trigger_typhoon_c70_dx37_dy35 typhoon + qwen2.5vl:3b, clip trigger_typhoon_c70_halfphase typhoon, clip control_typhoon_c70_phase0 healthy paired control trigger_071_nasa_contrast15 (+ NASA base) qwen2.5vl:3b, BOTH engines Each ships with a generator that reproduces it byte-identically, and two ship with a healthy control differing by one parameter -- a 35 px phase shift, or a contrast multiplier. Verdicts are n=5 in a fresh container, benign image first, positive control last. That ordering is load-bearing: a preceding trigger poisons the slot and the next image then reads as failing. Four silent confounds were caught this way (poisoned slot, a stale container answering on a clashing port, a control that could not fire, a log scraper reading the warm-up pass); each had produced a clean, plausible, wrong answer. Findings recorded in README.md: - Trigger sets are per-checkpoint; nothing transfers between them. - The Go engine (removed upstream at v0.30.0) needs a photograph; 297 generated images found nothing because the synthetic region sits ~30% short of the cliff -- shown by measurement, not inferred. - Gain amplifies structure an image already has and cannot create it. - HF activation screening is anti-predictive and must not be used. - The ggml triggers do not transfer to mlx-cuda, but with no MLX positive control that is "does not transfer", not "immune". Consolidated from 16 commits: the falsified HF proxy tooling and its ~115 KB of rows are dropped in favour of the finding, and the clip meter now lives in llama/compat/801-clip-node-stats-meter.patch (PR #228). Image rights in IMAGE-PROVENANCE.md: the checkerboards are generated and carry no third-party rights; the photograph is a US federal government work (NASA/Bill Ingalls, nasa_id 20040421_exp9_02). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Carries the diagnostic that found the qwen2.5-vl fp16-accumulate fault into the compat band, so it is one build away rather than one investigation away.
What it does
OLLAMA_CLIP_NODE_STATS=<name substring>(or*) logs, for every matching clip graph node, itsmax_abs, counts above 32k/49k/60k, and non-finite counts.Run under
GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32to read magnitudes without overflowing;hr = max_abs / 65504is the fp16 headroom.Inert unless enabled — without the variable the eval callback is never registered. No behaviour change to a normal build or run.
Why carry it rather than rebuild per incident
The fault it found is invisible from the product surface: garbled text out, and the cause is 3 elements of 15.7 M overflowing at
v.blk.31.ffn_down. Without an instrument, debugging degenerates into sampling — and sampling cannot distinguish no fault from not found yet. Concretely, a blind sweep of 297 generated images found nothing on one engine; the meter showed why in about an hour (the whole synthetic region sits ~30 % short of the cliff).It is also not specific to this bug. The eval callback already sees every node, so the filter covers all mtmd models and any node — including diffing two backends node-by-node, which is the shape of several defects this fork has carried (
903-fix-mmq-ids-padding, the mlx-cuda thrashing check, this one).New 8xx band
0xxcompat shims,9xxupstream defect fixes — a diagnostic is neither, and unlike both it is not waiting on anything, so it is not expected to leave. Documented inllama/compat/README.md, including that an 8xx apply failure after a pin bump means regenerate, not delete.Verification
clip_ctxconstructor).b10488: full compat band incl.801applies and themtmdtarget builds.Caveat worth keeping
An uncalibrated meter fails silently, not loudly — this one returned a correctly-sized array of zeros on one backend during development. Calibrate against a known trigger before trusting any run. Recorded in the README.