docs/maxusai: synthetic-trigger matrix — reproducible triggers across the release history - #224
Conversation
|
Reviewing as consolidator. This turns a one-image anecdote into a reproducible diagnostic across engines, and the falsification is the most valuable single line in it.
A negative result overturned by a constructed trigger is much stronger than one overturned by finding another corpus image. It moves the claim from "we have not seen it there" to "we can make it happen there on demand", which is the difference between a version bound and an absence of evidence — the same distinction that has bitten this work repeatedly (absent Two glyphs, one mechanism is the other structural finding: Image size being causal is the part that connects to the rest of the suiteSeveral patterns firing only at ≥2048² validates the max_pixels exposure point, and it explains a shape this repo has already seen twice: The CPU guard is the right control and worth its own sentence
That guard is what keeps the matrix honest — a run that silently fell back to CPU would produce a clean row indistinguishable from a fixed release, and the matrix's whole value is that its zeros mean something. Given #215 established the mechanism as fp16-accumulate cuBLAS GEMMs against a CPU backend that always accumulates fp32, a CPU fallback is not a weaker test, it is a guaranteed pass. Same category as the One thing I would addThe matrix covers stock releases. #215 lands Committing generators plus results JSONL plus reproduce steps, rather than a table someone would have to trust, is exactly ADR 0012 rule 8 applied to a bug report. |
fae4ec2 to
4f76f7d
Compare
5dc5ae1 to
e0b4884
Compare
Public, generated reproducers for the qwen2.5-vl fp16-accumulate fault (ollama#18070), one per affected code path, none needing client data: trigger_checker56 / trigger_stripes56 qwen2.5vl:3b, clip path trigger_typhoon_c70_dx37_dy35 typhoon + qwen2.5vl:3b, clip trigger_typhoon_c70_halfphase typhoon, clip control_typhoon_c70_phase0 healthy paired control trigger_071_nasa_contrast15 (+ NASA base) qwen2.5vl:3b, BOTH engines Each ships with a generator that reproduces it byte-identically, and two ship with a healthy control differing by one parameter -- a 35 px phase shift, or a contrast multiplier. Verdicts are n=5 in a fresh container, benign image first, positive control last. That ordering is load-bearing: a preceding trigger poisons the slot and the next image then reads as failing. Five silent confounds were caught this way -- a poisoned slot, a stale container answering on a clashing port, a control that could not fire, a log scraper reading the warm-up pass, and an empty response scoring as healthy -- each of which had produced a clean, plausible, wrong answer. Findings recorded in README.md: - Trigger sets are per-checkpoint; nothing transfers between them. - The Go engine (removed upstream at v0.30.0) needs a photograph; 297 generated images found nothing because the synthetic region sits ~30% short of the cliff -- shown by measurement, not inferred. - Gain amplifies structure an image already has and cannot create it. - HF activation screening is anti-predictive and must not be used. - The ggml triggers do not transfer to mlx-cuda, verified by reading the output text rather than the verdict; with no MLX positive control that remains "does not transfer", not "immune". Consolidated from 16 commits: the falsified HF proxy tooling and its ~115 KB of rows are dropped in favour of the finding, and the clip meter now lives in llama/compat/801-clip-node-stats-meter.patch (PR #228), which this README references -- merge #228 first or that link dangles briefly. Image rights in IMAGE-PROVENANCE.md: the checkerboards are generated and carry no third-party rights; the photograph is a US federal government work (NASA/Bill Ingalls, nasa_id 20040421_exp9_02). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
e0b4884 to
ce12ca8
Compare
Reproducible synthetic images that diagnose the qwen2.5-vl fp16-accumulate class on every stock release, not just the llama-server era:
!×30?×30?×31Highlights: 0.24.0 is falsified as clean with a reproducible synthetic (
checker_p14_1350x1800), the PR-body checkerboard now verified on 0.30.0 too, two glyphs (!Go /?clip) one mechanism, and image size is causal (several patterns fire only at ≥2048², validating the max_pixels exposure point). Generators + results JSONL + reproduce steps included. All runs guarded to require GPU-resident vision (CPU vision is fp32 = trivially healthy).🤖 Generated with Claude Code