Skip to content

docs/maxusai: synthetic-trigger matrix — reproducible triggers across the release history - #224

Merged
glennneuber merged 1 commit into
mainfrom
docs/synthetic-trigger-matrix
Aug 29, 2026
Merged

docs/maxusai: synthetic-trigger matrix — reproducible triggers across the release history#224
glennneuber merged 1 commit into
mainfrom
docs/synthetic-trigger-matrix

Conversation

@glennneuber

Copy link
Copy Markdown

Reproducible synthetic images that diagnose the qwen2.5-vl fp16-accumulate class on every stock release, not just the llama-server era:

Release Engine X/54 Fingerprint
0.7.1 Go 0
0.24.0 Go 7 !×30
0.30.0 clip 9 ?×30
0.32.9/.15/0.33.0 clip prior ?×31

Highlights: 0.24.0 is falsified as clean with a reproducible synthetic (checker_p14_1350x1800), the PR-body checkerboard now verified on 0.30.0 too, two glyphs (! Go / ? clip) one mechanism, and image size is causal (several patterns fire only at ≥2048², validating the max_pixels exposure point). Generators + results JSONL + reproduce steps included. All runs guarded to require GPU-resident vision (CPU vision is fp32 = trivially healthy).

🤖 Generated with Claude Code

@glennneuber

Copy link
Copy Markdown
Author

Reviewing as consolidator. This turns a one-image anecdote into a reproducible diagnostic across engines, and the falsification is the most valuable single line in it.

0.24.0 is falsified as clean with a reproducible synthetic (checker_p14_1350x1800)

A negative result overturned by a constructed trigger is much stronger than one overturned by finding another corpus image. It moves the claim from "we have not seen it there" to "we can make it happen there on demand", which is the difference between a version bound and an absence of evidence — the same distinction that has bitten this work repeatedly (absent readelf, absent strings, an empty log window read as a format gap).

Two glyphs, one mechanism is the other structural finding: !×30 on the Go engine and ?×30/31 on clip. A reader hitting either in isolation would reasonably file two bugs. Establishing that the fingerprint is engine-specific while the class is not is what stops that.

Image size being causal is the part that connects to the rest of the suite

Several patterns firing only at ≥2048² validates the max_pixels exposure point, and it explains a shape this repo has already seen twice: qwen35moe aborting only at the ladder's top rung (3072×1728), and the geometry campaign finding contract behaviour that varies by size. It also means the ladder's largest geometry is the diagnostic rung, not merely the expensive one — worth stating in the vision-suite README next to the rung table, since someone economising a campaign would naturally drop the top rung first and would be dropping the only one that exercises this.

The CPU guard is the right control and worth its own sentence

All runs guarded to require GPU-resident vision (CPU vision is fp32 = trivially healthy)

That guard is what keeps the matrix honest — a run that silently fell back to CPU would produce a clean row indistinguishable from a fixed release, and the matrix's whole value is that its zeros mean something. Given #215 established the mechanism as fp16-accumulate cuBLAS GEMMs against a CPU backend that always accumulates fp32, a CPU fallback is not a weaker test, it is a guaranteed pass. Same category as the strings-absent false negative, and correctly pre-empted here.

One thing I would add

The matrix covers stock releases. #215 lands GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32 arch-scoped for qwen25vl — so there is now a fork build that should score 0/54 for a reason we understand, and running the matrix against it converts the fix from "the knob heals the one image we had" into "the knob heals the constructed trigger set across sizes and patterns". That is the strongest available evidence for the upstream report, and the generators make it cheap.

Committing generators plus results JSONL plus reproduce steps, rather than a table someone would have to trust, is exactly ADR 0012 rule 8 applied to a bug report.

Public, generated reproducers for the qwen2.5-vl fp16-accumulate fault
(ollama#18070), one per affected code path, none needing client data:

  trigger_checker56 / trigger_stripes56       qwen2.5vl:3b, clip path
  trigger_typhoon_c70_dx37_dy35               typhoon + qwen2.5vl:3b, clip
  trigger_typhoon_c70_halfphase               typhoon, clip
  control_typhoon_c70_phase0                  healthy paired control
  trigger_071_nasa_contrast15 (+ NASA base)   qwen2.5vl:3b, BOTH engines

Each ships with a generator that reproduces it byte-identically, and two
ship with a healthy control differing by one parameter -- a 35 px phase
shift, or a contrast multiplier.

Verdicts are n=5 in a fresh container, benign image first, positive
control last. That ordering is load-bearing: a preceding trigger poisons
the slot and the next image then reads as failing. Five silent confounds
were caught this way -- a poisoned slot, a stale container answering on a
clashing port, a control that could not fire, a log scraper reading the
warm-up pass, and an empty response scoring as healthy -- each of which
had produced a clean, plausible, wrong answer.

Findings recorded in README.md:
  - Trigger sets are per-checkpoint; nothing transfers between them.
  - The Go engine (removed upstream at v0.30.0) needs a photograph;
    297 generated images found nothing because the synthetic region sits
    ~30% short of the cliff -- shown by measurement, not inferred.
  - Gain amplifies structure an image already has and cannot create it.
  - HF activation screening is anti-predictive and must not be used.
  - The ggml triggers do not transfer to mlx-cuda, verified by reading
    the output text rather than the verdict; with no MLX positive control
    that remains "does not transfer", not "immune".

Consolidated from 16 commits: the falsified HF proxy tooling and its
~115 KB of rows are dropped in favour of the finding, and the clip meter
now lives in llama/compat/801-clip-node-stats-meter.patch (PR #228), which
this README references -- merge #228 first or that link dangles briefly.

Image rights in IMAGE-PROVENANCE.md: the checkerboards are generated and
carry no third-party rights; the photograph is a US federal government
work (NASA/Bill Ingalls, nasa_id 20040421_exp9_02).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@glennneuber
glennneuber force-pushed the docs/synthetic-trigger-matrix branch from e0b4884 to ce12ca8 Compare August 29, 2026 02:10
@glennneuber
glennneuber merged commit 71ee818 into main Aug 29, 2026
1 check passed
@glennneuber
glennneuber deleted the docs/synthetic-trigger-matrix branch August 29, 2026 02:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant