QVAC-23075 feat: add VisionPsy Nano support and its Flash preprocessing rule

[yingying0906]

Overview

What problem does this PR solve?

• VisionPsy Nano is not supported at all. Its published mmprojs declare clip.projector_type = "custom", so fabric cannot load them.
• Its two checkpoints, base and Flash, are indistinguishable to fabric. They differ in exactly one thing, resize_to_max_side_len, but their mmprojs declare identical vision hparams, so nothing in the GGUF separates them and Flash silently ran base preprocessing, inflating a 256x256 image into a 4x4 grid of upscaled slices.
• get_tensor and set_tensor disagreed about the Adreno q8_0 transpose: set_tensor returns early for views and always decides on the parent, while get_tensor asked about the view, so the two could pair a transposed writer with a non-transposed reader.

How does it solve it?

• Routes VisionPsy through clip_graph_siglip and reuses the llava-uhd preprocessing, since it is a siglip encoder with an idefics3-style pixel-shuffle merge and a single mm_fc projector. Four deltas from idefics3, all taken from get_image_string() in the reference processors.py: position embeddings interpolated to the actual patch grid, overview image emitted first, the delimiters <|global_image|> and <row_%d_col_%d> with no fake token and no row-end token, and an <image: N> label before each image once a prompt carries more than one. When the slice grid is 1x1 the overview and the single slice are the same crop, so it sends one of them, and it sends the slice: the reference returns the split patch untouched for that grid, and the slice is rendered bicubic where the overview is bilinear.
• Adds a read-only alias table so the published mmprojs load. The alias is gated on general.name as well as the projector string, { "custom", "VisionPsyNano", PROJECTOR_TYPE_VISIONPSY }, because "custom" is not vendor-specific and another model shipping that string must not be silently loaded as VisionPsy; it would get idefics3 preprocessing and a hard <|global_image|> vocab requirement. PROJECTOR_TYPE_NAMES stays canonical and visionpsy is what we write out.
• Adds a null check on the VisionPsy vocab lookup so it throws instead of silently splicing LLAMA_TOKEN_NULL into the prompt if <|global_image|> is missing. lookup_token() itself is unchanged and still returns LLAMA_TOKEN_NULL on a miss for every other projector type, this check is scoped to VisionPsy only. The throw is conditional on there being a vocab to look in, because mtmd_get_memory_usage builds a context with no text model and every token misses there. Without that, every llama-server start with this mmproj lost its mmproj VRAM reservation.
• Adds clip.vision.preproc_no_upscale, read from the GGUF when present, overridable through mtmd_context_params.image_no_upscale and a new --image-no-upscale on|off. Tri-state, -1 keeps the model default, so every existing caller is unaffected. The published mmprojs do not carry the key, so today the flag selects the variant; a re-converted Flash mmproj would need no flag.
• Computes both sizing rules in double, not float32. The reference is Python floats throughout, and where the true product or quotient lands on a multiple of the slice size, float32 rounding pushes it past and the ceil buys a whole extra row or column. On the no-upscale short side, float32 differs from the reference in 171 of every size pair up to 3000x3000 and exact integer arithmetic in 92, double in none. On the shared aspect-preserving refine in calc_size_preserved_ratio, float32 differs in 10, all of them 4:3 or 3:4: 960x720 refined to 2048x2048 and sliced 4x4 where the reference gives 2048x1536 and 4x3, so the image gained 4 slices and 256 image tokens.
• Keeps the idefics3 overview sourced from the original image. An earlier revision of this PR derived it from the refined image, on the grounds that both references refine first, but that is wrong under one padding mode and a downgrade under the other, so it was reverted in a812964c9. Under PAD_CEIL, which every idefics3 member except VisionPsy keeps, the refined image is letterboxed, so the bars would land inside the global image; both references stretch at each step and never letterbox. Under PAD_NONE, which is VisionPsy, the refined resize is a plain stretch, so stretching it again to (p, p) is the same geometry as stretching the original once, and all it adds is a second resample through a 2x2 bilinear tap with no prefilter where the reference resize is antialiased. slice_image and slice_instructions are byte-identical to v10069.0.0.
• Stops letterboxing the VisionPsy refined image. The projector inherited image_pad_rf = PAD_CEIL, so the refined image was aspect-preserved inside black bars where DynamicResize.forward stretches straight to the target. On the 640x488 test image that left 176 black columns on each side of a 1024x512 frame, about a third of the encoded pixels. VisionPsy has its own hparams case, so idefics3 is untouched.
• Keeps the NUL terminator out of string_format in clip-impl.h, which returned std::string(buf.data(), buf.size()) over a buffer sized size + 1. mtmd.cpp does not include common.h, so that is the overload the <image: N> label is built with, and the tokenizer got all size + 1 bytes: llama-tokenize on this vocab gives [44, 5028, 42, 216, 32, 46] for the label and one extra token for the same bytes plus the NUL, so every image in a multi-image prompt carried a garbage token the checkpoint never saw. common/common.cpp already had it right.
• Fails the load, rather than at preprocess time, on metadata the sizing rule cannot use. image_size == 0 reaches GGML_ASSERT(align_size > 0) and aborts the process at the first image, so both idefics3-style projectors reject it. A missing cap makes the refined size {0,0}, an empty grid and an overview-only encode, 64 image tokens where the model expects hundreds: VisionPsy rejects that too, idefics3 warns instead, because the shipped ggml-org/SmolVLM-500M-Instruct-GGUF mmproj carries no cap and still has to load. A cap below one slice is rejected on the no-upscale path, where it would otherwise reach std::clamp with lo > hi, which is undefined behaviour rather than a bad size.
• Documents in mtmd.h that consumers must rebuild against these headers on every libmtmd update. mtmd_context_params is passed by value and gains fields while SOVERSION stays 0, so an old binary linked against a new library can write past the caller's struct. Not bumping SOVERSION: upstream set it to 0 in add version to all shared object files ggml-org/llama.cpp#17091 and has appended fields in mtmd: add mtmd_context_params::warmup option ggml-org/llama.cpp#17652, mtmd: add batching API ggml-org/llama.cpp#24384 and mtmd: add load progress callback ggml-org/llama.cpp#24865 without touching it, and our consumers all build fabric from source through a pinned vcpkg port, so there is no old binary to link.
• Makes get_tensor reconstruct the whole q8_0 parent and read the view out of it at view_offs, the same shape set_tensor already used, and asks enable_adreno_trans_weight about the parent rather than the view. That predicate gates the transpose in set_tensor, the restore in get_tensor and the adreno gemm/gemv selection in mul_mat as a unit, so the three must agree. Sizing the staging buffer and the dispatch from the view was the other half of the same bug: the SoA buffers cover the parent and the transposed layout is indexed by the parent's row count, so a 4096x1 view of a 4096x4096 parent allocated about 4.3 KB and the kernel restored 64 rows into it, about 278.5 KB. kernel_restore_block_q8_0_trans also gets the row guard its rounded-up dispatch needs, which closes the same overflow for any parent whose row count is not a multiple of 64. The q4_0 and mxfp4 readback paths still size from the view and ignore view_offs; that is pre-existing, untouched here, and needs Adreno hardware to confirm.

Three things reviewers may want to check:

• The sizing rule is ported from DynamicResize._get_new_hw in the published processor, not from the vendor patch that was circulated. That patch aligns the refined size to patch_size * n_merge, 64, and gives the overview an aspect-preserving size, where the published model aligns to vit_img_size, 512, and resizes the global image to a square. I verified ours line by line against custom_transforms.py and processors.py on qvac/VisionPsy-Nano-460M-Flash.
• This branch is based on temp-10069, not master. The addon requires qvac-fabric >= 10069.0.0, and master is a separate, older line: it carries neither common/finetune.h nor mtmd_input_text.text_len, so a master-based fabric fails every addon prebuild platform at compile time. master and v10069.0.0 share an ancestor from 2026-04-17 and have diverged 401 and 1616 commits since.
• The vocab-divisibility abort this work originally chased, GGML_ASSERT(M % 4 == 0) with M = ne1 on output.weight and token_embd.weight (960 x 49218, VisionPsy's vocab is 49218), is already fixed on this base, and more strictly: temp-10069 also requires ne2 == 1 && ne3 == 1. The commit that carried our version cherry-picks to empty here and has been dropped.

Additional information

How was it tested?

• tests/test-mtmd-preproc-sizing pins the refined size, the grid and the slice count against the reference rule for 24 cases, both variants, from 256x256 to 4096x3072 and including the extreme aspect ratios. The expected values come from DynamicResize._get_new_hw transcribed to Python and evaluated in doubles, and the Python is in the test header. Putting float32 back fails 960x720, 720x960 and 1920x1440 while 1440x1080 still passes, which is why the class needs several members.
• tests/test-clip-preproc-metadata covers the loader validation on metadata-only mmprojs generated at runtime, so nothing is committed and nothing is downloaded: 16 cases over zero image_size and zero cap with the flag both on and off, a cap below one slice, idefics3 with no cap key loading and warning, and the whole override matrix, -1 keeping the GGUF value, 0 and 1 both applying, off against a GGUF that says on being announced, and a projector that does not read the flag warning.
• tests/test-clip-projector-alias pins the alias to general.name, so custom with any other name stays unknown.
• tools/mtmd/tests.sh checks the tile structure per row rather than only the answer text: 34 image encodes for the base row, which passes two images so the ordinal labels are exercised, and 3 for the Flash row. Both rows also pin the prompt sequence over the -v assembly log, the ordinals, the first and last delimiter of each grid, the slice grid line, the chunk total and the overview position. Mutation tested: blanking ord_img_tmpl leaves both the encode count and the chunk total untouched and is caught only by the ordinal patterns, and flipping ov_img_first leaves them untouched, still answers "the new york times", and is caught only by the overview-position patterns. A 1x1 grid needs an image whose long side is at most 512 and the only committed image is 640x488, so the single-tile path is pinned at the sizing level only.
• No Adreno q8_0 view readback test: no CI leg runs OpenCL on an Adreno device, so it would never execute. The fix is verified by dispatch instead.
• Flash effect on Metal, q8_0. Base is untouched, still 13 slices for 1024x768:

image	slices off to on	encode ms	prefill tokens
256x256	17 to 1	474 to 38	1118 to 78
640x480	13 to 3	363 to 92	858 to 208
1024x768	13 to 5	361 to 139	858 to 338

• DocVQA ANLS at n=30 is 0.825 versus 0.824, unchanged as expected, since those scans exceed the 2048 cap where the two rules converge.
• Backend matrix, vlm-benchmark run Fixed tokenizer.model not found error when model dir is symlink ggml-org/llama.cpp#325, cognitive preset, 175 samples, 0 errors, addon built from this branch. Quality flat at 70.9 to 71.4 everywhere. Desktop Metal is verified locally on an M5 Pro rather than in CI, so it has no row:

backend	device	vision enc ms	quality
Vulkan	RTX 4000 Ada	40.2	71.0
Metal	iPhone 17	1081.7	71.4
CPU	iPhone 17	not recorded	70.9
Vulkan (LLM only, see below)	Pixel 9 Mali	5991.2	71.0
CPU	Pixel 9	5478.5	71.0
CPU	S25	2274.5	71.0

Correction on the Pixel 9 row. That leg ran the language model on Vulkan but the vision encoder on CPU. The addon auto-defaults the projector backend by GPU class (LlamaModel.cpp): GPU on Adreno 800+, CPU on Mali and on any tier it cannot identify. The device logs from run 31409445243 say so directly, four times per device:

multimodal projector backend: CPU (auto-default, Mali GPU)      pixel9
multimodal projector backend: GPU (auto-default, Adreno 800+)   s25

which is also why Pixel 9 GPU encodes slower than its own CPU leg, 7223 ms against 6617 ms. So OpenCL is genuinely validated for the vision encoder on S25 Adreno, and Vulkan for the vision encoder is validated on desktop only. Mali Vulkan for the encoder has not been measured; reaching it needs mmproj-use-gpu set explicitly, which the benchmark now permits.

• The vocab-divisibility fix was verified on hardware before it was found to be redundant here, run 31189679101, q4_0 on an S25 Ultra at smoke. The Adreno leg appears in the report and passes, where before the app crashed and the leg was absent entirely. Tile count matches CPU exactly, so the OpenCL path does the same work rather than skipping it. The equivalent guard on this base is upstream's:

leg	encode ms	TTFT ms	decode tps	tiles
linux GPU Vulkan	141.5	244	386.2	13
s25 CPU	6088.1	7991	105.5	13
s25 GPU Adreno OpenCL	3002.9	4288	79.7	13

• Confirmed again at scale, run 31200494289, Flash q4_0 at the cognitive preset, 125 samples across five legs, 0 errors. Adreno holds up over 25 samples and is now the fastest mobile leg, ahead of its own CPU and both Pixel 9 legs, at identical output quality:

leg	quality %	encode ms	TTFT ms	decode tps
linux GPU Vulkan	71.4	44.5	82	335.3
s25 GPU Adreno OpenCL	71.0	1160.4	1617	70.9
s25 CPU	71.0	2172.2	2696	91.2
pixel9 CPU	71.0	6807.6	8682	13.6
pixel9 GPU Mali Vulkan	71.0	8027.8	10686	35.9

• Local acceptance matrix passes 11 of 11 on Metal and CPU: both checkpoints across q8_0, q4_0, iq3_m and bf16, flash-attn on and off, and a two-image prompt.
• Both checkpoints are registered in the vision test list and answer "the new york times" to the harness default prompt, so they pass the existing assertion.
• resize_position_embeddings still takes its early return for this model on both variants, since every crop is 512x512 and the patch grid is always 32x32. Confirmed with GGML_SCHED_DEBUG=2, zero UPSCALE nodes across every image size.

Last full matrix, run 31457923076, all jobs green, taken 2026-08-11

Run ggml-org#335, cognitive preset, addon@candidate built from this fabric branch through the bench
overlay, 130 rows, 0 errors. First run where every job passed: 10 prebuild platforms, the
desktop leg and both Device Farm legs.

leg	quality %	mmproj enc ms	tiles	TTFT ms	decode tps
linux GPU Vulkan	71.4	50.2	3 and 5	94.6	354.0
s25 GPU Adreno OpenCL	71.4	1174.2	3 and 5	1642.0	77.6
s25 CPU	67.4	2163.9	3 and 5	2617.9	132.7
pixel9 CPU	67.4	6519.3	3 and 5	8190.5	19.2
pixel9 GPU Vulkan	71.4	6777.5	3 and 5	9938.0	37.1

Run 31459302593 then forced the projector onto the Mali GPU, which is the one path the routine
legs cannot reach, and it executes:

multimodal projector backend: GPU (mmproj-use-gpu override)     x4, pixel9

Pixel 9, 0 errors, encode 10322 ms forced onto the GPU against 7005 ms on the CPU in the same
run. So Vulkan works for the vision encoder on Mali and is slower there, which is why the addon
defaults it to CPU on that GPU class.

The older device runs in this section predate several changes that alter what the encoder sees: the 1x1 grid now sends the bicubic slice instead of the bilinear overview, a multi-image prompt now carries <image: N> labels without a trailing NUL token, the aspect-preserving refine is computed in double, and the refined image is no longer letterboxed. Every quality number in this section, run ggml-org#335 included, was taken before 2026-08-13 and is therefore stale, and the overview revert in a812964c9 moves the encoder's input again. They are kept for the Adreno history and for the backend coverage they demonstrate, not as current quality figures. A re-run at this head is outstanding. The slice-count table was recomputed here and is unchanged, since none of these changes alter the grid. The earlier run 31392540850 was cancelled: it failed at prebuild on all nine platforms because the overlay still pinned a master-based fabric, which is what surfaced the wrong-base problem above.

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1217241821002482

[github-actions]

Review Status

Current Status: ❌ PENDING
Approvals so far: none

Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member.

[yingying0906]

Consumer side is tetherto/qvac#3725. Device evidence for both, including the Adreno fix, is on the tetherto/qvac branch bench/QVAC-23075-visionpsy-vlm, which carries the vcpkg overlay. It was PR tetherto/qvac#3726, now closed, and those run records are still readable there.

[tobi-legan]

QA follow-up: VisionPsy coverage gaps

The core code findings I had are already covered inline, so not duplicating those here. From a QA perspective, I would still like the following coverage before this is treated as merge-ready:

• Add deterministic VisionPsy preprocessing golden tests against the HF processor for base and Flash sizes, including 960x720, 256x256, and 1024x768. Assert refined size, grid, slice count, and overview pixel source/order, not just final answer text.
• Add an Adreno Q8_0 view readback regression test with a transposed parent and smaller view, checking no overflow and byte-for-byte correct restore.
• Add tokenizer/chunk sequence tests for VisionPsy: 1x1 emits <|global_image|> plus the refined slice only; multi-tile emits overview first, then row/col slice delimiters; multi-image prompts emit <image: N> ordinals.
• Add loader/CLI tests for image_no_upscale: default -1 preserves GGUF, on/off override, bad values reject, non-idefics3 models ignore with warning, and invalid no-upscale metadata rejects both image_size == 0 and longest_edge < image_size.
• Add projector alias tests: clip.projector_type = "custom" plus general.name == "VisionPsyNano" resolves to VisionPsy, while custom with any other name remains unknown.

[tobi-legan]

Requesting changes based on the native correctness issues and QA gaps already posted.

Blocking items: the Adreno Q8_0 view readback can overflow/corrupt because the predicate uses the parent layout while restore allocation/dimensions still use the view; VisionPsy preprocessing still has reference-compatibility risks around resize precision and overview generation.

Please also cover the loader validation, ABI/rebuild story, preprocessing golden tests, Adreno view regression, and VisionPsy token/chunk sequence tests before this is merge-ready.

[iancris]

overview_from_refined is set unconditionally in the shared idefics3 preprocessor, but only VisionPsy opts out of PAD_CEIL, so every idefics3 overview now has the letterbox bars baked into it.

bdd9b1e97 is right that the reference refines first and takes the global image from that result. But this flag is set for the whole idefics3 family, while the opt-out from aspect-preserving padding is set for one member of it: clip.cpp:1541 gives VisionPsy image_pad_rf = PAD_NONE, and the PROJECTOR_TYPE_IDEFICS3 case at clip.cpp:1525-1530 sets none, keeping the PAD_CEIL default. slice_image then builds refined_img letterboxed (mtmd-image.cpp:758-759) and resizes that into the overview (:762-763).

HF stretches at both steps, so the idefics3 overview used to match the reference and now does not. At image_size=384, preproc_image_size=1920, a 1000x300 input leaves 96px bars top and bottom — 25% of the overview is black.

Reachable with no flag: it needs only clip.vision.preproc_image_size, which conversion/smolvlm.py:30-31 writes unconditionally for every model converted here. CI misses it because the three mmprojs tests.sh downloads predate that key and take the overview-only path.

Distinct from the ordering thread on mtmd-image.cpp:1295, which asked for this change and got it — the gap is that it was gated on projector family rather than on whether the refined image is padded.

Suggested fix: tie the flag to the padding mode, so it disables itself wherever the refined image is letterboxed and no projector has to remember to opt out.

instructions.overview_from_refined = (hparams.image_pad_rf == PAD_NONE);

[yingying0906]

Taking it. Reverted the overview source in a812964, so the overview comes from the original again for the whole family. Gating on the padding mode fixes the bars but leaves the second resample on VisionPsy, which is your other thread, so the single resize covers both. slice_image and slice_instructions are byte-identical to v10069.0.0 again.

[iancris]

Under PAD_NONE the new overview path is geometrically identical to the old one, so the only thing it changes is resampling quality — and it makes it worse.

"Stretch original to refined, then stretch refined to (p, p)" and "stretch original to (p, p)" are the same geometry once padding is out of the picture. For VisionPsy specifically, deriving the overview from refined_img therefore buys no correctness and costs resolution: resize_bilinear (mtmd-image.cpp:233-258) is a 2x2 tap with no antialiasing, while the reference torchvision.transforms.functional.resize defaults to antialias=True.

On the base checkpoint with the committed test-1.jpeg, a mild 640x488 -> 512x512 resize becomes a 3x bicubic upscale to 2048x2048 followed by a 4x downscale back to 512x512. A 2x2 tap at that reduction averages 4 of every 16 source pixels and discards the other 12, where an antialiased kernel weighs all of them. The affected embedding is the overview — the first thing the model sees, behind <|global_image|> — on any image below the 2048 cap.

Worth noting this is not fixed by gating overview_from_refined on the padding mode: VisionPsy is PAD_NONE, so it keeps the refined source either way.

Suggested fix: the cheapest option also resolves the letterboxing comment on line 1303 — keep sourcing the overview from the original wherever the refined image is an unpadded stretch, since the two are equivalent there. If the refined source is worth keeping for its own sake, img_tool::resize needs an antialiased or box-filter path for downscales beyond about 2x.

[yingying0906]

Confirmed. image_resize_algo_ov is RESIZE_ALGO_BILINEAR and image_pad_ov is PAD_NONE, so it routes to resize_bilinear, a 2x2 tap with no prefilter, and test-1.jpeg refines 640x488 to 2048x2048, so the overview was a 3x upscale then a 4x downscale. Reverted in a812964.