QVAC-23075 test[skiplog]: DO NOT MERGE, overlay-pinned CI proof for VisionPsy Nano - #3726
QVAC-23075 test[skiplog]: DO NOT MERGE, overlay-pinned CI proof for VisionPsy Nano#3726yingying0906 wants to merge 3 commits into
Conversation
License compliance — cleanNo new dependency license findings in this PR. Warn-only (shadow) mode — this check does not block merges yet. Updated automatically by the canonical license compliance workflow. NOTICE presence (advisory)Missing NOTICE (advisory, does not block):
|
Workflow security (shadow mode)zizmor found 1117 finding(s) in Findings are annotated inline on the changed files and listed in the job summary. Reproduce locally: pipx run zizmor==1.27.0 --offline .github/ |
|
CI evidence for #3725. Not for merging: the overlay pins a fabric branch SHA. Closes unmerged once tetherto/qvac-fabric-llm.cpp#205 is tagged and published to the vcpkg registry. |
… benchmark The native CLI can select VisionPsy Flash's preprocessing rule with a flag, but the addon could not, so every addon leg, which is what the phones run, measured Flash weights under base preprocessing. Nothing in the published Flash mmproj distinguishes it from the base one, so there is no way to infer it. Add image-no-upscale / image_no_upscale to the addon load-config handlers, forward it into mtmd_context_params, and declare it in the SDK config schema. The handler only ever writes 0 or 1. Leaving the key out keeps fabric's -1 model-default sentinel, so no existing caller changes behaviour; there is a unit test for exactly that. Needs the fabric side released first, since the key does not compile without mtmd_context_params.image_no_upscale. Pin the VisionPsy blobs in models.manifest.json and add catalog entries for q4_0, q8_0 and iq3_m across both checkpoints. Select the preprocessing rule on both engines, cliArgs for the native CLI and a new per-model addonConfig field for the addon. Three benchmark bugs are fixed along the way, all independent of VisionPsy: several-sources ignored matrix_models and always ran config.sourcesModel, so a dispatch asking for another model measured qwen and stayed green. A CLI-source dispatch also forced an addon leg, which then failed on a projector type that only exists on a fabric branch, before the CLI step ran. Encode-ms parsing only matched "image slice encoded in N ms" from mtmd_helper_eval_chunks, which llama-mtmd-cli never calls; it logs "mtmd batch encoding done in N ms" instead, so ttft_ms was empty on every CLI leg. The report's own mmproj-enc and tiles columns hold a second copy of that regex in aggregate.js, so they stayed blank after the parser fix. Fix both, and surface rss_mb, which was read from /usr/bin/time -v and then thrown away. Measured over run #325, cognitive preset, 175 samples, 0 errors: quality is flat at 70.9 to 71.4 across Vulkan on RTX 4000 Ada, Metal and CPU on iPhone 17, Mali Vulkan and CPU on Pixel 9, and CPU on S25. The addon key is proven live on device, not only in unit tests: image-no-upscale: on appears in the addon config in logcat, and vision_enc_tiles is 3 for a 2x1 slice grid and 5 for a 2x2, where base preprocessing gives 13 to 17.
#3195 made models.manifest.json the only source of model URLs, keyed by modelName, but the vlm-benchmark catalog kept its own reg-* names, and those are not manifest keys, so resolveModelEntry throws before an addon leg reaches the disk. CLI legs never hit it because the workflow curls those blobs itself, which is why it stayed green. Three of the five blobs are byte-identical to manifest entries that already exist under other keys, so repointing modelName migrates them with no new pins. The two mmprojs need real entries, both warm: false so they stay out of every cache-models warm step. Only the qwen and gemma entries are affected. The VisionPsy entries were pinned correctly when they were added.
…isionPsy mobile Points the addon's vcpkg dependency at the VisionPsy fabric branch HEAD so mobile legs, which build the addon rather than a native CLI, compile against PROJECTOR_TYPE_VISIONPSY, mtmd_context_params.image_no_upscale and the Adreno q8_0 transpose guard. The published qvac-fabric version has none of them. Never merge this. The REF is a branch SHA, and it dies with the registry publish.
6f10d2e to
ebde13e
Compare
VLM Matrix BenchmarkRun #331 — full report VLM Matrix — several-sources / cognitive (run #331)Mode: several sources (engine varies; model fixed) · Engine: addon Preset: one fixed model across inference engines · quality = lmms-eval (VQA / ANLS / relaxed / MC), equal-weight mean across tasks. 1 · HighlightsInference engines on the same model: addon@candidate. Quality — overall % per source
Speed — mmproj-encode ms per source (lower = faster)
2 · DetailsSources — resolved versions
Models & origins (Source = Registry / HF / S3 / URL · pinned commits)
Provenance — hardware & softwarelinux · gpu (runner
pixel9 — Google Pixel 9 (AWS Device Farm)
s25 — Samsung Galaxy S25 Ultra (AWS Device Farm)
Quality (%)
Quality by task (% — higher better, mean across platforms; one column per source)
Speed
Peak memory (RSS)
3 · Test Results (per platform)
4 · Image samples
|
|
Closing. This never merges by design, and as a PR it reads as a 21-file duplicate of #3725: it is that branch plus one overlay commit, so GitHub shows the full distance from main. The branch bench/QVAC-23075-visionpsy-vlm stays pushed and keeps working. CI and benchmark dispatches run off a branch ref and need no PR: gh workflow run on-pr-llm-llamacpp.yml -R tetherto/qvac --ref bench/QVAC-23075-visionpsy-vlm Evidence links moved to #3725. |
1 participant
🎯 What problem does this PR solve?
qvac-fabric, because the key it forwards,mtmd_context_params.image_no_upscale, only exists on a fabric branch. So the real PR's own CI is red for a reason that has nothing to do with whether the code is correct.📝 How does it solve it?
qvac-fabricto the VisionPsy fabric branch HEAD.android-arm64, so the mobile legs run branch-built native code on real phones.The overlapping diff is deliberate.
on-pr-*.ymltriggers only onbranches: [main], so a PR based on the companion branch would run no CI at all and prove nothing.🧪 How was it tested?
Run 31189679101,
visionpsy-q4at smoke on an S25 Ultra and a Linux GPU runner:The Adreno leg is the interesting one. It used to abort during model load and was absent from the report entirely; it now passes with the same 13 tiles its CPU leg reports, so the OpenCL path is doing the same work rather than skipping it.
Broader matrix, run #325, cognitive preset, 175 samples, 0 errors: quality flat at 70.9 to 71.4 across Vulkan on RTX 4000 Ada, Metal and CPU on iPhone 17, Mali Vulkan and CPU on Pixel 9, and CPU on S25.
This PR must never be merged. The overlay pins a branch SHA, and a branch SHA must not reach
main. It exists only as CI evidence and will be closed unmerged once the fabric tag is published to the vcpkg registry, which makes the overlay unnecessary.