Skip to content

QVAC-23075 feat[api]: expose image_no_upscale in the SDK config schema - #3854

Merged
gianni-cor merged 11 commits into
mainfrom
feat/QVAC-23075-visionpsy-sdk
Aug 19, 2026
Merged

QVAC-23075 feat[api]: expose image_no_upscale in the SDK config schema#3854
gianni-cor merged 11 commits into
mainfrom
feat/QVAC-23075-visionpsy-sdk

Conversation

@yingying0906

@yingying0906 yingying0906 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

🎯 What problem does this PR solve?

  • The VisionPsy Flash checkpoint and the base one are indistinguishable through the SDK. Their mmprojs declare identical vision hparams, so a Flash checkpoint loaded without this key silently runs base preprocessing. That changes the image token count, so it moves both accuracy and encode time.

📝 How does it solve it?

  • Bumped LLM addon to 0.44.0
  • image_no_upscale on the llamacpp completion config, an optional "on" | "off" that overrides the idefics3-style preprocessing rule the GGUF declares.
  • "on" rounds the image's long side up to a whole number of slices and caps it, so an image below the cap keeps its own resolution and becomes far fewer slices.
  • "off" always stretches the long side to the cap.
  • Unset keeps the model's own value, so existing callers are unaffected. There is deliberately no default, so the addon keeps the GGUF value rather than being forced onto base preprocessing.
  • The two sdk-python/_generated files are in the same commit on purpose. pr-checks-sdk-python.yml runs generate.py --check and fails with "Generated Python client is stale" if contract/schema.json moves without them, which then cascades into merge-guard / validate-pr.
  • Replaces SmolVLM2 with VisionPsy Flash Q4 in the existing desktop, mobile, and Electron vision E2E tests. Bonus: This reduces the total download size by about 130 MB.

🧪 How was it tested?

CI run: https://github.com/tetherto/qvac/actions/runs/32269149688?pr=3854

  • SDK and inference unit tests validate accepted, rejected, and unset image_no_upscale values.
  • A new E2E test processes the same 64×64 image with "on" and "off" and verifies that "on" uses less than half as many prompt tokens.

🔌 API Changes

await sdk.loadModel({
  modelType: ModelType.llamacppCompletion,
  modelSrc: VISIONPSY_NANO_460M_MULTIMODAL_Q4_K_M,
  modelConfig: {
    projectionModelSrc: MMPROJ_VISIONPSY_NANO_460M_MULTIMODAL_Q8_0,
    image_no_upscale: 'on'
  }
})

@yingying0906
yingying0906 requested review from a team as code owners August 14, 2026 04:19
@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Review Status

Current Status: ✅ APPROVED
Approvals so far: Team Lead: 1, Member: 1

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

License compliance — findings detected (warn-only)

Critical: 0 · High: 7 · Medium: 0

Dependency License Scope Severity Outcome
@qvac/classification-ggml@^0.19.1 (none detected) runtime High blocks
@qvac/embed-llamacpp@^0.33.0 (none detected) runtime High blocks
@qvac/llm-llamacpp@^0.44.0 (none detected) runtime High blocks
@qvac/ocr-ggml@^0.17.0 (none detected) runtime High blocks
@qvac/translation-nmtcpp@^0.9.0 (none detected) runtime High blocks
@qvac/tts-ggml@^0.7.4 (none detected) runtime High blocks
@qvac/vla-ggml@^0.20.0 (none detected) runtime High blocks

How to resolve a blocking finding:

  • Remove or replace the disallowed dependency, or
  • If the license is genuinely acceptable, run the compliance SKILL and record the decision in .github/license-allowlist.yml (CODEOWNERS-reviewed), or
  • For a one-off, a maintainer can apply the license-override label (High findings only; Critical cannot be overridden).

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

  • ./.github/actions/release-merge-guard
  • ./docs/website
  • ./packages/ggml-coload-smoke
  • ./packages/fabric/test/integration
  • ./packages/inference-addon-cpp/mobile
  • ./packages/sdk/e2e
  • ./packages/llm-llamacpp/benchmarks/performance
  • ./packages/llm-llamacpp/benchmarks/server
  • ./packages/vla-ggml/sim/server
  • ./packages/embed-llamacpp/benchmarks/performance
  • ./packages/embed-llamacpp/benchmarks/server
  • ./packages/asr-ggml/benchmarks/server

yingying0906 added a commit that referenced this pull request Aug 14, 2026
Parses the idefics3-style preprocessing override out of the load config and forwards it to
the vision context, so a caller can say "on" or "off" instead of being stuck with whatever
the GGUF declares. Unset leaves the model's own value alone.

This is what separates the VisionPsy Flash checkpoint from the base one, whose mmprojs are
otherwise indistinguishable, so a Flash checkpoint loaded without it silently runs base
preprocessing. It changes the image token count, so it moves both accuracy and encode time.

LoadConfigHandlers parses the string into common_params, and MtmdLlmContext::initVisionContext
copies it into mtmd_context_params next to image_tile_mode. Unit coverage for the parse sits
with the other load-config cases.

Needs the fabric side, tetherto/qvac-fabric-llm.cpp#205, which adds image_no_upscale to
common_params and mtmd_context_params. cpp-lint stays red here until that merges and the
registry publishes the next fabric version.

Split out at Gianfranco's request. The SDK schema is #3854 and the VLM benchmark is #3855.
yingying0906 added a commit that referenced this pull request Aug 14, 2026
Parses the idefics3-style preprocessing override out of the load config and forwards it to
the vision context, so a caller can say "on" or "off" instead of being stuck with whatever
the GGUF declares. Unset leaves the model's own value alone.

This is what separates the VisionPsy Flash checkpoint from the base one, whose mmprojs are
otherwise indistinguishable, so a Flash checkpoint loaded without it silently runs base
preprocessing. It changes the image token count, so it moves both accuracy and encode time.

LoadConfigHandlers parses the string into common_params, and MtmdLlmContext::initVisionContext
copies it into mtmd_context_params next to image_tile_mode. Unit coverage for the parse sits
with the other load-config cases.

Needs the fabric side, tetherto/qvac-fabric-llm.cpp#205, which adds image_no_upscale to
common_params and mtmd_context_params. cpp-lint stays red here until that merges and the
registry publishes the next fabric version.

Split by area. The SDK schema is #3854 and the VLM benchmark is #3855.
@yingying0906
yingying0906 force-pushed the feat/QVAC-23075-visionpsy-sdk branch from 70b4bd1 to ab3bbf3 Compare August 14, 2026 04:25
yingying0906 added a commit that referenced this pull request Aug 14, 2026
Adds the base and Flash VisionPsy checkpoints to the model catalog, with their manifest
entries, and the plumbing the comparison needed to be fair:

- `resolve-cli-model.cjs` resolves a spec to the blob the CLI legs load, so an addon leg and
  a CLI leg run the same bytes at the same ctx_size.
- `cli-args.cjs` carries a catalog entry's `cliArgs` to the CLI legs, since model-specific
  flags such as VisionPsy Flash's `--image-no-upscale` are fabric-fork additions that
  upstream-cli aborts on.
- `stdout-parser.js` and `aggregate.js` read vision-encode timing and score the new rows.

`package.json` registers the new `__tests__` in `test:prestage` so they run in CI. The
manifest gains the VisionPsy blobs the catalog points at; without them the catalog resolves
to keys that do not exist.

Split by area. The addon side stays in #3725 and the SDK schema is #3854.
@yingying0906
yingying0906 force-pushed the feat/QVAC-23075-visionpsy-sdk branch from ab3bbf3 to 66ba8b1 Compare August 14, 2026 04:42
@yingying0906 yingying0906 reopened this Aug 14, 2026
Adds the idefics3-style preprocessing switch to the llamacpp completion config so a caller
can override what the GGUF says. "on" rounds the image's long side up to a whole number of
slices and caps it, so an image below the cap keeps its own resolution and becomes far fewer
slices; "off" always stretches the long side to the cap. Unset keeps the model's own value.

This is what separates the VisionPsy Flash checkpoint from the base one, whose mmprojs are
otherwise indistinguishable, so a Flash checkpoint loaded without it silently runs base
preprocessing. It changes the image token count, so it moves both accuracy and encode time.

Additive and optional, so [api] rather than [bc]. The generated Python client is regenerated
in the same commit because pr-checks-sdk-python.yml runs generate.py --check and fails the
merge guard on a stale client.
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@maxim-smotrov maxim-smotrov added test-e2e-smoke Triggers smoke e2e test suite [Currently SDK-only] and removed test-e2e-smoke Triggers smoke e2e test suite [Currently SDK-only] labels Aug 19, 2026
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

QVAC E2E — ios — ❌ failed

Totals: 90/106 passed · 2 failed · 97.8% · 859s
Config: suite=smoke · filter=(none) · exclude=(none)
Device pool: iPhone 16 Pro - iOS 18 — ☀️ highly available
View run · Artifacts: reports · Device Farm logs

Results by section

  • vla: 0/4 ❌

Failed tests

  • vla-hparams-shape: Setup failed: Failed to load model: Failed to load weights, error: failed to load SmolVLA model from: /var/mobile/Containers/Data/Application/C3AD2F51-18C7-469C-9041-78990EB252CE/Documents/.qvac/mode…
  • vla-run-synthetic-shape: Setup failed: Failed to load model: Failed to load weights, error: failed to load SmolVLA model from: /var/mobile/Containers/Data/Application/C3AD2F51-18C7-469C-9041-78990EB252CE/Documents/.qvac/mode…

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

QVAC E2E — android — ✅ all tests passed (92/106, 1987s)

Config: suite=smoke · filter=(none) · exclude=(none)
Device pool: Google Pixel 10 Pro - Android 16 — ☀️ highly available
View run · Artifacts: reports · Device Farm logs · Device Farm logs

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

QVAC E2E — windows — ✅ all tests passed (106/106, 450s)

Config: suite=smoke · filter=(none) · exclude=(none)
View run · Artifacts: reports

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

QVAC E2E — linux — ✅ all tests passed (106/106, 323s)

Config: suite=smoke · filter=(none) · exclude=(none)
View run · Artifacts: reports

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

QVAC E2E — macos — ✅ all tests passed (106/106, 226s)

Config: suite=smoke · filter=(none) · exclude=(none)
View run · Artifacts: reports

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test-e2e-smoke Triggers smoke e2e test suite [Currently SDK-only]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants