Skip to content

fix(sdk): resolve QAIRT top-level precision as quant alias - #1247

Closed
MAN$I VERMA (mansiverma897993) wants to merge 1 commit into
qualcomm:mainfrom
mansiverma897993:fix/qairt-precision-roundtrip
Closed

fix(sdk): resolve QAIRT top-level precision as quant alias#1247
MAN$I VERMA (mansiverma897993) wants to merge 1 commit into
qualcomm:mainfrom
mansiverma897993:fix/qairt-precision-roundtrip

Conversation

@mansiverma897993

Copy link
Copy Markdown
Contributor

Fixes #1242

Root cause

QAIRT (AI Hub) manifests store their single model_file entry under the placeholder key "N/A" and record the real precision (e.g. W4A16) on the manifest's top-level Precision field. The two consumers of that manifest disagree:

  • Listing (geniex_model_list_detailed, used by geniex list and serve /v1/models) deliberately surfaces the top-level precision instead of the N/A placeholder — so /v1/models advertises qualcomm/Qwen3-4B-Instruct-2507:W4A16.
  • Resolution (resolve_model_paths, reached from /v1/chat/completions via ModelGetPaths) only matched model_file keys — W4A16 isn't one, so the advertised id failed with quantization 'W4A16' not found.

Any OpenAI-compatible client that auto-populates the model id from /v1/models (AnythingLLM in the report) therefore couldn't chat at all.

Fix

The issue offered two options — strip the suffix from the listing, or make completions accept it. Stripping would regress the human-meaningful precision display that the listing intentionally surfaces, so this PR takes the second option: resolve_model_paths now accepts the manifest's top-level precision (case-insensitively, mirroring the FFI's quant-tag normalization) as an alias for the "N/A" entry. Advertised ids round-trip; exact model_file keys still win over the alias; unknown quants still error; QuantNotDownloaded semantics are preserved.

This also fixes the same round-trip for geniex run name:W4A16 on the CLI, and makes /v1/models/{id} (which already accepted the suffix) consistent with completions.

Testing

  • 5 new unit tests in crates/core/src/paths.rs: alias hit, case-insensitive alias, alias-but-not-downloaded, unknown quant still errors, exact key wins over alias.
  • Full suites green locally: cargo test -p model-manager-core -p model-manager-ffi (171 core tests passed), cargo fmt --all --check clean.
  • No public header / FFI change — the fix is entirely inside the core resolver.

I don't have Snapdragon hardware to run geniex serve end-to-end — Novatoris could you confirm with AnythingLLM once a patched build is available? (Together with #1246 this should make the AnythingLLM flow fully work.)

QAIRT manifests key model_file under "N/A" and record the real
precision (e.g. W4A16) on the manifest's top-level field. Listings
(geniex list, serve /v1/models) surface that precision as the model id
suffix, but resolve_model_paths only matched model_file keys — so the
advertised "name:W4A16" failed with QuantNotFound when echoed back to
/v1/chat/completions, breaking OpenAI clients that auto-populate the
model id from /v1/models. Accept the top-level precision
(case-insensitively) as an alias for the "N/A" entry; exact model_file
keys still win.

Fixes qualcomm#1242

Signed-off-by: mansiverma897993 <vmansi756@gmail.com>
@Novatoris

Copy link
Copy Markdown

MAN$I VERMA (@mansiverma897993) Thank you for the quick fix. Could you provide a pre-built Windows ARM64 test binary that includes both this fix (#1247) and #1246? I am happy to test on Snapdragon hardware.

@mansiverma897993

MAN$I VERMA (mansiverma897993) commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Novatoris Unfortunately I can't provide a pre-built binary I don't have a Windows ARM64 less resources I have to build environment (the full build needs the Hexagon SDK, OpenCL SDK and WDK, and #1247 requires rebuilding the SDK bridge too).

I've pushed a combined branch with both fixes (#1246 + #1247) if you're able to build locally:
https://github.com/mansiverma897993/GenieX/tree/test/serve-openai-compat (build steps in notes/build.md)

Alan Zhu (@alanzhuly) could CI produce a Windows ARM64 test artifact from a PR branch, or could these fixes go into the next pre-release build? That would let Novatoris verify both fixes end-to-end on Snapdragon hardware.

@mengshengwu

Copy link
Copy Markdown
Contributor

MAN$I VERMA (@mansiverma897993) Hi Mansi, thanks for your efforts. Apologies for the delayed reply.
The model manager has recently undergone refactoring, so this PR is now outdated. I will follow up on this task later.
To resolve the issue, the model manager needs to support the model:quant format, consistent with llama.cpp.

@mengshengwu

Copy link
Copy Markdown
Contributor

Superseded by #1261, which addresses this at the manifest layer (keys QAIRT ModelFile by the precision label directly) rather than adding an alias to the resolver — closing this in favor of that approach.

@mansiverma897993

MAN$I VERMA (mansiverma897993) commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

MAN$I VERMA (MAN$I VERMA (@mansiverma897993)) Hi Mansi, thanks for your efforts. Apologies for the delayed reply. The model manager has recently undergone refactoring, so this PR is now outdated. I will follow up on this task later. To resolve the issue, the model manager needs to support the model:quant format, consistent with llama.cpp.

Np this is up to you ....and apart from these I have raised 2 more PR when you have chance take a look on it too .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

serve — /v1/models returns model ID with precision suffix, breaking OpenAI-compatible clients

3 participants