fix(sdk): resolve QAIRT top-level precision as quant alias - #1247
fix(sdk): resolve QAIRT top-level precision as quant alias#1247MAN$I VERMA (mansiverma897993) wants to merge 1 commit into
Conversation
QAIRT manifests key model_file under "N/A" and record the real precision (e.g. W4A16) on the manifest's top-level field. Listings (geniex list, serve /v1/models) surface that precision as the model id suffix, but resolve_model_paths only matched model_file keys — so the advertised "name:W4A16" failed with QuantNotFound when echoed back to /v1/chat/completions, breaking OpenAI clients that auto-populate the model id from /v1/models. Accept the top-level precision (case-insensitively) as an alias for the "N/A" entry; exact model_file keys still win. Fixes qualcomm#1242 Signed-off-by: mansiverma897993 <vmansi756@gmail.com>
|
MAN$I VERMA (@mansiverma897993) Thank you for the quick fix. Could you provide a pre-built Windows ARM64 test binary that includes both this fix (#1247) and #1246? I am happy to test on Snapdragon hardware. |
|
Novatoris Unfortunately I can't provide a pre-built binary I don't have a Windows ARM64 less resources I have to build environment (the full build needs the Hexagon SDK, OpenCL SDK and WDK, and #1247 requires rebuilding the SDK bridge too). I've pushed a combined branch with both fixes (#1246 + #1247) if you're able to build locally: Alan Zhu (@alanzhuly) could CI produce a Windows ARM64 test artifact from a PR branch, or could these fixes go into the next pre-release build? That would let Novatoris verify both fixes end-to-end on Snapdragon hardware. |
|
MAN$I VERMA (@mansiverma897993) Hi Mansi, thanks for your efforts. Apologies for the delayed reply. |
|
Superseded by #1261, which addresses this at the manifest layer (keys QAIRT ModelFile by the precision label directly) rather than adding an alias to the resolver — closing this in favor of that approach. |
Np this is up to you ....and apart from these I have raised 2 more PR when you have chance take a look on it too . |
Fixes #1242
Root cause
QAIRT (AI Hub) manifests store their single
model_fileentry under the placeholder key"N/A"and record the real precision (e.g.W4A16) on the manifest's top-levelPrecisionfield. The two consumers of that manifest disagree:geniex_model_list_detailed, used bygeniex listand serve/v1/models) deliberately surfaces the top-level precision instead of theN/Aplaceholder — so/v1/modelsadvertisesqualcomm/Qwen3-4B-Instruct-2507:W4A16.resolve_model_paths, reached from/v1/chat/completionsviaModelGetPaths) only matchedmodel_filekeys —W4A16isn't one, so the advertised id failed withquantization 'W4A16' not found.Any OpenAI-compatible client that auto-populates the model id from
/v1/models(AnythingLLM in the report) therefore couldn't chat at all.Fix
The issue offered two options — strip the suffix from the listing, or make completions accept it. Stripping would regress the human-meaningful precision display that the listing intentionally surfaces, so this PR takes the second option:
resolve_model_pathsnow accepts the manifest's top-level precision (case-insensitively, mirroring the FFI's quant-tag normalization) as an alias for the"N/A"entry. Advertised ids round-trip; exactmodel_filekeys still win over the alias; unknown quants still error;QuantNotDownloadedsemantics are preserved.This also fixes the same round-trip for
geniex run name:W4A16on the CLI, and makes/v1/models/{id}(which already accepted the suffix) consistent with completions.Testing
crates/core/src/paths.rs: alias hit, case-insensitive alias, alias-but-not-downloaded, unknown quant still errors, exact key wins over alias.cargo test -p model-manager-core -p model-manager-ffi(171 core tests passed),cargo fmt --all --checkclean.I don't have Snapdragon hardware to run
geniex serveend-to-end — Novatoris could you confirm with AnythingLLM once a patched build is available? (Together with #1246 this should make the AnythingLLM flow fully work.)