You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Qwen3.5-MoE on CUDA: load ExLlamaV3 (EXL3) checkpoints #1149
Native CUDA Qwen3.5-MoE has no loader for ExLlamaV3 (EXL3) checkpoints. I would like to add one, as two small pull requests, and I am opening this first because CONTRIBUTING.md asks for an issue before a new checkpoint format.
What: a Hugging Face-layout Qwen3.5-MoE checkpoint whose config.json says quantization_config.quant_method: "exl3" (turboderp's ExLlamaV3 trellis format, mul1 codebook, per-tensor widths) would load and serve on the existing cuda.exl3 Dense and grouped-expert kernels, with bf16 embeddings. The test checkpoint is https://huggingface.co/Mia-AiLab/Ornith-1.5-122B-EXL3-EAQv1-4-bpw (Ornith-1.5-122B, about 4 bits per weight, 60 GiB on the device). A second, independent change lets the Qwen vision tower read Hugging Face tensor names (model.visual.*).
Why: that checkpoint serves from one GB10 (121 GiB unified memory), where the MLX 4-bit form of the same model is larger.
Scope and limits: serving only, with the generic EXL3 kernels. There are no tuned prompt or decode kernels and no drafter (--drafter is refused for an EXL3 checkpoint), so speed is modest. The MTP layer and drafting are left out on purpose.
Related open work, not duplicated here: #1137 (dense BF16 Qwen3.5 scoring, touches the same geometry, engine and a cuda_bf16 file, so my files are named *_bf16_rows), #1145 (ModelOpt NVFP4 for Qwen3.5), #1113 (EXL3 for Flash Next), #1125 (Ornith on Metal), #1047 (padding rows in the routed experts).
Done by #1150 (1ef9c5c..b34f9e7): ExLlamaV3 (EXL3) Qwen3.5-MoE checkpoints load and serve on native CUDA. Ornith-1.5-122B EXL3 4 bpw is checked on a GB10, including prompts past 2,048 tokens. Drafters are refused for EXL3 for now (Exl3DrafterUnsupported).
Activity
ashhart commentedon Oct 25, 2026
Done by #1150 (1ef9c5c..b34f9e7): ExLlamaV3 (EXL3) Qwen3.5-MoE checkpoints load and serve on native CUDA. Ornith-1.5-122B EXL3 4 bpw is checked on a GB10, including prompts past 2,048 tokens. Drafters are refused for EXL3 for now (
Exl3DrafterUnsupported).- Synthema, Ash's AI Assistant.