Skip to content

Qwen3.5-MoE on CUDA: load ExLlamaV3 (EXL3) checkpoints #1149

Closed

Description

@MiaAI-Lab

Native CUDA Qwen3.5-MoE has no loader for ExLlamaV3 (EXL3) checkpoints. I would like to add one, as two small pull requests, and I am opening this first because CONTRIBUTING.md asks for an issue before a new checkpoint format.

What: a Hugging Face-layout Qwen3.5-MoE checkpoint whose config.json says quantization_config.quant_method: "exl3" (turboderp's ExLlamaV3 trellis format, mul1 codebook, per-tensor widths) would load and serve on the existing cuda.exl3 Dense and grouped-expert kernels, with bf16 embeddings. The test checkpoint is https://huggingface.co/Mia-AiLab/Ornith-1.5-122B-EXL3-EAQv1-4-bpw (Ornith-1.5-122B, about 4 bits per weight, 60 GiB on the device). A second, independent change lets the Qwen vision tower read Hugging Face tensor names (model.visual.*).

Why: that checkpoint serves from one GB10 (121 GiB unified memory), where the MLX 4-bit form of the same model is larger.

Scope and limits: serving only, with the generic EXL3 kernels. There are no tuned prompt or decode kernels and no drafter (--drafter is refused for an EXL3 checkpoint), so speed is modest. The MTP layer and drafting are left out on purpose.

Related open work, not duplicated here: #1137 (dense BF16 Qwen3.5 scoring, touches the same geometry, engine and a cuda_bf16 file, so my files are named *_bf16_rows), #1145 (ModelOpt NVFP4 for Qwen3.5), #1113 (EXL3 for Flash Next), #1125 (Ornith on Metal), #1047 (padding rows in the routed experts).

Credit: the format and kernel design are turboderp's ExLlamaV3 (https://github.com/turboderp-org/exllamav3).

Activity

  1. ashhart commented on Oct 25, 2026

    @ashhart
    Owner

    Done by #1150 (1ef9c5c..b34f9e7): ExLlamaV3 (EXL3) Qwen3.5-MoE checkpoints load and serve on native CUDA. Ornith-1.5-122B EXL3 4 bpw is checked on a GB10, including prompts past 2,048 tokens. Drafters are refused for EXL3 for now (Exl3DrafterUnsupported).

    - Synthema, Ash's AI Assistant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions