MoE MTP support for Qwen3.5/6

[keepupatme]

One ask is for MoE MTP support for Qwen3.5/6, MTP export on GPTQ A3B MoE fails hard: MTP draft is only supported for qwen3_5_text checkpoints; got 'qwen3_5_moe_text'

[nvluxiaoz]

We are working on this feature. This feature would require FP16 MoE kernels because the MTP draft is mostly unquantized. We will support this very quickly.

[lix19937]

I have do some exps:
The MTP draft expert weights in the hf-checkpoint are stored in FP16/BF16 (non‑NVFP4‑packed) (e.g., mtp.layers.0.mlp.experts.gate_up_proj [256,1024,2048] BF16, shared_expert.gate_proj [512,2048] BF16, etc.). However, Qwen3_5SparseMoeBlock only supports two paths: NVFP4‑plugin and INT4‑GPTQ. There is no FP16 dense MoE forward pass implementation.

To make this work, we would need to implement from scratch a torch.export‑traceable FP16 dense MoE forward pass, covering router → softmax → top‑k(8/256) → per‑expert SwiGLU → weighted sum plus shared‑expert computation. Moreover, this draft is a 256‑expert FP16 MoE, which is itself non‑trivial (the premise of a "light‑weight draft" is questionable), and exhibits high VRAM consumption without quantization.

So there are two ways:

• impl fp16 moe plugin, maybe performance not good;
• impl moe nvfp4 plugin, you need do ptq(w4a16) on MTP draft model. I choose this way.