GLM 4.5 4.6 MOE

[voipmonitor]

Hello,

@mobicham is gemlite supposed to work with MoE models GLM 4.6 ? Either sglang or vllm? Did anyone tried? Would love to test with 4x 6000 PRO

If yes, iw there any doc how to create quantised model and run it?

[mobicham]

Hey @voipmonitor , the current version only supports dense matmuls, not MoEs. I have some gemv MoE kernels already that will add in the next days when I have time, but as of today they are not integrated.

[voipmonitor]

Hey @voipmonitor , the current version only supports dense matmuls, not MoEs. I have some gemv MoE kernels already that will add in the next days when I have time, but as of today they are not integrated.

thanl you, once you will have something and need help with testing, I have 8x 6000 pro machine. Ultimately my goal is to discover if fp4 large models are usefull on this hardware or fp8 is minimum.

[mobicham]

The MoE kernels I have right now are for weight-only int8/int4, not fp4. You don't need fp4 for weight-only quantization, int4 is better quality and faster

[voipmonitor]

but isnt nvfp4/mxfp4 microscaling always better than only in4 weights? Benchmarks shows that nvfp4 is very close to fp8 under 1% precision difference?

[mobicham]

Not really. mxfp4 is pretty bad out-of-the-box, nvfp4 quality is better but still far from fp8/mxfp8. int4 with algorithms like HQQ is much better in quality than nvfp4. mxfp4/nvfp4 only make sense when both the activations and the weights are quantized + you have a compute-bound problem (large prefill / large batch-size decoding / attention). So for single user inference for example, INT4 is actually a better choice than mxfp4/nvfp4, unless you have very large context-size and you can't benefit well from prefix caching

[voipmonitor]

Not really. mxfp4 is pretty bad out-of-the-box, nvfp4 quality is better but still far from fp8/mxfp8. int4 with algorithms like HQQ is much better in quality than nvfp4. mxfp4/nvfp4 only make sense when both the activations and the weights are quantized + you have a compute-bound problem (large prefill / large batch-size decoding / attention). So for single user inference for example, INT4 is actually a better choice than mxfp4/nvfp4, unless you have very large context-size and you can't benefit well from prefix caching

thank you. with my current hw setup which is 4x rtx 6000 pro, I'm memory bound in decode. I'm seeking for the best combination for glm 4.6 model which is still accurate enough and can utilise the compute power of the rtx - which I'm not able with fp8 - though 48 tokens/sec with 50 000 prefill token is not bad. Thus I'm seeking for int4/nvfp4 possibilites and there is literally not much public information. The HQQ / int4 - do you know if someone already tried glm 4.6 with hqq/int4 on sm120?

[mobicham]

Have you tried running INT4 via vLLM, maybe with this ?
I am not sure it will work well with tensor parallel. We do have tensor parallel support for INT4/hqq/gemlite via torchao but only for dense layers not MoEs.