TL;DR
The int4 converter now supports per-tensor-type mixed precision, letting each weight category (shared expert, o_proj, kv_b_proj, attention projections, dense MLP) be quantized to a different bit width independently. The research-backed plan: put the 3 most error-sensitive tensors at int8, everything else at grouped int4 — for only +5.3 GB RAM vs pure int4, protecting the tensors that compound error on every token while keeping the 372 GB expert pool at 4-bit.
Branch: windows-dev (commit 8f2ae4d)
Why
Not all tensors suffer equally from int4 quantization. In a MoE model, three categories of weights compound their error on every single token:
- Shared expert (
shared_experts.*) — fires on every token regardless of routing. A quantization error here pollutes the output of every forward pass, with no averaging from other experts.
- o_proj (
self_attn.o_proj) — reconstructs the attention output. It's the biggest attention tensor and the last matmul before the residual stream. Errors here directly corrupt the residual.
- kv_b_proj (
self_attn.kv_b_proj) — reconstructs the KV cache on every decode step. Once corrupted, the error persists in the KV cache for the rest of the conversation — it's not a one-shot error, it's cumulative.
The other ~95% of the model's weight mass (routed experts, dense MLP, other attention projections) is either:
- Routed (only 8 of 256 experts fire per token, so error is isolated, not compounded), or
- Lower-sensitivity (the tensor is large but its per-element contribution to the output is small).
A pure int4 model crushes fine-grained weight information uniformly. A mixed-precision approach protects the compounding tensors at int8 while keeping the bulk at int4.
What changed
c/tools/convert_fp8_to_int4.py — the classify() function was split from a coarse 3-type scheme (resident / expert / io) into 5 sub-types:
| Type |
Tensors |
Why separated |
sh |
shared expert (gate/up/down) |
Fires every token, highest sensitivity |
o |
o_proj |
Reconstructs output, biggest attn tensor |
kvb |
kv_b_proj |
Reconstructs KV cache on every decode |
attn |
q_a_proj, q_b_proj, kv_a_proj |
Other attention projections |
dmlp |
dense MLP gate/up/down (first 3 layers) |
Dense MLP before the sparse layers |
New CLI arguments (each defaults to --ebits for backward compatibility):
--shared-bits N # bits for shared expert (default: ebits)
--o-bits N # bits for o_proj
--kvb-bits N # bits for kv_b_proj
--attn-bits N # bits for other attention projections
--dmlp-bits N # bits for dense MLP
Recommended plan
python tools/convert_fp8_to_int4.py \
--indir /path/to/GLM-5.2-FP8 \
--outdir /path/to/glm52_i4_mixed \
--ebits 4 --io-bits 8 --group-size 128 \
--shared-bits 8 --o-bits 8 --kvb-bits 8
This puts sh, o, and kvb at int8; everything else at grouped int4 (fmt=4, gs=128).
Cost analysis
| Tensor type |
Count |
Per-tensor size |
int4 vs int8 delta |
| shared expert |
3×78 layers |
~2 MB each |
+6 MB |
| o_proj |
78 |
~3 MB each |
+234 MB |
| kv_b_proj |
78 |
~1.5 MB each |
+117 MB |
| Total extra RAM |
|
|
~+5.3 GB |
vs. the 372 GB expert pool that stays at int4 — the mixed-precision overhead is 1.4% of total resident weight memory.
Impact on the program
- Backward compatible: with no
--*-bits flags, behavior is identical to the previous converter. Existing int4 models are unaffected.
- Composes with grouped quantization (
--group-size 128): the two features are orthogonal — grouped scales reduce per-tensor error, mixed precision protects the most sensitive tensors.
- Composes with EXPERT_BUDGET: budget trims the expert union; mixed precision improves the quality of the resident (non-expert) weights. They address different axes.
- No engine changes needed: the engine already supports int8 (fmt=1) and int4 (fmt=2/4) tensors side-by-side — format is auto-detected per-tensor from the
.qs scale array size. The converter simply emits different formats for different tensors.
What's needed before merge
- Production quality A/B: convert the real model with
--shared-bits 8 --o-bits 8 --kvb-bits 8 vs pure int4, compare output coherence on a warm cache.
- Sensitivity ranking validation: the 3-tensor ranking (sh > o > kvb) is research-backed but not yet empirically confirmed on GLM-5.2. An ablation (int8 one category at a time) would confirm the ranking.
Prior art
- QuantTrio/GLM-5.2-Int4-Int8Mix: mixed int4/int8 with channel-wise scales for GLM-5.2
- HOBBIT (arXiv 2411.01433): mixed-precision expert offloading — sensitivity-aware bit allocation
- MxMoE (ICML 2025): MoE experts exhibit divergent quantization sensitivity; uniform quant is suboptimal
- Automated Fine-Grained MoE Quantization (ACL 2025): layer/expert-wise sensitivity variation
TL;DR
The int4 converter now supports per-tensor-type mixed precision, letting each weight category (shared expert, o_proj, kv_b_proj, attention projections, dense MLP) be quantized to a different bit width independently. The research-backed plan: put the 3 most error-sensitive tensors at int8, everything else at grouped int4 — for only +5.3 GB RAM vs pure int4, protecting the tensors that compound error on every token while keeping the 372 GB expert pool at 4-bit.
Branch:
windows-dev(commit8f2ae4d)Why
Not all tensors suffer equally from int4 quantization. In a MoE model, three categories of weights compound their error on every single token:
shared_experts.*) — fires on every token regardless of routing. A quantization error here pollutes the output of every forward pass, with no averaging from other experts.self_attn.o_proj) — reconstructs the attention output. It's the biggest attention tensor and the last matmul before the residual stream. Errors here directly corrupt the residual.self_attn.kv_b_proj) — reconstructs the KV cache on every decode step. Once corrupted, the error persists in the KV cache for the rest of the conversation — it's not a one-shot error, it's cumulative.The other ~95% of the model's weight mass (routed experts, dense MLP, other attention projections) is either:
A pure int4 model crushes fine-grained weight information uniformly. A mixed-precision approach protects the compounding tensors at int8 while keeping the bulk at int4.
What changed
c/tools/convert_fp8_to_int4.py— theclassify()function was split from a coarse 3-type scheme (resident / expert / io) into 5 sub-types:shokvbattndmlpNew CLI arguments (each defaults to
--ebitsfor backward compatibility):Recommended plan
This puts
sh,o, andkvbat int8; everything else at grouped int4 (fmt=4, gs=128).Cost analysis
vs. the 372 GB expert pool that stays at int4 — the mixed-precision overhead is 1.4% of total resident weight memory.
Impact on the program
--*-bitsflags, behavior is identical to the previous converter. Existing int4 models are unaffected.--group-size 128): the two features are orthogonal — grouped scales reduce per-tensor error, mixed precision protects the most sensitive tensors..qsscale array size. The converter simply emits different formats for different tensors.What's needed before merge
--shared-bits 8 --o-bits 8 --kvb-bits 8vs pure int4, compare output coherence on a warm cache.Prior art