Test-model generators: FP8 emission + expert weight unfusing (fixes silent expert-skip bug in converter)

[woolcoxm]

TL;DR

The test-model generators (make_glm_oracle.py, make_glm_bench_model.py, glm_fp8_emit.py) now:

1. Emit weights as FP8 e4m3 + 128×128 block scale_inv in the exact layout of the real GLM-5.2-FP8 checkpoint, so the converter's FP8→int4 dequant path can be exercised locally without the 379 GB download.
2. Unfuse HF's 3-D gate_up_proj expert weights into the per-expert 2-D layout (gate_proj/up_proj/down_proj) that the real checkpoint, the converter, and the C engine all expect.

Two bugs were found and fixed in the process.

Branch: windows-dev (commits 4788685, 91b1035, d9b6a2d)

---

Why

The converter (convert_fp8_to_int4.py) has a dequant path for FP8 e4m3 + 128×128 block scales. Before this change, the only way to exercise that path was to download the 379 GB GLM-5.2-FP8 checkpoint. There was no local fixture that could validate the FP8→int4 pipeline, meaning bugs in the dequant path would only surface during the multi-hour production conversion.

The FP8 emission gap

glm_fp8_emit.py is a new shared helper that:

• Block-quantizes 2-D tensors to FP8 e4m3 with 128×128 block scales (FBGEMM/TransformerEngine convention: scale = amax(block)/448, stored as a multiplier)
• Keeps norms, router, 1-D biases, and 3-D tensors at f32 (matching the converter's classify() + ndim != 2 guard)
• Provides fp8_block_dequantize (exact inverse) for the oracle's reference round-trip

Both generators gained a --fp8 flag. The oracle additionally round-trips weights through FP8 before computing ref_glm.json, so the reference reflects exactly the FP8 model the converter ingests.

Bug 1: Fused expert weights (found during validation)

HF's GlmMoeDsaForCausalLM fuses gate_proj + up_proj into a single 3-D gate_up_proj tensor [E, 2*M, I] for compute efficiency. But the real GLM-5.2-FP8 checkpoint stores experts unfused as per-expert 2-D tensors:

model.layers.10.mlp.experts.0.gate_proj.weight       (2048, 6144)   FP8 e4m3
model.layers.10.mlp.experts.0.gate_proj.weight_scale_inv  (16, 48)   F32
model.layers.10.mlp.experts.0.up_proj.weight          (2048, 6144)   FP8 e4m3
model.layers.10.mlp.experts.0.up_proj.weight_scale_inv   (16, 48)   F32
model.layers.10.mlp.experts.0.down_proj.weight        (6144, 2048)   FP8 e4m3
model.layers.10.mlp.experts.0.down_proj.weight_scale_inv (48, 16)    F32

The converter's classify() matches ".mlp.experts." in name, but its ndim != 2 guard silently skipped the 3-D fused tensors. The C engine then crashed with missing tensor: model.layers.3.mlp.experts.5.gate_proj.weight because it expects per-expert 2-D tensors.

Fix: unfuse_experts() in glm_fp8_emit.py splits gate_up_proj [E, 2M, I] → per-expert gate_proj [M, I] + up_proj [M, I], and down_proj [E, I, M] → per-expert down_proj [M_out, I]. Called after reference generation (the model needs fused weights for forward()/generate()) but before saving, in both generators, for both FP8 and bf16 paths.

Bug 2: Oracle FP8 round-trip guard

make_glm_oracle.py's FP8 round-trip loop used p.dim() < 2 to skip non-quantizable tensors. But 3-D fused expert tensors have dim() == 3, which passed the guard and crashed fp8_block_quantize (which expects exactly 2-D). Fixed to p.dim() != 2, matching the converter's guard.

What changed

File	Change
c/tools/glm_fp8_emit.py	New: fp8_block_quantize, fp8_block_dequantize, keep_f32, state_dict_to_fp8, save_fp8_safetensors, unfuse_experts
c/tools/make_glm_oracle.py	--fp8 flag; round-trips weights through FP8 before computing reference; unfuses experts before saving; fixed dim guard
c/tools/make_glm_bench_model.py	--fp8 flag; unfuses experts before saving; writes config.json manually in FP8 path (bypasses save_pretrained)

Impact on the program

• Enables local testing of the full conversion chain: generator --fp8 → converter --indir → engine load, all without a network download.
• Fixes a silent data-loss bug: without unfusing, the converter would skip all expert weights and the engine would crash. This would have manifested as a broken production conversion once the FP8 download completed.
• Backward compatible: without --fp8, both generators behave exactly as before (bf16 output, save_pretrained path).
• Validated end-to-end: 570 FP8 tensors + 629 scale_inv sidecars → converter produces per-row int4 (fmt=2) and grouped int4 (fmt=4) → engine loads both clean, fmt=4 auto-detected.

Verification

generator --fp8 → 570 e4m3 tensors + 629 scale_inv (was 90 when fused)
converter --group-size 0   → per-row int4 (fmt=2), engine loads clean, 0 missing-tensor errors
converter --group-size 128 → grouped int4 (fmt=4), 8-16x more scales per tensor, engine loads clean
dequant error: grouped 1.14-1.22x lower than per-row vs FP8 source

[JustVugg]

Tracked by PR #244 (open, in review) — FP8 emission + expert weight unfusing, including the fix for the silent expert-skip bug in the converter. Keeping open until #244 merges.

[JustVugg]

Delivered — closing. This is already on dev; it just never got its own PR closed.

The content arrived inside 69f65d5 / 71c262c / 2c6946c (your "windows-dev" commits), which bundled grouped quantization + mixed precision + expert budget + the download tool together. Because the pieces landed as one commit rather than as their own PRs, the issues tracking them stayed open even though the code shipped. #243 and #244 were ghosts of the same kind — I trial-merged #243 onto dev and the diff came back empty.

Verified in the tree rather than assumed:

• c/tools/glm_fp8_emit.py exists on dev with both save_fp8_safetensors and unfuse_experts.
• make_glm_oracle.py --fp8 computes ref_glm.json after the FP8 round-trip, so the reference matches exactly what the converter ingests. That detail is the whole value: make_glm_oracle.py on transformers 5.5.0 produces an oracle the engine scores 25/32 against (expected 32/32) #281 was the same class of bug from the other side — an oracle built against a different forward pass than the engine implements, scoring 25/32 against a reference that was itself wrong.