GLM-5.3-Flash on CUDA: read nvidia/GLM-5.3-Flash-NVFP4 (ModelOpt NVFP4) as stored, W4A16 #308

What

GLM's CUDA engine reads the MLX 4-bit and EXL3 checkpoints. I would like it to read NVIDIA's ModelOpt NVFP4
checkpoint, nvidia/GLM-5.3-Flash-NVFP4 (revision 423acf37583782c51c142d145aef733d72943d93), without converting
it. I have a branch for it and would like to know what it needs to land before I open the pull request.

The checkpoint

• The routed experts (layers 3-44) and the dense MLP (layers 0-2) are ModelOpt NVFP4: e2m1 codes, an e4m3 scale per
  16 inputs, an fp32 weight_scale_2 per tensor. input_scale is stored as well (static activations).
• Everything else is BF16: every attention projection, the shared experts, the router, the LM head and embeddings,
  and the whole MTP layer, its 288 routed experts included.
• quantization_config.kv_cache_scheme declares an FP8 KV cache (static, 8-bit float); the checkpoint stores no KV
  scales. That is a separate question, in its own issue.

What the branch does (W4A16, the checkpoint's bytes as stored)

• Weights stay as stored. Routed experts load into the existing NVFP4 Experts4 blocks (the grouped kernel Flash Next
  uses) with weight_scale_2 as the per-expert multiplier. The dense MLP runs on Fp4Linear (gate and up apart,
  since their tensor scales differ). The shared expert runs as a BF16 MLP, as for EXL3.
• Activations stay BF16 (W4A16). input_scale is not read, and --precision checkpoint named on the command line is
  refused, because the grouped NVFP4 expert kernel takes bf16 rows only. The checkpoint's own FP4 x FP4 math is a
  separate step (as Flash Next CUDA: routed NVFP4 experts in the checkpoint's own math (FP4 x FP4), prompts −9…−10 % #211 did for Flash Next).
• One change to stored weights, for drafting only: the MTP layer's BF16 routed experts are packed to NVFP4 by
  ModelOpt's recipe at load, a projection at a time. TensorFold has no BF16 grouped expert kernel; these experts only
  draft, and verification never reads them, so drafted replies still equal "draft": false ones. It saves about
  4.85 GiB a rank. I would like to know whether you prefer a BF16 path for them instead.
• Two ranks split each weight_scale with its codes, so both keep whole 16-input groups and the grouped kernel's
  32-wide tiles. I checked that on all 147,661 tensor headers of the real checkpoint.
• NVIDIA stores the KDA conv taps as fp32. The KDA kernels already widened each tap to fp32, so they now read fp32 taps
  and the loader keeps the stored values. MLX and EXL3 store the taps bf16; widened at load, they give the same bits
  as before.
• The startup estimate counts what the loader holds: about 90.9 GiB a rank on the real headers.

Questions

1. Is W4A16 with the checkpoint's bytes as stored the right first step, with FP4 x FP4 later?
2. Is packing the MTP layer's BF16 experts to NVFP4 for drafting acceptable?
3. The header test uses a 435 KB gzipped dump of the real checkpoint's safetensors headers
   (tests/fixtures/glm53_flash_nvfp4/headers.json.gz, names, dtypes and shapes only). Keep it, shrink it to the
   tensors the checks need, or drop the test?
4. A follow-up gives NVFP4 routed experts a prompt-shaped kernel. It changes prompt bits (each pair's sum in a
   different order from decode's, the same for any chunking), as the MLX experts' prefill form does. Is that
   acceptable for a family where prompt and decode bits already differ?

Tested on two DGX Sparks (GB10, TP=2). I have no Mac in this loop, and the change touches CUDA only.

Written with Weave (My Symbiotic AI)