convert: support compressed-tensors mixed-precision NVFP4 checkpoints

[danielhanchen]

Problem

Every NVFP4 checkpoint Unsloth publishes is refused by convert_hf_to_gguf.py:

Traceback (most recent call last):
  File "convert_hf_to_gguf.py", line 311, in main
    model_instance.write()
  File "conversion/base.py", line 1100, in write
    self.prepare_tensors()
  File "conversion/qwen.py", line 150, in prepare_tensors
    super().prepare_tensors()
  File "conversion/base.py", line 924, in prepare_tensors
    self.dequant_model()
  File "conversion/base.py", line 506, in dequant_model
    raise NotImplementedError("Can't handle multiple config groups for compressed-tensors yet")
NotImplementedError: Can't handle multiple config groups for compressed-tensors yet

This reproduces on unsloth/Qwen3.6-35B-A3B-NVFP4, unsloth/Qwen3.6-35B-A3B-NVFP4-Fast
and unsloth/Qwen3.6-27B-NVFP4. ggml-org/llama.cpp master has the same two all(...)
gates (conversion/base.py:492 and :861) and the same raise at :496, so it is
refused there too.

Why

These are compressed-tensors "format": "mixed-precision" checkpoints with two config
groups, not one:

group	format	weights	targets
group_0	float-quantized	8 bit, strategy: channel	self_attn.(q|k|v|o)_proj, linear_attn.(in_proj_qkv|in_proj_z|out_proj), lm_head (and on the non-Fast 35B also the layer 32 to 39 experts)
group_1	nvfp4-pack-quantized	4 bit, group_size: 16, scale_dtype: torch.float8_e4m3fn	mlp.experts.*.(gate|up|down)_proj, shared_expert.(gate|up|down)_proj

The nvfp4_compressed_tensors gate demands that all groups be
nvfp4-pack-quantized, so a checkpoint that is NVFP4 for the experts and FP8 for the
attention is classified as neither, and falls into the multi-group raise.

Fix

Four hunks in conversion/base.py:

1. and 2. all(...) becomes any(...) in both copies of the gate (dequant_model and
   prepare_tensors), so a mixed-precision checkpoint with an NVFP4 group is recognised
   as NVFP4.
2. _generate_nvfp4_tensors identifies NVFP4 tensors by dtype and block geometry
   (weight uint8, scale float8_e4m3fn, one scale per 16 values) rather than by
   scale.ndim >= 2 alone. This matters because the FP8 group's per-channel
   weight_scale is shape [out, 1], which is also rank 2, so without the extra test
   the FP8 tensors were fed to _nvfp4_pack.
3. The nvfp4_compressed_tensors branch of dequant_model was a bare pass. It now
   dequantizes whatever weight_scale entries survive _generate_nvfp4_tensors (which
   in a mixed-precision checkpoint are exactly the non-NVFP4 group) using the same
   dequant_simple call the float-quantized branch uses, and drops the unused
   input_scale, k_scale and v_scale sidecars.

Verification

convert_hf_to_gguf.py \
  ~/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-NVFP4-Fast/snapshots/1c3f884b.../ \
  --outfile Qwen3.6-35B-A3B-NVFP4-Fast.gguf --outtype auto --fp8-as-q8

Converts in about six minutes on a DGX Spark and writes general.file_type = 39
(MOSTLY_NVFP4), 1233 tensors, 21.32 GiB:

ggml type	tensors	GiB	share	example
NVFP4	240	16.941	79.5 %	blk.0.ffn_down_exps.weight [512, 2048, 256]
BF16	72	2.527	11.9 %	token_embd.weight [2048, 248320]
Q8_0	131	1.769	8.3 %	output.weight [2048, 248320]
F32	790	0.085	0.4 %	blk.0.ffn_down_exps.scale [256]

480 of the F32 tensors are the weight_scale_2 and input_scale sidecars written as
.scale and .input_scale.

Measured on a DGX Spark (GB10, sm_121a), nvidia-smi -lgc 300,2100, observed
clocks.sm 2093 MHz on every cell, against the same model as MXFP4_MOE and as
unsloth/Qwen3.6-35B-A3B-GGUF UD-Q4_K_XL:

cell	Q4_K_XL (20.82 GiB)	MXFP4_MOE (18.42 GiB)	NVFP4 (21.32 GiB)
prefill npp 2048 npl 8, S_PP t/s	2145.1 / 2174.4	2466.3 / 2459.0	2345.8 / 2360.8
decode npp 128 ntg 128 npl 1, S_TG t/s	65.74	68.03	67.28
decode npl 8, S_TG t/s	207.57	226.99	224.90
decode npl 32, S_TG t/s	325.40	341.48	340.22
wikitext-2 test PPL, c 2048, 8 chunks	4.8213 +/- 0.1228	4.9970 +/- 0.1289	4.9348 +/- 0.1269

nsys on the NVFP4 prefill cell confirms the Blackwell FP4 path is what runs:
42.3 % of GPU time in mul_mat_q<(ggml_type)40, 128, false> plus 2.5 % in
quantize_mmq_nvfp4<...>, the FP4 activation quantiser, so it is genuine W4A4.

KL divergence over 4 chunks with the NVFP4 GGUF as the base (no BF16 checkpoint was
available for an absolute reference):

test	mean KLD	99 % KLD	RMS delta p	same top-1
Q4_K_XL	0.0561 +/- 0.0023	0.634	7.28 %	90.32 %
MXFP4_MOE	0.0980 +/- 0.0038	1.031	9.66 %	86.88 %
NVFP4 with -ctk q8_0 -ctv q8_0	0.0381 +/- 0.0021	0.430	6.19 %	91.59 %

Two things this does not do

• The KV cache scales are dropped. All three checkpoints carry
  quantization_config.kv_cache_scheme = 8 bit float, per-tensor, static_minmax, and
  ship self_attn.k_scale / self_attn.v_scale for the full-attention layers. There is
  no GGUF representation for them, so they are discarded and llama.cpp's KV cache stays
  F16 unless the user passes -ctk / -ctv. The FP8 KV the checkpoint was calibrated
  for is not reproduced.
• The input_scale sidecars are written but unused. llama-model.cpp loads them as
  TENSOR_NOT_REQUIRED, but the MMQ activation quantiser quantize_mmq_nvfp4 computes
  its own dynamic per-row global scale (row_amax / (6 * 448)) at runtime and then does
  a five candidate local search over neighbouring UE4M3 codes per 16 element sub-block.
  The checkpoint's statically calibrated activation scale is therefore ignored, which is
  a deliberate difference from the vLLM CUTLASS path but worth being explicit about.

Draft because the size result deserves a follow-up: the convert-only path cannot touch
what the HF checkpoint left unquantized, so token_embd and the ignore-listed
linear_attn.in_proj_a / in_proj_b stay BF16 and the NVFP4 GGUF ends up larger than
Q4_K_XL. An --outtype style override for the unquantized remainder would fix that.

Follow-up measured after this PR was opened

Use --outtype q8_0, not --outtype auto. A convert-only path cannot touch what the HF
checkpoint never quantized, so with --outtype auto the ignore-listed token_embd and
linear_attn.in_proj_a / in_proj_b stay BF16 (2.53 GiB) and the file lands at 21.32 GiB, larger
than Q4_K_XL. With --outtype q8_0 --fp8-as-q8 it is 20.14 GiB:

ggml type	tensors	GiB	share
NVFP4	240	16.941	84.1 %
Q8_0	203	3.111	15.5 %
F32	790	0.085	0.4 %

The 240 NVFP4 tensors and all 480 .scale / .input_scale sidecars are byte-identical between the
two builds (sha256 over the concatenated NVFP4 tensor bytes matches). llama-quantize cannot do
this shrink instead: --tensor-type is only consulted inside llama_tensor_get_type, which is
only reached when tensor_allows_quantization returns true, and that function begins with
if (params->only_copy) return false; (src/llama-quant.cpp:289), while without COPY the NVFP4
expert tensors are eligible for requantization and there is no way to exempt them.

The q8_0 build also measures slightly better: PPL 4.9027 +/- 0.1260 against 4.9348 +/- 0.1269, and
prefill 2405.0 t/s against 2344.9 t/s, decode at npl 32 338.1 t/s against 339.7 t/s.

The GGUF reproduces the source checkpoint. Feeding vLLM the exact token ids that
llama-perplexity --kl-divergence-base scored (read out of the .dat header, so there is no
tokenizer mismatch) and scoring the same second-half positions with prompt_logprobs=20, over 4092
positions:

	PPL
vLLM 0.28.0, FlashInferCutlassNvFp4LinearKernel + FLASHINFER_CUTLASS MoE	6.3145
llama.cpp, this NVFP4 GGUF	6.3371 (+0.36 %)

Top-20 KL(vLLM || GGUF), renormalised over vLLM's top-20 support: 0.042921 nats, top-1 agreement
90.69 %. The weights are bit-identical between the two, so that residual is the KV cache and the
activation quantiser described above.