quant: AVX2 matmul_e8 — fmt=6 decode was 92% of the time on a real model

[steve-m]

fmt=6 (E8/IQ3) landed its container (#465, #501) and a CUDA kernel (#627), but on the CPU it still runs the scalar reference kernel — and on a real model that dominates everything else.

Measured on GLM-5.2 (744B, E8 container), 32-token greedy decode:

PROFILE: expert-disk 40.7s service / 1.1s wait | expert-matmul 206.6s | attention 12.7s

206.6 s of a 224.1 s decode — 92% — inside matmul_e8, with disk service fully overlapped (1.1 s of actual wait). The format's own arithmetic was the wall, not I/O. This is the same shape of problem AVX2 matmul_i3 fixed for int3-g64.

The kernel

The format suggests its own vectorisation: one 8-weight lane is two 4-dim grid rows (8 contiguous codebook bytes) plus 8 signs — exactly one AVX2 register. So a lane decodes straight into a register and FMAs against x, instead of being expanded into a stack buffer and read back:

• the 8 grid bytes widen with a single vpmovzxbd;
• the sign byte (the 7 stored bits plus the parity-derived 8th) expands to 8 lane masks with AND/CMPEQ against a bit-select vector, applied as an XOR of the float sign bit. That removes the per-weight branch, which is what made the scalar expansion expensive;
• the grid's half-unit convention folds into the sub-scale, so magnitudes need no separate scaling;
• two accumulators over a super-block's 32 FMAs keep it off a single dependency chain, and one horizontal add per 256 weights replaces one per 32.

The scalar path stays for the ragged tail and for non-AVX2 hosts. The converter enforces whole 256-blocks, so the fast path is the one real containers take.

Numbers

Kernel, single-threaded: 280.8 → 4141.3 Mw/s (14.7×) — 44.8 → 3.0 ms per [2048, 6144] expert tensor.

End to end on GLM-5.2 (RX 9070 box, 32-token greedy decode, cold cache), identical text produced in every configuration:

	prefill	decode	expert-matmul
before	91.3 s	224.1 s (0.14 tok/s)	206.6 s
after	16.7 s	38.6 s (0.83 tok/s)	18.7 s

5.9× decode, 5.5× prefill. Decode becomes disk-bound again (40.3 s service, 3.3 s wait) rather than compute-bound, which is the regime the streaming design targets.

Correctness

tests/test_e8_kernel green against its existing fixture (worst rel 1.46e-06). AVX2 and scalar agree to 2.2e-07 relative L2 over a 512-row random container — that is accumulation order, not a semantic difference. make check 220.