quant: AVX-512 arm for matmul_i3 (fmt=5 int3-g64)

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic
(We/our = the joint work; me/my = @monotophic's own hardware and containers).

What this is

The follow-up the code asked for: matmul_i3's comment reads "x86 stays
scalar for now (follow-up)" — this adds the x86 vector arm, gated
__AVX512F__ && __AVX512BW__. The int3-g64 group geometry maps naturally onto
AVX-512: one full 64-value group per iteration — the 16B low plane through an
extended variant of matmul_i2's SSE unpack, the 8B high plane loaded directly
as a __mmask64 driving a masked +4, four sign-extend/convert/FMA chunks,
one reduce per group. The per-group structure is preserved exactly: FMA
reorder is confined WITHIN a group; the cross-group acc += a*srow[g] chain
stays in scalar order. Partial tails and non-AVX-512 builds keep the scalar
loop; NEON is untouched (the two arms are an #if/#elif chain).

Runtime envelope mirrors the existing i4 AVX-512 precedent: I3_AVX512=0
kill-switch and an I3_AVX512_TEST startup selftest that compares the vector
arm against the scalar decode on a fixed group and hard-fails on mismatch.
The selftest compares EXACTLY (!= on floats) by construction: every term is
an integer with any partial sum bounded by 8320 < 2^24, so f32 summation is
order-invariant there — the compare is exact-by-design, not tolerance-lite.

Measured (Zen 5 / Strix Halo, GLM-5.2 744B real int3-g64 container)

• Decode: 0.28 → 0.61–0.69 tok/s end-to-end (2.2–2.5×), 64-token greedy,
  two prompts × two reps, interleaved.
• Temp-0 generated tokens IDENTICAL, vector vs scalar (kill-switch A/B),
  all four pairs — rep-stable.
• Kernel accuracy on silicon: independent harness max rel err 2.44e-07
  vs f32 reference (discipline: 2e-4), adversarial patterns + lane probes.
• Selftest bite: three seeded error classes (mask inversion, bias, lane swap)
  each fail it on silicon (578/961/1425 != 1034).

Capstone matrix

requirement	decisive evidence
scalar/NEON paths untouched	arm64 colibri.o byte-identical, base vs branch
gate needs exactly F+BW	EVEX byte-audit of the object: no VL/DQ/VBMI/VNNI
numerics correct on silicon	test_int3 (incl. new I=33/I=7 tails) + selftest + 2.44e-07 harness
selftest actually bites	3/3 seeded mutations fail it on-host
observable behavior unchanged	temp-0 token A/B identical on the real container
it's worth having	2.2–2.5× end-to-end decode on a 744B real workload

Fuller instrument/result table

• Cross-compile gates: full TU at -mavx512f -mavx512bw, -march=x86-64-v4,
  and a znver5-like +VL/DQ/VNNI set — zero warnings under -Wall -Wextra.
• arm64 (NEON) build: zero warnings; extended test_int3 green; object-level
  byte-identity vs base proves the gate compiles to nothing off-x86.
• test_int3 failures now name the failing (I,S,O) tuple.
• Selftest exactness bound: w ∈ [−4,3], x[k] = ±(k+1), |any partial| ≤ 8320,
  integers exact in f32 under any grouping; Σx ≠ 0 so uniform bias can't cancel.
• Independent harness: own packer + own reference built from the format spec,
  shapes I ∈ {7,33,64,65,100,128,192,255} × O ∈ {1,7} × S ∈ {1,4}, hostile
  scales ±1e18/±1e-18, 256-point lane-permutation probe — all clean on Zen 5.
• A/B protocol: MTP=0, --cap 96 --ram 90 --ngen 64 --temp 0, same
  container/profile both arms, I3_AVX512=0 vs 1, reps interleaved.

Sibling work: #654 landed AVX2 for fmt=6 this week ("fmt=6 was 92% of decode
on a scalar kernel") — same class of win; this PR covers fmt=5's version of
that gap. VNNI/IDOT composition stays out of scope per the kernel's own note
(int8 activations don't compose with per-group accumulation without a
restructure); the Zen 5 host is instrumented for that follow-up when wanted.

Durable vs current-state: kernel, gate, and runtime envelope are durable;
all tok/s figures are calibrations at (Zen 5 Strix Halo, GLM-5.2 744B
int3-g64 container, dev fa599e0-era, 2026-07-28) and re-derive with host or
container.