perf(cpu): AVX2 beats both 512-bit tiers in the Q8_0 row-major dot product, but ComputeRows prefers them

[kkokosa]

Observation

The row-major A/B added in #414's benchmark (RowMajorVnniBenchmarks) compares all three tiers ComputeRows dispatches between. On an AVX-512 host (AVX-512F+CD+BW+DQ+VL+VBMI, .NET 10.0.10, BDN 0.14.0, 3 warmup / 10 iterations, single-threaded), baseline is the AVX-512 maddubs kernel that #399's VNNI kernel displaced:

Kernel	K=512	ratio	K=4096	ratio
VecDotQ8_0Avx512_4Rows (maddubs, pre-#399)	162.08 ns	1.01	816.83 ns	1.00
VecDotQ8_0Vnni_4Rows (#399, currently shipped)	108.78 ns	0.67	795.41 ns	0.97
VecDotQ8_0Avx2_4Rows (AVX2 maddubs)	89.62 ns	0.56	695.54 ns	0.85

Two things follow:

1. perf(cpu): use AVX-VNNI vpdpbusd for the Q8_0 4-row dot product #399 was a genuine improvement over what it replaced — 0.67x at K=512, 0.97x at K=4096 against the AVX-512 maddubs tier.
2. But plain AVX2 beats both, by 1.21x over VNNI at K=4096 and 1.21x at K=512. And ComputeRows (src/DotLLM.Cpu/Kernels/MatMul.cs:130) tiers Avx512BW && AvxVnni -> Avx512BW -> Avx2, so on this hardware the fastest of the three is the one tier that never gets selected.

This is the same shape as the #414 result on the R4 kernel, where the 256-bit AVX2 kernel beat the VNNI one by 1.8-1.9x. Two independent kernel families, same ordering: the 512-bit dual-block variants lose to 256-bit single-block. Consistent with the mechanism — Q8_0's per-block fp16 scale forces a convert-scale-FMA every 32 elements, and the Vector256->Vector512 assembly plus scale broadcast-and-insert costs more than the wider FMA saves.

Why this is not simply "flip the dispatch order"

ComputeRows is the decode path (GEMV, n == 1), and decode is memory-bandwidth-bound, not kernel-throughput-bound. A 1.21x microkernel win may not survive end-to-end at all. Measured earlier in the same investigation: decode on Llama-3.2-1B Q8_0 runs well under this machine's memory ceiling, so the kernel may not be the limiter.

So the microbenchmark justifies investigating, not changing. The change needs an end-to-end decode measurement before it is defensible.

Proposed work

1. Measure end-to-end decode tok/s with the dispatch preferring AVX2, against main, on Llama-3.2-1B Q8_0 — the number that decides this.
2. If it moves: reorder the tiers, with a comment recording the measurement, and keep the wider kernels reachable for hardware that orders them differently.
3. If it does not move: document that the tier ordering is performance-neutral for decode and close. That result is worth having written down, because it tells us decode work belongs in traffic reduction rather than kernel width.
4. Either way, re-run RowMajorVnniBenchmarks on a second microarchitecture before concluding anything general. One CPU is one CPU — @jamesburton's perf(cpu/matmul): AVX-512 VNNI for Q8_0 outer-product (AvxVnni.V512 / VPDPBUSD-512) — requires .NET 11 bump #322 numbers are Zen5 and ordered these differently.

Acceptance criteria

• End-to-end decode measured for both dispatch orders on the same build and model.
• Any dispatch change carries the measurement in a comment, and parity tests still pass on every reachable tier.
• The negative case is documented rather than silently dropped.

References

• src/DotLLM.Cpu/Kernels/MatMul.cs — ComputeRows (:130) and the three 4-row kernels (:696, :875, :975)
• benchmarks/DotLLM.Benchmarks/R4VnniBenchmarks.cs — RowMajorVnniBenchmarks
• perf(cpu): the R4 Q8_0 prefill kernel has no AVX-512 or VNNI tier #414 — the R4 prefill experiment that produced these numbers
• perf(cpu): use AVX-VNNI vpdpbusd for the Q8_0 4-row dot product #399 — introduced the VNNI tier
• perf(cpu/matmul): AVX-512 VNNI for Q8_0 outer-product (AvxVnni.V512 / VPDPBUSD-512) — requires .NET 11 bump #322 — @jamesburton's Zen5 investigation; the ILP argument for 256-bit over 512-bit originates there

[jamesburton]

Ran RowMajorVnniBenchmarks on Zen5 (Strix Halo, Ryzen AI Max 395, 16C/32T, .NET 10.0.3, AVX-512F+CD+BW+DQ+VL+VBMI) — a second microarchitecture as requested:

Kernel	K=512	K=4096
VecDotQ8_0Avx512_4Rows (maddubs, pre-#399)	104.70 ns (1.00x)	804.73 ns (1.00x)
VecDotQ8_0Vnni_4Rows (#399, shipped)	104.14 ns (0.99x)	772.30 ns (0.96x)
VecDotQ8_0Avx2_4Rows	92.24 ns (0.88x)	704.57 ns (0.88x)

Confirms the headline finding on a second microarchitecture: AVX2 beats both 512-bit tiers at both K, and ComputeRows's dispatch order (Avx512BW && AvxVnni -> Avx512BW -> Avx2) never reaches it.

One nuance worth flagging vs. the AVX-512 host numbers in the issue body: there, VNNI was a real win over the old AVX-512 maddubs kernel it replaced (0.67x at K=512 — #399 was a genuine improvement over what it displaced). On Zen5 that gap nearly disappears — VNNI and AVX-512 maddubs are statistically indistinguishable at K=512 (0.99 ± 0.03) and only a modest win at K=4096 (0.96). So #399's win over the kernel it replaced looks host-dependent; the "AVX2 is fastest and the dispatch can't reach it" conclusion does not — that part replicates cleanly.

Also: this only needs mainstream AvxVnni.IsSupported / Avx512BW.IsSupported (not the .V512 preview surface from #322's later comments) — Strix Halo hits both tiers natively on the shipping .NET 10 SDK, no dev-runtime build required for this particular measurement.

---

Side note since this is the first real back-and-forth on this in a while — happy to keep running these. Beyond this Zen5 box (128GB unified memory, also useful for Vulkan/iGPU cross-checks), the fork now has an Intel Meteor Lake laptop and two RTX 3060 12GB CUDA boxes available, so AVX2/AVX-512/VNNI x86 variants plus CUDA/Vulkan comparisons are all reachable fairly quickly on request.

Also worth flagging: quite a bit has landed on our dev branch while these threads sat idle — broader quantization format support (I2_S/BitNet ternary, PQ2_0, more K-quants), multi-device benchmarking infrastructure, LoRA (train/serve, multi-adapter, across CPU/CUDA/Vulkan), and various CUDA graph-capture/attention-kernel fixes, among other things. Some of that already has draft PRs open against main; a fair amount more is dev-only and not yet PR'd, mostly for lack of a live thread to PR into rather than any real blocker. Happy to open focused PRs for whichever pieces are of interest — I'll check what's already built vs. what still needs isolating into its own branch before saying anything is missing.

[jamesburton]

Ran items 1 and 4 on Zen 5 (Ryzen AI Max+ 395 "Strix Halo", 16C/32T, .NET 10.0.10, BDN 0.14.0,
3 warmup / 10 iterations). Summary: the ordering replicates, the magnitude does not, and it does
not survive end-to-end. My recommendation is not to reorder — that is acceptance criterion 3.

Item 4 — second microarchitecture

One correction first: the issue says my #322 numbers "are Zen5 and ordered these differently". They
do not — a fresh run here gives the same ordering as your host. Three clean runs, means:

Kernel	K=512	ratio	K=4096	ratio
VecDotQ8_0Avx512_4Rows (maddubs, pre-#399)	106.8 ns	1.00	780.6 ns	1.00
VecDotQ8_0Vnni_4Rows (#399, shipped)	102.7 ns	0.96	781.2 ns	1.00
VecDotQ8_0Avx2_4Rows	93.4 ns	0.87	722.5 ns	0.93

AVX2 is fastest on both hosts. But the margin is much smaller here — 1.10x over VNNI at K=512 and
1.08x at K=4096, against your 1.21x at both. And #399's own gain is roughly nil on this part:
VNNI ties the AVX-512 maddubs kernel at K=4096 (781.2 vs 780.6 ns) where you measured 0.97.

Likely mechanism, consistent with both: this part executes AVX-512 on 256-bit datapaths (my
hardware notes record it as F+CD+BW+DQ+VL+VBMI+VNNI (256-bit)). A 512-bit kernel is then issued as
two 256-bit operations — no throughput gained, while the Vector256->Vector512 assembly and the
scale broadcast-and-insert are still paid. That predicts "512-bit ties 256-bit at best, loses by the
shuffling overhead", which is what both hosts show, differing only in how much overhead.

A methodological caveat worth recording. My first two runs of this benchmark gave
AVX-512@K=4096 of 1066.7 ns and 796.1 ns — a 34% spread — because a test suite was running. Clean,
the three runs are 785.1 / 774.6 / 782.1. On a unified-memory part background load distorts these
microbenchmarks badly, so single runs here are not trustworthy.

Item 1 — end-to-end decode

Llama-3.2-1B-Instruct Q8_0, 128 tokens, greedy, all cores. A/B within one build via a temporary
DOTLLM_CPU_Q8_ROWS=avx2 override, so the two arms differ only in the tier selected — comparing two
builds would confound the kernel with everything else.

First, the path is genuinely reachable: I instrumented entry counts and a decode run enters the
row-major ComputeRows 43,840 times against 33,025 for the R4 kernel. (Not what I expected — Q8_0 is
repackable and RepackWeights() runs unconditionally at load, so I had assumed decode was entirely
R4. It is not.) Worth knowing the counters themselves cost 33.3 -> 18.7 tok/s from false sharing;
they were removed before timing.

The result depends on run order, which is the trap here. Alternating pairs, whichever arm ran
first won by about 10% — so a naive A/B returns whichever answer its ordering implies. Balancing
positions (n=16 per arm, each arm appearing equally in both slots):

	mean	sd
default dispatch (VNNI tier)	42.08 tok/s	2.61
forced AVX2	40.85 tok/s	2.09

Difference +1.23 tok/s (+3.0%) in favour of the current dispatch, SE 0.84, t = 1.47 — not
significant. Position-matched, the direction is consistent in both slots (+2.0% in slot 1, +4.0% in
slot 2), so if there is a real effect it favours leaving the tiering alone.

So the 1.08-1.10x microkernel win does not survive end-to-end, exactly as you predicted for a
bandwidth-bound path — and reordering to prefer AVX2 would if anything cost a little. I have not
opened a PR; the change I would have made is the one the measurement says not to make.

Happy to push the DOTLLM_CPU_Q8_ROWS override as a small diagnostic if you want the A/B repeatable
on other hosts — it is three lines and off by default. Otherwise this can close as documented-negative.