Drop the row-major weight duplicate for Q8_0 (blocked on fused decode still reading it)

[kkokosa]

Follow-up to #397 (Part C). #398, #399 and #400 are merged; this is the remaining piece.

Update: the unfused R4 decode approach has been measured and rejected (+16.56% on decode, 3/9 paired wins) — the dispatch fusion is worth ~17%. A separate finding means byte-identical output is not achievable by any decode-side approach. Revised acceptance criteria and the llama.cpp comparison are in this comment, which supersedes the criteria below.

The approaches below were originally numbered "option 1/2/3"; that earlier comment still uses those numbers. The mapping is: option 1 = R4 fused decode, option 2 = unfused R4 decode, option 3 = partial retention.

Goal

Stop holding Q8_0 linear weights in two layouts at once. Today every repackable weight exists as both the memory-mapped row-major original and an R4-interleaved copy in committed memory, roughly doubling peak footprint.

Measured on bartowski/Llama-3.2-1B-Instruct-GGUF Q8_0 (1,260 MiB file), sampling the process every 25 ms:

reported Weights (mapped)	1,252 MiB
reported Repacked (R4, committed)	1,252 MiB
OS peak private	1,414 MiB
OS peak working set	2,690 MiB

Dropping the row-major copy for Q8_0 would take peak working set to roughly 1.4 GB — about half.

Where things stand after #400

#400 routed Q8_0 prefill through GemmR4TiledQ8_0, so multi-token work no longer touches row-major. That was the prerequisite for this issue, but it did not finish the job.

Auditing every consumer of the row-major pointers in TransformerModel:

Projection	Prefill	Decode
Q, K, V	R4 (#400)	row-major — FusedQkvDecode -> MatMul.FusedDecodeGemv3
Gate, Up	R4 (#400)	row-major — FusedGateUpDecode -> MatMul.FusedDecodeGemv2
O, Down, LM head	R4	R4 (GemmInterleaved, n=1)

So 5 of 7 projections per layer still read row-major on every decode step. The fused decode kernels exist to batch three (or two) GEMVs into a single pool.Dispatch(), saving dispatch overhead per layer — and they take row-major pointers (MatMulFusedDecode.cs, e.g. case QuantizationType.Q8_0: GemvQ8_0(weights, ...)).

The row-major copy therefore cannot be dropped today. Removing it would break decode.

Candidate approaches

R4 fused decode — the target

Port FusedDecodeGemv3/FusedDecodeGemv2 to consume the interleaved layout, dispatching to ComputeRowsQ8_0Interleaved per projection inside one pool dispatch. Keeps the fusion benefit and frees row-major. Most work, best outcome.

Unfused R4 decode — measured, rejected

Route Q/K/V/Gate/Up decode through the existing GemmInterleaved n=1 path, which is already R4, giving up the fused dispatch. Much smaller change. Measured at +16.56% on decode (3/9 paired wins) — not viable. Its value was diagnostic: it established that the dispatch batching is worth ~17%, which is the bar the R4 fused decode has to clear.

Partial retention — not recommended

Keep row-major only for the tensors the fused path needs and drop it for the rest. Saves little and complicates the loader; listed for completeness.

Also still holding row-major

Independent of decode, these keep row-major alive regardless:

• Rows below InterleavedMinRowBytes (1024) fall back in both GemmInterleaved and GemvInterleaved. For Q8_0 that is roughly K < 964.
• K-quants and Q5_0 use row-major for prefill entirely — GemmR4TiledQ8_0 is Q8_0-only, and the K-quant interleaved wrappers were a measured regression in Step 26 (Step 26: Outer-product tiled matmul — kernels + investigation (blocked on AVX2) #61). Porting the perf(cpu): route Q8_0 prefill through an L2-tiled R4 GEMM #400 tiling to ComputeRowsQ4_KInterleaved and friends is the equivalent follow-up for those formats and would benefit Q4_K_M models, which is what most users run.
• token_embd.weight is deliberately excluded from repacking (random row access) and should stay mapped. For this model it is ~266 MiB and a 611-token prompt touches at most 611 of 128,256 rows.

Note that for tied-embedding models (TransformerWeights.cs, LM head aliases token_embd.weight) that tensor is currently resident in both layouts.

Acceptance criteria

• For a Q8_0 model where every linear weight is repackable and above the row-stride threshold, no forward-pass path reads the row-major linear weights.
• The repacked buffers are built without requiring the row-major copy to stay resident afterwards — i.e. the mapping may be released or left cold for those tensors.
• Peak working set for Llama-3.2-1B-Instruct Q8_0 drops materially against the 2,690 MiB baseline above; before/after recorded in the PR using the same sampling method.
• dotllm run --json memory reporting (added in fix(cli): count R4 repacked buffers in reported memory #398) stays accurate after the change.
• Decode tok/s does not regress — must be measured, paired and interleaved, on both .NET 10 and .NET 11. The unfused R4 decode approach failed this by +16.56%.
• Prefill tok/s does not regress against post-perf(cpu): route Q8_0 prefill through an L2-tiled R4 GEMM #400 numbers.
• Byte-identical generated output for a fixed-seed greedy run. Disproven — not achievable by any decode-side option, because the two layouts require kernels with different accumulation order and decode is autoregressive. See the linked comment for the replacement criterion.
• Existing kernel tests pass; new coverage for whichever decode path is introduced.

References

• src/DotLLM.Models/Architectures/TransformerModel.cs — FusedQkvDecode, FusedGateUpDecode, GemmInterleaved, GemvInterleaved, InterleavedMinRowBytes
• src/DotLLM.Cpu/Kernels/MatMulFusedDecode.cs — FusedDecodeGemv3, FusedDecodeGemv2
• src/DotLLM.Cpu/Kernels/MatMul.cs — GemmR4TiledQ8_0 (perf(cpu): route Q8_0 prefill through an L2-tiled R4 GEMM #400), ComputeRowsQ8_0Interleaved
• src/DotLLM.Models/Architectures/TransformerWeights.cs — RepackWeights, RepackedBytes
• R4 repacking: extend to prefill via interleaved GEMM, and fix ~2x under-reported memory #397 (parent), fix(cli): count R4 repacked buffers in reported memory #398 (memory reporting), perf(cpu): route Q8_0 prefill through an L2-tiled R4 GEMM #400 (prefill on R4)
• docs/QUANTIZATION.md, docs/ARCHITECTURE.md

[kkokosa]

Option 2 measured — rejected. The fusion is worth ~17% of decode.

Implemented option 2 (route Q8_0 decode Q/K/V/Gate/Up through the existing R4 GemmInterleaved n=1 path instead of FusedDecodeGemv3/Gemv2) purely as a measurement. Kept on a local branch exp/401-option2-r4-decode, not proposed for merge.

Decode-heavy config — short prompt, 256 generated tokens, --seed 42. Baseline (main) and candidate built to separate directories from one tree and run interleaved, 9 pairs, .NET 10:

	decode median	throughput
main (fused, row-major)	9,545 ms	26.7 tok/s
option 2 (unfused, R4)	11,181 ms	22.8 tok/s

3/9 paired wins for the candidate; mean delta +16.56% slower.

So the single-pool.Dispatch() batching in FusedDecodeGemv3/Gemv2 is worth roughly 17% of decode across 16 layers x 256 tokens. Option 2 is not viable, and any design that drops the fusion pays this.

Correctness finding: byte-identical output is unachievable for any decode-side option

Output diverged between the two variants — 1268 vs 1215 characters.

This is not a bug in the experiment. The row-major kernel (VecDotQ8_0Vnni_4Rows, Vector512 dual-block) and the R4 kernel (VecDotQ8_0Avx2_4RowsR4, Vector256 single-block) accumulate in different orders. The per-value differences are last-bit, but decode is autoregressive: across 256 steps a near-tie in the greedy argmax eventually flips and the sequences separate.

Prefill's switch to R4 in #400 stayed byte-identical because a single forward pass gives far fewer opportunities to diverge.

This applies to option 1 as well — the two layouts require different kernels, which is the entire point, so accumulation order necessarily changes. The acceptance criterion above ("byte-identical generated output for a fixed-seed greedy run") therefore cannot be met by any version of this work and needs replacing with something like logits-within-tolerance or unchanged quality on a fixed eval set.

Worth noting this is normal for the domain — llama.cpp does not guarantee bit-identical output across ISA paths either, and this repo's own kernel tests use 1e-2f tolerances rather than bit-equality for exactly this reason.

What llama.cpp does — they never face this problem

Two structural reasons:

1. Their repacked layout serves both phases. llama.cpp ships GEMV and GEMM kernels over the interleaved layout (PR #9921, "online flow for aarch64 Q4_0 GEMV/GEMM kernels"; the Q4_K repack work is described as improving "both prefill and token generation"). They repack into a CPU_REPACK buffer type where the tensor's data becomes interleaved rather than being duplicated, so there is no row-major copy to retain. vLLM does the equivalent for Marlin via replace_parameter.

2. They have no fused multi-projection dispatch to preserve. ggml's CPU backend processes graph ops one at a time — a thread per core held on a spinlock barrier, released to compute slices of one op's output, barriered again before the next. Q, K and V are three separate MUL_MAT nodes with a barrier between each. llama.cpp pays the per-op synchronisation that FusedDecodeGemv3 exists to avoid.

Reframing option 1

This makes option 1 more attractive than it looked, not less.

The fused decode is not an obstacle inherited from somewhere — it is an optimisation llama.cpp does not have, now measured at ~17% of decode. Option 1 is therefore "llama.cpp's single-layout design plus dotLLM's dispatch fusion", a combination neither reference implementation has. There is no prior art to copy for the hard part.

dotLLM already has both halves in isolation: ComputeRowsQ8_0Interleaved is a working R4 GEMV and already serves O/Down/LM-head at decode today. The missing piece is the group-aligned partitioning.

The concrete difficulty: FusedProjection slices work by row (Weights + projStart * WeightRowBytes). R4 can only be sliced on 4-row group boundaries. So FusedDecodeGemv3R4 needs partitioning over total groups across three projections of differing M, with per-projection tail rows handled separately, while preserving the single-dispatch property that the 17% depends on.

Revised acceptance criteria (superseding the list above)

• For a Q8_0 model where every linear weight is repackable and above the row-stride threshold, no forward-pass path reads the row-major linear weights.
• Peak working set for Llama-3.2-1B-Instruct Q8_0 drops materially against the 2,690 MiB baseline; before/after measured with the same 25 ms sampling method.
• Decode tok/s does not regress against main — paired and interleaved, both runtimes. This is the criterion option 2 failed by 16.56%.
• Prefill tok/s does not regress against post-perf(cpu): route Q8_0 prefill through an L2-tiled R4 GEMM #400 numbers.
• Byte-identical generated output — not achievable, see above. Replace with: logits match the row-major path within tolerance for a fixed input, and generated text remains coherent and on-topic for a fixed-seed run.
• dotllm run --json memory reporting stays accurate.
• New kernel tests for the R4 fused decode path against the existing row-major reference, using the repo's tolerance convention.

Alternative worth weighing first

Porting #400's tiling to K-quant prefill (ComputeRowsQ4_KInterleaved and friends) reuses a structure already proven to win, has no correctness complication, and benefits Q4_K_M models — which is what most users actually run. It does not advance this issue, but it may be higher value per unit of effort. Filing that separately is probably the right move.