ggml-cuda: pad matmul weight rows to avoid cache-set aliasing

[roberteg16]

Overview

A weight whose packed row size is a multiple of the cache set-index stride puts every row in the same cache sets, so a matmul walking down a column thrashes a single set. Adding one cache line to the row stride spreads the rows out again.

ggml_backend_cuda_buffer_init_tensor rewrites nb[1..3] for an eligible weight and get_alloc_size reserves the extra bytes, zeroed so the gaps cannot hold NaNs. The loader marks candidates with a new GGML_TENSOR_FLAG_PAD_ROWS and uploads the packed file data with a strided 2D copy. The matmul paths are unchanged: cuBLAS, mmf, mmvf and mmq all take their leading dimension from nb[1] already.

Quantized weights are excluded: those kernels read the row stride as nb[1]/type_size, and a 128 B pad would break that division being exact.

Tuning

The alias stride and the pad follow from the cache geometry (line size times number of sets), which neither CUDA nor HIP report — hipDeviceProp_t::l2CacheSize is documented as always returning 0 — so they are tabulated per architecture in ggml_cuda_row_pad_params_for(). Only RDNA3.5 has an entry (2048/128); any other architecture keeps packed rows until it has been measured.

variable	effect
GGML_CUDA_ROW_ALIAS_STRIDE	override the row size that triggers padding
GGML_CUDA_ROW_PAD	override the bytes added to the row stride
GGML_CUDA_NO_PAD_WEIGHTS	disable the padding

Settings are resolved and validated once per device. A stride that is not a power of two, a pad that is a multiple of it, or a pad that would violate the nb[1] % (2*type_size) requirement of mmf/mmvf all fall back to no padding rather than degrading silently.

Results

gfx1151, llama-bench -ngl 999 -p 128,4096 -n 128 -d 128 -ub 2048 -r 10, padding on vs off. Each measurement is the median of its repetitions, each cell the median across 4-6 rounds that alternate which side runs first. spread is the largest range between rounds on either side, so a delta below it is not separable from run-to-run variation.

model	test	off	on	delta	spread
gemma-3-4b BF16	pp128@d128	1144.57	2024.34	+76.86%	4.85%
gemma-4-E2B BF16	pp128@d128	2056.70	2590.53	+25.96%	1.15%
gemma-3-4b BF16	pp4096@d128	2275.95	2546.84	+11.90%	0.77%
gemma-4-E2B BF16	tg128@d128	42.04	43.40	+3.23%	2.52%
Qwen3.6-35B-A3B UD-Q4_K_M	tg128@d128	59.75	60.57	+1.37%	0.10%
Qwen3.5-35B-A3B Q4_K_M	tg128@d128	59.15	59.88	+1.23%	0.27%
gemma-3-4b BF16	tg128@d128	26.97	27.25	+1.07%	0.17%

The gain concentrates in small-batch prefill, where a short run of output rows streams the whole weight matrix and the weight-side cache behaviour dominates.

Q4_K_M models are largely unaffected: their large GEMM weights are quantized and therefore excluded, leaving only the MoE router and ssm_alpha/ssm_beta, which show up as the ~+1.2-1.4% decode gain on the two MoE models. Across all 21 model/test cells the worst delta is -0.47%, within its own spread.

Two notes on scope: the +3.23% figure is the least separated from its spread and should be read as indicative, and the two BF16 models are the only non-quantized GGUFs that fit in 32 GiB of VRAM, so the large-gain result rests on two models of the same family.

[roberteg16]

@mgehre-amd padding works amazing for bf16/f16/f32 checkpoints but for GGUF-quantized ones only a handful of layers stay in f32. For these models, this optimization is mostly shown only on decode and you can see it is around +1-1.4% . Do you have any thoughts on whether we ought to merge it? Should we wait for bf16/f16 use cases?

[mgehre-amd]

@mgehre-amd padding works amazing for bf16/f16/f32 checkpoints but for GGUF-quantized ones only a handful of layers stay in f32. For these models, this optimization is mostly shown only on decode and you can see it is around +1-1.4% . Do you have any thoughts on whether we ought to merge it? Should we wait for bf16/f16 use cases?

A win for bf16/f16/f32 checkpoints is still a considerable win. I think we should get it merged and upstreamed.

[roberteg16]

Added a second commit with the idea of making this more upstreamable. Now every arch can define their desired values.

[mgehre-amd]

Thanks!