CUDA: dispatch the chunked gated_delta_net in CU mode when it fills the device

[roberteg16]

What

The fused GDN kernel is bound by issue pressure on the LDS pipe, not by bandwidth or arithmetic: SQ_LDS_BANK_CONFLICT / SQ_LDS_IDX_ACTIVE is 0.00%, MemUnitBusy is 38.6%, and SQ_INSTS_VALU / SQ_INSTS_LDS is 3.03 against 7-9 for the token-by-token kernel it replaced.

CU mode confines a block to one CU, which halves the LDS contention domain. It costs no residency here: a WGP holds floor(128 KB / 57.25 KB) = 2 of these blocks in WGP mode and 2*floor(64 KB / 57.25 KB) = 2 in CU mode, so the resident block count is the same either way.

It is applied per kernel with __attribute__((target("cumode"))), which flips .amdhsa_workgroup_processor_mode for that kernel alone — no build-system change, and nothing else in the backend is affected.

The gate

CU mode only pays once every CU has a block to run. A block occupies one CU in CU mode but spreads over the whole WGP in WGP mode, so below one block per WGP the second CU of each WGP sits idle. Cost per op at 512 tokens against block count (H_v * n_seqs), on gfx1151 where nsm = 20 WGPs:

blocks	4	8	16	20	24	32	40	48
delta	+39.8%	+39.6%	+33.5%	+19.1%	-17.3%	-9.0%	-9.2%	-9.2%

The crossover falls exactly on nsm, so the launch picks between two entry points on H * n_seqs > nsm. The work-group processor mode is a function attribute rather than something a template parameter can select, so the two entry points share one inlined body and differ only in that attribute.

Results

Op time, interleaved with alternating order, medians of 8-10:

H_v	blocks	path	64 tok	512 tok	4096 tok
16 (Qwen3.5-0.8B)	16	WGP	—	+0.5%	+0.3%
24	24	CU	—	-8.3%	—
32 (Qwen3.5/3.6-35B-A3B)	32	CU	-7.5%	-6.4%	-9.0%
48 (Qwen3.6-27B)	48	CU	—	-8.1%	—

The head counts come from the models themselves, not from an assumption: the 35B stages q/k as [128, 16, T, 1] but v as [128, 32, T, 1] with state [128, 128, 32, 1], so it is 32 v-heads behind a 2-way GQA, and the dispatch keys on nev1 (v-heads), not on the q/k head count.

Head counts on the CU path are 6.4-10.6% cheaper at every sequence length from 64 to 4096 tokens. Head counts below the threshold keep the WGP path and are unchanged; without the gate, H_v = 16 would have regressed 23-34%.

End to end

GDN is now only ~7% of prefill, so a tenth off the op predicts about +0.7% throughput. That is close to the noise floor, so this was measured with the build order alternated every iteration and with the median of samples_ts after discarding the first repetition — llama-bench's first repetition is systematically cold (a 0.8B pp128 series reads 4572, 8368, 8419, 8401, ...) and avg_ts averages over it, which reads ~6% low and swamps an effect this size.

Six iterations per cell, -ub 512:

model	path	pp128	pp4096
Qwen3.5-0.8B (H_v=16)	WGP	+0.00%	no signal
Qwen3.5-35B-A3B (H_v=32)	CU	+0.54%	+0.67%

The 35B pp4096 cell is the one that clears the noise: the two sample ranges barely touch (1784-1792 against 1769-1783), and +0.67% is what the op-level -9% predicts.

The 0.8B keeps the WGP path, and a further ten iterations paired per iteration confirm it is untouched: pp128 comes out at a +0.04% median with the new build ahead in 5 of 10 pairs. Its pp4096 cell reads -0.29% but two of the ten pairs are the first and last measurement of the series, both of which landed on the baseline and both elevated (10316 and 10074 against a ~9600 steady state); dropping them leaves -0.19% with pair deltas scattered from -0.79% to +1.3%. There is also no mechanism for a real loss there — the refactor moved the WGP kernel by +0.3 to +0.5% at op level, and GDN is a small enough share of that model's prefill that this is ~0.03% end to end.

Beyond throughput, the op does the same work in ~10% less GPU time and energy, which matters on an APU and leaves more of the device for other ops.

Testing

test-backend-ops -o GATED_DELTA_NET: 42/42 pass with the path enabled and disabled. Both entry points report VGPR 91, zero spill, zero scratch, 16 waves/SIMD and LDS 58624 B, and .amdhsa_workgroup_processor_mode 1 and 0 respectively.

[mgehre-amd]

This is the first time where I see CU-mode have a non-null effect.

[roberteg16]

This is the first time where I see CU-mode have a non-null effect.

Me too 👀 . I am still verifying this is not a wash

[mgehre-amd]

Does this produce larger end2end wins at higher contexts? Otherwise I think a +0.67% win is too small to go through the maintenance and upstreaming hussle.

[roberteg16]

Does this produce larger end2end wins at higher contexts? Otherwise I think a +0.67% win is too small to go through the maintenance and upstreaming hussle.

Kernel wise the win is around 6-9%. e2e tho, since we already improved GDN quite a lot is not that noticeable. If we happen to keep optimizing the rest of the model this +0.67% could become +2%.

We can keep this PR as a draft until that happens if you prefer.