Summary
test-backend-ops -o GATED_DELTA_NET fails 0/18 on both gfx1201 (RDNA4, R9700) and gfx1100 (RDNA3, 7900 XTX), ERR ≈ 0.5–0.7 on every config. After tracing the CUDA kernel against the CPU reference, this is not a numerical bug in the GPU attn/state computation — it's an output-contract gap: the op's output tensor was extended with an intermediate_states region that the CUDA kernel populates but the CPU reference does not, so the correctness oracle compares real data against an unwritten region and fails unconditionally.
Environment
- Fork base:
ac06e543 (perf/luce-verify-kernels); reproduced identically on the older 30c9d7d (backport/ggml-cuda-correctness-groupA).
- GPUs: gfx1201 and gfx1100, ROCm/HIP. Arch-independent (identical failure set on both).
- Only
type=f32 configs exist; all 18 fail.
Root cause
The op packs three regions into one output tensor (ggml/src/ggml.c, ggml_gated_delta_net):
[ attn_output: n_tokens*n_seqs | final_state: S_v*n_seqs | intermediate_states: S_v*n_tokens*n_seqs ] (rows of width S_v*H)
- CUDA (
ggml/src/ggml-cuda/gated_delta_net.cu) writes all three regions — it stores the per-token running state into intermediate_states via store_inter_state(...) inside the token loop (the dflash spec-decode extension).
- CPU reference (
ggml/src/ggml-cpu/ops.cpp, ggml_compute_forward_gated_delta_net_one_chunk) writes only attn_output + final_state. It never touches the intermediate_states region.
test-backend-ops compares the entire output tensor against the CPU oracle, so the unwritten intermediate_states region mismatches the CUDA-computed values → 100% FAIL, independent of kernel correctness.
What is verified correct (by line-by-line comparison to the CPU reference)
The CUDA attn/final_state math matches the CPU reference for both paths:
- gate:
kda → each state row *= exp(g[i]); scalar → whole state *= exp(g[0])
delta[j] = (v[j] − Σᵢ S[i][j]·k[i]) · beta- fused outer-product update
S[i][j] += k[i]·delta[j]
attn[j] = (Σᵢ S_updated[i][j]·q[i]) · scale, with scale = 1/√S_v (the host passes the same 1/√S_v the CPU hardcodes)
Warp size is consistent (ggml_cuda_get_physical_warp_size() = 32 on RDNA, matching the runtime block_dims.x), and q/k share strides (asserted neq1 == nek1, ggml_are_same_stride). So the attn + final_state outputs that models actually consume are computed correctly on GPU; only the intermediate_states region (used by dflash spec-decode, not by standard decode) is unvalidated.
Recommended fix (CPU-reference side)
Make the CPU reference produce the same output contract as the CUDA kernel:
- Chain mode (covers ~16/18 configs): in
ggml_compute_forward_gated_delta_net_one_chunk, after each token t's state update, write the running transposed state (S_v*S_v floats) into the intermediate_states region at the matching offset:
dst->data + attn_score_elems + final_state_elems + (iv3*n_tokens*H + iv1)*S_v*S_v + t*(S_v*S_v*H) — same transposed layout as final_state, so a straight memcpy of the per-token state works.
- Tree mode (the 2
parent_ids configs): the CPU reference doesn't implement tree-mode recurrence (parent_ids) at all; it needs the same parent-state reload semantics as the CUDA TREE_MODE path.
- Persist buffer: when an external persist buffer is used the intermediate states go there instead of into
dst — the CPU path should mirror that.
These are the intended CPU semantics of the dflash extensions, so this is best owned by the kernel author. Happy to prep the chain-mode CPU change if useful.
Notes
- Discovered while bringing up the lucebox stack for GPU-kernel work on the R9700; GDN isn't on our critical path (standard llama / softmax targets don't use it), so this is low urgency for us — filing for visibility/tracking.
- Separately, a real RoPE numerical bug in this fork (mod-2π pre-reduction before angle scaling, breaking all
freq_scale≠1/freq_factor≠1 configs) is fixed on GeometricAGI/lucebox-ggml branch fix/rope-fp64-deferred-reduction and staged to upstream here.
🤖 Generated with Claude Code
Summary
test-backend-ops -o GATED_DELTA_NETfails 0/18 on both gfx1201 (RDNA4, R9700) and gfx1100 (RDNA3, 7900 XTX), ERR ≈ 0.5–0.7 on every config. After tracing the CUDA kernel against the CPU reference, this is not a numerical bug in the GPU attn/state computation — it's an output-contract gap: the op's output tensor was extended with anintermediate_statesregion that the CUDA kernel populates but the CPU reference does not, so the correctness oracle compares real data against an unwritten region and fails unconditionally.Environment
ac06e543(perf/luce-verify-kernels); reproduced identically on the older30c9d7d(backport/ggml-cuda-correctness-groupA).type=f32configs exist; all 18 fail.Root cause
The op packs three regions into one output tensor (
ggml/src/ggml.c,ggml_gated_delta_net):ggml/src/ggml-cuda/gated_delta_net.cu) writes all three regions — it stores the per-token running state intointermediate_statesviastore_inter_state(...)inside the token loop (the dflash spec-decode extension).ggml/src/ggml-cpu/ops.cpp,ggml_compute_forward_gated_delta_net_one_chunk) writes onlyattn_output+final_state. It never touches theintermediate_statesregion.test-backend-opscompares the entire output tensor against the CPU oracle, so the unwrittenintermediate_statesregion mismatches the CUDA-computed values → 100% FAIL, independent of kernel correctness.What is verified correct (by line-by-line comparison to the CPU reference)
The CUDA attn/final_state math matches the CPU reference for both paths:
kda→ each state row*= exp(g[i]); scalar → whole state*= exp(g[0])delta[j] = (v[j] − Σᵢ S[i][j]·k[i]) · betaS[i][j] += k[i]·delta[j]attn[j] = (Σᵢ S_updated[i][j]·q[i]) · scale, withscale = 1/√S_v(the host passes the same1/√S_vthe CPU hardcodes)Warp size is consistent (
ggml_cuda_get_physical_warp_size()= 32 on RDNA, matching the runtimeblock_dims.x), andq/kshare strides (assertedneq1 == nek1,ggml_are_same_stride). So the attn + final_state outputs that models actually consume are computed correctly on GPU; only theintermediate_statesregion (used by dflash spec-decode, not by standard decode) is unvalidated.Recommended fix (CPU-reference side)
Make the CPU reference produce the same output contract as the CUDA kernel:
ggml_compute_forward_gated_delta_net_one_chunk, after each tokent's state update, write the running transposed state (S_v*S_vfloats) into theintermediate_statesregion at the matching offset:dst->data + attn_score_elems + final_state_elems + (iv3*n_tokens*H + iv1)*S_v*S_v + t*(S_v*S_v*H)— same transposed layout asfinal_state, so a straightmemcpyof the per-token state works.parent_idsconfigs): the CPU reference doesn't implement tree-mode recurrence (parent_ids) at all; it needs the same parent-state reload semantics as the CUDATREE_MODEpath.dst— the CPU path should mirror that.These are the intended CPU semantics of the dflash extensions, so this is best owned by the kernel author. Happy to prep the chain-mode CPU change if useful.
Notes
freq_scale≠1/freq_factor≠1configs) is fixed onGeometricAGI/lucebox-ggmlbranchfix/rope-fp64-deferred-reductionand staged to upstream here.🤖 Generated with Claude Code