Summary
Found while implementing CUDA MTP (#253, PR #286). Both `SpeculativeDecoder` and `MtpSpeculativeDecoder`'s batched draft-verify-accept flow have no rollback mechanism for a rejected token's effect on recurrent trunk state (`IGdnState` — Gated DeltaNet, used by `Qwen3HybridDense` and other hybrid architectures).
Why this matters
For position-indexed state (attention KV-cache), a rejected token's KV-cache entry can simply be ignored/overwritten on the next round — position addressing means stale entries beyond the accepted length are naturally unreachable. But `IGdnState` is a pure sequential recurrence with no position addressing — every token position updates the running state in place. If a draft token is verified and rejected, the batched verify forward pass has already updated the recurrent state as if that token were accepted, and there's no way to roll that update back. The recurrent state after a rejection round is contaminated by the rejected token(s)' effect, and current code has no mechanism to correct it.
Where this was found
While building a real-model CUDA correctness test for MTP self-speculative decoding, which specifically exercises `Qwen3HybridDense` (an architecture with GDN layers). Two related but distinct bugs in the same area were found and fixed in #253/PR #286 (a catchup-forward double-processing bug, and a test-harness position off-by-one) — this rollback gap is the remaining, deeper architectural issue neither of those fixes addressed, left for its own investigation.
Affected
- `src/DotLLM.Engine/SpeculativeDecoder.cs` (the original two-model draft-verify-accept decoder)
- `src/DotLLM.Engine/MtpSpeculativeDecoder.cs` (the new MTP self-speculative decoder)
- Both CPU and CUDA backends identically — this is an engine-layer/state-model gap, not a kernel bug.
Scope for the fix
Needs investigation into what a correct rollback looks like for `IGdnState` specifically — likely either (a) checkpointing recurrent state before a verify round and restoring on partial rejection, or (b) recomputing the recurrent state from the accepted-token boundary forward. Check whether any existing GDN state checkpoint/restore mechanism exists elsewhere in this codebase (e.g. for KV-cache rollback on rejection in `SpeculativeDecoder` today — how does that currently handle partial rejection for attention-only models, and can the same pattern extend to GDN state, or does it need something structurally different given the no-position-addressing property).
Impact today
Any speculative decoding (regular two-model or MTP) run against a GDN/hybrid-recurrent architecture (`Qwen3HybridDense` and siblings) with partial draft rejection may currently produce subtly incorrect recurrent state after a rejection round, which could compound over a long generation. Non-recurrent (pure attention) architectures are unaffected. Severity/frequency not yet measured — needs its own investigation.
Summary
Found while implementing CUDA MTP (#253, PR #286). Both `SpeculativeDecoder` and `MtpSpeculativeDecoder`'s batched draft-verify-accept flow have no rollback mechanism for a rejected token's effect on recurrent trunk state (`IGdnState` — Gated DeltaNet, used by `Qwen3HybridDense` and other hybrid architectures).
Why this matters
For position-indexed state (attention KV-cache), a rejected token's KV-cache entry can simply be ignored/overwritten on the next round — position addressing means stale entries beyond the accepted length are naturally unreachable. But `IGdnState` is a pure sequential recurrence with no position addressing — every token position updates the running state in place. If a draft token is verified and rejected, the batched verify forward pass has already updated the recurrent state as if that token were accepted, and there's no way to roll that update back. The recurrent state after a rejection round is contaminated by the rejected token(s)' effect, and current code has no mechanism to correct it.
Where this was found
While building a real-model CUDA correctness test for MTP self-speculative decoding, which specifically exercises `Qwen3HybridDense` (an architecture with GDN layers). Two related but distinct bugs in the same area were found and fixed in #253/PR #286 (a catchup-forward double-processing bug, and a test-harness position off-by-one) — this rollback gap is the remaining, deeper architectural issue neither of those fixes addressed, left for its own investigation.
Affected
Scope for the fix
Needs investigation into what a correct rollback looks like for `IGdnState` specifically — likely either (a) checkpointing recurrent state before a verify round and restoring on partial rejection, or (b) recomputing the recurrent state from the accepted-token boundary forward. Check whether any existing GDN state checkpoint/restore mechanism exists elsewhere in this codebase (e.g. for KV-cache rollback on rejection in `SpeculativeDecoder` today — how does that currently handle partial rejection for attention-only models, and can the same pattern extend to GDN state, or does it need something structurally different given the no-position-addressing property).
Impact today
Any speculative decoding (regular two-model or MTP) run against a GDN/hybrid-recurrent architecture (`Qwen3HybridDense` and siblings) with partial draft rejection may currently produce subtly incorrect recurrent state after a rejection round, which could compound over a long generation. Non-recurrent (pure attention) architectures are unaffected. Severity/frequency not yet measured — needs its own investigation.