spec: greedy output diverges from non-speculative baseline under batched verification on CUDA (deterministic numerics; draft-simple alone reproduces, DFlash2 amplifies)

[indubwestep]

Summary

With speculative decoding enabled (temp 0, fixed seed, otherwise identical settings), greedy output diverges from the identical non-speculative run. The divergence is deterministic (reruns are byte-identical) and is classified as CUDA numerical divergence under batched speculative verification, not a logic bug: the plain draft-simple path — which exercises batched verification with none of the DFlash2 code — reproduces it on one workload, while the DFlash2 path amplifies it (diverges on both workloads tested, and earlier).

Outputs remain coherent; this is a bit-exactness concern only, and speculative acceptance is unaffected. No fix is claimed here — this is a documentation/confirmation request.

Environment

• llama.cpp at PR spec : add DFlash2 support (local convolution + candidate selector) #27342 head: 5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4 (local build, MSVC 19.44, CUDA 12.4, sm_86)
• OS: Windows 11 x86_64
• GPU: NVIDIA RTX 3090 Ti 24 GB
• Target: empero-ai/Qwen3.8-27B-Ridge-GGUF Qwen3.8-27B-Ridge-3.7bpw.gguf (IQ2_M)
• Draft: incoai/Qwen3.8-27B-DFlash2-GGUF Qwen3.8-27B-DFlash2-Q4_K_M.gguf (1,143,006,752 bytes; SHA-256 18a380efc9b7ed8d88677fc895f5c11ae170653434ee378f7348f715c14d0594)

Repro (llama-cli, 6 runs)

Baseline (--spec-type none):

llama-cli -m <target> -ngl 999 --flash-attn on -c 8192 \
  --cache-type-k q4_0 --cache-type-v q4_0 -b 256 -ub 128 -t 8 \
  --temp 0 -s 42 --no-display-prompt -no-cnv -st \
  -n 256 -f prompt_short.txt < nul

Speculative variants add:

  -md <draft> --spec-draft-ngl 999 --spec-draft-n-max 7 --spec-draft-p-min 0 \
  --spec-type draft-simple        # or draft-dflash

Run the same with -n 512 -f prompt_coding.txt. Note: --draft-model is not a valid flag in this build; use -md (cli) / --spec-draft-model (server).

Results (token-level, same-vocab llama-tokenize)

workload	none vs draft-simple	none vs draft-dflash
short (n=256)	diverge at gen-token ~109	diverge at gen-token ~49
coding (n=512)	identical 512/512	diverge at gen-token ~372

Stripped-output MD5 (static banner/prompt-echo removed; generation content only):

run	short	coding
none	db0b78ede21fcb32543217e5cbfd437f (1330 B)	1d69d9619a1bde9249dac21e6fccaf99 (2482 B)
draft-simple	c55e97dec9b537272f72dc578e1f00dc (1346 B)	1d69d9619a1bde9249dac21e6fccaf99 (2482 B) — = none
draft-dflash	25dfca084f4cec36bc8f71f5eaa8da24 (1401 B)	24d531f596c8aa41c1119ea01bd824d7 (2459 B)

Reruns are byte-identical (determinism confirmed).

Example near-tie flips (short workload): " storage of information." vs ", recall, forgetting"; ". Too many" vs "syllables. Too". Coding workload: digit flip "1, 2" vs "8, 4" in generated code.

Analysis

• Batched verification alone (draft-simple, no DFlash hooks) perturbs CUDA target logits at near-ties: it diverges on the short workload yet is byte-identical on coding → near-tie dependent, not systematic.
• draft-dflash diverges on both workloads and earlier: DFlash2's extraction graph (llama_set_embeddings_layer_inp graph-output marking) amplifies the numeric shift but is not the sole cause (proven by the draft-simple divergence).
• Deterministic across reruns → consistent with CUDA fp-non-associativity in the batched verification decode (e.g. n_max=7 draft batch in one flash-attn batch vs sequential target decode) and/or graph-extraction side effects.
• Supporting data point: z-lab's Metal validation of DFlash2 achieved byte-identical greedy output (see dflash: zero-fill draft-cache holes left by mtmd chunks and reused prefixes z-lab/llama.cpp-fork#1), suggesting a platform-numeric component.

Questions

1. Is bit-exact greedy output expected to be preserved under batched speculative verification on CUDA? (MTP presumably shows the same class of divergence.)
2. Is there a supported way to force the sequential target-decode path during verification, for bit-exact comparisons?
3. If this is accepted as generic numerics, it may be worth a short note in PR spec : add DFlash2 support (local convolution + candidate selector) #27342 so downstream users don't mistake it for output corruption.

Cross-links: PR #27342 (DFlash2), companion issue mtmd vision position-gap, z-lab#1 (Metal byte-identical validation).

[thc1006]

Second CUDA data point, on a different quantisation and OS, plus a width structure that may narrow this.

RTX 3090 (sm_86), driver 610.43.02, Debian 13, built from c060ca9 with CUDA 13.3. Target is unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL rather than your IQ2_M and KV is q8_0 rather than q4_0, so this is a different point in the same space. -c 8192 -fa on --parallel 1, greedy, cache_prompt: false, 400 tokens, 25 prompts, 5 passes, fresh server per arm per pass. The draft-dflash arms are PR #27342 at d1a522f and are compared only against a no-speculation baseline built from that same branch.

The divergence reproduces and is deterministic, as you found. 76 to 80 % of requests differ from their baseline depending on the arm, forking at a median 23 % into the text, and every arm is byte-identical to itself across all five passes.

What might be worth following is that the fork position is a function of verification width. Running n_max 2, 3, 4, 5 and 7, so widths 3, 4, 5, 6 and 8, the character index of the first difference is identical inside two groups and different between them: widths 3 and 4 agree on 25 of 25 prompts, widths 5, 6 and 8 agree on 25 of 25, and the two groups land somewhere different on 14 of them. Same structure in all five passes. The grouping crosses drafter types, so draft-mtp at width 6 sits with draft-dflash at widths 5 and 8 rather than with the other draft-mtp arms.

It is not monotone either. One prompt diverges at widths 3 and 4 and is byte-identical to baseline at 5, 6 and 8.

ggml/src/ggml-cuda/mmvq.cu changes at that width. ggml_cuda_should_use_mmvq returns ne11 <= MMVQ_MAX_BATCH_SIZE, which is 8 on this fallthrough, so widths 3 through 8 all stay inside MMVQ and the kernel family is not what moves. calc_nwarps does move: sm_86 is not RDNA, GCN, CDNA or DGX Spark and the Turing table is gated below AMPERE, so it lands on MMVQ_PARAMETERS_GENERIC, which gives 4 warps for ncols_dst 1 to 4, 2 warps for 5 to 8, and 1 above that. The warp count feeds blocks_per_iter, so each thread walks a different subset of K, and it sets how many partial sums are folded out of tmp_shared, so the reduction tree differs too.

That is one boundary coinciding on one architecture at one quantisation, which is consistent with the mechanism and is not proof of it. n_max 1 through 8 is running now. That puts widths 2 and 7 in for the first time and adds width 9, where calc_nwarps drops to one warp and should_use_mmvq stops returning true at the same time. If the grouping tracks the table those come out as {2,3,4}, {5,6,7,8} and {9} on its own; if width 2 or 7 lands in the wrong group it is tracking something else and I will post that.

Three caveats. My KV is q8_0, and @snick525 showed in #25618 that q8_0 KV moves greedy output with no speculation on either side; it is held constant across every arm here so it should not produce a width-dependent grouping, but the f16 control is queued rather than run. Your target is IQ2_M and mine is UD-Q4_K_XL, so the grouping may not transfer. And I have not run draft-simple against this target, so I cannot speak to your result that it stays identical on the coding workload, which is the more interesting half of your table.

The per-prompt fork table and the rest of the setup are in #25618 (comment). Happy to run other controls on this card if any would help.

[Apoze]

Hi,
I ran in a similar problem. Did some diging with an AI agent and ended up with some tests, resutts and a fix for my case on an RTX 5090. Here are more informations:

Test Environment:
RTX 5090 (SM120), CUDA 13.3, pilote 610.88
Base tested: f1357e4

Symptom:
Vanilla and MTP being deterministic but diverging in greedy.

Root causes identified:

1. An incorrect MMVF selection order;
2. Different Flash Attention paths and reductions depending on batch width.

Results after fix:

• Exact vanilla ↔ MTP parity for n_max = 1..3;
• divergence still present for n_max >= 4 (not fixed in my branch).

Tests:

• Exact comparisons: 75/75
• Long context: exact
• Flash Attention: 2936/2936
• MUL_MAT: 1193/1193

Branches (The first one is the fix and the other one is where you can find my tests, results and relevants files)

• Fix: https://github.com/Apoze/llama.cpp/tree/fix/cuda-speculative-batch-invariance
• Evidence / reproduction: https://github.com/Apoze/llama.cpp/blob/evidence/cuda-speculative-batch-invariance/evidence/cuda-speculative-batch-invariance

If you want me to create a PR or two, do tell me. Or you can just take what you are interrested in the code right away.