qwen35: batch DFlash2 decode for Qwen3.8-27B concurrency

[Graffioh]

Stack

This PR is stacked directly on #625 at 910a2086. It contains seven #642 commits, ending at 308f630f.

The final #625 change after the benchmarked parent was test-only: server/test/test_server_unit.cpp now expects disk-cache version 2. It does not change the server or runtime source.

Summary

Add fixed-width DFlash2 speculative decoding to Qwen3.8-27B continuous batching.

• Keep packed autoregressive and prefill service for prompt-bearing rounds.
• Batch DFlash2 draft generation, selector scoring, and fixed-chain verification across eligible decode lanes.
• Share one target forward between speculative lanes and autoregressive fallback lanes.
• Promote accepted K/V rows, target features, GDN state, convolution history, and output bursts as one transaction.
• Keep convolution, masks, positions, and draft KV state lane-local.
• Preserve the existing single-request --ddtree path from qwen35: DSpark speculative decoding (Qwen3.8-27B drafters) #625.

The concurrent path requires a selector-enabled local draft, same-device placement, paged full attention, and greedy sampling. Distribution-preserving sampled speculation is in the stacked follow-up #654.

DFlash2 sliding window

--draft-swa 2048 changes the effective window without replacing the checkpoint's per-layer SWA pattern. Metadata-backed mixed and all-full patterns are preserved; legacy inference runs only when the metadata is absent. Applying the override twice is idempotent.

The tested Qwen3.8 DFlash2 draft reports SWA layers: 5/5 (window=2048). Three post-window lanes at committed positions 65, 81, and 133 matched singleton execution with max_abs=0 on both gfx1201 and gfx1151.

Run concurrent DFlash2

env ROCR_VISIBLE_DEVICES=0 \
  DFLASH_SINGLE_CHAIN_CHECKPOINT_F32=1 \
  DFLASH_FAST_ROLLBACK_THRESHOLD=1 \
  LUCE_Q8_MEMO=1 \
  DFLASH_KV_ROTATE=0 \
  ./server/build-hip/dflash_server /path/to/Qwen3.8-27B-IQ4_XS.gguf \
  --draft /path/to/Qwen3.8-27B-DFlash2-Q8_0.gguf \
  --target-device hip:0 --draft-device hip:0 \
  --draft-block-size 8 --draft-swa 2048 --draft-residency persistent \
  --paged-attention --fa-window 0 \
  --max-concurrency 6 --max-ctx 4096 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --admission-coalesce-ms 20 \
  --prefix-cache-slots 0 --prefill-cache-slots 0 \
  --model-name qwen38 --host 127.0.0.1 --port 18080

Startup must report SWA layers: 5/5 (window=2048) and [parallel-chain] fixed DFlash2 width=8. A speculative cohort reports [draft-kv-batch] packed backbone ready.

W8 versus W16

I screened W8 and W16 with fresh servers, the same target, draft, prompt manifest, and SWA 2048 setting.

C	W8 goodput, two runs	W16 goodput, two runs	Median change
3	90.72, 90.52	90.69, 90.69	+0.08%
6	107.49, 107.66	98.80, 98.53	-8.28%

The refreshed #625 matrix produced 106.68 tok/s at C6 with W8, still 8.1% above the W16 median. W16 is noise-level at C3 and materially worse at C6, so this PR keeps the checkpoint-native W8 path. The experimental W16 prewarm commit is not in the stack.

Refreshed Qwen3.8-27B concurrency results

Each cell starts a fresh server. C is the number of live requests. Each completed cell contains 60 requests with temperature 0, natural EOS, and at most 64 output tokens. Prefix caching is disabled.

The deterministic manifest contains 15 entries from each checked-in code, GSM, math, and agent suite. Prompts contain 43 to 2,643 tokens, with a median of 107.5. Both modes use the same pure-IQ4_XS target, 4,096-token context, Q8_0 K/V, paged full attention, and 20 ms admission window. DFlash2 adds the Q8_0 draft, W8, and SWA 2048.

GPU	DFlash2	C	Success	Goodput tok/s	Output-window tok/s	Median TTFT s	Max TTFT s
R9700	OFF	1	60/60	28.31	35.80	0.241	2.650
R9700	ON	1	60/60	62.38	111.04	0.197	2.628
R9700	OFF	2	60/60	39.74	53.12	0.399	3.363
R9700	ON	2	60/60	81.68	162.97	0.319	3.343
R9700	OFF	3	60/60	48.06	67.12	0.450	5.138
R9700	ON	3	60/60	89.69	187.59	0.419	5.271
R9700	OFF	4	60/60	62.12	95.09	0.578	5.433
R9700	ON	4	60/60	96.02	209.02	0.515	5.505
R9700	OFF	5	60/60	69.66	112.58	0.618	6.304
R9700	ON	5	60/60	99.67	218.92	0.655	6.483
R9700	OFF	6	60/60	78.50	134.32	0.792	7.132
R9700	ON	6	60/60	106.68	246.32	0.811	7.174
Strix Halo	OFF	1	60/60	10.45	13.75	0.704	9.685
Strix Halo	ON	1	60/60	21.67	44.34	0.544	9.638
Strix Halo	OFF	2	60/60	15.78	23.69	0.975	12.564
Strix Halo	ON	2	60/60	30.28	66.87	0.840	10.292
Strix Halo	OFF	3	60/60	19.91	30.27	1.195	15.916

The refreshed R9700 matrix is complete: 720/720 requests, zero failures, and a 1.36x to 2.20x DFlash2 speedup over AR. The isolated Strix run completed 300/300 valid requests through C2 plus C3 AR. An unrelated shared-host CI job killed the C3 DFlash server and immediately started another GPU build, so that interrupted cell and C4-C6 are excluded rather than reported as measurements.

Goodput is total generated output divided by suite wall time, including prompt processing and scheduler retirement. Output-window throughput counts generated output between the first and last streamed tokens.

Artifact identity:

• Runtime benchmark head: fd1e7fbcf1f9321d51eab2dd0198809cb550de13
• Final head: 308f630f7eb379a1fd91902becd4ea22ab249101
• Runtime benchmark parent: qwen35: DSpark speculative decoding (Qwen3.8-27B drafters) #625 85e727625e4cca2b15acdc6e71934f020c3c65ac
• Final parent: qwen35: DSpark speculative decoding (Qwen3.8-27B drafters) #625 910a2086de7a7483f92699549353a133d936eaea; the only delta is the test expectation described above
• Target SHA-256: 150fa0d5e30a7971ee258b6e42c2b33d778e4fc9dca18d55a1797a860921a816
• Draft SHA-256: 777cd45c4f78dfcca13b3e857dc7ff70ca7d9ab589ed50e7e2b338466cc62f4a
• Manifest-file SHA-256: 4636a62b616ba3d429276d70328e80f02518f69ee74ee39ee6133d02506503ed
• Canonical {messages} SHA-256: 72f03c07c46bec9ec32679a2dff937956a4c3d506792a65d4b6d903ad27bee13

Verification

• git range-diff reports all seven qwen35: batch DFlash2 decode for Qwen3.8-27B concurrency #642 commits patch-equivalent across the final qwen35: DSpark speculative decoding (Qwen3.8-27B drafters) #625 restack.
• Built the final gfx1201 server and server-unit binary at 308f630f.
• Passed the disk-cache version-2 test inherited from qwen35: DSpark speculative decoding (Qwen3.8-27B drafters) #625.
• Passed five host SWA policy cases and the real three-lane SWA qualification on gfx1201 and gfx1151.
• Passed the focused rollback-policy, slot-manager, batched-GDN, journal, gallocr, transaction, KV-pool, paged-tree, and ROCTX checks before the final test-only parent update.
• Confirmed W8, SWA, and packed-backbone markers in every completed DFlash2 cell.
• Passed git diff --check.

Scope

This PR excludes adaptive admission, cost models, DSpark batching, KVFlash, PFlash, and per-request sampling policy. #651 batches draft projections across lanes. #654 adds distribution-preserving sampled speculation.

[Graffioh]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 48 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Graffioh]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 53 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Graffioh]

renamed this into replay log in PR 651

[cubic-dev-ai]

5 issues found across 62 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/common/dflash2_head.cpp">

<violation number="1" location="server/src/common/dflash2_head.cpp:91">
P1: When singleton selector scoring receives selector codebooks whose vocabulary differs from the target, this validation accepts the layout and later gathers codebook rows using target-generated IDs. Populate `target_output_vocab` from `target.lm_head_tensor()` (and the declared target vocabulary when available) before validating, or reject the mismatch before `ggml_get_rows()`.</violation>
</file>

<file name="server/src/common/feature_gate.cpp">

<violation number="1" location="server/src/common/feature_gate.cpp:267">
P2: When the drafter width is below 16, this gate reserves 16 scratch tokens per lane while `Qwen35Backend` reserves the actual draft width. It unnecessarily rejects valid `--kv-pool-tokens` values; derive this cap from the resolved draft width or enforce the address limit only where that width is known.</violation>
</file>

<file name="server/deps/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp">

<violation number="1" location="server/deps/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:15790">
P2: When intermediate output is enabled or a transition journal is requested, this predicate still admits the op to Vulkan even though the shader leaves those regions unwritten. Require `op_params[0] == 1` and `op_params[3] == 0`, or add shader support for those output regions.</violation>
</file>

<file name="server/deps/llama.cpp/ggml/src/ggml.c">

<violation number="1" location="server/deps/llama.cpp/ggml/src/ggml.c:5770">
P2: When `max_kv_seq_len` is near `INT32_MAX` and `block_size` is greater than one, this assertion admits the value, but the paged-attention launcher overflows while rounding `live_tokens` to blocks. Bound the launch bound including the rounding term, or perform that calculation in 64-bit before launching.</violation>
</file>

<file name="server/test/test_draft_swa_multilane.cpp">

<violation number="1" location="server/test/test_draft_swa_multilane.cpp:157">
P2: The test always fails before running its intended check: it passes the never-initialized `DraftFeatureMirror unused_ring` (target_feat=nullptr, cap=0) to `draft_kv_begin_step`, and every committed lane has n_new>0 so `copy_feature_ring_range_to_tensor` is called on the empty ring and returns false, making `begin_step` fail with "[draft-kv] feature copy failed" (or the bulk-append equivalent). The lane mask check, single-lane forward, and single-vs-packed comparison are therefore all unreachable. Initialize and populate the feature ring (draft_feature_mirror_init + fill committed rows) before the lane loop, matching how the production caller syncs the ring into begin_step.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Graffioh]

@cubic-dev-ai review this PR changes related to qwen 3.8 27b DFlash 2 concurrency implementation

[cubic-dev-ai]

@cubic-dev-ai review this PR changes related to qwen 3.8 27b DFlash 2 concurrency implementation

@Graffioh I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 62 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 8 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>