feat(ds4): add monolithic Strix Halo concurrent serving

[Graffioh]

Scope

This rebuilds the concurrency work for DeepSeek4 only. Qwen source and GGML kernels are unchanged.

It supersedes the experimental direction in #594, #595, #596 and Graffioh#16, while incorporating the MoE concurrency discussion. The supported topology is deliberately narrow: one local HIP gfx1151 Strix Halo with the complete model and all experts resident.

Implementation

• Adds a model-neutral continuous scheduler and SeqEngine boundary with admission, cancellation, fair prompt progress, slow-client isolation, and reusable slot lifecycle handling.
• Extends the shared paged allocator for reservations and rolling decode headroom.
• Adds DeepSeek4 persistent paged raw MLA, compressed MLA, and indexer state using 128-token model pages.
• Lowers up to 16 independent lanes into an exact gathered DeepSeek4 graph. Live decode rows share a weight pass; selected prompt rows advance one exact token per sequence in that same pass.
• Keeps sampling history and device block-table state isolated per reusable sequence slot.
• Auto-enables paged mode for --max-concurrency 2..16 and exposes optional physical-pool sizing through --kv-pool-tokens.

Guardrails

This path fails closed for CUDA, non-gfx1151 HIP devices, layer or remote target splits, DFLASH_DS4_MOE_TP, drafts/DSpark, DDTree, PFlash/KVFlash, fused decode, approximate prefill, windowed attention, and prefix-cache parking. It requires the full model to remain resident on the single Strix Halo.

Strix Halo concurrency screening

Model GGUF: DeepSeek-V4-Flash-ROCMFP2-STRIX.gguf
Device: one local HIP gfx1151 Strix Halo (Radeon 8060S), with the full model and experts resident.
Protocol: one fresh server and same-concurrency warm-up per level; short ragged, disjoint prompts; exact prefill; greedy streaming; 256 forced completion tokens per request; one repeat.

Concurrent requests	token/s
1	10.50 tok/s
8	25.88 tok/s
16	24.43 tok/s

No llama.cpp comparison is stated yet: the available llama-server rejects this ROCmFP GGUF at load time (invalid ggml type 101). A compatible llama.cpp/DeepSeek4 ROCmFP build (or a compatible GGUF) is required before publishing paired DS4-vs-llama results. This is a one-repeat screening, not a performance claim; use five paired repeats once that baseline is available.

Validation

• Built the HIP gfx1151 server and all focused concurrency targets, including test_deepseek4_unit.
• 358 selected scheduler, server, feature-gate, slot-manager, contract, page-layout, and paged-cache checks pass.
• Paged allocator suite: 13/13 pass.
• Model-backed DeepSeek4 screening completed at C=1, C=4, C=8, and C=16 (29/29 requests; 7,424/7,424 completion tokens; zero failures).
• Final publication commit is byte-identical to the validated worktree.
• Qwen directories, Qwen tests, and GGML kernels have no diff from upstream.

[Graffioh]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

3 issues found across 25 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/deepseek4/deepseek4_graph.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_graph.cpp:7249">
P2: For long-running ratio-4 lanes, crossing each compression boundary rebuilds the whole-model graph instead of replaying it, causing recurring latency spikes and reducing serving throughput. Use a capacity-padded compressed-history topology with runtime visibility masking, or otherwise keep the graph shape stable across row-count changes.</violation>
</file>

<file name="server/src/common/concurrency/seq_slot_manager.cpp">

<violation number="1" location="server/src/common/concurrency/seq_slot_manager.cpp:175">
P2: When a stochastic request uses `seed=0`, this branch replaces the requested deterministic stream with `std::random_device`, so repeated requests cannot reproduce. Use `needs_logit_processing()` alone as the seed gate and seed the RNG with zero.</violation>
</file>

<file name="server/src/deepseek4/deepseek4_backend.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_backend.cpp:527">
P2: When hybrid placement uses ratio-4 layers, `estimate_ds4_cache_bytes` underestimates the indexer compressor state by one factor of two. Charge the actual `2 * index_dim` state width so the expert budget does not leave the subsequent cache allocation without enough VRAM.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: For long-running ratio-4 lanes, crossing each compression boundary rebuilds the whole-model graph instead of replaying it, causing recurring latency spikes and reducing serving throughput. Use a capacity-padded compressed-history topology with runtime visibility masking, or otherwise keep the graph shape stable across row-count changes.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_graph.cpp, line 7249:

<comment>For long-running ratio-4 lanes, crossing each compression boundary rebuilds the whole-model graph instead of replaying it, causing recurring latency spikes and reducing serving throughput. Use a capacity-padded compressed-history topology with runtime visibility masking, or otherwise keep the graph shape stable across row-count changes.</comment>

<file context>
@@ -6854,6 +7156,259 @@ static bool initialize_layer_range_cache(
+                cache.plan.physical_blocks, ratio, prepared[(size_t) il])) return false;
+        for (const auto & row : prepared[(size_t) il]) {
+            key.push_back((int64_t) row.raw_history.size());
+            key.push_back((int64_t) row.compressed_history.size());
+            key.push_back(row.slot < 0 ? -1 :
+                (ratio ? row.position % ratio : row.position % DS4_PAGE_TOKENS));
</file context>

[cubic-dev-ai]

P2: When a stochastic request uses seed=0, this branch replaces the requested deterministic stream with std::random_device, so repeated requests cannot reproduce. Use needs_logit_processing() alone as the seed gate and seed the RNG with zero.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/common/concurrency/seq_slot_manager.cpp, line 175:

<comment>When a stochastic request uses `seed=0`, this branch replaces the requested deterministic stream with `std::random_device`, so repeated requests cannot reproduce. Use `needs_logit_processing()` alone as the seed gate and seed the RNG with zero.</comment>

<file context>
@@ -0,0 +1,278 @@
+    s.sample_history = prompt;
+    // Same predicate the engine uses to pick CPU sampling over GPU argmax:
+    // a seed only means anything when the sampler actually draws.
+    if (sampler.needs_logit_processing() && sampler.seed != 0) {
+        s.rng.seed(sampler.seed);
+    } else {
</file context>

[cubic-dev-ai]

P2: When hybrid placement uses ratio-4 layers, estimate_ds4_cache_bytes underestimates the indexer compressor state by one factor of two. Charge the actual 2 * index_dim state width so the expert budget does not leave the subsequent cache allocation without enough VRAM.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_backend.cpp, line 527:

<comment>When hybrid placement uses ratio-4 layers, `estimate_ds4_cache_bytes` underestimates the indexer compressor state by one factor of two. Charge the actual `2 * index_dim` state width so the expert budget does not leave the subsequent cache allocation without enough VRAM.</comment>

<file context>
@@ -503,20 +515,19 @@ static uint64_t estimate_ds4_cache_bytes(const DeepSeek4Weights & w, int max_ctx
+            // only in fixed-size state scratch and does not scale with context.
+            const size_t index_dim = (size_t) w.n_indexer_head_dim;
+            total_bytes += comp_cap * index_dim * sizeof(uint16_t);
+            total_bytes += state_rows * index_dim * sizeof(float) * 2;
+            total_bytes += (size_t) 2 * 2 * ratio *
+                           (size_t) w.n_indexer_head * index_dim *
</file context>

Suggested change

	total_bytes += state_rows * index_dim * sizeof(float) * 2;
	total_bytes += state_rows * (2 * index_dim) * sizeof(float) * 2;