perf(ds4): batch exact prefill through width-4 vector kernels

[dusterbloom]

Summary

• batch exact DeepSeek4 prefill in compressor-safe groups of at most four tokens
• use the existing quantized matrix-vector path only for those exact width-2-to-4 feed-forward graphs
• keep speculative verification, approximate prefill, and unsupported hybrid paths unchanged
• add no option or environment flag

Result

Ryzen AI Max+ 395 / gfx1151, ROCm 7.2.2, exact prefill, no drafter or caches.

Fresh current-main smoke (f5475131, one warmup then three interleaved runs):

Prompt	Main median	PR median	Lower time
64 tokens	3341.6 ms	2776.9 ms	16.90%
128 tokens	6710.7 ms	5602.6 ms	16.51%

The frozen five-run qualification measured 17.72% and 16.62% lower total prefill time respectively. The expert feed-forward phase was about 37% lower. Its 95% gain intervals were [17.13%, 18.33%] and [16.21%, 17.37%].

Correctness

The frozen Q=1 versus Q=4 qualification was byte-identical across ordered route IDs, normalized route weights, attention and feed-forward recurrent state, every-position F32 logits, and committed tokens.

After rebasing on current main, the server built with -j4 and these focused tests passed 4/4:

• test_deepseek4_hc_cuda
• test_rocmfpx_mmq
• deepseek4_mmid_grouped_cuda
• deepseek4_unit

All current-main benchmark responses emitted the same first token.

Scope

Three files, 21 insertions, 6 deletions. This PR claims the exact all-hot M64/M128 prefill result only; long-context and speculative decode are separate qualifications.

[cubic-dev-ai]

2 issues found across 3 files

_{You’re at about 90% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/deepseek4/deepseek4_graph.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_graph.cpp:6976">
P2: When `LUCE_MMVQ_MAX_NCOLS` is set below 4, this scope ignores the operator's explicit dispatch threshold and forces MMVQ for every quantized matmul in the exact batch. Preserve an explicit environment threshold, or scope the selection to the qualified feed-forward operations.

(Based on your team's feedback about respecting MMVQ dispatch overrides.)</violation>
</file>

<file name="server/deps/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu">

<violation number="1" location="server/deps/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2792">
P2: When the DS4 fused MMQ-pair path is skipped because ncols<=override, the fallback only lands on the quantized matrix-vector path if the grouped MMVQ batch ceiling (get_mmvq_mmid_max_batch) is >= ncols for the qtype. For an ids/grouped DS4 feed-forward, ncols is src1->ne[2], but the override ceiling is designed for plain mul_mat's src1->ne[1]; the two limits are not the same constant. If a qtype's mmid MMVQ ceiling is below ncols, this gate silently drops the fused path and the node falls back to the slower unfused MMQ/dequant path instead of the intended MMVQ win.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When LUCE_MMVQ_MAX_NCOLS is set below 4, this scope ignores the operator's explicit dispatch threshold and forces MMVQ for every quantized matmul in the exact batch. Preserve an explicit environment threshold, or scope the selection to the qualified feed-forward operations.

(Based on your team's feedback about respecting MMVQ dispatch overrides.)

View Feedback

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_graph.cpp, line 6976:

<comment>When `LUCE_MMVQ_MAX_NCOLS` is set below 4, this scope ignores the operator's explicit dispatch threshold and forces MMVQ for every quantized matmul in the exact batch. Preserve an explicit environment threshold, or scope the selection to the qualified feed-forward operations.

(Based on your team's feedback about respecting MMVQ dispatch overrides.) </comment>

<file context>
@@ -6965,7 +6965,17 @@ bool deepseek4_step_layer_range(
+        exact_prefill_band ? 4 : n_tokens);
+    const bool exact_multi_token_band =
+        exact_prefill_band && n_tokens > 1 && n_tokens <= 4;
+    ScopedCudaGraphOverrides exact_mmvq_scope(
+        /*disable_graphs=*/false,
+        /*mmvq_max_ncols=*/exact_multi_token_band ? 4 : 0);
</file context>

Suggested change

	ScopedCudaGraphOverrides exact_mmvq_scope(
	/*disable_graphs=*/false,
	/*mmvq_max_ncols=*/exact_multi_token_band ? 4 : 0);
	if (first_chunk > 0 && first_chunk < n_tokens &&
	const int exact_mmvq_max_ncols =
	exact_multi_token_band && std::getenv("LUCE_MMVQ_MAX_NCOLS") == nullptr
	? 4 : 0;
	ScopedCudaGraphOverrides exact_mmvq_scope(
	/*disable_graphs=*/false,
	/*mmvq_max_ncols=*/exact_mmvq_max_ncols);

[cubic-dev-ai]

P2: When the DS4 fused MMQ-pair path is skipped because ncols<=override, the fallback only lands on the quantized matrix-vector path if the grouped MMVQ batch ceiling (get_mmvq_mmid_max_batch) is >= ncols for the qtype. For an ids/grouped DS4 feed-forward, ncols is src1->ne[2], but the override ceiling is designed for plain mul_mat's src1->ne[1]; the two limits are not the same constant. If a qtype's mmid MMVQ ceiling is below ncols, this gate silently drops the fused path and the node falls back to the slower unfused MMQ/dequant path instead of the intended MMVQ win.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/deps/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu, line 2792:

<comment>When the DS4 fused MMQ-pair path is skipped because ncols<=override, the fallback only lands on the quantized matrix-vector path if the grouped MMVQ batch ceiling (get_mmvq_mmid_max_batch) is >= ncols for the qtype. For an ids/grouped DS4 feed-forward, ncols is src1->ne[2], but the override ceiling is designed for plain mul_mat's src1->ne[1]; the two limits are not the same constant. If a qtype's mmid MMVQ ceiling is below ncols, this gate silently drops the fused path and the node falls back to the slower unfused MMQ/dequant path instead of the intended MMVQ win.</comment>

<file context>
@@ -2787,7 +2787,9 @@ static bool ggml_cuda_try_fuse_mul_mat_glu(
-                ids ? src0->ne[2] : /*n_experts=*/0)) {
+                ids ? src0->ne[2] : /*n_experts=*/0) &&
+            !(ggml_cuda_mmvq_max_ncols_override > 0 &&
+              ncols <= ggml_cuda_mmvq_max_ncols_override)) {
             ggml_cuda_mul_mat_q_pair(
                 ctx, up->src[0], gate->src[0], src1, ids, up, gate);
</file context>