perf(gfx11): optimize Qwen DFlash speculative verify

crates/hip-bridge/map.md

@@ -23,21 +23,21 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 | File | Lines | Public items | Tests |
 |---|---:|---:|---:|
-| [`src/error.rs`](src/error.rs) | 70 | 8 | 0 |
-| [`src/ffi.rs`](src/ffi.rs) | 1,772 | 96 | 0 |
+| [`src/error.rs`](src/error.rs) | 132 | 10 | 2 |
+| [`src/ffi.rs`](src/ffi.rs) | 1,799 | 98 | 0 |
 | [`src/kernarg.rs`](src/kernarg.rs) | 178 | 14 | 3 |
-| [`src/lib.rs`](src/lib.rs) | 181 | 18 | 2 |
+| [`src/lib.rs`](src/lib.rs) | 222 | 19 | 2 |
 | [`src/rccl.rs`](src/rccl.rs) | 470 | 18 | 0 |
 | [`src/rocblas.rs`](src/rocblas.rs) | 744 | 17 | 3 |
 | [`src/rocsolver.rs`](src/rocsolver.rs) | 374 | 12 | 3 |
-| [`src/vmm.rs`](src/vmm.rs) | 566 | 12 | 4 |
+| [`src/vmm.rs`](src/vmm.rs) | 567 | 12 | 4 |
 
 ### Public API surface
 
-- [`src/error.rs`](src/error.rs): `HipErrorCode`, `HipResult`, `HIP_ERROR_INVALID_IMAGE`, `HIP_ERROR_PEER_ACCESS_UNSUPPORTED`, `HIP_ERROR_PEER_ACCESS_ALREADY_ENABLED`, `HIP_ERROR_PEER_ACCESS_NOT_ENABLED`, `HipError`, `new`
-- [`src/ffi.rs`](src/ffi.rs): `launch_counters`, `record`, `record_bytes`, `time_ns`, `count`, `bytes`, `reset`, `HipMemGenericAllocationHandle`, `HIP_MEM_LOCATION_TYPE_DEVICE`, `HIP_MEM_ALLOCATION_TYPE_PINNED`, `HIP_MEM_ACCESS_FLAGS_PROT_READ_WRITE`, `HIP_MEM_ALLOCATION_GRANULARITY_MINIMUM`, +84 more
+- [`src/error.rs`](src/error.rs): `HipErrorCode`, `HipResult`, `HIP_ERROR_INVALID_IMAGE`, `HIP_ERROR_PEER_ACCESS_UNSUPPORTED`, `HIP_ERROR_PEER_ACCESS_ALREADY_ENABLED`, `HIP_ERROR_PEER_ACCESS_NOT_ENABLED`, `LaunchContext`, `HipError`, `new`, `with_kernel`
+- [`src/ffi.rs`](src/ffi.rs): `launch_counters`, `record`, `record_bytes`, `time_ns`, `count`, `bytes`, `reset`, `HipMemGenericAllocationHandle`, `HIP_ERROR_NOT_READY`, `HIP_MEM_LOCATION_TYPE_DEVICE`, `HIP_MEM_ALLOCATION_TYPE_PINNED`, `HIP_MEM_ACCESS_FLAGS_PROT_READ_WRITE`, +86 more
 - [`src/kernarg.rs`](src/kernarg.rs): `KernargBlob`, `new`, `with_capacity`, `len`, `is_empty`, `push_ptr`, `push_u32`, `push_i32`, `push_f32`, `push_u64`, `pad_to`, `as_mut_slice`, +2 more
-- [`src/lib.rs`](src/lib.rs): `error`, `ffi`, `kernarg`, `rccl`, `rocblas`, `rocsolver`, `vmm`, `MemcpyKind`, `MemoryType`, `from_raw`, `DeviceBuffer`, `as_ptr`, +6 more
+- [`src/lib.rs`](src/lib.rs): `error`, `ffi`, `kernarg`, `rccl`, `rocblas`, `rocsolver`, `vmm`, `MemcpyKind`, `MemoryType`, `from_raw`, `DeviceBuffer`, `as_ptr`, +7 more
 - [`src/rccl.rs`](src/rccl.rs): `NCCL_SUCCESS`, `RcclError`, `RcclResult`, `RcclDataType`, `RcclRedOp`, `RcclComms`, `init_all`, `len`, `is_empty`, `version`, `group_start`, `group_end`, +6 more
 - [`src/rocblas.rs`](src/rocblas.rs): `RocblasError`, `RocblasResult`, `ROCBLAS_STATUS_SUCCESS`, `ROCBLAS_STATUS_INVALID_VALUE`, `RocblasOperation`, `RocblasDatatype`, `RocblasGemmAlgo`, `Rocblas`, `load`, `set_stream`, `handle`, `has_dgemm`, +5 more
 - [`src/rocsolver.rs`](src/rocsolver.rs): `ROCSOLVER_STATUS_SUCCESS`, `RocblasFill`, `RocblasDiagonal`, `RocsolverError`, `RocsolverResult`, `Rocsolver`, `load`, `load_from_handle`, `has_dpotri`, `dpotrf`, `dtrtri`, `dpotri`
@@ -56,6 +56,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 ### Totals
 
-- 8 modules · 4,355 lines · 195 public items · 15 tests · 8 examples
+- 8 modules · 4,486 lines · 200 public items · 17 tests · 8 examples
 
 <!-- crate-map:generated:end -->

crates/hipfire-arch-gemma4/map.md

@@ -49,7 +49,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 ### Dependencies (from `Cargo.toml`)
 
-- path: `hip-bridge`, `hipfire-dispatch`, `hipfire-runtime`, `rdna-compute`
+- path: `hip-bridge`, `hipfire-config`, `hipfire-dispatch`, `hipfire-runtime`, `rdna-compute`
 - external: `serde`, `serde_json`
 - dev: —
 - build: —

crates/hipfire-arch-muse-glimmer/map.md

@@ -43,7 +43,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 ### Dependencies (from `Cargo.toml`)
 
-- path: `hip-bridge`, `hipfire-dispatch`, `hipfire-runtime`, `rdna-compute`
+- path: `hip-bridge`, `hipfire-config`, `hipfire-dispatch`, `hipfire-runtime`, `rdna-compute`
 - external: `serde`, `serde_json`
 - dev: —
 - build: —

crates/hipfire-arch-qwen35-vl/map.md

@@ -26,7 +26,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 | [`src/image.rs`](src/image.rs) | 423 | 4 | 3 |
 | [`src/lib.rs`](src/lib.rs) | 40 | 4 | 0 |
 | [`src/mrope.rs`](src/mrope.rs) | 103 | 5 | 0 |
-| [`src/qwen35_vl.rs`](src/qwen35_vl.rs) | 1,241 | 9 | 11 |
+| [`src/qwen35_vl.rs`](src/qwen35_vl.rs) | 1,238 | 9 | 11 |
 
 ### Public API surface
 
@@ -38,7 +38,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 ### Dependencies (from `Cargo.toml`)
 
-- path: `hip-bridge`, `hipfire-runtime`, `rdna-compute`
+- path: `hip-bridge`, `hipfire-config`, `hipfire-runtime`, `rdna-compute`
 - external: `image`, `serde`, `serde_json`
 - dev: —
 - build: —
@@ -49,6 +49,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 ### Totals
 
-- 5 modules · 1,899 lines · 23 public items · 41 tests · 0 examples
+- 5 modules · 1,896 lines · 23 public items · 41 tests · 0 examples
 
 <!-- crate-map:generated:end -->

crates/hipfire-arch-qwen35/map.md

@@ -25,7 +25,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 | [`src/arch.rs`](src/arch.rs) | 94 | 1 | 0 |
 | [`src/arch_model.rs`](src/arch_model.rs) | 101 | 0 | 0 |
 | [`src/carrier.rs`](src/carrier.rs) | 784 | 3 | 7 |
-| [`src/dflash_spec.rs`](src/dflash_spec.rs) | 1,451 | 11 | 16 |
+| [`src/dflash_spec.rs`](src/dflash_spec.rs) | 1,540 | 11 | 16 |
 | [`src/dflash_verify_pm4.rs`](src/dflash_verify_pm4.rs) | 739 | 35 | 9 |
 | [`src/forward_slots.rs`](src/forward_slots.rs) | 3,154 | 14 | 3 |
 | [`src/grammar_config.rs`](src/grammar_config.rs) | 215 | 2 | 4 |
@@ -50,7 +50,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 | [`src/slot_batch.rs`](src/slot_batch.rs) | 123 | 4 | 6 |
 | [`src/spec_emit.rs`](src/spec_emit.rs) | 908 | 4 | 12 |
 | [`src/spec_impl.rs`](src/spec_impl.rs) | 643 | 1 | 0 |
-| [`src/speculative.rs`](src/speculative.rs) | 7,743 | 69 | 13 |
+| [`src/speculative.rs`](src/speculative.rs) | 7,748 | 69 | 13 |
 
 ### Public API surface
 
@@ -97,6 +97,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 ### Totals
 
-- 29 modules · 57,281 lines · 436 public items · 189 tests · 4 examples
+- 29 modules · 57,375 lines · 436 public items · 189 tests · 4 examples
 
 <!-- crate-map:generated:end -->

crates/hipfire-arch-qwen35/src/dflash_spec.rs

@@ -19,11 +19,11 @@ use crate::speculative::{
     spec_step_dflash, xorshift_next_unit, DdtreeScratch, DeltaNetSnapshot, GdnTape,
     HiddenStateRingBuffer, ModelSlot, SpecStepResult, VerifyScratch,
 };
-use hipfire_runtime::dflash::{DflashConfig, DflashScratch, DflashWeights};
+use hipfire_runtime::dflash::{DflashConfig, DflashScratch, DflashWeights, TargetHiddenLogMark};
 use hipfire_runtime::hfq::HfqFile;
 use hipfire_runtime::spec::{
-    request_rng_state, EvictRetain, PrefillOutcome, SpecGrammar, SpecRequestConfig, SpecStep,
-    SpecTarget, Speculator,
+    request_rng_state, terminal_prefix_replay, EvictRetain, PrefillOutcome, SpecGrammar,
+    SpecRequestConfig, SpecStep, SpecTarget, Speculator,
 };
 use rdna_compute::Gpu;
 use std::path::Path;
@@ -550,6 +550,14 @@ pub struct DflashSpeculator {
     resume_enabled: bool,
     ck_interval: usize,
     ck_cap: usize,
+    last_window: Option<DflashWindowMark>,
+}
+
+#[derive(Clone, Copy, Debug)]
+struct DflashWindowMark {
+    position: usize,
+    seed: u32,
+    target_hidden: TargetHiddenLogMark,
 }
 
 impl DflashSpeculator {
@@ -574,6 +582,7 @@ impl DflashSpeculator {
             resume_enabled,
             ck_interval,
             ck_cap,
+            last_window: None,
         }
     }
 
@@ -604,6 +613,7 @@ impl Speculator for DflashSpeculator {
         resume_from: Option<usize>,
         abort: &dyn Fn() -> bool,
     ) -> Result<PrefillOutcome, String> {
+        self.last_window = None;
         let slot = target
             .as_any_mut()
             .downcast_mut::<ModelSlot>()
@@ -852,6 +862,12 @@ impl Speculator for DflashSpeculator {
         // accepted drafts + bonus = emit; max accepted drafts = max_emit - 1.
         let max_accept = Some(max_emit.saturating_sub(1));
 
+        let window_mark = DflashWindowMark {
+            position,
+            seed,
+            target_hidden: self.df.draft_scratch.thlog.mark(),
+        };
+
         // Two-way dispatch: DDTree-batched (SWOR) when a tree is configured
         // (never for DFlash2 selector — load refused construction), else
         // chain-mode DFlash. Selector chain uses sparse-q rejection at temp>0.
@@ -935,14 +951,17 @@ impl Speculator for DflashSpeculator {
             )
         };
 
-        result
+        let lowered = result
             .map(lower_qwen35)
             // Defense only — accept stage already committed ≤ max_emit.
             .map(|s| s.cap_emit(max_emit))
-            .map_err(|e| e.to_string())
+            .map_err(|e| e.to_string());
+        self.last_window = lowered.as_ref().ok().map(|_| window_mark);
+        lowered
     }
 
     fn on_evict(&mut self, gpu: &mut Gpu, retain: &EvictRetain) -> Result<(), String> {
+        self.last_window = None;
         // Compact the drafter's cached target-hidden rows to match the target KV
         // after the FlashCASK eviction the daemon already applied to the target.
         let ne = self.df.draft_config.num_extract();
@@ -963,12 +982,81 @@ impl Speculator for DflashSpeculator {
         // divergent-render checkpoint ring (the target KV/recurrent reset is the
         // daemon's job — it owns the bundle).
         self.df.draft_scratch.reset_upload_tracking();
+        self.last_window = None;
         for (_, snap) in self.checkpoints.drain(..) {
             snap.free_gpu(gpu);
         }
         Ok(())
     }
 
+    fn repair_terminal_prefix(
+        &mut self,
+        gpu: &mut Gpu,
+        target: &mut dyn SpecTarget,
+        window_start: usize,
+        window_seed: u32,
+        consumed: &[u32],
+    ) -> Result<bool, String> {
+        let mark = self
+            .last_window
+            .take()
+            .ok_or("DflashSpeculator: no completed window available for terminal repair")?;
+        if mark.position != window_start || mark.seed != window_seed {
+            return Err(format!(
+                "DflashSpeculator: terminal repair window mismatch (saved pos={} seed={}, requested pos={} seed={})",
+                mark.position, mark.seed, window_start, window_seed
+            ));
+        }
+
+        let slot = target
+            .as_any_mut()
+            .downcast_mut::<ModelSlot>()
+            .ok_or("DflashSpeculator: target is not a Qwen3.5 ModelSlot")?;
+        self.df
+            .target_snap
+            .restore_to(&mut slot.dn_state, gpu)
+            .map_err(|e| format!("DeltaNetSnapshot::restore_to: {e}"))?;
+        self.df.draft_scratch.thlog.restore(mark.target_hidden)?;
+
+        // Before the ordinary terminal flush, target state must include the
+        // old pending seed and every consumed token except the new pending
+        // terminal token. For consumed=[] there is nothing to replay.
+        let replay = terminal_prefix_replay(window_seed, consumed);
+        if replay.is_empty() {
+            return Ok(true);
+        }
+
+        let aborted = seed_target_hidden_suffix_abortable(
+            gpu,
+            slot,
+            &mut self.df.hidden_rb,
+            &replay,
+            window_start,
+            &|| false,
+            self.resume_enabled.then_some(&mut self.checkpoints),
+            self.ck_interval,
+            self.ck_cap,
+        )
+        .map_err(|e| e.to_string())?;
+        debug_assert!(!aborted, "terminal repair uses a non-aborting callback");
+        scatter_hidden_block_to_interleaved(
+            gpu,
+            &self.df.hidden_rb,
+            &self.df.draft_scratch.target_hidden,
+            window_start,
+            replay.len(),
+            replay.len(),
+            self.df.draft_scratch.ctx_modulus(),
+        )
+        .map_err(|e| e.to_string())?;
+        let co = slot.kv_cache_mut().map(|kv| kv.compact_offset).unwrap_or(0) as i32;
+        self.df
+            .draft_scratch
+            .thlog
+            .append_committed(window_start, replay.len(), co);
+        Ok(true)
+    }
+
     fn reset_state_evidence(&self) -> Option<hipfire_runtime::spec::SpecResetEvidence> {
         let th = &self.df.draft_scratch.thlog;
         Some(hipfire_runtime::spec::SpecResetEvidence {
@@ -1028,6 +1116,7 @@ impl Speculator for DflashSpeculator {
         self.sample_top_k = cfg.top_k;
         self.sample_cactus = cfg.cactus_delta;
         self.rng_state = request_rng_state(cfg.rng_seed);
+        self.last_window = None;
     }
 
     fn requires_greedy(&self) -> bool {

crates/hipfire-arch-qwen35/src/speculative.rs

@@ -231,6 +231,11 @@ fn dflash_verify_graph_env_eligible(
     if arch == "gfx1100" && is_mq_v2 {
         return env_value == Some("1");
     }
+    if arch == "gfx1100" && output_dtype == rdna_compute::DType::MQ4G256 {
+        // Direct batched HIP is faster for the measured Qwen MQ4 workload;
+        // keep graph available as an explicit diagnostic opt-in.
+        return env_value == Some("1");
+    }
     true
 }
 

crates/hipfire-config/map.md

@@ -23,13 +23,13 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 | File | Lines | Public items | Tests |
 |---|---:|---:|---:|
 | [`src/bin/hipfire-rocm-resolve.rs`](src/bin/hipfire-rocm-resolve.rs) | 105 | 0 | 0 |
-| [`src/lib.rs`](src/lib.rs) | 5,159 | 79 | 28 |
+| [`src/lib.rs`](src/lib.rs) | 5,226 | 80 | 30 |
 | [`src/rocm.rs`](src/rocm.rs) | 2,460 | 39 | 37 |
 
 ### Public API surface
 
 - [`src/bin/hipfire-rocm-resolve.rs`](src/bin/hipfire-rocm-resolve.rs): —
-- [`src/lib.rs`](src/lib.rs): `rocm`, `CONFIG_SCHEMA_VERSION`, `ConfigError`, `Result`, `ConfigValue`, `DeviceSelector`, `Deepseek4ComputePlacement`, `Deepseek4CompressorCache`, `kind`, `ConfigCategory`, `ConfigScope`, `DefaultValue`, +67 more
+- [`src/lib.rs`](src/lib.rs): `rocm`, `CONFIG_SCHEMA_VERSION`, `ConfigError`, `Result`, `ConfigValue`, `DeviceSelector`, `Deepseek4ComputePlacement`, `Deepseek4CompressorCache`, `kind`, `ConfigCategory`, `ConfigScope`, `DefaultValue`, +68 more
 - [`src/rocm.rs`](src/rocm.rs): `DEVICE_COMPILERS`, `configured_root`, `has_configured_root`, `configured_compiler`, `has_configured_compiler`, `configured_compiler_from`, `is_strict_rocm`, `strict_from`, `strict_from_str`, `CompilerSource`, `ResolvedToolchain`, `version_for_root`, +27 more
 
 ### Dependencies (from `Cargo.toml`)
@@ -41,10 +41,10 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 ### Reverse dependencies
 
-- workspace crates with a path dependency on this crate: `hip-bridge`, `hipfire-arch-cohere2moe`, `hipfire-arch-deepseek4`, `hipfire-arch-dots-ocr`, `hipfire-arch-lfm2moe`, `hipfire-arch-maple`, `hipfire-arch-minimax`, `hipfire-arch-qwen2`, `hipfire-arch-qwen35`, `hipfire-cli`, `hipfire-client`, `hipfire-daemon`, `hipfire-dispatch`, `hipfire-ds4-parent`, `hipfire-engine`, `hipfire-generate`, `hipfire-loader`, `hipfire-pflash`, `hipfire-quantize`, `hipfire-reap`, `hipfire-registry`, `hipfire-runtime`, `hipfire-tui`, `hsa-bridge`, `rdna-compute`, `redline-dispatch`, `redline-rocr`, `saddle-lab`, `saddle-quant`
+- workspace crates with a path dependency on this crate: `hip-bridge`, `hipfire-arch-cohere2moe`, `hipfire-arch-deepseek4`, `hipfire-arch-dots-ocr`, `hipfire-arch-gemma4`, `hipfire-arch-lfm2moe`, `hipfire-arch-maple`, `hipfire-arch-minimax`, `hipfire-arch-muse-glimmer`, `hipfire-arch-qwen2`, `hipfire-arch-qwen35`, `hipfire-arch-qwen35-vl`, `hipfire-cli`, `hipfire-client`, `hipfire-daemon`, `hipfire-dispatch`, `hipfire-ds4-parent`, `hipfire-engine`, `hipfire-generate`, `hipfire-loader`, `hipfire-pflash`, `hipfire-quantize`, `hipfire-reap`, `hipfire-registry`, `hipfire-runtime`, `hipfire-tui`, `hsa-bridge`, `rdna-compute`, `redline-dispatch`, `redline-rocr`, `saddle-lab`, `saddle-quant`
 
 ### Totals
 
-- 3 modules · 7,724 lines · 118 public items · 65 tests · 0 examples
+- 3 modules · 7,791 lines · 119 public items · 67 tests · 0 examples
 
 <!-- crate-map:generated:end -->

crates/hipfire-dispatch/map.md

@@ -24,7 +24,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 |---|---:|---:|---:|
 | [`src/context.rs`](src/context.rs) | 67 | 5 | 0 |
 | [`src/coverage_tests.rs`](src/coverage_tests.rs) | 1,841 | 0 | 21 |
-| [`src/families/attention.rs`](src/families/attention.rs) | 2,239 | 9 | 10 |
+| [`src/families/attention.rs`](src/families/attention.rs) | 2,268 | 9 | 10 |
 | [`src/families/fused_qkv.rs`](src/families/fused_qkv.rs) | 1,585 | 8 | 1 |
 | [`src/families/gemm.rs`](src/families/gemm.rs) | 649 | 7 | 2 |
 | [`src/families/gemv.rs`](src/families/gemv.rs) | 624 | 20 | 0 |
@@ -40,11 +40,11 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 | [`src/ops/delta_net.rs`](src/ops/delta_net.rs) | 289 | 6 | 0 |
 | [`src/ops/mla.rs`](src/ops/mla.rs) | 329 | 5 | 0 |
 | [`src/ops/mod.rs`](src/ops/mod.rs) | 4 | 2 | 0 |
-| [`src/pipeline/mod.rs`](src/pipeline/mod.rs) | 4,119 | 18 | 16 |
+| [`src/pipeline/mod.rs`](src/pipeline/mod.rs) | 4,125 | 18 | 16 |
 | [`src/pipeline/steps.rs`](src/pipeline/steps.rs) | 1,559 | 5 | 13 |
 | [`src/pipeline/superop.rs`](src/pipeline/superop.rs) | 598 | 17 | 4 |
 | [`src/resource/mod.rs`](src/resource/mod.rs) | 19 | 3 | 0 |
-| [`src/tables/attention_table.rs`](src/tables/attention_table.rs) | 448 | 1 | 0 |
+| [`src/tables/attention_table.rs`](src/tables/attention_table.rs) | 458 | 1 | 0 |
 | [`src/tables/fused_qkv_table.rs`](src/tables/fused_qkv_table.rs) | 240 | 1 | 0 |
 | [`src/tables/gemm_table.rs`](src/tables/gemm_table.rs) | 512 | 1 | 0 |
 | [`src/tables/gemv_table.rs`](src/tables/gemv_table.rs) | 189 | 1 | 0 |
@@ -53,7 +53,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 | [`src/tables/rotation_table.rs`](src/tables/rotation_table.rs) | 42 | 1 | 0 |
 | [`src/tests.rs`](src/tests.rs) | 2,524 | 0 | 114 |
 | [`src/traits.rs`](src/traits.rs) | 7 | 1 | 0 |
-| [`src/types.rs`](src/types.rs) | 981 | 23 | 2 |
+| [`src/types.rs`](src/types.rs) | 982 | 23 | 2 |
 
 ### Public API surface
 
@@ -103,6 +103,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 ### Totals
 
-- 32 modules · 22,291 lines · 223 public items · 238 tests · 0 examples
+- 32 modules · 22,337 lines · 223 public items · 238 tests · 0 examples
 
 <!-- crate-map:generated:end -->