diff --git a/crates/hip-bridge/map.md b/crates/hip-bridge/map.md index 7b800c929..47c7d2bd3 100644 --- a/crates/hip-bridge/map.md +++ b/crates/hip-bridge/map.md @@ -23,21 +23,21 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | File | Lines | Public items | Tests | |---|---:|---:|---:| -| [`src/error.rs`](src/error.rs) | 70 | 8 | 0 | -| [`src/ffi.rs`](src/ffi.rs) | 1,772 | 96 | 0 | +| [`src/error.rs`](src/error.rs) | 132 | 10 | 2 | +| [`src/ffi.rs`](src/ffi.rs) | 1,799 | 98 | 0 | | [`src/kernarg.rs`](src/kernarg.rs) | 178 | 14 | 3 | -| [`src/lib.rs`](src/lib.rs) | 181 | 18 | 2 | +| [`src/lib.rs`](src/lib.rs) | 222 | 19 | 2 | | [`src/rccl.rs`](src/rccl.rs) | 470 | 18 | 0 | | [`src/rocblas.rs`](src/rocblas.rs) | 744 | 17 | 3 | | [`src/rocsolver.rs`](src/rocsolver.rs) | 374 | 12 | 3 | -| [`src/vmm.rs`](src/vmm.rs) | 566 | 12 | 4 | +| [`src/vmm.rs`](src/vmm.rs) | 567 | 12 | 4 | ### Public API surface -- [`src/error.rs`](src/error.rs): `HipErrorCode`, `HipResult`, `HIP_ERROR_INVALID_IMAGE`, `HIP_ERROR_PEER_ACCESS_UNSUPPORTED`, `HIP_ERROR_PEER_ACCESS_ALREADY_ENABLED`, `HIP_ERROR_PEER_ACCESS_NOT_ENABLED`, `HipError`, `new` -- [`src/ffi.rs`](src/ffi.rs): `launch_counters`, `record`, `record_bytes`, `time_ns`, `count`, `bytes`, `reset`, `HipMemGenericAllocationHandle`, `HIP_MEM_LOCATION_TYPE_DEVICE`, `HIP_MEM_ALLOCATION_TYPE_PINNED`, `HIP_MEM_ACCESS_FLAGS_PROT_READ_WRITE`, `HIP_MEM_ALLOCATION_GRANULARITY_MINIMUM`, +84 more +- [`src/error.rs`](src/error.rs): `HipErrorCode`, `HipResult`, `HIP_ERROR_INVALID_IMAGE`, `HIP_ERROR_PEER_ACCESS_UNSUPPORTED`, `HIP_ERROR_PEER_ACCESS_ALREADY_ENABLED`, `HIP_ERROR_PEER_ACCESS_NOT_ENABLED`, `LaunchContext`, `HipError`, `new`, `with_kernel` +- [`src/ffi.rs`](src/ffi.rs): `launch_counters`, `record`, `record_bytes`, `time_ns`, `count`, `bytes`, `reset`, `HipMemGenericAllocationHandle`, `HIP_ERROR_NOT_READY`, `HIP_MEM_LOCATION_TYPE_DEVICE`, `HIP_MEM_ALLOCATION_TYPE_PINNED`, `HIP_MEM_ACCESS_FLAGS_PROT_READ_WRITE`, +86 more - [`src/kernarg.rs`](src/kernarg.rs): `KernargBlob`, `new`, `with_capacity`, `len`, `is_empty`, `push_ptr`, `push_u32`, `push_i32`, `push_f32`, `push_u64`, `pad_to`, `as_mut_slice`, +2 more -- [`src/lib.rs`](src/lib.rs): `error`, `ffi`, `kernarg`, `rccl`, `rocblas`, `rocsolver`, `vmm`, `MemcpyKind`, `MemoryType`, `from_raw`, `DeviceBuffer`, `as_ptr`, +6 more +- [`src/lib.rs`](src/lib.rs): `error`, `ffi`, `kernarg`, `rccl`, `rocblas`, `rocsolver`, `vmm`, `MemcpyKind`, `MemoryType`, `from_raw`, `DeviceBuffer`, `as_ptr`, +7 more - [`src/rccl.rs`](src/rccl.rs): `NCCL_SUCCESS`, `RcclError`, `RcclResult`, `RcclDataType`, `RcclRedOp`, `RcclComms`, `init_all`, `len`, `is_empty`, `version`, `group_start`, `group_end`, +6 more - [`src/rocblas.rs`](src/rocblas.rs): `RocblasError`, `RocblasResult`, `ROCBLAS_STATUS_SUCCESS`, `ROCBLAS_STATUS_INVALID_VALUE`, `RocblasOperation`, `RocblasDatatype`, `RocblasGemmAlgo`, `Rocblas`, `load`, `set_stream`, `handle`, `has_dgemm`, +5 more - [`src/rocsolver.rs`](src/rocsolver.rs): `ROCSOLVER_STATUS_SUCCESS`, `RocblasFill`, `RocblasDiagonal`, `RocsolverError`, `RocsolverResult`, `Rocsolver`, `load`, `load_from_handle`, `has_dpotri`, `dpotrf`, `dtrtri`, `dpotri` @@ -56,6 +56,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Totals -- 8 modules · 4,355 lines · 195 public items · 15 tests · 8 examples +- 8 modules · 4,486 lines · 200 public items · 17 tests · 8 examples diff --git a/crates/hipfire-arch-gemma4/map.md b/crates/hipfire-arch-gemma4/map.md index 99e65166a..c2ba41cd8 100644 --- a/crates/hipfire-arch-gemma4/map.md +++ b/crates/hipfire-arch-gemma4/map.md @@ -49,7 +49,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Dependencies (from `Cargo.toml`) -- path: `hip-bridge`, `hipfire-dispatch`, `hipfire-runtime`, `rdna-compute` +- path: `hip-bridge`, `hipfire-config`, `hipfire-dispatch`, `hipfire-runtime`, `rdna-compute` - external: `serde`, `serde_json` - dev: — - build: — diff --git a/crates/hipfire-arch-muse-glimmer/map.md b/crates/hipfire-arch-muse-glimmer/map.md index 460183fe1..be8540324 100644 --- a/crates/hipfire-arch-muse-glimmer/map.md +++ b/crates/hipfire-arch-muse-glimmer/map.md @@ -43,7 +43,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Dependencies (from `Cargo.toml`) -- path: `hip-bridge`, `hipfire-dispatch`, `hipfire-runtime`, `rdna-compute` +- path: `hip-bridge`, `hipfire-config`, `hipfire-dispatch`, `hipfire-runtime`, `rdna-compute` - external: `serde`, `serde_json` - dev: — - build: — diff --git a/crates/hipfire-arch-qwen35-vl/map.md b/crates/hipfire-arch-qwen35-vl/map.md index 8f53a556b..005da3156 100644 --- a/crates/hipfire-arch-qwen35-vl/map.md +++ b/crates/hipfire-arch-qwen35-vl/map.md @@ -26,7 +26,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | [`src/image.rs`](src/image.rs) | 423 | 4 | 3 | | [`src/lib.rs`](src/lib.rs) | 40 | 4 | 0 | | [`src/mrope.rs`](src/mrope.rs) | 103 | 5 | 0 | -| [`src/qwen35_vl.rs`](src/qwen35_vl.rs) | 1,241 | 9 | 11 | +| [`src/qwen35_vl.rs`](src/qwen35_vl.rs) | 1,238 | 9 | 11 | ### Public API surface @@ -38,7 +38,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Dependencies (from `Cargo.toml`) -- path: `hip-bridge`, `hipfire-runtime`, `rdna-compute` +- path: `hip-bridge`, `hipfire-config`, `hipfire-runtime`, `rdna-compute` - external: `image`, `serde`, `serde_json` - dev: — - build: — @@ -49,6 +49,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Totals -- 5 modules · 1,899 lines · 23 public items · 41 tests · 0 examples +- 5 modules · 1,896 lines · 23 public items · 41 tests · 0 examples diff --git a/crates/hipfire-arch-qwen35/map.md b/crates/hipfire-arch-qwen35/map.md index 94a8f3efd..600c21b16 100644 --- a/crates/hipfire-arch-qwen35/map.md +++ b/crates/hipfire-arch-qwen35/map.md @@ -25,7 +25,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | [`src/arch.rs`](src/arch.rs) | 94 | 1 | 0 | | [`src/arch_model.rs`](src/arch_model.rs) | 101 | 0 | 0 | | [`src/carrier.rs`](src/carrier.rs) | 784 | 3 | 7 | -| [`src/dflash_spec.rs`](src/dflash_spec.rs) | 1,451 | 11 | 16 | +| [`src/dflash_spec.rs`](src/dflash_spec.rs) | 1,540 | 11 | 16 | | [`src/dflash_verify_pm4.rs`](src/dflash_verify_pm4.rs) | 739 | 35 | 9 | | [`src/forward_slots.rs`](src/forward_slots.rs) | 3,154 | 14 | 3 | | [`src/grammar_config.rs`](src/grammar_config.rs) | 215 | 2 | 4 | @@ -50,7 +50,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | [`src/slot_batch.rs`](src/slot_batch.rs) | 123 | 4 | 6 | | [`src/spec_emit.rs`](src/spec_emit.rs) | 908 | 4 | 12 | | [`src/spec_impl.rs`](src/spec_impl.rs) | 643 | 1 | 0 | -| [`src/speculative.rs`](src/speculative.rs) | 7,743 | 69 | 13 | +| [`src/speculative.rs`](src/speculative.rs) | 7,748 | 69 | 13 | ### Public API surface @@ -97,6 +97,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Totals -- 29 modules · 57,281 lines · 436 public items · 189 tests · 4 examples +- 29 modules · 57,375 lines · 436 public items · 189 tests · 4 examples diff --git a/crates/hipfire-arch-qwen35/src/dflash_spec.rs b/crates/hipfire-arch-qwen35/src/dflash_spec.rs index 2c3119000..8b143e697 100644 --- a/crates/hipfire-arch-qwen35/src/dflash_spec.rs +++ b/crates/hipfire-arch-qwen35/src/dflash_spec.rs @@ -19,11 +19,11 @@ use crate::speculative::{ spec_step_dflash, xorshift_next_unit, DdtreeScratch, DeltaNetSnapshot, GdnTape, HiddenStateRingBuffer, ModelSlot, SpecStepResult, VerifyScratch, }; -use hipfire_runtime::dflash::{DflashConfig, DflashScratch, DflashWeights}; +use hipfire_runtime::dflash::{DflashConfig, DflashScratch, DflashWeights, TargetHiddenLogMark}; use hipfire_runtime::hfq::HfqFile; use hipfire_runtime::spec::{ - request_rng_state, EvictRetain, PrefillOutcome, SpecGrammar, SpecRequestConfig, SpecStep, - SpecTarget, Speculator, + request_rng_state, terminal_prefix_replay, EvictRetain, PrefillOutcome, SpecGrammar, + SpecRequestConfig, SpecStep, SpecTarget, Speculator, }; use rdna_compute::Gpu; use std::path::Path; @@ -550,6 +550,14 @@ pub struct DflashSpeculator { resume_enabled: bool, ck_interval: usize, ck_cap: usize, + last_window: Option, +} + +#[derive(Clone, Copy, Debug)] +struct DflashWindowMark { + position: usize, + seed: u32, + target_hidden: TargetHiddenLogMark, } impl DflashSpeculator { @@ -574,6 +582,7 @@ impl DflashSpeculator { resume_enabled, ck_interval, ck_cap, + last_window: None, } } @@ -604,6 +613,7 @@ impl Speculator for DflashSpeculator { resume_from: Option, abort: &dyn Fn() -> bool, ) -> Result { + self.last_window = None; let slot = target .as_any_mut() .downcast_mut::() @@ -852,6 +862,12 @@ impl Speculator for DflashSpeculator { // accepted drafts + bonus = emit; max accepted drafts = max_emit - 1. let max_accept = Some(max_emit.saturating_sub(1)); + let window_mark = DflashWindowMark { + position, + seed, + target_hidden: self.df.draft_scratch.thlog.mark(), + }; + // Two-way dispatch: DDTree-batched (SWOR) when a tree is configured // (never for DFlash2 selector — load refused construction), else // chain-mode DFlash. Selector chain uses sparse-q rejection at temp>0. @@ -935,14 +951,17 @@ impl Speculator for DflashSpeculator { ) }; - result + let lowered = result .map(lower_qwen35) // Defense only — accept stage already committed ≤ max_emit. .map(|s| s.cap_emit(max_emit)) - .map_err(|e| e.to_string()) + .map_err(|e| e.to_string()); + self.last_window = lowered.as_ref().ok().map(|_| window_mark); + lowered } fn on_evict(&mut self, gpu: &mut Gpu, retain: &EvictRetain) -> Result<(), String> { + self.last_window = None; // Compact the drafter's cached target-hidden rows to match the target KV // after the FlashCASK eviction the daemon already applied to the target. let ne = self.df.draft_config.num_extract(); @@ -963,12 +982,81 @@ impl Speculator for DflashSpeculator { // divergent-render checkpoint ring (the target KV/recurrent reset is the // daemon's job — it owns the bundle). self.df.draft_scratch.reset_upload_tracking(); + self.last_window = None; for (_, snap) in self.checkpoints.drain(..) { snap.free_gpu(gpu); } Ok(()) } + fn repair_terminal_prefix( + &mut self, + gpu: &mut Gpu, + target: &mut dyn SpecTarget, + window_start: usize, + window_seed: u32, + consumed: &[u32], + ) -> Result { + let mark = self + .last_window + .take() + .ok_or("DflashSpeculator: no completed window available for terminal repair")?; + if mark.position != window_start || mark.seed != window_seed { + return Err(format!( + "DflashSpeculator: terminal repair window mismatch (saved pos={} seed={}, requested pos={} seed={})", + mark.position, mark.seed, window_start, window_seed + )); + } + + let slot = target + .as_any_mut() + .downcast_mut::() + .ok_or("DflashSpeculator: target is not a Qwen3.5 ModelSlot")?; + self.df + .target_snap + .restore_to(&mut slot.dn_state, gpu) + .map_err(|e| format!("DeltaNetSnapshot::restore_to: {e}"))?; + self.df.draft_scratch.thlog.restore(mark.target_hidden)?; + + // Before the ordinary terminal flush, target state must include the + // old pending seed and every consumed token except the new pending + // terminal token. For consumed=[] there is nothing to replay. + let replay = terminal_prefix_replay(window_seed, consumed); + if replay.is_empty() { + return Ok(true); + } + + let aborted = seed_target_hidden_suffix_abortable( + gpu, + slot, + &mut self.df.hidden_rb, + &replay, + window_start, + &|| false, + self.resume_enabled.then_some(&mut self.checkpoints), + self.ck_interval, + self.ck_cap, + ) + .map_err(|e| e.to_string())?; + debug_assert!(!aborted, "terminal repair uses a non-aborting callback"); + scatter_hidden_block_to_interleaved( + gpu, + &self.df.hidden_rb, + &self.df.draft_scratch.target_hidden, + window_start, + replay.len(), + replay.len(), + self.df.draft_scratch.ctx_modulus(), + ) + .map_err(|e| e.to_string())?; + let co = slot.kv_cache_mut().map(|kv| kv.compact_offset).unwrap_or(0) as i32; + self.df + .draft_scratch + .thlog + .append_committed(window_start, replay.len(), co); + Ok(true) + } + fn reset_state_evidence(&self) -> Option { let th = &self.df.draft_scratch.thlog; Some(hipfire_runtime::spec::SpecResetEvidence { @@ -1028,6 +1116,7 @@ impl Speculator for DflashSpeculator { self.sample_top_k = cfg.top_k; self.sample_cactus = cfg.cactus_delta; self.rng_state = request_rng_state(cfg.rng_seed); + self.last_window = None; } fn requires_greedy(&self) -> bool { diff --git a/crates/hipfire-arch-qwen35/src/speculative.rs b/crates/hipfire-arch-qwen35/src/speculative.rs index a06bb3495..87bf84f17 100644 --- a/crates/hipfire-arch-qwen35/src/speculative.rs +++ b/crates/hipfire-arch-qwen35/src/speculative.rs @@ -231,6 +231,11 @@ fn dflash_verify_graph_env_eligible( if arch == "gfx1100" && is_mq_v2 { return env_value == Some("1"); } + if arch == "gfx1100" && output_dtype == rdna_compute::DType::MQ4G256 { + // Direct batched HIP is faster for the measured Qwen MQ4 workload; + // keep graph available as an explicit diagnostic opt-in. + return env_value == Some("1"); + } true } diff --git a/crates/hipfire-config/map.md b/crates/hipfire-config/map.md index ba071529c..6d3de6ef9 100644 --- a/crates/hipfire-config/map.md +++ b/crates/hipfire-config/map.md @@ -23,13 +23,13 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | File | Lines | Public items | Tests | |---|---:|---:|---:| | [`src/bin/hipfire-rocm-resolve.rs`](src/bin/hipfire-rocm-resolve.rs) | 105 | 0 | 0 | -| [`src/lib.rs`](src/lib.rs) | 5,159 | 79 | 28 | +| [`src/lib.rs`](src/lib.rs) | 5,226 | 80 | 30 | | [`src/rocm.rs`](src/rocm.rs) | 2,460 | 39 | 37 | ### Public API surface - [`src/bin/hipfire-rocm-resolve.rs`](src/bin/hipfire-rocm-resolve.rs): — -- [`src/lib.rs`](src/lib.rs): `rocm`, `CONFIG_SCHEMA_VERSION`, `ConfigError`, `Result`, `ConfigValue`, `DeviceSelector`, `Deepseek4ComputePlacement`, `Deepseek4CompressorCache`, `kind`, `ConfigCategory`, `ConfigScope`, `DefaultValue`, +67 more +- [`src/lib.rs`](src/lib.rs): `rocm`, `CONFIG_SCHEMA_VERSION`, `ConfigError`, `Result`, `ConfigValue`, `DeviceSelector`, `Deepseek4ComputePlacement`, `Deepseek4CompressorCache`, `kind`, `ConfigCategory`, `ConfigScope`, `DefaultValue`, +68 more - [`src/rocm.rs`](src/rocm.rs): `DEVICE_COMPILERS`, `configured_root`, `has_configured_root`, `configured_compiler`, `has_configured_compiler`, `configured_compiler_from`, `is_strict_rocm`, `strict_from`, `strict_from_str`, `CompilerSource`, `ResolvedToolchain`, `version_for_root`, +27 more ### Dependencies (from `Cargo.toml`) @@ -41,10 +41,10 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Reverse dependencies -- workspace crates with a path dependency on this crate: `hip-bridge`, `hipfire-arch-cohere2moe`, `hipfire-arch-deepseek4`, `hipfire-arch-dots-ocr`, `hipfire-arch-lfm2moe`, `hipfire-arch-maple`, `hipfire-arch-minimax`, `hipfire-arch-qwen2`, `hipfire-arch-qwen35`, `hipfire-cli`, `hipfire-client`, `hipfire-daemon`, `hipfire-dispatch`, `hipfire-ds4-parent`, `hipfire-engine`, `hipfire-generate`, `hipfire-loader`, `hipfire-pflash`, `hipfire-quantize`, `hipfire-reap`, `hipfire-registry`, `hipfire-runtime`, `hipfire-tui`, `hsa-bridge`, `rdna-compute`, `redline-dispatch`, `redline-rocr`, `saddle-lab`, `saddle-quant` +- workspace crates with a path dependency on this crate: `hip-bridge`, `hipfire-arch-cohere2moe`, `hipfire-arch-deepseek4`, `hipfire-arch-dots-ocr`, `hipfire-arch-gemma4`, `hipfire-arch-lfm2moe`, `hipfire-arch-maple`, `hipfire-arch-minimax`, `hipfire-arch-muse-glimmer`, `hipfire-arch-qwen2`, `hipfire-arch-qwen35`, `hipfire-arch-qwen35-vl`, `hipfire-cli`, `hipfire-client`, `hipfire-daemon`, `hipfire-dispatch`, `hipfire-ds4-parent`, `hipfire-engine`, `hipfire-generate`, `hipfire-loader`, `hipfire-pflash`, `hipfire-quantize`, `hipfire-reap`, `hipfire-registry`, `hipfire-runtime`, `hipfire-tui`, `hsa-bridge`, `rdna-compute`, `redline-dispatch`, `redline-rocr`, `saddle-lab`, `saddle-quant` ### Totals -- 3 modules · 7,724 lines · 118 public items · 65 tests · 0 examples +- 3 modules · 7,791 lines · 119 public items · 67 tests · 0 examples diff --git a/crates/hipfire-dispatch/map.md b/crates/hipfire-dispatch/map.md index 6852915c9..c6bc82b97 100644 --- a/crates/hipfire-dispatch/map.md +++ b/crates/hipfire-dispatch/map.md @@ -24,7 +24,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside |---|---:|---:|---:| | [`src/context.rs`](src/context.rs) | 67 | 5 | 0 | | [`src/coverage_tests.rs`](src/coverage_tests.rs) | 1,841 | 0 | 21 | -| [`src/families/attention.rs`](src/families/attention.rs) | 2,239 | 9 | 10 | +| [`src/families/attention.rs`](src/families/attention.rs) | 2,268 | 9 | 10 | | [`src/families/fused_qkv.rs`](src/families/fused_qkv.rs) | 1,585 | 8 | 1 | | [`src/families/gemm.rs`](src/families/gemm.rs) | 649 | 7 | 2 | | [`src/families/gemv.rs`](src/families/gemv.rs) | 624 | 20 | 0 | @@ -40,11 +40,11 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | [`src/ops/delta_net.rs`](src/ops/delta_net.rs) | 289 | 6 | 0 | | [`src/ops/mla.rs`](src/ops/mla.rs) | 329 | 5 | 0 | | [`src/ops/mod.rs`](src/ops/mod.rs) | 4 | 2 | 0 | -| [`src/pipeline/mod.rs`](src/pipeline/mod.rs) | 4,119 | 18 | 16 | +| [`src/pipeline/mod.rs`](src/pipeline/mod.rs) | 4,125 | 18 | 16 | | [`src/pipeline/steps.rs`](src/pipeline/steps.rs) | 1,559 | 5 | 13 | | [`src/pipeline/superop.rs`](src/pipeline/superop.rs) | 598 | 17 | 4 | | [`src/resource/mod.rs`](src/resource/mod.rs) | 19 | 3 | 0 | -| [`src/tables/attention_table.rs`](src/tables/attention_table.rs) | 448 | 1 | 0 | +| [`src/tables/attention_table.rs`](src/tables/attention_table.rs) | 458 | 1 | 0 | | [`src/tables/fused_qkv_table.rs`](src/tables/fused_qkv_table.rs) | 240 | 1 | 0 | | [`src/tables/gemm_table.rs`](src/tables/gemm_table.rs) | 512 | 1 | 0 | | [`src/tables/gemv_table.rs`](src/tables/gemv_table.rs) | 189 | 1 | 0 | @@ -53,7 +53,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | [`src/tables/rotation_table.rs`](src/tables/rotation_table.rs) | 42 | 1 | 0 | | [`src/tests.rs`](src/tests.rs) | 2,524 | 0 | 114 | | [`src/traits.rs`](src/traits.rs) | 7 | 1 | 0 | -| [`src/types.rs`](src/types.rs) | 981 | 23 | 2 | +| [`src/types.rs`](src/types.rs) | 982 | 23 | 2 | ### Public API surface @@ -103,6 +103,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Totals -- 32 modules · 22,291 lines · 223 public items · 238 tests · 0 examples +- 32 modules · 22,337 lines · 223 public items · 238 tests · 0 examples diff --git a/crates/hipfire-dispatch/src/families/attention.rs b/crates/hipfire-dispatch/src/families/attention.rs index 51e323cd2..632d39794 100644 --- a/crates/hipfire-dispatch/src/families/attention.rs +++ b/crates/hipfire-dispatch/src/families/attention.rs @@ -262,6 +262,14 @@ fn dispatch_full_attention( Ok(()) } // ── Non-causal, F32 K/V ── + TileImpl::DflashN64 => { + debug_assert_eq!(key, AttnFullF32); + hip!(gpu.attention_dflash_wmma_n64_f32( + io.q, io.k, io.v, io.out, io.n, io.seq_len, + io.n_heads, io.n_kv_heads, io.head_dim, + ))?; + Ok(()) + } TileImpl::DflashM32 => { debug_assert_eq!(key, AttnFullF32); hip!(gpu.attention_dflash_wmma_m32_f32( @@ -1681,8 +1689,28 @@ fn dispatch_attend( // gate. The scalar variant keeps its measured break-even. // It computes in f16 (relative L2 ~1e-3 vs the f32 // reference) — a real precision/speed trade, hence opt-in. - let variant = hipfire_config::developer_var("HIPFIRE_FLASH_PREFILL_KERNEL") - .unwrap_or_else(|_| "wmma".to_owned()); + let variant_override = hipfire_config::developer_var("HIPFIRE_FLASH_PREFILL_KERNEL").ok(); + let variant = variant_override.clone().unwrap_or_else(|| { + if gpu.arch.starts_with("gfx11") + && ctx.workload == crate::context::DispatchWorkload::SpeculativeVerify + { + "batched".to_owned() + } else { + "wmma".to_owned() + } + }); + // Explicit A/B route for speculative verify. Batched + // flash keeps all query rows in one tiled launch and + // avoids the slower query-tiled WMMA path on gfx11. + if variant == "batched" { + let fp = io.flash_partials.unwrap(); + return hip!(gpu.attention_flash_q8_0_batched_masked( + io.q, io.k_cache, io.v_cache, io.output, io.positions(), + io.n_heads, io.n_kv_heads, io.head_dim, + io.physical_cap, io.max_ctx_len, io.batch_size, fp, + io.tree_bias, io.block_start, io.block_cols, + )); + } // Kernel bounds: Q8_0 blocks are 32 dims wide, and O_frags // is a fixed float8_t[MAX_D_CHUNKS=16] => head_dim <= 256. let wmma_ok = variant != "scalar" @@ -2163,6 +2191,7 @@ mod tests { TileImpl::DflashV5, TileImpl::DflashV5Gfx12, TileImpl::DflashN128, + TileImpl::DflashN64, TileImpl::DflashM32, TileImpl::DflashWmmaF32, TileImpl::DflashScalar, diff --git a/crates/hipfire-dispatch/src/tables/attention_table.rs b/crates/hipfire-dispatch/src/tables/attention_table.rs index f6a5f4a78..32d08d432 100644 --- a/crates/hipfire-dispatch/src/tables/attention_table.rs +++ b/crates/hipfire-dispatch/src/tables/attention_table.rs @@ -386,7 +386,17 @@ pub fn populate(registry: &mut KernelRegistry) { }); // No scalar floor for F16 — fall to AttnFullF32 at caller level. - // AttnFullF32: non-causal, F32 K/V + registry.register(KernelVariant { + key: KernelKey::AttnFullF32, + arch_required: ArchPredicate::HasWmmaW32, + shape_gate: Some(ShapePredicate::And(&[ + ShapePredicate::BatchEq(16), + ShapePredicate::HeadDimEq(128), + ])), + steps: &[PipelineOp::Attend], + has_awq: false, + tile: TileImpl::DflashN64, + }); registry.register(KernelVariant { key: KernelKey::AttnFullF32, arch_required: ArchPredicate::HasWmma, diff --git a/crates/hipfire-dispatch/src/types.rs b/crates/hipfire-dispatch/src/types.rs index a191089a0..2ddae83e3 100644 --- a/crates/hipfire-dispatch/src/types.rs +++ b/crates/hipfire-dispatch/src/types.rs @@ -76,6 +76,7 @@ pub enum TileImpl { DflashV5Gfx12, DflashN128, // Vision/dflash F32-K/V rungs + DflashN64, DflashM32, DflashWmmaF32, // Causal (F16-K/V rungs) diff --git a/crates/hipfire-generate/map.md b/crates/hipfire-generate/map.md index f59f344a3..22b9e0cf6 100644 --- a/crates/hipfire-generate/map.md +++ b/crates/hipfire-generate/map.md @@ -27,9 +27,9 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | [`src/common.rs`](src/common.rs) | 1,569 | 48 | 1 | | [`src/dense.rs`](src/dense.rs) | 8,455 | 91 | 4 | | [`src/lib.rs`](src/lib.rs) | 58 | 7 | 0 | -| [`src/qwen.rs`](src/qwen.rs) | 6,312 | 60 | 1 | +| [`src/qwen.rs`](src/qwen.rs) | 6,432 | 62 | 1 | | [`src/redline.rs`](src/redline.rs) | 4,361 | 49 | 1 | -| [`src/vision.rs`](src/vision.rs) | 2,942 | 9 | 8 | +| [`src/vision.rs`](src/vision.rs) | 2,939 | 9 | 8 | ### Public API surface @@ -38,7 +38,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside - [`src/common.rs`](src/common.rs): `asst_turn_fingerprint`, `strip_think_for_fingerprint`, `normalize_asst_turn_for_fingerprint`, `emit_spec_cancel_after_rollback`, `RollbackEpilogue`, `production_fail_closed_rollback`, `production_fail_closed_rollback_live`, `emit_fail_closed_error`, `ds4_gen_start_contract_version`, `gen_start_contract_version_for_arch`, `Ds4MalformedTerminalAction`, `ds4_malformed_terminal_action`, +36 more - [`src/dense.rs`](src/dense.rs): `glimmer_turn_key`, `emit_active_attempt_error`, `Ds4SpecWireTerminal`, `ds4_spec_wire_terminal`, `ds4_cache_action`, `ds4_ar_client_abort`, `GlimmerSpecMode`, `glimmer_spec_admission`, `write_error_envelope`, `generate_deepseek4_spec`, `generate_deepseek4`, `ds4_heterogeneous_client_abort`, +79 more - [`src/lib.rs`](src/lib.rs): `common`, `ar`, `qwen`, `dense`, `vision`, `redline`, `batch` -- [`src/qwen.rs`](src/qwen.rs): `EpSampling`, `generate_ep`, `ep_emit_token`, `ep_serve_qwen35_dense_tp`, `ep_emit_done`, `ep_reset_after_abort`, `ep_emit_abort`, `ep_serve_ds4`, `ep_serve_minimax`, `qwen_history_tool_render`, `plan_prompt_cache`, `plan_from_rendered`, +48 more +- [`src/qwen.rs`](src/qwen.rs): `EpSampling`, `generate_ep`, `ep_emit_token`, `ep_serve_qwen35_dense_tp`, `ep_emit_done`, `ep_reset_after_abort`, `ep_emit_abort`, `ep_serve_ds4`, `ep_serve_minimax`, `qwen_history_tool_render`, `plan_prompt_cache`, `plan_from_rendered`, +50 more - [`src/redline.rs`](src/redline.rs): `RedlineQwenSnapshot`, `json`, `RedlineDeepseek4Snapshot`, `RedlineDsparkVerifySnapshot`, `RedlineSnapshot`, `logits`, `kv`, `recurrent`, `gdn_frame`, `redline_qwen_snapshot`, `redline_deepseek4_snapshot`, `RedlineLfm2MoeSnapshot`, +37 more - [`src/vision.rs`](src/vision.rs): `ImageSource`, `GenerateVLParams`, `vl_no_eviction_kv_cap`, `generate_vl`, `generate_vl_dots_ocr`, `decode_vl_dots_ocr_ngram`, `run_dots_ocr_ngram_loop`, `generate_dots_ocr_text`, `generate_lfm2_vl` @@ -55,6 +55,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Totals -- 8 modules · 31,785 lines · 321 public items · 224 tests · 0 examples +- 8 modules · 31,902 lines · 323 public items · 225 tests · 0 examples diff --git a/crates/hipfire-generate/src/qwen.rs b/crates/hipfire-generate/src/qwen.rs index f91daf1ae..a37875ebb 100644 --- a/crates/hipfire-generate/src/qwen.rs +++ b/crates/hipfire-generate/src/qwen.rs @@ -3335,6 +3335,7 @@ pub fn generate_spec( // Fast path exit conditions (mirrors the dflash_spec_demo outer loop). // `!first_token_is_eos` short-circuits the entire spec loop when the prefill's // first sampled token was already a terminator (see the guard above). + let mut terminal_cache_invalidated = false; while !first_token_is_eos && generated < max_tokens { // Decode-side abort (dflash path). See the matching block in // `generate()` for rationale. Without this, a Pi cancel @@ -3367,6 +3368,7 @@ pub fn generate_spec( // matcher so the fused step constrains drafts in-place. `emit.grammar()`'s // borrow ends when `step` returns, before the per-token `emit.observe`. let max_emit = max_tokens.saturating_sub(generated); + let window_seed = seed_token; let step = match spec.step( gpu, slot, @@ -3515,9 +3517,10 @@ pub fn generate_spec( }; // Strict-prefix semantic stop: drop unobserved speculative tail from - // target + drafter via conservative reset + production prefill of the - // exact KV-resident prefix (`spec_prefix_realign_plan`). Full-window - // observe keeps the step's already-committed GPU state. + // target + drafter. Continue-generation paths rebuild the exact + // KV-resident prefix; completed requests reset and invalidate cache + // metadata so they do not pay a full-history replay before returning. + // Full-window observe keeps the step's already-committed GPU state. // // Capacity-aware: admit BEFORE reset/prefill. Realign is full-history // replay after reset (compact_offset cleared) — never overrun @@ -3525,7 +3528,95 @@ pub fn generate_spec( // state from an invalid oversize history. Abort/prefill errors share // the single fail-closed terminal (no second done/error). let keep = consumed.min(committed_tail.len()); - if keep < committed_tail.len() { + let strict_prefix_action = + spec_strict_prefix_action(keep, committed_tail.len(), hit_eos || think_cap_hit); + if strict_prefix_action == SpecStrictPrefixAction::RepairForTerminal { + // Prefer a window-local repair: restore the pre-window target/drafter + // snapshot and replay only the consumed prefix. Speculators without + // that capability retain the conservative reset + cache invalidation. + let repaired = match spec.repair_terminal_prefix( + gpu, + slot, + position_before, + window_seed, + &committed_tail[..keep], + ) { + Ok(repaired) => repaired, + Err(e) => { + let ep = production_fail_closed_rollback_live( + &mut m.seq_pos, + &mut m.conversation_tokens, + &mut m.prefill_checkpoints, + &mut m.dflash_checkpoints, + &mut m.asst_turn_cache, + gpu, + slot, + spec.as_mut(), + ); + emit_fail_closed_error( + stdout, + Some(id), + &format!("terminal prefix repair failed: {e}"), + "gpu", + true, + &ep, + ); + drop(guard); + return None; + } + }; + if repaired + && hipfire_config::developer_var("HIPFIRE_QWEN_CACHE_TRACE") + .ok() + .as_deref() + == Some("1") + { + eprintln!( + "[qwen-cache terminal-repair] window_start={} consumed={} replayed={}", + position_before, keep, keep + ); + } + let reset_error = if repaired { + None + } else { + slot.reset_recurrent(gpu) + .err() + .map(|e| format!("reset_recurrent: {e}")) + .or_else(|| { + spec.reset_for_realign(gpu) + .err() + .map(|e| format!("spec.reset_for_realign: {e}")) + }) + }; + if let Some(msg) = reset_error { + let ep = production_fail_closed_rollback_live( + &mut m.seq_pos, + &mut m.conversation_tokens, + &mut m.prefill_checkpoints, + &mut m.dflash_checkpoints, + &mut m.asst_turn_cache, + gpu, + slot, + spec.as_mut(), + ); + emit_fail_closed_error( + stdout, + Some(id), + &format!("terminal prefix reset failed: {msg}"), + "gpu", + true, + &ep, + ); + drop(guard); + return None; + } + if !repaired { + terminal_cache_invalidated = true; + pending_seed_committable = false; + position = 0; + } + } + if strict_prefix_action == SpecStrictPrefixAction::Realign { let plan = spec_prefix_realign_plan(&prompt_tokens, first_token, &raw_decode); let compact_offset = slot.kv_cache_mut().map(|kv| kv.compact_offset).unwrap_or(0); if let Err(msg) = spec_prefix_realign_admit( @@ -3885,7 +3976,12 @@ pub fn generate_spec( // only the decoded portion (`emitted`), making the next non-dflash turn // full-reset because no system/user prefix was present. // Host raw/conversation stay exact even when client events were held. - m.conversation_tokens = { + m.conversation_tokens = if terminal_cache_invalidated { + free_checkpoints(&mut m.prefill_checkpoints, gpu); + free_checkpoints(&mut m.dflash_checkpoints, gpu); + m.asst_turn_cache.clear(); + Vec::new() + } else { let mut v = Vec::with_capacity(prompt_tokens.len() + emitted.len()); v.extend_from_slice(&prompt_tokens); v.extend_from_slice(&emitted); @@ -5554,6 +5650,30 @@ pub fn spec_should_flush_pending_seed( !grammar_violated && pending_seed_committable } +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum SpecStrictPrefixAction { + None, + Realign, + RepairForTerminal, +} + +/// Decide how to repair target and drafter state after observing only a strict +/// prefix of a speculative window. Terminal requests prefer a window-local +/// rollback; speculators without that capability fall back to a full reset. +pub fn spec_strict_prefix_action( + consumed: usize, + committed: usize, + terminal: bool, +) -> SpecStrictPrefixAction { + if consumed >= committed { + SpecStrictPrefixAction::None + } else if terminal { + SpecStrictPrefixAction::RepairForTerminal + } else { + SpecStrictPrefixAction::Realign + } +} + pub fn spec_prefix_realign_plan( prompt: &[u32], first_token: u32, diff --git a/crates/hipfire-generate/tests/qwen_dflash_semantic_terminal_tests.rs b/crates/hipfire-generate/tests/qwen_dflash_semantic_terminal_tests.rs index 38a8e59d8..88ee15f11 100644 --- a/crates/hipfire-generate/tests/qwen_dflash_semantic_terminal_tests.rs +++ b/crates/hipfire-generate/tests/qwen_dflash_semantic_terminal_tests.rs @@ -621,10 +621,46 @@ use hipfire_runtime::emit_text::extract_tool_calls_from_text; } #[test] - fn terminal_marker_mid_window_strict_prefix_realigns() { + fn terminal_strict_prefix_uses_window_repair() { + use hipfire_generate::qwen::{ + spec_strict_prefix_action, SpecStrictPrefixAction, + }; + + assert_eq!( + spec_strict_prefix_action(9, 11, true), + SpecStrictPrefixAction::RepairForTerminal + ); + + use hipfire_runtime::spec::terminal_prefix_replay; + assert_eq!( + terminal_prefix_replay(7, &[]).as_slice(), + &[] as &[u32] + ); + assert_eq!(terminal_prefix_replay(7, &[8]).as_slice(), &[7]); + assert_eq!( + terminal_prefix_replay(7, &[8, 9, 10]).as_slice(), + &[7, 8, 9] + ); + assert_eq!( + spec_strict_prefix_action(9, 11, false), + SpecStrictPrefixAction::Realign + ); + assert_eq!( + spec_strict_prefix_action(11, 11, true), + SpecStrictPrefixAction::None + ); + assert_eq!( + spec_strict_prefix_action(12, 11, false), + SpecStrictPrefixAction::None + ); + } + + #[test] + fn terminal_marker_mid_window_tracks_exact_host_prefix() { // Spec window emits body + im_end + unobserved tail. Semantic loop - // consumes only through the terminal marker; host + realign plan must - // land exactly on that prefix (no unobserved tail in conversation or KV). + // consumes only through the terminal marker; host bookkeeping must + // exclude the unobserved tail; window-local repair replays this exact + // prefix while leaving the terminal token for the ordinary flush. let tok = test_tokenizer(); let prompt = vec![4u32, 5]; let first_token = tok.encode("hi")[0]; diff --git a/crates/hipfire-runtime/map.md b/crates/hipfire-runtime/map.md index 1995fd991..f47661cca 100644 --- a/crates/hipfire-runtime/map.md +++ b/crates/hipfire-runtime/map.md @@ -38,7 +38,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | [`src/cpu_router.rs`](src/cpu_router.rs) | 200 | 4 | 3 | | [`src/ddtree.rs`](src/ddtree.rs) | 2,046 | 17 | 24 | | [`src/device_mesh.rs`](src/device_mesh.rs) | 582 | 21 | 8 | -| [`src/dflash.rs`](src/dflash.rs) | 3,460 | 44 | 4 | +| [`src/dflash.rs`](src/dflash.rs) | 3,514 | 47 | 5 | | [`src/dflash_generic.rs`](src/dflash_generic.rs) | 1,365 | 3 | 13 | | [`src/dspark_block_controller.rs`](src/dspark_block_controller.rs) | 442 | 0 | 10 | | [`src/dspark_core.rs`](src/dspark_core.rs) | 1,773 | 11 | 0 | @@ -59,7 +59,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | [`src/loop_guard.rs`](src/loop_guard.rs) | 194 | 8 | 4 | | [`src/model_load.rs`](src/model_load.rs) | 117 | 8 | 1 | | [`src/model_source.rs`](src/model_source.rs) | 92 | 4 | 0 | -| [`src/multi_gpu.rs`](src/multi_gpu.rs) | 2,141 | 36 | 9 | +| [`src/multi_gpu.rs`](src/multi_gpu.rs) | 2,150 | 36 | 9 | | [`src/ngram_mod.rs`](src/ngram_mod.rs) | 484 | 11 | 13 | | [`src/paro.rs`](src/paro.rs) | 424 | 9 | 3 | | [`src/prefix.rs`](src/prefix.rs) | 109 | 3 | 6 | @@ -70,7 +70,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | [`src/semantic.rs`](src/semantic.rs) | 773 | 29 | 16 | | [`src/serve/mod.rs`](src/serve/mod.rs) | 215 | 12 | 3 | | [`src/session_table.rs`](src/session_table.rs) | 654 | 19 | 24 | -| [`src/spec.rs`](src/spec.rs) | 1,923 | 35 | 15 | +| [`src/spec.rs`](src/spec.rs) | 1,954 | 36 | 15 | | [`src/spec_ngram.rs`](src/spec_ngram.rs) | 491 | 4 | 3 | | [`src/swap/mod.rs`](src/swap/mod.rs) | 176 | 10 | 3 | | [`src/swap/snapshot.rs`](src/swap/snapshot.rs) | 392 | 9 | 7 | @@ -99,7 +99,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside - [`src/cpu_router.rs`](src/cpu_router.rs): `CpuRouter`, `from_f32_weights`, `compute_topk`, `TopK` - [`src/ddtree.rs`](src/ddtree.rs): `DdNode`, `DdTree`, `num_nodes`, `ancestors_of`, `build_ddtree_tree`, `build_ddtree_tree_with_cutoff`, `build_ddtree_tree_bounded`, `follow_verified_tree`, `naive_sample_chain`, `swor_draft_candidates`, `sample_host_nucleus`, `dump_pq_jsonl`, +5 more - [`src/device_mesh.rs`](src/device_mesh.rs): `MeshEpoch`, `as_u64`, `DimKind`, `Axis`, `CollectiveHint`, `MeshError`, `DeviceMesh`, `rect`, `single`, `axes`, `epoch`, `n_devices`, +9 more -- [`src/dflash.rs`](src/dflash.rs): `DflashConfig`, `num_extract`, `kv_dim`, `q_dim`, `runtime_block_size`, `from_hfq`, `DflashLayerWeights`, `SelectorCodebook`, `get_f32`, `DflashWeights`, `has_candidate_selector`, `load`, +32 more +- [`src/dflash.rs`](src/dflash.rs): `DflashConfig`, `num_extract`, `kv_dim`, `q_dim`, `runtime_block_size`, `from_hfq`, `DflashLayerWeights`, `SelectorCodebook`, `get_f32`, `DflashWeights`, `has_candidate_selector`, `load`, +35 more - [`src/dflash_generic.rs`](src/dflash_generic.rs): `dense_tree_verify_nodes`, `GenericDflashSpeculator`, `build_generic_dflash_speculator` - [`src/dspark_block_controller.rs`](src/dspark_block_controller.rs): — - [`src/dspark_core.rs`](src/dspark_core.rs): `DsparkConfig`, `from_metadata_json`, `DsparkWeights`, `DsparkBody`, `DraftResult`, `main_proj_ingest`, `main_proj_ingest_batched`, `noise_block_ids`, `run_heads`, `DsparkDrafter`, `build_dspark_speculator` @@ -131,7 +131,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside - [`src/semantic.rs`](src/semantic.rs): `AttemptId`, `fn`, `VisibleText`, `as_str`, `into_string`, `MalformedProtocol`, `new`, `detail`, `TerminalReason`, `TerminalOutcome`, `CommittedToken`, `SemanticEvent`, +17 more - [`src/serve/mod.rs`](src/serve/mod.rs): `SubmitRequest`, `Continuation`, `tokens`, `DoneReason`, `Event`, `send_event`, `EngineStats`, `note_admitted`, `note_rejected`, `note_eviction`, `note_restore`, `note_prefix_hit` - [`src/session_table.rs`](src/session_table.rs): `SessionId`, `Residency`, `Session`, `SessionTable`, `open`, `close`, `get`, `get_mut`, `begin_turn`, `find_continuation`, `confirm_reentry`, `touch`, +7 more -- [`src/spec.rs`](src/spec.rs): `SpecStep`, `new`, `cap_emit`, `GreedyAccept`, `accept_greedy_prefix`, `SpecTarget`, `SpecTargetGuard`, `InPlaceGuard`, `SpecScratch`, `SpecAdvance`, `SpecGrammar`, `PrefillOutcome`, +23 more +- [`src/spec.rs`](src/spec.rs): `SpecStep`, `new`, `cap_emit`, `terminal_prefix_replay`, `GreedyAccept`, `accept_greedy_prefix`, `SpecTarget`, `SpecTargetGuard`, `InPlaceGuard`, `SpecScratch`, `SpecAdvance`, `SpecGrammar`, +24 more - [`src/spec_ngram.rs`](src/spec_ngram.rs): `BlockDrafter`, `NgramDrafter`, `new`, `ChainSpeculator` - [`src/swap/mod.rs`](src/swap/mod.rs): `snapshot`, `store`, `SwapError`, `DEFAULT_HOST_BUDGET_BYTES`, `SwapManager`, `new`, `stats`, `park`, `unpark`, `forget` - [`src/swap/snapshot.rs`](src/swap/snapshot.rs): `checksum_of`, `SnapshotStamp`, `SlotSnapshot`, `expected_len`, `validate`, `to_bytes`, `from_bytes`, `capture_slot`, `restore_slot` @@ -156,6 +156,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Totals -- 58 modules · 53,079 lines · 893 public items · 617 tests · 132 examples +- 58 modules · 53,173 lines · 897 public items · 618 tests · 132 examples diff --git a/crates/hipfire-runtime/src/dflash.rs b/crates/hipfire-runtime/src/dflash.rs index 1f7ce9ad3..8b2bd90dd 100644 --- a/crates/hipfire-runtime/src/dflash.rs +++ b/crates/hipfire-runtime/src/dflash.rs @@ -1003,6 +1003,14 @@ impl DflashWeights { /// the #462 class): that error is now defined out of existence — it does not /// compile. mod target_hidden_log { + #[derive(Clone, Copy, Debug)] + pub struct TargetHiddenLogMark { + uploaded_rows: usize, + abs_positions_len: usize, + proj_cached_rows: usize, + full_cached_rows: usize, + } + /// See module-level intent. Construct via [`TargetHiddenLog::new`]. #[derive(Default)] pub struct TargetHiddenLog { @@ -1042,6 +1050,33 @@ mod target_hidden_log { self.full_cached_rows } + /// Lightweight rollback point for one speculative window. The backing + /// tensors are append-only here, so restoring metadata is sufficient; + /// stale tail rows are ignored and overwritten by the next append. + pub fn mark(&self) -> TargetHiddenLogMark { + TargetHiddenLogMark { + uploaded_rows: self.uploaded_rows, + abs_positions_len: self.abs_positions.len(), + proj_cached_rows: self.proj_cached_rows, + full_cached_rows: self.full_cached_rows, + } + } + + pub fn restore(&mut self, mark: TargetHiddenLogMark) -> Result<(), String> { + if mark.abs_positions_len > self.abs_positions.len() { + return Err(format!( + "target-hidden rollback mark {} exceeds live rows {}", + mark.abs_positions_len, + self.abs_positions.len() + )); + } + self.abs_positions.truncate(mark.abs_positions_len); + self.uploaded_rows = mark.uploaded_rows; + self.proj_cached_rows = mark.proj_cached_rows; + self.full_cached_rows = mark.full_cached_rows; + Ok(()) + } + // ── invariant-preserving mutations ──────────────────────────────── /// New-prompt / session boundary: forget all GPU-resident rows. pub fn reset(&mut self) { @@ -1148,7 +1183,7 @@ mod target_hidden_log { } } } -pub use target_hidden_log::TargetHiddenLog; +pub use target_hidden_log::{TargetHiddenLog, TargetHiddenLogMark}; // ─── Scratch ─────────────────────────────────────────────────────────────── @@ -3036,7 +3071,7 @@ pub fn draft_forward_opts( #[cfg(test)] mod ring_tests { - use super::ring_segments; + use super::{ring_segments, TargetHiddenLog}; #[test] fn identity_modulus_is_single_segment() { @@ -3075,6 +3110,25 @@ mod ring_tests { } } } + + #[test] + fn target_hidden_log_restores_a_speculative_append() { + let mut log = TargetHiddenLog::new(); + log.seed_prompt(4); + log.mark_proj_cached(3); + log.mark_full_cached(4); + let mark = log.mark(); + + log.append_committed(4, 2, 0); + log.mark_proj_cached(6); + assert_eq!(log.abs_positions(), &[0, 1, 2, 3, 4, 5]); + + log.restore(mark).unwrap(); + assert_eq!(log.uploaded_rows(), 4); + assert_eq!(log.abs_positions(), &[0, 1, 2, 3]); + assert_eq!(log.proj_cached_rows(), 3); + assert_eq!(log.full_cached_rows(), 4); + } } // ─── Candidate selector (DFlash2 chain-only) ─────────────────────────────── diff --git a/crates/hipfire-runtime/src/spec.rs b/crates/hipfire-runtime/src/spec.rs index 1499f231a..8dfc51e1d 100644 --- a/crates/hipfire-runtime/src/spec.rs +++ b/crates/hipfire-runtime/src/spec.rs @@ -112,6 +112,18 @@ impl SpecStep { } } +/// Tokens that must be re-forwarded after restoring a speculative window's +/// pre-verify snapshot. The final consumed token remains pending and is +/// committed by the caller's ordinary terminal flush. +pub fn terminal_prefix_replay(window_seed: u32, consumed: &[u32]) -> SmallVec<[u32; 8]> { + let mut replay = SmallVec::with_capacity(consumed.len()); + if !consumed.is_empty() { + replay.push(window_seed); + replay.extend_from_slice(&consumed[..consumed.len() - 1]); + } + replay +} + /// Outcome of the shared greedy accept-prefix rule ([`accept_greedy_prefix`]). #[derive(Debug, Clone, PartialEq, Eq)] pub struct GreedyAccept { @@ -763,6 +775,25 @@ pub trait Speculator { Ok(false) } + /// Repair a terminal that consumed only a strict prefix of the most recent + /// speculative window. Implementations with a retained pre-window snapshot + /// restore it and replay only the state-committable prefix, leaving the last + /// consumed token pending for the caller's normal terminal flush. + /// + /// Returns `true` when the resident target and drafter caches are repaired. + /// The default is unsupported; callers retain the conservative reset path. + fn repair_terminal_prefix( + &mut self, + gpu: &mut Gpu, + target: &mut dyn SpecTarget, + window_start: usize, + window_seed: u32, + consumed: &[u32], + ) -> Result { + let _ = (gpu, target, window_start, window_seed, consumed); + Ok(false) + } + /// Rewind drafter-LOCAL state for a fresh conversation. The target's KV / /// recurrent state is the daemon's concern (it owns the bundle); this clears /// only the drafter's own scratch + checkpoint ring. diff --git a/crates/rdna-compute/map.md b/crates/rdna-compute/map.md index c8b7ea6b3..dd5db0155 100644 --- a/crates/rdna-compute/map.md +++ b/crates/rdna-compute/map.md @@ -24,25 +24,25 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | File | Lines | Public items | Tests | |---|---:|---:|---:| | [`src/arch_caps.rs`](src/arch_caps.rs) | 712 | 55 | 19 | -| [`src/attention.rs`](src/attention.rs) | 14,946 | 211 | 3 | +| [`src/attention.rs`](src/attention.rs) | 14,872 | 211 | 3 | | [`src/bin/hipfire-kernel-hash.rs`](src/bin/hipfire-kernel-hash.rs) | 142 | 0 | 0 | | [`src/cdna/gfx942.rs`](src/cdna/gfx942.rs) | 578 | 10 | 1 | | [`src/cdna/mod.rs`](src/cdna/mod.rs) | 11 | 1 | 0 | -| [`src/compiler.rs`](src/compiler.rs) | 2,266 | 8 | 24 | -| [`src/dispatch.rs`](src/dispatch.rs) | 5,224 | 113 | 14 | +| [`src/compiler.rs`](src/compiler.rs) | 2,506 | 8 | 28 | +| [`src/dispatch.rs`](src/dispatch.rs) | 5,388 | 117 | 19 | | [`src/embedding.rs`](src/embedding.rs) | 410 | 10 | 0 | -| [`src/feature_flags.rs`](src/feature_flags.rs) | 908 | 12 | 6 | +| [`src/feature_flags.rs`](src/feature_flags.rs) | 915 | 12 | 6 | | [`src/flash_attn_ck.rs`](src/flash_attn_ck.rs) | 1,775 | 26 | 15 | -| [`src/gemm.rs`](src/gemm.rs) | 36,086 | 437 | 0 | +| [`src/gemm.rs`](src/gemm.rs) | 35,995 | 437 | 0 | | [`src/gemma4_ext.rs`](src/gemma4_ext.rs) | 542 | 18 | 0 | | [`src/gemma4_ops.rs`](src/gemma4_ops.rs) | 83 | 1 | 0 | -| [`src/gemv.rs`](src/gemv.rs) | 15,991 | 235 | 0 | +| [`src/gemv.rs`](src/gemv.rs) | 15,620 | 235 | 0 | | [`src/graph.rs`](src/graph.rs) | 556 | 33 | 0 | | [`src/kernels.rs`](src/kernels.rs) | 8,088 | 1228 | 37 | | [`src/kv_slots.rs`](src/kv_slots.rs) | 420 | 9 | 10 | | [`src/lib.rs`](src/lib.rs) | 88 | 26 | 1 | | [`src/moe.rs`](src/moe.rs) | 1,742 | 27 | 0 | -| [`src/norm.rs`](src/norm.rs) | 6,170 | 90 | 0 | +| [`src/norm.rs`](src/norm.rs) | 6,165 | 90 | 0 | | [`src/pool.rs`](src/pool.rs) | 96 | 5 | 0 | | [`src/profile.rs`](src/profile.rs) | 358 | 43 | 0 | | [`src/profile_rocprof.rs`](src/profile_rocprof.rs) | 343 | 6 | 4 | @@ -50,8 +50,8 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside | [`src/rdna/gfx1201.rs`](src/rdna/gfx1201.rs) | 414 | 7 | 0 | | [`src/rdna/mod.rs`](src/rdna/mod.rs) | 11 | 1 | 0 | | [`src/replay.rs`](src/replay.rs) | 9,126 | 78 | 81 | -| [`src/sampling.rs`](src/sampling.rs) | 1,765 | 22 | 3 | -| [`src/scratch.rs`](src/scratch.rs) | 1,415 | 21 | 0 | +| [`src/sampling.rs`](src/sampling.rs) | 1,753 | 22 | 3 | +| [`src/scratch.rs`](src/scratch.rs) | 1,419 | 21 | 0 | | [`src/slot_pool.rs`](src/slot_pool.rs) | 236 | 11 | 7 | ### Public API surface @@ -62,7 +62,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside - [`src/cdna/gfx942.rs`](src/cdna/gfx942.rs): `Gfx942Device`, `try_gfx942`, `mq2_lloyd_moe_gate_up_wave64`, `mq2_lloyd_moe_gate_up_wave64x8_candidate`, `mq_rotate_x_wave64_batched`, `mq2_lloyd_moe_down_expanded_wave64`, `mq2_lloyd_moe_down_residual_wave64`, `indexer_top_k_buf_parallel`, `grouped_olora_e8`, `grouped_olora_e8_wave64x4_candidate` - [`src/cdna/mod.rs`](src/cdna/mod.rs): `gfx942` - [`src/compiler.rs`](src/compiler.rs): `KernelCompiler`, `new`, `compiled_kernels`, `register_func_artifact`, `packaging_hash`, `packaging_hash_for`, `compile`, `compile_batch` -- [`src/dispatch.rs`](src/dispatch.rs): `LLOYD_MQ3_GROUP_BYTES`, `LLOYD_MQ4_GROUP_BYTES`, `GL_MQ2_GROUP_IDX_BYTES`, `GL_MQ3_GROUP_IDX_BYTES`, `MQ4V2_GROUP_BYTES`, `MQ4C_GROUP_BYTES`, `MQ6G256V2_GROUP_BYTES`, `MQ5G256V2_GROUP_BYTES`, `MQ3G256V2_GROUP_BYTES`, `MQ2G256V2_GROUP_BYTES`, `GL_GROUP_SCALE_BYTES`, `GL_CB2`, +101 more +- [`src/dispatch.rs`](src/dispatch.rs): `LLOYD_MQ3_GROUP_BYTES`, `LLOYD_MQ4_GROUP_BYTES`, `GL_MQ2_GROUP_IDX_BYTES`, `GL_MQ3_GROUP_IDX_BYTES`, `MQ4V2_GROUP_BYTES`, `MQ4C_GROUP_BYTES`, `MQ6G256V2_GROUP_BYTES`, `MQ5G256V2_GROUP_BYTES`, `MQ3G256V2_GROUP_BYTES`, `MQ2G256V2_GROUP_BYTES`, `GL_GROUP_SCALE_BYTES`, `GL_CB2`, +105 more - [`src/embedding.rs`](src/embedding.rs): `embedding_lookup`, `embedding_lookup_q8`, `embedding_lookup_q8_buf_broadcast`, `embedding_lookup_q4k`, `embedding_lookup_hfq4g256`, `embedding_lookup_q8_batched`, `embedding_lookup_f16_batched`, `embedding_lookup_hfq4g256_batched`, `embedding_lookup_hfq4g128`, `embedding_lookup_hfq4g128_batched` - [`src/feature_flags.rs`](src/feature_flags.rs): `Mb4Mode`, `FeatureFlags`, `from_process_config`, `from_active_config`, `gemv_dp4a_enabled`, `ddtree_logw_cutoff_value`, `gemv_prefetch_enabled`, `gfx942_lds_gemv_enabled`, `hfq3_mmq_layer_gate_pass`, `fp16_disabled_for_current_layer`, `hfq4_mmq_gfx906_y64_enabled`, `for_test` - [`src/flash_attn_ck.rs`](src/flash_attn_ck.rs): `FLASH_ATTN_CK_ABI_VERSION`, `FlashAttnCkDType`, `FlashAttnCkArch`, `FlashAttnCkKvFormat`, `FLASH_ATTN_CK_CAP_CAUSAL`, `FLASH_ATTN_CK_CAP_GQA`, `FlashAttnCkCapability`, `FlashAttnCkRequest`, `FlashAttnCkPrefillInput`, `FlashAttnCkRejectReason`, `select_q8_d256_prefill`, `supports`, +14 more @@ -100,6 +100,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Totals -- 30 modules · 110,994 lines · 2757 public items · 225 tests · 192 examples +- 30 modules · 110,856 lines · 2761 public items · 234 tests · 192 examples diff --git a/crates/saddle-core/map.md b/crates/saddle-core/map.md index fb3217536..a3d4d7432 100644 --- a/crates/saddle-core/map.md +++ b/crates/saddle-core/map.md @@ -37,7 +37,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside |---|---:|---:|---:| | [`src/caps.rs`](src/caps.rs) | 286 | 11 | 3 | | [`src/grammar.rs`](src/grammar.rs) | 3,977 | 20 | 106 | -| [`src/kv.rs`](src/kv.rs) | 4,308 | 78 | 9 | +| [`src/kv.rs`](src/kv.rs) | 4,306 | 78 | 9 | | [`src/lib.rs`](src/lib.rs) | 76 | 6 | 0 | | [`src/logprobs.rs`](src/logprobs.rs) | 191 | 3 | 8 | | [`src/sampling.rs`](src/sampling.rs) | 52 | 2 | 0 | @@ -66,6 +66,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside ### Totals -- 7 modules · 9,213 lines · 122 public items · 127 tests · 0 examples +- 7 modules · 9,211 lines · 122 public items · 127 tests · 0 examples