fix(vram): free weight sidecars on unload; record the Ep mesh axis on EP loads

AGENTS.md

@@ -220,27 +220,27 @@ works, what to measure, what counts as pass/fail.
 
 ### Pull the model + draft you want to test
 
-Targets and drafts are independent pulls — drafts auto-discover their
-target by filename when the daemon loads:
+`hipfire pull <tag>` fetches the target plus its registry-declared
+DFlash draft sidecar (same mechanism as the MTP/DSpark sidecars):
 
 ```bash
 # 27B Qwen 3.5 (the canonical perf-test target):
-hipfire pull qwen3.5:27b           # 15 GB target
-hipfire pull qwen3.5:27b-draft     # 0.92 GB DFlash draft
+hipfire pull qwen3.5:27b           # 15 GB target + 0.92 GB DFlash draft sidecar
 
 # 27B Qwen 3.6 (refresh):
-hipfire pull qwen3.6:27b           # 15 GB target
-hipfire pull qwen3.6:27b-draft     # 0.92 GB DFlash draft
+hipfire pull qwen3.6:27b           # 15 GB target + 0.92 GB DFlash draft sidecar
 
 # 9B Qwen 3.5 (smaller, faster sanity-check):
-hipfire pull qwen3.5:9b            # 5.3 GB target
-hipfire pull qwen3.5:9b-draft      # 0.55 GB DFlash draft
+hipfire pull qwen3.5:9b            # 5.3 GB target + 0.55 GB DFlash draft sidecar
 ```
 
-Files land at `~/.hipfire/models/<canonical-name>` matching the
-daemon's auto-discovery pattern (`qwen3{ver}-{size}-dflash-{quant}.hfq`).
-**Do not rename.** Renaming breaks the auto-discovery and DFlash falls
-back to AR silently.
+Standalone `*-draft` tags (`hipfire pull qwen3.5:27b-draft`) still work —
+they address the same file for anyone who wants the draft alone.
+
+Files land at `~/.hipfire/models/<canonical-name>`.
+**Do not rename.** Load resolves the draft by its registry-declared
+filename; renaming breaks the pairing — `dflash_mode auto` then runs AR
+(one warning line), `on` fails the load.
 
 ### Verify md5s after pull (paranoid mode)
 
@@ -330,7 +330,8 @@ Standalone: `cargo run --release -p hipfire-runtime --example encode_prompt -- M
 - `hipfire-models/qwen3.6-27b/qwen36-27b-dflash-mq4.hfq` (+ the 3.6 27B
   target `hipfire-models/qwen3.6-27b/qwen3.6-27b.mq4`)
 
-Pullable via `hipfire pull qwen3.{5,6}:{9b,27b}-draft` and `hipfire pull qwen3.6:27b`.
+`hipfire pull <target>` fetches the target plus its draft sidecar;
+standalone drafts stay pullable via `hipfire pull qwen3.{5,6}:{9b,27b}-draft`.
 
 ---
 
@@ -355,6 +356,7 @@ hipfire bench <model> --runs 5 --warmups 3 --max-tokens 128 --json
 | `--spec` | `off`/`dflash`/`mtp`/`ngram`/`dspark`/`auto` |
 | `--backend` | `noslots` (sequential daemon) / `slots` / `batch` / `both` |
 | `--workload` | `stateless` / `multiturn` / `both` |
+| `--prompt-file PATH` | verbatim prompt bytes for the run; JSON records `prompt_tokens`/`prompt_md5`/`prompt_chars` plus a `warnings` caveat below 256 tokens |
 | `--kv-mode`, `--kv-backend` | KV format and allocator |
 | `--reasoning-on` | off by default: a reasoning model cannot close `<think>` inside the token budget, and the daemon fails that turn closed |
 
@@ -429,17 +431,17 @@ reassurance.
 If you're testing an actual user UX flow:
 
 ```bash
-hipfire pull qwen3.5:9b
-hipfire pull qwen3.5:9b-draft
+hipfire pull qwen3.5:9b                # target + draft sidecar in one pull
 hipfire config set dflash_mode auto    # opt in (default since 2026-04-26: off)
 hipfire run qwen3.5:9b "Write a Python function to find the longest substring without repeating characters"
-# expected: daemon logs '[hipfire] DFlash draft detected: ...'
+# expected: loader logs 'DFlash draft loaded: ...'
 # response generates at ≥250 tok/s on a 9B target with a paired draft
 ```
 
 Without the `dflash_mode auto` config, `hipfire run` runs pure AR
-even when a paired draft is on disk — the daemon explicitly logs
-`[hipfire] DFlash disabled (dflash_mode=off).` This is the "I pulled
+even when a paired draft is on disk. `dflash_mode on` instead requires
+the sidecar and fails the load when it is missing; `developer.dflash_draft`
+or `run --model-draft` overrides the sidecar. This is the "I pulled
 the draft but DFlash isn't firing" pitfall.
 
 ---
@@ -584,8 +586,8 @@ against the A3B MoE DFlash perfmaxx line.
 | 3.6-A3B DFlash 68.6 tok/s vs AR 135 tok/s (50% loss) | 3.6 draft trained on 3.5 traces; target distribution mismatch on code. τ=1.22 on hard code. | Use AR mode for 3.6-A3B. Draft mismatch is expected and no 3.6 retrain is planned — Path C (`feat/mtp-dflash-training`) is dead/out-of-scope, not a forthcoming fix. 3.5-A3B DFlash works (τ=4.91). |
 | `hipMalloc out of memory` at hidden_rb | Long ctx (≥16K real tokens) + 27B + asym3 = tight on 24 GB | Reduce ctx, use a smaller target, or wait for the bounded-rolling-buffer trick (roadmap) |
 | `tok/s` below expected on long-ctx | KV cache growth — prefill is fine but decode slows past ~2K | Test at small ctx first, then scale |
-| daemon doesn't auto-find draft | Filename doesn't match `qwen3{ver}-{size}-dflash-{quant}.hfq` | Don't rename the file after pull |
-| `[hipfire] DFlash disabled (dflash_mode=off)` | Default flipped to `off` in 35265c6 (post-2026-04-26). Pulling a draft does NOT auto-enable DFlash anymore. | `hipfire config set dflash_mode auto` (or `on`); or per-model `hipfire config qwen3.5:9b set dflash_mode on` |
+| daemon doesn't pair a pulled draft | Renamed draft file, or pulled before the sidecar existed | Don't rename files after pull; re-run `hipfire pull <tag>` to fetch the registry-declared sidecar |
+| `[hipfire-daemon] dflash_mode=off — skipping draft load` | Default flipped to `off` in 35265c6 (post-2026-04-26). Pulling a draft does NOT auto-enable DFlash anymore. | `hipfire config set dflash_mode auto` (or `on`); or per-model `hipfire config qwen3.5:9b set dflash_mode on` |
 | "Numbers don't match the README" | Forgot `HIPFIRE_NORMALIZE_PROMPT=1` (pre-2026-04-26) | Now default ON. Pull latest. If you opted out via `prompt_normalize=false`, that overrides the default — flip back. |
 | "27B DFlash regressed 30-40% suddenly" | PR #32 (cleanup-dead-wmma-kernels) on master removed `gemm_hfq4g256_residual_wmma{,2,_k4}.hip` thinking dead. Dispatch fell back to slower variants. | Verify against canonical 199 tok/s @ max=120 with default flags. If kernel files missing in `kernels/src/`, `git checkout` from a known-good commit (see commit 9a2c667 for the full recovery context). |
 | `HIPFIRE_GRAPH=1` reports plausible tok/s but output is garbage | Dangling stack-pointer kernargs from raw `self.hip.launch_kernel(...)` calls in `forward_scratch_layers` (kv_cache_write_*, attention_flash_*, fused_qkv_hfq4g256, rmsnorm_batched, rope_partial_interleaved_f32, gated_delta_net_q8, etc.) — captured pointers dangle past `end_graph_capture` | Bench tok/s alone never proves graph correctness. Always coherence-gate or eyeball under `HIPFIRE_GRAPH=1`. Fix: migrate every raw-launch helper used in forward_scratch_layers to `launch_maybe_blob` (model after `conv1d_silu_split_f32_n`). |
@@ -602,7 +604,7 @@ against the A3B MoE DFlash perfmaxx line.
 | `HIPFIRE_PROMPT_HEAT_LIMIT` | Max rows in heat dump | 64 |
 | `HIPFIRE_KV_MODE` | Override kv_cache config | (config) |
 | `HIPFIRE_ATTN_FLASH` | Override flash_mode config | (config) |
-|`HIPFIRE_DFLASH_DRAFT`|Force a specific draft path. Empty string = explicit opt-out|(filename auto-match alongside target)|
+|`HIPFIRE_DFLASH_DRAFT`|Force a specific draft path, overriding the registry sidecar. Empty string = explicit opt-out|(unset: registry sidecar when `dflash_mode` is `auto`/`on`)|
 |`HIPFIRE_DFLASH_CTX_CAP`|Max rows for draft context-indexed structures (target_hidden, draft K/V caches, hidden ring). Bounds draft-side VRAM on large-`max_seq` serve loads; over-cap requests fall back to AR (identical output, slower). `0` = uncapped legacy.|8192|
 |`HIPFIRE_DFLASH_WINDOW`|Windowed draft context (NInfer pattern): SWA over the last W rows on draft layers 0..n-2 + full-attention last layer reaching min(physical_cap, 4W). Draft VRAM pins at W regardless of `max_seq`; past-W requests degrade τ instead of falling back to AR. Refused with CASK eviction. `0`/unset = Legacy (cap + AR fallback).|0 (off)|
 | `HIPFIRE_LM_HEAD_F16` | `auto`/`native` keeps qt=1 lm_head as F16; `f32`/`legacy` expands to F32 | auto/native |

crates/hipfire-arch-gemma4/map.md

@@ -24,26 +24,26 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 | File | Lines | Public items | Tests |
 |---|---:|---:|---:|
 | [`src/arch.rs`](src/arch.rs) | 77 | 2 | 1 |
-| [`src/carrier.rs`](src/carrier.rs) | 203 | 4 | 1 |
+| [`src/carrier.rs`](src/carrier.rs) | 261 | 6 | 2 |
 | [`src/config.rs`](src/config.rs) | 880 | 19 | 12 |
 | [`src/drafter.rs`](src/drafter.rs) | 1,074 | 15 | 0 |
 | [`src/forward.rs`](src/forward.rs) | 2,665 | 6 | 4 |
 | [`src/gemma4.rs`](src/gemma4.rs) | 1,088 | 13 | 0 |
 | [`src/gemma4_vision.rs`](src/gemma4_vision.rs) | 16 | 3 | 0 |
-| [`src/lib.rs`](src/lib.rs) | 48 | 8 | 0 |
+| [`src/lib.rs`](src/lib.rs) | 51 | 8 | 0 |
 | [`src/lowered.rs`](src/lowered.rs) | 5,876 | 35 | 0 |
 | [`src/speculative.rs`](src/speculative.rs) | 252 | 6 | 0 |
 
 ### Public API surface
 
 - [`src/arch.rs`](src/arch.rs): `ARCH_ID`, `Gemma4`
-- [`src/carrier.rs`](src/carrier.rs): `Gemma4EagerBundle`, `Gemma4LoweredBundle`, `Gemma4Bundle`, `load_gemma4_bundle`
+- [`src/carrier.rs`](src/carrier.rs): `LOWERED_GENERATE_REFUSAL`, `gemma4_lowered_refusal`, `Gemma4EagerBundle`, `Gemma4LoweredBundle`, `Gemma4Bundle`, `load_gemma4_bundle`
 - [`src/config.rs`](src/config.rs): `LayerType`, `RopeType`, `Gemma4ESeriesVariant`, `Gemma4Config`, `from_hfq`, `from_metadata_json`, `n_full_layers`, `n_sliding_layers`, `n_full_kv_slots`, `n_sliding_kv_slots`, `max_head_dim`, `max_q_dim`, +7 more
 - [`src/drafter.rs`](src/drafter.rs): `DRAFTER_ARCH_ID`, `Gemma4DrafterConfig`, `from_hfq`, `pre_proj_in`, `max_q_dim`, `max_head_dim`, `DrafterLayerWeights`, `Gemma4DrafterWeights`, `load`, `free_gpu`, `Gemma4DrafterScratch`, `new`, +3 more
 - [`src/forward.rs`](src/forward.rs): `decode_step`, `decode_step_capture`, `decode_step_with_graph`, `supports_batched_prefill`, `forward_batch`, `forward_batch_spec`
 - [`src/gemma4.rs`](src/gemma4.rs): `SlidingLayerWeights`, `FullLayerWeights`, `LayerWeights`, `PerLayerBranchWeights`, `PerLayerInputWeights`, `Gemma4Weights`, `load`, `free_gpu`, `Gemma4State`, `new`, `new_with_max_seq`, `new_with_fwht3_max_seq`, +1 more
 - [`src/gemma4_vision.rs`](src/gemma4_vision.rs): `Gemma4VisionConfig`, `Gemma4VisionWeights`, `Gemma4VisionScratch`
-- [`src/lib.rs`](src/lib.rs): `arch`, `config`, `drafter`, `forward`, `gemma4`, `lowered`, `speculative`, `carrier`
+- [`src/lib.rs`](src/lib.rs): `arch`, `carrier`, `config`, `drafter`, `forward`, `gemma4`, `lowered`, `speculative`
 - [`src/lowered.rs`](src/lowered.rs): `wmma_prefill_enabled`, `batched_prefill_enabled`, `LayerType`, `RopeType`, `Gemma4Config`, `config_from_hfq`, `SlidingLayerWeights`, `FullLayerWeights`, `MoeExpertWeights`, `MoeLayerExtras`, `LayerWeights`, `Gemma4Weights`, +23 more
 - [`src/speculative.rs`](src/speculative.rs): `SpecStepOut`, `Gemma4SpecScratch`, `new`, `set_seed_hidden_from`, `free`, `spec_step_gemma4_eagle`
 
@@ -60,6 +60,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 ### Totals
 
-- 10 modules · 12,179 lines · 111 public items · 18 tests · 8 examples
+- 10 modules · 12,240 lines · 113 public items · 19 tests · 8 examples
 
 <!-- crate-map:generated:end -->

crates/hipfire-arch-gemma4/src/carrier.rs

@@ -39,6 +39,29 @@ fn gemma4_validate_drafter_route(is_e_series: bool, has_drafter: bool) -> Result
     Ok(())
 }
 
+/// Generate-time refusal for the lowered/MoE path, shared with the load-time
+/// admission below so both name the same combination and remedy. The generate
+/// body is eager-dense-only; a lowered load must never reach it.
+pub const LOWERED_GENERATE_REFUSAL: &str = "gemma4 lowered/MoE generate not yet wired on this build (eager dense only) —                  reload without batched/WMMA prefill opt-in or the MoE variant";
+
+/// Admission decision for the lowered path, before any device allocation.
+/// Pure so the contract is unit-testable. Returns the refusal reason when the
+/// (model, option) combination would select `lowered` via [`gemma4_use_lowered`],
+/// which generate cannot serve — fail here instead of after a full weight/KV
+/// upload. `None` means the eager dense path serves the combination.
+pub fn gemma4_lowered_refusal(
+    enable_moe_block: bool,
+    want_batched: bool,
+    has_drafter: bool,
+    is_e_series: bool,
+) -> Option<&'static str> {
+    if gemma4_use_lowered(enable_moe_block, want_batched, has_drafter, is_e_series) {
+        Some(LOWERED_GENERATE_REFUSAL)
+    } else {
+        None
+    }
+}
+
 // ─── Bundle types ─────────────────────────────────────────────────────────
 
 pub struct Gemma4EagerBundle {
@@ -119,6 +142,22 @@ pub fn load_gemma4_bundle(src: ModelSource, ctx: &mut LoadCtx) -> Result<Gemma4B
     } else {
         false
     };
+    // Admission: the lowered/MoE path has no generate arm (the generate body is
+    // eager-dense-only and refuses post-load). Fail here — after the host-side
+    // config/env parse but before `lowered::load_weights` (first device
+    // allocation) — instead of after a full weight/scratch/KV upload.
+    // The `if use_lowered` block below is retained for the future lowered serve
+    // path; it is unreachable while this refusal stands.
+    if let Some(lcfg) = &lowered_cfg {
+        if let Some(reason) = gemma4_lowered_refusal(
+            lcfg.enable_moe_block,
+            want_batched,
+            ctx.gemma4_drafter_path.is_some(),
+            is_e_series,
+        ) {
+            return Err(reason.to_string());
+        }
+    }
     if use_lowered {
         let lcfg = lowered_cfg.unwrap();
         let (n_sliding_layers, n_full_layers) = lowered_kv_layer_counts(&lcfg.layer_types);
@@ -200,4 +239,23 @@ mod tests {
             .collect::<Vec<_>>();
         assert_eq!(lowered_kv_layer_counts(&layer_types), (40, 8));
     }
+
+    #[test]
+    fn lowered_admission_refuses_what_generate_cannot_serve() {
+        // MoE variant: refused with the combination and remedy named.
+        let err = gemma4_lowered_refusal(true, false, false, false)
+            .expect("MoE lowered must be refused at admission");
+        assert!(err.contains("lowered/MoE"), "reason names the state: {err}");
+        assert!(err.contains("eager dense only"), "reason: {err}");
+        // Dense with batched/WMMA prefill opt-in and no drafter: refused.
+        assert!(gemma4_lowered_refusal(false, true, false, false).is_some());
+        // Adjacent supported: eager dense (no opt-in), drafter-kept-eager,
+        // and E-series all admit.
+        assert_eq!(gemma4_lowered_refusal(false, false, false, false), None);
+        assert_eq!(gemma4_lowered_refusal(false, true, true, false), None);
+        assert_eq!(gemma4_lowered_refusal(false, true, false, true), None);
+        // MoE always selects lowered even with a drafter requested, so it is
+        // refused too (the drafter route only keeps *dense* eager).
+        assert!(gemma4_lowered_refusal(true, false, true, false).is_some());
+    }
 }

crates/hipfire-arch-gemma4/src/lib.rs

@@ -28,14 +28,17 @@
 //! `gelu_tanh_f32`, `logit_softcap_f32`, plus the shared GEMV path.
 
 pub mod arch;
+pub mod carrier;
 pub mod config;
 pub mod drafter;
 pub mod forward;
 pub mod gemma4;
 pub mod lowered;
 pub mod speculative;
-pub mod carrier;
-pub use carrier::{load_gemma4_bundle, Gemma4Bundle, Gemma4EagerBundle, Gemma4LoweredBundle};
+pub use carrier::{
+    gemma4_lowered_refusal, load_gemma4_bundle, Gemma4Bundle, Gemma4EagerBundle,
+    Gemma4LoweredBundle, LOWERED_GENERATE_REFUSAL,
+};
 
 pub use arch::{Gemma4, ARCH_ID};
 pub use config::{Gemma4Config, LayerType, RopeType};

crates/hipfire-arch-qwen2/map.md

@@ -26,7 +26,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 | [`src/arch_model.rs`](src/arch_model.rs) | 45 | 0 | 0 |
 | [`src/carrier.rs`](src/carrier.rs) | 76 | 2 | 0 |
 | [`src/lib.rs`](src/lib.rs) | 85 | 5 | 0 |
-| [`src/qwen2.rs`](src/qwen2.rs) | 2,355 | 23 | 7 |
+| [`src/qwen2.rs`](src/qwen2.rs) | 2,358 | 23 | 7 |
 | [`src/spec_impl.rs`](src/spec_impl.rs) | 199 | 1 | 0 |
 
 ### Public API surface
@@ -51,6 +51,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 ### Totals
 
-- 6 modules · 2,849 lines · 32 public items · 8 tests · 4 examples
+- 6 modules · 2,852 lines · 32 public items · 8 tests · 4 examples
 
 <!-- crate-map:generated:end -->

crates/hipfire-arch-qwen2/src/qwen2.rs

@@ -276,26 +276,29 @@ impl Qwen2Weights {
 
     /// Release every GPU buffer back to the pool. Consumes self.
     /// Mirrors `LlamaWeights::free_gpu` and `Qwen35Weights::free_gpu`
-    /// — the daemon calls this on unload to actually return VRAM.
+    /// — the daemon calls this on unload to actually return VRAM. Linear
+    /// weights go through `WeightTensor::free_all` so any PARO / AWQ sidecar
+    /// (none are allocated for Qwen2 today, but the forward already reads
+    /// `awq_scale`) is released with the buffer instead of leaking per reload.
     pub fn free_gpu(self, gpu: &mut Gpu) {
         let _ = gpu.free_tensor(self.token_embd);
         let _ = gpu.free_tensor(self.output_norm);
         if !self.tied_lm_head {
-            let _ = gpu.free_tensor(self.output.buf);
+            self.output.free_all(gpu);
         }
         for l in self.layers {
             let _ = gpu.free_tensor(l.attn_norm);
-            let _ = gpu.free_tensor(l.wq.buf);
+            l.wq.free_all(gpu);
             let _ = gpu.free_tensor(l.wq_bias);
-            let _ = gpu.free_tensor(l.wk.buf);
+            l.wk.free_all(gpu);
             let _ = gpu.free_tensor(l.wk_bias);
-            let _ = gpu.free_tensor(l.wv.buf);
+            l.wv.free_all(gpu);
             let _ = gpu.free_tensor(l.wv_bias);
-            let _ = gpu.free_tensor(l.wo.buf);
+            l.wo.free_all(gpu);
             let _ = gpu.free_tensor(l.ffn_norm);
-            let _ = gpu.free_tensor(l.w_gate.buf);
-            let _ = gpu.free_tensor(l.w_up.buf);
-            let _ = gpu.free_tensor(l.w_down.buf);
+            l.w_gate.free_all(gpu);
+            l.w_up.free_all(gpu);
+            l.w_down.free_all(gpu);
         }
     }
 }

crates/hipfire-arch-qwen35/map.md

@@ -42,7 +42,7 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 | [`src/qwen35/ep_batch.rs`](src/qwen35/ep_batch.rs) | 4,800 | 20 | 7 |
 | [`src/qwen35/forward.rs`](src/qwen35/forward.rs) | 6,255 | 31 | 12 |
 | [`src/qwen35/load.rs`](src/qwen35/load.rs) | 4,906 | 10 | 0 |
-| [`src/qwen35/prefill.rs`](src/qwen35/prefill.rs) | 9,312 | 11 | 48 |
+| [`src/qwen35/prefill.rs`](src/qwen35/prefill.rs) | 9,381 | 11 | 49 |
 | [`src/qwen35/weights.rs`](src/qwen35/weights.rs) | 1,971 | 43 | 10 |
 | [`src/qwen35.rs`](src/qwen35.rs) | 63 | 7 | 0 |
 | [`src/scheduler.rs`](src/scheduler.rs) | 142 | 3 | 4 |
@@ -97,6 +97,6 @@ _Generated by `scripts/check-crate-maps.py` from the tree — do not edit inside
 
 ### Totals
 
-- 29 modules · 57,281 lines · 436 public items · 189 tests · 4 examples
+- 29 modules · 57,350 lines · 436 public items · 190 tests · 4 examples
 
 <!-- crate-map:generated:end -->