Skip to content

fix(runtime): make DFlash construction transactional - #736

Closed
fivetide wants to merge 1 commit into
warpfront:betafrom
fivetide:replan/g4-dflash-construction
Closed

fix(runtime): make DFlash construction transactional#736
fivetide wants to merge 1 commit into
warpfront:betafrom
fivetide:replan/g4-dflash-construction

Conversation

@fivetide

@fivetide fivetide commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Problem

DflashWeights::load and the base/windowed DflashScratch constructors transfer GPU owners through multi-step construction without RAII. Ordinary late errors could strand completed layers, staged weights, optional scratch, or window-extension allocations.

Change

  • stage global/per-layer DFlash weights until complete publication and roll back completed layers plus unpublished owners on error
  • upload raw DFlash weights through the reusable GPU pool, returning the allocation if H2D copy fails
  • stage every base-scratch allocation and free the ledger in reverse order on failure
  • keep the fully owned base scratch as the window-extension rollback owner, including the replaced position buffer
  • add private deterministic boundaries for final-owner and final-layer fault injection; public constructors and outer GenericDflashStaging / Qwen35 or_free! ownership remain unchanged
  • add ignored gfx1151 regressions for late weight, completed-layer, base-scratch, and window-extension failure followed by immediate retry

Verification

Final head 53bba32a5 on upstream/beta fdb750d6d:

  • cargo test -p hipfire-runtime --lib construction_tests -- --ignored --test-threads=1 --nocapture: 4 passed on gfx1151
  • cargo test -p hipfire-runtime --lib: 609 passed, 4 ignored
  • scoped rustfmt: clean
  • cargo clippy -p hipfire-runtime --lib --no-deps: completed with repository warnings only
  • cargo build and cargo build --release: passed
  • runtime crate map regenerated and checker clean
  • static ownership review: no remaining blocking finding

Hardware/lifecycle:

  • GPU: AMD Radeon 8060S, gfx1151, HSA runtime 1.18
  • target: /home/bjoern/.hipfire/models/qwen3.5-27b.mq4, md5 e1c9480a3fa54ad0a0f25ca18105510a
  • draft: /home/bjoern/.hipfire/models/qwen35-27b-dflash-mq4.hfq, md5 7b6df2a4ee1c8d933f0a52e187d1860b
  • release hipfire md5 e101abe74e886fd845dd42215e1b59a8; daemon md5 1e1cba2b78dc7796a276c020da56f059
  • paired DFlash serve_harness.py battery: 5/5 coherent inspected outputs, draft loaded, every request logged drafter=dflash
  • a fresh-daemon paired DFlash chain immediately afterward: 5/5 coherent inspected outputs, draft reloaded, every request logged drafter=dflash, prefix checkpoint reuse observed
  • both harnesses terminated their daemons after the run, exercising target/draft unload between the battery and chain reload

Scope

This is G4.2 only. It does not replace outer DFlash staging, change successful generation semantics, or claim malformed-HFQ panic recovery. G4 milestone acceptance remains blocked on the remaining tracker tasks and the final composed-head matrix.

@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

Holding #736 after independent review of the beta-composed candidate22f7e2ba6. Two real leaf paths still escape the claimed construction transaction: (1) hfq_tensor_f32 plus F16-lift/F32 matrix arms use Gpu::upload_f32, which allocates then returns on memcpy_htod failure without freeing the new owner before it reaches gt!/wt! staging; (2) AWQ scale attachment uses load_awq_scale -> direct-HIP upload_raw(...).ok(), leaking on copy failure and silently converting failure to missing scale. Successful direct allocations also bypass the pool, so pool_stats-only retry tests do not prove the AWQ invariant. Please make these uploads pool-aware/free-on-copy-failure and propagate sidecar errors, with a focused after-allocation copy-failure regression. Move the new changelog line to Unreleased as well. The reviewer specifically found the scratch tests current before-retry baselines valid for their final-allocation seams; do NOT move them based on our preliminary hypothesis. No #736 code has landed.

@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

Maintainer-integrated into beta at 24fc65c, preserving the original change with cherry-pick provenance and adding the review-required corrections. This closes the original PR as ported, not as an unmodified merge.

Fixes: F32/raw leaf uploads return their pool allocation on copy failure; DFlash AWQ sidecars use pooled allocation and propagate errors while releasing the trunk; changelog entry moved to Unreleased. Leaf regression oracle warms two simultaneously live slots before testing cumulative allocation counts (the original empty-pool baseline was invalid).

Parent-run proof on gfx1201 / Radeon AI PRO R9700 / HIP 7.15:

  • All 6 construction rollback tests passed, including all ignored GPU cases.
  • Fresh isolated release CLI+daemon build passed.
  • Canonical Qwen3.8-27B MQ4XT with pinned MQ4V2 draft: serve battery 5/5 and related-turn chain 5/5, DFlash active throughout, no empty/attractor/runaway flags. Read all decoded answers; coherent. Chain exercised cached-prefix reuse (256 tokens).
  • Changed Rust formatting and generated runtime map checks passed.

Trunk SHA256: 9f91556f7e0431a077d03756a7102d0154108757289e6e5fe9a2d204c0c9eeb7
Draft SHA256: d0a74a232a0e2166d889f823e91e0fbf778d21dd9668d7de055cdecb065401bc
Served CLI MD5: 24a62a65fbc11b3367adc8545f2cb6ac
Served daemon MD5: d1295ea923d9d74cd091364b5dce68f8
Served build ecdc606 is production-identical to final head apart from formatting; subsequent changes affect tests/map only.
Local JSON receipts: /home/kaden/ClaudeCode/warpfront/_pr_smokes/repair736-{battery,chain}.json. No performance claim and no master promotion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants