Skip to content

fix(qwen35): retain construction owners until publication - #735

Closed
fivetide wants to merge 2 commits into
warpfront:betafrom
fivetide:replan/g4-qwen35-construction
Closed

fix(qwen35): retain construction owners until publication#735
fivetide wants to merge 2 commits into
warpfront:betafrom
fivetide:replan/g4-qwen35-construction

Conversation

@fivetide

@fivetide fivetide commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

G4.1b — Qwen35 construction ownership

Depends on #731. This PR is stacked on its prefill-scratch ownership fix; once #731 lands, this diff reduces to the remaining decode-batch and per-layer construction transactions.

Changes

  • stage the seven Qwen35DecodeBatchState output tensors as actual owners until complete publication; no borrowed aliases participate in rollback
  • stage every norm/raw/projection/MoE owner across all four load_layer variants
  • add the narrow WeightBackend::free_tensor rollback seam for HFQ and PaRo backends
  • preserve packed-expert, AWQ, PaRo shared-sidecar, EP dummy, and paged empty-expert ownership in the authoritative MoE cleanup helper
  • add deterministic real-GPU late-failure/retry regression coverage for decode, dense-layer, and MoE-boundary construction
  • update crate maps and the v0.3.1 changelog entry

Final-head evidence

Head: 229f5f54f8771e41eb81dd35cb18a5361f6a6ae0 (rebased onto current beta).

  • cargo test -p hipfire-runtime --lib: 609 passed
  • cargo test -p hipfire-arch-qwen35 --lib: 191 passed, 4 ignored
  • each of the 3 new ignored real-GPU rollback tests: passed on gfx1151
  • cargo build --release: passed
  • scoped rustfmt check: passed
  • crate-map check: passed
  • dense Qwen3.5-27B DFlash battery: 5/5 coherent decoded outputs, 0 empty, 0 attractors; daemon confirmed the draft loaded and drafter=dflash for all requests
  • Qwen3.6-35B-A3B AR battery: 5/5 coherent decoded outputs, 0 empty, 0 attractors
  • after each battery unloaded its daemon, the same model immediately reloaded and decoded the committed merge-sort fixture (253c7ac50857fe6d0e10fb0d2c5e35c0) coherently

Final artifacts:

  • hipfire md5: 09cc1504492ff0628608a58434ce7c60
  • daemon md5: c6db7ff700b3b17b48461736e6f584f5
  • Qwen3.5-27B target md5: e1c9480a3fa54ad0a0f25ca18105510a
  • Qwen3.5-27B draft md5: 7b6df2a4ee1c8d933f0a52e187d1860b
  • Qwen3.6-35B-A3B target md5: edde51ec1dac0f2bd42cff5ef1cb8944

Scoped cargo clippy --no-deps exits successfully with the repository's existing warning set. The advisory workspace CI Clippy job fails on the pre-existing approx_constant lint in examples/test_dflash_hidden_scatter_gfx1100.rs; required build, unit-test, rustfmt, cargo-deny, and gate jobs pass.

Bjoern Agent added 2 commits September 8, 2026 17:14
The rollback ledger stored DeviceBuffer::alias() views, which free_tensor
refuses to release. Stage real GpuTensor owners and publish them only after
every mandatory/optional allocation succeeds; unwind failures in reverse.

GPU fault regression on gfx1151/HIP7.2: warm54 allocations; fail the final
allocation and retry. Before:107 fresh allocations (53 buffers lost).
After:54, full warm-pool reuse. Default qwen35 lib tests191 passed/1 ignored;
the ignored GPU fault test was explicitly run and passed. Scoped clippy
completed with existing warnings; crate maps43/43, ratchets21/21 pass.

Rebuilt CLI/daemon, verified qwen3.6:27b DFlash factual+5-turn chain and
qwen3.6:35b-a3b AR factual:7 turns stop, no empty/runaway/attractor outputs.
Matched128-token DFlash request is byte-identical before/after (both hit
that explicit cap); separate512-token run completes at148 tokens.

Evidence scope: first independent G4 allocation-error fix, not G4 milestone
acceptance, numerical/state admission, physical multi-GPU, or performance.
@fivetide
fivetide force-pushed the replan/g4-qwen35-construction branch from bd653c5 to 229f5f5 Compare September 8, 2026 15:21
@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

Landed into beta as800ddf740: original exclusive229f5f54 cherry-picked with -x after #731, avoiding duplicate prefill commit7b599a84. Independent static review approved. Exact composed candidate246a26f passed all4 GPU rollback regressions on gfx1201/HIP7.15 in a unique Cargo target (an earlier stale shared-target run was discarded). Dense canonical XT+DFlash battery5/5 and chain5/5 passed with decoded text inspected; Ornith MQ4R AR battery5/5 passed at2048 tokens with replay-route-proof logging enabled. Initial256-token Ornith run had2 reasoning-budget-truncated/empty turns and is NOT a passing receipt. Candidate-to-landed source diff contains only separately reviewed #738/#740 tooling fixes. Scope caveat: nested MoE constructors can still leak before returning an owner; that predates and lies outside this bounded returned-owner publication fix. #736 remains held for separate leaf-copy/AWQ atomicity defects.

@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

Decoded output receipts for the #731+#735 composed candidate (not a performance comparison). All three accepted runs inspected manually; original JSON retained under /home/kaden/ClaudeCode/warpfront/_pr_smokes/. CLI MD5 b95f46207e8629c46deccadf67599b35; daemon ee68c70fdb50200c663b38a63ab77772.

Canonical XT DFlash battery (256 tokens)

Turn1, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=43ca0d15712d3dfb777b51ae76d8fd5f

Decoded assistant answer
def merge_sorted(a, b):
    """Merge two sorted lists into a single sorted list."""
    result = []
    i, j = 0, 0

    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1

    result.extend(a[i:])
    result.extend(b[j:])

    return result

Turn2, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=640e0fd4f55996cb175a422f0a12cef5

Decoded assistant answer

Step 1: Find the distance for the first part.

[
\text{Distance} = \text{Speed} \times \text{Time}
]

[
60 \text{ mph} \times 2.5 \text{ hours} = 150 \text{ miles}
]

Step 2: Find the distance for the second part.

[
40 \text{ mph} \times 1.5 \text{ hours} = 60 \text{ miles}
]

Step 3: Add the distances together.

[
150 \text{ miles} + 60 \text{ miles} = 210 \text{ miles}
]

Final answer: 210 miles

Turn3, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=8f66b4c97988825bd8e7840aaf44357e

Decoded assistant answer

Earth's axis is tilted about 23.5 degrees relative to its orbit around the Sun. As Earth orbits the Sun, different hemispheres receive more direct sunlight and longer days at different times of the year. This variation in sunlight intensity and day length causes the seasons.

Turn4, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=8fe0ad36f61bcf4992cc9df81cdf3817

Decoded assistant answer

Elias, the lighthouse keeper, found a glass bottle balanced on the black rocks after the storm. Inside was not a message, but a tiny silver key, warm as if it had been held just now. He walked to the locked sea door he had never dared to open. The key fit, and beyond it the sea glowed with lights he had only ever dreamed of.

Turn5, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=8bed8e2d056dc1d47dccae9d32dbecf4

Decoded assistant answer
  1. Write clear, descriptive names for functions, variables, and modules.
  2. Keep functions small and focused on a single responsibility.
  3. Add tests to verify behavior and guard against regressions.
  4. Document non-obvious decisions and complex logic with concise comments.
  5. Refactor regularly to remove duplication and simplify structure.

Canonical XT DFlash chain (2048 tokens)

Turn1, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=43ca0d15712d3dfb777b51ae76d8fd5f

Decoded assistant answer
def merge_sorted(a, b):
    """Merge two sorted lists into a single sorted list."""
    result = []
    i, j = 0, 0

    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1

    result.extend(a[i:])
    result.extend(b[j:])

    return result

Turn2, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=640e0fd4f55996cb175a422f0a12cef5

Decoded assistant answer

Step 1: Distance at 60 mph for 2.5 hours
[
60 \times 2.5 = 150 \text{ miles}
]

Step 2: Distance at 40 mph for 1.5 hours
[
40 \times 1.5 = 60 \text{ miles}
]

Step 3: Total distance
[
150 + 60 = 210 \text{ miles}
]

Final answer: 210 miles

Turn3, finish=stop, cached=256, empty=False, attractor=False, runaway=False, prompt_md5=8f66b4c97988825bd8e7840aaf44357e

Decoded assistant answer

Earth's axial tilt of about 23.5 degrees causes different hemispheres to receive varying angles of sunlight throughout the year. As Earth orbits the Sun, the tilted axis makes one hemisphere lean toward the Sun, producing summer, while the other leans away, producing winter. This cycle repeats annually, creating the pattern of spring, summer, autumn, and winter.

Turn4, finish=stop, cached=256, empty=False, attractor=False, runaway=False, prompt_md5=8fe0ad36f61bcf4992cc9df81cdf3817

Decoded assistant answer

Every morning, Elias checked the rocks below his lighthouse for barnacles, broken glass, and the usual driftwood. One gray dawn, he found a brass compass tied to a bottle, its needle still trembling despite the tide. The glass label bore his own name, written in his late father's hand. He turned it over, and on the reverse was tomorrow's date in fresh ink.

Turn5, finish=stop, cached=256, empty=False, attractor=False, runaway=False, prompt_md5=8bed8e2d056dc1d47dccae9d32dbecf4

Decoded assistant answer
  1. Write small functions that do one thing clearly.
  2. Use descriptive names for variables, functions, and files.
  3. Keep functions short and avoid deep nesting.
  4. Add meaningful comments only where logic is unclear.
  5. Write tests to protect behavior and support refactoring.

Ornith MQ4R AR battery (2048 tokens)

Turn1, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=43ca0d15712d3dfb777b51ae76d8fd5f

Decoded assistant answer

Here's a clean implementation using the classic two-pointer "merge" approach (the same step merge sort uses):

def merge_sorted(a, b):
    """Merge two sorted lists into a single sorted list without using sorted()."""
    result = []
    i = j = 0

    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1

    # Append whatever remains (one of the lists will be exhausted already)
    result.extend(a[i:])
    result.extend(b[j:])
    return result

How it works:

  • Two pointers i and j walk through lists a and b simultaneously.
  • At each step, whichever element is smaller is appended, keeping the result sorted.
  • When one list is exhausted, the rest of the other list is simply appended.

Example:

print(merge_sorted([1, 3, 5], [2, 4, 6]))   # [1, 2, 3, 4, 5, 6]
print(merge_sorted([], [1, 2]))             # [1, 2]
print(merge_sorted([1, 2], [2, 3]))         # [1, 2, 2, 3]

This runs in O(n + m) time, which is optimal since every element must be visited at least once.

Turn2, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=640e0fd4f55996cb175a422f0a12cef5

Decoded assistant answer

Train Distance Calculation

Step 1: First leg of the trip

$$60 \text{ mph} \times 2.5 \text{ hours} = 150 \text{ miles}$$

Step 2: Second leg of the trip

$$40 \text{ mph} \times 1.5 \text{ hours} = 60 \text{ miles}$$

Step 3: Add both distances together

$$150 \text{ miles} + 60 \text{ miles} = 210 \text{ miles}$$

Final Answer

The train traveled a total of 210 miles.

Turn3, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=8f66b4c97988825bd8e7840aaf44357e

Decoded assistant answer

The seasons on Earth are primarily caused by the approximately 23.5-degree tilt of Earth's axis relative to its orbital path around the Sun. Because this axis remains pointed in a fixed direction as Earth orbits, different hemispheres are angled toward or away from the Sun at different times of the year. When a hemisphere is tilted toward the Sun, it receives more direct sunlight and longer days, producing summer, while a tilt away from the Sun creates the colder conditions of winter.

Turn4, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=8fe0ad36f61bcf4992cc9df81cdf3817

Decoded assistant answer

For twenty years, old Tomás had patrolled the same black rocks at low tide, knowing every splintered hull and torn net the sea chose to return. But on this bitter November morning, something glimmered among the barnacled stones that no storm could have carried. He knelt in the freezing spray and brushed away the seaweed to reveal a small wooden chest, its hinges fused by salt and its lid carved with the name of a ship he had buried decades ago. Inside, wrapped in oilcloth, lay a letter addressed in his own young man's hand, and a single silver ring he had thrown into the dark the night he left home.

Turn5, finish=stop, cached=0, empty=False, attractor=False, runaway=False, prompt_md5=8bed8e2d056dc1d47dccae9d32dbecf4

Decoded assistant answer
  1. Keep functions small and focused, each doing a single job so they're easy to understand and test.
  2. Use descriptive names for variables, functions, and modules that reveal their intent.
  3. Write consistent, minimal comments that explain "why" rather than restating the code.
  4. Avoid duplication by extracting reusable logic instead of copying and pasting.
  5. Follow a consistent code style and run automated formatting tools to keep things uniform.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants