Skip to content

perf(draft): pack dynamic-convolution projections across lanes - #657

Closed
Graffioh wants to merge 20 commits into
Luce-Org:mainfrom
Graffioh:codex/qwen38-dflash2-dynconv-pack
Closed

perf(draft): pack dynamic-convolution projections across lanes#657
Graffioh wants to merge 20 commits into
Luce-Org:mainfrom
Graffioh:codex/qwen38-dflash2-dynconv-pack

Conversation

@Graffioh

@Graffioh Graffioh commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Depends on #656. Review the top commit, 00ba24f1; the diff collapses to this mechanism after #656 merges.

Summary

  • pack normalized hidden columns across active draft lanes before the shared attention and MLP dynamic-convolution coefficient projections
  • slice projected coefficients back into lane views before every convolution apply
  • keep convolution history, temporal shifts, positions, masks, cache writes, RoPE, attention, and apply steps lane-local
  • retain the singular projection path for C1, preserving the existing graph
  • add a real-model multilane isolation check with distinct inputs and positions

The production change is 25 additions and 6 removals in one file. It retains no legacy multilane runtime path.

R9700 fixed-C A/B/A

Qwen3.8 27B IQ4_XS target, Q8 DFlash2 width 8, Q8 KV, HumanEval raw code prompts, ten fixed-width waves, 256 forced output tokens, temperature 0. Normal runs provide latency and goodput; matched ROCprof runs provide dispatch and device-time evidence.

C Baseline A draft p50 Candidate Baseline A2 Improvement vs weaker baseline Dispatch delta / fixed round
1 5.3470 ms 5.3441 ms 5.3488 ms 0.06% 0
2 6.8028 ms 6.5815 ms 6.7946 ms 3.14% -10
3 8.6155 ms 8.2460 ms 8.6070 ms 4.19% -20
4 10.2278 ms 9.6294 ms 10.2264 ms 5.84% -30

The mean minimum A/B/A draft-compute improvement at C2-C4 is 4.39%. Complete output digests match at C1-C4, all 300 requests return 256 tokens, and captures report zero dropped records.

Against the A/A2 bracket mean, end-to-end goodput improves by 0.79%, 0.23%, 0.43%, and 0.47% at C1-C4. Those are marginal system gains. The keep decision rests on the small production diff plus two separately measured effects: materially lower draft compute and ten fewer speculative dispatches per added lane.

The launch accounting is exact per added lane: ten Q8 coefficient-input quantize launches and ten coefficient MMQ launches disappear, while ten packing concat launches are added. Fixed-C elementwise device time rises by 0.88%, 1.36%, and 1.56% at C2-C4, but coefficient GEMM time falls enough for the draft phase to win. TTFT is mixed and is not a justification for this change.

Strix Halo

The same normal A/B/A protocol passes output parity, fixed-token accounting, capture integrity, and lower-C drain accounting. Its timing is performance-inconclusive: the unchanged C1 path moves by 8.65% against the bracket mean and A/A2 draft drift reaches 15.47%. It is retained as correctness and harness evidence, not as support for the performance decision.

Verification

  • exact HIP build
  • test_inference_profile
  • test_observability
  • test_qwen35_roctx
  • test_seq_engine_contract (15 checks)
  • real Q8 three-lane dynamic-convolution isolation check; all lane outputs have max absolute difference 0
  • R9700 C1-C4 normal A/B/A and matched ROCprof comparison
  • Strix Halo C1-C4 normal A/B/A integrity comparison
  • git diff --check

The device-resident selector handoff and the R9700 target MMQ C2-to-C3 cliff remain separate follow-ups.

Review in cubic

Restore the paged-attention C ABI through a separate tree entry point. Reject unsupported GDN variants, initialize root-only chain trees, and size draft metadata from graph capacity.
Keep GDN journals inside the result buffer, restore the 10-source tensor layout, validate every tree destination before mutation, and allow selector widths that use the CPU top-k fallback.
Validate singleton selector vocabularies and token IDs before codebook gathers. Reject unsupported GDN output layouts on Vulkan, and use overflow-safe ceil division in paged attention.
Preserve paged-KV allocation order across rollback and destroy draft graphs before park frees their weights.

Reject invalid GDN replay layouts and malformed SWA metadata. Exercise both grouped and scalar GDN dispatch paths.
Add fixed-width DFlash2 concurrent decoding for Qwen3.8 and batch the shared dense draft projections across active lanes.

Keep convolution, cache updates, RoPE, masks, and attention lane-local. Preserve transactional state promotion and fixed-chain validation through C6.
@Graffioh

Copy link
Copy Markdown
Contributor Author

Folded into #659. The surviving head e9103d7 contains this PR's exact dynamic-convolution packing commit (00ba24f), preserves its causal evidence, and passes the combined stack checks. Closing this draft to keep one #651 concurrency follow-up.

@Graffioh Graffioh closed this Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant