diff --git a/.gitignore b/.gitignore index 966acb97a854..a2b2735d7309 100644 --- a/.gitignore +++ b/.gitignore @@ -203,3 +203,39 @@ Makefile tests/perf-baseline/**/*.log tests/perf-baseline/**/rocprof-*/ tests/perf-baseline/calibration-logs/ + +# --- rdna4_fp8_gemm / dvgpr spike: local build artifacts, scratch + result logs (2026-07) --- +# Tracked source (.s/.cpp/.cu/.cuh, CMakeLists.txt, *_DESIGN.md, etc.) is added explicitly; +# everything below is local-only and must stay out of the tree. +# repo-root run dumps +/results.db +/results.json +/results.sysinfo.txt +# scoped scratch doc + build output dirs +ggml/src/ggml-cuda/aiter-integration/vulkan-port-scope.md +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/out/ +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/bench/gemm_fp8_levers +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_pm4/ref_gfx12/ +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/build/ +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/rga_out/ +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/rga_ml8/ +# compiler/assembler dumps, disassembly, objects, scratch copies, generated viz (dvgpr_occ) +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/*.err +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/*.amdisa +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/*.co +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/*.new +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/phase_profile.html +# result / benchmark logs (explicit prefixes so tracked CMakeLists.txt / TRITON_PIN.txt survive) +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/dynfull_*.txt +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/fairrace_*.txt +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/mbml8_*.txt +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/nofeed_*.txt +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/o_pf_yield_test.txt +# compiled sim/probe binaries + their scratch sources +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/lean_bounds_sim +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/lean_bounds_sim.cpp +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/simdprobe +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/simdprobe.hip +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/rocprof_smoke +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/rocprof_smoke.hip +ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/t_ctrl diff --git a/docs/superpowers/plans/2026-07-01-dsws-phaseb-pool-economy.md b/docs/superpowers/plans/2026-07-01-dsws-phaseb-pool-economy.md new file mode 100644 index 000000000000..87f9ec45f27c --- /dev/null +++ b/docs/superpowers/plans/2026-07-01-dsws-phaseb-pool-economy.md @@ -0,0 +1,417 @@ +# DSWS Phase-B Pool Economy Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make a converted wave execute its new role's code, and turn the fixed +8-wave partition into a lean-start grow-into-budget pool that rebalances its +{compute, A-feed, B-feed} mix at runtime. + +**Architecture:** One scalar (`s59`)-driven dispatcher handles **re-dispatch** +(reached from a `_quiesce` bail; lands on each role's `_follow`, scalar-only). +**First-time entry is DISTINCT** — it seeds `s59` then runs the full role entry +(`.Lcompute`/`.Lafeed`/`.Lbfeed` → `_alloc` → `_init` → `_follow`); it must NOT +reuse `.Ldispatch`, which skips `_alloc`/`_init` (the Pool-T7 brick: corrected +2026-07-02, committed `b01c722df`). Every wave already launches lean-32 (compute +grows to `NFV` per-rowblk on demand and shrinks back — Phase-A behavior); +"seeding" a role is therefore just setting `s59` in the existing lean partition +arms and falling into that role's existing entry, with **no launch-time grow +added**. The only `s_alloc_vgpr` GROW in the whole design remains `conv_apply`'s, +already audited. +`BUDGET` is retuned to the real per-SIMD ceiling so feed→compute grows can +succeed (bidirectional balancing within the launched mix). A cooldown `K` damps +thrash; a `DSWS2_FORCE` hook gives a deterministic first GPU proof. All offline +tasks precede three SUPERVISED GPU gates. Scaling the *launched* wave count +above the current mix sum (the 12–16-wave "bigger pool") is a follow-on once +conversions are silicon-proven — it needs an `occ_dispatch` (not-ours) dims +change and is out of scope here. + +**Tech Stack:** Hand-written RDNA4/gfx1201 (wave32) assembly (`occ_kernel_dsws.s`), +a C++ CPU control model (`dsws_ctrl_model.cpp` + gtest-free asserts), the +`build_dsws.sh` / `rga_check.sh` offline gates, and the `occ_dispatch` PM4 +harness for supervised GPU runs. Base: committed HEAD `f6dda4ccf`, branch +`feat/dsws-phaseb-conversion`. + +## Global Constraints + +- A GPU brick is a **BUG**, never a tax. Any hang/DMESG-fault/oracle-`bad>0`/DIAG-mismatch = **full STOP + report + bisect**; NEVER auto-advance to the next variant. +- **kmbandy greenlights EVERY GPU dispatch individually.** Tasks 7–9 each STOP for explicit go before touching the GPU. +- Supervised GPU dispatch env only: `ML8_POOL=1 ML8_COOP_CHUNK=8 ML8_COOP_CHUNK_MAXS=0.75 ML8_COOP_STREAM=1`, `timeout 30`. **NEVER pass `--gl2c`.** `ML8_COOP_STREAM=1` always. +- **No `s_barrier`** anywhere (mixed dyn-VGPR + `s_barrier` hard-deadlocks). LDS-atomic busy-wait only. +- `occ_kernel_coop.s` is **NEVER modified** — known-good reference. All work is additive in `occ_kernel_dsws.s`. +- **All new code gated under `.if DSWS2_CONV`.** With `DSWS2_CONV=0` the assembled object MUST be byte-identical (sha256) to before the change. This is a per-task gate. +- **OOR-poison rule (SPEC §4, #1 brick risk):** any VGPR temp with index > v15 read *before* an `s_alloc_vgpr` GROW completes is poison. Every pre-grow LDS/atomic temp must be ≤ v15; scalar temps ≤ s65. +- **Reserved-register contract:** `occ_sample` clobbers s60/s61(+dst s55/s56); `try_gate` clobbers s62–s65 + v5/v6/v7; `reserve_try` clobbers s62/s63; `conv_dec_floor` clobbers s52/s53/s65 + v5/v6/v7; `conv_apply` clobbers s52/s53/s54. Persistent Phase-B scalars s57(dir)/s58(s_win)/s59(role) sit outside all clobbers. Whole-kernel-live: s67(mask)/s68(shift)/s69(chunkHi, claimer)/s24(wid)/s35(last-epoch). **s66 (n_kseg) is dead after the prologue** and is the designated free persistent scalar for the cooldown counter. +- Config held at G=6, SEGK=64, FM=2, FN=4. `NFV=112`, `VLEAN=32`, conversion delta `NFV−VLEAN=80`. Role slot ids: `NCOMP_SLOT=24 / NAFEED_SLOT=28 / NBFEED_SLOT=32`. +- **Commit to git only when kmbandy explicitly asks.** Otherwise leave the tree dirty and report (overrides the per-task commit step below — do the `git add`/`commit` step only on request). +- Pre-existing dirty files (`occ_kernel_coop.s`, `occ_dispatch.cpp`, `fp8_oracle.*`, `mt_pagedattn*`) are **not ours** — never stage them. +- Shell is **fish** — run any `read <<<` / `set -- $var` snippet via `bash script.sh`, not directly. +- Round-table discipline: implement → adversarial review → kmbandy greenlights each GPU dispatch. + +--- + +## File Structure + +- **`dsws_ctrl_model.cpp`** + **`test_dsws_ctrl_model.cpp`** (modify) — CPU source of truth. Add: `role_dispatch()` (slot id → role enum), `cooldown_step()` (per-wave counter), `seed_mix()` + pool invariant `pool_fits_lean()`, and a `quiesce_ready` variant parameterized on `N_POOL`. Tests transcribe the asm's decisions. +- **`occ_kernel_dsws.s`** (modify, additive, all under `.if DSWS2_CONV`) — the `.Ldispatch` trampoline; the 3 tail-branch replacements; the `s59` seed assignment in the existing lean partition arms (no launch grow); the cooldown gate on the decision; the `DSWS2_FORCE` hook; new defsyms `CONV_COOLDOWN / DSWS2_FORCE / DSWS2_FORCE_WID / DSWS2_FORCE_DIR / DSWS2_FORCE_EPOCH`, and a `BUDGET` retune knob. **No `N_POOL`/`SEED_*` defsyms** — pool size == launched mix sum (`WAVES`); seed == launch partition. +- **`build_dsws.sh`** (modify) — thread `CONV_COOLDOWN/BUDGET` and the `DSWS2_FORCE*` defsyms through `mk2()`; keep no-conversion bins buildable. +- **`rga_check.sh`** (already `KSRC`-overridable) — used as-is for the live-VGPR/spill gate. +- **`scratchpad/run_pool_gates.sh`** (create, GPU tasks only) — supervised-gate driver (env above, `timeout 30`, dmesg check, abort-on-anomaly), modeled on the Phase-A `run_mix_gates.sh`. + +--- + +## Task 1: CPU control model — dispatch, cooldown, seed/pool invariants, N_POOL quiesce + +**Files:** +- Modify: `dsws_ctrl_model.cpp`, `test_dsws_ctrl_model.cpp` + +**Interfaces:** +- Produces: `enum Role { COMPUTE, AFEED, BFEED }`; `Role role_dispatch(uint32_t slot_id)` (24→COMPUTE, 28→AFEED, 32→BFEED); `uint32_t cooldown_step(uint32_t cd)` (saturating `cd?cd-1:0`); `bool in_cooldown(uint32_t cd)`; `bool pool_fits_lean(uint32_t n_pool, uint32_t vlean, uint32_t budget)` (`n_pool*vlean <= budget`); `bool quiesce_ready_pool(uint32_t quiesce_cnt, uint32_t n_pool)` (`quiesce_cnt >= n_pool-1`). These mirror the asm exactly. + +- [ ] **Step 1: Write the failing tests** (append to `test_dsws_ctrl_model.cpp`, inside `main()` before the final PASS print): + +```cpp + // ---- Task 1: dispatch + cooldown + pool invariants ---- + assert(role_dispatch(24) == COMPUTE); + assert(role_dispatch(28) == AFEED); + assert(role_dispatch(32) == BFEED); + // cooldown counts down and saturates at 0; in_cooldown true iff >0 + assert(cooldown_step(3) == 2 && cooldown_step(1) == 0 && cooldown_step(0) == 0); + assert(in_cooldown(1) && !in_cooldown(0)); + // no-parking budget invariant: 16 lean waves fit iff budget >= 512 + assert( pool_fits_lean(16, 32, 512)); + assert(!pool_fits_lean(16, 32, 511)); + // quiesce cross-check generalizes WAVES-1 -> N_POOL-1 + assert( quiesce_ready_pool(11, 12) && !quiesce_ready_pool(10, 12)); + printf("dsws_ctrl_model: dispatch/cooldown/pool OK\n"); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `cd && g++ -std=c++17 -O2 -pthread test_dsws_ctrl_model.cpp -o /tmp/test_dsws_ctrl 2>&1 | head` +Expected: FAIL — `role_dispatch`/`cooldown_step`/`pool_fits_lean`/`quiesce_ready_pool` not declared. + +- [ ] **Step 3: Implement in `dsws_ctrl_model.cpp`** + +```cpp +enum Role { COMPUTE, AFEED, BFEED }; +inline Role role_dispatch(uint32_t slot_id) { + return slot_id == 24 ? COMPUTE : (slot_id == 28 ? AFEED : BFEED); +} +inline uint32_t cooldown_step(uint32_t cd) { return cd ? cd - 1 : 0; } +inline bool in_cooldown(uint32_t cd) { return cd > 0; } +inline bool pool_fits_lean(uint32_t n_pool, uint32_t vlean, uint32_t budget) { + return (uint64_t)n_pool * vlean <= budget; +} +inline bool quiesce_ready_pool(uint32_t quiesce_cnt, uint32_t n_pool) { + return quiesce_cnt >= n_pool - 1; +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd && g++ -std=c++17 -O2 -pthread test_dsws_ctrl_model.cpp -o /tmp/test_dsws_ctrl && /tmp/test_dsws_ctrl` +Expected: `dsws_ctrl_model: dispatch/cooldown/pool OK` then `dsws_ctrl_model: ALL PASS`. + +- [ ] **Step 5: Commit** (*only if kmbandy asks*): `feat(dsws): CPU model for dispatch/cooldown/pool invariants` + +--- + +## Task 2: Seed `s59` in the lean partition arms + BUDGET retune (no launch grow) + +**Files:** +- Modify: `occ_kernel_dsws.s` (entry role branch region ~L597–629; defsym/BUDGET block ~L128–131 / ~L403); `build_dsws.sh` (`mk2()`) + +**Interfaces:** +- Consumes: `NCOMP_SLOT/NAFEED_SLOT/NBFEED_SLOT`, `VLEAN`, `WAVES`. +- Produces: every non-claimer wave records its launch role slot id in `s59` before it reaches the dispatcher. `BUDGET` overridable via `-defsym`. Consumed by Tasks 3–5. +- **Correction vs spec §3/§6 (kmbandy-approved 2026-07-01):** every wave already launches lean-32 (compute grows to `NFV` per-rowblk, not at init). Seeding is `s59`-assignment only — **no launch-time `s_alloc_vgpr`**, so no new OOR grow site. `N_POOL`/`SEED_*` are dropped (pool size == `WAVES` == launched mix sum; seed == the existing wid partition). + +- [ ] **Step 1: BUDGET retune knob + no-parking invariant** (near the existing `BUDGET` defsym ~L403). The default stays the launch-footprint (Phase-A conservation); a `-defsym BUDGET=` override supplies real per-SIMD headroom for feed→compute grows. Add the compile-time no-parking assert: + +```asm +// compile-time no-parking invariant: every launched wave must fit lean at once +.if (WAVES * VLEAN) > BUDGET + .error "WAVES*VLEAN exceeds BUDGET — pool cannot stay all-lean (parking is out of scope)" +.endif +``` + +(`BUDGET` itself already exists behind `.ifndef BUDGET` at ~L403 — do not redefine it; only add the invariant, placed after both `WAVES` and `BUDGET` are set.) + +- [ ] **Step 2: Seed `s59` in the existing partition arms.** In the entry role branch (~L625–629), which already routes wid→`.Lbfeed`/`.Lafeed`/`.Lcompute`, add — under `.if DSWS2_CONV` — a one-instruction `s59` write on each arm so the dispatcher/re-dispatch can read the role. Do **not** restructure the branch and do **not** add any `s_alloc_vgpr`. Keep the `.else` (CONV=0) path byte-identical. Structure (`wid`=s24): + +```asm +.if DSWS2_CONV + s_cmp_lt_u32 s24, NBFEED + s_cbranch_scc1 .Lseed_bfeed + s_cmp_lt_u32 s24, (NBFEED+NAFEED) + s_cbranch_scc1 .Lseed_afeed + s_mov_b32 s59, NCOMP_SLOT + s_branch .Lcompute // FIRST entry: full role entry (_alloc/_init/_follow). Do NOT retarget to .Ldispatch (Pool-T7 brick). +.Lseed_afeed: + s_mov_b32 s59, NAFEED_SLOT + s_branch .Lafeed +.Lseed_bfeed: + s_mov_b32 s59, NBFEED_SLOT + s_branch .Lbfeed +.else + s_cmp_lt_u32 s24, NBFEED + s_cbranch_scc1 .Lbfeed + s_cmp_lt_u32 s24, (NBFEED+NAFEED) + s_cbranch_scc1 .Lafeed + s_branch .Lcompute +.endif +``` + +**Verify (name in report):** `s59` is written on every non-claimer arm before any `_follow`; no `s_alloc_vgpr` added; `s24`(wid) intact; the CONV=0 `.else` arm is character-identical to the pre-edit branch. + +- [ ] **Step 3: Thread `BUDGET` (and later `CONV_COOLDOWN`, `DSWS2_FORCE*`) through `build_dsws.sh mk2()`** — add `-defsym,BUDGET=${BUDGET:-...}` passthrough (default = the current launch-footprint expression), leaving existing calls unchanged. + +- [ ] **Step 4: Assemble + byte-identity + RGA.** Run: + +```bash +cd ; L=/opt/rocm/llvm/bin +# byte-identity at CONV=0 (baseline = same defsyms assembled from HEAD f6dda4ccf): +$L/clang -x assembler -target amdgcn-amd-amdhsa -mcpu=gfx1201 -Wa,-defsym,DSWS2=1 -Wa,-defsym,DSWS2_CONV=0 \ + -Wa,-defsym,FM=2 -Wa,-defsym,FN=4 -Wa,-defsym,G=6 -Wa,-defsym,SEGK=64 -Wa,-defsym,SAFEPROBE=1 \ + -Wa,-defsym,NCOMP=4 -Wa,-defsym,NAFEED=2 -Wa,-defsym,NBFEED=2 -c occ_kernel_dsws.s -o /tmp/c0_after.o \ + && sha256sum /tmp/c0_after.o +# CONV=1 assembles across all three mixes: +for MIX in "4 2 2" "6 1 1" "2 3 3"; do read NC NA NB <<< "$MIX" + $L/clang -x assembler -target amdgcn-amd-amdhsa -mcpu=gfx1201 -Wa,-defsym,DSWS2=1 -Wa,-defsym,DSWS2_CONV=1 \ + -Wa,-defsym,FM=2 -Wa,-defsym,FN=4 -Wa,-defsym,G=6 -Wa,-defsym,SEGK=64 -Wa,-defsym,SAFEPROBE=1 -Wa,-defsym,DIAG=1 \ + -Wa,-defsym,NCOMP=$NC -Wa,-defsym,NAFEED=$NA -Wa,-defsym,NBFEED=$NB -c occ_kernel_dsws.s -o /tmp/t2.o \ + 2>/tmp/t2.err && echo "$MIX ASSEMBLE_OK" || { echo "$MIX FAIL"; sed -n '1,15p' /tmp/t2.err; } +done +``` +*(Run via `bash script.sh` — fish won't `read <<<`.)* Expected: CONV=0 sha256 == the HEAD baseline sha256 (assemble HEAD's `occ_kernel_dsws.s` with the identical CONV=0 line to get the baseline); all three `ASSEMBLE_OK`. Then `KSRC=occ_kernel_dsws.s bash rga_check.sh t2_pool DSWS2=1 DSWS2_CONV=1 NCOMP=4 NAFEED=2 NBFEED=2` → `SGPR_SPILLS=0 VGPR_SPILLS=0`. + +- [ ] **Step 5: Commit** (*only if kmbandy asks*): `feat(dsws): seed s59 in lean partition arms + BUDGET headroom knob` + +--- + +## Task 3: The `.Ldispatch` RE-DISPATCH trampoline + tail-branch replacements + +> **CORRECTION 2026-07-02 (Pool-T7 brick, committed `b01c722df`).** `.Ldispatch` is +> **RE-DISPATCH ONLY** — reached from a `_quiesce` bail, never from first-time entry. +> The original Step 3 below flipped the Task-2 seed arms to `.Ldispatch`; that skipped +> `_alloc`/`_init` on first entry and hung the first CONV=1 dispatch (desktop brick). +> Step 3 is corrected: the seed arms **stay** on the full role entry. + +**Files:** +- Modify: `occ_kernel_dsws.s` (add `.Ldispatch`; replace the 3 `_quiesce`-bail tail branches at L907, L951, L1097. **Leave the Task-2 seed arms on `.Lcompute`/`.Lafeed`/`.Lbfeed` — do NOT retarget them to `.Ldispatch`.**) + +**Interfaces:** +- Consumes: `s59` (seeded in Task 2, flipped by `conv_apply`), the `_follow` labels `.Lcompute_follow`/`.Lafeed_follow`/`.Lbfeed_follow`. +- Produces: per-epoch role RE-dispatch after a bail. A bailed non-claimer wave routes to the `_follow` of the role `s59` names. (First entry does not use this — it runs the full role `_alloc`/`_init`/`_follow`.) + +- [ ] **Step 1: Add the trampoline** (under `.if DSWS2_CONV`, placed after the three role bodies so all `_follow` labels are in scope): + +```asm +.if DSWS2_CONV +.Ldispatch: // scalar-only, wave-uniform; s59 = current role slot id + s_cmp_eq_u32 s59, NCOMP_SLOT + s_cbranch_scc1 .Lcompute_follow + s_cmp_eq_u32 s59, NAFEED_SLOT + s_cbranch_scc1 .Lafeed_follow + s_branch .Lbfeed_follow +.endif +``` + +- [ ] **Step 2: Replace the three tail branches** — under `.if DSWS2_CONV` route to `.Ldispatch`; `.else` keep the verbatim Phase-A branch (byte-identity). At L907 (bfeed), L951 (afeed), and L1097 (compute, after `.Lcmp_quiesce`): + +```asm +.if DSWS2_CONV + s_branch .Ldispatch +.else + s_branch .Lbfeed_follow // (afeed: .Lafeed_follow ; compute: .Lcompute_follow) +.endif +``` + +- [ ] **Step 3: Leave the Task-2 seed arms on the full role entry.** Do **NOT** retarget them to `.Ldispatch`. First-time entry must run `_alloc` (the `s_alloc_vgpr 32` allocator handshake) and `_init` (the `INITFLAG==0xACED` LDS rendezvous + `s35=0` epoch seed) before its first `_follow` — all of which live only inside the role entry. Only a wave that has *already* run those once (i.e. after a `_quiesce` bail) may use `.Ldispatch`. **Verify (name in report):** the seed arms still branch to `.Lcompute`/`.Lafeed`/`.Lbfeed`; `.Ldispatch` is reached only from the 3 bail tail-branches; on re-dispatch it lands on `_follow` (NOT `_alloc`/`_init`) and `s35` is preserved so the re-dispatched wave waits for the next epoch. *(Original Step 3 flipped these to `.Ldispatch` → Pool-T7 brick; corrected 2026-07-02.)* + +- [ ] **Step 4: Assemble + byte-identity + RGA** — same commands as Task 2 Step 4. Expected: CONV=0 sha256 unchanged from baseline; 3× `ASSEMBLE_OK`; RGA 0 spills. Additionally grep the object disassembly is not required, but confirm no `s_barrier` was introduced: `grep -c s_barrier occ_kernel_dsws.s` → unchanged from HEAD. + +- [ ] **Step 5: Commit** (*only if kmbandy asks*): `feat(dsws): s59 re-dispatch trampoline (bail → _follow; first entry stays on role entry)` + +--- + +## Task 4: Cooldown `K` gate on the conversion decision + +**Files:** +- Modify: `occ_kernel_dsws.s` (decision blocks at the three bails ~L897–902 bfeed, L941–946 afeed, L1071–1091 compute; seed init in Task 2) + +**Interfaces:** +- Consumes: `s66` (designated free persistent scalar, dead after prologue), `CONV_COOLDOWN` defsym. +- Produces: a wave skips the watermark decision while `s66 > 0`; on a committed conversion `s66` resets to `CONV_COOLDOWN`; `s66` decrements once per epoch. + +- [ ] **Step 1: Add the defsym** (with the DSWS2_CONV defsyms): + +```asm +.ifndef CONV_COOLDOWN + .set CONV_COOLDOWN, 0 // default 0 = spec-faithful (no cooldown); storm sets 0; >0 damps thrash +.endif +``` + +- [ ] **Step 2: Init `s66=0` at seed** (Task 2 entry, before `.Ldispatch`): one `s_mov_b32 s66, 0` on the common seed path so every wave starts un-cooled. **Verify** all pre-existing `s66` refs are prologue-only (`grep -n '\bs66\b' occ_kernel_dsws.s` — confirm none live past the shift/mask derivation) before repurposing it. + +- [ ] **Step 3: Gate each decision + decrement.** At each bail, before `occ_sample`, add (under `.if DSWS2_CONV`): if `in_cooldown` skip straight to the `QUIESCE_CNT` bump; else run the decision. Always decrement once per epoch. Pattern (compute bail shown; mirror at afeed/bfeed, branching to the role's `_quiesce` label): + +```asm +.if CONV_COOLDOWN > 0 + s_cmp_eq_u32 s66, 0 + s_cbranch_scc0 .Lcmp_cooldn // s66>0 -> in cooldown, skip decision +.endif + // ... existing occ_sample + watermark_decision + try_gate + conv_apply ... + // inside conv_apply commit path (on \won), reset cooldown: s_mov_b32 s66, CONV_COOLDOWN +.Lcmp_cooldn: +.if CONV_COOLDOWN > 0 + s_cmp_eq_u32 s66, 0 // decrement once per epoch (saturating) + s_cbranch_scc1 .Lcmp_quiesce + s_sub_i32 s66, s66, 1 +.endif +.Lcmp_quiesce: + // ... existing QUIESCE_CNT bump ... +``` + +The cooldown reset on commit is added inside `conv_apply`'s win path (after `s59` flip, `s_mov_b32 s66, CONV_COOLDOWN`), gated `.if CONV_COOLDOWN > 0`. **Verify:** with `CONV_COOLDOWN=0` (default) none of this code emits bytes → byte-identity vs Task 3 preserved. + +- [ ] **Step 4: Assemble both K settings + CPU model.** Assemble CONV=1 with `CONV_COOLDOWN=0` (expect byte-identical to Task 3) and with `CONV_COOLDOWN=4` (expect `ASSEMBLE_OK`, RGA 0 spills). Re-run `test_dsws_ctrl_model` → `ALL PASS`. + +- [ ] **Step 5: Commit** (*only if kmbandy asks*): `feat(dsws): conversion cooldown K (default 0, tunable damper)` + +--- + +## Task 5: `DSWS2_FORCE` deterministic bring-up hook + +**Files:** +- Modify: `occ_kernel_dsws.s` (decision blocks); defsym block; `build_dsws.sh` + +**Interfaces:** +- Consumes: `s24` (wid), `s35`/epoch, the `conv_apply` commit. +- Produces: with `DSWS2_FORCE=1`, exactly the wave `DSWS2_FORCE_WID` converts direction `DSWS2_FORCE_DIR` at epoch `DSWS2_FORCE_EPOCH`, watermarks bypassed; every other wave never converts. + +- [ ] **Step 1: Add defsyms:** + +```asm +.ifndef DSWS2_FORCE + .set DSWS2_FORCE, 0 +.endif +.ifndef DSWS2_FORCE_WID + .set DSWS2_FORCE_WID, 0 +.endif +.ifndef DSWS2_FORCE_DIR + .set DSWS2_FORCE_DIR, 0 // 0/1 = compute->A/B ; 2/3 = A/B->compute +.endif +.ifndef DSWS2_FORCE_EPOCH + .set DSWS2_FORCE_EPOCH, 1 +.endif +``` + +- [ ] **Step 2: Add the forced path** at each bail, gated `.if DSWS2_CONV && DSWS2_FORCE`, taken INSTEAD of the watermark decision: convert iff `s24 == DSWS2_FORCE_WID` AND current epoch (`s35`) `== DSWS2_FORCE_EPOCH` AND this bail's role matches the source of `DSWS2_FORCE_DIR`; then `try_gate DSWS2_FORCE_DIR` + `conv_apply` with the matching src/dst slots. Bypass the watermark compare entirely. Only the site whose dir source matches emits the `conv_apply` (the dir is a compile-time immediate — a `.if DSWS2_FORCE_DIR == N` selects the site). **Verify:** with `DSWS2_FORCE=0` no bytes emit → byte-identity preserved. + +- [ ] **Step 3: Thread the four `DSWS2_FORCE*` defsyms through `build_dsws.sh mk2()`.** + +- [ ] **Step 4: Assemble matrix.** Assemble CONV=1 `DSWS2_FORCE=0` (byte-identical to Task 4 at K=0) and `DSWS2_FORCE=1 DSWS2_FORCE_WID=4 DSWS2_FORCE_DIR=0 DSWS2_FORCE_EPOCH=2` → `ASSEMBLE_OK`, RGA 0 spills. (For `4c2a2b` the partition is wids 0–1 B-feed, 2–3 A-feed, 4–7 compute; wid 4 is a compute seed, so `DIR=0` = compute→B-feed exercises a shrink at a known wave/epoch.) + +- [ ] **Step 5: Commit** (*only if kmbandy asks*): `feat(dsws): DSWS2_FORCE deterministic bring-up hook` + +--- + +## Task 6: Offline integration gate (consolidation) + +**Files:** +- Modify: none (verification-only task); optionally `build_dsws.sh` (add pool bins to the default build set) + +**Interfaces:** +- Consumes: everything from Tasks 1–5. +- Produces: the green offline package kmbandy sees before greenlighting Task 7. + +- [ ] **Step 1: Full assemble matrix.** For each mix × {CONV=0, CONV=1/K=0, CONV=1/K=4, CONV=1/FORCE=1}: assemble → `ASSEMBLE_OK`. CONV=0 sha256 == HEAD baseline for every mix. + +- [ ] **Step 2: RGA live-VGPR within BUDGET.** For each CONV=1 mix at the intended `BUDGET`: `KSRC=occ_kernel_dsws.s bash rga_check.sh pool_ ...` → `SGPR_SPILLS=0 VGPR_SPILLS=0` and reported max-VGPR ≤ the per-wave `NFV`. The `WAVES*VLEAN ≤ BUDGET` no-parking invariant is proven at assemble time by the `.error` guard (Task 2). + +- [ ] **Step 3: CPU model** — `test_dsws_ctrl_model` → `ALL PASS`. + +- [ ] **Step 4: Dry-print** — `DSWS2_DRYRUN=1 DSWS_NCOMP=4 DSWS_NAFEED=2 DSWS_NBFEED=2 DSWS2_NKSEG=1 ./occ_dispatch --dsws2 4c2a2b 2>&1 | grep -iE "NCOMP|n_kseg|tier|REFUSE"` → prints the mix, no `REFUSE`. + +- [ ] **Step 5: Write the offline-green summary** to `.superpowers/sdd/progress.md` and STOP for kmbandy greenlight. Commit the whole offline series (*only if kmbandy asks*). + +--- + +## Task 7: [SUPERVISED GPU] CONV=1 inert-safe re-baseline gate (conversions dormant) + +**Files:** +- Create: `scratchpad/run_pool_gates.sh` + +**Interfaces:** +- Consumes: the CONV=1 bins with conversions suppressed. +- Produces: the first silicon proof that the Phase-B machinery (seed `s59`, `.Ldispatch`, snapshot/quiesce sentinels — none of which has ever run on GPU) is inert-safe: with conversions dormant it reproduces the Phase-A green. Isolates "did the machinery regress the substrate" from "does conversion work" (Task 8). + +- [ ] **Step 1: Build the dormant bins** — CONV=1 with watermarks unreachable so no ticket ever fires: `DSWS2_FORCE=0` and set the watermark thresholds to make `watermark_decision` never trigger (e.g. `CTRL_LOW=0` so `occ_X < 0` is never true, and `CTRL_HIGH_*` ≥ the ring depth). Confirm from the disassembly/DIAG that zero conversions occur. All three mixes, 0-spill. + +- [ ] **Step 2: STOP — request greenlight.** Present the offline-green package (Task 6) and state plainly this is the first-ever CONV=1 GPU run; conversions are provably dormant. + +- [ ] **Step 3: Run the re-baseline sweep** — the Phase-A driver pattern in `scratchpad/run_pool_gates.sh`: 3 mixes × {n_kseg 1, 8}, env `ML8_POOL=1 ML8_COOP_CHUNK=8 ML8_COOP_CHUNK_MAXS=0.75 ML8_COOP_STREAM=1`, `timeout 30`, dmesg check, abort-on-anomaly. Expected per gate: `dsws2 oracle CLEAN`, `ok=1536 bad=0`, `occ[0]=0`, exit 0, DIAG `occ[29]` == agree (0 mismatch — the snapshot/quiesce cross-check's first silicon check), dmesg silent. + +- [ ] **Step 4: On ANY anomaly** (bad>0, 124/hang, dmesg fault, occ[29] mismatch): **full STOP + bisect**; do not run Task 8. A failure here is a machinery-regression, isolated from conversion behavior. + +- [ ] **Step 5: Record** the milestone in `.superpowers/sdd/progress.md` and a KG note. + +--- + +## Task 8: [SUPERVISED GPU] Force-convert gate + +**Files:** +- Modify: `scratchpad/run_pool_gates.sh` (force params) + +**Interfaces:** +- Consumes: `DSWS2_FORCE` bins from Task 5. +- Produces: the deterministic first proof that a converted wave runs its new role's code bit-exact. + +- [ ] **Step 1: Build the forced bin** — `DSWS2_CONV=1 DSWS2_FORCE=1 DSWS2_FORCE_WID=4 DSWS2_FORCE_DIR=0 DSWS2_FORCE_EPOCH=2`, one mix (`4c2a2b`; wid 4 = a compute seed), 0-spill confirmed. + +- [ ] **Step 2: STOP — request greenlight.** Present the Task 7 inert-safe clean result + the exact forced-conversion parameters. Do not proceed without explicit go. + +- [ ] **Step 3: Run the forced gate** — env `ML8_POOL=1 ML8_COOP_CHUNK=8 ML8_COOP_CHUNK_MAXS=0.75 ML8_COOP_STREAM=1`, `timeout 30`, `DSWS2_NKSEG` 1 then 8. Expected per run: `dsws2 oracle CLEAN`, `ok=1536 bad=0`, `occ[0]=0`, exit 0, DIAG `occ[29]` == agree (0 mismatch), dmesg silent. + +- [ ] **Step 4: On ANY anomaly** (bad>0, 124/hang, dmesg fault, occ[29] mismatch): **full STOP + bisect**; do not run Task 9. + +- [ ] **Step 5: Record** the milestone in `.superpowers/sdd/progress.md` and a KG note. + +--- + +## Task 9: [SUPERVISED GPU] Dynamic-mix gate (watermark-driven conversions) + +**Files:** Modify: `scratchpad/run_pool_gates.sh` (watermark params) + +**Interfaces:** Consumes: the K=0 CONV=1 pool bins. Produces: proof the watermark balancer moves roles while the oracle stays clean. + +- [ ] **Step 1: Build** the CONV=1 `DSWS2_FORCE=0 CONV_COOLDOWN=0` pool bins for all three mixes, 0-spill. + +- [ ] **Step 2: STOP — request greenlight.** Present the Task 8 clean result + the watermark settings that will actually fire conversions. + +- [ ] **Step 3: Run the dynamic sweep** — 3 mixes × {n_kseg 1, 8}, env + `timeout 30` + dmesg check, abort-on-anomaly. Expected per gate: `oracle CLEAN`, `bad=0`, `occ[0]=0`, `occ[29]` agree, exit 0. + +- [ ] **Step 4: On ANY anomaly: full STOP + bisect;** do not run Task 10. + +- [ ] **Step 5: Record** milestone + KG note. + +--- + +## Task 10: [SUPERVISED GPU] Storm gate (race-hunt) + +**Files:** Modify: `scratchpad/run_pool_gates.sh` (storm params) + +**Interfaces:** Consumes: CONV=1 pool bins. Produces: the lock-free race-hunt at maximal conversion frequency. + +- [ ] **Step 1: Build** CONV=1 with tight watermarks + `EPOCH_SHIFT=0` + `CONV_COOLDOWN=0`. + +- [ ] **Step 2: STOP — request greenlight.** + +- [ ] **Step 3: Run** each mix ×10 repeats, env + `timeout 30` + dmesg check. Expected every repeat: `oracle CLEAN`, `bad=0`, `occ[0]=0`, `occ[29]` agree. + +- [ ] **Step 4: On ANY anomaly: full STOP + bisect.** + +- [ ] **Step 5: Record** the final milestone; dispatch the whole-branch review (subagent-driven-development final review) and then superpowers:finishing-a-development-branch. + +--- diff --git a/docs/superpowers/plans/2026-07-02-dsws-rolling-dynvgpr-envelope.md b/docs/superpowers/plans/2026-07-02-dsws-rolling-dynvgpr-envelope.md new file mode 100644 index 000000000000..c38646aa8ce1 --- /dev/null +++ b/docs/superpowers/plans/2026-07-02-dsws-rolling-dynvgpr-envelope.md @@ -0,0 +1,509 @@ +# DSWS Rolling dyn-VGPR Sum-Envelope Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the multi-grower dyn-VGPR collision unreachable by routing every per-rowblk compute burst grow through the shared `vgpr_reserved` sum-envelope, and realize the rolling trapezoid (`lean → reserve → grow → WMMA burst → flush → shrink → release`) with real split-K. + +**Architecture:** Off the confirmed root cause (bare `s_alloc_vgpr NFV` SCC-retry with no envelope, `occ_kernel_dsws.s:1158`), we (1) prove the enveloped reserve/spin/release semantics in the CPU reference model under a thread race, (2) transcribe them into the kernel behind a new `DSWS2_ENVELOPE` symbol that defaults to byte-identical-off, (3) gate on assemble/RGA/byte-identity offline, then (4) one supervised GPU oracle run at `n_kseg>1`. Stagger + conversion-reserve reconciliation are the full-design follow-ups after the first gate. + +**Tech Stack:** GCN/RDNA4 hand-assembly (`gfx1201`, `occ_kernel_dsws.s`); C++17 CPU reference model (`dsws_ctrl_model.cpp` + std::thread test harnesses); LLVM `clang`/`llvm-objcopy` assemble + `sha256sum` byte-identity; RGA static spill analysis. + +**Spec:** `docs/superpowers/specs/2026-07-02-dsws-rolling-dynvgpr-envelope-design.md` + +**Working dir (all paths relative to it unless absolute):** `/home/kmbandy/GitHub/llama.cpp/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ` + +## Global Constraints + +- **Byte-identity is a hard gate.** With `DSWS2_ENVELOPE=0 && DSWS2_STAGGER=0`, the `.text` MUST be bit-for-bit identical to HEAD (`7b9680508`): `DSWS2_CONV=0` → `4840B`, sha256 prefix `e5ec5e50`; `DSWS2_CONV=1 && DIAG=0` → `7920B`, sha256 prefix `049818c6`. (The earlier `e296b846` figure in the handoff/spec was stale — measured pre-instrumentation-commit; ground truth re-derived from `git show HEAD:occ_kernel_dsws.s` on 2026-07-02.) Every kernel change is verified against these before proceeding. +- **`NFV=112 < 128`** (default `SQ_DYN_VGPR` cap). Never emit a grow > 128 VGPR (no operator umr flip). +- **No `s_barrier` anywhere. Never pass `--gl2c`.** +- **NEVER modify or `git add`** `occ_kernel_coop.s`, `occ_dispatch.cpp`, `fp8_oracle.cpp`, `fp8_oracle.h` (shared/not-ours). They may be dirty; leave them. +- **No GPU dispatch is auto-run.** kmbandy greenlights EVERY GPU dispatch individually. Tasks 8 is operator-triggered. +- **Commit to git ONLY when kmbandy explicitly asks.** Steps below that say "Commit" are gated on that ask; stage only OURS files. +- **After any `DSWS2_ENVELOPE=1` build+dispatch, restore the safe `CONV=0` 4840B/`e5ec5e50` bin immediately and unconditionally** (footgun removal). +- **ALL GPU data-collection writes (occ stream, dmesg/fault capture, fire scripts) go to REAL DISK (`/home/kmbandy/dsws_gpu_logs/`, btrfs), NEVER `/tmp` (tmpfs).** A brick forces a reboot that wipes tmpfs — writing the stream there destroys the very evidence the `ML8_COOP_STREAM` safeguard exists to preserve. (Lesson: 2026-07-02 brick — stream + fire script both lost to the reboot because they were in the tmpfs scratchpad.) Build intermediates (`.o`/`.text`) may stay in scratch since they are regenerable. +- **Never confound an untested feature with the change under test in one GPU dispatch.** `n_kseg>1` split-K addressing has NEVER been GPU-run or oracle-validated; it must be validated on its own before any `n_kseg>1` dispatch. The envelope gate runs at `n_kseg=1` (proven-safe geometry). +- Constants (from the kernel, fixed at `FM=2 FN=4 G=6 SEGK=64`): `NFV=112`, `VLEAN=32`, `Δ=NFV−VLEAN=80`, `WAVES=8` (4c2a2b), `KSEG_STEPS=SEGK/16=4`. + +--- + +## File Structure + +- `dsws_ctrl_model.cpp` (MODIFY) — reference semantics. Add `reserve_spin` + `reserve_release` (the compute-burst envelope helpers). Pure, header-style; no new includes. +- `test_dsws_ctrl_model.cpp` (MODIFY) — add the single-thread envelope-invariant asserts. +- `test_dsws_envelope_race.cpp` (CREATE) — std::thread forward-progress race for the compute-burst reserve loop (mirrors `test_dsws_quiesce_race.cpp`). +- `occ_kernel_dsws.s` (MODIFY) — new symbols + `BUDGET` envelope default + assemble guard; `VRESV_OFF` init inversion; `.Lcompute_reserve`/grow/release in the claim loop; DIAG envelope telemetry. All behind `DSWS2_ENVELOPE` / `DIAG`. +- `build_dsws.sh` (MODIFY) — `mk2` passthrough for `DSWS2_ENVELOPE`/`PEAK_CONC`/`DSWS2_STAGGER`/`STAGGER_PERIOD`, defaults mirroring the in-file `.ifndef` (unset ⇒ byte-identical). +- `scratchpad/fire_envelope_oracle.sh` (CREATE, Task 8) — supervised build+dispatch+restore for the GPU gate. + +Post-first-gate (full design): `occ_kernel_dsws.s` stagger phase-token (Task 9), `conv_apply` delta reconciliation (Task 10). + +--- + +## Task 1: CPU model — compute-burst reserve/release + envelope invariant + +**Files:** +- Modify: `dsws_ctrl_model.cpp` (add two helpers after `reserve_grow`, ~line 54) +- Test: `test_dsws_ctrl_model.cpp` (add asserts in `main`, before the final PASS print) + +**Interfaces:** +- Consumes: `bool reserve_grow(std::atomic& resv, uint32_t delta, uint32_t budget)` (existing, `dsws_ctrl_model.cpp:47`). +- Produces: + - `void reserve_spin(std::atomic& resv, uint32_t delta, uint32_t budget, uint64_t& spins)` — reserve `+delta`, spin-retry on over-budget, count backoffs in `spins`. + - `void reserve_release(std::atomic& resv, uint32_t delta)` — release `−delta` (models the post-shrink `lds_fetch_add VRESV_OFF, -Δ`). + +- [ ] **Step 1: Write the failing test** — append to `test_dsws_ctrl_model.cpp` `main`, just before `printf("dsws_ctrl_model: ALL PASS\n");`: + +```cpp + // ---- envelope invariant: at most PEAK_CONC concurrent peaks fit; release frees a slot ---- + { + const uint32_t VLEAN = 32, NFV = 112, D = NFV - VLEAN; // Δ = 80 + const uint32_t WAVES = 8, PEAK_CONC = 2; + const uint32_t BUDGET = WAVES * VLEAN + PEAK_CONC * D; // 256 + 160 = 416 + std::atomic resv{WAVES * VLEAN}; // init: everyone lean + uint64_t spins = 0; + reserve_spin(resv, D, BUDGET, spins); assert(spins == 0); // 1st peak fits + reserve_spin(resv, D, BUDGET, spins); assert(spins == 0); // 2nd peak fits (== PEAK_CONC) + assert(reserve_grow(resv, D, BUDGET) == false); // 3rd over-budget -> rejected + undone + assert(resv.load() == WAVES * VLEAN + PEAK_CONC * D); // undo left counter exactly at 2 peaks + reserve_release(resv, D); // free one slot + assert(reserve_grow(resv, D, BUDGET) == true); // now the 3rd fits + reserve_release(resv, D); reserve_release(resv, D); + assert(resv.load() == WAVES * VLEAN); // conservation: back to all-lean + printf("dsws_ctrl_model: envelope invariant OK\n"); + } +``` + +- [ ] **Step 2: Run to verify it fails** — Run: `cd /home/kmbandy/GitHub/llama.cpp/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ && g++ -std=c++17 -O2 -pthread test_dsws_ctrl_model.cpp -o /tmp/tcm && /tmp/tcm` + Expected: FAIL — compile error `reserve_spin`/`reserve_release` not declared. + +- [ ] **Step 3: Add the helpers** — in `dsws_ctrl_model.cpp`, immediately after the `reserve_grow` function (closing brace ~line 54): + +```cpp +// Compute-burst reserve with spin-retry (models .Lcompute_reserve): reserve +delta against the +// sum-envelope; on over-budget, reserve_grow has already undone its add, so back off and retry. +// `spins` accumulates the backoff count (permit-starvation depth). Bounded when >=1 peak fits. +static inline void reserve_spin(std::atomic& resv, uint32_t delta, + uint32_t budget, uint64_t& spins) { + while (!reserve_grow(resv, delta, budget)) ++spins; +} +// Release a booked burst (models the post-shrink lds_fetch_add VRESV_OFF, -delta). Never fails. +static inline void reserve_release(std::atomic& resv, uint32_t delta) { + resv.fetch_sub(delta, std::memory_order_acq_rel); +} +``` + +- [ ] **Step 4: Run to verify it passes** — Run: `g++ -std=c++17 -O2 -pthread test_dsws_ctrl_model.cpp -o /tmp/tcm && /tmp/tcm` + Expected: PASS — prints `envelope invariant OK` then `ALL PASS`. + +- [ ] **Step 5: Commit** (only if kmbandy asked) + +```bash +git add dsws_ctrl_model.cpp test_dsws_ctrl_model.cpp +git commit -m "feat(dsws): CPU model reserve_spin/reserve_release + envelope invariant test" +``` + +--- + +## Task 2: CPU model — forward-progress thread race + +**Files:** +- Create: `test_dsws_envelope_race.cpp` + +**Interfaces:** +- Consumes: `reserve_spin`, `reserve_release` (Task 1). +- Produces: standalone executable; exit 0 == no stall reproduced. + +- [ ] **Step 1: Write the failing test** — create `test_dsws_envelope_race.cpp`: + +```cpp +// Forward-progress race for the DSWS compute-burst sum-envelope (.Lcompute_reserve/grow/release). +// NCOMP threads each loop {claim rowblk -> reserve_spin(+Δ) -> bounded burst -> reserve_release(-Δ)}. +// A watchdog trips if any thread fails to finish its rowblks (models a permit-starvation hang on CPU +// what cannot be safely bisected on the compositor-attached gfx1201). Target: 0 stalls for PEAK_CONC>=1. +#include "dsws_ctrl_model.cpp" +#include +#include +#include +#include +#include + +static constexpr uint32_t VLEAN = 32, NFV = 112, D = NFV - VLEAN; // Δ = 80 +static constexpr uint32_t WAVES = 8, NCOMP = 4; +static constexpr uint32_t ROWBLKS = 64; // rowblks each compute wave completes +static constexpr uint64_t SPIN_LIMIT = 50'000'000ull; + +static bool run_once(uint32_t peak_conc) { + const uint32_t budget = WAVES * VLEAN + peak_conc * D; + std::atomic resv{WAVES * VLEAN}; + std::atomic done{0}; + std::atomic stalled{false}; + std::vector ts; + for (uint32_t w = 0; w < NCOMP; ++w) { + ts.emplace_back([&]{ + for (uint32_t r = 0; r < ROWBLKS && !stalled.load(); ++r) { + uint64_t spins = 0; + while (!reserve_grow(resv, D, budget)) { + if (++spins > SPIN_LIMIT) { stalled.store(true); return; } + std::this_thread::yield(); + } + // bounded "burst": a few atomic touches, then release + for (int k = 0; k < 8; ++k) done.fetch_add(0); + reserve_release(resv, D); + } + done.fetch_add(1); + }); + } + for (auto& t : ts) t.join(); + bool ok = !stalled.load() && done.load() >= NCOMP && resv.load() == WAVES * VLEAN; + if (!ok) printf(" STALL peak_conc=%u resv=%u done=%u stalled=%d\n", + peak_conc, resv.load(), done.load(), (int)stalled.load()); + return !ok; +} + +int main() { + const int TRIALS = 200; + int fails = 0; + for (uint32_t pc = 1; pc <= 3; ++pc) { + int f = 0; + for (int i = 0; i < TRIALS; ++i) if (run_once(pc)) f++; + printf("[peak_conc=%u] %d/%d trials stalled\n", pc, f, TRIALS); + fails += f; + } + if (fails == 0) printf("dsws_envelope_race: NO STALL — envelope guarantees forward progress\n"); + else printf("dsws_envelope_race: STALL REPRODUCED (%d)\n", fails); + return fails ? 1 : 0; +} +``` + +- [ ] **Step 2: Run to verify it builds & passes** — Run: `g++ -std=c++17 -O2 -pthread test_dsws_envelope_race.cpp -o /tmp/ter && /tmp/ter; echo "exit=$?"` + Expected: `[peak_conc=1] 0/200 … [peak_conc=3] 0/200`, `NO STALL`, `exit=0`. (If it stalls at `peak_conc=1`, the envelope math is wrong — stop and re-derive before touching asm.) + +- [ ] **Step 3: Commit** (only if kmbandy asked) + +```bash +git add test_dsws_envelope_race.cpp +git commit -m "test(dsws): forward-progress thread race for the compute-burst sum-envelope" +``` + +--- + +## Task 3: Kernel — new symbols + BUDGET envelope default + assemble guard + +**Files:** +- Modify: `occ_kernel_dsws.s` (symbol block near `:125`; `BUDGET` block `:441`; guard `:447`) + +**Interfaces:** +- Produces symbols: `DSWS2_ENVELOPE` (0), `PEAK_CONC` (2), `DSWS2_STAGGER` (0), `STAGGER_PERIOD` (`NCOMP`); envelope-mode `BUDGET` default. + +- [ ] **Step 1: Add the symbols** — in `occ_kernel_dsws.s`, alongside the other `.ifndef` symbol defaults (after the `DSWS2_CONV`/`CONV_COOLDOWN` block, ~`:130`): + +```asm +.ifndef DSWS2_ENVELOPE + .set DSWS2_ENVELOPE, 0 // 1 = route the per-rowblk compute burst grow through the vgpr_reserved +.endif // sum-envelope. 0 = HEAD (bare .Lcompute_grow) -> .text byte-identical. +.ifndef PEAK_CONC + .set PEAK_CONC, 2 // concurrent compute peaks the budget admits (R3 sweep). Used iff ENVELOPE=1. +.endif +.ifndef DSWS2_STAGGER + .set DSWS2_STAGGER, 0 // 1 = lock-free phase-token stagger (Task 9). 0 -> emergent envelope stagger. +.endif +.ifndef STAGGER_PERIOD + .set STAGGER_PERIOD, NCOMP // phase slots in the stagger ring (R3 sweep). Used iff STAGGER=1. +.endif +``` + +- [ ] **Step 2: Make BUDGET envelope-aware** — replace the `BUDGET` `.ifndef` block (`:441`): + +```asm +.ifndef BUDGET +.if DSWS2_ENVELOPE + .set BUDGET, (WAVES*VLEAN + PEAK_CONC*(NFV-VLEAN)) // rolling: lean floor + concurrent-peak headroom +.else + .set BUDGET, (NCOMP*NFV + (NAFEED+NBFEED)*VLEAN) // static-fat (HEAD) — unchanged +.endif +.endif +``` + +- [ ] **Step 3: Add the forward-progress guard** — inside the existing `.if DSWS2_CONV` guard block (`:445-450`, next to the `WAVES*VLEAN > BUDGET` `.error`): + +```asm +.if DSWS2_ENVELOPE +.if (WAVES*VLEAN + (NFV-VLEAN)) > BUDGET + .error "ENVELOPE: BUDGET admits < 1 concurrent peak — forward progress impossible" +.endif +.endif +``` + +- [ ] **Step 4: Byte-identity gate** — assemble at the byte-identical config (`ENVELOPE` unset ⇒ 0) and confirm sha unchanged. Run: + +```bash +cd /home/kmbandy/GitHub/llama.cpp/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ +L=/opt/rocm/llvm/bin +"$L/clang" -x assembler -target amdgcn-amd-amdhsa -mcpu=gfx1201 \ + -Wa,-defsym,DSWS2=1 -Wa,-defsym,FM=2 -Wa,-defsym,FN=4 -Wa,-defsym,G=6 -Wa,-defsym,SEGK=64 \ + -Wa,-defsym,SAFEPROBE=1 -Wa,-defsym,DIAG=1 -Wa,-defsym,NCOMP=4 -Wa,-defsym,NAFEED=2 -Wa,-defsym,NBFEED=2 \ + -Wa,-defsym,DSWS2_CONV=0 -c occ_kernel_dsws.s -o /tmp/bi.o \ +&& "$L/llvm-objcopy" -O binary --only-section=.text /tmp/bi.o /tmp/bi.text \ +&& echo "CONV=0: $(wc -c < /tmp/bi.text)B $(sha256sum /tmp/bi.text | cut -c1-8) (expect 4840B e5ec5e50)" +``` + Expected: `CONV=0: 4840B e5ec5e50`. If it differs, a symbol block emitted bytes — a `.set`/`.ifndef` must be inert; fix before proceeding. + +- [ ] **Step 5: Commit** (only if kmbandy asked) + +```bash +git add occ_kernel_dsws.s +git commit -m "feat(dsws): envelope symbols + BUDGET envelope-mode default + forward-progress guard" +``` + +--- + +## Task 4: Kernel — VRESV init inversion under ENVELOPE + +**Files:** +- Modify: `occ_kernel_dsws.s` (`VRESV_OFF` seed, `:744`) + +- [ ] **Step 1: Invert the init under envelope mode** — replace the single `lds_put VRESV_OFF, …` at `:744`: + +```asm +.if DSWS2_ENVELOPE + lds_put VRESV_OFF, (WAVES*VLEAN) // rolling: everyone lean; counter books peaks +.else + lds_put VRESV_OFF, (NCOMP*NFV + (NAFEED+NBFEED)*VLEAN) // HEAD (unchanged) +.endif +``` + +- [ ] **Step 2: Byte-identity gate** — re-run the Task 3 Step 4 block (ENVELOPE unset). Expected unchanged: `CONV=0: 4840B e5ec5e50`. (The `.else` arm is the only one emitted when `ENVELOPE=0`, and it is textually identical to the original `lds_put`.) + +- [ ] **Step 3: Commit** (only if kmbandy asked) + +```bash +git add occ_kernel_dsws.s +git commit -m "feat(dsws): invert VRESV init to lean-floor under DSWS2_ENVELOPE" +``` + +--- + +## Task 5: Kernel — reserve/grow/release in the compute claim loop + +**Files:** +- Modify: `occ_kernel_dsws.s` (`.Lcompute_claim`/`.Lcompute_grow` `:1151-1161`; `.Lcompute_shrink` `:1221-1224`) + +**Interfaces:** +- Consumes: `reserve_try delta, won` macro (`:497`), `VRESV_OFF=52`, `NFV`/`VLEAN`, `SLEEPN`. +- Registers: `s54` = reserve `won`; `s62/s63` = `reserve_try` internal scratch (dead here — conv runs only at `.Lcompute_drained`). No new persistent reg. + +- [ ] **Step 1: Add reserve-before-grow** — in the `.if DYNVGPR` block between `s_wait_storecnt 0x0` (`:1157`) and `.Lcompute_grow` (`:1158`), insert: + +```asm +.if DSWS2_ENVELOPE +.Lcompute_reserve: + reserve_try +(NFV-VLEAN), s54 // s54 = won (1 = booked, prev+Δ ≤ BUDGET) + s_cmp_eq_u32 s54, 0 + s_cbranch_scc0 .Lcompute_grow // won -> grow (SCC==0 when s54!=0) + s_sleep SLEEPN // over budget -> back off AT LEAN (reserve_try already undid its add) + s_branch .Lcompute_reserve +.endif +``` + (The existing `.Lcompute_grow:` label and its `s_alloc_vgpr NFV` SCC-retry stay exactly as-is, immediately below.) + +- [ ] **Step 2: Add release-after-shrink** — in the `.if DYNVGPR` block, after the `.Lcompute_shrink` SCC-retry (`:1223`) and before `lds_inc ROWBLK_DONE_OFF` (`:1225`): + +```asm +.if DSWS2_ENVELOPE + lds_fetch_add s54, VRESV_OFF, -(NFV-VLEAN) // release −Δ; wave is lean-32 here (v-temps ≤v15, OOR-safe) +.endif +``` + +- [ ] **Step 3: Byte-identity gate (ENVELOPE=0)** — re-run the Task 3 Step 4 block. Expected unchanged: `CONV=0: 4840B e5ec5e50`. Also verify `CONV=1/DIAG=0`: + +```bash +cd /home/kmbandy/GitHub/llama.cpp/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ +L=/opt/rocm/llvm/bin +"$L/clang" -x assembler -target amdgcn-amd-amdhsa -mcpu=gfx1201 \ + -Wa,-defsym,DSWS2=1 -Wa,-defsym,FM=2 -Wa,-defsym,FN=4 -Wa,-defsym,G=6 -Wa,-defsym,SEGK=64 \ + -Wa,-defsym,SAFEPROBE=1 -Wa,-defsym,DIAG=0 -Wa,-defsym,NCOMP=4 -Wa,-defsym,NAFEED=2 -Wa,-defsym,NBFEED=2 \ + -Wa,-defsym,DSWS2_CONV=1 -c occ_kernel_dsws.s -o /tmp/bi1.o \ +&& "$L/llvm-objcopy" -O binary --only-section=.text /tmp/bi1.o /tmp/bi1.text \ +&& echo "CONV=1/DIAG=0: $(sha256sum /tmp/bi1.text | cut -c1-8) (expect 049818c6)" +``` + Expected: `CONV=1/DIAG=0: 049818c6`. + +- [ ] **Step 4: Assemble the ENVELOPE=1 bin (smoke)** — confirm the new path assembles clean and grows are ≤128: + +```bash +"$L/clang" -x assembler -target amdgcn-amd-amdhsa -mcpu=gfx1201 \ + -Wa,-defsym,DSWS2=1 -Wa,-defsym,FM=2 -Wa,-defsym,FN=4 -Wa,-defsym,G=6 -Wa,-defsym,SEGK=64 \ + -Wa,-defsym,SAFEPROBE=1 -Wa,-defsym,DIAG=0 -Wa,-defsym,NCOMP=4 -Wa,-defsym,NAFEED=2 -Wa,-defsym,NBFEED=2 \ + -Wa,-defsym,DSWS2_CONV=1 -Wa,-defsym,DSWS2_ENVELOPE=1 -Wa,-defsym,PEAK_CONC=2 \ + -c occ_kernel_dsws.s -o /tmp/env.o && echo "ENVELOPE=1 assembles OK" +``` + Expected: `ENVELOPE=1 assembles OK` (no `.error`; guard passes since `336 ≤ 416`). + +- [ ] **Step 5: Commit** (only if kmbandy asked) + +```bash +git add occ_kernel_dsws.s +git commit -m "feat(dsws): envelope the per-rowblk compute burst grow (.Lcompute_reserve/release)" +``` + +--- + +## Task 6: Kernel — DIAG envelope telemetry + +**Files:** +- Modify: `occ_kernel_dsws.s` (DIAG wedge-frame block `:893-931`; reserve/release sites for event counters) + +**Interfaces:** +- Free occ slots (verified against used set `{24,28,40,44,60,76,84,88,104,108,112,116}`): occ[8]=32, occ[9]=36, occ[12]=48, occ[13]=52, occ[14]=56. + +> **Scope decision (2026-07-02, implemented):** Task 6 landed the **claimer-side readback only** — instantaneous `vgpr_reserved` → occ[8] and `PEAK_CONC` → occ[14], inside the existing lane-0 DIAG wedge block (reuses `s52`). This is the headline diagnostic: a wedge with `occ[8]==BUDGET` is permit-starvation, anything below points elsewhere. It touches only the claimer's cold advance-gate poll — **not** the compute hot path — avoiding a fresh heisenbug (recall: adding stores to the compute poll loop perturbed timing and hid the original race). The per-wave reserve-spin / grow / release counters (occ[9]/[12]/[13]) are **deferred** to a follow-up: they require carrying scalars across the WMMA burst (clobber risk) or LDS accumulation (hot-path atomics), for resolution the `vgpr_reserved` readback already covers at the first gate. Add them only if the first GPU gate wedges ambiguously. + +- [ ] **Step 1: Add a per-wave reserve-spin + event counters** — carry three scalars across the burst (choose from the compute-loop dead band; `s55`/`s56`/`s57` are free outside `.Lcompute_drained`). At `.Lcompute_reserve` increment a spin counter on each backoff; at grow and release increment event counters. In the reserve loop (Task 5 Step 1), under `.if DSWS2_ENVELOPE && DIAG` only, bump `s55` (reserve-spin) before `s_sleep`, and bump `s56` (grow events) after a win / `s57` (release events) at the release site. Keep these purely additive and DIAG-gated so `DIAG=0` stays byte-identical. + +```asm +; in .Lcompute_reserve, before s_sleep SLEEPN: +.if DSWS2_ENVELOPE && DIAG + s_add_u32 s55, s55, 1 ; reserve-spin depth (saturating in practice) +.endif +; at the win fall-through (just before/after .Lcompute_grow's alloc), and at the release: +.if DSWS2_ENVELOPE && DIAG + s_add_u32 s56, s56, 1 ; (grow events) — at grow + s_add_u32 s57, s57, 1 ; (release events) — at release +.endif +``` + +- [ ] **Step 2: Publish envelope telemetry in the wedge frame** — in the claimer DIAG block (`:912-929`, inside the lane-0 exec mask, alongside the existing `global_store_b32` markers), under `.if DSWS2_ENVELOPE && DIAG` add reads of `VRESV_OFF` (high-water via running max in a scalar) and stores of the envelope slots: + +```asm +.if DSWS2_ENVELOPE && DIAG + lds_get s52, VRESV_OFF + v_mov_b32 v14, s52 + global_store_b32 v4, v14, s[0:1] offset:32 scope:SCOPE_DEV // occ[8] vgpr_reserved (peak Σ readback) + v_mov_b32 v14, PEAK_CONC + global_store_b32 v4, v14, s[0:1] offset:56 scope:SCOPE_DEV // occ[14] PEAK_CONC echo (config readback) +.endif +``` + (Per-wave `s55`/`s56`/`s57` — reserve-spin / grow / release — are published to occ[9]=36 / occ[12]=48 / occ[13]=52 from the compute wave's own lane-0 path at drain, same idiom as `epoch_mark`.) + +- [ ] **Step 3: Byte-identity gate (DIAG=0)** — re-run Task 3 Step 4 (CONV=0) and Task 5 Step 3 (CONV=1/DIAG=0). Expected unchanged: `4840B e5ec5e50` and `7920B 049818c6`. (All new stores are under `.if …&& DIAG`.) + +- [ ] **Step 4: Assemble ENVELOPE=1/DIAG=1 (smoke)** — re-run Task 5 Step 4 with `-Wa,-defsym,DIAG=1`. Expected: assembles OK. + +- [ ] **Step 5: Commit** (only if kmbandy asked) + +```bash +git add occ_kernel_dsws.s +git commit -m "feat(dsws): DIAG envelope telemetry (vgpr_reserved HWM, reserve-spin, grow/release counts)" +``` + +--- + +## Task 7: build_dsws.sh passthrough + assemble matrix + RGA gate + +**Files:** +- Modify: `build_dsws.sh` (`mk2`, add envelope defsym passthrough, defaults mirroring the in-file `.ifndef`) + +- [ ] **Step 1: Add passthrough to `mk2`** — in `build_dsws.sh` `mk2` (the `clang` invocation, `:30-36`), append (mirroring the existing `DSWS2_FORCE`/`BUDGET` passthrough pattern so an unset env is byte-identical): + +```bash + -Wa,-defsym,DSWS2_ENVELOPE=${DSWS2_ENVELOPE:-0} -Wa,-defsym,PEAK_CONC=${PEAK_CONC:-2} \ + -Wa,-defsym,DSWS2_STAGGER=${DSWS2_STAGGER:-0} -Wa,-defsym,STAGGER_PERIOD=${STAGGER_PERIOD:-4} \ +``` + Also make the `mk2` `budget` local respect envelope mode: when `DSWS2_ENVELOPE=1` and `$BUDGET` unset, default to `$(( 8*32 + ${PEAK_CONC:-2}*80 ))`; else the existing `$(( $1*112 + ($2+$3)*32 ))`. + +- [ ] **Step 2: Assemble matrix** — build all four corners and confirm the byte-identity ones and that the envelope ones assemble: + +```bash +cd /home/kmbandy/GitHub/llama.cpp/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ +DIAG=1 DSWS2_ENVELOPE=0 ./build_dsws.sh norga # baseline path (byte-identical family) +DIAG=1 DSWS2_ENVELOPE=1 PEAK_CONC=2 ./build_dsws.sh norga +``` + Expected: both print `OK … occ_dsws2_4c2a2b_gd.bin`. (Note: `mk2`'s output bin name is fixed; the envelope build OVERWRITES it — re-run the ENVELOPE=0 build or restore the safe bin before any GPU work.) + +- [ ] **Step 3: RGA 0-spill gate** — run the static spill analysis on the ENVELOPE=1 build (bar: 0 spill, max-live ≤ prior 84): + +```bash +KSRC=occ_kernel_dsws.s DIAG=0 DSWS2=1 DSWS2_CONV=1 DSWS2_ENVELOPE=1 PEAK_CONC=2 \ + ./rga_check.sh env_4c2a2b FM=2 FN=4 G=6 SEGK=64 NCOMP=4 NAFEED=2 NBFEED=2 SAFEPROBE=1 2>&1 \ + | grep -E "gfx1201,|livereg|spill" || true +``` + Expected: `livereg`/spill line shows 0 spill; max-live ≤ 84. If spill > 0, the reserve/release added live pressure — investigate the scratch assignment before the GPU gate. + +- [ ] **Step 4: Restore the safe CONV=0 bin** (footgun removal) — rebuild the safe bin so no envelope bin is left installed: + +```bash +DIAG=1 DSWS2_ENVELOPE=0 ./build_dsws.sh norga +echo "installed: $(wc -c < occ_dsws2_4c2a2b_gd.bin)B $(sha256sum occ_dsws2_4c2a2b_gd.bin | cut -c1-8)" +``` + Note: the default `mk2` build is `CONV=0`-equivalent only if the bin matches the safe sha; if `build_dsws.sh` builds `DIAG=1` the bin won't be the 4840B CONV=0 — confirm the exact safe bin (`e5ec5e50`, 4840B, `DIAG=1 CONV=0`) is what's installed, matching `fire_wedge_diag.sh`'s restore. + +- [ ] **Step 5: Commit** (only if kmbandy asked) + +```bash +git add build_dsws.sh +git commit -m "build(dsws): mk2 passthrough for DSWS2_ENVELOPE/PEAK_CONC/STAGGER" +``` + +--- + +## Task 8: Supervised GPU gate — envelope ISOLATED at n_kseg=1 (OPERATOR-TRIGGERED) + +> **This task is NOT auto-run. kmbandy greenlights it individually.** The agent prepares the script and STOPS. A hang/fault can MODE1-reset the desktop — accepted, but only with the safeguards intact. +> +> **REVISED after the 2026-07-02 brick.** The first attempt confounded the envelope with `n_kseg=2` (never validated) and page-faulted (OOB global address, gfxhub → MES unrecoverable → MODE1). The envelope is LDS-atomic only and cannot cause a global page fault; the fault was the untested split-K addressing. **This gate now isolates the envelope on proven-safe `n_kseg=1`** — the only change under test is the collision fix. `n_kseg>1` is validated separately (Task 8b) before any `n_kseg>1` dispatch. + +**Files:** +- Create: `/home/kmbandy/dsws_gpu_logs/fire_envelope_oracle.sh` (REAL DISK — not scratchpad/tmpfs). + +- [ ] **Step 1: Write the fire script** — build `ENVELOPE=1/CONV=1/DIAG=0 PEAK_CONC=2`, run the safeguarded dispatch at **`n_kseg=1`** (`DSWS2_NKSEG=1`), stream occ + dmesg to **real disk** (`~/dsws_gpu_logs/`), then UNCONDITIONALLY restore the safe `CONV=0` 4840B/`e5ec5e50` bin. Safeguards: quiesce gate ON, chunk watchdog 0.75s, `timeout 30`, restore-with-sha-check. (Script already written at `~/dsws_gpu_logs/fire_envelope_oracle.sh`, default `NKSEG=1 DIAG_RUN=0`.) + +```bash +ML8_POOL=1 ML8_COOP_CHUNK=8 ML8_COOP_CHUNK_MAXS=0.75 ML8_COOP_STREAM=1 \ + DSWS_NCOMP=4 DSWS_NAFEED=2 DSWS_NBFEED=2 DSWS2_NKSEG=1 \ + timeout 30 ./occ_dispatch --dsws2 4c2a2b 2>"$STREAM" # $STREAM on /home (btrfs), NOT /tmp +``` + +- [ ] **Step 2: STOP — request greenlight.** Present the script path and exact config. Do not run it. + +- [ ] **Step 3 (on greenlight): run + read out.** Expected: `ok=1536 bad=0`, `occ[0]=0`, dmesg delta 0. DIAG=0 first so a clean pass proves the *envelope* (not DIAG perturbation) fixed the collision. If it wedges (forward-progress hang, NOT a page fault): re-run `DIAG_RUN=1` to read `occ[8]` (`==416` ⇒ permit-starvation). If it *page-faults*: that's an addressing bug, not the envelope — investigate offline. + +- [ ] **Step 4: Confirm safe bin restored** — `sha256sum occ_dsws2_4c2a2b_gd.bin | cut -c1-8` == `e5ec5e50`, size 4840B. Confirm the stream persisted to `~/dsws_gpu_logs/`. + +## Task 8b: Validate n_kseg>1 split-K addressing (SEPARATE, before any n_kseg>1 dispatch) + +> `n_kseg>1` addressing has never been validated. The 2026-07-02 page fault was in the `ksi`-dependent A/B base math (`A += ksi*SEGK`, `B += ksi*KSEG_STEPS*NT*256`) — and `SAFEPROBE` clamps the per-lane *vaddr* (v8/v9/v10) + `ti`, NOT the scalar segment base, so it did not catch it. + +- [ ] Build a CPU model of the `n_kseg>1` (t,ksi) decode + A/B/C address computation; assert every computed global offset stays within the host-allocated buffer bounds (`occ_dispatch` `Amax`/`Bmax` at :1374/:1377) for all `sti ∈ [0,TOTAL_super)`. Only after that passes offline, gate one supervised `n_kseg=2` oracle dispatch (LOOSE tier) — ISOLATED from the envelope (i.e. `ENVELOPE=0` first, to prove the addressing alone). + +--- + +## Task 9 (post-first-gate): Stagger phase-token — full design + +> Build only after Task 8 is green. The envelope alone fixes the collision; the stagger is the occupancy/feed-fungibility refinement (R3). Whether explicit stagger beats emergent envelope staggering is an empirical sweep — this task adds the mechanism; the sweep (Task 8 Step 5 sweep, extended) decides if it earns its bytes. + +**Files:** +- Modify: `occ_kernel_dsws.s` — `STAGGER_TOK_OFF=100` LDS word (extend the overlap guard `:184`); a `.Lcompute_phase` poll before `.Lcompute_reserve`, gated `.if DSWS2_STAGGER`. +- Modify: `dsws_ctrl_model.cpp` + a stagger-fairness test — model the phase ring keeps `≈PEAK_CONC` peaks active and phase-spread. + +- [ ] Add `STAGGER_TOK_OFF=100` + guard; CPU-model the phase-ring fairness (TDD); transcribe the lock-free `lds_fetch_add STAGGER_TOK_OFF,1` mod `STAGGER_PERIOD` poll; byte-identity at `STAGGER=0`; assemble/RGA; (greenlit) GPU sweep `STAGGER ∈ {0,1} × STAGGER_PERIOD`. + +--- + +## Task 10 (post-first-gate): Conversion-reserve reconciliation + +> Land before conversions go live WITH the envelope. Under `DSWS2_ENVELOPE=1`, a feed↔compute conversion books zero peak VGPR (both roles lean at rest); the burst reserve is the only peak booking. + +**Files:** +- Modify: `occ_kernel_dsws.s` — the four `conv_apply` grow call sites (`:1010/1019/1084/1093`) pass `delta=0` and `alloc_sz=32` under `.if DSWS2_ENVELOPE`; shrink sites (`:1253/1257/1267/1274`) already lean. +- Modify: `dsws_ctrl_model.cpp` + test — model that conversion is delta-neutral under envelope mode and the burst reserve carries all peak accounting; assert no double-booking. + +- [ ] TDD the delta-neutral conversion in the CPU model; transcribe the `.if DSWS2_ENVELOPE` delta/alloc change; byte-identity at `ENVELOPE=0`; assemble/RGA; (greenlit) GPU gate with conversions LIVE + envelope at `n_kseg>1`. + +--- + +## Self-Review + +- **Spec coverage:** §2 accounting inversion → Task 4 (+ Task 1 model). §4.1 envelope reserve/release → Task 5 (+ Tasks 1-2 model). §4.2 split-K → Task 8 (`DSWS2_NKSEG=2`; no kernel change needed). §4.3 stagger → Task 9. §4.4 conversion reconciliation → Task 10. §4.5 DIAG telemetry → Task 6. §3 symbols/BUDGET → Task 3. §7 CPU-model → Tasks 1-2. §8 validation ladder → Tasks 1-2 (model) → 7 (assemble/RGA/byte-identity) → 8 (GPU) → 9 (sweep). §9 safety → Global Constraints + Task 8 gating. All covered. +- **Placeholder scan:** Task 6 Step 1 describes the counter placement in prose with the exact `.if …&& DIAG` guards and register choices (`s55/s56/s57`) rather than a single fixed diff, because the three bump sites are in distinct blocks; the code fragments and slot map are concrete. No TBD/TODO elsewhere. +- **Type consistency:** `reserve_spin(resv, delta, budget, spins)` and `reserve_release(resv, delta)` names/signatures match between Task 1 (definition), Task 1 test, and Task 2 race. `VRESV_OFF=52`, `PEAK_CONC`, `NFV`/`VLEAN`/`Δ`, occ slot offsets consistent across Tasks 3-8. diff --git a/docs/superpowers/specs/2026-07-01-dsws-phaseb-pool-economy-design.md b/docs/superpowers/specs/2026-07-01-dsws-phaseb-pool-economy-design.md new file mode 100644 index 000000000000..9db9da8347bb --- /dev/null +++ b/docs/superpowers/specs/2026-07-01-dsws-phaseb-pool-economy-design.md @@ -0,0 +1,282 @@ +# DSWS Phase B — Universal Dynamic Dispatch + Grow-into-Budget Pool Economy + +**Status:** design (brainstormed 2026-07-01, kmbandy). Extends and builds on the +committed Phase-B substrate (Tasks 1–5, branch `feat/dsws-phaseb-conversion`, +HEAD `6f3e36f45`). + +**Goal:** Make a wave that commits a role conversion actually *execute the new +role's code*, and generalize the fixed 8-wave partition into an adaptive +wave-role economy on a lean-start pool — realizing the DSWS vision where the +kernel re-balances its {compute, A-feed, B-feed} mix into the VGPR budget at +runtime, per shape, per moment. + +**Non-goal:** parking/dormant waves (explicitly dropped as YAGNI — see §7). + +--- + +## 0. Context: what already exists (Tasks 1–5) + +The committed substrate provides, gated under `.if DSWS2_CONV` (byte-identical +to Phase A when off): + +- **Sensing** (`occ_sample`) — per-role occupancy proxy in `s55/s56`. +- **Ticket** (`try_gate \dir,\swin`) — single-winner `(dir,epoch)` LDS-CAS. +- **Reservation** (`reserve_try \delta,\won`) — signed-delta VGPR envelope vs + `BUDGET`; shrink (−80) always wins, grow (+80) aborts over budget. +- **Bail-time commit** (`conv_apply`) — floor-guarded role-slot swap + VGPR + resize (`s_alloc_vgpr`) + `s59` role-register flip, ordered strictly *before* + the `QUIESCE_CNT` bump. +- **Snapshot/quiesce** — claimer snapshots the live role mix into the next + epoch's parity half of `SNAP_BASE` at broadcast; per-role claim-count + sentinels are sized from that snapshot; a `QUIESCE_CNT ≥ WAVES−1` cross-check + is the safety net; a DIAG `occ[29]` flag asserts the two agree. + +**The gap this spec closes:** `conv_apply` flips `s59` but nothing reads it — a +converted wave changes its role-slot count and VGPR footprint yet keeps running +its *original* role's code path. Conversion is currently accounting-only. + +**Key constants (from `occ_kernel_dsws.s`):** `NFV=112` (fat compute VGPR), +`VLEAN=32` (lean feed VGPR), conversion delta `NFV−VLEAN=80`, role slot ids +`NCOMP_SLOT=24 / NAFEED_SLOT=28 / NBFEED_SLOT=32`, `SNAP_BASE=72` (u32[6], +parity-doubled {nC,nA,nB}), `QUIESCE_CNT_OFF=96`. + +--- + +## 1. Architecture + +Under `DSWS2_CONV=1` every wave launches lean at `VLEAN=32` (already Phase-A +behavior — compute grows to `NFV` per-rowblk on demand and shrinks back; nothing +launches fat). The launched wave count *is* the pool size (`WAVES` = the mix +sum). wid-0 is the claimer. Each non-claimer wave is **seeded** with its launch +role by writing `s59` in the existing wid partition — **no launch-time grow**. +Role lives entirely in `s59`, and a single dispatcher routes every wave — at +entry and after every terminal bail — to the code for the role `s59` names. +Waves rebalance by growing into / shedding VGPR budget (bounded by +`reserve_try`/`BUDGET`), **never** by forking, merging, or parking. Scaling the +launched wave count above the current mix sum (the 12–16-wave "bigger pool" for a +larger feed multiplier) is a follow-on requiring an `occ_dispatch` dims change; +the mechanism is proven first as role rebalancing *within* the launched mix. + +**Wave-count invariant (the hardware truth):** the launched wave count is fixed +for the kernel's life; `s_alloc_vgpr` resizes *one existing wave*, it does not +spawn or retire waves. The {compute, A, B} multiplier from the vision +(shed 1 fat compute → several lean feed light up) is realized in the **VGPR +budget**, not the wave count: shrinking a compute wave frees `80` VGPR of budget +that other waves can grow into or that lets a feed-heavy mix stay within budget. +Because feed waves are cheap (`32` VGPR), the pool is launched large enough that +a feed-heavy mix has many active feed waves. + +**Byte-identity invariant:** all Model-B launch/dispatch lives under +`.if DSWS2_CONV`. With `DSWS2_CONV=0` the kernel assembles byte-identical to the +proven Phase-A fixed-partition kernel. Task 6's re-baseline runs `DSWS2_CONV=0` +as the untouched golden reference. + +--- + +## 2. The dispatcher (universal dynamic dispatch) + +**Today:** role is a wid-partition branch taken once at entry (`occ_kernel_dsws.s` +~L604–629); each role's `_follow` loop ends with an unconditional +`s_branch .L_follow` (L907/L951/compute equivalent). + +**New:** replace those three tail branches with one shared, scalar-only, +wave-uniform trampoline: + +```asm +.Ldispatch: // s59 = current role slot id (24/28/32) + s_cmp_eq_u32 s59, NCOMP_SLOT + s_cbranch_scc1 .Lcompute_follow + s_cmp_eq_u32 s59, NAFEED_SLOT + s_cbranch_scc1 .Lafeed_follow + s_branch .Lbfeed_follow +``` + +> **CORRECTION 2026-07-02 (Pool-T7 brick root cause).** `.Ldispatch` is a +> **RE-DISPATCH-ONLY** trampoline — it is reached from a per-super-tile `_quiesce` +> bail, **never from first-time entry**. Entry and re-dispatch are **NOT** the same +> mechanism: first entry must run the full role `_alloc`/`_init` (see §3), and only +> re-dispatch may land on `_follow`. The original text below said entry could reuse +> `.Ldispatch`; that routing skipped `_alloc`/`_init` on first entry (no +> `s_alloc_vgpr 32` handshake, no INITFLAG rendezvous, garbage `s35`) and hung the +> first CONV=1 dispatch → desktop brick. Points 1–2 hold **for re-dispatch**; point 3 +> is corrected below. + +Three properties make this safe and cheap: + +1. **On RE-DISPATCH, lands on `_follow`, skipping `_alloc`/`_init`.** `conv_apply` + already set the wave's VGPR footprint to the target role's size, and the wave + already ran `_alloc`/`_init` once at first entry. Re-running `_alloc` would + wrongly resize the wave; re-running `_init` would reset `s35 = 0`, breaking the + "wait for the *next* epoch" contract (INITFLAG is written once, `0xACED`, and is + never cleared — so it is the `s35` reset, not INITFLAG, that `_follow` preserves). + **First entry must NOT take this path** — it has run neither `_alloc` nor `_init`. +2. **Preserves the last-seen-epoch register** (`s35`). A re-dispatched wave + therefore waits for the *next* epoch in its new role rather than + re-processing the epoch it just converted in. +3. **Entry and re-dispatch are DISTINCT.** First entry runs the full role entry + (`.Lcompute`/`.Lafeed`/`.Lbfeed` → `_alloc` → `_init` → `_follow`), with the + wid-branch *seeding* `s59` first so the role register is set before any later + re-dispatch reads it. Only after a bail does a wave use `.Ldispatch`. A + non-converting wave, once past first entry, reads back its own role each epoch + via `.Ldispatch` and loops exactly as today. + +The claimer (wid-0) is **excluded**: it branches to `.Lclaimer` at entry (before +the seed/`.Ldispatch` path), runs its own `.Lclaim_loop`, never converts, and is +never counted among the `N_POOL−1` bailers. The dispatcher governs non-claimer +waves only. + +Cost: initialize `s59` at each seed role (part of §3), the trampoline (~6 +lines), replace 3 tail branches. Entirely additive; the proven `_follow` bodies +are untouched. `s59` read + `s_branch` is scalar-only → **zero** new OOR +exposure (§6). + +Rejected alternatives: per-bail inline role check (less DRY, no benefit); +unified per-epoch dispatch loop that collapses the three `_follow` bodies +(rewrites GPU-proven control flow for no functional gain). + +--- + +## 3. Launch & init (seed the lean partition) + +Every wave already `s_alloc_vgpr 32` at entry (feeds, claimer, **and** compute — +compute grows to `NFV` per-rowblk inside its loop and shrinks back). wid-0 → +claimer. So launch/init is unchanged from Phase A except one addition: + +**Seed `s59` by wid** (under `DSWS2_CONV`), in the *existing* partition arms — +`[0,NBFEED)` → B-feed, `[NBFEED,NBFEED+NAFEED)` → A-feed, rest → compute — each +arm gains a single `s_mov_b32 s59, ` and then **falls into the full role +entry** (`.Lbfeed`/`.Lafeed`/`.Lcompute` → `_alloc` → `_init` → `_follow`). +**It must NOT branch to `.Ldispatch`** (that skips `_alloc`/`_init` on a wave that +has run neither → the Pool-T7 brick; see §2 correction). `.Ldispatch` is entered +only later, from a `_quiesce` bail. **No launch-time `s_alloc_vgpr` is *added*** — +the seed reuses the role entry's existing `_alloc` (compute's `NFV` footprint is +still handled per-rowblk). There are no `N_POOL`/`SEED_*` defsyms: the launched mix +`NCOMP/NAFEED/NBFEED` *is* the seed and its sum *is* the pool size. + +**Defsyms:** + +- `BUDGET` — retuned to the real per-SIMD VGPR ceiling (headroom for growth), + **not** the zero-headroom launch-footprint default Task 5 inherited. This is + the knob that makes feed→compute grows able to succeed. Validated by the RGA + live-VGPR gate and a real dispatch — never a guessed constant. Compile-time + no-parking invariant: `WAVES × VLEAN ≤ BUDGET` (every wave always fits lean). +- `VRESV` seed = `NCOMP*NFV + (NAFEED+NBFEED)*VLEAN` (unchanged; the live + reservation the economy adjusts from). + +--- + +## 4. Conversion policy (sense → decide → commit → re-dispatch) + +At each terminal bail (`conv_apply` site), a wave runs Task 5's decision, plus +two additions: + +- **Cooldown `K`** — a per-wave scalar counts down one per epoch; while `> 0` + the wave skips the watermark check entirely (no ticket race, no conversion). + On a committed conversion, reset to `K`. Default `K = 0` (spec-faithful; the + storm gate sets 0 for maximal thrash). `K > 0` is a churn damper available if + `s_alloc_vgpr` grow/shrink frequency starts bricking runs — its value is + reducing the *number* of OOR windows per run. The cooldown counter joins the + persistent scalar set (outside `s60–s65`, alongside `s57/s58/s59`). +- **Force-convert bring-up hook** — a `DSWS2_FORCE` defsym path where a + designated wid converts a chosen dir at a chosen epoch, watermarks bypassed; + everything else static. The deterministic first-proof lever (§7). Gated, off + by default, emits zero bytes when off. + +Direction semantics unchanged from Task 5: `occ_X < CTRL_LOW` → compute starved +for X → shrink one compute → feed-X (delta −80, always wins); `occ_X > +CTRL_HIGH_X` → feed-X over-serving → grow one feed-X → compute (delta +80, may +abort over `BUDGET` → stay). Floor guard keeps every role ≥ 1 wave. + +After commit (or no-op), the wave bumps `QUIESCE_CNT` (Task 5 ordering) and +falls into `.Ldispatch`. + +--- + +## 5. Quiesce reconciliation (the correctness crux) + +**Invariant that must hold every epoch:** *the claimer's per-epoch snapshot of +the role mix equals the role mix the waves actually execute that epoch.* + +Re-dispatch closes Task 5's loop: + +1. A converting wave commits its slot swap in epoch *E* at bail time, **before** + bumping `QUIESCE_CNT` (Task 5 ordering). +2. The claimer waits for quiesce (all `N_POOL−1` non-claimer waves bailed **and** + the snapshot sentinels), then snapshots the **live** (now-updated) slot counts + into the *E+1* parity half of `SNAP_BASE`, resets `QUIESCE_CNT`, and bumps the + epoch last. +3. The converting wave, having flipped `s59`, enters its **new** role's `_follow` + via `.Ldispatch` and wakes for *E+1*. +4. In *E+1* the wave contributes to the claim counters **as its new role** — + exactly the mix the claimer snapshotted for *E+1*. Sentinels match observed + contributions. + +The `QUIESCE_CNT ≥ N_POOL−1` cross-check holds unchanged: **every wave bails +exactly once per epoch regardless of role** (no parking), so the count is +role-agnostic and invariant under conversion. The DIAG `occ[29]` flag asserts +snapshot-sentinels ⟺ quiesce-count agree; with re-dispatch wired it reads +*agree* even when conversions fire (whereas Task 5 alone would diverge — the +divergence the flag was built to catch). Task 7's dynamic gate verifies this on +silicon. + +--- + +## 6. OOR-poison window (SPEC §4, #1 brick risk) + +- **The dispatcher adds zero OOR exposure:** `.Ldispatch` is `s59` read + + `s_branch`, scalar-only. A re-dispatched wave enters its new `_follow` with its + footprint already correctly sized (`conv_apply` closed its grow window before + the bump) and no pending grow. +- **`conv_apply`'s `s_alloc_vgpr` grow** is unchanged from Task 5's audit: waves + are lean-32 at every bail, every pre-grow LDS/atomic temp ≤ v15. **This is the + only GROW in the design** — seeding adds none (compute's per-rowblk grow is the + existing, already-audited Phase-A path). + +--- + +## 7. Failure modes, floors, and no-parking + +- **Role floors:** `conv_dec_floor` keeps compute ≥ 1 and each feed ≥ 1 — a role + can never empty. +- **Budget safety:** `N_POOL × VLEAN ≤ BUDGET` guarantees the lean floor always + fits; `reserve_try` guarantees grows only commit within the real ceiling; the + atomic-add-then-undo-on-abort path leaves `vgpr_reserved` exactly restored (no + leaked reservation, Task 5-verified). +- **No parking (YAGNI):** because the pool is sized so all waves stay + active-lean, a wave never needs to go dormant. Surplus capacity does useful + lean-feed work instead of sleeping. This is what preserves the simple + `QUIESCE_CNT ≥ N_POOL−1` accounting and avoids a race-prone wake/sleep + handshake on the brick-risk path. + +--- + +## 8. Validation ladder + +Offline (no GPU) → then supervised GPU gates, each needing an individual +greenlight; any brick/hang/DMESG-fault/DIAG-mismatch = full STOP + bisect, never +auto-advance. + +1. **Offline** — assemble all mixes `DSWS2_CONV=1 DIAG=1` → `ASSEMBLE_OK`; RGA + `SGPR_SPILLS=0 VGPR_SPILLS=0` and live-VGPR within `BUDGET`; `DSWS2_CONV=0` + sha256 byte-identical to pre-change; CPU control model `ALL PASS`; dry-print + sanity. +2. **Force-convert gate (SUPERVISED)** — `DSWS2_FORCE` one designated wave, one + dir, one known epoch, watermarks bypassed. Expect oracle-CLEAN, `bad=0`, + `occ[0]=0`, DIAG `occ[29]` agree, dmesg silent. The deterministic debut of a + wave executing a converted footprint's role code. +3. **Dynamic gate (SUPERVISED)** — watermark-driven conversions fire; oracle + stays CLEAN as roles move; `occ[29]` agree across all mixes × tiers. +4. **Storm gate (SUPERVISED)** — `K=0`, `EPOCH_SHIFT=0`, ×10 repeats — lock-free + race-hunt under maximal conversion frequency. + +--- + +## 9. Open parameters (resolved during implementation, not guessed) + +- `BUDGET`, `K` — empirically tuned defsyms with the principled defaults in + §3–§4; final values set by RGA + real-dispatch measurement on the target ml8 + shapes, not chosen a priori. (No `N_POOL`/`SEED_*`: pool size == launched mix + sum, seed == launch partition — see §3 correction.) +- Persistent-scalar assignment for the cooldown counter (must sit outside + `s60–s65` and every macro's clobber set; candidates alongside `s57/s58/s59`) — + fixed at implementation time against the live register map. diff --git a/docs/superpowers/specs/2026-07-02-dsws-rolling-dynvgpr-envelope-design.md b/docs/superpowers/specs/2026-07-02-dsws-rolling-dynvgpr-envelope-design.md new file mode 100644 index 000000000000..5e76dd1a59c6 --- /dev/null +++ b/docs/superpowers/specs/2026-07-02-dsws-rolling-dynvgpr-envelope-design.md @@ -0,0 +1,350 @@ +# DSWS rolling dyn-VGPR — the sum-envelope grow economy (detailed design) + +**Date:** 2026-07-02 +**Status:** design, pre-implementation +**Kernel:** `ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_dsws.s` (`--dsws2` path) +**Predecessors:** `HANDOFF_DSWS_ROLLING_DYNVGPR.md` §4 (approved architecture), `SPEC_WAVESPEC.md` §EXTENSION (R0–R4 traveling-peak), +`SPEC_DSWS_SUBSTRATE_V2.md` / `SPEC_DSWS_PHASEB_CONVERSION.md` (`vgpr_reserved`/`VRESV` envelope, `conv_apply`). + +This spec turns the handoff §4 architecture into exact LDS offsets, register/scratch assignments, macro definitions, asm +placement, host-streamed DIAG slots, and CPU-model interfaces. It is written to be implemented directly. + +--- + +## 0. One-paragraph problem & fix + +`occ_kernel_dsws.s:1158` `.Lcompute_grow` is a **bare `s_alloc_vgpr NFV` SCC-retry with no sum-envelope reservation and no +stagger**. All `NCOMP` compute waves claim a rowblk and issue `s_alloc_vgpr NFV=112` in lockstep, colliding on the shared +per-SIMD dyn-VGPR pool → intermittent forward-progress hang (ISA §3.3.3.2: HW deadlock-avoidance guarantees only *one* +wave/SIMD can reach max allocation). Confirmed root cause (KG `21cc0d63`); the heisenbug (adding DIAG made it run clean) +proves a timing race, not a logic bug. **Fix:** route every per-rowblk compute grow through the existing `vgpr_reserved` +sum-envelope (`reserve_try`, `VRESV_OFF=52`) with a spin-at-lean retry, so at most `PEAK_CONC` waves hold peak at once and +the collision is **unreachable by construction**. Then (a later validation rung) add a lock-free stagger so peaks stay +phase-spread and freed budget flows to whichever role is the current bottleneck. + +--- + +## 1. Goals / non-goals + +**Goals** +- G1. Make the multi-grower collision unreachable: enforce `Σ instantaneous VGPR allocations ≤ BUDGET` at every instant, + gating the *per-rowblk compute burst grow* through the same `vgpr_reserved` counter role-conversion already uses. +- G2. Realize the rolling trapezoid `lean → reserve → grow → WMMA burst → flush → shrink → release` per rowblk/kseg, with + real split-K (`n_kseg > 1`) so the peak is a brief window and the pool genuinely churns. +- G3. Bake in envelope telemetry (DIAG-gated) so a wedge is a readout (permit-starvation vs elsewhere), not a mystery. +- G4. Keep the door open to a lock-free stagger (phase-spread peaks; feed-fungibility timing) as a tuning layer on top. +- G5. **Byte-identity:** every new instruction sits behind a new assembler symbol; with the new features off, the `.text` + is bit-for-bit identical to HEAD at `DSWS2_CONV=0` (`e5ec5e50`) and `DSWS2_CONV=1/DIAG=0` (`e296b846`). + +**Non-goals (YAGNI)** +- N1. No wave parking / dynamic launch-count change. `WAVES` is fixed; every wave fits lean at once (`WAVES*VLEAN ≤ BUDGET`). +- N2. No `s_barrier` anywhere. Coordination stays LDS-atomic + epoch/phase polling. +- N3. No `NFV > 128` (stays under the default `SQ_DYN_VGPR` cap; no operator umr flip). +- N4. The role-conversion **watermark controller** (occ_sample → try_gate → conv_apply) is unchanged in *policy*. This spec + only changes what conversion *reserves* against the envelope (see §4.4), and only under the new envelope mode. + +--- + +## 2. The invariant (the one law) + +Let `B = BUDGET` (per-SIMD VGPR sum-envelope ceiling), `V_peak = NFV = 112`, `V_lean = VLEAN = 32`, `Δ = V_peak − V_lean = 80`. + +``` +vgpr_reserved = Σ over resident waves of (that wave's current booked allocation) +INVARIANT: vgpr_reserved ≤ BUDGET at every instant +``` + +**Accounting model (this is the crux — it inverts the static-fat init).** Today `VRESV_OFF` is initialized to +`NCOMP*NFV + (NAFEED+NBFEED)*VLEAN` — i.e. it books **all compute waves at full peak permanently** (the static substrate, +where compute waves camp fat for ~95% of K). That leaves **zero** headroom for a transient per-burst reserve, so it cannot +express the rolling model. The rolling model is the inverse: + +- **Every wave is `VLEAN` at rest** (compute, A-feed, B-feed, claimer). Role identity = *which loop the wave runs*, not a + permanent VGPR footprint. +- The reservation counter books **transient burst peaks only.** Init `vgpr_reserved = WAVES*VLEAN` (everyone lean). +- A compute wave taking a rowblk/kseg books `+Δ` (reserve → grow), and releases `−Δ` on shrink. Net zero per rowblk. +- `BUDGET = WAVES*VLEAN + PEAK_CONC*Δ`, where `PEAK_CONC` = the number of concurrent compute peaks the SIMD VGPR file + admits. At most `PEAK_CONC` waves can hold peak simultaneously; the rest spin at lean until budget frees. + +**Concurrent-peak count** (WAVESPEC §EXTENSION worked example): a symmetric trapezoid averages `≈ V_peak/2`, so resident +peaks `≈ B/(V_peak/2) = 2×` the all-at-peak count. `PEAK_CONC` is the sweep knob (R3); start at 2 (proven-safe: 2 peaks = +`2*112 + 6*32 = 416 ≤ B`), sweep up while `Σ ≤ B` holds with slack ≥ one grow-step (the brick guard). + +Reconciliation with the existing conversion envelope (§4.4): under envelope mode a feed↔compute **conversion books no peak +VGPR** — both source and dest roles are `VLEAN` at rest, so the flip is a zero-delta reservation. Only the *burst* books +`+Δ`. This folds the conversion reserve into the per-burst reserve — the unification the handoff calls for — and supersedes +`SPEC_DSWS_PHASEB_CONVERSION.md`'s `+(NFV−VLEAN)` feed→compute conversion reserve *when `DSWS2_ENVELOPE=1`*. + +--- + +## 3. New assembler symbols (all default to the byte-identical value) + +```asm +.ifndef DSWS2_ENVELOPE + .set DSWS2_ENVELOPE, 0 // 1 = route the per-rowblk compute burst grow through the vgpr_reserved sum-envelope. +.endif // 0 = HEAD behavior (bare .Lcompute_grow) -> .text byte-identical. +.ifndef PEAK_CONC + .set PEAK_CONC, 2 // concurrent compute peaks the budget admits (R3 sweep axis). Only used when ENVELOPE=1. +.endif +.ifndef DSWS2_STAGGER + .set DSWS2_STAGGER, 0 // 1 = enable the lock-free phase-token stagger (R3 refinement on top of ENVELOPE). +.endif +.ifndef STAGGER_PERIOD + .set STAGGER_PERIOD, NCOMP // phase slots in the stagger ring (R3 sweep axis). Only used when STAGGER=1. +.endif +``` + +`BUDGET` (already a `-defsym`, `occ_kernel_dsws.s:441`) gains an envelope-mode default: + +```asm +.ifndef BUDGET +.if DSWS2_ENVELOPE + .set BUDGET, (WAVES*VLEAN + PEAK_CONC*(NFV-VLEAN)) // rolling: lean floor + concurrent-peak headroom +.else + .set BUDGET, (NCOMP*NFV + (NAFEED+NBFEED)*VLEAN) // static-fat (HEAD) — unchanged +.endif +.endif +``` + +Compile-time guards (extend the existing `.error` block at `:447`): +```asm +.if DSWS2_ENVELOPE + .if (WAVES*VLEAN + (NFV-VLEAN)) > BUDGET + .error "ENVELOPE: BUDGET admits < 1 concurrent peak — forward progress impossible" + .endif +.endif +``` + +`DSWS2_NKSEG` is **host-side only** (dispatch env; already plumbed) — it sets `KT`, and the kernel derives +`n_kseg = KT >> NKSEG_SHIFT` in the prologue (`:621`). No kernel symbol; split-K churn is turned on purely by dispatching +with `DSWS2_NKSEG=` (N>1). `n_kseg=1` remains the degenerate high-duty case (handled free by `DECODE_STI`). + +--- + +## 4. Mechanism + +### 4.1 Envelope the per-rowblk compute grow (the correctness fix) + +**Init.** Replace the `VRESV_OFF` seed (`occ_kernel_dsws.s:744`) under envelope mode: +```asm +.if DSWS2_ENVELOPE + lds_put VRESV_OFF, (WAVES*VLEAN) // rolling: everyone lean; counter books transient peaks +.else + lds_put VRESV_OFF, (NCOMP*NFV + (NAFEED+NBFEED)*VLEAN) // HEAD (unchanged) +.endif +``` + +**Reserve before grow, release after shrink.** The current claim loop (`:1151–1226`): +```asm +.Lcompute_claim: + lds_fetch_add s33, ROWBLK_NEXT_OFF, 1 + s_cmp_ge_u32 s33, G + s_cbranch_scc1 .Lcompute_drained +.if DYNVGPR + s_wait_loadcnt 0x0 + s_wait_storecnt 0x0 + // >>> NEW (ENVELOPE): reserve +Δ against vgpr_reserved; spin-at-lean-retry on over-budget <<< +.if DSWS2_ENVELOPE +.Lcompute_reserve: + reserve_try +(NFV-VLEAN), s54 // s54 = won (1 = booked, ≤BUDGET); reserve_try scratch s62/s63 + s_cmp_eq_u32 s54, 0 + s_cbranch_scc0 .Lcompute_grow // won -> grow + s_sleep SLEEPN // over budget -> back off AT LEAN (reserve_try already undid its add) + s_branch .Lcompute_reserve +.endif +.Lcompute_grow: + s_alloc_vgpr NFV // SCC-retry stays as the HW backstop + s_cbranch_scc0 .Lcompute_grow +.endif + ...zero ACC / WMMA burst over KSEG_STEPS / flush to C via global_atomic_add_f32... + s_wait_storecnt 0x0 // atomic-adds READ ACC -> drain before shrink frees ACC +.if DYNVGPR +.Lcompute_shrink: + s_alloc_vgpr 32 + s_cbranch_scc0 .Lcompute_shrink + // >>> NEW (ENVELOPE): release −Δ; wave is lean-32 here so v-temps ≤v15 (OOR-safe) <<< +.if DSWS2_ENVELOPE + lds_fetch_add s54, VRESV_OFF, -(NFV-VLEAN) // mirror of the reserve; shrink already committed +.endif +.endif + lds_inc ROWBLK_DONE_OFF + s_branch .Lcompute_claim +``` + +**Why the collision is now unreachable.** `reserve_try` (`:497`) atomically adds `+Δ`, validates `prev+Δ ≤ BUDGET`, and +UNDOES on failure — the LDS atomic serializes concurrent grows. With `BUDGET = WAVES*VLEAN + PEAK_CONC*Δ`, at most +`PEAK_CONC` waves can be past the reserve gate (holding peak) at once; every other claimant spins in `.Lcompute_reserve` at +lean-32. "All `NCOMP` grow at once" cannot occur. This is exactly the ISA §3.3.3.2 "software scheme for forward progress": +reserve-slack is the guarantee. + +**Forward progress.** `PEAK_CONC ≥ 1` (guarded at assemble time) ⇒ at least one waiter always fits. Each burst is bounded +(`KSEG_STEPS` WMMA steps) and unconditionally reaches `shrink → release`, monotonically freeing `Δ`, so a waiter always +eventually proceeds. No wave holds peak indefinitely. (Validated by the CPU race-model, §7.) + +**OOR safety (SPEC 4 #1 brick rule).** The reserve runs while the wave is lean-32 (pre-grow); `reserve_try`'s only vector +temps are the `lds_fetch_add` v11/v14 pair (`RP_A`/`RP_D`, ≤v15) — same as `conv_apply`'s proven pre-grow window. The +release runs *after* the shrink, wave back at lean-32, again ≤v15. No VGPR > v15 is sourced before a grow. + +**Scratch register budget at the claim loop.** At `.Lcompute_claim`, the conversion scratch band `s52–s65` is dead (conv +only runs at `.Lcompute_drained`). Assignment: `s54` = reserve `won`; `s62/s63` = `reserve_try` internal scratch (already +its documented scratch). Live-across-burst regs untouched: `s33` (rowblk), `s38` (C tile-term), `s35` (epoch), +`s19/s30/s31` (mblk/tcol/ksi), `s67/s68` (mask/shift), `s6/s7`/`s28/s29` (C base). No new persistent register. + +**Transient-blip note.** A failed `reserve_try` does add-then-undo, a momentary over-count another wave could observe and +spuriously back off from. This is not a correctness violation (no one ever exceeds `BUDGET`); worst case is one extra +`s_sleep SLEEPN` backoff. The monotonic shrink-frees-budget property rules out livelock (proven in the CPU race-model). We +keep the proven `reserve_try` primitive rather than introduce a peek-without-commit variant. + +### 4.2 Real split-K churn (`n_kseg > 1`) + +The per-rowblk `grow → WMMA(KSEG_STEPS) → flush → shrink` is **already the trapezoid** — each rowblk covers one `SEGK` +segment (`KSEG_STEPS = SEGK/16` k-steps) and flushes fp32 partials to C via `global_atomic_add_f32` +(`:1205–1219`, `scope:SCOPE_DEV`). With `n_kseg > 1`, `DECODE_STI` splits `sti → (t, ksi)` so a compute wave processes one +`ksi`'s `SEGK`-segment per super-tile; partials from all `ksi` accumulate into the **same** ksi-independent C cell +(`s38 = ti*(G*FM*FN*1024)`, ksi-independent — `:1147`). **No accumulator persists across a wave's ksegs** (the accumulate- +into-C path handles the sum), so the peak is brief and duty drops as `n_kseg` rises. Split-K is fully plumbed; the only +change to exercise it is dispatching with `DSWS2_NKSEG=N` (N>1). **Open question resolved:** per-kseg flush → immediate +release is confirmed the intent and matches the existing structure — the wave shrinks to lean between rowblks/ksegs; no +path pins the peak across ksegs. + +**Master knob = burst length.** Here burst length = `KSEG_STEPS` (WMMA steps a wave holds peak per rowblk) × rowblks held +before shrink (currently 1 — shrink every rowblk). Short burst → more churn/occupancy, more grow/shrink overhead; long +burst → amortized overhead, higher duty. Optimum = shortest burst that keeps the WMMA pipe full. `SEGK` (⇒ `KSEG_STEPS`) +and `n_kseg` are the R3 sweep axes alongside `PEAK_CONC`. + +### 4.3 The stagger (R3 refinement — phase-spread peaks + feed fungibility) + +The envelope alone yields **emergent demand-driven staggering**: with `PEAK_CONC < NCOMP`, waves take turns at peak. The +explicit stagger *coordinates* those turns to (1) keep compute saturated (always `≈PEAK_CONC` peaks active and phase- +spread), (2) keep peaks from bunching, and (3) time the per-burst release so **feed waves can grow the moment the ring +counters say feed is the bottleneck** — freed VGPR flows to the current bottleneck, per-moment (the adaptive core). + +**Mechanism (lock-free, no `s_barrier`).** A ring of `STAGGER_PERIOD` phase slots in a new LDS word `STAGGER_TOK_OFF` +(§5). Reuse the prod/cons counter style (`lds_fetch_add` + compare). Before the reserve, a compute wave claims the next +phase slot (`lds_fetch_add STAGGER_TOK_OFF, 1`, mod `STAGGER_PERIOD`) and only proceeds to `.Lcompute_reserve` when its +slot is "open" (its phase's turn), else `s_sleep SLEEPN` and re-poll. The envelope remains the hard floor; the stagger is +scheduling on top (it can never admit more than `BUDGET` allows). Gated under `DSWS2_STAGGER`; `STAGGER=0` ⇒ pure emergent +envelope staggering, byte-identical-when-off. + +**Whether explicit stagger beats emergent** is an empirical R3 question (measure WMMA-pipe saturation and feed-wait % both +ways). The spec defines the mechanism; the sweep decides if it earns its bytes. + +### 4.4 Conversion-reserve reconciliation (envelope mode only) + +When `DSWS2_ENVELOPE=1`, a feed↔compute conversion is a **zero-delta** reservation (both roles lean at rest). Concretely, +under envelope mode the four `conv_apply` call sites pass `delta = 0` for the feed→compute grow direction, and the +compute→feed direction likewise (the wave is already lean; the burst reserve is what books peak). The `s_alloc_vgpr` +in `conv_apply` for feed→compute becomes `s_alloc_vgpr 32` (stay lean — the burst grows later, enveloped), not `NFV`. + +**Scoping:** the first GPU correctness gate runs with **conversions dormant** (the exact CONV=1-dormant / CONV=0 configs +already on silicon), so this reconciliation is not exercised at the first gate — the envelope-on-the-burst fix stands +alone. The `conv_apply` delta change lands as a scoped follow-up (its own CPU-model gate) before conversions go live with +the envelope. This is test *ordering*, not scope-cutting: the full design includes both. + +### 4.5 Instrumentation baked in (DIAG-gated, byte-identical when `DIAG=0`) + +Add envelope telemetry to host-streamed occ slots at the claimer advance-gate wedge frame (`:893` `.if DIAG` block, next +to the existing snapshot dump), and at the reserve/release sites. All under `.if DSWS2_ENVELOPE && DIAG`, lane-0-masked, +`global_store_b32 v4, …, s[0:1] offset:` (the existing idiom): + +| datum | meaning | occ slot (byte off) | +|---|---|---| +| `vgpr_reserved` high-water | peak `Σ` observed — headroom check vs `BUDGET` | occ[8] = 32 | +| reserve-spin count (per wave, saturating) | permit-starvation depth — "wedge = permit wait" vs elsewhere | occ[9] = 36 | +| grow events | bursts entered | occ[12] = 48 | +| shrink/release events | bursts completed (grow==release ⇒ no leak) | occ[13] = 52 | +| `PEAK_CONC` echo | budget config readback | occ[14] = 56 | + +(Free slots verified against the used set {24,28,40,44,60,76,84,88,104,108,112,116}; occ[8/9/12/13/14] are unused. The +existing wedge-frame markers — ROWBLK/AROW/BFRAG DONE/NEXT, QUIESCE, per-role epochs — stay as-is.) A wedge then reads out +*directly* whether it is permit-starvation (high reserve-spin, `vgpr_reserved` pinned at `BUDGET`) or elsewhere. + +--- + +## 5. LDS layout delta + +Control region is `0..255` below the resident region (`BRES_OFF=256`). Current occupancy: control words `0..68`, +`SNAP_BASE=72` (u32[6] → 72..95), `QUIESCE_CNT_OFF=96`. One new word: + +```asm +.set STAGGER_TOK_OFF, 100 // u32 phase-ring counter (only touched when DSWS2_STAGGER=1) +``` + +Extend the existing overlap guard (`:184`) to include `STAGGER_TOK_OFF < BRES_OFF`. `vgpr_reserved` reuses the existing +`VRESV_OFF=52` (no new slot). No resident-region repoint (all new state stays in the `0..255` control gap), preserving the +`DSWS2_CONV=0` byte-identity. + +--- + +## 6. Assembler-symbol / register / offset summary (implementation checklist) + +- **New symbols:** `DSWS2_ENVELOPE` (0), `PEAK_CONC` (2), `DSWS2_STAGGER` (0), `STAGGER_PERIOD` (`NCOMP`). `BUDGET` gains an + envelope-mode default. All default to the byte-identical value. +- **New LDS:** `STAGGER_TOK_OFF=100` (u32). Reuse `VRESV_OFF=52`. +- **New labels:** `.Lcompute_reserve` (spin-retry reserve loop). Optional `.Lcompute_phase` (stagger poll) under STAGGER. +- **Registers:** `s54` = reserve `won`; `s62/s63` = `reserve_try` scratch (unchanged); no new persistent SGPR/VGPR. Reserve + runs pre-grow (lean, ≤v15); release runs post-shrink (lean, ≤v15). +- **Edited sites:** `VRESV_OFF` init (`:744`); `.Lcompute_claim/_grow` (`:1151–1161`); `.Lcompute_shrink` (`:1221–1224`); + `BUDGET` block (`:441`); the `.error` guard (`:447`); the DIAG wedge frame (`:893`); (envelope-mode) the four + `conv_apply` deltas (`:1010/1019/1084/1093` grows; `:1253/1257/1267/1274` shrinks) — follow-up, §4.4. + +--- + +## 7. CPU-model interfaces (extend `dsws_ctrl_model.cpp` + `test_dsws_ctrl_model.cpp`) + +The existing `reserve_grow(resv, delta, budget) -> bool` (`dsws_ctrl_model.cpp:47`) already models the atomic +add/validate/undo. Add, matching the asm exactly: + +1. **Signed / spin reserve** — a compute-burst reserve helper: reserve `+Δ`; on fail, back off and retry (models + `.Lcompute_reserve`). Pure function over a shared `std::atomic` reservation + `budget`. +2. **Envelope invariant test** — assert `vgpr_reserved ≤ BUDGET` holds across every interleaving; assert grow-count == + release-count at quiescence (no lost/leaked reservation); assert `vgpr_reserved` returns to `WAVES*VLEAN` when all + bursts complete. +3. **Forward-progress race test** — `std::thread` model of `NCOMP` compute waves each looping {claim rowblk → spin-reserve + → burst (bounded) → release}, `PEAK_CONC ∈ {1,2,…}`, `{seq_cst, relaxed}`, N trials. Watchdog trips iff any wave fails + to complete its rowblks (reproduces a permit-starvation hang on CPU). Target: **0 stalls** for all `PEAK_CONC ≥ 1` + (mirrors the existing `test_dsws_quiesce_race.cpp` structure and its 0/400 result). + +--- + +## 8. Validation ladder (test ordering — not scope-cutting) + +1. **CPU model** — envelope invariant + forward-progress race (§7) green; existing model tests still pass. +2. **Assemble matrix** — `{ENVELOPE,STAGGER} × {CONV 0/1} × {DIAG 0/1}` assemble clean; RGA 0-spill; max-live ≤ prior. +3. **Byte-identity** — `ENVELOPE=0 && STAGGER=0`: `.text` sha == HEAD at `CONV=0` (`e5ec5e50`, 4840B) and + `CONV=1/DIAG=0` (`e296b846`). Non-negotiable gate. +4. **One supervised GPU oracle run** — `ENVELOPE=1`, conversions dormant, **`n_kseg > 1`**, `PEAK_CONC=2`, the safeguarded + config (§9). Target `ok=1536 bad=0`, dmesg silent. kmbandy greenlights it individually. +5. **Sweep** — `PEAK_CONC × SEGK/n_kseg × (STAGGER on/off, STAGGER_PERIOD)`; rocprof + RGA each cell; watch resident-wave + count, WMMA-pipe %, feed-wait %, `vgpr_reserved` high-water (must keep slack ≥ one grow-step — the brick guard). + +--- + +## 9. Safety & operating rules (HARD — kmbandy standing orders) + +- **No GPU isolation.** gfx1201 drives the desktop; a hang can MODE1-reset it. Bricks are accepted **only with safeguards.** +- **kmbandy greenlights EVERY GPU dispatch individually.** Never auto-dispatch. +- **Safeguarded, proven-non-bricking dispatch config** (keep all of it; only `DSWS2_NKSEG` and the envelope build change): + ``` + ML8_POOL=1 ML8_COOP_CHUNK=8 ML8_COOP_CHUNK_MAXS=0.75 ML8_COOP_STREAM=1 \ + DSWS_NCOMP=4 DSWS_NAFEED=2 DSWS_NBFEED=2 DSWS2_NKSEG= \ + timeout 30 ./occ_dispatch --dsws2 4c2a2b + ``` +- **NEVER disable the quiesce gate** (no `SKIPQ`/force `s51=1`) — removes the clean chunk-watchdog self-abort → bricks. +- **Build the `ENVELOPE=1` bin only at dispatch time; restore the safe `CONV=0` 4840B/`e5ec5e50` bin immediately after, + unconditionally** (footgun removal — see `scratchpad/fire_wedge_diag.sh`). Never leave a fat/`ENVELOPE` bin installed. +- **`NFV=112 < 128`** (default `SQ_DYN_VGPR` cap) — never exceed 128 VGPR without the operator-only umr flip. +- **Never modify / stage** `occ_kernel_coop.s`, `occ_dispatch.cpp`, `fp8_oracle.*` (shared/not-ours). Never pass `--gl2c`. +- **Commit to git ONLY when kmbandy asks.** + +--- + +## 10. References + +- `HANDOFF_DSWS_ROLLING_DYNVGPR.md` §4 — approved architecture (this spec's source). +- `SPEC_WAVESPEC.md` §EXTENSION — traveling-peak, sum-envelope reasoning, R0–R4 build order. +- `SPEC_DSWS_SUBSTRATE_V2.md` / `SPEC_DSWS_PHASEB_CONVERSION.md` — `vgpr_reserved`/`VRESV`, `conv_apply`, `reserve_try`. +- `MAD305_DSWS_MASTER.md:78–81` — split-K headroom framing (its "staggering buys nothing" clause describes `n_kseg=1`; + corrected here for real split-K). +- KG: `1fdd5784` (traveling peak), `21cc0d63` (root cause = multi-grower collision), `98614f24`/`1cef3378`/`a241842c` + (ISA §3.3.3.2), `2009aa16` (dyn-VGPR OOR rule), `d1a6d529` (entry-fix ≠ residual-hang fix). diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_EMERGENT_ECONOMY_DESIGN.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_EMERGENT_ECONOMY_DESIGN.md new file mode 100644 index 000000000000..ae6e0d1f03ed --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_EMERGENT_ECONOMY_DESIGN.md @@ -0,0 +1,185 @@ +# DSWS Emergent Wave-Economy — Design (Step 1 of the "feed + kill-the-8-wave-limit" work) + +**Date:** 2026-07-05 +**Status:** design, pre-plan. Follows `DSWS_STAGGER_DESIGN.md` / `DSWS_STAGGER_PLAN.md` (write-once + stagger, both oracle-clean to 14 waves/WG). +**Council:** Fable + Codex consulted; verdict + corrections baked into §5. + +--- + +## 1. Goal + +Stop specifying the wave mix at all. The VGPR-budget accounting we just built should *decide* the +wave economy at runtime — how many waves are fat compute vs lean feed emerges from the physical VGPR +file, moment to moment. **No artificial wave-count limit.** Owner's words: *"we built this elegant +accounting… so why are we even specifying waves? Let the accounting decide; set no artificial limit."* + +This is **Step 1 of two**. It removes the baked `NCOMP/NAFEED/NBFEED` mix and lets the economy emerge. +It is **not** the operand-overlap feed fix (`POOL_N=2` via `SEGK=16`) — that is **Step 2**, a separate, +separately-measured experiment (§7). The two are orthogonal and must not be conflated (§5, Risk B). + +--- + +## 2. The mechanical truth (why this is a subtraction, not a rewrite) + +Grounded in the current `occ_kernel_dsws_flow.s`: + +- **Concurrent-fat already emerges from hardware.** The per-burst grow (`.Lflow_compute`) is a raw + `s_alloc_vgpr NFV` → `s_cbranch_scc0 .Lflow_growfail`. SCC=0 means the *physical VGPR file* could not + grant the grow. That is the moat and the true cap. The `BUDGET` / `reserve_try` LDS ledger + (`VRESV_OFF=52`) is used **only** by `conv_apply` (the old role-conversion envelope), which is + **dormant** in the flow build. So we do **not** need to build a budget mechanism — the hardware is it. +- **No runtime role emergence exists today.** The coordinator's sense/nudge is deferred + (`.Lflow_coord_period`: *"static launch mix for the first flow build"*). A wave reads a **static** + `ROLE[wid]` each loop and simply *is* that role; the coordinator only ever rewrites `ROLE[wid]` to + `ROLE_RETIRE` at the end. +- **The only fungibility is the coast.** A COMPUTE wave with nothing staged, or whose grow fails, + runs feed code (`.Lflow_coast` → helps whichever of A/B is behind) and returns to `.Lflow_loop`. + Every path (`.Lflow_feed`, `.Lflow_feed_empty` = `s_sleep;branch`, `.Lflow_stage_adv`, bank-done, + coast) loops back to the head, which re-reads `ROLE[wid]` and re-dispatches. So a **compute-seeded + wave already oscillates compute↔feed** by (staged-availability × hardware-grow-success) each + iteration. This flow is **asymmetric**: compute→feed only. A feed-seeded wave never tries compute. + +**Consequence:** the "baked mix" problem is entirely in (a) the *seed* (feed-seeded waves are stuck +feeding) and (b) the assemble-time coupling of launch count / guards to `NCOMP/NAFEED/NBFEED`. Fix the +seed to be compute-biased with a minimal feed floor, decouple the launch count, and the economy emerges +using machinery that already exists and is already oracle-clean. + +--- + +## 3. Design + +### 3.1 Compute-biased seed with a static liveness floor +Replace the mix-proportional seed (`wid coordinator (runs lean B-feed between ASSIGN duties, as today) +wid 1 -> ROLE_AFEED (static A-feed floor) +wid 2 -> ROLE_BFEED (static B-feed floor) [wid0 already B-class, so this is the 2nd B] +wid 3 .. W-1 -> ROLE_COMPUTE (everything else) +``` + +- The floor (`≥1` dedicated A-feed + `≥1` dedicated B-feed, plus the coordinator's B-class feeding) is + a **liveness invariant**, not a mix (§5, Risk A). At `POOL_N=1` the pipeline is depth-1 ping-pong; + liveness *requires* someone always able to fill the operand pool regardless of the compute cohort's + phase. This guarantees it deterministically, cheaply, with no spin. +- Everything above the floor is COMPUTE-seeded. Those waves self-distribute: when work is staged and a + grow succeeds they burst; when the budget is full (`growfail`) or nothing is staged they coast to + feed and fill the pool; next iteration they try compute again. **Concurrent-fat = whatever the + physical VGPR file admits** (the `growfail` boundary). No number is imposed. +- The floor is fixed by `wid`, independent of any `NCOMP/NAFEED/NBFEED`. Those three symbols are + **deleted** from the kernel. + +### 3.2 Decouple the launch count from the mix +- Introduce a single wave-count defsym (e.g. `WAVES` becomes a standalone `-defsym`, default the + swept-safe max). It no longer equals `NCOMP+NAFEED+NBFEED`. Everything sized off `WAVES` (the + 32-wave mailbox region, the retire `.rept WAVES`, guards) keeps working — it already tolerates up + to 30 (see below). +- `G` (rowblk-group size = LDS bank count) is **structural**, currently written `.set G, 6 // = NCOMP_MAX`. + Make `G` its own defsym (default 6), *not* tied to the launch mix. `ACC_N` (banks) stays coupled to `G`. + +### 3.3 `BUDGET` becomes a physical constant, decoupled from the mix +- `BUDGET` today = `NCOMP*NFV + (NAFEED+NBFEED)*VLEAN` (mix-derived). With the mix gone, redefine it as + a **physical VGPR budget** defsym `VBUDGET` (the VGPR-file credit ceiling), independent of wave count. +- This does **not** change concurrent-fat behavior (the hardware `s_alloc_vgpr` is the real cap; the + ledger is dormant). Its only jobs now are (i) keeping the assemble-time sanity guards meaningful and + (ii) being available if we ever re-arm `conv_apply`. Keep the guards as physical sanity checks: + - `WAVES*VLEAN ≤ VBUDGET` (all waves fit lean) + - `WAVES*VLEAN + (NFV-VLEAN) ≤ VBUDGET` (≥1 wave can grow — liveness for compute progress) + +### 3.4 Host auto-derives `W_launch` (Option 3 — no wave number anywhere) +The host picks the launch count as the **min of all real caps**, expecting VGPR to almost never be the +winner: + +``` +W_launch = min( occupancy_cap, // swept empirically (resident waves/SIMD, ~16 wave32) + coordinator_cap = 30, // mailbox-tail squat (§3.5); hard structural ceiling + lean_fit = (VBUDGET - (NFV-VLEAN)) / VLEAN ) // all-lean + one grow of headroom +``` + +- Emergence belongs on the **fat/lean time profile** (the ledger/hardware already deliver it) and on + the **role mix above the floor** — **not** on resident wave count, which is an + occupancy-and-coordinator decision the ledger never measured. +- The host stops reading `DSWS_NCOMP/NAFEED/NBFEED` for the flow path and stops naming the bin per-mix. + One flow bin, built at `WAVES=W_launch`; the host launches `W_launch*32` threads/WG. +- `occupancy_cap` is the one unknown → sweep it (§7). Until swept, gate `W_launch` at a conservative + value and raise it by measurement. + +### 3.5 Mailbox / coordinator safety at larger counts +- The mailbox already reserves a full 32-wave region: `ROLE_BASE=20 … SLOTC_BASE=148`. The coordinator + state squats at `COORD_KSI_OFF=140` / `COORD_T_OFF=144` = `ROLE[30]` / `ROLE[31]`. Hence **`WAVES ≤ 30` + is safe today**; `WAVES` of 31–32 would corrupt coord state **silently**. +- `W_launch` is therefore hard-capped at **30** in §3.4. This is comfortably above the expected + occupancy cap (~16 wave32), so we do **not** need to relocate coord state for Step 1. (If a future + step wants 31–32, move `COORD_KSI/T` past the 32-wave region — out of scope here.) + +--- + +## 4. What we deliberately do NOT build + +- **Option 2 (fully fungible, one uniform wave body): rejected.** Council-unanimous. Dyn-VGPR already + decouples launch footprint from peak, so fusion doesn't save VGPR; its real cost is that feed would + become a *residual of compute contention* (you feed only when a grow fails), coupling the bottleneck + resource to the noisy grow-failure rate the stagger is built to damp. Keeping distinct role bodies + keeps feed a **controlled variable with a floor**. Also: it's a rewrite of a just-clean kernel. +- **Re-arming the coordinator sense/nudge / `conv_apply` conversion:** not needed. The compute-biased + seed + coast + hardware grow-fail deliver emergence without it. `conv_apply` stays dormant. +- **The `POOL_N=2` / `SEGK=16` feed-overlap fix:** that is Step 2 (§7), not this change. + +--- + +## 5. Risks & mitigations (from the council) + +- **Risk A — liveness / stampede (both agents).** All-lean-unassigned could stampede all-compute + (nobody feeds → compute stalls on an empty pool) or all-feed (nothing computes). **Mitigation:** the + static feed floor in §3.1 is a *correctness invariant*, not tuning. At `POOL_N=1` the single operand + slot also self-serializes, and `.Lflow_feed_empty` yields safely; deadman covers any stall. +- **Risk B — this is ONE change or TWO? (both agents): TWO.** Emergent mix governs *who* works; + `POOL_N` governs *whether feed and compute overlap at all*. Orthogonal. Killing the wave cap alone can + even **regress** (more waves eligible to grab compute → more pressure on the one slot). The feed win + is gated on `POOL_N=2` (`SEGK=16`), which is not free: halving SEGK doubles K-segments → doubles + `ds_add` + `RBDONE` churn **and halves the WMMA burst length**, attacking the stagger's grow/shrink + amortization. **Mitigation:** ship Step 1, re-confirm oracle + stagger self-maintenance, measure; + *then* do Step 2 with its own before/after. +- **Risk C — the VGPR file is NOT the binding cap (Fable; the one I underweighted).** `VBUDGET`/VLEAN + math lands ~45 waves, but occupancy (~16 wave32) and the coordinator ≤30 bind first. **Mitigation:** + §3.4 derives `W_launch = min(occupancy, 30, lean_fit)` — VGPR is a sanity ceiling, not the driver. +- **Risk D — coordinator correctness under dynamic churn (Codex).** Oracle-clean at 14 waves does not + prove stability near the physical boundary where grow-fails, coasting, pool state, and completer + election interact. **Mitigation:** strongest oracle + repeats at each new wave count; watch for the + first sign of a completer/mailbox race and root-cause immediately (do not defer). + +--- + +## 6. Acceptance criteria (Step 1 done = all hold) + +1. **Correctness:** oracle `bad=0`, `max_rel=0`, `occ[0]=0` at the new (larger) `W_launch`, with repeats + (Risk D). Re-confirm at the old 4c2a2b-equivalent count too. +2. **No mix anywhere:** kernel has no `NCOMP/NAFEED/NBFEED`; host does not read `DSWS_NCOMP/AFEED/BFEED` + for the flow path and does not name the bin per-mix; `W_launch` is host-derived. +3. **The stagger engages:** `grow-fail` (STAGINSTR `occ[73]`) flips from **0 → >0** — the proof the + physical VGPR file finally binds and the stagger's coast-repulsion is doing real work. Peak resident + waves (`occ[1]`) rises meaningfully above the old 8. +4. **Stagger self-maintains:** no wedge/brick; deadman `occ[0]` stays 0; coast fraction sane (not ~100% + feed-starve *and* not ~0% — some genuine grow-fail repulsion). + +Measurement uses the STAGINSTR counters already in place: `occ[70]=coast`, `[71]=computed`, `[72]=feed`, +`[73]=growfail`, plus `occ[1]` resident peak and `occ[58]` peak concurrent fat. + +--- + +## 7. Step 2 (sketch — separate experiment, not part of this plan) + +`POOL_N=2` double-buffer so feed and compute overlap instead of serializing on one operand slot. +Needs `SEGK=16` to halve operand footprint and fit two slots + `ACC_N=6` banks in 64 KB. Costs: 2× K-seg +count (2× `ds_add`/`RBDONE` churn), halved WMMA burst (stagger amortization pressure), and the "ksi=0 +write vs add" first-segment handling must become an explicit bank-zero (two ksi in flight otherwise race +the write/add). Land and measure on its own after Step 1 is clean. + +--- + +## 8. Open calibration items (resolved during the plan, not before) + +- `VBUDGET` exact value (physical VGPR-file credit for R9700 wave32) — sweep/confirm; default generous. +- `occupancy_cap` — sweep resident waves/SIMD to find the real ceiling below 30. +- Whether the coordinator (wid0) B-feed duty + `wid1` A-feed + `wid2` B-feed is the right floor, or + `1×A + 1×B` (drop the redundant 2nd B) suffices — decide by the liveness sweep. diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_EMERGENT_ECONOMY_PLAN.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_EMERGENT_ECONOMY_PLAN.md new file mode 100644 index 000000000000..335feda54136 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_EMERGENT_ECONOMY_PLAN.md @@ -0,0 +1,363 @@ +# DSWS Emergent Wave-Economy — Implementation Plan (Step 1) + +> **For agentic workers:** implement task-by-task. Steps use checkbox (`- [ ]`) syntax. Spec: +> `DSWS_EMERGENT_ECONOMY_DESIGN.md`. This is Step 1 (kill the baked mix); Step 2 (POOL_N=2 feed +> overlap) is a separate later plan. + +**Goal:** Delete the baked `NCOMP/NAFEED/NBFEED` wave mix. Launch one generous lean pool seeded +compute-biased with a static feed floor; the wave economy (who is fat compute vs lean feed) emerges at +runtime from the hardware `s_alloc_vgpr` grow-fail + the existing coast. Launch count is host-derived, +capped at 30. + +**Architecture:** A *subtraction* on `occ_kernel_dsws_flow.s` (concurrent-fat already emerges from +hardware; the only fungibility is the coast; roles are static-from-seed). Change the seed to +compute-biased+floor, decouple `WAVES` from the deleted mix, make `BUDGET` a physical constant +(`VBUDGET`), and have the host derive `W_launch` and load a single per-count flow bin. + +**Tech Stack:** amdgcn-amd-amdhsa assembly (gfx1201 wave32), C++17 host (raw PM4/KFD), bash build. + +## Global Constraints + +- **NO GPU dispatch without kmbandy's individual greenlight.** Tasks 1-3 verify OFFLINE (assemble / + compile / dry-run only). Task 4 is the single greenlit GPU checkpoint. +- GPU data logs → `/home/kmbandy/dsws_gpu_logs/` (real disk), never tmpfs. DEADMAN on. Never `--gl2c`. + Keep SAFEPROBE + bounds gate. Commit to git ONLY when kmbandy asks. +- `WAVES ≤ 30` is a hard structural ceiling (coordinator state squats at `ROLE[30]/[31]`). +- Do not re-arm `conv_apply` / coordinator sense-nudge. Do not touch `POOL_N`/`SEGK` (that's Step 2). +- Run config for all builds/runs stays: `SEGK=32 POOL_N=1 ACC_N=6 G=6` (+ `TFPROBE=1 STAGINSTR=1`). + +--- + +### Task 1: Kernel — delete the baked mix, physical BUDGET, compute-biased seed + +**Files:** +- Modify: `occ_kernel_dsws_flow.s` (5 edits below) + +**Interfaces:** +- Consumes: `-defsym,WAVES=` and `-defsym,VBUDGET=` from the build (Task 2). +- Produces: a bin whose per-WG role economy emerges from a compute-biased seed; no `NCOMP/NAFEED/NBFEED`. + +- [ ] **Step 1a: Cut the dangling `STAGGER_PERIOD → NCOMP` reference (line ~202-204).** + +Replace: +``` +.ifndef STAGGER_PERIOD + .set STAGGER_PERIOD, NCOMP // phase slots in the stagger ring (R3 sweep). Used iff STAGGER=1. +.endif +``` +with: +``` +.ifndef STAGGER_PERIOD + .set STAGGER_PERIOD, 4 // phase slots in the stagger ring (R3 sweep). Used iff STAGGER=1 (inert here). +.endif +``` + +- [ ] **Step 1b: Fix the `G` comment (line ~131-133).** `G` is already `.ifndef`-guarded and + defsym-driven — only decouple the misleading comment: +``` +.ifndef G + .set G, 6 // cooperative M-extent (rowblks per super-tile) = LDS accumulator-bank count (ACC_N) +.endif +``` + +- [ ] **Step 1c: Replace the role-count block with a standalone `WAVES` + mailbox guard (line ~362-374).** + +Replace: +``` +.if DSWS2 + // ---- role counts (lifted from coop's `.ifndef NCOMP` etc., gated under DSWS2) ---- + .ifndef NCOMP + .set NCOMP, 4 // compute waves (fat, dyn-grow). Compute floor >= 1. + .endif + .ifndef NAFEED + .set NAFEED, 2 // A-feed waves (lean). Feed floor >= 1. + .endif + .ifndef NBFEED + .set NBFEED, 2 // B-feed waves (lean). Feed floor >= 1. + .endif + .set WAVES, (NCOMP + NAFEED + NBFEED) // total waves launched per WG (harness dims must match) +.endif +``` +with: +``` +.if DSWS2 + // ---- launch wave count (EMERGENT economy: NO baked compute/feed mix; roles emerge at runtime) ---- + .ifndef WAVES + .set WAVES, 16 // waves/WG launched; host launches the SAME count. + .endif + .if WAVES > 30 + .error "WAVES>30 collides with COORD_KSI/T at ROLE[30]/ROLE[31] -- relocate coord state first" + .endif +.endif +``` + +- [ ] **Step 1d: Make `BUDGET` physical (`VBUDGET`) + add always-on sanity guards (line ~881-900).** + +Replace the whole `.ifndef BUDGET … .endif` block AND the two gated guard blocks: +``` +.ifndef BUDGET +.if DSWS2_ENVELOPE + .set BUDGET, (WAVES*VLEAN + PEAK_CONC*(NFV-VLEAN)) // rolling: lean floor + concurrent-peak headroom +.else + .set BUDGET, (NCOMP*NFV + (NAFEED+NBFEED)*VLEAN) // = VRESV_OFF init (conservation ceiling) +.endif +.endif + +.if DSWS2_CONV +// compile-time no-parking invariant: every launched wave must fit lean at once +.if (WAVES * VLEAN) > BUDGET + .error "WAVES*VLEAN exceeds BUDGET — pool cannot stay all-lean (parking is out of scope)" +.endif +.endif +.if DSWS2_ENVELOPE +// forward-progress: the budget must admit at least one concurrent peak or a claimed wave can never grow +.if (WAVES*VLEAN + (NFV-VLEAN)) > BUDGET + .error "ENVELOPE: BUDGET admits < 1 concurrent peak — forward progress impossible" +.endif +.endif +``` +with: +``` +.ifndef VBUDGET + .set VBUDGET, 1536 // physical VGPR-file credit ceiling (R9700 wave32, per SIMD). Calibrate. +.endif // Sanity ceiling only: the hardware s_alloc_vgpr is the real concurrent-fat cap. +.ifndef BUDGET +.if DSWS2_ENVELOPE + .set BUDGET, (WAVES*VLEAN + PEAK_CONC*(NFV-VLEAN)) // rolling: lean floor + concurrent-peak headroom +.else + .set BUDGET, VBUDGET // EMERGENT: budget is PHYSICAL, not mix-derived (ledger is dormant; conv-only). +.endif +.endif + +// emergent-economy PHYSICAL sanity (always on): all waves fit lean, and >=1 can grow. +.if (WAVES * VLEAN) > BUDGET + .error "WAVES*VLEAN exceeds VBUDGET -- pool cannot stay all-lean" +.endif +.if (WAVES*VLEAN + (NFV-VLEAN)) > BUDGET + .error "VBUDGET admits < 1 concurrent grow -- compute can never make progress" +.endif +``` + +- [ ] **Step 1e: Replace the seed with compute-biased + static floor (line ~1345-1357).** + +Replace: +``` + // seed per-wave ROLE mailbox with the launch mix: wid BFEED (wid0=coordinator, B-class), + // wid AFEED, else COMPUTE + .set w, 0 + .rept WAVES + .if w < NBFEED + lds_put (ROLE_BASE + w*4), ROLE_BFEED + .elseif w < (NBFEED + NAFEED) + lds_put (ROLE_BASE + w*4), ROLE_AFEED + .else + lds_put (ROLE_BASE + w*4), ROLE_COMPUTE + .endif + .set w, w+1 + .endr +``` +with: +``` + // EMERGENT economy seed: minimal liveness FLOOR + everything else COMPUTE. wid0=coordinator (runs + // lean B-feed between ASSIGN duties), wid1=dedicated A-feed, wid2=dedicated B-feed; wid>=3=COMPUTE. + // Excess compute waves self-distribute to feed via .Lflow_coast; concurrent-fat emerges from the + // hardware s_alloc_vgpr grow-fail. NO baked NCOMP/NAFEED/NBFEED. + .set w, 0 + .rept WAVES + .if w == 0 + lds_put (ROLE_BASE + w*4), ROLE_BFEED + .elseif w == 1 + lds_put (ROLE_BASE + w*4), ROLE_AFEED + .elseif w == 2 + lds_put (ROLE_BASE + w*4), ROLE_BFEED + .else + lds_put (ROLE_BASE + w*4), ROLE_COMPUTE + .endif + .set w, w+1 + .endr +``` + +- [ ] **Step 1f: Verify no live `NCOMP/NAFEED/NBFEED` references remain (comments OK).** + +Run: `grep -n "NCOMP\|NAFEED\|NBFEED" occ_kernel_dsws_flow.s` +Expected: only comment lines (e.g. the `conv_apply`/`conv_dec_floor` doc comments ~912/995). No `.set`, +no expression use. If any live use remains, resolve before building. + +- [ ] **Step 1g: Assemble clean at WAVES=16 (offline, safe).** + +Run: `WAVES=16 VBUDGET=1536 SEGK=32 POOL_N=1 ACC_N=6 G=6 TFPROBE=1 STAGINSTR=1 ./build_flow.sh` +(requires Task 2 first — build_flow.sh must pass the new defsyms). Do Task 2 before 1g. +Expected: `OK occ_dsws2_w16_flow_gd.bin (B .text)`. Then check no register spill: +`grep -i "spill" /tmp/flow_build.err || echo "0-spill"` → `0-spill`. + +- [ ] **Step 1h: Assemble at WAVES=8 too (regression sanity).** + +Run: `WAVES=8 VBUDGET=1536 SEGK=32 POOL_N=1 ACC_N=6 G=6 TFPROBE=1 STAGINSTR=1 ./build_flow.sh` +Expected: `OK occ_dsws2_w8_flow_gd.bin`, 0-spill. (Seed differs from the old 4c2a2b — this is the +intended behavior change, not a byte-identity target.) + +--- + +### Task 2: Build — `build_flow.sh` passes `WAVES`/`VBUDGET`, drops the mix + +**Files:** +- Modify: `build_flow.sh` (the `mkflow` defsym line + tag + the `c/a/b` arg plumbing) + +**Interfaces:** +- Consumes: env `WAVES`, `VBUDGET` (plus existing `SEGK/ACC_N/G/POOL_N/...`). +- Produces: `occ_dsws2_w_flow_gd.bin` (name no longer carries the mix). + +- [ ] **Step 2a: Rewrite `mkflow` to defsym `WAVES`/`VBUDGET`, drop `NCOMP/NAFEED/NBFEED`, rename tag.** + +In `mkflow`, change the tag line to: +``` + local tag="occ_dsws2_w${WAVES:-16}_flow_gd" +``` +In the `clang` invocation, replace the mix defsyms +`-Wa,-defsym,NCOMP=$1 -Wa,-defsym,NAFEED=$2 -Wa,-defsym,NBFEED=$3` with: +``` + -Wa,-defsym,WAVES=${WAVES:-16} -Wa,-defsym,VBUDGET=${VBUDGET:-1536} \ +``` +Also add `-Wa,-defsym,G=${G:-6}` to the defsym list if not already present (Task 1b relies on `G` being +a defsym; the current script passes `G=6` literally — keep it or make it `${G:-6}`). + +- [ ] **Step 2b: Drop the positional `c/a/b` handling at the call site.** + +Replace the bottom block: +``` +c=${1:-4}; a=${2:-2}; b=${3:-2} +echo "== FIX 1 flow bin (occ_kernel_dsws_flow.s; G=6 SEGK=64 FM=2 FN=4 POOL_N=${POOL_N:-3}) ==" +mkflow "$c" "$a" "$b" +``` +with: +``` +echo "== flow bin (occ_kernel_dsws_flow.s; EMERGENT mix; WAVES=${WAVES:-16} G=${G:-6} SEGK=${SEGK:-64} POOL_N=${POOL_N:-3} VBUDGET=${VBUDGET:-1536}) ==" +mkflow +``` +And change the `mkflow()` header comment/params — it no longer takes `$1 $2 $3`. + +- [ ] **Step 2c: Build succeeds (this is also Task 1g/1h).** + +Run: `WAVES=16 VBUDGET=1536 SEGK=32 POOL_N=1 ACC_N=6 G=6 TFPROBE=1 STAGINSTR=1 ./build_flow.sh` +Expected: `OK occ_dsws2_w16_flow_gd.bin (B .text) [POOL_N=1 ...]`, exit 0. + +--- + +### Task 3: Host — derive `W_launch`, single per-count flow bin, skip mix for flow + +**Files:** +- Modify: `occ_dispatch.cpp` — the DSWS2 dispatch block (~6040-6127) and the flow bin-name (~6113). + +**Interfaces:** +- Consumes: env `FLOW_WAVES` (default 16), `DSWS2_FLOW`. Physical constants `NFV=112`,`VLEAN=32` for the + sanity check (or reuse existing helpers if present). +- Produces: launches `W_launch*32` threads/WG; loads `occ_dsws2_w_flow_gd.bin`; `run_dsws2` + receives `(W_launch, 0, 0)` so `WAVES_LAUNCH == W_launch` (C-sizing uses `Gv`, unaffected). + +- [ ] **Step 3a: Derive `W_launch` and override the mix for the flow path (just before the bin block, ~6104).** + +Insert, right after `char dswsBin[160];` and before the `if (getenv("DSWS2_FLOW"))` name switch: +```cpp + // EMERGENT economy (flow path): no baked mix. Launch a host-derived pool, capped at 30 + // (coordinator mailbox squat) and sanity-checked against the lean-fit budget. + if (getenv("DSWS2_FLOW")) { + uint32_t Wlaunch = getenv("FLOW_WAVES") ? (uint32_t)atoi(getenv("FLOW_WAVES")) : 16u; + if (Wlaunch < 4) Wlaunch = 4; // floor(3) + >=1 compute + if (Wlaunch > 30) { printf(" [flow] FLOW_WAVES=%u > 30 (coord cap) -> clamping to 30\n", Wlaunch); Wlaunch = 30; } + const uint32_t VB = getenv("FLOW_VBUDGET") ? (uint32_t)atoi(getenv("FLOW_VBUDGET")) : 1536u; + const uint32_t leanFit = (VB - (112u - 32u)) / 32u; // (VBUDGET-(NFV-VLEAN))/VLEAN + if (Wlaunch > leanFit) + printf(" [flow] WARNING FLOW_WAVES=%u exceeds lean-fit=%u for VBUDGET=%u (bin's .error will catch a real overflow)\n", Wlaunch, leanFit, VB); + c.nComp = Wlaunch; c.nAfeed = 0; c.nBfeed = 0; // WAVES_LAUNCH = Wlaunch inside run_dsws2 + } +``` + +- [ ] **Step 3b: Name the flow bin by wave count (~6113).** + +Change the flow branch: +```cpp + if (getenv("DSWS2_FLOW")) + snprintf(dswsBin, sizeof dswsBin, "occ_dsws2_%uc%ua%ub_flow_gd.bin", c.nComp, c.nAfeed, c.nBfeed); +``` +to: +```cpp + if (getenv("DSWS2_FLOW")) + snprintf(dswsBin, sizeof dswsBin, "occ_dsws2_w%u_flow_gd.bin", c.nComp); // c.nComp == W_launch (3a) +``` + +- [ ] **Step 3c: Skip the positional-mix REFUSE for the flow path (~6055-6076).** + +Wrap the `if (g_posMixArg) { sscanf … REFUSE … }` block so it is skipped when `DSWS2_FLOW` is set: +```cpp + if (!getenv("DSWS2_FLOW") && g_posMixArg) { + // ... existing positional-mix parse + mismatch REFUSE ... + } +``` +(The `nComp<1` / `N()` sum checks below stay; with `(Wlaunch,0,0)` they pass: `nComp>=4`.) + +- [ ] **Step 3d: Compile the host (offline, safe).** + +Run: +``` +clang++ -std=c++17 -O2 -I ../dvgpr_pm4/vendor/compat -I ../dvgpr_pm4/vendor -I ../dvgpr_pm4 -I /opt/rocm/include \ + occ_dispatch.cpp fp8_oracle.cpp ../dvgpr_pm4/vendor/PM4Packet.cpp ../dvgpr_pm4/vendor/BasePacket.cpp \ + /opt/rocm/lib/libhsakmt.a -ldrm_amdgpu -ldrm -lnuma -lpthread -ldl -lrt -o occ_dispatch +``` +Expected: builds to `occ_dispatch` (hsakmt linter warnings are pre-existing noise). No new errors. + +- [ ] **Step 3e: Dry-run prints the derived launch (offline, safe — DSWS2_DRYRUN, no GPU).** + +Run: +``` +DSWS2_FLOW=1 DSWS2_DRYRUN=1 DSWS2_SEGK=32 DSWS2_ACC_N=6 FLOW_POOL_N=1 FLOW_WAVES=16 \ + DSWS2_ORACLE_MTL=3 DSWS2_ORACLE_NTL=8 DSWS2_NKSEG=64 ./occ_dispatch --dsws2 +``` +Expected: prints `waves/WG=16(=16c0a0b)` and the bin name `occ_dsws2_w16_flow_gd.bin`, returns before any +GPU submit. If `DSWS2_DRYRUN` isn't wired for the flow path, verify by inspection that the derivation + +bin-name print are correct and proceed (do NOT submit to GPU here). + +--- + +### Task 4: GPU checkpoint — oracle-clean at the new pool + emergence proof (GREENLIT ONLY) + +**Files:** none (verification only). + +**Interfaces:** consumes the Task-1/2/3 artifacts (`occ_dsws2_w16_flow_gd.bin`, `occ_dspatch`). + +> **STOP — do not run any command in this task without kmbandy's explicit per-dispatch greenlight.** + +- [ ] **Step 4a: Oracle at W=16 (greenlit).** Log to `/home/kmbandy/dsws_gpu_logs/`. + +Run (after greenlight): +``` +DSWS2_FLOW=1 DSWS2_SEGK=32 DSWS2_ACC_N=6 FLOW_POOL_N=1 FLOW_WAVES=16 \ + DSWS2_ORACLE_MTL=3 DSWS2_ORACLE_NTL=8 DSWS2_NKSEG=64 ML8_POOL=24 \ + ./occ_dispatch --dsws2 2>&1 | tee /home/kmbandy/dsws_gpu_logs/emergent_w16_$(date +%s).log +``` +Expected (acceptance §6): `bad=0`, `max_rel=0`, `occ[0]=0`. Repeat 2-3× for the race check (Risk D). + +- [ ] **Step 4b: Read the emergence proof from STAGINSTR.** + +From the same run's readout: `occ[73]` (grow-fail) should be **> 0** (was 0 at 8 waves — proof the +physical VGPR file finally binds and the stagger's coast-repulsion engages), and `occ[1]` (peak resident +waves) should rise meaningfully above 8. Also sanity-check the coast fraction (`occ[70]`) is neither ~100% +(pure feed-starve) nor ~0%. + +- [ ] **Step 4c: Re-confirm at W=8 (greenlit).** Build `WAVES=8 …` bin (Task 1h) and run with + `FLOW_WAVES=8`; expect `bad=0` — confirms the seed change didn't regress the small pool. + +- [ ] **Step 4d: Record results** in a short note (and memory) — TF, grow-fail, occ[1] at W=8/16, and + whether emergence engaged. Do NOT commit to git unless kmbandy asks. + +--- + +## Self-Review + +- **Spec coverage:** §3.1 seed → Task 1e; §3.2 decouple WAVES/G → 1b/1c; §3.3 physical BUDGET+guards → + 1d; §3.4 host W_launch derive + single bin → Task 3; §3.5 ≤30 cap → 1c guard + 3a clamp; §6 acceptance + → Task 4. Step 2 (POOL_N=2) intentionally excluded. +- **Placeholder scan:** `VBUDGET=1536` and `FLOW_WAVES=16` are explicit calibratable defaults, not TBDs; + Task 4 pins the empirical acceptance numbers. +- **Type/name consistency:** bin name `occ_dsws2_w_flow_gd.bin` matches between Task 2a (tag) and Task + 3b (host). `WAVES` defsym name matches kernel (1c) ↔ build (2a). `(W_launch,0,0)` (3a) makes + `WAVES_LAUNCH` (run_dsws2) == the built `WAVES`. diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_GRESIDENT_DESIGN.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_GRESIDENT_DESIGN.md new file mode 100644 index 000000000000..755d0a5fcb10 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_GRESIDENT_DESIGN.md @@ -0,0 +1,201 @@ +# DSWS — burst-owns-a-ksi-run: engaging the dyn-VGPR stagger without a flush + +**Date:** 2026-07-06 **Kernel:** `occ_kernel_dsws_flow.s` (gfx1201 / RDNA4 / R9700, wave32, raw PM4) +**Status:** design **v2** — folds council review (Fable). Supersedes the shelved `POOL_N=2` framing, the +rejected `WOFLUSH` (atomic-flush) lever, and v1's *count-only* concurrent-fat model. + +> **What changed v1 → v2 (why read this):** v1 modeled concurrent-fat as a pure *count* problem +> (`POOL_N·G_resident`) and proposed shrinking banks (`G_resident < G`) to afford a high `POOL_N`. Fable's +> review found three defects in that: **H1** a silent correctness race in the reduction, **H3** an +> off-by-256 that made every "feasible 64KB" row actually overflow, **H5** a group/tile-boundary bank +> collision the partition scheme creates. v2 adopts Fable's **burst-owns-a-ksi-run**: it makes the +> reduction commutative (kills H1), keeps **whole-tile groups** (`G_resident=G`, no boundary → kills H5), +> and reframes concurrent-fat as **count × duty-cycle** with **burst length** as a free knob on the +> duty axis that costs *no* LDS. Bank-shrink (`G_resident throughput lever behind a proven bind. + +--- + +## 1. The problem (measured, not inferred) + +The dyn-VGPR "traveling-peak" stagger only does work if enough compute waves are **simultaneously fat** +that the per-SIMD VGPR pool (~1536 VGPR; a fat wave = NFV≈112 → ~13 fat/SIMD) actually **binds** → +`s_alloc_vgpr` starts failing → `grow-fail > 0`. + +**Run 4 (2026-07-05):** full grid (128 WGs, saturated 256-tile 3072×1024×2048 shape), `grow-fail = 0`, +98.8% coast. The pool never binds. Root cause is the LDS accumulator: + +- Write-once-C keeps each rowblk's fp32 C sub-tile in an **8KB LDS bank**, persisting across all + `n_kseg` split-K segments; `G=6` rowblks → **48KB** of banks. +- LDS/WG = 57,600B → only **~1 WG/CU** resident (gfx1201 CU ≈ 64KB LDS) → few fat compute waves/SIMD. + Nothing forces waves to compete for VGPR. + +## 2. The trilemma (why this is hard) — and the escape + +The K-reduction running-sum must live *somewhere* on-chip to be cheap, and to stagger (short fat bursts) +the fat accumulator must be **evicted between bursts** — a dyn-VGPR *shrink* is only legal if its contents +are safe first. v1 framed the three eviction targets as a hard trilemma: + +| Running-sum home | Cost | +|---|---| +| **VGPR, whole K** (grind) | square wave — no trapezoid, no stagger; wave held fat entire K-loop | +| **LDS bank** (write-once) | 48KB persistent → caps occupancy at 1 WG/CU → **pool never binds** | +| **Global atomic** (WOFLUSH) | RMW round-trip + same-cell serialization (the 26–51% FLUSH we deleted) | + +**The escape (Fable):** the trilemma is false because "VGPR whole-K" and "LDS bank every ksi" are the two +*endpoints* of a knob, not the only choices. Let a fat burst own a **run of J consecutive ksi** for one +rowblk: accumulate those J ksi in the **VGPR ACC** (no LDS touch between them), then do **one** `ds_add` +merge of that partial into the rowblk's LDS bank. `J = n_kseg` is the whole-K endpoint (square wave); +`J = 1` is the write-once endpoint (bank every ksi). **J is a continuous duty-cycle knob between them**, +and it costs no LDS. The bank still persists across bursts (it holds the running sum of merged runs), but +it is *touched* only once per burst, not once per ksi. + +WOFLUSH stays rejected: it re-introduces the exact atomic-flush wait write-once removed. + +## 3. The insight — concurrent-fat is *count × duty-cycle*, not count + +v1's error: it treated instantaneous-fat as `min(POOL_N·G_resident, WAVES)` — a pure **count**. But a +wave is only fat *while it grinds*; between bursts it goes thin (fetch operands, merge, re-claim). So: + +> **instantaneous-fat ≈ (waves resident) × (fraction of time each is fat) = count × duty-cycle.** + +To bind the pool you need instantaneous-fat/SIMD ≳ 13–14. Two independent levers get you there: + +1. **Count** — more concurrent `(rowblk, ksi-run)` claims live at once (`POOL_N`, `G_resident`, `WAVES`). + Costs LDS (operand slots + banks). This was v1's only lever. +2. **Duty-cycle** — longer fat bursts (**larger J**) → each resident wave spends more of its life fat → + more are simultaneously fat. Costs **no LDS**. This is the lever v1 was blind to. + +Burst length is the free knob on the duty axis. That is what lets us keep **whole-tile groups** +(`G_resident = G`, 48KB banks, 1 WG/CU — the config Run 4 *already* had) and still reach the bind, by +cranking J instead of shrinking banks. + +**The H1 fix that makes any of this legal:** the reduction must be **order-independent**. Today it is not — +`ksi==0` does `ds_store` (init the bank), `ksi>0` does `ds_add`; with concurrent runs of the same rowblk +the init can land *after* an add → silent wrong C. v2: **zero-initialize the banks up front, then *every* +merge (all ksi, all runs) is `ds_add`.** `ds_add` is atomic per-op in LDS, so concurrent merges into one +bank serialize correctly and the result is order-independent. (fp32 add is non-associative, so ordering +still perturbs rounding — but write-once *already* has non-deterministic `ds_add` order across ksi and +passes oracle `bad=0` within tolerance; v2 adds no new nondeterminism class.) + +## 4. The binding model (corrected arithmetic — H3 fixed) + +Per-SIMD fat waves ≈ `instantaneous-fat / 2` (a WG spans 2 SIMDs). Bind ⇒ need **≳ 13–14 fat/SIMD**. + +**LDS constraint (the H3 fix):** `256 + POOL_N·OPSTRIDE + G_resident·8192 ≤ 65536`. +Measured OPSTRIDE: **8192 @ SEGK=32, 4096 @ SEGK=16** (from Run-4 `LDS=57600 = 256+8192+6·8192`). +*(v1 omitted the +256 header; every "64KB" row there actually summed to 65792 = 256 **over** the limit and +the kernel's `.error` would have fired.)* + +### Path A — whole-tile groups (`G_resident = G = 6`), the correctness+bind proof + +Dodges the boundary problem (§7) entirely. Banks = 48KB fixed → **1 WG/CU**. Binding rides on **duty-cycle +(J)**, since the count ceiling is low: + +| SEGK | OPSTRIDE | POOL_N max | banks | LDS (+256) | count = POOL_N·G | +|---|---|---|---|---|---| +| 32 | 8192 | 1 | 48KB | 57,600 | 6 ← **Run 4** | +| 16 | 4096 | 3 | 48KB | 61,696 | 18 | + +Both are **below** the count-threshold of ~26 — so Path A **cannot bind by count alone**; it must bind by +**duty-cycle**. That is the whole point: crank **J** (burst length) up until enough of the ≤30 resident +waves are simultaneously fat. This is the cheapest possible first step — it changes *scheduling* +(burst granularity), not the LDS layout, and reuses Run-4's exact bank geometry. + +### Path B — partition (`G_resident < G`), an *optional* throughput lever behind a proven bind + +Only if Path A binds but leaves throughput on the table do we shrink banks to raise the **count** lever +too. Needs the §7 boundary resolution. Corrected feasible points (SEGK=16, OPSTRIDE=4096, 1 WG/CU), +requiring `POOL_N·4096 + G_resident·8192 ≤ 65280`: + +| POOL_N | G_resident | LDS (+256) | count = POOL_N·G_r | binds by count? | +|---|---|---|---|---| +| 7 | 4 | 61,696 | 28 | **yes** (≥26) | +| 9 | 3 | 61,696 | 27 | **yes** | +| 5 | 5 | 61,696 | 25 | marginal | + +**Key finding (revised):** one WG at `WAVES=30` can bind *either* by count (Path B, `POOL_N·G_resident≳26`) +*or* by duty-cycle (Path A, large J) *or* by both. Path A is the milestone-1 default because it is +correct-by-construction (no boundary hazard) and free (no layout change). `SEGK=16` (halved operand slots) +is what buys headroom on *both* paths. A further lever — dropping banks small enough for **2 WG/CU** — is +noted but not pursued until 1-WG/CU binding is characterized. + +## 5. B-stationary residency + the cache math (why re-fetch is free) + +*(Relevant only to Path B, where splitting a tile into `G/G_resident` groups re-consumes B once per group. +Path A re-fetches nothing new.)* Measured gfx1201 caches: **L2 = 8MB, L3/Infinity-Cache = 64MB** +(`rocminfo`, chip 0x7551). + +- One B panel `B[:,tcol]` = `K·FN·16` fp8 = **128KB** (oracle) … **576KB** (real ml8 down, K=9216). +- Under **B-stationary** scan order (co-resident WGs share `tcol` — the existing `KMAJOR` hook), the L2 + working set = a *handful* of shared panels = **0.5–4.6MB** → fits the 8MB L2 with margin. +- Safety net: the **entire B** fits L3 for our shapes — real down B = 9216×2560 fp8 = **23.6MB < 64MB**; + oracle B = 2MB < 8MB L2. After first touch, every B (re)access is **at worst an L3 hit, never HBM.** + +Re-fetching B per group is a cache hit **by construction**. (DE frame: a partitioned hash-agg — fewer +group-by states in `work_mem`/LDS → more concurrent workers; the re-scan of the small dimension table B +stays in the buffer pool.) + +## 6. Design + +- **Reduction (H1 fix, both paths):** at rowblk-claim time, **zero-initialize** the `G_resident` banks + (`ds_store 0`). Drop the `ksi==0` special case. *Every* merge is `ds_add_f32`. +- **Burst = a ksi-run:** a fat compute wave claims `(tile, rowblk, ksi_lo)` and grinds **J** consecutive + ksi `[ksi_lo, ksi_lo+J)` in the **VGPR ACC** (WMMA-accumulate, no LDS between them), then does **one** + `ds_add` merge of the ACC into the rowblk's bank, shrinks, and re-claims. **J is a build/dispatch knob.** +- **Enumeration:** coordinator emits `(tile, rowblk, ksi_run)` where `ksi_run` indexes `⌈n_kseg/J⌉` + runs. Completer stores each rowblk's bank to C **once** (plain `global_store`, no atomic) after its last + run merges; banks recycle. +- **Path A (default):** `G_resident = G` (whole tile, all rowblks' banks co-resident). No group axis, so + no group-boundary drain — only the existing tile-boundary drain. `SEGK=16` → `POOL_N ≤ 3`. +- **Path B (optional, post-bind):** `ACC_N = G_resident < G`; add a `group ∈ [0, G/G_resident)` axis; + B re-staged per group from L2/L3; requires §7 drain-before-admit. *(Impl caveat: OPSTRIDE/operand-staging + currently bake `G=6`; Path B must re-derive staging for `G_resident`.)* +- **Waves:** `WAVES = 30` (kernel cap; barrier/mailbox already supports ≤30). +- **Unchanged:** on-chip `ds_add_f32` reduction (fp32), count-to-WAVES exit barrier, deadman throttle, + chunking + compositor yield. + +## 7. Correctness + +- fp32 on-chip reduction preserved → oracle `bad=0` expected (same math class as write-once). +- **H1 (was a silent race — now resolved):** zero-init banks + all-`ds_add` ⇒ order-independent ⇒ correct + under any number of concurrent same-rowblk runs. This is a *precondition* for J>1 and for `POOL_N>1`. +- **Path A boundary hazard: none.** Whole-tile groups have no intra-tile group boundary; the pool can only + span a tile boundary, handled by the existing tile-drain. +- **Path B boundary hazard (H5):** with `POOL_N>1` the pool can span two groups whose rowblks map to the + *same* (group-relative) banks; group g+1's runs would `ds_add` into group g's not-yet-stored sums. + **Resolution = drain-before-admit:** the coordinator does not admit group g+1 until group g's banks are + stored + `s_wait` (serializes at *group* granularity only). Slot-indexed banks (`POOL_N·G_resident` + banks) are rejected — too much LDS. This is why Path A goes first. + +## 8. Gates (sequenced, per kmbandy) + +1. **Correctness** — builds, W-anything, oracle `bad=0`, brick-free. Flag-gated so `J=1, G_resident=G` + is byte-identical to today's write-once (the safe fallback). +2. **Bind** — at a work-heavy shape, sweep **J** (Path A) until `grow-fail > 0` — the proof the stagger + engages. *This is the milestone Run 4 failed, and J is the new lever to clear it.* +3. **Is the bind the *real* constraint? (the honesty gate — Fable's caveat):** duty-cycle headroom only + converts to throughput **if VGPR is the binding occupancy constraint.** A prior TF finding hinted one + cap was **structural (tile geometry), not VGPR.** So at Gate 2 also record whether TF *moves* with J — + if `grow-fail>0` but TF is flat, the wall is elsewhere and Path B/POOL_N gymnastics won't help. Measure + before building more. +4. **Throughput** — TF vs the write-once bin; sweep `J × POOL_N × (G_resident) × SEGK × WAVES`. + +## 9. Open questions for the council + +1. Does `ds_add` **contention on a shared bank** (many concurrent same-rowblk runs merging into one bank[r]) + become the new wall as J shrinks / POOL_N grows? (J>1 *reduces* merge frequency — is it enough?) +2. Is `instantaneous-fat = count × duty-cycle` the right silicon model, or does something else cap first + (claim-rate, coordinator throughput, feed bandwidth, the structural/tile-geometry cap from Gate 3)? +3. Optimal `J` for a target shape — and does the best J leave *enough* occupancy headroom to still stagger, + or does binding Path A force J so high it's effectively a square wave (H2's 91%-duty fragility)? +4. If Path A binds but Gate-3 says the wall is structural, is Path B worth building at all — or does the + answer become "fix tile geometry," not "shrink banks"? +5. Any reason B-stationary + Path-B re-staging thrashes L2 in a way the §5 math misses? + +## 10. Instrumentation TODO (before Gate 2) + +Wire the **dead `occ[58]` FATMAX** counter (defined, never written; non-TRACE) to record the running max of +instantaneous-fat, so Gate 2 can *measure* duty-cycle × count directly instead of inferring it from +`grow-fail`. `occ[57]` FATLIVE (current fat count) pairs with it. Without this we can only see *whether* the +pool binds, not *how close* a non-binding config came — which is exactly what J-sweeping needs. diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_STAGGER_DESIGN.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_STAGGER_DESIGN.md new file mode 100644 index 000000000000..507385e4d4af --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_STAGGER_DESIGN.md @@ -0,0 +1,222 @@ +# DSWS Stagger — Design (2026-07-05) + +The "S" in DSWS (Dynamic **Staggered** Wave-Spec). This builds the staggered traveling-peak +occupancy layer on top of the write-once-C foundation, inside the existing flow economy +(`occ_kernel_dsws_flow.s`). It supersedes the split-K atomic-flush compute path and fills the +occupancy/latency headroom the write-once calibration exposed. + +Sibling docs: `FLOW_ECONOMY_DESIGN.md` (the mailbox/coordinator economy this extends), +`RING_SLOTS_DESIGN.md`, `SPEC_WAVESPEC.md` (MAD-305 traveling-peak origin). + +--- + +## 1. Where we are (the measured starting line) + +- **Write-once-C is calibrated.** The grind kernel (`occ_kernel_grind.s`) — which *is* the + rowblk-owner shape (1 wave/WG, full-K accumulate in VGPR, one `global_store`) — peaks at + **1.5 TF at pool=64** on 576×512×2048, 2×4 tile, oracle-clean. That is ~2× over split-K's + shape-invariant ~0.7 TF. Write-once-C is a real but **bounded ~2×** lever at our size (the + earlier "10×" was a large-shape artifact: grind 6.9 TF at 1920×2048×2048). +- **The wall at our size is NOT memory bandwidth.** Back-of-envelope: 0.813 ms for ~1.2 GFLOP, + moving ~10–30 MB → ~30 GB/s against a card that does hundreds. We are **occupancy/latency-bound**: + at the 1.5 TF peak, occupancy is **0.5 waves/SIMD (3.1%)**. A lone wave streaming a full-K + accumulator has almost nothing to overlap its load latency against. +- **Stagger is aimed at exactly this wall:** raise *effective* occupancy so load-latency stalls are + hidden, without paying more than average VGPR footprint per wave. + +## 2. Thesis: temporal register multiplexing (the traveling peak) + +Governing rule (kmbandy / MAD-305): **the sum of all resident waves' *instantaneous* VGPR +allocations must stay ≤ the per-SIMD budget B at every instant.** If a compute wave's fat period is +*brief* (a trapezoid, not a square wave), its time-averaged footprint ≈ peak/2, so you can pack ~2× +as many resident waves into B as their peak would allow. Fatness becomes **temporal, not +simultaneous** — this is CDNA "ping-pong" expressed through `s_alloc_vgpr`. + +**Precondition = short bursts.** A full-K accumulator held at peak for the entire K-loop is a +~95%-duty **square wave**: avg ≈ peak, staggering buys nothing, and high pool oversubscribes the +per-SIMD file → the classic grow-spin deadlock. This is why the earlier "stagger" attempt (full-K, +phase-offset independent waves, no feedback) **died** — lockstep re-synchronized it. Split-K short +bursts create the **trapezoid duty cycle** that makes both the packing *and* the stagger-maintaining +feedback (§5) physically possible. Split-K is not a side optimization; it is the precondition. + +## 3. Substrate: split-K short bursts → per-rowblk LDS accumulator → write-once C + +The compute path changes from "one K-segment, atomic-flush to C" to a **WG-local reduction that +writes each C cell once** — and the pin unit is a **rowblk-group that shares each B fetch** (NOT +serialized rowblks, which would cost `G×` B traffic and is rejected): + +- **Enumeration change:** the global work counter `occ[20]` enumerates **`(mblk,tcol)` tiles**, not + `(mblk,tcol,ksi)` segments. A WG pins a tile and walks `ksi = 0..n_kseg-1` internally (`SL_KSI` + cursor). The host terminal count drops the `n_kseg` factor. +- **Rowblk-group sharing B (the reuse-preserving core):** a **burst = one K-segment computed across a + group of `g` rowblks that share the single `B[ksi]` fetch**, each rowblk accumulating into its own + **LDS bank** (`ACC_N = g` banks, 8 KB each). This keeps the *segment-outer* ordering that gives + today's B-reuse: `B[ksi]` is fetched once and consumed by all `g` rowblks. `g = G` (=6) = **identical + reuse to the current kernel, zero regression**. Within a burst the wave iterates the `g` rowblks + (WMMA into VGPR → `ds_add_f32` into that rowblk's bank → next), so VGPR peak is **one** rowblk's + accumulator while the `g` banks live in LDS. Then the wave shrinks — the burst is still just one + segment of K (brief → trapezoid → staggerable). +- **Write-once:** after the `n_kseg`-segment walk, each of the `g` banks holds a rowblk's full-K sum; + each is `global_store`d **once**. C-write traffic drops `n_kseg`× with **no loss of B-reuse**. +- **Reduction primitive: `ds_add_f32`** (confirmed present on gfx1201: encodings `D8540000` / + `ds_add_rtn_f32 D9E40000`; final confirmation is the oracle on a GPU run). A compute wave's burst: + `try-grow → WMMA one K-segment into VGPR acc → ds_add_f32 into the 8 KB LDS rowblk accumulator → + shrink`. The **last** segment to complete the rowblk (its done-counter hits `n_kseg`) emits **one + `global_store`** of the accumulator to C — no atomics on the C-write, no per-cell amplification. +- **Fallback A (deterministic):** if `ds_add_f32` misbehaves on hardware, fall back to a + frontier-ordered `ds_store`+`v_add` reduction (segment `s` adds after `s−1`, gated by a counter — + reuses DRAIN_HEAD machinery). A is bit-exact; B is not (FP add non-associative) — but B is **not a + determinism regression**, since today's `global_atomic_add_f32` is already unordered. +- **Bursts, not epochs.** In the current flow kernel a compute wave grows *once* on adopting the + compute role and stays fat across many rowblks. For the trapezoid we need grow/shrink **per burst** + (per K-segment, or per short window of K-segments). Burst length is the master knob (§5). +- **Accumulator lifecycle:** on claiming a rowblk, zero its 8 KB LDS accumulator (`ds_store` zeros, + or first-segment-writes-instead-of-adds); each segment `ds_add_f32`s its partial; the completer + (`RBDONE == n_kseg`) `global_store`s once, then the accumulator region is recycled for the next + rowblk. Codex's forward-progress rule holds: the completer must observe completion only *after* all + prior segments have drained their DS ops (`s_wait_dscnt 0` before the done-increment). + +**LDS budget (a real constraint, not a footnote).** `ACC_N = g` = the rowblk-group size = number of +banks that share each `B[ksi]` fetch. **Target `g = G = 6` for full B-reuse parity.** The banks have +**rowblk-group lifetime** (live across the whole `n_kseg` walk), longer than the **segment lifetime** +of operands. Budget: `OP_BASE + operands + g*8192 ≤ 65536`. At `g=6` the banks alone are 49152 B, so +the operand region must be **lean**: stage `B[ksi]` (4096) once per segment, and **stream A +per-rowblk** (`FM` frags = 2048) rather than staging all `g` rowblks' A at once (12288). That gives +`256 + (4096 + 2048) + 49152 = 55552 ≤ 65536` — `g=6` fits with room, full reuse, no double-buffer on +A. Alternatives if the A-stream stalls compute: `g=5` (40960 banks + a 16 KB double-buffered operand +group = 57600, B re-fetched 1.2×) — a clean fallback that still keeps ~full reuse. The exact operand +packing (A-stream vs small double-buffer) is **build-stage-1 sizing**, driven by whether A-streaming +starves the burst; `g` itself stays at `G` unless the packing forces `g=5`. + +## 4. The envelope: coast-on-grow-fail (no per-wave poll) + +The per-SIMD budget is enforced by the **hardware allocator itself**, not by any software counter a +wave reads. `s_alloc_vgpr` grow sets **SCC** (SCC0 = allocation failed). Today the kernel *spins* on +failure (`s_cbranch_scc0 .Lca_alloc` — the unbounded grow-spin the deadman does **not** cover; the +reason we're pinned to M=576). We replace the spin with **coast**: + +- **Try-grow; on SCC0 (budget full) → COAST** (run lean feed code, do useful work) and return to the + loop head to try again next lap. No shared budget counter, no reservation, no wait-for-permission + poll. The hardware allocator *is* the floodgate; coast *is* the water flowing around it. +- **This is explicitly NOT a reserve-or-wait gate.** A software "reserve your delta, wait if the + counter is full" scheme re-introduces the published-value poll (FOLLOW_WAIT) the flow economy + removed. Rejected for that reason. +- **Invariant (load-bearing): commit nothing until the grow succeeds.** Claim the rowblk *after* the + grow; flip any role/population state *after* the grow. On SCC0 the wave has touched **zero** shared + state — a failed grow is a complete no-op to the accounting. Ordering is rigid: + `try-grow → (SCC0) coast, nothing committed` / `(SCC1) now claim + WMMA + count`. +- **Closes the brick gap.** Because coast returns to the loop head — where the deadman watches — the + former uncovered grow-spin becomes a deadman-covered graceful coast. **Stagger and scale-safety are + the same change.** + +## 5. Maintaining the stagger: a self-organizing equilibrium + +We do **not** launch staggered and hope it sticks (that is precisely what the dead attempt did; +lockstep is an attractor and it re-synchronized). Instead the coast-on-fail gate is a **restoring +force** that creates and holds the stagger: + +- When two waves drift toward the same phase (both want peak at once), the SIMD budget can hold only + ~k fat waves; the (k+1)th's grow **fails → it coasts → it is time-shifted away** from the collision. + The gate **repels coincident peaks.** The system self-organizes into "≈k fat at staggered phases" + and *stays* there, because every attempt to clump triggers a re-spacing coast. +- **The stagger is maintained dynamically by the gate, not preserved from launch.** A launch-time + phase offset (stagger each wave's first grow) is a **warm-up** that speeds convergence and avoids an + initial thundering herd — but it is not the mechanism. +- **Burst length is the master knob.** Long burst = better matrix efficiency + amortized grow cost, + but higher duty → weaker stagger (toward the dead square-wave). Short burst = max stagger but more + grow/shrink overhead and needs more concurrent waves to fill the WMMA pipe. **Optimum = shortest + burst that keeps WMMA saturated.** We sweep it. +- **Convergence quality is empirical.** The gate guarantees no over-budget and provides the restoring + force, but whether it settles into a good equilibrium (pipe-saturating, low thrash) vs. a mediocre + one (collide-and-coast oscillation, or too few fat to fill the pipe) is measured, not assumed (§8). + +## 6. Accounting consistency under coast + +The flow economy's hard state tracks **work done, not intended roles.** Frontiers and done-counters +(`RBDONE`, `BFDONE`/`ARDONE`, ASSIGN/STAGE/DRAIN heads) are incremented by *whoever actually does the +work*. The ROLE mailbox is a soft **bias**, never a contract. Therefore a compute-biased wave that +coasts (grow failed) is fully consistent: + +- It didn't bump the compute counter — truthful (it didn't compute). If it fed while coasting, it + bumped feed counters — truthful. **No lie enters the ledger.** +- The rowblk it would have computed stays unclaimed (claim happens *after* grow, §4); the DRAIN + frontier simply doesn't advance for it until *someone* computes it. Nothing lost or double-counted. +- **Self-heals on survey:** the coordinator reads actual progress off the frontiers and re-biases the + mailbox. Coast is transparent to accounting except as "no compute progress from that wave this lap," + which is the truth. +- **Survey tuning (control, not correctness):** the coordinator must not misread budget-*saturated* + compute lag as "assign more compute waves" (that would starve feed — a milder cousin of the old + runaway). The survey should look at **budget saturation** (grow-fail rate, concurrent-fat count), + not raw compute lag. Coast makes a mis-bias *safe* (the extra compute-biased wave just coasts), but + we design the survey to see saturation. + +## 7. The handoff gap (coast-on-fail's one real cost) + +`s_alloc_vgpr` allocates registers free *at that instant*; it cannot reserve against a neighbor's +*future* shrink. So a wave that calls grow a hair before the fat wave shrinks fails and coasts. This +is handled correctly and does **not** waste the slot — **provided coast is a short useful detour with +a prompt retry:** the wave feeds one small unit, retries grow, and catches the shrink a beat later; +the freed registers get reused. The cost is a small **utilization gap** at each handoff (the window +between a shrink and the next wave's next grow attempt). + +- We **do not** pre-engineer this away with a proactive "shrinking-now" handshake — that re-introduces + the published-signal poll and is fragile (mis-timed → over budget anyway). YAGNI: coast-on-fail + first, **measure the gap** (§8), build a tighter handoff only if data demands it. +- **Coast granularity is the gap dial:** one small feed unit per retry — enough to be useful and not + spin (spinning on grow burns issue cycles and contends with the very waves it waits on), short + enough to catch the shrink window. Swept alongside burst length. + +## 8. Instrumentation (required from the start, not bolted on) + +We must *see* the equilibrium form. `PHASEPROBE`-style, occ-buffer atomics, gated (zero bytes when +off): + +- **Concurrent-fat-wave count** — each grow `atomic_inc`s, each shrink `atomic_dec`s; `atomic_max` + captures the peak; a time-integral gives average concurrent-fat. The direct readout of "is the + stagger working" (≈k steady vs. clump-to-peak-then-zero lockstep) and of "sum vs. budget." +- **Grow-fail / coast rate** — count SCC0 grows and cycles-coasted-before-success. This *is* the + handoff-gap measurement (§7). +- **Fat duty cycle** — fraction of a compute wave's life spent fat (the trapezoid shape; warns if we + drifted back toward the square-wave). +- **Confirm per-SIMD budget B** — never directly measured on gfx1201 (MAD-305 R0). Pin it down + empirically (grow until fail from a known-idle SIMD); every packing calc depends on it. + +## 9. Correctness, oracle, safety + +- **Oracle-gated every run.** Expect bit-exact-to-tolerance (grind's tight tier rel 5e-3 / abs 1e-2). + B's reduction is unordered (non-associative FP) — same acceptance basis as today's atomic flush. +- **Deadman now covers the former grow-spin** (coast returns to loop head). Scale-stress becomes safe + in a way it wasn't: this is a safety *improvement*. +- **GPU runs are individually greenlit** by kmbandy; logs to real disk `/home/kmbandy/dsws_gpu_logs`, + never tmpfs. Single-shot first, then sustained (`DSWS2_TARGET_SECS`) with greenlight. Keep + SAFEPROBE + bounds gate; never `--gl2c`. + +## 10. Build stages (incremental, measure between) + +1. **Write path first:** convert the flow compute body to split-K short burst → `ds_add_f32` into an + 8 KB per-rowblk LDS accumulator → single `global_store` on rowblk completion. Grow/shrink still + per-role-epoch (no stagger yet). Oracle-verify write-once at 576×512×2048; confirm it matches the + grind's write-once number (~1.5 TF) inside the economy. **Gate: bit-exact + not slower than split-K.** +2. **Per-burst grow/shrink + coast-on-fail:** move grow/shrink to per-burst; replace the grow-spin + with coast-on-SCC0; add the launch-offset warm-up. Add the §8 instrumentation. **Gate: no wedge, + deadman-clean at scale; instrumentation shows concurrent-fat > 1 and a real duty cycle.** +3. **Tune the equilibrium:** sweep burst length and coast granularity; read grow-fail rate, concurrent + -fat vs. budget, duty cycle; find the shortest burst that keeps WMMA saturated. **Gate: TF beats the + 1.5 TF write-once baseline; occupancy (concurrent-fat) meaningfully > 0.5 waves/SIMD.** + +## 11. Success criteria & open risks + +**Success:** at 576×512×2048, 2×4 tile, staggered write-once beats the 1.5 TF write-once baseline +with concurrent-fat-waves > the grind's 0.5/SIMD and a healthy trapezoid duty cycle, oracle-clean, no +wedge. (Tile size is the *next* foundational step and is held fixed here.) + +**Risks / open questions (measure, don't assume):** +- **Feed may not keep N fat waves fed.** If the feed path (operand staging) can't supply more + concurrent compute waves, stagger raises occupancy but not throughput. The instrumentation will show + compute waves starving on operands even when fat — if so, the lever is the feed, not occupancy. +- **The gate may settle into a mediocre equilibrium** (collide-and-coast thrash, or under-fill). + Burst length + coast granularity are the dials; §8 tells us which way to turn. +- **Per-SIMD vs. per-WG budget placement.** Waves in a WG spread across SIMDs; the gate is per-SIMD + (hardware). Confirm the economy's compute-wave count doesn't systematically overload one SIMD. +- **`ds_add_f32` hardware behavior** — assembler-accepted; oracle on real hardware is the proof. A is + the fallback. diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_STAGGER_PLAN.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_STAGGER_PLAN.md new file mode 100644 index 000000000000..57c46e8c865b --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/DSWS_STAGGER_PLAN.md @@ -0,0 +1,165 @@ +# DSWS Stagger Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: use superpowers:executing-plans (inline, checkpoint-gated) to implement this plan. Steps use checkbox (`- [ ]`) syntax. **Every GPU run is a hard checkpoint requiring kmbandy's individual greenlight — a subagent MUST NOT dispatch `./occ_dispatch` on its own.** + +**Goal:** Build the staggered traveling-peak occupancy layer on the write-once-C foundation, inside `occ_kernel_dsws_flow.s`, converting the split-K atomic-flush compute path into a WG-local per-rowblk LDS reduction with self-maintaining stagger. + +**Architecture:** Three incremental stages, each ending in an oracle-verified GPU run. Stage 1 changes the *write path* (segment atomic-flush → per-rowblk `ds_add_f32` LDS accumulator → single `global_store`) with no stagger. Stage 2 adds *per-burst* grow/shrink + coast-on-grow-fail + a launch-offset warm-up + instrumentation. Stage 3 tunes the equilibrium (burst length, coast granularity). + +**Tech Stack:** GCN assembly (gfx1201, wave32), raw PM4/KFD dispatch via `occ_dispatch.cpp`, `/opt/rocm/llvm/bin` toolchain, oracle-gated correctness. + +**Spec:** `DSWS_STAGGER_DESIGN.md` (same directory). Read it before starting. + +## Global Constraints + +- Target `amdgcn-amd-amdhsa` `-mcpu=gfx1201`, wave32. Build via `build_flow.sh` (defsym passthrough). +- **0-spill is mandatory** — any `scratch`/spill reference is a build failure. +- **LDS ≤ 65536 B/WG.** The kernel already asserts `LDS_TOTAL_FLOW > 65536 → .error`; keep it. +- **Oracle gate every run:** `bad=0`, and for write-once the tight tier (`rel 5e-3 / abs 1e-2`). A run with `bad>0` fails the task. +- **Safety:** `DEADMAN=1` on every run; SAFEPROBE + bounds gate on; never `--gl2c`; logs to real disk `/home/kmbandy/dsws_gpu_logs/`, never tmpfs. Single-shot before sustained. +- **GPU runs are individually greenlit by kmbandy.** Offline assemble/spill/LDS checks are free; dispatches are not. +- **Commit only when kmbandy explicitly asks.** Do not `git commit` as a plan step unless told. +- **Hold tile size fixed at FM=2 FN=4** — tile size is the *next* foundational step, out of scope here. +- Baseline to beat: **write-once grind = 1.5 TF @ pool 64, 576×512×2048** (occupancy 0.5 waves/SIMD). Split-K baseline ~0.7 TF. + +--- + +## File Structure + +- `occ_kernel_dsws_flow.s` — the kernel. All mechanism changes land here. +- `occ_dispatch.cpp` — host. Only the flow-path LDS-byte computation (~line 1803 / ~6004) and, if added, instrumentation readout in `run_dsws2`. +- `build_flow.sh` — add any new defsyms (`ACC_N`, `BURST`, `COAST_GRAN`, `STAGGER_WARMUP`, `STAGINSTR`). +- `DSWS_STAGGER_DESIGN.md` — reference; update its "build status" as stages land. + +--- + +## Stage 1 — Write-once path into the economy (no stagger yet) + +### Task 1: LDS accumulator pool + +**Files:** +- Modify: `occ_kernel_dsws_flow.s` (LDS layout block, ~lines 318–338) +- Modify: `occ_dispatch.cpp` (flow LDS-byte calc, ~1803 and ~6004) +- Modify: `build_flow.sh` (add `ACC_N` defsym) + +**Interfaces:** +- Produces: `ACC_BASE` (LDS byte offset), `ACC_STRIDE = 8192` (FM*FN*1024), `ACC_N` (defsym, default 2), macro `acc_of slot → acc_slot` mapping a live rowblk to one of `ACC_N` accumulator banks, and `ACC_TOTAL = ACC_N*8192`. + +- [ ] **Step 1:** Add defsym `ACC_N` (default 2) in `build_flow.sh` and a matching `.ifndef ACC_N / .set ACC_N, 2` in the kernel near the other pool defsyms. +- [ ] **Step 2:** In the LDS layout block, place the accumulator pool after the operand pool: `.set ACC_BASE, (OP_BASE + POOL_N*OPSTRIDE)` and `.set ACC_STRIDE, (FM*FN*1024)` (=8192) and `.set LDS_TOTAL_FLOW, (ACC_BASE + ACC_N*ACC_STRIDE)`. Keep the existing `.if LDS_TOTAL_FLOW > 65536 → .error`. +- [ ] **Step 3:** Set `POOL_N=2, ACC_N=2` as the default co-budget (`256 + 2*16384 + 2*8192 = 49408 ≤ 65536`). Leave `POOL_N=3, ACC_N=1` (`57600`) as a documented alternate. +- [ ] **Step 4:** In `occ_dispatch.cpp`, change the flow-path `ldsBytesRaw` to `256 + poolSlots*operandBytes + accN*8192` so the host requests the matching group-segment size. Read `accN` from env `DSWS2_ACC_N` (default 2). Both call sites (~1803, ~6004). +- [ ] **Step 5 (offline gate):** `POOL_N=2 ACC_N=2 ./build_flow.sh 4 2 2` → assemble clean, **0 spill** (`llvm-objdump -d … | grep -c scratch` = 0), no LDS `.error`. Record the `.text` byte size. + +_No GPU run this task — layout only._ + +### Task 2: Per-rowblk LDS reduction (segment atomic-flush → `ds_add_f32` + single store) + +**Files:** +- Modify: `occ_kernel_dsws_flow.s` — coordinator ASSIGN (work-unit granularity), slot fields, `DECODE_STI`, `.Lflow_compute` body (~1441–1537). + +**Interfaces:** +- Consumes: `ACC_BASE/ACC_STRIDE/ACC_N/acc_of` (Task 1). +- Produces: a compute path where all `n_kseg` K-segments of a rowblk `(mblk,tcol,r)` reduce into one LDS accumulator bank and the completer writes C once. + +**Design (this is a work-model change, implement to these invariants — not a copy-paste):** + +The current work unit is a super-tile `(mblk,tcol,ksi)` whose G rowblks each `global_atomic_add_f32`-flush. Change the unit so a **rowblk's `n_kseg` segments are WG-local-reduced**: + +- **Pin:** the coordinator's ASSIGN hands a slot a rowblk-group for a fixed `(mblk,tcol)` and the slot walks `ksi = 0..n_kseg-1` as its segment work-list (the existing `SL_STI` STAMP carries `(mblk,tcol)`; `ksi` becomes an internal per-slot counter, not part of the claimed unit). +- **Accumulator bind:** on a slot beginning a rowblk `r`, bind an accumulator bank `acc_of(slot)` and **zero** its `ACC_STRIDE` bytes (or mark first-segment-writes-instead-of-adds). +- **Per segment (the burst):** compute wave does `KSEG_STEPS` WMMA into VGPR acc (unchanged 1454–1495), then instead of the `global_atomic_add_f32` block (1504–1518), `ds_add_f32` each of the `FM*FN*8` accumulator elements into `ACC_BASE + acc_slot*ACC_STRIDE + frag*1024 + e*4`. Then `s_wait_dscnt 0x0` before advancing the segment done-counter. +- **Completer:** when a rowblk's segment done-counter reaches `n_kseg`, exactly one wave `global_store`s the `FM*FN` fragments from the LDS accumulator bank to `C` at the address the current code already computes (1496–1503), then the bank is recycled. **Ordering invariant (Codex):** the completer must observe completion only after all prior segments `s_wait_dscnt 0` — do not let the done-increment race the DS adds. +- **Invariant carried from spec §4:** rowblk/segment claims still happen in the compute body (after any grow), so this task does not touch grow ordering. + +- [ ] **Step 1:** Introduce a per-slot `SL_KSI` counter and a per-slot/per-rowblk `SL_SEGDONE` counter in the slot control block (extend `SLOTC_STRIDE` if needed; keep `SLOTC_BASE + POOL_N*SLOTC_STRIDE ≤ OP_BASE`, the existing assert). +- [ ] **Step 2:** Add the accumulator-zero on rowblk start (a `ds_store` loop over `ACC_STRIDE`, or a first-segment write path). +- [ ] **Step 3:** Replace the `.if NOCFLUSH==0 … global_atomic_add_f32 …` flush (1504–1518) with the `ds_add_f32` reduction into `acc_of(slot)`, followed by `s_wait_dscnt 0x0`. +- [ ] **Step 4:** Add the completer branch: on `SL_SEGDONE == n_kseg`, single `global_store` of the accumulator bank to C, then reset the bank/counters for reuse and advance the DRAIN frontier. +- [ ] **Step 5:** Update the coordinator ASSIGN + `DECODE_STI` so `ksi` is derived per-slot internally rather than from the claimed unit. +- [ ] **Step 6 (offline gate):** `POOL_N=2 ACC_N=2 ./build_flow.sh 4 2 2` → clean, 0-spill, no LDS error. + +### Task 3: Oracle-verify write-once inside the economy (CHECKPOINT — greenlit) + +**Files:** none (run only). + +- [ ] **Step 1:** Confirm host build current (`ls -la occ_dispatch`), flow bin built from Task 2. +- [ ] **Step 2 (GREENLIT GPU RUN):** `DSWS2_FLOW=1 ML8_POOL=64 ./occ_dispatch --dsws2 4c2a2b 2>&1 | tee /home/kmbandy/dsws_gpu_logs/dsws2_flow_wo_s1.log` +- [ ] **Step 3 (gate):** oracle `bad=0`, `max_rel` within tight tier, `occ[0]=0` (clean drain), no wedge. TF should land near the grind's write-once (~1.5 TF) — it will not yet stagger, so do not expect more. **A regression below split-K's ~0.7 or any `bad>0` fails the task.** +- [ ] **Step 4:** Record TF + oracle in the design doc's build-status. If green, Stage 1 done: write-once now lives in the multi-wave economy. + +--- + +## Stage 2 — Per-burst grow/shrink + coast-on-fail + instrumentation + +### Task 4: Instrumentation (land it BEFORE the stagger so we can see it work) + +**Files:** +- Modify: `occ_kernel_dsws_flow.s` (new `STAGINSTR`-gated atomics + occ offsets), `occ_dispatch.cpp` (`run_dsws2` readout), `build_flow.sh` (`STAGINSTR` defsym, default 0). + +**Interfaces:** +- Produces occ counters: `FATNOW` (live concurrent-fat), `FATPEAK` (`atomic_max`), `FATACC` (time-integral or sample sum), `GROWFAIL` (SCC0 count), `COASTCYC` (cycles coasted before a successful grow), `DUTYFAT`/`DUTYTOT` (fat vs total cycles). All `.if STAGINSTR` gated → zero bytes when off. + +- [ ] **Step 1:** Reserve occ slots for the six counters (past the existing `PH_*`/`ALLLIVE` block); document offsets. +- [ ] **Step 2:** On a successful grow: `global_atomic_add_u32 FATNOW,+1`; `global_atomic_max_u32 FATPEAK,FATNOW`. On shrink: `FATNOW,-1`. All `.if STAGINSTR`. +- [ ] **Step 3:** On grow SCC0: `GROWFAIL,+1`; accumulate coasted RTC into `COASTCYC` (reuse the deadman RTC read pattern). +- [ ] **Step 4:** Duty: stamp RTC at grow and shrink; add the fat span to `DUTYFAT`, total lap span to `DUTYTOT`. +- [ ] **Step 5:** `occ_dispatch.cpp` `run_dsws2`: print `FATPEAK`, `FATACC/laps`, `GROWFAIL`, mean `COASTCYC`, `DUTYFAT/DUTYTOT` when `STAGINSTR`. +- [ ] **Step 6 (offline gate):** build with `STAGINSTR=0` → byte-identical to pre-Task-4 bin (proves gating is clean); build `STAGINSTR=1` → 0-spill. + +### Task 5: Per-burst grow/shrink + coast-on-grow-fail + +**Files:** Modify `occ_kernel_dsws_flow.s` — the grow macro (`conv_apply` / `.Lca_alloc` ~1039–1044, `.Lflow_grow` ~1417) and the compute burst loop. + +**Interfaces:** +- Consumes: Task 4 counters. +- Produces: a compute wave that grows per burst, and on grow SCC0 **coasts** (branch to `.Lflow_coast`) instead of spinning. + +**Design (spec §4 — the load-bearing invariant is "commit nothing until grow succeeds"):** + +- [ ] **Step 1:** Move grow/shrink from role-epoch to **per-burst**: the compute wave, at the top of each segment burst, is lean; it `try-grow`s, does the segment, `ds_add`s, then `shrink`s back to lean. `BURST` defsym (default 1 = one K-segment per grow) sets how many segments per grow before shrink. +- [ ] **Step 2:** Replace the grow spin `s_cbranch_scc0 .Lca_alloc` with **coast**: `s_cbranch_scc0 .Lflow_coast` (SCC0 = budget full → coast lean). Ensure the rowblk/segment **claim happens only after** SCC1 (grow success) — audit that no `lds_fetch_add`/counter write precedes the grow on the burst path. +- [ ] **Step 3:** Confirm coast returns to the loop head (`.Lflow_loop`) where `deadman_check` runs — this is what closes the grow-spin brick gap. Verify no path reaches `s_alloc_vgpr` retry without a loop-head pass. +- [ ] **Step 4 (offline gate):** `BURST=1 STAGINSTR=1 POOL_N=2 ACC_N=2 ./build_flow.sh 4 2 2` → clean, 0-spill. + +### Task 6: Launch-offset warm-up + +**Files:** Modify `occ_kernel_dsws_flow.s` — entry, after `.Lflow_alloc`. + +- [ ] **Step 1:** Add `STAGGER_WARMUP` defsym (default 1). When set, compute-class wave `wid` delays its first grow by `wid * WARMUP_STEP` cycles via `s_sleep`/RTC-spin (spread initial phases). Warm-up only — steady state is maintained by the gate. +- [ ] **Step 2 (offline gate):** build clean, 0-spill. + +### Task 7: Stage-2 verify (CHECKPOINT — greenlit, single-shot then sustained) + +- [ ] **Step 1 (GREENLIT single-shot):** `DSWS2_FLOW=1 STAGINSTR=1 ML8_POOL=64 576×512×2048 ./occ_dispatch --dsws2 4c2a2b | tee /home/kmbandy/dsws_gpu_logs/dsws2_flow_stag_s2.log` +- [ ] **Step 2 (gate — correctness + safety):** `bad=0`, `occ[0]=0`, no wedge. Instrumentation must show `FATPEAK > 1` and a real duty cycle (`DUTYFAT/DUTYTOT` between ~0.3 and ~0.9) — i.e., the stagger is actually forming, not lockstep (peak-then-zero) or square-wave (duty ≈ 1). +- [ ] **Step 3 (GREENLIT sustained):** re-run with `DSWS2_TARGET_SECS=2` → deadman-clean, no wedge across the sweep (proves the brick gap is closed). Record `GROWFAIL` rate + mean `COASTCYC` (the handoff gap). +- [ ] **Step 4:** Record all counters + TF in the design doc build-status. Gate to Stage 3: no wedge, `FATPEAK>1`, healthy duty. + +--- + +## Stage 3 — Tune the equilibrium + +### Task 8: Burst-length × coast-granularity sweep (CHECKPOINT — greenlit) + +**Files:** Modify `occ_kernel_dsws_flow.s` (add `COAST_GRAN` defsym = feed units per grow-retry), `build_flow.sh`. + +- [ ] **Step 1:** Add `COAST_GRAN` defsym (default 1 = one feed unit per retry). Wire the coast fallback to do `COAST_GRAN` feed units then retry grow. +- [ ] **Step 2 (offline):** build the sweep matrix: `BURST ∈ {1,2,4}` × `COAST_GRAN ∈ {1,2}` × `POOL_N/ACC_N ∈ {2/2, 3/1}`, all `STAGINSTR=1`, 0-spill each. +- [ ] **Step 3 (GREENLIT sweep):** run each at 576×512×2048, ML8_POOL swept {32,64,128}, logs to `/home/kmbandy/dsws_gpu_logs/`. Read TF + `FATPEAK`/`FATACC` (concurrent-fat vs budget) + `GROWFAIL` + `COASTCYC` + duty. +- [ ] **Step 4 (gate — SUCCESS CRITERION):** find the config where TF **beats 1.5 TF** with concurrent-fat meaningfully **> 0.5 waves/SIMD** and low thrash (`GROWFAIL` not dominating), oracle-clean. Record the winning `(BURST, COAST_GRAN, POOL_N, ACC_N)`. +- [ ] **Step 5:** If nothing beats 1.5 TF: consult the instrumentation to decide which risk fired — feed-capped (compute waves fat but starved on operands → the lever is the feed, not occupancy), or bad equilibrium (thrash / under-fill → burst/coast dials). Record the diagnosis; do not thrash on fixes — bring the finding back for a decision. + +--- + +## Self-Review (spec coverage) + +- **Spec §3 (substrate: split-K bursts → LDS `ds_add_f32` → write-once):** Tasks 1–3. ✓ +- **Spec §4 (coast-on-grow-fail, commit-after-grow, closes brick gap):** Task 5. ✓ +- **Spec §5 (self-maintaining stagger + launch warm-up + burst knob):** Tasks 5, 6, 8. ✓ +- **Spec §6 (accounting consistency under coast):** enforced by Task 5 Step 2 (claim-after-grow) + Task 2 completer ordering. ✓ +- **Spec §7 (handoff gap, coast granularity dial, measured):** Task 4 (`COASTCYC`), Task 8 (`COAST_GRAN`). ✓ +- **Spec §8 (instrumentation: fat count/peak, grow-fail, coast, duty, confirm B):** Task 4. (Per-SIMD budget B: `FATPEAK` at saturation × peak footprint gives an empirical read — note it in Task 7.) ✓ +- **Spec §9 (oracle, deadman, greenlit runs):** every CHECKPOINT task. ✓ +- **Spec §10 (3-stage incremental):** stage structure. ✓ +- **Spec §11 (risks: feed cap, equilibrium quality, per-SIMD placement, ds_add_f32 hw):** Task 3 (ds_add_f32 hw proof via oracle), Task 8 Step 5 (feed cap / equilibrium diagnosis). ✓ diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/FLOW_ECONOMY_DESIGN.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/FLOW_ECONOMY_DESIGN.md new file mode 100644 index 000000000000..463cdbe9d091 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/FLOW_ECONOMY_DESIGN.md @@ -0,0 +1,230 @@ +# DSWS v2 — Fix #1 (the real one): the non-blocking FLOW economy + +Design spec. This **supersedes the ring's consumption model** (`RING_SLOTS_DESIGN.md` / Fix #1a). The +ring cut the coordination wait in half but kept the *paradigm*: every super-tile boundary, every wave +still **polls a published flag** (`SL_GEN[DRAIN_IDX]`) and spins until it changes. That poll *is* +FOLLOW_WAIT. Measured: balancing rowblks made it worse (6c2a2b → 29%), adding feeds made STAGE worse +(4c3a3b → 20%) — because the poll/slot-sync is the disease, not the tuning. + +**The flood model (kmbandy):** water flowing past a floodgate doesn't stop and read the gate — it just +flows; the gate biases the flow. A wave should never ask "is the next super-tile ready?" It reads its +assigned role, grabs the next unit of work for that role, and goes. When the accounting lags, the wave +**keeps doing what it did last time** until the accounting catches up — "a bit of a perf hit, barely +noticeable." **There is no blocking read anywhere in a wave's hot loop.** That is the whole design. + +This is also fixes **#2** (wave count) and **#3** (feed:compute ratio) — they're the same knob: the +role mix, tuned in the background by the coordinator. Split-K's C-write amplification (FLUSH, the +measured 60–71%) is the memory-bandwidth floor we attack *after* this, as planned. + +--- + +## The one idea that makes it non-blocking (kmbandy's twist) + +Each wave has a mailbox slot `ROLE[wid]` in LDS that **always holds a valid role** — the last one the +coordinator wrote, or the launch role. So the wave never needs a "is my assignment fresh?" check: + +- Coordinator kept up → slot holds a new role → the wave switches (dyn-VGPR resize) and flows. +- Coordinator lagged → slot **still holds the old role** → the wave keeps being that and flows. + +The lag-fallback is **free** — a not-yet-updated mailbox literally contains the previous value. There is +no `s_cbranch`-back-to-sleep to write. FOLLOW_WAIT cannot exist because there is no publish to poll. + +--- + +## The wave hot loop (every wave, every cycle — the whole thing) + +``` +loop: + role = ROLE[wid] // read my assignment; stale == last role == coast (free) + if role != cur_role: + dyn-VGPR resize to role footprint // compute=NFV(112) / feed=VLEAN(32); at a lean boundary + cur_role = role + got = try_grab(role) // ONE atomic fetch-add on this role's work counter; no spin + if got: + do_work(role, got) // compute a rowblk / stage an A-rowblk / stage a B-frag + else: + // my channel is momentarily dry -> FUNGIBILITY: do the complementary work for one cycle, + // which almost always has some (compute dry => there's staging to do; feeds dry/pool-full + // => there's compute to do). Never spin. The coordinator's job is to make this rare. + alt = complement(role) + got2 = try_grab(alt) + if got2: do_work(alt, got2) else: s_sleep SLEEPN // true floor: both dry (rare transient) +``` + +No follow-loop. No `GEN` poll. No slot-boundary barrier between waves. The only `s_sleep` is the genuine +"there is literally no work anywhere this instant" floor, which the coordinator drives toward zero. + +--- + +## Three layers + +### 1. Work-flow layer — a 3-frontier pipeline, each advanced by a lock-free closer +A **deep operand pool** of `N` slots (reuse the ring's per-slot layout: BRES 4KB + ARES 12KB = 16KB +each). Super-tiles are consumed in claim order by local index `k`; slot = `k mod N`. **Three monotone +frontiers** (LDS u32), invariant `DRAIN ≤ STAGE ≤ ASSIGN ≤ DRAIN + N`: + +- **`ASSIGN_HEAD`** — next local index to assign a global super-tile to. **Single writer = coordinator + (wid0).** When `ASSIGN − DRAIN < N` (a slot is free) and work remains: claim `occ[20]`→`gsti`, **reset + that slot's counters, set `STAMP[slot]=gsti`, then `ASSIGN_HEAD++` (release LAST).** Because the free + gate means slot `ASSIGN mod N`'s prior occupant (`ASSIGN−N`) is already drained, the reset is a clean + single-writer-after-drain — no reset-race, no CAS. This is the ring's "reset only at FREE, only by one + writer, after the occupant drained" discipline, kept. +- **`STAGE_HEAD`** — oldest assigned-but-not-fully-staged index. **Feeds** stage slot `STAGE mod N` + cooperatively (`SL_BFNEXT`/`SL_ARNEXT` fetch-add → `BSTAGE_R`/`ASTAGE_R` → bump `SL_BFDONE`/`SL_ARDONE`, + verbatim from the ring). Any feed that observes slot `STAGE mod N` fully staged (`BFDONE==FN && + ARDONE==G`) does `ds_cmpstore(STAGE_HEAD, sh, sh+1)` — a lock-free monotone bump; losers just retry. +- **`DRAIN_HEAD`** — oldest not-fully-drained index. **Compute** pulls rowblks from slot `DRAIN mod N` + (`SL_RBNEXT` fetch-add, same barrier-free protocol as the ring). Any compute wave that observes slot + `DRAIN mod N` fully drained (`SL_RBDONE==G`) does `ds_cmpstore(DRAIN_HEAD, dh, dh+1)` — frees the slot + for the coordinator to reassign. + +`try_grab` per role is one atomic returning **work-or-empty**, never a spin: +- compute empty ⇔ `DRAIN >= STAGE` (nothing fully staged) → coast to feed. +- feed empty ⇔ `STAGE >= ASSIGN` (nothing assigned; coordinator behind) → coast; or pool-full for the + assigner. Over-claim (`fetch_add ≥ max`) → attempt the CAS-advance, then retry/empty. + +The CAS advances (`STAGE_HEAD`, `DRAIN_HEAD`) are the only lock-free arbitration, and they're monotone +counter bumps (idempotent, losers retry) — **not** the old `try_gate` role-election that caused the +reacting-controller bugs. `ASSIGN_HEAD` and every `STAMP`/reset is single-writer (wid0). + +### 2. Role-mailbox layer (the economy) — which counter a wave pulls from +`ROLE[wid]` ∈ {COMPUTE, AFEED, BFEED}. Written **only** by the coordinator (single writer → no races, +no CAS, no single-winner gate — the exact machinery that caused the `try_gate`/`conv` reacting-controller +bugs is *absent by construction*). Read by every wave each cycle. Determines the steady-state mix. + +### 3. Coordinator layer (wid0, a flowing worker with a side-duty) +`wid0` is a **lean feed worker** (it flows and stages like any feed) that **also**, on a slow cadence +(every `COORD_PERIOD` cycles), runs sense→nudge: + +- **Sense** (cheap, LDS-local): pool fill = `stagedFrontier − drainFrontier`. Low fill → compute is + out-running staging → operand-starved → **shift toward feeds**. High/full fill → feeds are ahead, + compute-bound → **shift toward compute**. (This is the prod/cons ring-occupancy sensor that already + exists.) +- **Nudge**: move **one** wave's `ROLE[wid]` toward the starved side (bias-on-next-adopt, not a live + yank — the wave switches at its next lean cycle boundary). One step per period. +- **Anti-thrash**: hysteresis band + a cooldown (min cycles between nudges), held in wid0's own + registers — trivial because there is exactly one decider. No shared thrash state, no distributed + epoch arbitration. + +Because waves **coast on lag** and a single GEMM's optimum is fixed, the coordinator can be as lazy as +we like: it settles the mix in a few nudges and then mostly senses. Its lag never stalls anyone. + +--- + +## LDS layout (64 KB; N = 3 pool) +``` +shared frontier + mailbox (bytes 0..~256): + STAGE_HEAD next super-tile index to stage (feeds claim; = occ[20] mirror or its own counter) + DRAIN_HEAD oldest not-fully-drained slot's super-tile index (compute advances) + RINGINIT barrier-free init publish flag (0xACED last) + ROLE[0..WAVES-1] per-wave mailbox (u32 each), coordinator-written +per-slot control block s in [0,N): (STAMP=super-tile id, GEN unused, SL_RBNEXT/RBDONE/BFNEXT/BFDONE/ + ARNEXT/ARDONE) — same fields as the ring +per-slot operands OP_BASE + s*OPSTRIDE (16 KB each): N=3 -> 48 KB + ~1 KB control < 64 KB (N=4 = 64KB, too tight) +``` +`N=3` gives compute up to 18 staged rowblks of runway — plenty to smooth super-tile boundaries so the +grab always finds ready work. (`N` is a defsym; 3 is the LDS-safe default, occupancy already 1 WG/WGP +above 32 KB either way.) + +--- + +## Fungibility (the coast) — a code-path flip, NOT a resize (this is the key to it being free) +Critical distinction that avoids resize churn: + +- **The coast flip is a CODE-PATH branch, not a `s_alloc_vgpr`.** A wave that gets "empty" from its own + role just branches to the complementary role's code for one cycle. No VGPR resize. +- **compute→feed coast is FREE.** A fat compute wave (112 VGPR) can run the lean feed code path as-is + (feed code only touches ≤v31; the extra VGPRs sit unused). So a compute wave starved for staged work + instantly helps *stage* — no resize, no churn — which refills the pool and lets it compute again next + cycle. **This is the common, high-value case** (compute starving is exactly the FOLLOW/STAGE wait), + and it costs nothing. It is what structurally deletes the wait. +- **feed→compute coast is NOT free** — compute needs the fat allocation (accumulators at v32+), so a + lean feed can't compute without growing. So a feed that finds the pool full **yields briefly** + (`s_sleep`) rather than panic-growing; the coordinator, sensing pool-full, moves feed waves to the + COMPUTE role via mailbox (a *deliberate* grow at adopt). Pool-full is the rare case (too many feeds), + so this floor is small. + +- **`s_alloc_vgpr` resize happens ONLY on a mailbox role change** (coordinator-directed, damped by + hysteresis/cooldown), at a lean cycle boundary. Grow (feed→compute, 32→112) is the brick-class op; + the multi-wave grow-stagger deadlock (ISA §3.3.3.2) is **out of scope** exactly as for the ring — + **stay at M=576**, per-rowblk grow proven safe there. Grow-stagger is its own later increment. + +So: coasts are free code-path branches (no churn); resizes are rare, deliberate, coordinator-driven. + +## Non-blocking correctness argument +- **No spin on a producer anywhere.** `try_grab` is one atomic returning work-or-empty; "empty" flips + to complementary work, never waits. The only `s_sleep` is the both-channels-dry transient. +- **No reset-race** — same barrier-free per-slot counter discipline as the ring: a slot's counters + reset only at FREE→FILLING, only after its occupant's `SL_RBDONE==G`, and compute can't bump + `SL_RBDONE` until `SL_BFDONE==FN && SL_ARDONE==G`. Carried over verbatim. +- **No mailbox race** — single writer (coordinator). Readers see a torn-free u32 (aligned word). +- **Termination** — a SENTINEL super-tile (past `chunkHi`) propagates: feeds staging it mark the slot + terminal; compute draining a terminal slot retires; the coordinator, seeing all work claimed + + drained, writes a RETIRE role to all mailboxes. `occ[0]` live-counter gate unchanged. +- **The true floor** = both channels dry for a wave = pool full *and* all staged rowblks claimed = a + transient other waves resolve within a rowblk. Measured as the residual; the coordinator minimizes it. + +## What carries over from the ring (not wasted) +Per-slot operand layout + `BSTAGE_R`/`ASTAGE_R`, slot-indexed counter macros (`lds_*_r`), the +barrier-free counter protocol, WMMA loop, split-K C-flush, PHASEPROBE/TFPROBE, host `DSWS2_RING`-style +LDS sizing (bump to N slots). **New file** `occ_kernel_dsws_flow.s` (fork of `occ_kernel_dsws_ring.s`); +the ring bin stays as a working reference; safe single-slot bins untouched. + +## Non-goals (deferred, explicit) +- **Split-K C-write amplification** (FLUSH 60–71%) — the bandwidth floor, addressed AFTER flow. +- **Grow-stagger** — stay at M=576; training-M needs it and it's a separate increment. +- **Cross-WG global mix** — the economy is per-WG (LDS rings are per-WG); coordinator is per-WG (wid0). + +## Test plan (scoreboard-driven, brick-safe) +1. Assemble clean, 0 spill, LDS ≤ 64 KB. +2. Greenlit `576×512×2048`, POOL=16, streamed: `occ[0]==0`, **oracle CLEAN**, dmesg delta 0. Restore. +3. PHASEPROBE run → **FOLLOW_WAIT should be ~0** (no poll left). Compare STAGE + the new both-dry floor + vs the ring's 25%. Success = FOLLOW gone, total wait materially below the ring, oracle clean. +4. TFPROBE/sustained-reps **wall-clock** — the number that actually matters; compare to ring + baseline. + +## Build status (2026-07-04) +- **Stage 1 (foundation) DONE:** `occ_kernel_dsws_flow.s` forked from the ring. LDS reworked to the + 3-frontier + mailbox + N=3 pool layout (`ASSIGN/STAGE/DRAIN_HEAD`, `ROLE_BASE` mailbox, + `SLOTC_BASE=148`, `OP_BASE=256`, `LDS_TOTAL_FLOW=49408<65536`; `POOL_N`/`COORD_PERIOD` defsyms). Host + `occ_dispatch.cpp`: `DSWS2_FLOW`→ldsBytes `256+POOL_N*16384` + bin `occ_dsws2__flow_gd.bin`. + `build_flow.sh` added. Descriptor group-seg 65536 (inherited). +- **Stages 2–4 (the unified loop) NOT YET CUT — file does not assemble yet** (old ring role loops still + reference removed symbols `FILL_IDX_OFF`/`DRAIN_IDX_OFF`/`RING_D`/`.Ldispatcher`). This is expected: + the flow control flow is a ground-up rewrite of the whole role section, all-or-nothing for assembly. + The ring bin (`occ_kernel_dsws_ring.s`, GPU-verified) and safe single-slot bins are untouched. + +**Resume plan — replace the ring's role section (branch + dispatcher + .Lbfeed/.Lafeed/.Lcompute) with +one unified flow section:** +1. **Coordinator (wid0):** LDS init (frontiers=0, mailboxes=launch mix, RINGINIT last); load chunkHi; + loop { ASSIGN duty: if `ASSIGN−DRAIN0, and wid0's feed part):** read `ROLE[wid]`; if `RETIRE`→retire; if + `role!=cur` resize (grow/shrink at lean boundary); dispatch: COMPUTE→`try_compute_grab` + (`DRAIN>=STAGE`?empty: fetch_add `SL_RBNEXT[DRAIN%N]`; `=ASSIGN`?empty: fetch_add + `SL_ARNEXT/SL_BFNEXT[STAGE%N]`→`ASTAGE_R`/`BSTAGE_R`, on both-done `ds_cmpstore(STAGE_HEAD)`); + **coast:** compute empty→run BFEED code (fat wave runs lean feed, FREE); feed empty→`s_sleep`. +3. Add a `lds_cmpstore_r` macro (ds_cmpstore_rtn_b32 wrapper) for the monotone frontier CAS advances. +4. Terminal: coordinator sets `FLOWTERM`+writes `ROLE_RETIRE` to all mailboxes once occ[20] past + chunkHi AND all drained; waves read `RETIRE`→`occ[0]` live--→endpgm. +5. Assemble clean (0 spill, LDS≤64KB, PP 0&1) → greenlit `576×512×2048` (occ0==0, oracle CLEAN, dmesg + 0) → PHASEPROBE (FOLLOW→~0) → TFPROBE wall vs ring+baseline. + +## DEADMAN watchdog (2026-07-04) — makes hangs recoverable, enables scale-stress +Every wave stamps its start RTC (`s[70:71]`, `s_sendmsg_rtn GET_REALTIME`); `deadman_check` at every +loop head (`.Lflow_loop`, `.Lflow_wait_init`) force-retires via the normal path if alive > +`DEADMAN_TICKS` (default 50M = 0.5s @ 100MHz, < host `chunkMaxS` 0.75s). A coordination hang (a frontier +that never advances) → all waves hit the deadline → all retire → `occ[0]→0`, queue idle, EOP fence fires +→ host sees a clean completion, **no wedge, no desktop brick**; the result is incomplete and the oracle +flags it. `DEADMAN=0` → zero bytes. **Validated** (1ms-deadline bin, greenlit): `occ[0]==0`, +`occ[20]=464`≠784 (fired early), oracle `bad=528` (incomplete, flagged), no hang/brick. +**Why it was needed:** the host's hang path did *not* destroy a hung queue ("brick-avoidance; process-exit +reclaims"), but process-exit reclaim of a live-spinning-wave queue faults the shared gfx1201 ring → +brick. The deadman drains the waves *before* that, in-kernel. +**Caveat:** covers the coordination-hang class (loop-head spins). Does NOT cover the `s_alloc_vgpr` +grow-spin (grow-stagger, ISA §3.3.3.2) — that never reaches a loop head; separate gate, M=576 only. + +## Rollout / safety +New file; ring + `c62568f6` bins untouched. Every dispatch kmbandy-greenlit, one at a time, streamed, +safe-bin restored after; halt on any INCOMPLETE; never `--gl2c`; stay at M=576. diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/HANDOFF_DSWS_ROLLING_DYNVGPR.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/HANDOFF_DSWS_ROLLING_DYNVGPR.md new file mode 100644 index 000000000000..dea3d0434ede --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/HANDOFF_DSWS_ROLLING_DYNVGPR.md @@ -0,0 +1,284 @@ +# HANDOFF — DSWS rolling dyn-VGPR (the sum-envelope grow economy) + +**Written 2026-07-02** for a fresh session that will write the detailed spec + plan + implementation. +This doc gives you the **architecture** (agreed with kmbandy) and the full **state of the world** so you +can start cold. It deliberately stops short of the detailed transcription spec — that is YOUR first task. + +Everything lives in `ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/` unless noted. +The kernel under work is **`occ_kernel_dsws.s`** (the "v2 substrate", `--dsws2` path). Do **not** touch +`occ_kernel_coop.s`, `occ_dispatch.cpp`, `fp8_oracle.*` — they are shared/not-ours siblings (leave dirty, +never stage). + +--- + +## 0. TL;DR — what to do next + +1. Read this doc, then the referenced repo docs (§7) and the SDD ledger (`.superpowers/sdd/progress.md`). +2. Write the **detailed design spec** for the rolling dyn-VGPR grow economy (§4 is the architecture; you + turn it into exact LDS offsets, register assignments, macro definitions, asm placement, CPU-model + interfaces). Use the brainstorming→writing-plans flow. Save to `docs/superpowers/specs/`. +3. Then plan (`docs/superpowers/plans/`) and implement, offline gates first, one supervised GPU oracle run. + +**The one-line problem:** the DSWS compute waves currently grow to peak (`NFV=112`) via a bare +`s_alloc_vgpr` SCC-retry with **no sum-envelope and no stagger**, so all `NCOMP` compute waves grow in +lockstep and collide on the shared dyn-VGPR pool → intermittent forward-progress hang (ISA §3.3.3.2). +**The fix:** route every compute grow through the shared `vgpr_reserved` sum-envelope + stagger the +grows, so peaks never all coincide and freed budget flows to whatever role the bottleneck needs. + +--- + +## 1. Where we are (git + disk state, 2026-07-02) + +- **HEAD = `a7212a605`** on branch `feat/dsws-phaseb-conversion` (base `66eec7f54`). +- Relevant commits (newest first): + - `a7212a605` docs(dsws): correct entry != re-dispatch in pool-economy spec + plan + - `b01c722df` **fix(dsws): route first-time wave entry through _alloc/_init** — the Pool-T7 *brick* fix + (see §3). Message overstates it as "the Pool-T7 brick" fix: it fixed the desktop-**brick** failure mode + but NOT the residual hang. Leave the commit; the record is corrected in the ledger + KG. + - `3dbb8c985` feat(dsws): pool-economy offline machinery (Tasks 1-6) + - `5421751` feat(dsws): CPU model for Phase-B snapshot/quiesce +- **Uncommitted OURS (working tree):** + - `occ_kernel_dsws.s` — the **wedge-frame DIAG instrumentation** (see §6). Byte-identity verified: at + `DSWS2_CONV=0` and at `DSWS2_CONV=1/DIAG=0` the `.text` sha is IDENTICAL to HEAD (`e5ec5e50` / + `e296b846`). The markers only materialize at `CONV=1 && DIAG=1`. **Safe to commit.** + - `test_dsws_quiesce_race.cpp` (untracked) — the offline `std::thread` QUIESCE/epoch race harness (§5). + - `lean_bounds_sim.cpp` (untracked) — pre-existing, not ours, ignore. +- **Installed bins:** all three `occ_dsws2_{4c2a2b,6c1a1b,2c3a3b}_gd.bin` restored to the **safe CONV=0 + 4840B** build (`e5ec5e50` / `bb8cb230` / `066b73a4`). No CONV=1 bin is installed (footgun rule). +- The 4 not-ours siblings are dirty; never stage them. + +--- + +## 2. What DSWS is (one paragraph, so you have the frame) + +**Dynamic-Split Wave-Specialization** — one persistent-wave fp8 GEMM kernel on gfx1201 (RDNA4/R9700, +wave32) that re-balances its mix of **{fat compute / lean A-feed / lean B-feed}** wave roles to the +runtime bottleneck, using raw-PM4-armed **dyn-VGPR** (`s_alloc_vgpr`, which HIP cannot express — the moat) +to make the per-SIMD VGPR budget **fungible across roles**. One WG = `WAVES` waves: wid0 = pinned +"claimer" (super-tile broadcaster + B-feed-class), the rest are followers in three roles. Coordination is +**barrier-free** (no `s_barrier`) via LDS atomics + epoch polling. Split-K decomposes K into `SEGK`-sized +segments accumulated into C via `global_atomic_add_f32` (ksi-independent C term). Role mix is written +`{NCOMP,NAFEED,NBFEED}` (e.g. `4c2a2b` = 4 compute, 2 A-feed, 2 B-feed). Phase-A = static substrate +(`DSWS2_CONV=0`, GPU-green). Phase-B = the adaptive conversion economy (`DSWS2_CONV=1`). + +Read `MAD305_DSWS_MASTER.md` for the full vision; `SPEC_DSWS_SUBSTRATE_V2.md` for the substrate; +`SPEC_DSWS_PHASEB_CONVERSION.md` for the conversion economy. + +--- + +## 3. How we got here — the debugging arc (so you don't re-walk it) + +1. **Pool-T7 first GPU gate (CONV=1, conversions dormant) BRICKED** — MODE1 reset, rebooted the desktop + (gfx1201 shares the graphics ring with Hyprland). CONV=0 was green across all mixes. +2. **Entry-fix (`b01c722df`)** — root-caused offline: the CONV=1 **seed block routed first-time wave entry + through the `.Ldispatch` re-dispatch trampoline**, which lands on `_follow` and skips `_alloc`/`_init` + (the `s_alloc_vgpr 32` handshake, the `INITFLAG==0xACED` rendezvous, and `s35=0`). Re-pointed the seed + arms at the full role entry (`.Lcompute/.Lafeed/.Lbfeed`). This **fixed the desktop-BRICK failure mode** + — confirmed on silicon (re-attempt #2: dmesg silent, chunk watchdog self-aborted cleanly at ~24.8s). +3. **BUT a residual non-bricking hang remained** — chunk 1 (8 super-tiles) completes, the *next* dispatch + stalls at its first super-tile (`occ[20]` frozen, `occ[0]=1` claimer live). CONV=0 control ran clean + (`ok=1536 bad=0`), so CONV=1-dormant has a real liveness regression. +4. **Offline analysis RULED OUT the protocol logic:** a faithful `std::thread` model of the claimer+ + followers QUIESCE/epoch handshake (`test_dsws_quiesce_race.cpp`) → **0/400 stalls** (seq_cst *and* + relaxed). Codex static review (task-mr3u2ig0) independently found no source-level race. Snapshot LDS + offsets don't collide; snapshot values are constant in the dormant case (so `s50 == CONV=0`); + `lds_put/get/fetch_add` all `s_wait_dscnt` (ordering HW-enforced). Logic is sound. +5. **SKIPQ bisection BRICKED (do NOT repeat):** an uncommitted `DSWS2_DBG_SKIPQ` knob forced `s51=1` to + advance on the snapshot sentinels alone. Dispatched → hard hang (~8 min) → brick. **Disabling the + quiesce gate removes the chunk-watchdog's clean self-abort** and re-introduces the straggler bug the + gate prevents. **Never disable the quiesce gate on the compositor card.** +6. **HEISENBUG → root cause:** adding the wedge-frame DIAG instrumentation (extra occ stores in the + claimer poll loop + per-role markers) shifted timing and the CONV=1/DIAG=1 bin ran **CLEAN** + (`ok=1536 bad=0`, all 4 chunks, dmesg silent) where the stock bin hung. A timing-sensitive race that + instrumentation perturbs away ⇒ **not a logic bug**. kmbandy confirmed: it's the **known multi-grower + dyn-VGPR collision** — all `NCOMP` compute waves grow to 112 in lockstep and collide. + +**Root cause (CONFIRMED, KG `21cc0d63`):** `occ_kernel_dsws.s` `.Lcompute_grow` is a **bare +`s_alloc_vgpr NFV` SCC-retry with no sum-envelope reservation and no stagger**. The `vgpr_reserved`/`VRESV` +envelope exists but only gates **role-conversion** grows, not the per-rowblk compute grow. All compute +waves issue `s_alloc_vgpr` simultaneously → ISA §3.3.3.2 (HW deadlock-avoidance guarantees only *one* +wave/SIMD can reach max) → intermittent hang, worsened by the shared B-ring (`RINGD=2`) interaction. + +--- + +## 4. THE ARCHITECTURE (the fix) — build the detailed spec from this + +The design kmbandy and I converged on. **One pool, one law, split-K churn, stagger for saturation + +safety + fungibility.** This *unifies* the per-compute-wave grow with the role-conversion economy: they +are the same pool discipline, not two systems. + +### 4.1 One pool, one invariant +A single per-SIMD VGPR budget `BUDGET`. The only governing law: +``` +Σ instantaneous allocations over all resident waves ≤ BUDGET (at every instant) +``` +Every grow — a compute wave taking a kseg, OR a feed wave converting to compute — reserves against the +**same** `vgpr_reserved` counter. VGPR is fungible without regard to who grows next. That fungibility IS +the adaptive substrate. + +### 4.2 Real split-K (n_kseg > 1) makes the pool churn +Old full-K behavior: a compute wave grabs its fp32 accumulators and camps at peak for ~95% of the K-loop +(square wave — this is the `MAD305_DSWS_MASTER.md:78-81` "staggering buys nothing" case). With **real +split-K** each compute wave: **grow → compute ONE kseg → flush that partial to C via +`global_atomic_add_f32` → shrink to lean IMMEDIATELY → loop**. No accumulators persist across a wave's +ksegs (the accumulate-into-C path split-K already uses across waves handles the sum), so peak is a *brief* +window, duty drops, and the pool has genuine free headroom cycling through it. Split-K is **already +plumbed** in the substrate: `n_kseg` is derived in-kernel from `KT` and `KSEG_STEPS=SEGK/16`; `DECODE_STI` +splits `sti → (t, ksi)` and handles `n_kseg=1` for free (`shift=0, mask=0`). We've only ever run +`n_kseg=1` (degenerate high-duty). The real thing runs `n_kseg > 1`. + +### 4.3 Every grow through the shared sum-envelope (the collision fix) +Replace the bare `.Lcompute_grow` SCC-retry with the **enveloped reserve→grow→…→shrink→release**, the +same primitive `conv_apply` already uses for role conversion: +``` +reserve : old = atomic_add(vgpr_reserved, NFV-VLEAN); if old+(NFV-VLEAN) > BUDGET { atomic_sub; spin-at-lean; retry } +grow : s_alloc_vgpr NFV (SCC-retry stays as the hardware backstop) +burst : one kseg of WMMA +flush : global_atomic_add_f32 partials into C +shrink : s_alloc_vgpr VLEAN +release : atomic_sub(vgpr_reserved, NFV-VLEAN) +``` +Because the budget can never be exceeded, the "all NCOMP grow at once" collision is **unreachable by +construction** — only as many waves as fit can be at peak; the rest spin at lean until budget frees. +This is exactly the ISA §3.3.3.2 "software forward-progress scheme." Reserve/release run at lean-32 +(v≤15, OOR-safe) — before the grow and after the shrink. + +### 4.4 The stagger — three jobs at once +Phase-offset the compute waves' grow windows (lock-free LDS phase tokens, reuse the prod/cons counter +style — NO `s_barrier`) so: +1. **Compute stays saturated** — there is always *some* compute wave at peak doing WMMA. +2. **The collision can't recur** — grows don't all issue at the same instant (the envelope is the hard + floor; the stagger is the scheduling that keeps peaks spread). +3. **Feed fungibility** — the immediate per-kseg release frees budget so **feed waves can spawn/grow the + moment the ring counters say feed is the bottleneck.** The freed VGPR flows to wherever the current + bottleneck is, per-moment — the adaptive core of DSWS. + +**Master knob = burst length** (ksegs held at peak before shrink) and **stagger period / concurrent-peak +count**. These are the R3 sweep axes (`SPEC_WAVESPEC.md` R3). Short burst → more churn/occupancy but more +grow/shrink overhead; long burst → amortized overhead but higher duty. Optimum = shortest burst that keeps +the WMMA pipe full. + +### 4.5 Instrumentation baked in from the start (kmbandy mandate) +"Build the real thing with the real instrumentation so if/when it breaks, the data pinpoints it — not a +blind brick." On top of the wedge-frame markers already in the tree (§6), add grow-envelope telemetry to +host-streamed occ slots: `vgpr_reserved` high-water, per-wave reserve-spin count (permit-wait), +grow/shrink/release event counts, per-role epoch. A wedge becomes a readout (permit-starvation vs +elsewhere), not a mystery. All DIAG-gated so `DIAG=0` stays byte-identical. + +### 4.6 Validation ladder +CPU-model the envelope invariant (Σ ≤ BUDGET, no lost reservation, forward-progress under thread race — +extend `dsws_ctrl_model.cpp` + `test_dsws_ctrl_model.cpp`) → assemble matrix + RGA 0-spill → CONV=0 / +DIAG=0 byte-identity → **one** safeguarded GPU oracle run at `n_kseg>1` (`ok=1536 bad=0`) → then sweep +concurrent-peak × burst × stagger-period for occupancy/throughput. + +--- + +## 5. How this lines up with the existing repo design (reconciliation) + +kmbandy asked explicitly whether this matches what's already written. It does, **with one correction and +one addition** we made this session: + +- **`SPEC_WAVESPEC.md` EXTENSION (2026-06-25), "rolling dyn-VGPR / traveling peak" (KG `1fdd5784`)** — the + governing sum-envelope rule (Σ instantaneous ≤ B), the trapezoid grow→peak→shrink, the "three jobs" + (occupancy, brick-avoidance, §3.3.3.2-unreachable). This IS our design; we're realizing it on the DSWS + claim-based split-K substrate. Build order R0–R4 there. +- **`MAD305_DSWS_MASTER.md:78-81`** — "Split-K is the headroom-creator; split-K creates brief peak windows + = the room to reallocate VGPR between roles." Matches. **Correction:** its "staggering buys nothing" + clause describes **full-K / n_kseg=1** (95% duty square wave). With real split-K the peak is brief and + staggering both avoids the collision AND enables the fungibility. kmbandy's framing: the split-K/dyn-VGPR + combo is for *whatever's needed at that instant* — compute-per-kseg OR role conversion — not one or the + other. It's one pool. +- **`vgpr_reserved`/`VRESV` sum-envelope** (`SPEC_DSWS_SUBSTRATE_V2.md:65,165`, `SPEC_DSWS_PHASEB_ + CONVERSION.md:95,130`) — already built, currently applied only to role-conversion grows. **Addition:** + extend the *same* envelope to gate the per-rowblk/per-kseg compute grow (§4.3). Same counter, same + primitive — this is the unification, not a new mechanism. +- **`HYBRID_DESIGN.md`** — the grow-ordering rule (grow before loading operands; the DEFERGROW hazard), + the Σ≤~1152/SIMD budget, RGA livereg. Keep the grow-ordering rule; it's orthogonal and still applies. + +--- + +## 6. What's built & tested this session (all offline unless noted) + +- **Entry-fix (`b01c722df`, committed)** — first-entry through `_alloc/_init`. Fixed the desktop-brick; + confirmed on silicon. +- **CPU model (committed at `5421751`, extended offline)** — `dsws_ctrl_model.cpp` + + `test_dsws_ctrl_model.cpp`: watermark/epoch/gate/reserve + snapshot/quiesce + entry-contract repro. All + pass. +- **QUIESCE race harness (`test_dsws_quiesce_race.cpp`, uncommitted)** — `std::thread` claimer+7-follower + model, 400 trials × {seq_cst, relaxed}, **0 stalls** — ruled out the protocol race. +- **Wedge-frame DIAG instrumentation (`occ_kernel_dsws.s`, uncommitted, byte-identity verified)** — an + `epoch_mark` macro + a claimer wait-loop snapshot of all advance-gate counters, routed into the host's + existing `ML8_COOP_STREAM` occ slots. Stream field map: `cons`=ROWBLK_DONE, `tiles`=AROW_DONE, + `compPh`=BFRAG_DONE, `comp:dsB`=QUIESCE_CNT, `feed:tr/pub/rawTi`=ROWBLK/AROW/BFRAG_NEXT, + `roles[C/A/B]`=per-role last-bumped epoch. RGA 0-spill, max-live 84. +- **GPU (supervised, kmbandy-greenlit):** the instrumented CONV=1/DIAG=1 4c2a2b bin ran **CLEAN** + (`ok=1536 bad=0`, occ[0]=0, occ[20]=33, dmesg silent) — the heisenbug that pinned the root cause. + Bins restored to safe CONV=0 immediately after. + +--- + +## 7. References + +**Repo docs (this dir unless noted):** +- `MAD305_DSWS_MASTER.md` — the vision, the issue-port wall, split-K framing. +- `SPEC_DSWS_SUBSTRATE_V2.md` — the claim-based split-K substrate + role economy. +- `SPEC_DSWS_PHASEB_CONVERSION.md` — the conversion economy + `vgpr_reserved` envelope. +- `SPEC_WAVESPEC.md` §EXTENSION — the rolling dyn-VGPR "traveling peak" design (R0–R4). **Primary.** +- `HYBRID_DESIGN.md` — grow-ordering rule, Σ≤pool budget, RGA gates. +- `RESULT_DSWS.md` — the substrate's GPU history (the feed-scheduling starvation class + K-outer fix). +- `docs/superpowers/specs/2026-07-01-dsws-phaseb-pool-economy-design.md` + the matching plan — the + pool-economy universal-dispatch design (Tasks 1-6 offline built, GPU gates T7-T10). +- `.superpowers/sdd/progress.md` — the SDD ledger; the full blow-by-blow incl the brick/SKIPQ/heisenbug. + +**KG (mneme) notes — search these:** +- `1fdd5784` rolling dyn-VGPR / traveling peak (THE architecture). +- `21cc0d63` root cause confirmed = multi-grower collision (this session). +- `d1a6d529` correction: entry-fix fixed the brick, not the residual hang. +- `2009aa16` dyn-VGPR OOR rule (no v>15 pre-grow) — already satisfied in the kernel. +- `98614f24` / `1cef3378` / `a241842c` ISA §3.3.3.2 multi-grower deadlock (authoritative). +- `11d86889` / `7c697c80` never dispatch a dyn kernel above the live SQ_DYN_VGPR cap (reboot lesson). + +--- + +## 8. Safety & operating rules (HARD — kmbandy standing orders) + +- **No GPU isolation exists.** gfx1201 drives the desktop; a hang can MODE1-reset it. **Bricks are + accepted** ("if it bricks, it bricks — we reboot and go from there") **but only with safeguards.** +- **kmbandy greenlights EVERY GPU dispatch individually.** Never auto-dispatch. +- **The safeguarded, proven-non-bricking dispatch config** (keep ALL of these): + ``` + ML8_POOL=1 ML8_COOP_CHUNK=8 ML8_COOP_CHUNK_MAXS=0.75 ML8_COOP_STREAM=1 \ + DSWS_NCOMP=4 DSWS_NAFEED=2 DSWS_NBFEED=2 DSWS2_NKSEG= \ + timeout 30 ./occ_dispatch --dsws2 4c2a2b + ``` + The chunk watchdog (`ML8_COOP_CHUNK_MAXS=0.75`) self-aborts a stalled chunk cleanly; `ML8_COOP_STREAM=1` + streams occ frames to disk so the wedge frame survives a hard hang. +- **NEVER disable the quiesce gate** (no `SKIPQ`-style force `s51=1`) — it removes the clean self-abort + and bricks. +- **Always rebuild+install the CONV=1 bin only at dispatch time, and restore the safe CONV=0 4840B bin + immediately after, unconditionally** (footgun removal — see `scratchpad/fire_wedge_diag.sh` for the + pattern). Never leave a CONV=1 bin installed. +- **`NFV=112 < 128` (the default `SQ_DYN_VGPR` cap)** — do NOT exceed 128 VGPR without the umr flip + (sudo, operator-only). Keep grows ≤128. +- **Never modify** `occ_kernel_coop.s`, `occ_dispatch.cpp`, `fp8_oracle.*` — shared/not-ours. Never stage + them. +- **Commit to git ONLY when kmbandy asks.** Never `--gl2c`. + +--- + +## 9. Your first tasks (next session) + +1. **Spec** the rolling dyn-VGPR grow economy from §4: exact LDS slot(s) for the compute grow envelope + (reuse `VRESV_OFF` or a sibling), the reserve/release macro, the stagger phase-token mechanism, where + they plug into `.Lcompute_claim`/`.Lcompute_grow`/`.Lcompute_shrink`, the per-kseg flush structure + (largely already there — each super-tile's ksi flushes to C), and the DIAG telemetry slots. Decide: + concurrent-peak count (start 1-2, sweep), burst length, stagger period. Save to + `docs/superpowers/specs/`. Self-review, then have kmbandy read it. +2. **Plan** (`docs/superpowers/plans/`) — offline tasks (CPU model envelope invariant → asm → assemble/RGA + → byte-identity) then the supervised GPU oracle gate at `n_kseg>1`. +3. **Build**, offline gates first. One greenlit GPU oracle run. Then the occupancy sweep. + +**Open question to resolve in the spec:** at `n_kseg>1`, confirm each kseg's partial flushes to C +immediately (atomic-add accumulate — it already does across super-tiles) so a compute wave truly shrinks +to lean between segments (true trapezoid), vs any path where accumulators pin the peak. kmbandy confirmed +the intent is per-kseg flush → immediate release. diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/MAD305_DSWS_MASTER.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/MAD305_DSWS_MASTER.md new file mode 100644 index 000000000000..ad2e1cf18c38 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/MAD305_DSWS_MASTER.md @@ -0,0 +1,172 @@ +# MAD-305 / DSWS — Master Pickup Doc (GPU kernel campaign) + +**Single source of truth for the dynamic-VGPR fp8 GEMM work.** Consolidated +2026-06-26 before a context compact. Read this first to resume cold. Detailed +history lives in the sibling docs + KG notes pointed to at the bottom. + +--- + +## 1. North star + the moat + +- **Goal:** 250–300 TF fp8 GEMM on **gfx1201** (AMD R9700 / Navi48, RDNA4, wave32). +- **The moat — dynamic VGPR (`s_alloc_vgpr`)** is armable on a compute queue **only + via raw PM4** writing **COMPUTE_PGM_RSRC2 bit 6** on a KFD queue (libhsakmt, + bypassing ROCr). HIP/the toolchain cannot emit it for compute. This is why the + whole harness dispatches via raw PM4 (`occ_dispatch.cpp` + `dvgpr_pm4/`), not HIP. + Confirmed: bit clear → reads 0; bit 6 set → `DYN_VGPR_EN` reads back. (MAD-304.) + +## 2. Current state (the numbers) + +- **fp8 WMMA ceiling = 307 TF** (15.9 WMMA/cyc); **272 TF** feed-free. One WMMA = + 16×16×16 = 8192 FLOP. R9700 = 64 CU × 2 SIMD32 = **128 SIMDs**. +- **Best real kernel ≈ 165.7 TF** (8×2 tile, KWINBPF double-buffer + s_setprio + + wide-A feed) on **square** shapes = **~52% of ceiling**, **8.34 WMMA/cyc**. + HIP 4×4-dbuf reference = 161 TF / 52.5% (VGPR 183/256, 16 frags = 128 acc VGPR, + occ 5 blocks = 20 waves). +- **On real ml8 dims = ~113 TF** (latest tile tuning, 3 verified fixes; commit + `99f063e9c`). ml8 shapes: **`down`** (ffn_down) M=2048 K=9216 N=2560; + **`down_pf`** (prefill) M=512. *Always bench on these, not throwaway squares.* + +## 3. THE WALL — it's the VALU issue port, NOT occupancy + +- Measured **~31 non-WMMA instructions issued per 32 WMMAs** (square shapes) → + caps throughput at **8.34 / 15.9 WMMA/cyc = 52%** of 307 TF. The issue port, + not memory and not occupancy, is the binding constraint. +- **Occupancy is FLAT**: minWaves 4→8 barely moves TF (~160 TF either way). So + raising occupancy is NOT the lever. +- **The lever = cut non-WMMA issues per WMMA** — via (a) more fragment reuse, or + (b) wave specialization so the math wave issues ~only WMMAs while other waves do + the address/feed VALU. This is what DSWS targets. +- Caveat: the 165 TF winner + the 31:32 ratio are **square** shapes. ml8 shapes + (tall-skinny `down`) differ — re-measure the issue mix ON ml8 with `--att`. + +## 4. SOLVED this week (both real, keep) + +### POOLTERM — pool≥2 teardown brick (FIXED, confirmed on silicon 2026-06-26) +- **Root cause:** the compute terminal in `occ_kernel_coop.s` was a **POOL=1-only + diagnostic stub** — compute counted tiles it processed (s57) and exited at + s57==TOTAL. Valid only when ONE WG owns all tiles. At pool≥2 the WGs SPLIT tiles + via the shared global atomic claim, so each compute gets ~1s) persistent dispatches starve + the compositor's gfx ring → brick. Fix: bound each dispatch to `ML8_COOP_CHUNK` + output tiles (claim `occW[5]=base`, `userdata[11]/s11=hi` terminal), drain on the + EOP fence, YIELD between chunks, `ML8_COOP_CHUNK_MAXS` guard. Sub-second bounded + dispatches run the real ml8 GEMM **imperceptibly** on the display GPU. + +## 5. DSWS — the design (the actual north-star architecture) + +**"Dynamic-Split Wave-Specialization" / adaptive wave-role economy.** ONE kernel +that re-balances its mix of **{fat compute waves (large VGPR, hold fp32 WMMA +accumulators) / lean A-feed waves / lean B-feed waves}** to the bottleneck **at +runtime**, using `s_alloc_vgpr` to make the VGPR budget **fungible across roles +asymmetrically** (shrink one fat compute ~112–256 VGPR → fund several lean feeders, +or vice-versa). Bottleneck sensed by the prod/cons ring counters already in +`occ_kernel_coop.s`. + +- **Split-K is the headroom-creator, NOT an occupancy play.** Full-K GEMM holds the + fp32 accumulators (~64 of 112 VGPR) at peak for the ENTIRE K-loop (~95% duty — + square wave, not trapezoid), so staggering buys nothing. Split-K creates brief + accumulator-peak windows = the room to reallocate VGPR between roles. +- **Why this beats static:** the issue-port wall (§3) is broken by letting the math + wave issue ~only WMMAs while feed waves absorb the address/feed VALU — and tuning + that ratio per-shape (and within a kernel) to whatever the runtime says is short. + +## 6. Prior art + the research — VERDICT (Murmur prior-art-scan, landed 2026-06-27) + +**Workflow `5ec8a958` (42 scouts on 6900xt + Qwen3.6-27B captain sift). Verdict: +the RUNTIME-ADAPTIVE piece is NOVEL; the lean/fat mechanism is established (borrow +it).** Ranked leads: + +1. **Runtime-adaptive producer:consumer wave-role rebalancing with dynamic per-wave + register reallocation is NOT in any known GPU GEMM/compute kernel** (HIGH conf — + unanimous negative across ~40 scouts: arXiv 2023–2026, AMD GPUOpen, NVIDIA + CUTLASS/CuTe, Triton, LLVM/ACO, forums). → DSWS's core (in-kernel ring-counter + sensing + runtime `s_alloc_vgpr` rebalancing) is **architecturally novel**. +2. **CUTLASS warp-spec = STATIC `setmaxnreg` / `warpgroup_reg_alloc` at launch** — + producer(DMA/TMA):consumer(MMA) split + register budgets fixed for the whole + kernel. The direct baseline to differentiate from; register fungibility is a + compile/launch-time optimization, not a runtime control loop. +3. **AMD `s_alloc_vgpr` (RDNA3.5/4) = static per-wave; no in-kernel sensing / runtime + shifting.** The primitive we repurpose; the ISA lacks native runtime-rebalancing + semantics → DSWS supplies its own coordination layer (= the contribution). +4. **Adjacent dynamic work (Stream-K, persistent kernels, WaveTune) balances at the + CTA/workgroup level, NOT intra-CTA wave roles or mid-flight register realloc** + (HIGH conf). Rules out the nearby paradigm a reviewer might conflate us with. + +**GAPS / must verify against primary sources before we stake the claim:** +- Exact CUTLASS `setmaxnreg` register ranges + granularity (the "24..256 step 8" + numbers) and documented HW restrictions — NOT extracted by any scout. +- Precise producer:consumer warp ratios in CUTLASS / CK-rocWMMA / Triton persistent. +- AMD `s_alloc_vgpr` HW rules re: barriers, occupancy, wave-launch sync in COMPUTE + shaders (vs the RT path) — needs ISA-level confirmation. (Ties to our own + barrier-vs-dyn-VGPR deadlock history — KG `8a9ce97f`/`17f209af`.) +- Novelty is partly absence-of-evidence; one citation (`WaveTune arXiv:2604.10187`) + is suspect (possible 8B-scout hallucination) — do not cite unverified. + +Full LEADS doc: handoff `141924b9` result in quantdb on mad-lab-2026. Captain banked +to claude__main KG. (This run also surfaced + fixed the engine bug that had been +truncating the captain's scout bundle — KG `22063f79`.) + +## 7. Key files + bins (all on mad-lab-main, committed to fork master) + +- `occ_kernel_coop.s` — cooperative kernel; **POOLTERM** fix; the prod/cons ring + counters DSWS will sense; SAFEPROBE bounds guard. +- `occ_kernel_wavespec.s` — static loader/compute role-split prototype (WS path). +- `occ_kernel_mbgemm.s`, `occ_kernel_wggemm2.s` — the 8×2 / KWINBPF winner lineage. +- `occ_dispatch.cpp` — raw-PM4 KFD harness; compositor-safe chunking; oracle gate. +- `build_coop.sh` / `build.sh` — defsym build matrices (FM,FN,P,RINGD,BATCH,DYNVGPR). +- `../dvgpr_pm4/` — PM4 packet defs + RSRC2 bit6 arming. +- **Detailed specs:** `SPEC_WAVESPEC.md` (DSWS v1+v2, wall numbers, prior art), + `MORNING_PICKUP.md` (day arc), `RESULTS_ml8_dynvgpr_gauntlet.md`, + `MAD305_LEVER_CATALOG.md`, `L4_LEAN_DESIGN.md`, the `RELATED_WORK_SWEEP` + + `NVIDIA_KERNEL_IDEAS_FOR_RDNA4` research docs. + +## 8. STANDING SAFETY (hard rules — do not violate) + +- **A GPU brick is a BUG, never an "accepted tax."** Recoverable-via-MODE1/reboot + ≠ acceptable. Root-cause and fix; don't route around. +- **THE USER decides EVERY GPU dispatch** — each individual dispatch, not one + blanket "go" for a session. A hang/timeout on an unproven kernel = full STOP + + report, NOT auto-fire the next variant. Do max work OFFLINE first (disasm, RGA, + static analysis); batch a diagnostic into ONE prepared dispatch. +- Only **sub-second bounded** dispatches are safe on the display GPU (R9700 drives + the monitors). Freeze dyn dispatch on any build known to leave the queue non-idle. +- **NEVER pass `--gl2c`** (MES-crash landmine). Keep SAFEPROBE clamp + bounds gate + + padding ON. Every run streams to disk (`ML8_COOP_STREAM=1`). +- Do NOT move displays to the eGPU 6900XT (can't init pre-login; also NOT the + single-GPU target user — the kernel must coexist with the compositor). + +## 9. NEXT STEPS (resume order for step 4) + +1. **Incorporate the Murmur prior-art leads** (research output) into the DSWS + novelty + setmaxnreg mechanics-to-copy. +2. **R0 occupancy attribution** — step pool up ONE at a time (gated, abort on any + non-clean exit); RGA showed compute peak-live ~81 vs HW-allocated 120 → trim NFV + toward ~96 = free occupancy. +3. **ML8_P=2** — one feed + TWO compute waves in one WG = the real reuse/throughput + lever (reuse 2.0/2.4/2.67 for P=2/3/4). P>1 needs its own terminal review. +4. **DSWS build** — brainstorm → spec → plan → TDD, on the POOLTERM substrate, + measured on ml8 `down`/`down_pf` with `--att` for the issue mix. + +## 10. Pointers + +- KG: `dac0bb8c` (POOLTERM root cause), `0a2cea44` (POOLTERM silicon confirm), + `63583120` (DSWS v2), `21827908` (compositor chunking), `34f29c00` (full-day + session summary), `58a41155` (GPU-dispatch discipline), `1630687a` (earlier + brick = OOB shader access root cause + the mandatory bounds-guard lesson). +- Jira epic: MAD-305 (under the 250-300 TF north star). MAD-304 = the PM4 dyn-VGPR + arming. MAD-300 = the WMMA ceiling + rocWMMA baseline. +- Branch/state: fork **master** (pushed `9ff961564`), all dvgpr work committed + (`a514395ec` snapshot + the upstream merge). diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/MORNING_2026-07-04.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/MORNING_2026-07-04.md new file mode 100644 index 000000000000..4f6443c63b2c --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/MORNING_2026-07-04.md @@ -0,0 +1,152 @@ +# DSWS — Morning pickup, 2026-07-04 + +Written end of 2026-07-03 session. **Two tasks for the morning (kmbandy's call), in order:** + +1. **Chase the "conversion system won't react" bug** — with the sensor fixed, the adaptive economy sat + frozen at 4/2/2 while throughput was catastrophic (0.1 TF) *and* a watermark trigger condition was met. + A non-reacting adaptive controller under catastrophic perf is a **bug**, not a "balanced" state. +2. **Explore decoupling accounting from wave launch** — stop gating waves on the accounting; launch waves + freely (grind-style) and have the accounting *intervene only when needed*, not sit on the per-super-tile + critical path. + +> META-NOTE TO SELF (I did the bad thing again): I hit a horrific number with zero adaptive reaction and +> started to conclude "adaptive is a dead end, pivot to grind." kmbandy correctly flagged that the *lack of +> reaction* is itself the evidence something is broken. Do NOT declare the adaptive path dead from a run +> where the adaptivity provably failed to fire. Measure/fix the actuation first. + +--- + +## TASK 1 — the conversion system doesn't react (chase it down) + +### What we saw +End of session I fixed the ring-occupancy **sensor** (details below). After the fix: +- Sensor reads REAL mid-drain occupancy: **occA = 4–6 (mean 4.7), occB = 4 (constant)** — verified in + `/home/kmbandy/dsws_gpu_logs/trace_fixed_4c2a2b.csv` and `trace_fixed2_4c2a2b.csv`. +- Roles held **static 4/2/2** the whole run. **convCount = 0.** +- Throughput **0.1 TF** (span ~910k ticks) — vs 0.7 TF for the same 4/2/2 at CONV=0, and 0.7 TF for the + OLD (broken-sensor) CONV=1 that ran away to 1/6/1. + +### Why "0 conversions" is a BUG, not "balanced" +The watermark rule (`occ_kernel_dsws.s`, thresholds ~L687-694): +- `occ_X < CTRL_LOW(=1)` → compute starved → compute→feed. +- `occ_X > CTRL_HIGH_X` → feed over-serving → feed→compute. `CTRL_HIGH_A=G-1=5`, `CTRL_HIGH_B=FN-1=3`. + +**occB = 4 every super-tile, and 4 > CTRL_HIGH_B(3).** So the B-feed wave's decision (site now reading +`lds_get s56, OCCB_PUB_OFF; s_cmp_gt_u32 s56, CTRL_HIGH_B` → dir 3 B-feed→compute) *should fire every +epoch*. It fires **zero** times. The actuation is not happening when its own condition is met. + +### Suspects to check (in rough priority) +1. **Floor block.** `conv_dec_floor` refuses to drop a role slot below 1. `NBFEED=2` but **wid0 (the + claimer) is B-feed-CLASS** — so there may be only **1 actual B-feed follower** that can run the decision, + and `NBFEED_SLOT` decrementing 2→1 may be blocked or the only decrementer is the claimer (which doesn't + run the follower decision path). Check: who actually executes the B-feed conversion decision, and does + the floor guard (`s_cmp_le_u32 s52, 1`) stop it at the launch count? +2. **try_gate epoch gate.** Conversions gate on `E = segcnt >> EPOCH_SHIFT` (default EPOCH_SHIFT=3). One + single-winner per (dir, epoch). If E advances slowly or the gate is always "already fired," the decision + loses. Earlier we found EPOCH_SHIFT=0 made FORCE conversions fire (=4). Re-check whether the real + watermark path ever wins a ticket at default EPOCH_SHIFT. +3. **Decision code not reached.** Do the feed waves even reach their `_quiesce` conversion decision at + default settings, or do they bail/retire earlier? Add a DIAG counter at each decision site (entered / + condition-met / ticket-won / committed) to see WHERE it dies. This is the "instrument it, don't reason" + move. +4. **Threshold semantics.** occB=4=FN (ring full). Is "full B-ring" really "over-serving" (convert away) or + "healthy" (compute never starves)? The CTRL_HIGH semantics may be miscalibrated — full ring might be the + GOOD state. But even so, #1-#3 are about the actuation firing AT ALL, which is the real bug. + +### The deeper thing the fix revealed (important context, not a conclusion) +Same 4/2/2 mix: **0.7 TF at CONV=0 but 0.1 TF at CONV=1.** The CONV=1 machinery adds ~7× per-super-tile +overhead. And the OLD runaway to 1/6/1 was accidentally *faster* (0.7) because **1 compute wave shortcuts +the per-super-tile quiesce handshake** (claimer synchronizes 1 over-claim instead of 4). So the adaptive +economy's apparent throughput was never feed-balancing — it was accidentally gaming the coordination +overhead by minimizing compute waves. **This is exactly why Task 2 matters.** + +--- + +## TASK 2 — decouple accounting from wave launch + +kmbandy's framing: *"why are we waiting for the accounting to launch a wave? We should just be launching +waves and only have the accounting intervene when it needs to."* + +### The problem it names +Today the accounting IS the critical path: persistent-wave WG + per-super-tile **epoch publish → feed → +compute → quiesce handshake**, with a fixed compile-time super-tile (192 WMMAs). Every super-tile pays the +full lockstep coordination. That's: +- the ~226-ticks/super-tile fixed cost that pins split-K at 0.7 TF **regardless of shape** (fingerprint: + 229 ticks/ST small shape vs 225 big — see session data), and +- the 7× CONV=1 penalty (handshake cost scales with compute-wave count), and +- the reason adaptivity is entangled/hard to make react cleanly. + +### The idea to explore +Launch waves and let them RUN (grind-style: 1 wave claims work, full-K in registers, no per-tile +rendezvous), and make the accounting an **asynchronous side-controller** that only intervenes when a +bottleneck actually appears — NOT a per-super-tile gate every wave waits on. Concretely, questions to design +around: +- What is the minimal "intervene when needed" signal? (e.g. a wave notices it stalled on feed N times → + raises a flag → a controller occasionally rebalances; waves don't block on a barrier each tile.) +- Can the dyn-VGPR grow/shrink stay (brief peak / stagger) WITHOUT the lockstep quiesce? i.e. keep the + register-multiplexing but drop the synchronous handshake. +- Does the accounting even need to be per-WG-LDS synchronous, or can it be a lazy global/periodic thing? +- Relationship to grind: grind already IS "launch and run, no accounting." The question is whether a + *light* async accounting layer on top of a grind-like base buys anything (it only matters if some + workload is occupancy/feed-bound — which so far nothing is; keep testing). + +### Hardware constraints to respect (established this session) +- A wave = 32 lockstep SIMD lanes with ONE uniform VGPR count. **Cannot** sub-specialize lanes or partition + a wave's VGPRs to roles without divergence/spill. Specialization is ≥ wave-granularity. Runtime-indexed + registers are a measured LOSER here. +- Residency: grind (1 wave/WG, 120 VGPR, 6144B LDS) caps ~640 resident WGs (LDS-limited). split-K pool is + clamped to 64 in the host (`occ_dispatch.cpp:1823` — `pool = poolD<64?poolD:64`). +- B ≈ 1536 VGPR/SIMD, 16 waves/SIMD (2048 total), per-wave dyn cap ~128 / static cap 256. + +--- + +## Session data map (so morning doesn't re-derive) + +**The headline comparison (fair, same shape 1920×2048×2048, oracle CLEAN both):** +| kernel | TF | bound by | +|---|---|---| +| split-K DSWS (dsws2) | 0.7 (flat, pool 16 & 64) | coordination (fixed 226-tick super-tile) | +| **grind (non-split-K, 1 tile/WG, full-K, write-once)** | **6.9** (pool=128; drops as pool↑) | **memory bandwidth** | + +- split-K is **shape-INVARIANT ~0.7** (every ml8 shape, mix, sleep, size) — fingerprint of the compile-time + fixed super-tile (192 WMMAs always; shape only changes the COUNT). +- grind **scales with shape** (576²→1.1, 1920×2048×2048→6.9) and is BW-bound (TF drops with more WGs). +- **C-write amplification is 67% of split-K's wall**: NOCFLUSH (skip the `global_atomic_add_f32` C flush, + keep split-K n_kseg=32) → 0.7→2.1 TF (3×). But it's WRITE VOLUME, not atomic contention: CSTORE probe + (non-atomic store, same count) = 0.5 (slower!), KMAJOR probe (spread contention in claim-time) = 0.7 + (no change, oracle GREEN). So: split-K writes each C cell n_kseg=32× → the fix that helps is fewer writes + (coordinated/WG-local reduction), NOT reordering. +- Tile size is the grind lever: 2×4 (current, 6.9) → 4×4 known ~147-155 TF fed in this project's prior work + (`gemm_fp8_levers`), 276 TF pure-WMMA. NOT YET TESTED on grind — a `defsym FM=4 FN=4` run at + 1920×2048×2048 is the obvious next data point. + +## State / where things are + +- **Kernels:** `occ_kernel_dsws.s` (split-K adaptive; has the UNCOMMITTED sensor fix + probe flags below), + `occ_kernel_grind.s` + `occ_kernel_grind.bin` (NEW control kernel, 2280B, 0-spill, oracle CLEAN). +- **Host:** `occ_dispatch.cpp` — `run_dsws2` (with TF/trace/occupancy/probe readouts) and NEW `run_grind` + (`--grind`, env `GRIND_M/N/K/KCHUNK/BIN`, default 576×512×2048; no pool clamp). Probe flags in the + dsws kernel: `NOCFLUSH`, `CSTORE`, `KMAJOR` (all default 0 = byte-identical), plus `TFPROBE`, `TRACE` + (per-super-tile CSV), and the occupancy/fat-wave counters. +- **Sensor fix (UNCOMMITTED, in `occ_kernel_dsws.s`, all `.if DSWS2_CONV`-gated → CONV=0 byte-identical + c62568f6):** new LDS words `OCCA_PUB_OFF`/`OCCB_PUB_OFF` (after QUIESCE_CNT); claimer tracks per-super-tile + ring PEAK (s73/s74) across its wait_done spins and publishes them at `.Lqc_q_ok`; the 3 conversion sites + (`.Lbfeed`/`.Lafeed`/`.Lcompute` decisions) now `lds_get` the published peak instead of `occ_sample` at + their own post-drain quiesce. This is the fix that killed the runaway but exposed the non-reaction bug. +- **Safe bins installed (verified):** `occ_dsws2_4c2a2b_gd.bin` c62568f6 (4872B, CONV=0/DIAG=1/SAFEPROBE), + `occ_dsws2_6c1a1b_gd.bin` f57c59d5, `occ_dsws2_2c3a3b_gd.bin` 2291f715. Nothing unsafe installed. +- **Committed HEAD:** `f2cc1f26e` (scaffolding) on `feat/dsws-phaseb-conversion`. Everything since (TFPROBE, + trace, occupancy probe, NOCFLUSH/CSTORE/KMAJOR probes, sensor fix, grind kernel + run_grind) is + **UNCOMMITTED** working-tree changes. +- **Repro commands:** + - CONV=1 trace (shows the frozen 4/2/2 + occA/occB): `LABEL=x MIX=4c2a2b SEGK=64 NKSEG=32 OMTL=3 ONTL=8 + POOL=16 bash /home/kmbandy/dsws_gpu_logs/fire_trace.sh` → `trace_x_4c2a2b.csv`. + - grind: `GRIND_M=1920 GRIND_N=2048 GRIND_K=2048 ML8_POOL=128 ./occ_dispatch --grind`. + - split-K fair: `ML8_POOL=16 ML8_COOP_CHUNK=0 DSWS2_SEGK=64 DSWS2_NKSEG=32 DSWS2_ORACLE_MTL=10 + DSWS2_ORACLE_NTL=32 ./occ_dispatch --dsws2 4c2a2b` (needs a TFPROBE=1 dsws2 bin installed; restore + c62568f6 after). + +## Standing safety +Every GPU dispatch is kmbandy-greenlit, one at a time. Never `--gl2c`. All data to REAL DISK +(`/home/kmbandy/dsws_gpu_logs/`), never tmpfs. Always restore the safe CONV=0 bin (c62568f6) after any +CONV=1 / probe / TFPROBE dispatch. A brick is a bug, not a tax. diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/MURMUR_ENGINE_TRIAGE.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/MURMUR_ENGINE_TRIAGE.md new file mode 100644 index 000000000000..f7c50885529e --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/MURMUR_ENGINE_TRIAGE.md @@ -0,0 +1,113 @@ +# Murmur Engine — Morning Triage (2026-06-26 night → pickup) + +## TL;DR +Spent the night making the Murmur dynamic-workflow engine produce clean research +output on the junk-drawer fleet. **FOUR real bugs root-caused + TDD-fixed.** The +LAST fix (removing openai-python) introduced a **concurrency regression** that is +the open blocker: under real swarm load every model call fails `model_unreachable` +even though the llama-servers are healthy (HTTP 200). The v4 run is PAUSED. + +## The four fixes that LANDED tonight (all TDD'd, all live) +1. **paged-KV seq_rm crash** — llama.cpp fork, **COMMITTED+PUSHED master `1d9d93906`**. + seq_rm tail-truncate left freed blocks as holes → `compute_slot_mapping` + GGML_ASSERT abort under swarm prompt-reuse. Fixed via `BlockTable::truncate` + restoring the `num_blocks==ceil(live/block)` invariant. Validated on silicon + (cards survived the load that bricked them). KG `b2ac012f`. +2. **MCP-bridge concurrency wedge** — `engine/mcp_bridge.py`. No per-call timeout + + per-request session storm → scouts hung 19 min. Added 45s `wait_for` + 8-way + semaphore. KG `6fdb764d`. +3. **forced-conclusion** — `engine/loop.py` + `types.py`. The agentic loop offered + tools EVERY turn and never forced a final answer → tool-happy models burned all + `max_turns`(8) searching, emitted no `content` → captured the raw reasoning + ramble. Fix: reserve the FINAL turn, strip tools, inject `_FINAL_TURN_NUDGE`; + `max_turns` 8→12. VALIDATED: v3 done scouts produced CLEAN FINDINGS (lens + 512–2970c, not 18k rambles). KG `a9ded2b5`. +4. **openai-python REMOVED** — `engine/model.py`. A prior session re-introduced + `openai.AsyncOpenAI` into the engine (contradicts the engine's whole purpose). + Rewrote `ModelClient` on raw httpx + explicit `timeout(read=300)`. 89/89 engine + tests, validated single-request live. KG `c306326c`. + ***** THIS IS THE ONE THAT BROKE UNDER LOAD ***** + +## THE OPEN BUG — morning task #1 +v4 workflow `ab88955c-0e07-40c3-8432-dad92880e725` (PAUSED): ALL agents fail +`handoff run error: model_unreachable`: +- 6900xt: **18 failed** (it was the HEALTHY card that carried v3 — this is a + REGRESSION), 8 in_progress (expired on pause) +- 480: 8 failed; 1070: 1 done (an 8996c ramble) + 7 failed +- **BUT `curl /health` on all three servers (6900xt:8092, 480:8097, 1070:8095) + returns HTTP 200 — the servers are UP.** + +**Diagnosis:** the httpx `ModelClient` swap regressed model calls UNDER CONCURRENCY. +v3 (openai client) had the 6900xt working (it carried the run solo). v4 (httpx) has +the 6900xt failing. Single-request httpx works (mock test + isolated 480 test +post-restart, both clean). Only fails under 42-scout × 12-turn concurrent load. +`engine/loop.py` maps ANY `model.stream` exception → `model_unreachable` (the +`except Exception as e: yield em.error(code="model_unreachable", message=str(e))`), +**MASKING the real error.** A 15-concurrent `ModelClient.stream` repro against the +6900xt HUNG (120s timeout, no output) — consistent with connection exhaustion/hang. + +**PRIME SUSPECT:** `ModelClient._raw_chunks` creates a FRESH `httpx.AsyncClient` +PER `stream()` call (`owns = self._client is None`). 42 scouts × up to 12 turns = +hundreds of concurrent AsyncClient creations to the same few hosts → likely +ephemeral-port / fd exhaustion OR pool issues → httpx raises/hangs. +**SECONDARY:** the `_raw_chunks` async-generator pattern (`async with +client.stream` + `yield` + `finally aclose`) under concurrency / early-close. + +## MORNING PLAN +1. **Unmask the real exception.** Either log `str(e)` at `engine/loop.py`'s + `except Exception` (only "model_unreachable" reaches the handoff result), or + re-run a concurrent `ModelClient.stream` repro that prints each task's exception + class (add a per-task `asyncio.wait_for` so a hang surfaces as TimeoutError). + Goal: see the actual httpx exception. +2. **LIKELY FIX: inject a SHARED app-level `httpx.AsyncClient`** with + `httpx.Limits(max_connections=N, max_keepalive_connections=M)` instead of + per-request creation. `ModelClient` ALREADY supports `client=` injection. The + dashboard handler (`mad-dashboard.py` ~2306) should build ONE shared AsyncClient + (module-level, with the explicit timeout + sane limits) and pass it to every + `ModelClient`. TDD: a concurrency test (N concurrent `stream()` → assert all + succeed). +3. **DO NOT revert to openai** (`model.py.pre-httpx.bak`). openai must stay OUT + (explicit design goal). Fix the httpx version forward. +4. **Re-validate:** fire a fresh murmur, confirm 6900xt succeeds + clean FINDINGS + + 480/1070 contribute. + +## ALSO OPEN (separate, lower priority) +- **480/1070 (Q5_K_M) tool-calling quality:** in isolated tests they give up + WITHOUT searching ("unable to locate") while the 6900xt (Q6_K) searches+fetches. + Possibly Q5-vs-Q6 tool-calling degradation. Address AFTER the concurrency fix. +- **2026 RAM pressure** (238 MiB free): the 480/1070 servers run `--ctx-size + 524288` (512k) — overkill for ~9k-context scouts. Drop to ~64k (frees RAM + + speeds them). The semantic-index embedder model itself is fine (n_ctx capped at + 512, `mt-embed.cpp:48`) — NOT the RAM eater. +- **Murmur v2 polish (banked):** tool-call budget, redispatch load-rebalance, + on-demand force-wrap control. (Note: the forced-conclusion fix likely supersedes + the earlier "scouts must call handoff_complete" idea — content capture works now.) + +## FILES TOUCHED (mad-lab-dash, NOT committed — backups alongside) +- `engine/model.py` (httpx rewrite) — `.pre-httpx.bak` +- `engine/loop.py` (forced-conclusion) — `.pre-forcedturn.bak` +- `engine/types.py` (max_turns 12) — `.pre-forcedturn.bak` +- `engine/mcp_bridge.py` (timeout+semaphore) — `.pre-timeout.bak` +- `engine/tests/{test_model,test_loop}.py` — new tests (+ test_mt-block-table, + test-paged-lifecycle in the llama.cpp fork) +- `mad-dashboard.py` + `engine/tests/smoke_{live,image}.py` (openai removal) — + `.pre-noopenai.bak` +- `~/.config/mad-lab-agents/config.json` (murmur prompts; tools trimmed by kmbandy) + — `.pre-v2prompt.bak` +- llama.cpp fork: **COMMITTED + pushed** (master `1d9d93906`). + +## RUN IDS +- v4 PAUSED (broken httpx-concurrency): `ab88955c-0e07-40c3-8432-dad92880e725` +- v3 (forced-conclusion validated; openai-timeout era; killed): `2b68541b-...` +- v1 (pipeline mechanically validated end-to-end; done): `1c2c3ef2-...` + +## INFRA NOTES +- mad-dashboard.service = SYSTEM service → needs kmbandy `sudo systemctl restart` + for engine code changes to load. (Tonight's restart "hung then came back" — worth + confirming it loaded cleanly; though the isolated post-restart httpx test DID work, + so the new code is live.) +- mad-lab-mcp.service (dispatcher + orchestrator + workflow tools) = SYSTEM service. +- Fire a murmur: `cd ~/GitHub/mad-lab-mcp && python3 fire_priorart.py` (on 2026). +- The whole point: prior-art scan for MAD-305/DSWS — does anyone do RUNTIME-ADAPTIVE + producer:consumer wave-role rebalancing (vs CUTLASS static setmaxnreg)? Feeds the + GPU kernel work (MAD305_DSWS_MASTER.md §6 on mad-lab-main). diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/PHASE_PROFILE.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/PHASE_PROFILE.md new file mode 100644 index 000000000000..1a10fc0a00a5 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/PHASE_PROFILE.md @@ -0,0 +1,172 @@ +# DSWS v2 — In-Kernel Phase Profile (compute-wave time breakdown) + +**What this is:** measured, in-kernel timing of where the DSWS v2 split-K compute wave actually +spends its cycles — *not* inferred from lever-pulls. Each compute wave stamps `GET_REALTIME` (RTC, +100 MHz) at every phase boundary, accumulates per-phase ticks in registers, and emits them once at +retire. Host sums across all compute waves and prints ticks + % per phase. + +This doc is the running record: every profiling run's **args, shape, and breakdown** goes in the +table below. Add rows; don't overwrite. + +--- + +## The six phases (compute wave critical path) + +| phase | what it measures | +|---|---| +| `FOLLOW_WAIT` | idle — spinning until the claimer publishes the next super-tile (coordination latency) | +| `STAGE_WAIT` | idle — spinning until the A/B feed waves have staged this super-tile's operands | +| `GROW` | rowblk claim + `s_alloc_vgpr` dyn-VGPR grow 32→112 | +| `WMMA` | LDS frag loads + `v_wmma_f32_16x16x16_fp8` compute over the SEGK segment | +| `FLUSH` | split-K C reduction — `global_atomic_add_f32` per (frag,elem), drained | +| `SHRINK` | `s_alloc_vgpr` dyn-VGPR shrink 112→32 | + +## Methodology / how to reproduce + +- **Kernel:** `occ_kernel_dsws.s`, gated on `-Wa,-defsym,PHASEPROBE=1`. Accumulators in SGPRs + `s78..s83` (last-stamp RTC in `s77`); single `phase_flush` atomic-add to `occ[64..69]` (bytes + 256–276, above the per-chunk memset) at compute retire. **PHASEPROBE=0 → `.text` byte-identical to + the safe production bin `c62568f6`.** +- **Build (per run):** `DSWS2=1 FM=2 FN=4 G=6 SEGK=64 SAFEPROBE=1 DIAG=0 TFPROBE=1 PHASEPROBE=1 + NOCFLUSH=<0|1> NCOMP= NAFEED= NBFEED= DSWS2_CONV=0` +- **Run (per run):** `ML8_POOL=16 ML8_COOP_CHUNK=0 DSWS2_SEGK=64 DSWS2_NKSEG=32 DSWS2_ORACLE_MTL=3 + DSWS2_ORACLE_NTL=8 ./occ_dispatch --dsws2 ` — **single pass** (no `DSWS2_TARGET_SECS`). +- **Shape:** `576×512×2048`, super-tile `192×64` (G=6 FM=2 FN=4), `n_kseg=32`, `TOTAL_super=768`, + `POOL=16` WGs. (M=576 = the proven-brick-safe regime; the distribution is shape-robust.) +- Logs on real disk: `~/dsws_gpu_logs/phase_ + +### Gate 1 dispatch #1 — HUNG (not bricked), 2026-06-29 + +First GPU dispatch of the campaign fired (user-greenlit). `4c2a2b` static, `down_pf`, single-tile, but at the +harness **default pool=64** (I left `ML8_POOL` unset). Result: **clean software deadlock**, NOT a brick — +`timeout 30` caught it, teardown deliberately did not destroy the queue, dmesg silent (zero ring/fence/fault/MES), +both GPUs responsive. Wedge frame (direct atomics, reliable): `maxlive=64 claim=64 live=1` → 63 WGs correctly +retired on `ti>=TOTAL`, the **one WG that won `ti=0` deadlocked** in produce/consume. All fine markers 0 — but +that was instrumentation blindness (see below), not proof of no progress. + +**Root cause of the blindness (2026-06-29):** `DIAG` defaults to 0 and `build_dsws.sh` never passed it → every +`mark` compiled to nothing. NOT the 2026-06-24 "v2 clobber" hypothesis (v2 is set once at kernel:276 and never +rewritten; the v2==0-gated `lds_put` provably works since the 63 WGs retired via LDS broadcast). Additionally the +live DSWS feed bodies (`.Lbfeed_role`/`.Lafeed_role`) had ZERO marks — the feed marks lived in the dead +`.Lfeed_role`. FIX (all offline, verified): `build_dsws.sh` now passes `DIAG=${DIAG:-1}`; added 10 marks to the +DSWS feed bodies (B: feedPhase/ti + slotok[18]/loadtr[19]/dsstore[20]/publish[21]; A: reached[28]/Aload[29]/ +dsstore[30]/publish[31] on fresh occ slots 28-31); harness zeros+prints occ[28..31] + a DIAGINIT line (occ23-27). +Gates: 6 DIAG=1 bins clean, 55 mark atomics in disasm, RGA livereg 82 / 0-spill, DSWS=0 still 1716B byte-identical. + +**Reflight plan (pool=1, awaiting greenlight):** matches the kernel's POOL=1 DIAGNOSTIC TERMINAL design, removes +the 64-WG claim-contention variable, and with DIAG=1 will localize the exact wedge instruction: +`ML8_COOP_STREAM=1 ML8_COOP_CHUNK=8 ML8_COOP_CHUNK_MAXS=0.75 ML8_POOL=1 DSWS_NCOMP=4 DSWS_NAFEED=2 DSWS_NBFEED=2` +`DSWS_RINGD=2 DSWS_DYN=0 DSWS_ONLY=down_pf DSWS_ORACLE_MTL=1 DSWS_ORACLE_NTL=1 timeout 30 ./occ_dispatch --dsws` + +### Gate 1 dispatch #2+#3 — root cause LOCALIZED + CONFIRMED, first DSWS green (2026-06-29) + +**#2 (pool=1, 4c2a2b):** HUNG again, but DIAG-instrumented this time. Wedge frame: +`INIT adm1 tmr1 lds1 rdv1 | feedPh2 compPh4 cons3 tiles0 | feed:tr2 pub3 comp:dsB2 wm3`. Decode: init/rendezvous +FULLY PASSED (kills the old init-deadlock fear for static); the protocol RAN and made real progress (B published 3 +steps, compute did 3 WMMA) then froze at step ~3 of 32. → a MID-K-LOOP producer/consumer stall, not an init wedge. + +**Root cause (LOCALIZED by evidence + code, CONFIRMED by #3):** the A-feed loop is **band-outer / K-inner** +(`occ_kernel_coop.s` `.Lafeed_role`: `.rept NCOMP` wraps the K-loop), so when an A-feed wave owns >1 band it +produces band b's ENTIRE 32-step K-loop before touching band b+NAFEED. The compute waves consuming the not-yet-fed +bands starve → never release their consumer counters → `min_cons` stays low → the shared B-ring (depth RINGD=2) +can't recycle slots → B and all compute jam → WG wedges at ~step RINGD. B-feed does it RIGHT (K-outer / frag-inner). + +**#3 (pool=1, 2c3a3b) — CONFIRMING PROBE: `oracle CLEAN ok=16 bad=0`.** With NAFEED=3 ≥ NCOMP=2 every band has +its own A-feed wave → zero band-sequential starvation → GREEN. Airtight: the only changed variable is bands-per- +A-feed-wave, and it flips green↔hang exactly as predicted. **FIRST oracle-green of the DSWS 3-role kernel — the +cooperative A+B-feed→compute→store protocol is NUMERICALLY CORRECT.** The hang was a pure feed-scheduling bug. + +All three dispatches HUNG-or-completed CLEANLY (no brick; dmesg silent, R9700 responsive each time). + +**THE FIX (applied + verified 2026-06-29):** restructured `.Lafeed_role` to **K-outer / band-inner** (mirror +B-feed): every K-step produces one step for ALL owned bands, so no compute band starves. `astep==k` fell out for +free (use the loop counter directly; recompute per-band saddr from trow each iter — lean feed has SGPR headroom). +Offline gates: 6 cells assemble, RGA livereg 82 / 0-spill, DSWS=0 still 1716B byte-identical. + +### Gate 1 dispatch #4 — FIX VALIDATED: 4c2a2b oracle CLEAN ok=32 bad=0 (2026-06-29) + +The headline cell that deadlocked at step 3 in #2 now runs the full 32-step tile and matches the fp8 oracle +exactly (32/32 frags), clean exit, no hang, no brick. + +### Gate 1 dispatch #5 — extreme case: 6c1a1b oracle CLEAN ok=48 bad=0 (2026-06-29) + +One A-feed wave cycling all 6 bands per K-step (192x64 tile) — the worst case for the old band-outer bug — +runs CLEAN, 48/48 frags. **Both cells that deadlocked (4c2a2b, 6c1a1b) now pass; 2c3a3b passed pre-fix.** +The K-outer A-feed fix is validated across the full band-count range. Pending: a 2c3a3b post-fix regression +re-check (low risk — NAFEED>=NCOMP means each wave owns <=1 band, the simplest K-outer case), then Gate 1 (T1.4) +is fully GREEN and the DSWS static 3-role substrate (Phase 1) is DONE. 5 GPU dispatches total, ZERO bricks. + +### ✅ GATE 1 (T1.4) PASSED — 2026-06-29 + +All three role-mix cells oracle-CLEAN at pool=1 single-tile down_pf, static (DYNVGPR=0): +**2c3a3b ok=16 / 4c2a2b ok=32 / 6c1a1b ok=48, all bad=0.** The DSWS static 3-role substrate (Phase 1) is DONE — +the cooperative fp8 GEMM with separate fat-compute / lean-A-feed / lean-B-feed waves is numerically correct across +the full role-mix range. 6 supervised GPU dispatches total today, ZERO bricks (every one hung-cleanly or completed; +dmesg silent each time). Net of fixes that got here: SAFEPROBE feed ti-clamps + SAFEPROBE=1 build; DIAG=1 + feed-body +marks (instrumentation was DIAG=0/uncompiled, not the old "v2 clobber"); and THE bug — A-feed band-outer/K-inner +loop starved un-fed compute bands → fixed to K-outer/band-inner. + +NEXT: Phase 2 (sensing + role slots + reservation counter), then Phase 3 (conversion/epoch-ticket + dyn-VGPR +DYNVGPR=1), then Phase 4 (adaptivity proof + tuning). NOTE the pool=64 path still hung in dispatch #1 — pool>1 +cross-WG behavior is a separate open item to revisit before any multi-WG perf run (Phase 1 gate is pool=1 by design). + +
(historical) Gate 1 pre-dispatch notes + +First display-GPU dispatch of the whole campaign. As of 2026-06-28: ZERO GPU dispatches run — all offline. + +**HARDENING PASS — DONE 2026-06-29 (all offline, zero GPU):** +1. ✅ **SAFEPROBE `ti`-clamps added to BOTH A-feed and B-feed decodes** (`occ_kernel_coop.s`), mirroring + compute lines 471-472; gated `.if SAFEPROBE` inside `.if DSWS`. The `s_min` clamp is the safety mechanism, + independent of the (possibly-flaky) diagnostic `mark`. On inspection the feeds were ALREADY covered on the + *vector* vaddr (the v8/v9/v10 clamps live in the common prologue, run by all waves) AND already retire on + `ti>=TOTAL` (terminal test) — the new clamp closes the one remaining asymmetry on the *scalar* saddr. +2. ✅ **Rebuilt all 6 bins with `SAFEPROBE=1`** — build_dsws.sh was passing it ZERO times → bins were + SAFEPROBE=0 = NO clamps compiled at all; THIS was the real lever. RGA re-gated: livereg **82**, + SCRATCH/SGPR_SPILLS/VGPR_SPILLS all **0**. DSWS=0 still byte-identical to `.clean_bins` (**1716B**). +3. ⏳ **First contact = single tile** (`DSWS_ORACLE_MTL=1 DSWS_ORACLE_NTL=1` → TOTAL=1) — dispatch-time env, + set in the gated command below. Awaiting user greenlight. + +**Brick-risk assessment (honest, 2026-06-28):** LOW-to-MODERATE. Static (`DYNVGPR=0`) **eliminates the +dyn-VGPR brick class** (#1 historical vector — no `s_alloc_vgpr`). Residual: (a) protocol-hang → teardown +wedge (200 lines of new hand-asm, never executed; `timeout 30` + 0.75s chunk-abort are the net, but +recovery-from-hang is less proven than clean completion); (b) OOB page-fault from a bad `ti` — mitigated +to ~proven-coop level by the hardening pass above. NOT claiming "can't brick." + +**Dispatch discipline:** set `ML8_COOP_CHUNK` (compositor-safe, opt-in; `ML8_COOP_CHUNK_MAXS=0.75`) + +`ML8_COOP_STREAM=1` + `timeout 30`. ONE gated dispatch at a time. Hang/brick = full STOP + report, never +auto-fire the next. Proposed first command (post-hardening, static, single-tile, down_pf): +`ML8_COOP_STREAM=1 ML8_COOP_CHUNK=8 ML8_COOP_CHUNK_MAXS=0.75 DSWS_NCOMP=4 DSWS_NAFEED=2 DSWS_NBFEED=2` +`DSWS_RINGD=2 DSWS_DYN=0 DSWS_ONLY=down_pf DSWS_ORACLE_MTL=1 DSWS_ORACLE_NTL=1 timeout 30 ./occ_dispatch --dsws` + +
+ +--- + +## Phase 2 — Sensing + role slots + reservation (offline, no GPU) — 2026-06-30 + +**Status:** Tasks 2.1, 2.2, 2.3 DONE + gated offline. Task 2.4 (supervised GPU sensor-sanity) is the next STOP-for-go dispatch. + +### T2.1 — controller LDS state (`occ_kernel_coop.s`, behind `.if DSWS`) +Appended after `CONS_A_OFF` so all proven offsets stay byte-identical: +`NCOMP_SLOT / NAFEED_SLOT / NBFEED_SLOT` (role slots), `GATE_OFF[4]` (one epoch-gate per conversion +direction: 0=comp→Afeed 1=comp→Bfeed 2=Afeed→comp 3=Bfeed→comp), `VRESV_OFF` (vgpr_reserved +envelope), `SEGCNT_OFF` (per-WG segments_processed). `LDS_TOTAL_DSWS` rolled forward; existing +`>65536 .error` now covers them. **Gate:** 6 bins assemble; RGA 0-spill, `USED_LDS=32768`, livereg +**V_fat=82**; DSWS=0 byte-identical (1716B). + +### T2.2 — CPU control-law model (already banked, re-verified) +`dsws_ctrl_model.cpp` + `test_dsws_ctrl_model.cpp` → `ALL PASS` (watermark_decision / epoch_of / +gate_try_win single-winner / reserve_grow envelope). This is the 1:1 semantic spec the Phase-3 asm transcribes. + +### T2.3 — leader state-init + read-only sensing + snapshot instrumentation +- **Reservation unit clarified:** the envelope uses the hardware **alloc** footprint, not live-peak. + Fat = `NFV` (=112 this tile, the actual `s_alloc_vgpr` target); lean = `VLEAN`=32 (the feeds' + `s_alloc_vgpr 32`). `vgpr_reserved` init = `NCOMP*NFV + (NAFEED+NBFEED)*VLEAN`. (RESULT table's + "V_fat=82" is live-peak, a different measurement; the grow-delta accounting `NFV−32=80` confirms + the envelope is in alloc units.) +- **Leader init** (extends the barrier-free INITFLAG publish, all lean-32 there): role slots ← launch + mix, gates ← 0, `vgpr_reserved` ← launch envelope, `segcnt` ← 0. Constants written by each wave's + lane-0 to WG-shared LDS (idempotent — identical values). +- **Read-only sensing (NO actuation):** + - New `mark_set off,val` macro = lane-0 **plain global_store** to `occ[off]` (vs `mark`'s atomic-MAX), + so the harness's 200 ms poll sees a sensor *oscillate* instead of latching its max. + - Compute **segment boundary** (once per tile): `occ_b = min(prod_b)−cons` → occ[32]; `occ_a = + prod_a[cid]−a_step` → occ[33] (both consumer-observed ring backlog, range [0,RINGD]/[0,RINGD_A]). + - Per-WG `segcnt` bump by lead compute (cid==0) on each segment (POOL=1: one tile = one WG segment). + - Role slots streamed once per compute wave (read LDS → occ[34/35/36]) — proves the controller-state + init round-trips through cross-wave LDS. +- **`occ_dispatch.cpp`:** DIAGFINE zero-loop extended to 14..36; new fields added to the 200 ms live + stream line (`SENS occ_b.. occ_a.. roles[../../..]`) and the TIMEOUT dump (`DSWS sensors` line). + +**DEVIATION from plan (single-writer sensor, deliberate):** the plan also names a "feed inter-frag" +sensing point. I made the **compute segment boundary the sole writer** of occ_a/occ_b. Having both the +producer (feed) and consumer (compute) write the same occ slot would race and corrupt the very +oscillation T2.4 is meant to validate. Feed-side pressure reads belong in Phase 3, where the feed +actually *actuates* on them (conversion trigger). occ_a/occ_b here are the clean consumer-side signal. + +**Offline gates (all green):** 6 cells assemble; RGA 0-spill, `USED_LDS=32768`, livereg **82**; +DSWS=0 still **1716B byte-identical**; `occ_dispatch` rebuilds clean (rc=0, 0 errors). Everything +UNCOMMITTED (commit-on-ask). **NEXT = T2.4 supervised dispatch (STOP for user go).** + +--- + +## BUG (found at T2.4, fixed offline) — multi-tile A-feed starvation — 2026-06-30 + +**Surfaced by:** the first MULTI-tile DSWS dispatch ever (T2.4 at 32 tiles, `DSWS_ORACLE_MTL=4 NTL=8`). +Gate 1 and the first T2.4 run were all **single-tile** (`MTL=NTL=1`), so the per-tile claim/feed loop +was never exercised. Single-tile = CLEAN; 32-tile = **COOP TIMEOUT** (`claim=2`, compute wedged into tile 1). + +**DIAGFINE evidence (the localization):** feed published B to step 35 (into tile 1 fine); **A-feed +`publish[31]=32`** — stuck at exactly one tile's worth (KT=32); compute `prodwait=32 consRel=33`, wedged +mid tile-1 K-loop. `roles[4/2/2]` streamed correctly (init round-trips); `occ_b/occ_a=0` only because +no compute segment ever *completed*. → compute is **A-starved**. + +**Root cause (confirmed by code, not assumption):** the A-feed published `prod_a[bnd] = k+1` using the +**per-tile** K counter `k` (s26), which **resets to 0 each tile**. The compute consumer's `a_step` +(s54) is **global/cumulative across tiles** (like the B-ring step). Tile 0: both 0..31 → match → CLEAN. +Tile ≥1: compute's `a_step` is 32→63 and waits `prod_a>32`, but the A-feed re-publishes `prod_a=1..32` +→ never exceeds 32 → permanent A-starvation deadlock. The B-feed was correct (global continuous step); +the A-feed was the lone asymmetry. **Pre-existing** in the Phase-1 K-outer A-feed rewrite — NOT caused +by the Phase-2 sensing (compute wedges before reaching the boundary sensor). + +**Fix:** introduce a **global cumulative A-step `s60` (gk)** in the A-feed, used for the ring protocol +(publish count, ring slot index, slot-free gate vs `cons_a`) — mirroring the B-feed and matching +compute's global `s54`. The per-tile `k` (s26) is kept for **A-matrix addressing only** (`k*16` within +the tile's K). `s60` inits to 0 once before `.Lafeed_loop`, increments per K-step, never resets. +(Note: first picked `s49` — caught in review that it's the `lds_get/lds_put` exec-save scratch; moved +to `s60`, 0 refs kernel-wide.) + +**Offline gates (green):** 6 bins assemble; RGA 0-spill, LDS 32768, livereg 82; DSWS=0 byte-identical +(1716B). **Needs a supervised MULTI-tile re-dispatch to verify the deadlock is gone** (this is also the +real T2.4 sensor-oscillation run: 32 segments → occ_a/occ_b vary across the 200ms stream). Uncommitted. + +--- + +### ✅ GATE T2.4 PASSED — sensors report sane + multi-tile clean — 2026-06-30 + +Four supervised dispatches (static DYN=0, pool=1, compositor-safe-chunked, ZERO bricks): +1. single-tile 4c2a2b → `oracle CLEAN ok=32 bad=0` (sensing code correct/safe; too fast to read sensors). +2. **32-tile → COOP TIMEOUT** → surfaced + root-caused the multi-tile A-feed starvation bug (above). +3. 32-tile, A-feed fix → `oracle CLEAN ok=1024 bad=0` (deadlock GONE) but occ read 0 (boundary-drained). +4. 32-tile, sensors relocated to consume points → **`oracle CLEAN ok=1024 bad=0`, `occ_b=1 occ_a=1`, + `roles[4/2/2]`** — sensors read real in-range backlog, NOT pinned. **PASS.** + +**Sensor-placement refinement (banked design note for Phase 3):** occ_a/occ_b are sampled at the per-K +CONSUME points (where the gate `prod>cons` guarantees occ ∈ [1,RINGD]), NOT at the segment boundary +(where the ring has drained → occ≈0, which would make the controller read "always starved"). The +segment boundary remains only the `segcnt` decision-clock tick. The eventual watermark LOW/HIGH must be +read against this consume-point occupancy, and likely wants a per-segment max/avg, not a single sample. + +**Phase 2 status: COMPLETE** (T2.1 LDS state, T2.2 control model, T2.3 init+sensing, T2.4 supervised +gate). Plus an out-of-band WIN: the **multi-tile substrate is now proven** (ok=1024) — Gate 1 had only +ever tested single-tile, so this de-risks all of Phase 3 (real GEMMs are multi-tile). Everything static +(DYN=0), UNCOMMITTED. Open items unchanged: pool>1 cross-WG (separate), dyn-VGPR arm (Phase 3). + +--- + +## Phase 3 — Conversion + epoch/ticket + role-agnostic terminal (offline) + +### T3.1 — epoch + lock-free gate-CAS ticket (no actuation) — 2026-06-30 +`try_gate dir, swin` macro transcribes dsws_ctrl_model.cpp `epoch_of` + `gate_try_win` EXACTLY: +`E = segcnt >> EPOCH_SHIFT`; read `g = gate[dir]`; lose if `g>=E`; else lane0 does an LDS compare-swap +`ds_cmpstore_rtn_b32 v6,v5,v6(=g),v7(=E)` and WINS iff returned-old == g. Single winner per (dir,epoch) +across all racing waves (the LDS atomic serializes). New defsyms `EPOCH_SHIFT=3`, `CTRL_LOW=1`, +`CTRL_HIGH=RINGD-1` (mirror the harness DSWS_* env). New `mark_inc off` macro = lane0 atomic-add counter. +**Unit call site** (compute boundary, ALL compute waves, NO actuation): race `try_gate 0` (compute→Afeed) +each segment, atomic-inc would-win counter `occ[39]`. occ_dispatch decodes `gateWin[39]`. + +**CAS operand-order note:** RDNA4 `ds_cmpstore_rtn_b32` is `(vaddr, vCMP, vDATA)` → `MEM=(MEM==vCMP)?vDATA:MEM` +(GCN's order was flipped). Assumed cmp=g, data=E. **Validated by:** the T3.1 micro-check (would-win must be +≤1 per epoch per dir — a wrong order gives double-winners or none) and the Gate-2 conversion-storm oracle. + +**Offline gates (green):** 6 bins assemble; RGA 0-spill, LDS 32768, livereg 83; DSWS=0 byte-identical +(1716B); occ_dispatch rebuilds rc=0. UNCOMMITTED. **Optional next:** T3.1 Step-3 supervised micro-check +(1 streamed dispatch; confirms `gateWin[39] ≈ #epochs`, NOT NCOMP×#epochs) — validates the gate-CAS before +3.2/3.3 build conversions on it. Else defer to Gate 2 (T3.5). + +### ✅ T3.1 micro-check PASSED (caught + fixed a gate-CAS operand-order bug) — 2026-06-30 +First micro-check run: `gateWin[39]=3` — ANOMALOUS (last chunk reaches only epoch E=1 once, so a correct +single-winner CAS gives exactly 1). Diagnosis: my `ds_cmpstore_rtn_b32` operand order was BACKWARDS. +Verified authoritatively OFFLINE via LLVM `cmpxchg` lowering (`clang -S` of an IR `cmpxchg ptr,%cmp,%new`): +`ds_cmpstore_rtn_b32 vdst,vaddr,vsrc0,vsrc1` is `MEM=(MEM==vsrc1)?vsrc0:MEM` -> **vsrc0=NEW, vsrc1=CMP** +(the GCN order, NOT flipped as I'd assumed). The swapped form compared against E and stored g, so gate +stayed 0 and `old==g` held for every racer -> all ~NCOMP waves "won" each epoch. FIX: swap the two source +operands (`... v7(=E,new), v6(=g,cmp)`). Re-dispatch (same config, kernel-only change): **`gateWin 3 -> 1`**, +oracle CLEAN ok=1024 bad=0, no brick. Single-winner ticket PROVEN. The gate-CAS is correct for 3.2/3.3. + +--- + +## SUBSTRATE v2 — claim-based work decomposition + split-K (2026-06-30) + +**Why v2:** Phase-3 review (3/3 consensus: kmbandy + Claude + Codex) found the proven coop substrate binds +the matrix WORK decomposition to compile-time role counts + wave identity (`rowblk=trow*P+cid`, A-band 1:1 +`cid` pairing, B-frag `owner=ni%NBFEED`). Naive role conversion therefore BRICKS (hang fires before any +wrong-output). Fix = decouple work from identity: make it CLAIMABLE by whoever holds the role. Split-K +folded in (makes B resident-per-segment → replay free → rowblk-count decouples from live nComp). New design +in `SPEC_DSWS_SUBSTRATE_V2.md`; plan in `PLAN_DSWS_SUBSTRATE_V2.md`. New kernel `occ_kernel_dsws.s` (coop +kernel NEVER touched). Config: G=6 SEGK=64 FM=2 FN=4, 8-wave WG, LDS 16640B. + +**Model/workflow:** Sonnet 5 implements; Opus + Codex review; kmbandy greenlights every GPU dispatch. + +**Phase A offline — DONE, oracle/RGA-gated (no GPU yet):** +- A1 scaffold + v2 LDS layout + harness `--dsws2` dry-print + `mk2` build. +- A2 tiered oracle `oracle_compare` (fp8_oracle.cpp) — TIGHT{5e-3,1e-2} for n_kseg=1, LOOSE{3e-2,2e-2} + for n_kseg>1 (the CPU wmma_ref chain is not bit-identical to GPU WMMA, so "exact" = the proven tight + tolerance, not bit-match). Self-test passes. +- A3–A7 datapath (claimer + resident B/A feeds + compute w/ `global_atomic_add_f32` partial-flush + + completion handshake + role-agnostic sentinel terminal). `global_atomic_add_f32` ENCODES on gfx1201. +- Host launch `run_dsws2` + tiered oracle wired. + +**Round-table caught 5 brick/correctness bugs OFFLINE (all fixed + re-gated: ASSEMBLE_OK, RGA +SGPR/VGPR_SPILLS=0, mk2 bin 4840B, harness compiles, dry-print clean):** +1. [Codex] Reset/quiesce straggler race — a compute wave increments ROWBLK_DONE then loops to claim; if + descheduled, the claimer resets ROWBLK_NEXT and the straggler claims row 0 of the next super-tile with + stale state. FIX: claimer advance-gate also drains the CLAIM counters (ROWBLK_NEXT≥G+NCOMP, + BFRAG_NEXT≥FN+NBFEED, AROW_NEXT≥G+NAFEED). [BRICK] +2. [Codex] n_kseg=1 magic-div overflow (ceil(2^32/1) truncates to 0 → t=0 all sti). FIX: shift/mask decode + (`ksi=sti&mask, t=sti>>shift`, shift=ctz(n_kseg)) — handles n_kseg=1 for free. [BRICK] +3. [Sonnet flagged→Opus→Codex corrected] kernargs s16/s17 undeliverable (PM4 only defines + COMPUTE_USER_DATA_0..15; all proven paths use 15) AND s16 doubly-used as the per-chunk terminal. Opus's + first fix was INCOMPLETE (dropped the chunk terminal) — Codex caught it. FINAL: 15 kernargs (s0..s14, + USER_SGPR=15); n_kseg derived in-kernel (KT>>NKSEG_SHIFT); chunk terminal memory-carried in occ[24]; + claimer publishes SENTINEL 0xFFFFFFFF at terminal, followers retire on it. [BRICK] +4. [Opus + Codex] C body never memset=0 (only the canary) → atomic-adds onto garbage. FIX: memset(C,0) + once before the chunk loop (not per-chunk, so split-K accumulates across chunks). [wrong-oracle] +5. [Codex] DSWS2 ignored positional mix arg + no role-floor/32-bit-overflow validation. FIXED. [correctness] + +**Phase-B carry-forward:** the quiesce sentinels use compile-time NCOMP/NAFEED/NBFEED (correct for STATIC +roles only); Phase-B conversion must switch to live role counts / epoch-snapshot drained counters. (In-code +note at `.Lclaimer_wait_done`.) + +**NEXT = A8 [SUPERVISED GPU — kmbandy greenlights], morning 2026-07-01.** First cell (n_kseg=1 TIGHT/exact, +compositor-safe): `ML8_POOL=1 ML8_COOP_CHUNK=8 ML8_COOP_CHUNK_MAXS=0.75 ML8_COOP_STREAM=1 DSWS2_NKSEG=1 +timeout 30 ./occ_dispatch --dsws2 4c2a2b` → expect ok=32 bad=0, occ[0]=0 clean, fence FIRED, no brick +(TOTAL_super=32 → 4 chunks of 8). Then n_kseg=8 LOOSE + mixes 6c1a1b/2c3a3b. Gate: static substrate +oracle-green both tiers before Phase B (conversion). All uncommitted. KG: b8c689cc (A8-ready contract), +48625333 (kernel round-table), 86e33108 (the blocker + 3/3 consensus). + +--- + +# 2026-07-03 — SUSPECT #2 FIXED (ti clamp) + CONV=1 + SPLIT-K + ADAPTIVE CONVERSION PROVEN ON SILICON + +Started the day with the CONV=1 substrate bricking (3 GPU MODE1 resets before the fix). Ended with a +**correct, adaptive, split-K-capable** wave-specialized kernel, brick-free across 15+ dispatches. + +## Root cause (the whole SUSPECT #2 saga was one symptom) +The CONV=1 hang/brick was a **racy-garbage tile-index -> out-of-buffer A/B/C scalar base -> gfxhub page +fault -> MODE1** (the COOP_STATUS.md:145 class). SAFEPROBE clamped only the per-lane vaddr (v8/v9/v10); its +own comment (occ_kernel_dsws.s:752) noted it "pairs with the future ti clamp" — which was **never +implemented**. A torn `sti` read during the claimer's per-super-tile republish decoded a garbage +`t -> mblk/tcol` and the scalar base went OOB. + +Everything the debugging chased before this — the rolling dyn-VGPR envelope, the device-scoped `gquiesce` +handshake, the bail-mark localization, the straggler/liveness hypothesis — was circling the symptom. The +envelope even ran GREEN isolated at CONV=0 (its plumbing is sound), proving it was NOT the blocker. + +## THE FIX (committed f0131142f) — 10 lines +`DECODE_STI`, under `SAFEPROBE`: `t = min(sti>>shift, TOTAL-1)` using `s11=TOTAL` (userdata, never clobbered) ++ `s36` scratch. Now every global address is provably in-buffer: t/mblk/tcol (new clamp), ksi (mask), r/f +(claim checks), v8/v9/v10 (existing). OOB unreachable by construction; a no-op for valid indices. +Crucially `s11=TOTAL=MTLsuper*NTL` is the **t-space** bound (NOT TOTAL_super), so the clamp is +**split-K-correct**, not just n_kseg=1-correct (Fable advisor confirmed). + +## GPU RESULTS (all SAFEPROBE=1, 4c2a2b unless noted, n_kseg=1 unless noted) +- CONV=0 clamped (`c62568f6`, 4872B): GREEN, ok=1536 bad=0 — no-op for valid indices, new safe baseline. +- CONV=1 clamped (`490db6e5`, 7952B): GREEN + CORRECT, **7/7** dispatches ok=1536 bad=0. Was 3x brick. +- Other mixes CONV=1: 6c1a1b 2/2, 2c3a3b 2/2 — all ok=1536 bad=0. +- **n_kseg=2 real split-K**: ok=1536 bad=0 max_rel=0 (EXACT), LOOSE tier, bounds gate A/B doubled + OK. + First split-K ever. n_kseg is a RUNTIME lever (`DSWS2_NKSEG=2`) — same .bin as n_kseg=1. + +## ADAPTIVE CONVERSION — PROVEN (the DSWS thesis, first time on silicon) +Added a DIAG conversion-commit counter (`occ[48]`, incremented in `conv_apply` at `.Lca_commit`; +byte-identical at DIAG=0) + host readout in `run_dsws2` completion (`[dsws2 CONVERSIONS] = N`). +- **Proof #1 (mechanical, DSWS2_FORCE=1 WID=4 DIR=0 EPOCH=1, EPOCH_SHIFT=0)**: forced wave4 compute->A-feed + -> counter **4** (1/chunk x4 chunks), ok=1536 bad=0. A wave switches role + re-dispatches, correct. +- **Proof #2 (adaptive, watermark-driven, ACTIVE defaults CTRL_LOW=1/HIGH_A=5/HIGH_B=3, EPOCH_SHIFT=0)**: + counter **52** — 52 NATURAL role-switches from runtime ring occupancy in one run, result EXACT. +- **GOTCHA**: `try_gate` gates on the CONTROLLER epoch `E=segcnt>>EPOCH_SHIFT`, NOT the wave's `s35`. With + default EPOCH_SHIFT=3, E only reaches 1 at super-tile 8, so conversions need EPOCH_SHIFT small (0 for + max reactivity). The prior "are waves switching?" was open because earlier runs used dormant thresholds + (CTRL_LOW=0/HIGH=6/4) OR were DIAG=0 (unobserved). + +## STATE / NEXT +- Committed: `f0131142f` (ti clamp fix), `f2cc1f26e` (gated envelope/gquiesce/bailmark scaffolding + offline + models + spec/plan). Uncommitted: the conversion-counter instrumentation (occ_kernel_dsws.s occ[48] + + occ_dispatch.cpp readout). +- Safe bins: CONV=0 clamped (4c2a2b c62568f6). occ_dispatch.cpp has ~544 lines pre-existing WIP (NOT ours + to bundle) + our ~15 lines (bail/conv readout). +- **NEXT = PERF.** The `--dsws2` path has NO throughput readout (oracle only). Wire a TF measurement into + `run_dsws2` (wall-span -> real WMMA count) for a baseline, then flip the envelope on and sweep occupancy. + The thesis (adaptivity beats the feed-bound wall) is UNPROVEN until there's a TF number. diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/RING_SLOTS_DESIGN.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/RING_SLOTS_DESIGN.md new file mode 100644 index 000000000000..ff07aa1467de --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/RING_SLOTS_DESIGN.md @@ -0,0 +1,200 @@ +# DSWS v2 — Fix #1a: D=2 double-buffered ring-of-slots pipeline + +Design spec. Kills the two measured waits (FOLLOW_WAIT 15–22%, STAGE_WAIT 26–49%) by pipelining +super-tiles instead of lockstepping one at a time. This is increment **1a** of Fix #1 (see +`PHASE_PROFILE.md` → "the 3 core fixes"). The paddle (1c) and #2/#3 come after. + +## Goal & success criteria +- **Goal:** compute waves never block on a central "next super-tile" publish, and rarely block on + feeds — because the next super-tile's operands are already staged in the other slot. +- **Success:** on `576×512×2048` (the brick-safe regime), oracle CLEAN, no brick, and the phase + profiler shows **FOLLOW_WAIT + STAGE_WAIT materially down** vs the current lockstep baseline + (4c2a2b: 16.6% + 35.5% = 52% today). WMMA/FLUSH share rises correspondingly (same absolute work). + +## Non-goals (explicitly deferred — NOT silently skipped) +- **Grow-stagger gate.** Multiple compute waves growing to 112 VGPR at once is the ISA §3.3.3.2 + deadlock (the M=1920 brick). The ring does **not** fix that — it's independent. 1a keeps the + current per-rowblk grow (proven safe at M=576) and stays at M=576. Grow-stagger is required before + training-M and is its own increment. +- **The paddle / role rebalance (1c), wave-count-from-budget (#2), feed:compute ratio (#3).** Later. +- **Split-K C-write amplification.** The long pole we address *after* the 3 fixes. + +## Approach: a NEW file, safe bin untouched +Build `occ_kernel_dsws_ring.s` (fork of `occ_kernel_dsws.s`). The current kernel and its proven +`c62568f6` bin are never edited. The ring bin is built to `occ_dsws2__gd.bin` for a run, then the +safe bin is restored (same discipline as every probe). WMMA / DECODE_STI / flush / oracle logic is +copied verbatim; only the LDS map + the claimer/feed/compute control loops change. + +## LDS map (64 KB, D=2) — decision A (raise cap 32→64 KB, keep SEGK=64) +Two slots, each with its own control words **and** its own operand buffers. Operands dominate: +``` +SLOT_STRIDE = 16 KB operands + control words, 256B-aligned + per slot s: + STI[s] super-tile id in this slot (0xFFFFFFFF = sentinel/retire) + GEN[s] monotonic refill generation (consumers detect new occupant by GEN change) + RB_NEXT[s] rowblk claim counter RB_DONE[s] rowblks computed+flushed + BF_NEXT[s] B-frag claim counter BF_DONE[s] B-frags staged + AR_NEXT[s] A-rowblk claim counter AR_DONE[s] A-rowblks staged + BRES[s] resident B operands (FN*16*SEGK = 4 KB) + ARES[s] resident A operands (G*16*FM*SEGK = 12 KB) +shared (single copy): FILL_IDX, DRAIN_IDX, INITFLAG, NCOMP/NAFEED/NBFEED slot census +total ≈ 2 × 16.4 KB + shared ≈ 33 KB (< 64 KB hardware limit; kernel cap raised to 65536) +``` +Host: `run_dsws2` allocates/declares 64 KB group segment for the ring bin (currently 16896B). One +constant to bump; guard stays `LDS ≤ 65536`. + +## Frontiers & steady state +Two moving indices into the D=2 ring: +- `FILL_IDX` — the slot the dispatcher just published and feeds are staging. +- `DRAIN_IDX` — the slot compute is consuming. + +Steady state overlaps **fill(other) with drain(current)**: +1. Compute drains `DRAIN_IDX` (claims rowblks, WMMA, flush). +2. Concurrently, dispatcher has published the next `gsti` into the other slot (`FILL_IDX`), and feeds + stage its A/B operands there. +3. When compute finishes `DRAIN_IDX` (`RB_DONE==G`) **and** `FILL_IDX` is fully staged + (`BF_DONE==FN && AR_DONE==G`): swap — compute moves to `FILL_IDX`, the drained slot is recycled. + +STAGE_WAIT is hidden whenever fill-time ≤ drain-time (feeds had all of drain(current) to prepare). +FOLLOW_WAIT collapses to just the swap check — no central publish to wait on. + +### Init & fill-sequencing (keep exactly one slot ahead) +- **Init:** `DRAIN_IDX=0`. The dispatcher fills slot 0, then slot 1, before compute drains — so once + compute starts on slot 0, slot 1 is already published (and staging). One look-ahead slot always. +- **One slot fills at a time.** All feed waves target the single `FILL_IDX`. The dispatcher advances + `FILL_IDX` to the next free slot **only after** the current fill slot is fully staged + (`BF_DONE==FN && AR_DONE==G` → READY). So feeds never split across slots; the "which slot do I + stage" question is just "read `FILL_IDX`." +- **Dispatcher invariant:** keep the non-`DRAIN_IDX` slot filled-or-filling. Concretely: whenever a + slot becomes FREE (its prior occupant's `RB_DONE==G`), the dispatcher claims the next `gsti` into it + and sets `FILL_IDX` to it. With D=2 this strictly alternates. The dispatcher's own waits (for a slot + to free, for staging to finish) are **off the compute critical path** — compute is draining the + other slot throughout. + +## Protocol refinements (derived while reading the substrate — 2026-07-04) +Two decisions that refine the pseudocode below, both discovered by mapping the spec onto the real +single-slot loops. They *simplify* correctness; record them so the barrier-free argument is reviewable. + +1. **`GEN` is a GLOBAL-monotonic publish counter, not per-slot.** The dispatcher keeps one running + counter (the old `EPOCH` role), `++` per publish, and writes it to the published slot's `SL_GEN` + **last** (release fence). Every follower tracks a single last-seen gen (`s35`, exactly as the old + `_follow` tracked epoch) and gates "new occupant in this slot" on `SL_GEN[idx] > s35`. A slow feed + that skips a whole generation (reads a *newer* FILL_IDX/gen than the one it last staged) is + **safe**: the slot it skipped was already fully staged by the other feed(s) before the dispatcher + advanced FILL_IDX off it (the dispatcher's `wait READY` gate guarantees this), and compute gates on + the DONE *counters*, not on which feed contributed. Old `SL_GEN` of any slot is always `< s35` for a + caught-up wave, so a stale read just spins — never double-stages. +2. **The per-super-tile QUIESCE over-claim handshake is DROPPED.** In the single-slot kernel the + claimer waited `*_NEXT >= depth + #role-waves` before resetting, so a descheduled straggler's next + `fetch_add` couldn't land on index 0 of the *next* super-tile against stale state. The ring makes + that race **structurally impossible**: a slot's counters reset only at FREE→FILLED, only by the + dispatcher, only after `SL_RBDONE >= G`; and compute cannot bump `SL_RBDONE` until + `SL_BFDONE==FN && SL_ARDONE==G`. So every in-flight feed claim on a slot is accounted (its DONE + increment) *before* that slot can be recycled — no straggler is ever mid-claim at reset. The DONE + counters alone gate recycling. (Feeds still over-claim `*_NEXT >= depth` to know when to STOP + claiming a slot; that is within-occupant and reset-safe.) Net: no `QUIESCE_CNT`, no + `*_NEXT >= depth+waves` waits, and the `NCOMP/NAFEED/NBFEED` LDS census is unused at CONV=0. + +Consequence for wave roles: `wid0` is the pure **dispatcher** (no BSTAGE — deviates from the old +claimer, which was B-feed-class). Effective B-stager count is `NBFEED-1`; pick the launch mix +accordingly (the exact feed:compute ratio is fix #3, not 1a). GEN starts at 1 for the first real +publish (`SL_GEN==0` marks a never-published slot → the dispatcher's initial fill). + +## Counter protocol (the barrier-free correctness — this is what bricks if wrong) +Each **slot owns its counters**; concurrent super-tiles never share one → the reset-race that forced +the original WG-wide quiesce cannot occur. Invariants: + +**Dispatcher (wid0), per free slot:** +``` +when slot s is FREE (RB_DONE[s]==G for its prior occupant, or s never used): + gsti = atomic_inc(occ[20]); if gsti >= chunkHi: publish STI[s]=SENTINEL, bump GEN[s], mark terminal + else: + reset RB_NEXT[s]=BF_NEXT[s]=AR_NEXT[s]=0 ; RB_DONE[s]=BF_DONE[s]=AR_DONE[s]=0 + STI[s] = gsti + GEN[s] += 1 // publish LAST: the GEN bump is the release fence for feeds+compute + FILL_IDX = s +``` +Reset happens **only** here, **only** by the dispatcher, **after** the prior occupant's `RB_DONE==G` +is observed. `GEN[s]` bumped last = single release point; feeds/compute gate on GEN, so they never see +half-reset state. + +**Feed wave (A or B), staging FILL_IDX:** +``` +loop: + wait GEN[FILL_IDX] advances (new occupant) // no central epoch; per-slot gen + sti = STI[FILL_IDX]; if sentinel: retire + decode; claim frags/rowblks from BF_NEXT/AR_NEXT[FILL_IDX]; stage into BRES/ARES[FILL_IDX] + on stage of each unit: BF_DONE/AR_DONE[FILL_IDX]++ // compute gates on DONE, not NEXT + terminal over-claim bounds the claim (unchanged threshold logic, now per-slot) +``` + +**Compute wave, draining DRAIN_IDX:** +``` +loop: + wait GEN[DRAIN_IDX] advances // slot pre-filled -> minimal FOLLOW_WAIT + sti = STI[DRAIN_IDX]; if sentinel: retire + decode; wait BF_DONE[DRAIN_IDX]==FN && AR_DONE[DRAIN_IDX]==G // minimal STAGE_WAIT if pre-staged + claim rowblks RB_NEXT[DRAIN_IDX]; grow; WMMA(BRES/ARES[DRAIN_IDX]); flush C; shrink; RB_DONE[DRAIN_IDX]++ + the compute wave whose RB_DONE++ makes it hit G advances the frontier: + if other slot READY (its BF_DONE==FN && AR_DONE==G): DRAIN_IDX = other; else spin until READY +``` +Frontier advance is done by exactly one wave (the one that closes the slot), lock-free, detected by +`RB_DONE==G`. The just-drained slot's `RB_DONE==G` is the dispatcher's signal to recycle it. + +## Correctness argument +- **No reset-race:** slot counters reset only at FREE→FILLED, only by the dispatcher, only after the + prior occupant's `RB_DONE==G`. A straggler still finishing occupant N of slot s cannot touch slot + s's next occupant's counters (they're the same slot but the reset waits for the straggler's DONE) — + and cannot touch the *other* slot's counters (separate memory). The original cross-super-tile + collision is structurally impossible. +- **No new deadlock:** the only wait cycles are (a) compute waits GEN (dispatcher bumps it — always + makes progress while work remains), (b) compute waits FILL ready (feeds always progress), (c) + dispatcher waits prior `RB_DONE==G` (compute always progresses). No cycle where each waits on the + other indefinitely while work remains. Termination: sentinel STI propagates through GEN to all + waves → retire. (The pre-existing multi-grow deadlock is unchanged and out of scope for 1a; M=576.) +- **dyn-VGPR:** unchanged per-rowblk grow/shrink (the profile says it's ~1%, safe at M=576). + +## What stays identical (copied verbatim) +DECODE_STI, BSTAGE/ASTAGE inner staging math, the WMMA loop, the C flush (split-K +`global_atomic_add_f32`), SAFEPROBE clamps, TFPROBE + PHASEPROBE instrumentation (ported so the +scoreboard works), the oracle. + +## Host changes (`run_dsws2`) +- Group-segment size 16896 → 65536 for the ring bin (env or bin-detected). Keep `LDS ≤ 65536` guard. +- Nothing else: same kernarg contract, same occ[20]/occ[24] claim bounds, same chunk/rep loop, same + phase-accumulator readout. + +## Test plan (scoreboard-driven, brick-safe) +1. Assemble; verify LDS ≤ 64 KB; 0 spill. +2. Run `576×512×2048`, POOL=16, single pass, streamed, `chunkMaxS` short. Gate: occ0==0, fence fired, + **oracle CLEAN**, dmesg delta 0. Restore safe bin. +3. PHASEPROBE run, same shape → compare FOLLOW_WAIT+STAGE_WAIT vs the current-lockstep baseline row in + `PHASE_PROFILE.md`. Success = the two waits materially down, oracle clean. +4. Only after clean + faster: consider D=3 (LDS permitting) and the next increments. + +## Build status (2026-07-04) — Stages 1–5 assembled clean, awaiting greenlit GPU test +Implemented in `occ_kernel_dsws_ring.s` (fork; safe bins untouched) + host `occ_dispatch.cpp`: +- **Stage 1 (foundation):** RING_D=2 layout (FILL_IDX/DRAIN_IDX/RINGINIT at 0/4/8; per-slot control + block SLOTC_BASE=32 stride 32 with SL_STI/GEN/RBNEXT/RBDONE/BFNEXT/BFDONE/ARNEXT/ARDONE; operands + OP_BASE=256 OPSTRIDE=16384, BRES_ROFF=0 ARES_ROFF=BRES_BYTES; LDS_TOTAL_RING=33024<65536). Added + ungated `lds_fetch_add_r`/`lds_inc_r`/`lds_put_r`. Descriptor group-seg 32768→65536. Foundation + verified byte-identical to pristine `.text`. +- **Stage 2 (dispatcher, was claimer):** regs s36=gen (global-monotonic), s37=fill_slot (alternates), + s34=fill_slot ctrl base, s17=sti, s69=chunkHi. Inits both slots + frontier, RINGINIT last; fill-free + loop; publish order STI→FILL_IDX→SL_GEN(last); wait-READY before alternating; terminal sentinels + fill_slot only + points FILL_IDX at it. +- **Stage 3 (feeds):** ring `BSTAGE_R`/`ASTAGE_R scb,sob`; loops gate on SL_GEN[FILL_IDX]>s35, read STI + (sentinel→retire), stage into slot ops. Feed regs: s35=gen, s38=FILL_IDX, s48=scb, s52=sob. +- **Stage 4 (compute):** gate on SL_GEN[DRAIN_IDX]>s35, wait SL_BFDONE==FN && SL_ARDONE==G, claim + SL_RBNEXT, grow(NFV), WMMA on v9+sob(+ARES_ROFF), flush C, shrink, `lds_fetch_add_r SL_RBDONE` + (old==G-1 ⇒ closer ⇒ `lds_put DRAIN_IDX = ^1`). Compute regs: s35=gen, s46=DRAIN_IDX, s48=scb, + s52=sob; PHASEPROBE ported. Assembles PHASEPROBE 0 & 1, **0 spill**, no dangling labels. +- **Stage 5 (host):** `DSWS2_RING=1` env ⇒ ldsBytes `256+2*16384=33024` + bin `occ_dsws2__ring_gd.bin` + (built by `build_ring.sh`). RINGD default 1 leaves the single-slot path byte-identical. + +**Next (Stage 5 finish): the greenlit GPU test — NOT yet run.** One dispatch, streamed, at the +brick-safe shape; oracle CLEAN + dmesg-delta-0 gate; then a PHASEPROBE=1 delta vs the 4c2a2b baseline. + +## Rollout / safety +- New file; current kernel + `c62568f6` bin untouched. Every ring dispatch is kmbandy-greenlit, one at + a time, streamed, safe-bin restored after. Never `--gl2c`. Stay at M=576 (no grow-stagger yet). diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/SPEC_DSWS_CONTROLLER.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/SPEC_DSWS_CONTROLLER.md new file mode 100644 index 000000000000..de2d332431a1 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/SPEC_DSWS_CONTROLLER.md @@ -0,0 +1,217 @@ +# Spec: DSWS Adaptive Wave-Role Controller (MAD-305) + +Date: 2026-06-27. Substrate: `occ_kernel_coop.s` (dyn-VGPR grow/shrink + split-K + +POOLTERM cooperative kernel, raw-PM4 dispatch on gfx1201/RDNA4 wave32). +Companions: [[MAD305_DSWS_MASTER.md]] (campaign master), [[SPEC_WAVESPEC.md]] +(static wave-spec lineage). Prior-art verdict (Murmur `5ec8a958`, KG `2dc2332d`): +the runtime-adaptive rebalancing is **novel**; the lean/fat mechanism is borrowed. + +## Goal + +One fp8 GEMM kernel that **senses its in-kernel bottleneck and rebalances its mix +of wave roles at runtime** to attack the measured wall (the VALU **issue port**: +~31 non-WMMA issues per 32 WMMA = 52% of the 307 TF ceiling). The lever is +wave-specialization — put feed/address instructions on *separate* waves so compute +waves issue near-pure WMMA — with the feed:compute split **found by the kernel per +shape, per moment** instead of hand-tuned and frozen at launch. + +This spec covers the **controller** (the novel "brain"). The static 3-role +substrate it sits on is a prerequisite build phase (Phase 1 below). + +## Scope decisions (settled in brainstorming, 2026-06-27) + +- **3 roles** from the start: `compute` (fat VGPR, holds fp32 accumulators), + `A-feed` (lean), `B-feed` (lean). Rationale: tests the actual novel thesis (the + multi-role economy), and on the `coop` substrate **both** A-feed and B-feed + relieve the issue-port wall — coop currently has compute load its *own* A direct, + so an A-feed role offloads those A-load issues off the compute stream. The + control-law code is role-count-parametric, so 3-role-first avoids throwaway. +- **Data path fixed for v1** (no runtime A-in-LDS-vs-direct switching). That is a + legitimate *future* second control axis (slow/structural cadence, richer sensing) + but it makes compute's hot loop branchy, which fights the issue-port wall. +- **Feed floor = 1; compute floor = 1.** Feeds can be driven down to their last + wave (throw everyone else at the bottleneck) but not to 0 — floor-0 would force a + dual-path (fed-vs-direct) branch into compute's K-loop, adding non-WMMA issues on + the exact waves we're trying to keep pure. Floor-0 is bundled with the future + data-path-switch extension. +- **Build sequencing:** static 3-role substrate oracle-green **first**, then layer + the controller. Do not couple "new A-feed role" and "new control law" in one step. + +## Architecture — the role economy + +A workgroup launches a **fixed N waves**. Each wave is in exactly one role at any +instant. The controller governs only the **partition** +`(n_compute, n_Afeed, n_Bfeed)` with `n_compute + n_Afeed + n_Bfeed = N`. The wave +count never changes; only the partition moves. + +State = three atomic LDS **role-count slots** (`n_compute`, `n_Afeed`, `n_Bfeed`) — +the single source of truth for the current mix. A conversion is one wave atomically +moving itself between slots, then physically re-roling (grow/shrink + switch job +loop), **only at a split-K segment boundary** (the one point a wave is safe to +change: partial flushed, accumulators released). + +Key property: **the controller moves the *target partition*; waves migrate +themselves toward it at their next safe boundary.** This is what keeps it lock-free +and barrier-free — no wave waits on another, no rendezvous. + +Invariants (always hold): +- Floors: `n_compute ≥ 1`, `n_Afeed ≥ 1`, `n_Bfeed ≥ 1`. +- Sum-envelope: `Σ instantaneous VGPR alloc < per-SIMD budget`, enforced at + grow-time (below), never by the controller's arithmetic. + +## Sensing — ring pressure + +Each ring (A-ring, B-ring; depth `RINGD`) is a supply/demand **pressure gauge**. +At its decision boundary a wave reads occupancy from LDS: + + occ_X = prod_count_X − min_cons_X (clamped to [0, RINGD]) + +Watermark bands give hysteresis (the **spatial** damping): +- `occ_X < LOW` → ring draining empty → **starved** for X. +- `occ_X > HIGH` → ring backing up full → **over-served** for X. +- `LOW ≤ occ_X ≤ HIGH` → dead-zone, no action. + +Both conversion directions read the *same* gauge from opposite sides: +- a **compute** wave (consumes from ring X) acts on `occ_X < LOW`; +- a **feed-X** wave (fills ring X) acts on its own `occ_X > HIGH`. + +## Control law — conversion, both directions + +**compute → feed-X** (compute is starved for X): +- Boundary: split-K **segment** boundary (compute holds accumulator state). +- Guard: `occ_X < LOW`, win the epoch ticket for this direction (below), + `n_compute > 1` (floor). +- Actuation: CAS-decrement `n_compute`, increment `n_feedX`, `s_alloc_vgpr` **SHRINK** + to lean (a shrink *always* succeeds), enter the feed loop. + +**feed-X → compute** (feed-X is over-serving): +- Boundary: any inter-frag point (a feed wave holds no accumulator state → far more + frequent safe boundaries), gated to the epoch. +- Guard: `occ_X > HIGH`, win the epoch ticket, `n_feedX > 1` (floor). +- Actuation: CAS-decrement `n_feedX`, increment `n_compute`, `s_alloc_vgpr` **GROW** + to fat — *gated on the sum-envelope* (below). On grow-fail: **abort cleanly** + (undo the slot move + reservation, stay feed this epoch). On success: enter the + compute loop (claim a K-segment, accumulate). + +Designed asymmetry (the good kind): the cheap-to-move role (lean feeds) reacts +**fast**; the expensive-to-move role (fat compute) reacts at **segment** granularity. +Shrink-always-succeeds / grow-can-abort means budget is *freed* promptly but +*consumed* only when proven-available. + +## Epoch + ticket — lock-free single-winner per window + +**Epoch** = a clock derived from work progress (per-workgroup): + + E = (segments_processed >> EPOCH_SHIFT) + +ticks every `2^EPOCH_SHIFT` segments. `EPOCH_SHIFT` is the **decision-cadence knob** +(small = reactive, large = damped). No one "advances" it; it's a function of +throughput. + +**Ticket** = one LDS `gate[d]` per conversion direction `d` (4 directions: +compute→Afeed, compute→Bfeed, Afeed→compute, Bfeed→compute), each holding the last +epoch in which `d` fired. To convert in direction `d` at epoch `E`: + + g = gate[d] + if g >= E: back off # d already fired this epoch + else: won = (atomic_cmpswap(gate[d], g, E) == g) + if won: proceed else: back off + +Exactly one wave's CAS succeeds when many race the same `g < E`. **≤1 conversion per +direction per `2^EPOCH_SHIFT` segments**, per workgroup — the **temporal** damping. + +This one CAS does triple duty: (1) anti-thrash, (2) the **phase-stagger** the +rolling-dyn-VGPR thesis needs — grows are automatically spread across epochs, never +simultaneous — and (3) keeps the sum-envelope satisfiable by construction (you can +never get N waves spiking VGPR at once). + +Combined damping: a feeder can't flip back to compute until **both** its ring exceeds +HIGH (spatial) **and** a fresh epoch ticket is available (temporal). Tuning surface = +`{LOW, HIGH, RINGD, EPOCH_SHIFT}` — small and interpretable. + +Gates + role slots + epoch clock are all **per-workgroup** (in each WG's LDS); every +WG self-balances independently with zero cross-WG coordination. + +## Safety & failure modes + +- **Sum-envelope, concretely:** one LDS counter `vgpr_reserved`. A feed→compute grow: + `r = atomic_add(vgpr_reserved, V_fat − V_lean)`; if `r + Δ > BUDGET` → + `atomic_sub` back and **abort**. compute→feed shrink: `atomic_sub` (always + succeeds). This reservation correctly arbitrates the ≤2 concurrent grows an epoch + permits (the atomic serializes them; the second to validate sees the first's + reservation and backs off). +- **Role floors:** CAS-guarded decrements (compute→feed needs `n_compute > 1`; + feed→compute needs `n_feedX > 1`). +- **No barrier, by construction:** pure LDS atomics + busy-wait flags (inherits + coop's `BUSYWAIT`); conversions add zero rendezvous → the dyn-VGPR / §3.3.3.2 + barrier deadlock is structurally absent. +- **POOLTERM terminal must be role-agnostic** (the one genuinely new requirement): + every role's loop checks the `ti ≥ TOTAL` terminal broadcast at the *same* boundary + it checks for conversion, so no wave is stranded in a role past drain, and a wave + that converts then sees the terminal exits immediately. +- **No livelock:** a full envelope just makes feed→compute grows keep aborting — the + wave stays feeding (productive); the mix holds at a safe, correct partition. No + state spins doing nothing. + +All failure modes degrade to "stay in current role, keep doing useful work" — never +to a brick, never to a stall. + +## Testing & verification + +- **CPU oracle = correctness ground truth, under ANY schedule.** DSWS is a pure + *performance* transform — the adaptive mix must never change the math. Gate: for + any (shape, mix, conversion schedule), stored C matches the fp8 e4m3 oracle. + Split-K flushes fp32 partials via atomic-add (nondeterministic ordering) → gate + with an fp32-reorder **tolerance** or a deterministic-accumulation gate mode + (carry coop's existing choice — resolve in planning). +- **Two-gate sequencing:** + - *Gate 1 — static 3-role green:* fixed `(n_compute, n_Afeed, n_Bfeed)`, no + controller, oracle-clean across shapes + several hand-set mixes. + - *Gate 2 — dynamic green:* same oracle, conversions firing. +- **Conversion-storm race stress:** oracle + many repeats under adversarial tuning + (tight watermarks + tiny `EPOCH_SHIFT` → maximal conversion rate) to expose + cross-wave races the strong oracle catches and `acc00` would miss. +- **Prove it adapts (not just runs):** instrument role-count slots + conversion + counters into the occ-snapshot stream. (a) asymmetric shape → mix moves the right + way and settles; (b) **start from a deliberately wrong mix** (e.g. all-compute on a + feed-bound shape) → controller converges to a better partition **and TF climbs**. +- **Success metric (v1)** on ml8 `down` (M=2048 K=9216 N=2560) / `down_pf` (M=512): + (a) oracle-correct, (b) converges to a mix that beats the static 3-role baseline + AND the 165 TF static winner, (c) demonstrably adapts when shape changes — + confirmed with `--att` showing **cut non-WMMA issues on the compute waves**. +- **Supervised GPU discipline:** display GPU = brick risk. Compositor-safe chunking + + per-chunk hang-abort; oracle (STORE=1) before perf (STORE=0); **one gated dispatch + at a time, no sweeps until proven safe**; stream to disk for brick forensics; user + greenlights each dispatch. Never `--gl2c`; SAFEPROBE + bounds gate stay on. + +## Build phases (high level; detailed plan is the next step) + +1. **Static 3-role substrate** on coop: add the A-feed role (A-ring in LDS, A-feed + waves) — *port* the proven A-LDS-share pattern from `occ_kernel_wggemm2.s` rather + than invent it. Fixed mix. **Gate 1** oracle-green. +2. **Sensing + role slots:** ring-occupancy reads, watermark bands, the LDS + role-count slots + `vgpr_reserved` reservation counter (no conversions yet — + read-only sensing + a STATIC mix still). Verify the sensors report sane occupancy. +3. **Conversion + epoch/ticket:** the CAS gate, both-direction conversion with floor + + envelope guards, role-agnostic POOLTERM terminal. **Gate 2** oracle-green + + conversion-storm stress. +4. **Adaptivity + tuning:** instrument role counts; converge-from-wrong-start proof; + sweep `{LOW, HIGH, RINGD, EPOCH_SHIFT}`; measure on ml8 `down`/`down_pf` with + `--att`. RESULT doc + KG bank. + +## Open details to resolve in planning + +- Oracle accumulation determinism vs tolerance (carry coop's choice). +- Exact source/owner of the per-WG `segments_processed` counter feeding `E`. +- A-feed ring sizing / LDS budget at the chosen tile (must fit A-ring + B-ring + + counters within 64 KB; confirm not occupancy-binding since we're not occupancy-maxxing). +- Fat/lean VGPR footprints `V_fat`, `V_lean` and the per-SIMD `BUDGET` constant + (from RGA on the static substrate). +- Feed wave's exact inter-frag check cadence (every frag vs every K-window). + +## Pointers + +- KG: `2dc2332d` (prior-art verdict), `dac0bb8c`/`0a2cea44` (POOLTERM), `5fedf098` + (rolling-peak origin), `dc9faf0d`/`63583120` (DSWS v1/v2), `8a9ce97f`/`17f209af` + (dyn-VGPR OOR-temp deadlock fix — the barrier-free precedent). +- Master: `MAD305_DSWS_MASTER.md` §5–§6. Jira epic MAD-305. Tasks #323/#324. diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/SPEC_DSWS_PHASEB_CONVERSION.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/SPEC_DSWS_PHASEB_CONVERSION.md new file mode 100644 index 000000000000..fafb6a271789 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/SPEC_DSWS_PHASEB_CONVERSION.md @@ -0,0 +1,222 @@ +# DSWS Substrate v2 — Phase B: Runtime Role Conversion (design) + +**Status:** approved design (2026-07-01). Realizes *step 2* of the build sequence in +`SPEC_DSWS_SUBSTRATE_V2.md` §6 ("Add conversion"). Phase A (static claim-based split-K +substrate) is GPU-proven green: all 3 role mixes × 2 split-K tiers oracle-CLEAN, bit-exact, +zero bricks (KG `0c5537e6`, 2026-07-01). This spec covers only the conversion actuation added +on top of that green substrate. + +**Goal (one sentence):** Let a workgroup move waves between {compute / A-feed / B-feed} at +runtime — the DSWS payload — on the claim-based v2 substrate, without orphaning output, jamming +a feed, or bricking, so the partition `(nComp, nAfeed, nBfeed)` self-tunes to the bottleneck. + +## Relationship to prior work + +- The **conversion control law** already exists and is unit-tested in `dsws_ctrl_model.cpp` + (`watermark_decision`, `epoch_of`, `gate_try_win`, `reserve_grow`) and is transcribed + exactly by the `try_gate` macro + reservation logic in `occ_kernel_coop.s`. Phase B **ports** + that proven actuation into `occ_kernel_dsws.s` and binds it to the claim-based role branch — + it does not re-derive it. +- The blocker that killed the *coop* substrate's conversion (work bound to wave identity; + KG `86e33108`) is structurally absent in v2: a converted wave inherits **no work item**, it + just changes its role tag and starts claiming from the dest role's counters. This spec relies + on that property and does not re-litigate it. + +## The two locked design decisions + +### Decision 1 — Quiesce model: per-epoch snapshot (+ `N−1` DIAG assert) + +The claimer's `.Lclaimer_wait_done` quiesce currently waits on **compile-time** +`NCOMP/NAFEED/NBFEED` (`ROWBLK_NEXT ≥ G+NCOMP`, `BFRAG_NEXT ≥ FN+NBFEED`, +`AROW_NEXT ≥ G+NAFEED`). Correct for static roles only. Once conversions move the partition, +those constants are wrong → the claimer advances early (straggler races the counter reset → +brick / stale-resident wrong-oracle) or waits forever (deadlock). + +**Fix:** the claimer snapshots the *live* role counts `(nComp, nAfeed, nBfeed)` at super-tile +broadcast into per-epoch LDS slots; the quiesce sentinels read the snapshot, not the constants: +`ROWBLK_NEXT ≥ G + snap.nComp`, `BFRAG_NEXT ≥ FN + snap.nBfeed`, `AROW_NEXT ≥ G + snap.nAfeed`. + +**Why this model** (vs. the two rejected alternatives): smallest diff from the GPU-proven +Phase-A quiesce (swap three constants for three LDS reads — tiny bisect surface); preserves the +per-operand tripwire (a jammed single role's counter is still caught *at* the quiesce, not only +downstream at the oracle); and maps 1:1 onto `dsws_ctrl_model.cpp` so the snapshot/quiesce +interaction is CPU-unit-testable offline before any GPU dispatch. +- Rejected: **role-agnostic `N−1` counter alone** — more robustly partition-independent, but + changes more proven code and blinds the per-operand tripwire. +- Rejected: **live per-role drained counters w/ mid-super-tile conversion** — reopens the exact + cross-wave ordering hazards v2 was built to avoid; reactivity payoff is moot given split-K's + already-short super-tiles. Deferred to Phase 4 only if measured need appears. + +**Safety net (the rejected alternative, kept as an assert):** the `N−1` counter is compiled in +as `QUIESCE_CNT` and, under `DIAG`, cross-checked against `Σ snapshot sentinels`. Because wave +count is fixed and `wid 0` (claimer) never converts, exactly `N−1` non-claimer waves are alive +each super-tile and each does exactly one terminal bail — so `QUIESCE_CNT == N−1` must coincide +with the three snapshot sentinels being satisfied. A disagreement is an ordering bug, caught +immediately (offline model + supervised runs). + +### Decision 2 — Actuation mechanism: bail-time commit (Approach A) + +A converting wave **decides** during the super-tile but **commits** at its terminal bail, +sequenced immediately *before* it increments `QUIESCE_CNT`. + +**Why this composes with Decision 1 for free:** the claimer's quiesce already waits for all +`N−1` bails. If the role-slot CAS (the commit) is ordered *before* the bail-count bump, then +"quiesce satisfied" *implies* "all conversions for this boundary have landed" — the quiesce +counter **is** the publish/snapshot handshake. The claimer therefore snapshots the E+1 +partition only after every commit is visible. No extra barrier, no second ordering, no +intent-buffer. (Rejected: claimer-mediated commit — splits the commit across two waves and adds +a claimer→wave-resize ordering, since `s_alloc_vgpr` must run on the converting wave anyway. +Rejected: immediate mid-tile commit + pending-counter — reintroduces mid-super-tile population +mutation = Option-3 hazard.) + +## Global constraints (inherited verbatim from `SPEC_DSWS_SUBSTRATE_V2.md`) + +- A GPU brick is a **BUG**, never a tax. A hang = full STOP + report; never auto-fire the next + variant. **The user greenlights EVERY GPU dispatch individually.** +- Display GPU → only compositor-safe chunked sub-second dispatches + (`ML8_POOL=1 ML8_COOP_CHUNK=8 ML8_COOP_CHUNK_MAXS=0.75 ML8_COOP_STREAM=1`, `timeout 30`). +- **NEVER `--gl2c`.** SAFEPROBE + bounds gate + padding stay ON. `ML8_COOP_STREAM=1` always. +- **No `s_barrier`** (mixed dyn-VGPR + `s_barrier` hard-deadlocks — proven). Pure LDS-atomic + busy-wait coordination only. +- `occ_kernel_coop.s` is **never modified** — known-good reference. All work is additive in + `occ_kernel_dsws.s`. +- Commit to git only when the user asks. Single-variable isolation; fix bugs, don't dodge them. + +## Section 1 — New LDS state + +**Symbols (inherited from `SPEC_DSWS_SUBSTRATE_V2.md` / the coop layer):** `N` = total waves/WG +(fixed at launch); `G=6` cooperative M-extent (= `nComp` ceiling); `FN=4`/`FM=2` N/M frag +counts; `NFV` = fat-compute VGPR target (grow size); `VLEAN=32` = lean feed VGPR; `BUDGET` = +per-SIMD VGPR budget the sum-envelope must not exceed; `RINGD` = ring depth; `SEGK=64`, +`n_kseg = KT/(SEGK/16)`. + +Ported from the coop actuation layer (same relative layout), plus two Phase-B additions: +- `GATE_OFF[4]` — `gate[dir]` = last epoch direction `dir` fired (the `try_gate` CAS target). +- `VRESV_OFF` — `vgpr_reserved` sum-envelope counter. Init = `nComp·NFV + (nAfeed+nBfeed)·VLEAN`. +- `SEGCNT_OFF` — per-WG decision clock source (`epoch = segcnt >> EPOCH_SHIFT`). +- Thresholds `CTRL_LOW`, `CTRL_HIGH = RINGD−1`, `EPOCH_SHIFT` (mirror `occ_dispatch` env). +- **NEW — `SNAP_NC/NA/NB`**, double-buffered by epoch parity (`[E&1]`) so the snapshot being + read for `quiesce(E)` is never clobbered by the claimer staging `E+1`. +- **NEW — `QUIESCE_CNT`** — role-agnostic `N−1` bail counter (Decision 1 safety net + advance + gate). + +The role-count slots `NCOMP_SLOT/NAFEED_SLOT/NBFEED_SLOT` and `VRESV_OFF` already exist in +`occ_kernel_dsws.s` (published by the claimer today); Phase B makes them **mutable** (CAS'd on +conversion) instead of write-once. LDS budget delta is a handful of u32 slots — re-assert the +32 KB group-segment fit (Phase A used 16640 B of 16896 B alloc; ample margin). + +## Section 2 — Sensing (faithful port) + +`occ_X` = ring producer minus consumer at the **consume point**: +`occ_A = prod_a − min(cons_a)`, `occ_B` analogously from the B-frag claim counters. Sampled +where the value is consumed, **not** at the segment boundary — the coop lesson (KG `0a3135b2`): +at the boundary the ring is drained so `occ ≈ 0` and the controller would read "always +starved." Fed to `watermark_decision(occ, CTRL_LOW, CTRL_HIGH)`: +`occ_X < LOW` → compute starved for X → shrink compute → feed-X; `occ_X > HIGH` → feed-X +over-serving → grow feed-X → compute. Read-only; no actuation here. + +## Section 3 — Conversion lifecycle (Approach A, per non-claimer wave) + +At each **kseg boundary** (= super-tile boundary; roles are frozen *within* a super-tile): + +1. **Sense** (§2) → `watermark_decision` → candidate direction `dir` (or none). +2. **Win ticket** — `try_gate(dir)`: `E = segcnt >> EPOCH_SHIFT`; win iff `gate[dir] < E` via + the single-winner LDS CAS (`ds_cmpstore_rtn_b32`, operand order per KG `9ed04f3c`: + `vsrc0=new=E`, `vsrc1=cmp=g`). ≤1 winner per `(dir, epoch)`. Non-winners continue unchanged. +3. **Work to terminal bail** in the current role (unchanged claim loop). +4. **Commit — ordered strictly before the `QUIESCE_CNT` bump:** + a. **Floor guard** — CAS-dec the source role slot only if `> 1` (compute floor ≥ 1, feed + floor ≥ 1). Fail → abort conversion, bail as current role. + b. **Reservation envelope** — `compute→feed` shrink: `atomic_sub(vgpr_reserved, NFV−VLEAN)` + (always succeeds). `feed→compute` grow: `atomic_add(vgpr_reserved, NFV−VLEAN)` then + validate `≤ BUDGET`; if over, `atomic_sub` + abort (stay in role this epoch). + c. **CAS role slots** — dec source, inc dest (bounded to ≤2 concurrent writers/boundary by + the ticket; plain atomic-LDS). + d. **Flip private role register** + `s_alloc_vgpr` GROW(`NFV`) / SHRINK(32), each SCC-retry + guarded. +5. **Bump `QUIESCE_CNT`** (every non-claimer wave does this exactly once/super-tile, converted + or not), then enter the dest role's claim loop (or re-enter own). + +**Claimer (pinned `wid 0`, never converts):** on `QUIESCE_CNT == N−1` → all commits landed → +reset per-super-tile counters, **snapshot** live `(nComp,nAfeed,nBfeed)` into the `[E+1 & 1]` +slots, clear `QUIESCE_CNT`, bump epoch (proven `TI_OFF`-before-`EPOCH_OFF` ordering). Terminal: +`sti ≥ TOTAL_super` remains the role-agnostic retire signal every role checks at its boundary. + +## Section 4 — Safety / anti-brick invariants + +- **Ordering contract (the crux):** commit (4c CAS) precedes the `QUIESCE_CNT` bump (step 5), + which precedes the claimer's E+1 snapshot. Therefore the claimer never snapshots a stale + partition. Self-enforced by the bail sequence; no barrier. +- **Floors** `nComp, nAfeed, nBfeed ≥ 1`; **ceiling** `nComp ≤ G` (asserted in the control + model — surplus compute waves would find `rowblk_next` exhausted and idle, but floors+ceiling + keep `nComp ≤ G` by construction). +- **`s_alloc_vgpr` OOR-poison guard (highest brick-risk item):** on RDNA4, any LDS/atomic temp + register reachable *before* a grow must be v14/v15 under dyn — a `>v15` source pre-grow is + poison (coop learned this; `occ_kernel_coop.s` gates every pre-grow-reachable temp to + v14/v15). Every new pre-grow-reachable temp in the conversion path (sense, ticket, floor, + envelope) inherits this constraint. Verified by RGA + the sense/ticket temps living in the + lean-safe register window. **This is the single most likely place to brick — reviewed + explicitly by the round table before the first dynamic-mix GPU dispatch.** +- **No `s_barrier`.** All coordination is LDS-atomic busy-wait. +- **Clock never converts:** `SEGCNT` is bumped by the pinned claimer (`wid 0`), which is + non-convertible — kills the clock-stall failure mode. + +## Section 5 — Control model changes (offline, TDD, no GPU) + +Extend `dsws_ctrl_model.cpp` + `test_dsws_ctrl_model.cpp`: +- Model the **snapshot/quiesce interaction**: a `snapshot(E)` reads role counts; conversions + mutate counts at the boundary; assert `quiesce(E)` uses `snap(E)` and that + `Σ snap(E) sentinels ⟺ QUIESCE_CNT == N−1` under arbitrary interleavings (thread-race test, + as the existing `gate_try_win` test does). +- Keep `watermark_decision / epoch_of / gate_try_win / reserve_grow` tests green (unchanged + semantics). All `ALL PASS` before any assemble. + +## Section 6 — Build sequence & gates (isolation-preserving) + +Each stage a supervised GPU gate; you greenlight each; brick = full STOP + bisect. Offline +before each: `dsws_ctrl_model` tests green, RGA 0-spill, dry-print sane. + +Config held at Phase-A values **`G=6, SEGK=64`** for stage 1 so the conversion code is the only +variable vs the green static gate. + +1. **Static-mix through the conversion path** — conversion code wired but watermarks set so + **none fire** (`CTRL_LOW=0` / unreachable). Must reproduce the Phase-A green (all 3 mixes × + both tiers, `ok=… bad=0`). Proves the ported actuation + snapshot machinery is inert-safe / + non-regressing. **[SUPERVISED GPU — the re-baseline gate.]** +2. **Dynamic-mix** — watermarks that *do* fire conversions; oracle stays green as roles move. + Start `n_kseg=1` TIGHT (exact) at one mix, then LOOSE, then the other mixes. **[SUPERVISED.]** +3. **Storm** — tight watermarks + `EPOCH_SHIFT = 0` + ×10 repeats: the lock-free race-hunt + (the strong-oracle-plus-repeats discipline that caught 136/552). **[SUPERVISED.]** + +Phase 4 (separate spec): adaptivity proof (converge-from-wrong-start), +`{LOW,HIGH,RINGD,EPOCH_SHIFT,G,SEGK}` sweep, `--att` issue-mix on ml8 `down`/`down_pf`. + +## Testing + +- **CPU oracle gate** (`fp8_oracle.cpp`): Tier-1 tight (`5e-3` rel / `1e-2` abs) at `n_kseg=1`; + Tier-2 loose (`3e-2` / `2e-2`) at `n_kseg>1`. Before every perf run and every kernel change. +- **RGA static gate:** 0-spill, live-VGPR within budget, every assemble. +- **Control-law unit tests:** extended per §5, `ALL PASS`. +- **Storm stress:** §6 stage 3. +- **DIAG cross-check:** `QUIESCE_CNT == N−1` ⟺ `Σ snapshot sentinels` (Decision-1 safety net). + +## Success metric + +Oracle-green through the **storm** at dynamic mix on both tiers, all mixes, **zero bricks** — +the partition provably moves at runtime with correct output. The DSWS *thesis* payoff (beats +static baseline + `--att` shows cut non-WMMA issues on compute waves, on ml8 `down`/`down_pf`) +is Phase 4, not a Phase-B gate. + +## Risks & open items + +- **`s_alloc_vgpr` OOR-poison in the conversion path** — highest brick risk (§4). Round-table + the register assignment before the first dynamic-mix dispatch. +- **Concurrent role-slot CAS** — bounded to ≤2 writers/boundary by the ticket; verify the CAS + is genuinely atomic-LDS and the floor-guard dec/inc pair can't transiently violate a floor. +- **Snapshot double-buffer parity** — confirm `[E&1]` indexing can't alias when the pool is + ≤2 super-tiles deep (degenerate tiny-shape oracle case). +- **`vgpr_reserved` under grow-abort** — the atomic_add-then-sub-on-over-budget must not leave a + transient over-count visible to a *second* concurrent grower; both are ticket-serialized per + epoch, but verify across directions. +- **Round-table structure** (kmbandy's): implement (Sonnet) → adversarial review (Fable + Codex) + → kmbandy greenlights each GPU dispatch. It caught 5 offline bricks in Phase A; keep it. diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/SPEC_DSWS_SUBSTRATE_V2.md b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/SPEC_DSWS_SUBSTRATE_V2.md new file mode 100644 index 000000000000..55bab7acef7a --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/SPEC_DSWS_SUBSTRATE_V2.md @@ -0,0 +1,222 @@ +# DSWS Substrate v2 — Claim-Based Work Decomposition + Split-K (design) + +**Status:** approved design (2026-06-30). Supersedes the Phase-3 *actuation* plan in +`PLAN_DSWS_CONTROLLER.md` (Tasks 3.2–3.4), which assumed role conversion could be bolted +onto the static-partition coop substrate. It cannot — see "The blocker" below. Phases 1–2 +(static 3-role substrate, sensing, role-count slots, gate-CAS ticket, reservation model) +remain valid and are reused. + +**Goal (one sentence):** Re-found the DSWS substrate so matrix work is *claimed by whatever +wave currently holds a role* rather than *owned by a wave's compile-time identity*, and fold +in split-K — so the controller can move waves between {compute / A-feed / B-feed} at runtime +without orphaning output, jamming a feed, or bricking. + +**Architecture (2-3 sentences):** A workgroup launches a fixed N waves. Work is a pool of +`(mblk, tcol, kseg)` super-tiles; a pinned claimer broadcasts the current super-tile, and the +live waves of each role drain shared atomic counters (compute claims rowblks, feeds claim +which operand fragment to stage) against resident-in-LDS A/B for that super-tile. Because work +is claimed, not owned, `nComp/nAfeed/nBfeed` can change at any per-kseg boundary with no +work-item handoff. + +**Tech stack:** Hand-written gfx1201 (RDNA4, wave32) assembly (`occ_kernel_dsws.s`, NEW file), +raw-PM4 dispatch (`occ_dispatch.cpp`), CPU fp8 e4m3 oracle (`fp8_oracle.cpp`), control law +(`dsws_ctrl_model.cpp`, unchanged). dyn-VGPR via `s_alloc_vgpr` (armed by PM4 RSRC2 bit 6). + +## Global constraints (verbatim, every task inherits these) + +- A GPU brick is a **BUG**, never an accepted tax. A hang = full STOP + report, never + auto-fire the next variant. +- **The user greenlights EVERY GPU dispatch individually.** Display GPU (R9700 drives the + monitors) → only sub-second, compositor-safe-chunked dispatches + (`ML8_POOL=1 ML8_COOP_CHUNK=8 ML8_COOP_CHUNK_MAXS=0.75 ML8_COOP_STREAM=1`, timeout 30). +- **NEVER pass `--gl2c`** (MES-crash landmine). SAFEPROBE + bounds gate + padding stay ON. +- Every run streams to disk (`ML8_COOP_STREAM=1`). +- Commit only when the user asks. +- Single-variable isolation; fix bugs, don't dodge them; never declare a wall from an + unmeasured assumption; don't blame the GPU/model prematurely. +- Barrier-free / lock-free: pure LDS atomics + busy-wait flags. **No `s_barrier`** (mixed + dyn-VGPR allocations + `s_barrier` hard-deadlock the GPU — proven). +- The proven `occ_kernel_coop.s` (1716B `DSWS=0` coop binary) is **never modified** — it is + the known-good reference. v2 is an additive new file. + +## The blocker (why v2 exists) + +3/3 consensus (kmbandy + Claude + Codex, session 019f19da, 2026-06-30; KG `86e33108`): the +proven coop substrate binds the matrix **work decomposition** to compile-time role counts and +physical wave identity (`wid`). Naive role conversion therefore **bricks** (the hang fires +before any wrong-output is even observable): + +1. Compute M-rows: `rowblk = trow*P + cid`, `P = NCOMP` compile-time, `cid` fixed from `wid`. + A departed `cid`'s rows have **no writer** → orphaned output. +2. A-feed: `NCOMP` bands paired 1:1 to compute `cid` via `prod_a[cid]/cons_a[cid]`. A departed + consumer → `prod_a` blocks at `RINGD_A` → **WG hangs**. B-feed's `min_cons` also scans the + `P` compute counters, so a departed `cid` wedges B production too. +3. B-feed frags: `owner = ni % NBFEED`; a converted-in wave with `b_id ≥ NBFEED` computes a + `prod_b` address that **overflows into A-ring/LDS storage** → memory corruption. + +Root cause: work bound to wave **identity** instead of being **claimable by whoever holds the +role**. v2 removes that binding. + +## Architecture — the role economy (unchanged from SPEC_DSWS_CONTROLLER.md) + +Fixed N waves/WG; each wave is in exactly one role at any instant. The controller governs only +the partition `(nComp, nAfeed, nBfeed)` with `nComp + nAfeed + nBfeed = N`. Wave count never +changes; only the partition moves. State = three atomic LDS role-count slots (already built, +T2.1). Floors: each role `≥ 1`. Sum-envelope: `Σ instantaneous VGPR alloc < per-SIMD budget`, +enforced at grow-time via the `vgpr_reserved` counter (already built, T2.2). + +## Section 1 — Work pool & claim model + +**Super-tile** = `(mblk, tcol, ksi)`. K-loop terms used throughout: `SEGK` = segment size +(K-elements per split-K segment, a compile-time knob); `n_kseg = KT / SEGK` = number of +segments; `ksi ∈ [0, n_kseg)` = segment index. +- `mblk` indexes a group of `G` consecutive rowblks — `G` = compile-time cooperative M-extent + (set to the launch's max compute count, `NCOMP_MAX`). Bounds resident A. +- `tcol` = the shared N-column tile (`FN` frags). `ksi` = the split-K segment index. +- Pool size = `(M / (G·16·FM)) × NTL × n_kseg` super-tiles. + +**Two claim levels** (reuse the proven "claimer claims + broadcasts, followers wait on epoch" +machinery): +- **Super-tile claim:** a pinned claimer wave (Section 3) does the existing `global_atomic_add` + to grab the next super-tile index `sti`, decodes `(mblk, tcol, kseg)`, publishes it, then + bumps epoch (the proven `TI_OFF`-before-`EPOCH_OFF` ordering). All WG waves work the same + current super-tile. +- **Rowblk claim (within a super-tile):** a per-super-tile LDS counter `rowblk_next`. Live + compute waves `ds`-atomic-add to grab the next `rowblk ∈ [0, G)`. Exhausted (`≥ G`) → this + super-tile's compute is done. + +**Coverage proof:** the super-tile pool covers every `(mblk, tcol, kseg)` once; `rowblk_next` +covers every rowblk in the group once. So every `(rowblk, tcol, kseg)` is computed exactly +once and `C[rowblk,tcol] = Σ_kseg partial`. Holds for **any** live `nComp ≤ G` — fewer compute +waves just drain `rowblk_next` slower. (If the controller ever set `nComp > G`, the extra waves +find the counter exhausted and idle/convert; floors+ceilings keep `nComp ≤ G`.) + +## Section 2 — Resident A/B lifetime & completion handshake + +Split-K keeps each segment's operands small enough to stage **resident** in LDS: +- **Resident B** for `(tcol, ksi)` = `FN·16` cols × `SEGK` K. Loaded by B-feed waves (each + claims which frag to stage from a frag counter — identity-free). +- **Resident A** for `(mblk, ksi)` = `G` rowblks × `16·FM` rows × `SEGK` K. Loaded by A-feed + waves (each claims which rowblk's A to stage from an A-load counter — identity-free). Compute + reads its claimed rowblk's A from this resident region (replayable — any compute wave + re-reads freely; this is what decouples rowblk-count from `nComp`). + +**Completion handshake** (barrier-free; the safety core): +- **`rowblk_done`** (per-super-tile LDS counter). A compute wave, after flushing its rowblk's + partial, `ds`-atomic-increments `rowblk_done`. +- **Claimer gate:** the claimer may claim/broadcast the *next* super-tile only once + `rowblk_done == G` for the current one (all rowblks computed *and* flushed). This frees the + resident A/B safely — a counter compare, not `s_barrier`. +- A compute wave that finishes its claimed rowblks before the super-tile is globally complete + spins on the completion gate (or attempts a conversion) rather than racing ahead — the + busy-wait discipline the kernel already uses. + +**Terminal:** super-tile claim returning `sti ≥ TOTAL_super` is the role-agnostic retire +signal; every role checks it at its decision boundary (replaces the per-tile POOLTERM, now +sub-tile aware). A just-converted wave re-checks immediately. + +## Section 3 — Role tags & the pinned claimer (simplification over a published role map) + +The claim-counter model makes a *published per-wave role/rank map* unnecessary — every role +claims work from shared counters, so no role needs a stable logical rank; count affects only +speed, never coverage. What remains: +- **Role-count slots** (already built, T2.1) — CAS'd on conversion; drive sensing + floor + guards. +- **Per-wave private current-role register** — the wave branches to its role loop; on + conversion it rewrites this register and jumps. Cross-wave visibility is carried by the + atomic work-counters themselves, not a broadcast map. +- **Pinned claimer + clock = physical `wid 0`** (never converts). Permanently owns super-tile + claiming, the `ti`/epoch broadcast, and the `SEGCNT` controller clock. This kills the + clock-stall failure mode — the clock owner can never convert away. + +This is a deliberate, documented deviation from the consensus "generationed role map" step: the +claim model dissolved the need, removing a class of cross-wave ordering hazards. + +## Section 4 — Partial-C reduction & tiered oracle + +Split-K's headroom requires a wave to do **one kseg then release** (brief VGPR peak), so +different waves compute different ksegs of the same `(rowblk,tcol)` → their partials combine +**across waves** → the low bits of `C` are no longer bit-deterministic (fp combine order). The +combine uses `global_atomic_add` of fp32 partials into `C`. + +**Tiered oracle** — tight tolerance where the risk is, looser only where split-K fp +reassociation forces it. (NB: the CPU `wmma_ref` chain is not bit-identical to the GPU fp8→fp32 +WMMA even today, so the established gate is already a *tight tolerance*, `5e-3` rel + `1e-2` abs, +not bit-exact — that is the discipline that caught 136/552.) +- **Tier 1 — tight.** Run the correctness gate at `n_kseg = 1`: one partial per `(rowblk,tcol)`, + no cross-wave combine. Reuse the **existing tight tolerance** (`5e-3` rel / `1e-2` abs). This + exercises all the dangerous new logic (claim, coverage, resident A/B, handshake, conversion — + identical code regardless of kseg count). +- **Tier 2 — loose.** Run `n_kseg > 1` against the reference with a looser tolerance + (`~3e-2` rel / `2e-2` abs). A structural combine bug (missed/double kseg) is a *large* error → + caught; only the benign split-K reassociation slips under the looser bound. + +Build-time check: confirm `global_atomic_add_f32` is encodable on gfx1201; if not, the combine +uses a CAS loop or a scratch-slot + final-reduction path (the scratch path also recovers exact +determinism at `n_kseg > 1` if ever needed). + +## Section 5 — Conversion actuation + +Reuses the already-built gate-CAS ticket (T3.1), reservation model (T2.2), and sensors (T2.3). +At a per-kseg boundary (frequent, sub-tile) a wave eligible to convert: +1. **Sense** ring/counter occupancy → watermark decision (`watermark_decision`). +2. **Win the epoch ticket** `try_gate(dir)` — single winner per `(dir, epoch)`. +3. **Floor guard** — CAS-dec the source role-count only if `> 1`. +4. **Reservation envelope** — `compute→feed` shrink: `atomic_sub vgpr_reserved` (always + succeeds). `feed→compute` grow: `atomic_add` then validate `≤ BUDGET`, else `atomic_sub` + + abort (stay in current role this epoch). +5. **Actuate** — CAS role slots (dec source, inc dest), flip own private role register, + `s_alloc_vgpr` GROW(`NFV`)/SHRINK(32) (each guarded by SCC-retry), jump to the dest role's + loop. + +**Payoff:** a converted wave inherits **no work item** — it changes its tag and starts claiming +from the dest role's counters like any other wave of that role. No orphaned rows, no rank +handoff, no jammed feed. Every blocker failure mode is structurally absent. Barrier-free +throughout. + +## Section 6 — File structure, build sequencing & gates + +**New file `occ_kernel_dsws.s`.** Diverges enough (resident A/B, split-K partial-combine, +claim-counters) that it is a clean new kernel; `occ_kernel_coop.s` stays pristine as the +known-good reference. Trade-off: no literal "byte-identical to 1716B" guard inside the new +file, bought with the proven kernel never being touched (smallest blast radius). + +**Build sequence (isolation within the folded scope — a brick/oracle break stays bisectable):** +1. **Static split-K + claim-counter substrate, fixed roles, no conversion.** Oracle-green at + `n_kseg = 1` (exact) *and* `n_kseg > 1` (tolerance). Proves claim-coverage + resident A/B + + partial-combine + completion handshake with static roles. RGA 0-spill. **[SUPERVISED GPU — + the big gate.]** +2. **Add conversion** (Section 5). Oracle-green static-mix, then dynamic-mix (conversions + firing), then the storm (tight watermarks, `EPOCH_SHIFT = 0`, ×10 repeats). **[SUPERVISED + GPU.]** +3. **Adaptivity proof + tuning** (Phase 4 carries over: converge-from-wrong-start, `{LOW, HIGH, + RINGD, EPOCH_SHIFT, G, SEGK}` sweep, `--att` issue-mix). **[SUPERVISED.]** + +Every GPU dispatch: compositor-safe chunked, one at a time, user greenlights each, brick = full +STOP + bisect. + +## Testing + +- **CPU oracle gate** (`fp8_oracle.cpp`): Tier-1 exact bit-match at `n_kseg = 1`; Tier-2 tight + tolerance at `n_kseg > 1`. Gate before every perf run and on every kernel change. +- **RGA static gate:** 0-spill, live-VGPR within budget, every assemble. +- **Control-law unit tests** (`test_dsws_ctrl_model.cpp`): unchanged, still `ALL PASS`. +- **Storm stress:** tight watermarks + `EPOCH_SHIFT = 0` + ×10 repeats — proves the lock-free + protocol has no conversion race (the strong-oracle-plus-repeats discipline that caught + 136/552 before). + +## Risks & open items + +- **LDS budget for resident A+B.** Resident A (`G·16·FM` rows × `SEGK`) + resident B (`FN·16` + cols × `SEGK`) + role/claim state + ring-free counters must fit the 32 KB group segment. + `G`, `SEGK`, `FM`, `FN` are the knobs; pick a first config that fits with margin and verify + in step 1. +- **`global_atomic_add_f32` on gfx1201** — verify encodable; fallback CAS loop / scratch path. +- **Partial-combine traffic** — split-K adds `n_kseg` atomic-adds per `(rowblk,tcol)`. The bet + (per the DSWS thesis) is that the issue-port offload + fungibility gain exceeds this overhead; + measured, not assumed, in Phase 4. +- **`G` vs `nComp` overshoot** — floors/ceilings must keep `nComp ≤ G`; assert in the + controller. +- **Claimer single point** — `wid 0` pinned as claimer means it is always a B-feed-class wave; + confirm the role economy still balances with one permanently-non-compute wave (it is one wave + of N; negligible, but noted). diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/build_dsws.sh b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/build_dsws.sh new file mode 100755 index 000000000000..87e550793440 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/build_dsws.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# build_dsws.sh — MAD-305 DSWS static 3-role substrate bins (occ_kernel_coop.s, DSWS=1). +# FM=2 FN=4 POOLTERM=1 fixed (the v1 coop tile; role counts are the swept defsyms). +# Bin name MUST match occ_dispatch.cpp --dsws: occ_dsws_cab_r[_dyn]_gd.bin +# OFFLINE/CPU only — assemble + RGA, no GPU. Usage: ./build_dsws.sh [static] [rga] +set -e +cd "$(dirname "$0")" +L=/opt/rocm/llvm/bin +fail=0 +mk() { # $1=NCOMP $2=NAFEED $3=NBFEED $4=RINGD $5=DYN + local dtag=""; [ "$5" = "1" ] && dtag="_dyn" + local tag="occ_dsws_${1}c${2}a${3}b_r${4}${dtag}_gd" + nice -19 ionice -c3 "$L/clang" -x assembler -target amdgcn-amd-amdhsa -mcpu=gfx1201 \ + -Wa,-defsym,DSWS=1 -Wa,-defsym,FM=2 -Wa,-defsym,FN=4 -Wa,-defsym,POOLTERM=1 -Wa,-defsym,SAFEPROBE=1 -Wa,-defsym,DIAG=${DIAG:-1} \ + -Wa,-defsym,NCOMP=$1 -Wa,-defsym,NAFEED=$2 -Wa,-defsym,NBFEED=$3 \ + -Wa,-defsym,RINGD=$4 -Wa,-defsym,DYNVGPR=$5 \ + -c occ_kernel_coop.s -o "$tag.o" 2>/tmp/dsws_build.err \ + && { "$L/llvm-objcopy" -O binary --only-section=.text "$tag.o" "$tag.bin"; echo " OK $tag.bin ($(wc -c < "$tag.bin")B)"; } \ + || { echo " FAIL $tag"; sed -n '1,15p' /tmp/dsws_build.err; fail=1; } +} + +mk2() { # $1=NCOMP $2=NAFEED $3=NBFEED (DSWS2 v2 substrate, occ_kernel_dsws.s; G=6 SEGK=64 FM=2 FN=4) + local tag="occ_dsws2_${1}c${2}a${3}b_gd" + # BUDGET passthrough: default mirrors the in-file .ifndef launch-footprint conservation ceiling + # (NCOMP*NFV + (NAFEED+NBFEED)*VLEAN, with NFV=112/VLEAN=32 at this file's fixed FM=2 FN=4) so an + # unset $BUDGET reproduces the existing default exactly; set $BUDGET to give real per-SIMD headroom. + # Envelope-mode budget = WAVES*VLEAN + PEAK_CONC*(NFV-VLEAN) = (c+a+b)*32 + PEAK_CONC*80; else static-fat. + local budget + if [ "${DSWS2_ENVELOPE:-0}" = "1" ]; then + budget="${BUDGET:-$(( ($1 + $2 + $3) * 32 + ${PEAK_CONC:-2} * 80 ))}" + else + budget="${BUDGET:-$(( $1 * 112 + ($2 + $3) * 32 ))}" + fi + # DSWS2_FORCE* passthrough (Task 5): defaults mirror the in-file .ifndef values exactly, so an + # unset env leaves every existing mk2 call byte-identical (DSWS2_FORCE=0 emits zero bytes). + nice -19 ionice -c3 "$L/clang" -x assembler -target amdgcn-amd-amdhsa -mcpu=gfx1201 \ + -Wa,-defsym,DSWS2=1 -Wa,-defsym,FM=2 -Wa,-defsym,FN=4 -Wa,-defsym,G=6 -Wa,-defsym,SEGK=64 \ + -Wa,-defsym,SAFEPROBE=1 -Wa,-defsym,DIAG=${DIAG:-1} -Wa,-defsym,BUDGET=$budget \ + -Wa,-defsym,NCOMP=$1 -Wa,-defsym,NAFEED=$2 -Wa,-defsym,NBFEED=$3 \ + -Wa,-defsym,DSWS2_FORCE=${DSWS2_FORCE:-0} -Wa,-defsym,DSWS2_FORCE_WID=${DSWS2_FORCE_WID:-0} \ + -Wa,-defsym,DSWS2_FORCE_DIR=${DSWS2_FORCE_DIR:-0} -Wa,-defsym,DSWS2_FORCE_EPOCH=${DSWS2_FORCE_EPOCH:-1} \ + -Wa,-defsym,DSWS2_ENVELOPE=${DSWS2_ENVELOPE:-0} -Wa,-defsym,PEAK_CONC=${PEAK_CONC:-2} \ + -Wa,-defsym,DSWS2_STAGGER=${DSWS2_STAGGER:-0} -Wa,-defsym,STAGGER_PERIOD=${STAGGER_PERIOD:-4} \ + -c occ_kernel_dsws.s -o "$tag.o" 2>/tmp/dsws2_build.err \ + && { "$L/llvm-objcopy" -O binary --only-section=.text "$tag.o" "$tag.bin"; echo " OK $tag.bin ($(wc -c < "$tag.bin")B)"; } \ + || { echo " FAIL $tag"; sed -n '1,15p' /tmp/dsws2_build.err; fail=1; } +} + +echo "== DSWS static 3-role bins (FM=2 FN=4 POOLTERM=1) ==" +# NCOMP NAFEED NBFEED RINGD DYN +mk 4 2 2 2 1 ; mk 4 2 2 2 0 # 4c2a2b +mk 6 1 1 2 1 ; mk 6 1 1 2 0 # 6c1a1b +mk 2 3 3 2 1 ; mk 2 3 3 2 0 # 2c3a3b +echo "dsws build done. fail=$fail" + +echo "== DSWS2 v2 substrate scaffold bin (occ_kernel_dsws.s; G=6 SEGK=64) ==" +mk2 4 2 2 # 4c2a2b +echo "dsws2 build done. fail=$fail" + +# RGA gate (offline static analysis; 0-spill is the bar). Runs by default unless 'norga' passed. +if [ "${1:-}" != "norga" ] && [ "${2:-}" != "norga" ]; then + echo "== RGA gate (4c2a2b dyn — compute peak-live is tile-fixed, representative) ==" + KSRC=occ_kernel_coop.s ./rga_check.sh dsws_build_4c2a2b \ + DSWS=1 FM=2 FN=4 NCOMP=4 NAFEED=2 NBFEED=2 RINGD=2 POOLTERM=1 SAFEPROBE=1 DYNVGPR=1 2>&1 \ + | grep -E "gfx1201,|livereg" || true +fi +exit $fail diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/build_flow.sh b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/build_flow.sh new file mode 100755 index 000000000000..c761f7303f65 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/build_flow.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# build_flow.sh — FIX 1 (flow economy) bin (occ_kernel_dsws_flow.s). N-deep pool + ROLE mailbox + +# coordinator. Bin name matches occ_dispatch.cpp DSWS2_FLOW path: occ_dsws2_cab_flow_gd.bin +# OFFLINE/CPU only. Usage: ./build_flow.sh [NCOMP NAFEED NBFEED] +# Env: POOL_N=3 PHASEPROBE={0|1} NOCFLUSH={0|1} CSTORE={0|1} SLEEPN=N COORD_PERIOD=N DIAG=0 +set -e +cd "$(dirname "$0")" +L=/opt/rocm/llvm/bin +fail=0 +mkflow() { # EMERGENT economy: no mix args. Env: WAVES VBUDGET G SEGK POOL_N ACC_N ... + local tag="occ_dsws2_w${WAVES:-16}_flow_gd" + nice -19 ionice -c3 "$L/clang" -x assembler -target amdgcn-amd-amdhsa -mcpu=gfx1201 \ + -Wa,-defsym,DSWS2=1 -Wa,-defsym,FM=2 -Wa,-defsym,FN=4 -Wa,-defsym,G=${G:-6} -Wa,-defsym,SEGK=${SEGK:-64} \ + -Wa,-defsym,SAFEPROBE=1 -Wa,-defsym,DIAG=${DIAG:-0} -Wa,-defsym,POOL_N=${POOL_N:-3} -Wa,-defsym,ACC_N=${ACC_N:-1} -Wa,-defsym,WOFLUSH=${WOFLUSH:-0} \ + -Wa,-defsym,WAVES=${WAVES:-16} -Wa,-defsym,VBUDGET=${VBUDGET:-1536} \ + -Wa,-defsym,PHASEPROBE=${PHASEPROBE:-0} -Wa,-defsym,NOCFLUSH=${NOCFLUSH:-0} -Wa,-defsym,CSTORE=${CSTORE:-0} \ + -Wa,-defsym,SLEEPN=${SLEEPN:-2} -Wa,-defsym,COORD_PERIOD=${COORD_PERIOD:-64} -Wa,-defsym,TFPROBE=${TFPROBE:-0} -Wa,-defsym,DEADMAN=${DEADMAN:-1} -Wa,-defsym,DEADMAN_TICKS=${DEADMAN_TICKS:-50000000} -Wa,-defsym,STAGINSTR=${STAGINSTR:-0} -Wa,-defsym,TRACE=${TRACE:-0} \ + -c occ_kernel_dsws_flow.s -o "$tag.o" 2>/tmp/flow_build.err \ + && { "$L/llvm-objcopy" -O binary --only-section=.text "$tag.o" "$tag.bin"; \ + echo " OK $tag.bin ($(wc -c < "$tag.bin")B .text) [POOL_N=${POOL_N:-3} PHASEPROBE=${PHASEPROBE:-0}]"; } \ + || { echo " FAIL $tag"; sed -n '1,25p' /tmp/flow_build.err; fail=1; } +} +echo "== flow bin (occ_kernel_dsws_flow.s; EMERGENT mix; WAVES=${WAVES:-16} G=${G:-6} SEGK=${SEGK:-64} POOL_N=${POOL_N:-3} VBUDGET=${VBUDGET:-1536}) ==" +mkflow +echo "flow build done. fail=$fail" +exit $fail diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/build_ring.sh b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/build_ring.sh new file mode 100755 index 000000000000..bfba9eced325 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/build_ring.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# build_ring.sh — FIX 1a D=2 double-buffered ring-of-slots bin (occ_kernel_dsws_ring.s). +# Bin name MUST match occ_dispatch.cpp DSWS2_RING path: occ_dsws2_cab_ring_gd.bin +# OFFLINE/CPU only — assemble + objcopy, NO GPU. Usage: ./build_ring.sh [NCOMP NAFEED NBFEED] +# Env: PHASEPROBE={0|1} NOCFLUSH={0|1} CSTORE={0|1} SLEEPN=N DIAG=0 +set -e +cd "$(dirname "$0")" +L=/opt/rocm/llvm/bin +fail=0 +mkring() { # $1=NCOMP $2=NAFEED $3=NBFEED + local tag="occ_dsws2_${1}c${2}a${3}b_ring_gd" + nice -19 ionice -c3 "$L/clang" -x assembler -target amdgcn-amd-amdhsa -mcpu=gfx1201 \ + -Wa,-defsym,DSWS2=1 -Wa,-defsym,FM=2 -Wa,-defsym,FN=4 -Wa,-defsym,G=6 -Wa,-defsym,SEGK=64 \ + -Wa,-defsym,SAFEPROBE=1 -Wa,-defsym,DIAG=${DIAG:-0} \ + -Wa,-defsym,NCOMP=$1 -Wa,-defsym,NAFEED=$2 -Wa,-defsym,NBFEED=$3 \ + -Wa,-defsym,PHASEPROBE=${PHASEPROBE:-0} -Wa,-defsym,NOCFLUSH=${NOCFLUSH:-0} -Wa,-defsym,CSTORE=${CSTORE:-0} \ + -Wa,-defsym,SLEEPN=${SLEEPN:-2} -Wa,-defsym,TFPROBE=${TFPROBE:-0} \ + -c occ_kernel_dsws_ring.s -o "$tag.o" 2>/tmp/ring_build.err \ + && { "$L/llvm-objcopy" -O binary --only-section=.text "$tag.o" "$tag.bin"; \ + echo " OK $tag.bin ($(wc -c < "$tag.bin")B .text) [PHASEPROBE=${PHASEPROBE:-0} NOCFLUSH=${NOCFLUSH:-0}]"; } \ + || { echo " FAIL $tag"; sed -n '1,20p' /tmp/ring_build.err; fail=1; } +} +c=${1:-4}; a=${2:-2}; b=${3:-2} +echo "== FIX 1a ring bin (occ_kernel_dsws_ring.s; G=6 SEGK=64 FM=2 FN=4 D=2) ==" +mkring "$c" "$a" "$b" +echo "ring build done. fail=$fail" +exit $fail diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/dsws_ctrl_model.cpp b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/dsws_ctrl_model.cpp new file mode 100644 index 000000000000..3a1d09ed4644 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/dsws_ctrl_model.cpp @@ -0,0 +1,147 @@ +// DSWS control-law reference model (SPEC_DSWS_CONTROLLER.md). Pure host functions that define the +// EXACT semantics the Phase-3 gfx1201 asm transcribes. Kept header-light (included directly by the +// test and, later, mirrored by hand-asm) so there is one source of truth for the protocol. +// +// watermark_decision -> the boundary band check (sensing -> action sign) +// epoch_of -> the per-WG decision clock E = segments_processed >> EPOCH_SHIFT +// gate_try_win -> the lock-free single-winner ticket (asm: ds_cmpst_b32 on gate[dir]) +// reserve_grow -> the sum-envelope reservation (asm: atomic_add/sub on vgpr_reserved) +#pragma once +#include +#include + +// Sensing -> action sign for ring X. Bands are STRICT (edges are dead-zone) so LOW==HIGH degenerates +// to "act only when strictly past the edge", and LOW +1 (ring draining empty: the consumer is STARVED for X -> wants more feed-X) +// occ > high -> -1 (ring backing up full: feed-X is OVER-SERVING -> a feed-X wave can leave) +// else -> 0 (dead-zone: hold) +static inline int watermark_decision(uint32_t occ, uint32_t low, uint32_t high) { + if (occ < low) return +1; + if (occ > high) return -1; + return 0; +} + +// Per-workgroup decision clock. Ticks every 2^shift segments of progress; no one "advances" it, it is +// purely a function of throughput. shift is the cadence knob (small=reactive, large=damped). +static inline uint32_t epoch_of(uint32_t segments_processed, uint32_t shift) { + return segments_processed >> shift; +} + +// Lock-free single-winner ticket for one conversion direction at epoch E. gate holds the last epoch in +// which this direction fired. Among many waves racing the same (g < E), exactly one CAS succeeds; the +// rest observe g advanced and back off. Guarantees <=1 conversion per direction per epoch, per WG. +// asm: g = ds_read gate[dir]; if g>=E -> lose; else old = ds_cmpst_b32(gate[dir], g, E); win iff old==g +static inline bool gate_try_win(std::atomic& gate, uint32_t E) { + uint32_t g = gate.load(std::memory_order_relaxed); + if (g >= E) return false; // direction already fired this (or a later) epoch + // compare_exchange_strong updates `g` to the observed value on failure -> mirrors ds_cmpst's + // returned-old semantics; a single retry loop is unnecessary because a failed CAS means someone + // else won this epoch (g becomes >=E), so we simply lose. + return gate.compare_exchange_strong(g, E, std::memory_order_acq_rel, std::memory_order_relaxed); +} + +// Sum-envelope reservation for a feed->compute GROW. Reserve first (atomic_add), then validate against +// budget; if the reservation would blow the envelope, undo it (atomic_sub) and reject. The atomic +// serializes concurrent grows: the second to validate sees the first's reservation and backs off. +// asm: r = atomic_add(vgpr_reserved, delta); if (r+delta) > budget -> atomic_sub(delta), abort +static inline bool reserve_grow(std::atomic& resv, uint32_t delta, uint32_t budget) { + uint32_t prev = resv.fetch_add(delta, std::memory_order_acq_rel); + if (prev + delta > budget) { + resv.fetch_sub(delta, std::memory_order_acq_rel); // over-budget: cleanly undo, stay current role + return false; + } + return true; +} + +// Compute-burst reserve with spin-retry (models .Lcompute_reserve): reserve +delta against the +// sum-envelope; on over-budget, reserve_grow has already undone its add, so back off and retry. +// `spins` accumulates the backoff count (permit-starvation depth). Bounded when >=1 peak fits. +static inline void reserve_spin(std::atomic& resv, uint32_t delta, + uint32_t budget, uint64_t& spins) { + while (!reserve_grow(resv, delta, budget)) ++spins; +} +// Release a booked burst (models the post-shrink lds_fetch_add VRESV_OFF, -delta). Never fails. +static inline void reserve_release(std::atomic& resv, uint32_t delta) { + resv.fetch_sub(delta, std::memory_order_acq_rel); +} + +struct WgSnap { uint32_t nC, nA, nB; }; + +static inline WgSnap snapshot_counts(uint32_t nC, uint32_t nA, uint32_t nB) { + return WgSnap{nC, nA, nB}; +} + +// Sentinels = work-threshold + snapshot role-count terminal bails (Phase A arithmetic, +// with compile-time constants replaced by the per-epoch snapshot). +static inline bool quiesce_ready(uint32_t rowblk_next, uint32_t bfrag_next, + uint32_t arow_next, const WgSnap& s, + uint32_t G, uint32_t FN) { + return rowblk_next >= (G + s.nC) + && bfrag_next >= (FN + s.nB) + && arow_next >= (G + s.nA); +} + +// Role-agnostic safety net: fixed N waves, wid0 claimer never bails -> exactly N-1 bails. +static inline bool quiesce_ready_nm1(uint32_t quiesce_cnt, uint32_t N) { + return quiesce_cnt >= (N - 1); +} + +// ---- Task 1: dispatch, cooldown, pool invariants ---- + +enum Role { COMPUTE, AFEED, BFEED }; + +inline Role role_dispatch(uint32_t slot_id) { + return slot_id == 24 ? COMPUTE : (slot_id == 28 ? AFEED : BFEED); +} + +inline uint32_t cooldown_step(uint32_t cd) { return cd ? cd - 1 : 0; } + +inline bool in_cooldown(uint32_t cd) { return cd > 0; } + +inline bool pool_fits_lean(uint32_t n_pool, uint32_t vlean, uint32_t budget) { + return (uint64_t)n_pool * vlean <= budget; +} + +inline bool quiesce_ready_pool(uint32_t quiesce_cnt, uint32_t n_pool) { + return quiesce_cnt >= n_pool - 1; +} + +// ---- Pool-T7 root cause: first-entry contract (dispatch vs re-dispatch) ---- +// +// A wave's FIRST-time entry and its RE-dispatch (after a per-super-tile bail) are NOT +// interchangeable. First entry must run, in order, the per-role _alloc and _init blocks: +// (a) ran_alloc -- s_alloc_vgpr 32, the DYNVGPR per-wave lean allocator handshake, +// (b) waited_initflag -- spin until the claimer publishes INITFLAG == 0xACED (LDS ready), +// (c) seeded_epoch -- s35 = 0, the local last-seen-epoch baseline, +// THEN falls into _follow. Re-dispatch legitimately skips (a)/(b)/(c): the wave already ran +// them once and conv_apply already sized its footprint -- so the scalar-only .Ldispatch +// trampoline lands straight on _follow. The bug: the seed arms pointed FIRST entry at +// .Ldispatch too, so first-time followers skip _alloc/_init and desync from the epoch clock. +struct WaveEntry { bool ran_alloc; bool waited_initflag; bool seeded_epoch; }; + +// Where a wave's first-time entry lands. LAND_FOLLOW == seed arms branch to .Ldispatch (the +// buggy routing); LAND_ROLE_ENTRY == seed arms branch to .Lbfeed/.Lafeed/.Lcompute (the fix). +enum EntryLanding { LAND_FOLLOW, LAND_ROLE_ENTRY }; + +inline WaveEntry simulate_first_entry(EntryLanding land) { + // Only the role entry labels chain _alloc -> _init -> _follow; landing on _follow skips both. + bool full = (land == LAND_ROLE_ENTRY); + return WaveEntry{full, full, full}; +} + +// A wave is correctly initialized for the follow/quiesce protocol iff it ran all three. +inline bool entry_safe(const WaveEntry& w) { + return w.ran_alloc && w.waited_initflag && w.seeded_epoch; +} + +// End-to-end handshake: does the claimer's per-super-tile QUIESCE_CNT >= WAVES-1 gate ever +// close? Every non-claimer follower (wid0 is the claimer and never bails) must reach its +// _quiesce bail once per super-tile; only a follower that entered safely is synced to the +// epoch clock and reliably does. An unsafe entry is a deterministic code-path defect (garbage +// s35 + skipped INITFLAG in identically-compiled waves), so ALL seed-entered followers desync +// -> zero reliable bumps -> the claimer spins forever in .Lclaimer_wait_done. +inline bool claimer_quiesce_converges(EntryLanding land, uint32_t waves) { + uint32_t n_followers = waves - 1; + uint32_t n_safe = entry_safe(simulate_first_entry(land)) ? n_followers : 0; + return quiesce_ready_pool(n_safe, waves); +} diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/fp8_oracle.cpp b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/fp8_oracle.cpp index aeadac94ca5d..6abea4b85fa7 100644 --- a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/fp8_oracle.cpp +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/fp8_oracle.cpp @@ -29,3 +29,42 @@ void wmma_ref_16x16x16(const uint8_t* A, const uint8_t* B, const float* C, float D[i * 16 + j] = acc; } } + +#include +// Tiered oracle comparison. bad = #elements exceeding rel*|ref|+abs_; max_rel = worst |got-ref|/|ref|. +OracleCmp oracle_compare(const float* got, const float* ref, long n, float rel, float abs_) { + OracleCmp r{true, 0, 0.0}; + for (long i = 0; i < n; ++i) { + float d = std::fabs(got[i] - ref[i]); + float thr = rel * std::fabs(ref[i]) + abs_; + double rl = (double)d / ((double)std::fabs(ref[i]) + 1e-30); + if (rl > r.max_rel) r.max_rel = rl; + if (d > thr) { r.ok = false; ++r.bad; } + } + return r; +} + +#ifdef ORACLE_SELFTEST +#include +#include +#include +int main() { + const long n = 256; + // ref ~ O(100) so the abs term doesn't dominate the rel term in the test. + std::vector ref(n), id(n), p01(n), p1(n), p5(n); + for (long i = 0; i < n; ++i) { + float v = 100.0f + 50.0f * (float)(i % 7); + ref[i] = v; id[i] = v; + p01[i] = v * 1.001f; // 0.1% + p1[i] = v * 1.01f; // 1% + p5[i] = v * 1.05f; // 5% + } + assert( oracle_compare(id.data(), ref.data(), n, 5e-3f, 1e-2f).ok); // identical -> tight ok + assert( oracle_compare(p01.data(), ref.data(), n, 5e-3f, 1e-2f).ok); // 0.1% -> tight ok + assert(!oracle_compare(p1.data(), ref.data(), n, 5e-3f, 1e-2f).ok); // 1% -> tight REJECTS + assert( oracle_compare(p1.data(), ref.data(), n, 3e-2f, 2e-2f).ok); // 1% -> loose ok + assert(!oracle_compare(p5.data(), ref.data(), n, 3e-2f, 2e-2f).ok); // 5% -> loose REJECTS + printf("ORACLE_SELFTEST all pass\n"); + return 0; +} +#endif diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/fp8_oracle.h b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/fp8_oracle.h index 826006d4c892..fd2c25b99cb2 100644 --- a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/fp8_oracle.h +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/fp8_oracle.h @@ -5,3 +5,9 @@ float fp8_e4m3_to_float(uint8_t b); // Reference D = A*B + C. A,B are 16x16 row-major e4m3 bytes; C,D are 16x16 row-major f32. void wmma_ref_16x16x16(const uint8_t* A, const uint8_t* B, const float* C, float* D); + +// Tiered oracle comparison (DSWS v2). Generalizes the inline gate fabs(got-ref) > rel*fabs(ref)+abs_. +// Tier 1 (n_kseg==1): TIGHT = {rel 5e-3, abs 1e-2} (the proven gate). +// Tier 2 (n_kseg>1): LOOSE = {rel 3e-2, abs 2e-2} (absorbs split-K reassociation). +struct OracleCmp { bool ok; long bad; double max_rel; }; +OracleCmp oracle_compare(const float* got, const float* ref, long n, float rel, float abs_); diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_dispatch.cpp b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_dispatch.cpp index bdc44865b9f9..ece9f3fa2ef2 100644 --- a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_dispatch.cpp +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_dispatch.cpp @@ -1321,9 +1321,13 @@ struct CoopResult { bool ok=false; uint32_t maxlive=0, total=0; uint64_t okFrags static CoopResult run_mbcoop(uint32_t node, const char* isaPath, bool dynvgpr, uint32_t pool, int M, int N, int K, int FM, int FN, int P, int RINGD, bool fullCheck, bool useGenDiv=false, - uint32_t reps=1, double targetSecs=0.0) { + uint32_t reps=1, double targetSecs=0.0, + int totalWaves=0, uint32_t ldsBytesOverride=0) { + // DSWS (MAD-305): totalWaves>0 launches N=NCOMP+NAFEED+NBFEED waves/WG (P=NCOMP for the C-store/oracle + // partition); ldsBytesOverride carries the larger DSWS LDS_TOTAL_DSWS. Both default to the proven 2-role + // coop behavior (1+P waves, coop LDS) so every existing caller is byte-identical. CoopResult res; - const int WAVES_LAUNCH = 1 + P; // 1 feed/claim wave + P compute waves + const int WAVES_LAUNCH = totalWaves > 0 ? totalWaves : (1 + P); // 1 feed + P compute (coop) | N (DSWS) const int TM = P*FM*16, TN = FN*16; // WG tile: P M-bands (each FM rows-of-16) x shared FN N-cols int MTL = M / TM, NTL = N / TN, NT = N / 16, KT = K / 16; uint32_t TOTAL = (uint32_t)MTL * NTL; @@ -1411,7 +1415,8 @@ static CoopResult run_mbcoop(uint32_t node, const char* isaPath, bool dynvgpr, u uint32_t vgprField = dynvgpr ? 4u : ((fatregs / 8) & 0x3fu); uint32_t rsrc1 = BuildPgmRsrc1(dynvgpr); rsrc1 = (rsrc1 & ~0x3fu) | (vgprField & 0x3fu); // LDS (Step 3 byte layout): B_ring[RINGD*FN*256] + prod_count(u32) + cons_count[P](u32) + tile_slot[3](u32) - uint32_t ldsBytes = (uint32_t)(RINGD * FN * 256 + 4 * (1 + P + 3)); + uint32_t ldsBytes = ldsBytesOverride > 0 ? ldsBytesOverride // DSWS: the full LDS_TOTAL_DSWS + : (uint32_t)(RINGD * FN * 256 + 4 * (1 + P + 3)); // coop: B-ring + prod/cons/ti/epoch/initflag ldsBytes = (ldsBytes + 0x1FFu) & ~0x1FFu; // round to 512B LDS granule uint32_t ldsU=0,ldsA=0,ldsG=0; uint32_t ldsBits = ldsRsrc2Bits(ldsBytes, &ldsU, &ldsA, &ldsG); uint32_t rsrc2 = (BuildPgmRsrc2(dynvgpr) & ~0x3eu) | (15u << RSRC2_USER_SGPR_SHIFT) | ldsBits; @@ -1458,7 +1463,7 @@ static CoopResult run_mbcoop(uint32_t node, const char* isaPath, bool dynvgpr, u uint32_t chunkHi = (base + chunkTiles < TOTAL) ? (base + chunkTiles) : TOTAL; userdata[11] = chunkHi; // s11 = this chunk's terminal tile (feed exits at claim>=hi; POOLTERM compute follows) occW[0]=0; occW[1]=0; occW[2]=0xFFFFFFFFu; occW[3]=0; occW[4]=0; occW[5]=base; *fenceW=0; occW[6]=occW[7]=occW[8]=occW[9]=occW[10]=occW[11]=occW[12]=occW[13]=0; - for (int qi=14; qi<28; ++qi) occW[qi]=0; // DIAGFINE markers: 14-21 hot-step, 22 raw-ti, 23-27 init-window + for (int qi=14; qi<52; ++qi) occW[qi]=0; // DIAGFINE: 14-21 hot-step, 22 raw-ti, 23-27 init, 28-31 A-feed, 32-36 sensors, 39 gate would-win count, 40-47 DSWS2_BAILMARK per-wave bail epochs, 48 conversion-commit count RingPlace(ring, PM4AcquireMemoryPacket(FAMILY_GFX12)); RingPlace(ring, PM4SetShaderRegPacket(mmCOMPUTE_START_X, dims, 8)); RingPlace(ring, PM4SetShaderRegPacket(mmCOMPUTE_PGM_LO, pgm, 6)); @@ -1484,10 +1489,12 @@ static CoopResult run_mbcoop(uint32_t node, const char* isaPath, bool dynvgpr, u while (true) { double now = now_s(); if (streamOn && (now - lastSnap) >= 0.2) { lastSnap = now; fprintf(stderr, "[occ +%5.2fs] live%u maxlive%u claim%u end%u | INIT adm%u tmr%u lds%u flag%u rdv%u | " - "feedPh%u compPh%u cons%u tiles%u | feed:tr%u pub%u comp:dsB%u wm%u rawTi%u | fence=%s\n", + "feedPh%u compPh%u cons%u tiles%u | feed:tr%u pub%u comp:dsB%u wm%u rawTi%u | SENS occ_b%u occ_a%u roles[%u/%u/%u] | BAIL[w1=%u w2=%u w3=%u w4=%u w5=%u w6=%u w7=%u] | fence=%s\n", now-t0, occW[0],occW[1],occW[5],occW[3], occW[23],occW[24],occW[25],occW[26],occW[27], occW[6],occW[7],occW[10],occW[11], occW[19],occW[21],occW[15],occW[17], occW[22], + occW[32],occW[33],occW[34],occW[35],occW[36], + occW[41],occW[42],occW[43],occW[44],occW[45],occW[46],occW[47], // DSWS2_BAILMARK per-wave bail epochs (wid1..7; wid0=claimer=occ[40]) (*fenceW==FENCE_VALUE)?"FIRED":"--"); fflush(stderr); } if (occW[1] > 0) admitted = true; @@ -1510,6 +1517,14 @@ static CoopResult run_mbcoop(uint32_t node, const char* isaPath, bool dynvgpr, u occW[18], occW[19], occW[20], occW[21]); fprintf(stderr, " DIAGFINE compute: prodwait[14]=%u dsloadB[15]=%u Aload[16]=%u wmma[17]=%u consRel[10]=%u rawTiMax[22]=%u\n", occW[14], occW[15], occW[16], occW[17], occW[10], occW[22]); + fprintf(stderr, " DIAGFINE Afeed : reached[28]=%u Aload[29]=%u dsstore[30]=%u publish[31]=%u | Bphase(occ6)=%u Bti(occ8)=%u\n", + occW[28], occW[29], occW[30], occW[31], occW[6], occW[8]); + fprintf(stderr, " DIAGINIT : adm[23]=%u tmr[24]=%u ldsinit[25]=%u initflag[26]=%u rdv[27]=%u\n", + occW[23], occW[24], occW[25], occW[26], occW[27]); + fprintf(stderr, " DSWS sensors : occ_b[32]=%u occ_a[33]=%u nComp[34]=%u nAfeed[35]=%u nBfeed[36]=%u gateWin[39]=%u\n", + occW[32], occW[33], occW[34], occW[35], occW[36], occW[39]); + fprintf(stderr, " DSWS2 BAILMARK : per-wave last-bailed epoch w1=%u w2=%u w3=%u w4=%u w5=%u w6=%u w7=%u (all==hung epoch => visibility; ONE stale => that wave is the STRAGGLER)\n", + occW[41], occW[42], occW[43], occW[44], occW[45], occW[46], occW[47]); repFail=true; break; } uint32_t gs=occW[2], ge=occW[3]; @@ -1546,6 +1561,15 @@ static CoopResult run_mbcoop(uint32_t node, const char* isaPath, bool dynvgpr, u while (*fenceW != FENCE_VALUE && (now_s() - tw) < 5.0) nanosleep(&ts, nullptr); } bool queueIdle = (*fenceW == FENCE_VALUE); if (!queueIdle) fprintf(stderr, " [teardown] WARN: EOP fence never fired in 5s; queue NON-IDLE -> NOT destroying (process-exit reclaims). Brick-avoidance.\n"); + // DSWS Phase-2 sensor readback (always print on clean exit -- a fast oracle finishes before the 200ms + // stream OR the timeout dump ever fires, so this is the guaranteed observation of the LAST chunk's + // final sensor sample). occ_b/occ_a must sit in [0,RINGD] (not pinned 0/RINGD); roles must = launch mix. + if (totalWaves > 0) { + fprintf(stderr, " [dsws sensors @clean-exit] occ_b=%u occ_a=%u roles[nComp=%u nAfeed=%u nBfeed=%u] gateWin(c->Afeed)[39]=%u (last chunk)\n", + occW[32], occW[33], occW[34], occW[35], occW[36], occW[39]); + fprintf(stderr, " [dsws CONVERSIONS] committed role-switches this run = %u (occ[48]; >0 => waves ADAPTIVELY switched role)\n", + occW[48]); + } res.ok = true; res.maxlive = lastMaxlive; res.total = lastTotal; res.wall = spanSum / (res.repsDone ? res.repsDone : 1); // mean per-rep span res.wallSum = spanSum; res.wallMin = spanMin; res.wallMax = spanMax; @@ -1611,6 +1635,652 @@ static CoopResult run_mbcoop(uint32_t node, const char* isaPath, bool dynvgpr, u return res; } +// --------------------------------------------------------------------------- +// MAD-305 DSWS v2 substrate launch (Task A8, PLAN_DSWS_SUBSTRATE_V2.md). Dispatches occ_kernel_dsws.s: a +// pinned wid-0 claimer broadcasts super-tiles from a pool; B-feed/A-feed/compute waves drain LDS atomic +// claim counters against resident-in-LDS A/B for the current super-tile; compute flushes fp32 partials via +// global_atomic_add_f32 (split-K segments accumulate into the SAME C cell, ksi-independent address). +// +// This MIRRORS run_mbcoop's PM4 launch/chunk/teardown/canary infrastructure byte-for-byte where the +// protocols line up (buffer alloc, VRAM guard, address bounds gate, compositor-safe chunking, fence/settle +// poll, guarded teardown, C-guard-tail canary) -- re-keyed to the v2 contract: +// * occ[20] (=occW[5], byte offset 20) is the SAME global claim counter coop uses for output-tile claims; +// v2 claims super-tile ids (`sti`) through it instead. +// * The v2 C address formula (ti*(G*FM*FN*1024) + r*(FM*FN*1024) + frag*1024 + lane*32 + e*4) is coop's +// formula with P (compute-wave count) replaced by G (super-tile M-extent) and cid replaced by r +// (claimed rowblk) -- so the oracle reuses unpack_D/oracle_compare exactly, decoding r where coop +// decoded cid. +// +// *** FIX 1 (round-table Opus+Codex pass) *** +// The original v2 kernarg contract called for COMPUTE_PGM_RSRC2.USER_SGPR=18 (s0..s17 hardware-preloaded) +// so n_kseg/TOTAL_super/magic_kseg could ride in as s15/s16/s17, written via a SECOND SET_SH_REG packet +// at register (COMPUTE_USER_DATA_0 + 16). That was never deliverable: every OTHER kernel in this harness +// uses <=15 user SGPRs, and the project's own pinned PM4 register reference (dvgpr_pm4/ref/gfx_7_2_d.h) +// defines COMPUTE_USER_DATA_0..15 ONLY (16 registers, covering s0..s15) -- there is no register defined +// for s16/s17 anywhere in this raw-PM4 path, and RESULT_WGGEMM.md's "raw-PM4 TGID is unavailable" probe +// already found SGPR delivery beyond s15 under raw-PM4 (CP-direct dispatch, MES bypassed) reads constant +// garbage, not a controllable value. On top of that, the per-chunk override of kernarg slot s16 (the +// chunk terminal bound) silently collided with s16 also being the would-be TOTAL_super kernarg -- two +// unrelated meanings on the same undeliverable slot. +// FIX 1 drops s15/s16/s17 from the kernarg contract entirely (USER_SGPR=15, s0..s14 only, ONE SET_SH_REG +// packet of 16 registers like every other proven path here, index 15 unused/padding). n_kseg is now +// DERIVED in-kernel from KT (s8); the chunk terminal bound is now MEMORY-CARRIED via occ[24] (occW[6], +// written once per chunk below) instead of riding in an undeliverable kernarg slot. See the KERNARG +// CONTRACT block at the top of occ_kernel_dsws.s for the full new scheme. +// --------------------------------------------------------------------------- +struct Dsws2Result { + bool ok = false; // true iff the run completed cleanly (fence fired, occ[0]==0) AND badFrags==0 + uint64_t okFrags = 0, badFrags = 0; + double maxRel = 0.0; + uint32_t occ0 = 0; // live-counter readback at last clean completion (expect 0) + uint32_t occClaim = 0; // occ[20] readback at last clean completion (global claim counter) + double tf = 0.0; // TFPROBE throughput (2*M*N*K / summed GPU-tick span); 0 if the bin has no tick capture + uint64_t wall = 0; // summed per-chunk GPU-tick span (occ[3]-occ[2], device busy ticks, excl host gaps) +}; + +static Dsws2Result run_dsws2(uint32_t node, const char* isaPath, + uint32_t nComp, uint32_t nAfeed, uint32_t nBfeed, + int Gv, int SEGKv, int FMc, int FNc, + int Mo, int No, int Ko, + float orel, float oabs, double freq_hz) { + Dsws2Result res; + const uint32_t WAVES_LAUNCH = nComp + nAfeed + nBfeed; + const int TMsuper = Gv*16*FMc, TN = FNc*16; // super-tile M rows, N-panel cols + if (TMsuper == 0 || TN == 0 || SEGKv <= 0 || Ko <= 0 || (Mo % TMsuper) || (No % TN) || (Ko % SEGKv)) { + fprintf(stderr, " [dsws2] geometry %dx%dx%d does not divide cleanly (G=%d SEGK=%d FM=%d FN=%d)\n", + Mo, No, Ko, Gv, SEGKv, FMc, FNc); + return res; + } + const int MTLsuper = Mo / TMsuper, NTL = No / TN, NT = No / 16, KT = Ko / 16; + const int KSEG_STEPS = SEGKv / 16; + const int n_kseg = Ko / SEGKv; + if (MTLsuper == 0 || NTL == 0 || n_kseg == 0 || KSEG_STEPS == 0) { + fprintf(stderr, " [dsws2] degenerate geometry (MTLsuper=%d NTL=%d n_kseg=%d KSEG_STEPS=%d)\n", + MTLsuper, NTL, n_kseg, KSEG_STEPS); + return res; + } + // FIX 1(k): the kernel now DERIVES n_kseg in-kernel as KT >> NKSEG_SHIFT (a plain shift) and uses it + // as a shift/mask pair (shift=ff1(n_kseg), mask=n_kseg-1) to split sti -> (t,ksi) -- both of those + // are only correct when n_kseg is a power of two (ff1 finds "the" single set bit; a non-power-of-two + // mask drops bits). Refuse rather than silently mis-decode every super-tile. + if ((n_kseg & (n_kseg - 1)) != 0) { + fprintf(stderr, " [dsws2] *** REFUSE: n_kseg=%d is not a power of two (required: the kernel derives " + "shift/mask from n_kseg via s_ff1_i32_b32) ***\n", n_kseg); + return res; + } + const uint32_t TOTAL = (uint32_t)MTLsuper * (uint32_t)NTL; // coop-compat output-tile count (C sizing) + const uint64_t TOTAL_super = (uint64_t)MTLsuper * (uint64_t)NTL * (uint64_t)n_kseg; // super-tile pool size + const uint32_t magic = (uint32_t)((0x100000000ULL + (uint64_t)NTL - 1) / (uint64_t)NTL); // ceil(2^32/NTL) + const uint32_t magicTotal = (uint32_t)((0x100000000ULL + (uint64_t)TOTAL - 1) / (uint64_t)TOTAL); // ceil(2^32/TOTAL) for KMAJOR ksi=sti/TOTAL + + static const uint8_t NICE[6] = {0x38,0x40,0x30,0xB8,0xC0,0xB0}; + std::vector Ah((size_t)Mo*Ko), Bh((size_t)Ko*No), Bshufh((size_t)Ko*No); + for (size_t i = 0; i < Ah.size(); ++i) Ah[i] = NICE[(i*7 + i/(size_t)Ko) % 6]; + for (size_t i = 0; i < Bh.size(); ++i) Bh[i] = NICE[(i*5 + (i/(size_t)No)*3) % 6]; + mbg_preshuffle_B(Bh.data(), Bshufh.data(), Ko, No); + + size_t isaLen = 0; uint8_t* isaBytes = ReadFile(isaPath, &isaLen); + if (!isaBytes) { fprintf(stderr, " [dsws2] cannot read kernel bin '%s'\n", isaPath); return res; } + GpuBuf isa = AllocGpu(node, (isaLen + 0xFFF) & ~0xFFFull, true, false); + GpuBuf occ = AllocGpu(node, 0x1000, false, true); + // ---- TRACE: per-super-tile time-series buffer (DSWS2_TRACE=1; requires a TRACE=1 kernel bin + single chunk). + // The claimer appends a 16-u32 row per super-tile (indexed by SEGCNT) capturing the live role mix, ring + // occupancy peak, conversions, and vresv. Host reads it back to CSV after the run. ---- + const bool traceOn = getenv("DSWS2_TRACE") != nullptr; + uint32_t traceMaxRows = 0; GpuBuf traceBuf{}; volatile uint32_t* traceW = nullptr; + if (traceOn) { + uint64_t want = TOTAL_super + 64; traceMaxRows = (uint32_t)(want > (1u<<21) ? (1u<<21) : want); + traceBuf = AllocGpu(node, ((uint64_t)traceMaxRows*64 + 0xFFF)&~0xFFFull, false, true); + traceW = (volatile uint32_t*)traceBuf.ptr; + if (!traceBuf.ptr) { fprintf(stderr, " [dsws2 trace] buffer alloc FAILED -> trace disabled\n"); traceW = nullptr; } + else memset((void*)traceW, 0, traceBuf.size); + } + // SAFETY PADDING (mirrors run_mbcoop): a guard tail after each operand so a small dyn off-by-one global + // access lands in mapped VRAM (observable wrong answer) instead of a page-fault brick. + uint64_t padB = (uint64_t)(getenv("ML8_COOP_PAD_MB") ? atoi(getenv("ML8_COOP_PAD_MB")) : 64) * 1024ull * 1024ull; + GpuBuf Ad = AllocGpu(node, ((Ah.size()+0xFFF)&~0xFFFull) + padB, false, true, /*deviceLocal*/true); + GpuBuf Bd = AllocGpu(node, ((Bshufh.size()+0xFFF)&~0xFFFull) + padB, false, true, /*deviceLocal*/true); + uint64_t cbytes = ((uint64_t)TOTAL * (uint64_t)((uint32_t)Gv*FMc*FNc*1024) + 0xFFF) & ~0xFFFull; // TOTAL output tiles x G*FM*FN frags x 256 f32 + GpuBuf C = AllocGpu(node, cbytes + padB, false, true, /*deviceLocal*/true); + GpuBuf fence = AllocGpu(node, 0x1000, false, true); + if (!(Ad.vram && Bd.vram && C.vram)) { + fprintf(stderr, "\n*** DSWS2 VRAM GUARD FAILED (%s): operands not device-local -> PERF/SAFETY INVALID ***\n", isaPath); + abort(); + } + // ---- ADDRESS BOUNDS GATE (MANDATORY, mirrors run_mbcoop's gate). Formulas re-derived from + // occ_kernel_dsws.s's ASTAGE/BSTAGE/.Lcompute address math (G replaces coop's P; r replaces cid). ---- + { + uint64_t Asize = (uint64_t)Ah.size(), Bsize = (uint64_t)Bshufh.size(), Csz = cbytes; + uint64_t rowblkAbsMax = (uint64_t)MTLsuper * (uint64_t)Gv - 1ull; + uint64_t Amax = rowblkAbsMax*(uint64_t)16*FMc*Ko + (uint64_t)(FMc-1)*16*Ko + + (uint64_t)(n_kseg-1)*SEGKv + (uint64_t)(KSEG_STEPS-1)*16 + + (uint64_t)15*Ko + 8 + 7; + uint64_t Bmax = (uint64_t)(NTL-1)*FNc*256 + (uint64_t)(n_kseg-1)*KSEG_STEPS*(uint64_t)NT*256 + + (uint64_t)(FNc-1)*256 + (uint64_t)(KSEG_STEPS-1)*(uint64_t)NT*256 + + (uint64_t)31*8 + 7; + uint64_t Cmax = (uint64_t)(TOTAL-1)*(uint64_t)Gv*FMc*FNc*1024 + (uint64_t)(Gv-1)*(uint64_t)FMc*FNc*1024 + + (uint64_t)(FMc*FNc-1)*1024 + (uint64_t)31*32 + (uint64_t)7*4 + 3; + bool aok = Amax < Asize, bok = Bmax < Bsize, cok = Cmax < Csz; + printf(" [dsws2 bounds] A last=%llu/%llu %s B last=%llu/%llu %s C last=%llu/%llu %s\n", + (unsigned long long)Amax,(unsigned long long)Asize, aok?"OK":"*OOB*", + (unsigned long long)Bmax,(unsigned long long)Bsize, bok?"OK":"*OOB*", + (unsigned long long)Cmax,(unsigned long long)Csz, cok?"OK":"*OOB*"); + if (!(aok && bok && cok)) { + fprintf(stderr, "\n*** DSWS2 ADDRESS BOUNDS GATE FAILED (%s) -> REFUSING to dispatch. ***\n", isaPath); + FreeGpu(fence); FreeGpu(C); FreeGpu(Bd); FreeGpu(Ad); FreeGpu(occ); FreeGpu(isa); + return res; + } + } + memcpy(isa.ptr, isaBytes, isaLen); free(isaBytes); + memcpy(Ad.ptr, Ah.data(), Ah.size()); + memcpy(Bd.ptr, Bshufh.data(), Bshufh.size()); + memset((char*)Ad.ptr + ((Ah.size()+0xFFF)&~0xFFFull), 0, padB); + memset((char*)Bd.ptr + ((Bshufh.size()+0xFFF)&~0xFFFull), 0, padB); + volatile uint32_t* occW = (volatile uint32_t*)occ.ptr; + volatile uint32_t* fenceW = (volatile uint32_t*)fence.ptr; + memset((void*)occW, 0, occ.size); // host zero-init: occ[0] live-count, occ[20] claim-counter, all reserved words + *fenceW = 0; + memset((char*)C.ptr + cbytes, 0, padB); // CANARY: zero the C guard tail (any nonzero after run = an OOB store) + // FIX 2: the kernel's compute role accumulates into C via global_atomic_add_f32 (split-K segments add + // into the SAME C cell) -- it never initializes a cell, so the host MUST zero the C data region before + // ANY dispatch (occ_kernel_dsws.s's KERNARG CONTRACT comment: "HOST MUST MEMSET C=0"). This was + // missing entirely (only the guard-tail canary was zeroed above). ONCE here, before the chunk loop -- + // NOT per chunk, so split-K segments claimed across separate chunk dispatches still accumulate. + memset((char*)C.ptr, 0, cbytes); + + Ring ring; ring.buf = AllocGpu(node, 0x10000, true, true); ring.dw = (uint32_t*)ring.buf.ptr; + ring.sizeDw = (uint32_t)(ring.buf.size / sizeof(uint32_t)); + CHECK(hsaKmtCreateQueue(node, HSA_QUEUE_COMPUTE, 100, HSA_QUEUE_PRIORITY_NORMAL, ring.buf.ptr, ring.buf.size, nullptr, &ring.res)); + + uint64_t shiftedIsa = ((uint64_t)isa.ptr) >> 8; + uint64_t occVa=(uint64_t)occ.ptr, aVa=(uint64_t)Ad.ptr, bVa=(uint64_t)Bd.ptr, cVa=(uint64_t)C.ptr, fenceVa=(uint64_t)fence.ptr; + uint32_t dims[8] = {0,0,0,(uint32_t)(WAVES_LAUNCH*32),1,1,0,0}; // NUM_THREAD_X = WAVES_LAUNCH*32 -> WAVES_LAUNCH waves/WG + uint32_t pgm[6] = {(uint32_t)shiftedIsa,(uint32_t)(shiftedIsa>>32)|(g_is_dgpu?0u:(1u<<8)),0,0,0,0}; + // DYNVGPR is baked DYNVGPR=1 into this bin's compute role (build_dsws.sh mk2 never overrides it) -- dyn-VGPR + // MUST be armed (RSRC2 bit6) to match; there is no static v2 bin to fall back to if it weren't. + const uint32_t vgprField = 4u; // lean 32-VGPR launch; compute waves s_alloc_vgpr-grow per claimed rowblk + uint32_t rsrc1 = BuildPgmRsrc1(true); rsrc1 = (rsrc1 & ~0x3fu) | (vgprField & 0x3fu); + // FIX 1 pools: DSWS2_FLOW=1 -> N-deep flow pool (LDS_TOTAL_FLOW = 256 + POOL_N*OPSTRIDE); + // DSWS2_RING=1 -> D=2 ring (33024); neither -> single-slot occ_kernel_dsws.s (16640, byte-identical). + uint32_t poolSlots = 1u; + if (getenv("DSWS2_FLOW")) poolSlots = getenv("FLOW_POOL_N") ? (uint32_t)atoi(getenv("FLOW_POOL_N")) : 3u; + else if (getenv("DSWS2_RING")) poolSlots = 2u; + uint32_t operandBytes = (uint32_t)(FNc*16*SEGKv) + (uint32_t)((uint32_t)Gv*16*FMc*SEGKv); // per-slot = 16384 + // FIX 1 STAGGER: flow adds a per-rowblk fp32 reduction accumulator pool (ACC_N banks x FM*FN*1024B) AFTER + // the operand pool. Must match the kernel's ACC_BASE/ACC_STRIDE/ACC_N (DSWS2_ACC_N, default 1; 0 for ring/single). + uint32_t accN = getenv("DSWS2_FLOW") ? (getenv("DSWS2_ACC_N") ? (uint32_t)atoi(getenv("DSWS2_ACC_N")) : 1u) : 0u; + uint32_t accBytes = accN * (uint32_t)(FMc*FNc*1024); + uint32_t ldsBytesRaw = 256u + poolSlots * operandBytes + accBytes; // flow POOL3/ACC1:57600 POOL2/ACC2:49408 + uint32_t ldsU=0, ldsA=0, ldsG=0; uint32_t ldsBits = ldsRsrc2Bits(ldsBytesRaw, &ldsU, &ldsA, &ldsG); + uint32_t rsrc2 = (BuildPgmRsrc2(true) & ~0x3eu) | (15u << RSRC2_USER_SGPR_SHIFT) | ldsBits; // USER_SGPR=15 (FIX 1h: dropped s15..s17) + uint32_t rsrc[2] = {rsrc1, rsrc2}; + printf(" [dsws2] %dx%dx%d super-tile=%dx%d (G=%d FM=%d FN=%d) TOTAL=%u TOTAL_super=%llu n_kseg=%d " + "waves/WG=%u(=%uc%ua%ub) LDS=%uB(alloc %uB) VGPR~%u dyn=1 RSRC2=0x%x\n", + Mo,No,Ko, TMsuper,TN, Gv,FMc,FNc, TOTAL, (unsigned long long)TOTAL_super, n_kseg, + WAVES_LAUNCH, nComp,nAfeed,nBfeed, ldsBytesRaw, ldsA, vgprField*8, rsrc2); + + // FIX 1(i): 15 kernargs (s0..s14) only -- n_kseg/TOTAL_super/magic_kseg dropped (derived in-kernel / + // memory-carried via occ[24], see occ_kernel_dsws.s KERNARG CONTRACT). Array is still 16 wide to match + // every other proven path's single 16-register SET_SH_REG packet; index 15 is unused padding (lands + // in the hardware's TGID_X slot, which this kernel does not read). + uint32_t userdata[16] = { + (uint32_t)occVa,(uint32_t)(occVa>>32), (uint32_t)aVa,(uint32_t)(aVa>>32), // s0:1 occ, s2:3 A + (uint32_t)bVa,(uint32_t)(bVa>>32), (uint32_t)cVa,(uint32_t)(cVa>>32), // s4:5 Bshuf, s6:7 C + (uint32_t)KT, (uint32_t)Ko, (uint32_t)(NT*256), TOTAL, // s8 KT, s9 K(bytes/row), s10 NTx256, s11 TOTAL + magic, (uint32_t)NTL, (uint32_t)(FNc*256), 0u }; // s12 magic, s13 NTL, s14 FNx256, [15] unused + uint32_t dispInit = BuildDispatchInitiator(); + + const uint32_t poolD = getenv("ML8_POOL") ? (uint32_t)atoi(getenv("ML8_POOL")) : 64u; + const uint32_t pool = poolD < 64u ? poolD : 64u; + const char* ydis = getenv("ML8_YIELD_DISABLE"); bool yieldOff = ydis && ydis[0]=='1'; + int yieldMs = getenv("ML8_YIELD_MS") ? atoi(getenv("ML8_YIELD_MS")) : 5; + if (yieldMs < 0) yieldMs = 0; + double yieldEvery = (getenv("ML8_YIELD_EVERY_MS") ? atoi(getenv("ML8_YIELD_EVERY_MS")) : 100) / 1000.0; // proven run_mbgemm cadence + if (yieldEvery <= 0.0) yieldEvery = 0.1; + const double timeoutS = 25.0; + // COMPOSITOR-SAFE CHUNKING (mirrors run_mbcoop): bound each dispatch to ML8_COOP_CHUNK super-tiles + // (claim starts at occ[20]=base, terminal bound occ[24]/occW[6]=chunkHi -- FIX 1(j), memory-carried + // since there is no deliverable kernarg slot for it) and yield between dispatches. + uint64_t chunkTilesEnv = getenv("ML8_COOP_CHUNK") ? (uint64_t)atoll(getenv("ML8_COOP_CHUNK")) : 0ull; + // FIX 1 STAGGER: the flow write-once kernel claims occ[20] as whole TILES (a WG owns a tile's n_kseg + // segments so its per-WG LDS banks sum a full tile); every other dsws2 path claims super-tiles. + const uint64_t claimTotal = getenv("DSWS2_FLOW") ? (uint64_t)TOTAL : TOTAL_super; + uint64_t chunkTiles = (chunkTilesEnv == 0ull || chunkTilesEnv > claimTotal) ? claimTotal : chunkTilesEnv; + uint64_t nChunks = (claimTotal + chunkTiles - 1ull) / chunkTiles; + double chunkMaxS = getenv("ML8_COOP_CHUNK_MAXS") ? atof(getenv("ML8_COOP_CHUNK_MAXS")) : 0.75; + if (chunkTiles < TOTAL_super) printf(" [dsws2] compositor-safe: %llu super-tiles/dispatch x %llu chunks (yield %dms between; abort chunk > %.2fs)\n", + (unsigned long long)chunkTiles, (unsigned long long)nChunks, yieldMs, chunkMaxS); + bool streamOn = getenv("ML8_COOP_STREAM") != nullptr; + uint32_t reslim[1]={0}, tmpring[1]={0}, restart[4]={0,0,0,0}; + bool allok = true; uint32_t lastOcc0 = 0, lastOcc20 = 0; uint32_t totalConv = 0; // occ[48] conv-commit count, summed across chunks (reset per chunk) + uint64_t sumSpan = 0; uint32_t spanChunks = 0; bool tfMissing = false; // TFPROBE: summed per-chunk GPU-tick span (occ[3]-occ[2]); tfMissing => bin has no tick capture + // SUSTAINED (DSWS2_REPS>1): re-run the whole chunked GEMM back-to-back, buffers reused, C re-zeroed per rep + // (split-K atomic-adds into C, so a repeated pass without reset would double it). Spans sum across ALL + // reps -> TF is over reps*(2MNK) work / total busy ticks (warm-clock steady state, not a cold ms blip). + // Per-rep span min/max -> the TF spread (glass-flat vs jittery), the trustworthiness signal. + uint32_t dswsReps = getenv("DSWS2_REPS") ? (uint32_t)atoi(getenv("DSWS2_REPS")) : 1u; + if (dswsReps < 1u) dswsReps = 1u; + double dswsTarget = getenv("DSWS2_TARGET_SECS") ? atof(getenv("DSWS2_TARGET_SECS")) : 0.0; // >0: rep until this many wall-secs + double repT0 = now_s(); + uint64_t repSpanMin = ~0ull, repSpanMax = 0; uint32_t repsDone = 0; + for (uint32_t rep = 0; ; ++rep) { + if (dswsTarget > 0.0) { if (rep > 0 && (now_s() - repT0) >= dswsTarget) break; } // duration-bounded + else { if (rep >= dswsReps) break; } // count-bounded + if (rep > 0) memset((char*)C.ptr, 0, cbytes); // split-K accumulation reset before each repeated pass + uint64_t repSpanBase = sumSpan; + // ML8_CHUNK_DIAG: per-chunk wall + STAGINSTR delta (coast/computed/feed/grow-fail). occ[70..73] are + // OUTSIDE the per-chunk memset (occ[0..63]) so they accumulate; snapshot before each chunk for the delta. + // A slow chunk with grow-fail/coast spiking + computed crawling == VGPR-starvation churn (compositor + // held the SIMD pool during the inter-chunk yield); a slow chunk that is mostly `computed` == real work. + const bool chunkDiag = getenv("ML8_CHUNK_DIAG") != nullptr; + for (uint64_t base = 0; base < claimTotal; base += chunkTiles) { + uint64_t chunkHi = (base + chunkTiles < claimTotal) ? (base + chunkTiles) : claimTotal; + uint32_t diagPrevCoast = occW[70], diagPrevComp = occW[71], diagPrevFeed = occW[72], diagPrevGF = occW[73]; + memset((void*)occW, 0, 0x100); // re-zero the control region (occ[0] live, occ[20] claim, reserved) each chunk + occW[5] = (uint32_t)base; // occ[20] (=occW[5]) claim counter starts at this chunk's base sti + occW[6] = (uint32_t)chunkHi; // FIX 1(j): occ[24] (=occW[6]) = this chunk's terminal sti bound (memory-carried) + occW[2] = 0xFFFFFFFFu; // TFPROBE: min-sentinel for the entry-tick atomic_min (occ[2]); occ[3] stays 0 (max) + occW[62] = magicTotal; // KMAJOR: magic(TOTAL) for the ksi=sti/TOTAL decode (ignored unless KMAJOR bin) + if (traceW) { uint64_t tva=(uint64_t)traceBuf.ptr; // TRACE: (re-)publish buffer VA + cap (memset above wiped occ[52..54]) + occW[52]=(uint32_t)tva; occW[53]=(uint32_t)(tva>>32); occW[54]=traceMaxRows; } + *fenceW = 0; + RingPlace(ring, PM4AcquireMemoryPacket(FAMILY_GFX12)); + RingPlace(ring, PM4SetShaderRegPacket(mmCOMPUTE_START_X, dims, 8)); + RingPlace(ring, PM4SetShaderRegPacket(mmCOMPUTE_PGM_LO, pgm, 6)); + RingPlace(ring, PM4SetShaderRegPacket(mmCOMPUTE_PGM_RSRC1, rsrc, 2)); + RingPlace(ring, PM4SetShaderRegPacket(mmCOMPUTE_RESOURCE_LIMITS, reslim, 1)); + RingPlace(ring, PM4SetShaderRegPacket(mmCOMPUTE_TMPRING_SIZE, tmpring, 1)); + RingPlace(ring, PM4SetShaderRegPacket(mmCOMPUTE_RESTART_X, restart, 4)); + RingPlace(ring, PM4SetShaderRegPacket(mmCOMPUTE_USER_DATA_0, userdata, 16)); // s0..s14 (FIX 1i: ONE packet, like every proven path) + RingPlace(ring, PM4DispatchDirectPacket(pool * (uint32_t)(WAVES_LAUNCH*32), 1, 1, dispInit)); // grid = pool WGs + // NOTE: CS_PARTIAL_FLUSH here was tried (2026-07-05) and STALLS too -- the CP-level terminal-store/wave + // drain is genuinely stuck at 16 waves; no host packet (drop-EOP, drop-ACQUIRE, PARTIAL_FLUSH) fixes it. + // The fix is shader-side (terminal store / dyn-VGPR wave retirement). See KG. + // Codex fix (2026-07-05): EOP RELEASE_MEM only on the FINAL chunk. The terminal-store-drain quirk stalls + // the EOP fence with no post-kernel traffic; a per-chunk stalled EOP sits in the in-order queue and + // BLOCKS the next chunk's DispatchDirect from ever launching (that was the chunk1 "occ20 stuck at base, + // 0 claims" hang). Non-last chunks serialize via the kernel-done gate (occ0==0 + settle) in the poll + // loop below; only the last chunk arms the real fence (guarded teardown handles it if it too stalls). + const bool lastChunk = (chunkHi >= claimTotal); + if (lastChunk) RingPlace(ring, PM4ReleaseMemoryPacket(FAMILY_GFX12, true, fenceVa, FENCE_VALUE)); + double t0 = now_s(); RingSubmit(ring); + bool done = false, admitted = false; double lastSnap = t0, lastYield = t0; + // Complete on the KERNEL'S OWN done-signal (occ0==0 + settle), NOT the EOP fence. The terminal C store's + // s_endpgm implicit drain stalls the EOP fence on this raw-PM4 path (COOP_STATUS.md; the proven coop + // path completes the same way via ML8_COOP_NOFENCE). The store IS issued -- the settle lets it land + // before the oracle reads C (a stale read fails the oracle, never a false CLEAN). The guarded teardown + // below still refuses to destroy a non-idle queue (brick-avoidance), so a lingering wave is reclaimed + // by process-exit, never a forced destroy. A real hang still trips the timeoutS bail -> forensics. + uint32_t lastEnd = 0; double lastEndChange = t0; + double settle = getenv("DSWS2_SETTLE") ? atof(getenv("DSWS2_SETTLE")) : 0.30; + while (true) { double now = now_s(); + // COMPOSITOR YIELD (proven run_mbgemm mechanism, was MISSING on the flow path): hand the gfx ring + // (Hyprland) an unconditional render+VGPR window every yieldEvery ms DURING the wait -- so a long or + // stuck dispatch can't starve the desktop's gfx ring into a ring-timeout MODE1 reset. Host sleep + // only; never enters the in-kernel TF span. This is the "let hyprland through every so often" logic. + if (!yieldOff && yieldMs > 0 && (now - lastYield) >= yieldEvery) { + struct timespec yts = { yieldMs/1000, (long)(yieldMs%1000)*1000000L }; + nanosleep(&yts, nullptr); lastYield = now_s(); + } + if (streamOn && (now - lastSnap) >= 0.2) { lastSnap = now; + fprintf(stderr, "[dsws2 +%5.2fs] occ0(live)=%u occ20(claim)=%u fence=%s\n", + now-t0, occW[0], occW[5], (*fenceW==FENCE_VALUE)?"FIRED":"--"); fflush(stderr); } + if (occW[0] > 0) admitted = true; + uint32_t end = occW[3]; if (end != lastEnd) { lastEnd = end; lastEndChange = now; } + bool ff = (*fenceW == FENCE_VALUE); + // done = fence fired, OR kernel-done: occ0==0 (all counted waves retired) + a wave stamped its exit + // tick (occ[3]!=0) + settled (store landed). Matches the coop gate. + if (admitted && occW[0]==0 && (ff || (end != 0 && (now - lastEndChange) > settle))) { done = true; break; } + if (now - t0 > timeoutS) break; + } + if (!done) { + fprintf(stderr, "\n*** DSWS2 TIMEOUT (chunk base=%llu hi=%llu): occ0(live)=%u occ20(claim)=%u fence=%s ***\n", + (unsigned long long)base, (unsigned long long)chunkHi, occW[0], occW[5], (*fenceW==FENCE_VALUE)?"FIRED":"--"); + // EMERGENT-economy timeout forensics: WHERE did it stall? (all slots stream live during the run) + // computed>0 & rising -> compute progressed (slow/contention); computed~0 & coast huge -> LIVELOCK. + fprintf(stderr, " [timeout forensics] residentPeak occ[1]=%u fatPeak occ[58]=%u fatResidual occ[57]=%u alllive-net occ[60]=%u (TRACE: >0 w/ occ0=0 == waves stuck PRE-LIVE at .Lflow_alloc)\n", + occW[1], occW[58], occW[57], occW[60]); + fprintf(stderr, " [timeout forensics] STAGINSTR coast occ[70]=%u computed occ[71]=%u feed-stages occ[72]=%u grow-fail occ[73]=%u\n", + occW[70], occW[71], occW[72], occW[73]); + allok = false; break; + } + if (chunkDiag) { + double cwall = now_s() - t0; + fprintf(stderr, " [chunk diag] base=%llu hi=%llu wall=%.3fs claim=%u STAGINSTR d: coast=%u computed=%u feed=%u grow-fail=%u%s\n", + (unsigned long long)base, (unsigned long long)chunkHi, cwall, occW[5], + occW[70]-diagPrevCoast, occW[71]-diagPrevComp, occW[72]-diagPrevFeed, occW[73]-diagPrevGF, + (cwall > 0.5) ? " <-- SLOW" : ""); + fflush(stderr); + } + lastOcc0 = occW[0]; lastOcc20 = occW[5]; totalConv += occW[48]; // accumulate this chunk's role-switch commits (DIAG conv counter) + // TFPROBE: read this chunk's device-busy span BEFORE the next iteration re-zeros occ[2]/occ[3]. A stamped + // chunk has occ[2] != 0xFFFFFFFF (entry min written) AND occ[3] != 0 (exit max written). Sum spans across + // chunks -> total GPU busy ticks for the whole GEMM (host inter-chunk gaps excluded). If unstamped, the + // bin lacks TFPROBE tick capture -> flag and skip (no bogus TF from the 0xFFFFFFFF sentinel). + { uint32_t gs = occW[2], ge = occW[3]; + if (gs != 0xFFFFFFFFu && ge != 0) { + sumSpan += (ge >= gs) ? (uint64_t)(ge - gs) : ((uint64_t)ge + 0x100000000ull - (uint64_t)gs); + spanChunks++; + } else tfMissing = true; } + if (!yieldOff && yieldMs > 0) { struct timespec ts = { yieldMs/1000, (long)(yieldMs%1000)*1000000L }; nanosleep(&ts, nullptr); } + if ((now_s() - t0) > chunkMaxS) { + fprintf(stderr, " [dsws2] WARN chunk @base%llu wall %.2fs > %.2fs cap -> ABORT remaining chunks\n", + (unsigned long long)base, now_s()-t0, chunkMaxS); + allok = false; break; + } + } + if (!allok) break; // rep loop: bail on any chunk failure/timeout + { uint64_t rs = sumSpan - repSpanBase; // this rep's busy-tick span (across its chunks) + if (rs > 0) { if (rs < repSpanMin) repSpanMin = rs; if (rs > repSpanMax) repSpanMax = rs; repsDone++; } } + } // ---- end SUSTAINED rep loop ---- + if (!allok) { + fprintf(stderr, " [teardown] dsws2 run did not complete cleanly -> NOT destroying queue (brick-avoidance; process-exit reclaims).\n"); + return res; + } + { double tw = now_s(); struct timespec ts = {0, 2000000L}; + while (*fenceW != FENCE_VALUE && (now_s() - tw) < 5.0) nanosleep(&ts, nullptr); } + bool queueIdle = (*fenceW == FENCE_VALUE); + if (!queueIdle) fprintf(stderr, " [teardown] WARN: EOP fence never fired in 5s; queue NON-IDLE -> NOT destroying (process-exit reclaims).\n"); + res.occ0 = lastOcc0; res.occClaim = lastOcc20; + printf(" [dsws2 alllive-net] occ[60]=%u peak-resident occ[1]=%u (TRACE build: occ[60]>0 w/ occ0=0 == waves stuck PRE-LIVE at .Lflow_alloc)\n", occW[60], occW[1]); + printf(" [dsws2 completion] occ[0](live)=%u (0=clean) occ[20](claim)=%u (NOTE: with pool>=1, each WG's pinned\n" + " claimer makes exactly one extra terminal over-claim past the bound, so the expected clean value is\n" + " chunkHi(last chunk)+#WGs-that-raced-the-last-claim, NOT exactly TOTAL_super=%llu -- treat occ[20] as a\n" + " 'did every WG's claimer reach a terminal claim' liveness signal, occ[0]==0 as the real completion gate)\n", + lastOcc0, lastOcc20, (unsigned long long)TOTAL_super); + printf(" [dsws2 CONVERSIONS] committed role-switches (occ[48], summed over chunks) = %u (>0 => waves ADAPTIVELY switched role)\n", totalConv); + { // STAGINSTR write-once diag: feed-vs-compute-bound. occ[70]=coast iters, [71]=computed, [72]=feed stages. + uint32_t coastIt = occW[70], compIt = occW[71], feedIt = occW[72], growFail = occW[73]; + if (coastIt + compIt > 0) { + double starve = 100.0 * (double)coastIt / (double)(coastIt + compIt); + printf(" [dsws2 STAGINSTR] compute-wave iters: coast=%u computed=%u feed-stages=%u grow-fail=%u\n" + " -> coast-frac=%.1f%% grow-fail(stagger-repulsion)=%u (%.1f%% of coasts)\n", + coastIt, compIt, feedIt, growFail, starve, growFail, + coastIt > 0 ? 100.0 * (double)growFail / (double)coastIt : 0.0); + } + } + if (traceOn) { + uint32_t fatPeak = occW[58], fatResidual = occW[57]; // FATMAX / FATLIVE (should end ~0 if balanced) + printf(" [dsws2 VGPR-BUDGET PROBE] peak concurrent FAT compute waves (occ[58]) = %u -> ~%u VGPR in flight (x NFV=112)" + " [residual live=%d]\n", fatPeak, fatPeak*112u, (int)fatResidual); + printf(" (per-SIMD B estimate = peak/128 SIMDs x 112; raise DSWS2_BUDGET/pool until this plateaus or s_alloc stalls)\n"); + uint32_t peakWaves = occW[1]; // occ[1] = peak concurrent RESIDENT waves (all roles), vs 2048 HW ceiling (16/SIMD) + printf(" [dsws2 OCCUPANCY] peak concurrent resident waves (occ[1]) = %u of 2048 HW max (%.1f%%, %.2f/SIMD) " + "launched = %u WGs x %u waves = %u\n", peakWaves, peakWaves/2048.0*100.0, peakWaves/128.0, + pool, WAVES_LAUNCH, pool*WAVES_LAUNCH); + } + + // ---- TFPROBE THROUGHPUT: total useful work / total device-busy span. Work = 2*M*N*K (split-K independent; + // the n_kseg segments reduce the SAME K, so total MACs = M*N*K regardless of how K is partitioned). Span + // = summed per-chunk (occ[3]-occ[2]) GPU ticks -> the on-chip busy time, immune to host launch/fence/poll + // overhead (the reason a host wall-clock is useless at these <1ms shapes). TF = 2*M*N*K*freq / span / 1e12. ---- + if (spanChunks > 0 && sumSpan > 0) { + res.wall = sumSpan; + double reps_eff = (repsDone > 0) ? (double)repsDone : 1.0; // work = reps_eff * (2MNK); span = sum over reps + double workAll = 2.0 * (double)Mo * (double)No * (double)Ko * reps_eff; + res.tf = workAll * freq_hz / (double)sumSpan / 1e12; + double perRepWork = 2.0 * (double)Mo * (double)No * (double)Ko; + double tfHi = (repSpanMax > 0) ? perRepWork * freq_hz / (double)repSpanMin / 1e12 : res.tf; // min span -> peak TF + double tfLo = (repSpanMax > 0) ? perRepWork * freq_hz / (double)repSpanMax / 1e12 : res.tf; // max span -> trough TF + printf(" [dsws2 THROUGHPUT] %dx%dx%d TF=%.1f (%.1f%% of 307 TF fp8 peak) span=%llu ticks / %u chunk(s) @ %.0f MHz\n", + Mo, No, Ko, res.tf, res.tf / 307.0 * 100.0, (unsigned long long)sumSpan, spanChunks, freq_hz / 1e6); + if (repsDone > 1) + printf(" [dsws2 SUSTAINED] reps=%u TF=%.1f mean (per-rep %.1f-%.1f, spread %.1f%%) -- glass-flat=trustworthy\n", + repsDone, res.tf, tfLo, tfHi, (tfHi > 0 ? (tfHi - tfLo) / tfHi * 100.0 : 0.0)); + } else { + printf(" [dsws2 THROUGHPUT] n/a -- bin has no TFPROBE tick capture (occ[2]/occ[3] unstamped%s). " + "Rebuild the bin with -Wa,-defsym,TFPROBE=1 to measure TF.\n", tfMissing ? "" : "; no chunks completed"); + } + + // ---- PHASEPROBE: in-kernel per-phase tick breakdown of the COMPUTE wave (the critical path). + // Accumulators at occ[64..69] (bytes 256..276) live ABOVE the 0x100 per-chunk memset -> they SUM + // over the whole run. u32 slots -> keep PHASEPROBE runs short (single/few passes) to avoid wrap; the + // DISTRIBUTION (%) is stable regardless. Ticks are summed across ALL compute waves (aggregate time + // in each phase), so % shows WHERE compute-wave time goes -- measured, not inferred. ---- + { + const char* phName[6] = {"FOLLOW_WAIT","STAGE_WAIT","GROW","WMMA","FLUSH","SHRINK"}; + uint64_t ph[6] = {0,0,0,0,0,0}, phSum = 0; + for (int i = 0; i < 6; i++) { ph[i] = (uint64_t)occW[64 + i]; phSum += ph[i]; } + if (phSum > 0) { + printf(" [dsws2 PHASE breakdown] compute-wave ticks by phase (summed over all waves+chunks):\n"); + printf(" %-12s %14s %6s %s\n", "phase", "ticks", "share", "what"); + const char* phWhat[6] = {"idle: waiting for claimer to publish next super-tile", + "idle: waiting for A/B feeds to stage operands", + "dyn-VGPR grow 32->112 (+ rowblk claim)", + "the actual fp8 WMMA compute", + "split-K C reduction (global_atomic_add_f32)", + "dyn-VGPR shrink 112->32"}; + for (int i = 0; i < 6; i++) { + double pct = 100.0 * (double)ph[i] / (double)phSum; + char bar[41]; int nb = (int)(pct / 2.5 + 0.5); if (nb > 40) nb = 40; + for (int k = 0; k < nb; k++) bar[k] = '#'; bar[nb] = 0; + printf(" %-12s %14llu %5.1f%% %-40s %s\n", phName[i], + (unsigned long long)ph[i], pct, bar, phWhat[i]); + } + printf(" %-12s %14llu\n", "TOTAL", (unsigned long long)phSum); + } + } + + // ---- TRACE dump: read the per-super-tile rows back to CSV (real disk). Rows are indexed by SEGCNT + // (1-based), so row 0 stays zero; skip all-zero (unwritten) rows. Single-chunk runs only (chunked + // runs reset SEGCNT per chunk -> rows overwrite). ---- + if (traceW) { + const char* csv = getenv("DSWS2_TRACE_CSV"); + char path[600]; + if (!csv) { snprintf(path, sizeof path, "/home/kmbandy/dsws_gpu_logs/trace_%dx%dx%d.csv", Mo, No, Ko); csv = path; } + FILE* tf = fopen(csv, "w"); + if (tf) { + fprintf(tf, "row,tick_lo,segcnt,epoch,nComp,nAfeed,nBfeed,occA,occB,convCount,vresv,sti,quiesce,tick_hi,chunkHi,wg_id\n"); + uint32_t nrows = 0; + for (uint32_t r = 0; r < traceMaxRows; ++r) { + const volatile uint32_t* row = traceW + (size_t)r*16; + bool nz = false; for (int i = 0; i < 16; ++i) if (row[i]) { nz = true; break; } + if (!nz) continue; + fprintf(tf, "%u,%u,%u,%u,%u,%u,%u,%u,%u,%u,%u,%u,%u,%u,%u,%u\n", + r, row[0],row[1],row[2],row[3],row[4],row[5],row[6],row[7],row[8],row[9],row[10],row[11],row[12],row[13],row[14]); + ++nrows; + } + fclose(tf); + printf(" [dsws2 trace] wrote %u rows -> %s\n", nrows, csv); + } else fprintf(stderr, " [dsws2 trace] cannot open '%s' for write\n", csv); + } + + // ---- CANARY scan (verbatim pattern from run_mbcoop): did any C store land past the data region? ---- + { const uint32_t* tail = (const uint32_t*)((const char*)C.ptr + cbytes); + uint64_t words = padB/4, firstNZ = ~0ull, lastNZ = 0, nzCount = 0; + for (uint64_t w=0; wr and P->G (the v2 G-extent). ---- + const float* Cf = (const float*)C.ptr; + // SAMPLED ORACLE: full-K CPU reference is ~O(TOTAL*G*KT) MACs -> minutes at training M (640 tiles, KT=128). + // DSWS2_ORACLE_STRIDE>1 checks every Nth output tile (still every rowblk/frag within it) so big-shape + // perf runs verify a representative subset cheaply. Default 1 = full check (unchanged for small shapes). + int ostride = getenv("DSWS2_ORACLE_STRIDE") ? atoi(getenv("DSWS2_ORACLE_STRIDE")) : 1; + if (ostride < 1) ostride = 1; + int nTilesChecked = 0; + for (int ti = 0; ti < (int)TOTAL; ti += ostride) { + nTilesChecked++; + int mblk = ti / NTL, tcol = ti % NTL; + for (int r = 0; r < Gv; ++r) { + for (int mi = 0; mi < FMc; ++mi) for (int ni = 0; ni < FNc; ++ni) { + int rowbase = mblk*TMsuper + r*(FMc*16) + mi*16; + int colbase = tcol*TN + ni*16; + float Cacc[256]; for (int i=0;i<256;i++) Cacc[i]=0.f; + uint8_t Ablk[256], Bblk[256]; float Dout[256]; + for (int kt = 0; kt < KT; ++kt) { + for (int i=0;i<16;i++) for (int j=0;j<16;j++) { + Ablk[i*16+j] = Ah[(size_t)(rowbase+i)*Ko + (kt*16+j)]; + Bblk[i*16+j] = Bh[(size_t)(kt*16+i)*No + (colbase+j)]; + } + wmma_ref_16x16x16(Ablk, Bblk, Cacc, Dout); + for (int i=0;i<256;i++) Cacc[i]=Dout[i]; + } + int frag = mi*FNc + ni; + size_t foff = (size_t)ti*(size_t)((uint32_t)Gv*FMc*FNc*256) + (size_t)r*(size_t)(FMc*FNc*256) + (size_t)frag*256; + float D[256]; unpack_D(Cf + foff, D); + OracleCmp cmp = oracle_compare(D, Cacc, 256, orel, oabs); + if (cmp.ok) res.okFrags++; else res.badFrags++; + if (cmp.max_rel > res.maxRel) res.maxRel = cmp.max_rel; + } + } + } + printf(" [dsws2 oracle] ok=%llu bad=%llu max_rel=%.4g tier=%s (rel=%.0e abs=%.0e) [%d/%u tiles checked, stride=%d]\n", + (unsigned long long)res.okFrags, (unsigned long long)res.badFrags, res.maxRel, + n_kseg==1?"TIGHT":"LOOSE", orel, oabs, nTilesChecked, TOTAL, ostride); + + if (queueIdle) { + CHECK(hsaKmtDestroyQueue(ring.res.QueueId)); + if (traceOn && traceBuf.ptr) FreeGpu(traceBuf); + FreeGpu(ring.buf); FreeGpu(fence); FreeGpu(C); FreeGpu(Bd); FreeGpu(Ad); FreeGpu(occ); FreeGpu(isa); + } + res.ok = allok && (res.badFrags == 0); + return res; +} + +// =========================================================================================== +// run_grind: the NON-split-K CONTROL. Launches occ_kernel_grind.bin (1 wave = 1 WG owns one +// FM*16 x FN*16 output tile, full-K in registers, writes C ONCE with plain global_store, NO +// split-K, NO C-atomic reduction). Directly comparable to run_dsws2's TF: does avoiding split-K's +// 32x C-write amplification beat the split-K kernel's ~2.1 TF? Static VGPR (the anti-moat control). +// Self-instrumented tick span (occ[2]/occ[3]) + maxlive (occ[1]) live in the kernel already. +// =========================================================================================== +struct GrindResult { bool ok=false; uint64_t okFrags=0, badFrags=0; double maxRel=0.0, tf=0.0; uint64_t wall=0; uint32_t maxlive=0, occ0=0; }; + +static GrindResult run_grind(uint32_t node, const char* isaPath, int FMc, int FNc, + int Mo, int No, int Ko, float orel, float oabs, double freq_hz) { + GrindResult res; + const int TM = FMc*16, TN = FNc*16; // grind output tile = FM*16 x FN*16 (32x64 @ 2x4) + if (TM==0 || TN==0 || Ko<=0 || (Mo%TM) || (No%TN) || (Ko%16)) { + fprintf(stderr, " [grind] geometry %dx%dx%d not tile-aligned (TM=%d TN=%d, K%%16)\n", Mo,No,Ko,TM,TN); return res; } + const int MTL = Mo/TM, NTL = No/TN, KT = Ko/16, NT = No/16; + const uint32_t TOTAL = (uint32_t)MTL*(uint32_t)NTL; // grind tile count (finer: no G, no ksi) + const uint32_t magic = (uint32_t)((0x100000000ULL + (uint64_t)NTL - 1)/(uint64_t)NTL); + const int KCHUNK = getenv("GRIND_KCHUNK") ? atoi(getenv("GRIND_KCHUNK")) : 4; + + static const uint8_t NICE[6] = {0x38,0x40,0x30,0xB8,0xC0,0xB0}; + std::vector Ah((size_t)Mo*Ko), Bh((size_t)Ko*No), Bshufh((size_t)Ko*No); + for (size_t i=0;i abort ***\n"); abort(); } + // address bounds gate: last A/B/C element the kernel can touch must be in-buffer. + { uint64_t Amax = (uint64_t)(MTL*FMc*16-1)*Ko + (uint64_t)(KT-1)*16 + 8 + 7; // max row=(MTL*FM*16-1), max kcol=(KT-1)*16+8+7 + uint64_t Bmax = (uint64_t)(NTL-1)*FNc*256 + (uint64_t)(KT-1)*(uint64_t)NT*256 + (uint64_t)(FNc-1)*256 + (uint64_t)31*8 + 7; + uint64_t Cmax = (uint64_t)(TOTAL-1)*(uint64_t)(FMc*FNc*1024) + (uint64_t)(FMc*FNc-1)*1024 + (uint64_t)31*32 + 7*4 + 3; + bool aok=Amax REFUSE ***\n"); + FreeGpu(fence);FreeGpu(C);FreeGpu(Bd);FreeGpu(Ad);FreeGpu(occ);FreeGpu(isa); return res; } } + memcpy(isa.ptr,isaBytes,isaLen); free(isaBytes); + memcpy(Ad.ptr,Ah.data(),Ah.size()); memcpy(Bd.ptr,Bshufh.data(),Bshufh.size()); + memset((char*)Ad.ptr+((Ah.size()+0xFFF)&~0xFFFull),0,padB); + memset((char*)Bd.ptr+((Bshufh.size()+0xFFF)&~0xFFFull),0,padB); + volatile uint32_t* occW=(volatile uint32_t*)occ.ptr; volatile uint32_t* fenceW=(volatile uint32_t*)fence.ptr; + memset((void*)occW,0,occ.size); *fenceW=0; + memset((char*)C.ptr,0,cbytes+padB); // grind writes each cell once; zero anyway (canary tail + clean) + + Ring ring; ring.buf=AllocGpu(node,0x10000,true,true); ring.dw=(uint32_t*)ring.buf.ptr; ring.sizeDw=(uint32_t)(ring.buf.size/sizeof(uint32_t)); + CHECK(hsaKmtCreateQueue(node,HSA_QUEUE_COMPUTE,100,HSA_QUEUE_PRIORITY_NORMAL,ring.buf.ptr,ring.buf.size,nullptr,&ring.res)); + uint64_t shiftedIsa=((uint64_t)isa.ptr)>>8; + uint64_t occVa=(uint64_t)occ.ptr,aVa=(uint64_t)Ad.ptr,bVa=(uint64_t)Bd.ptr,cVa=(uint64_t)C.ptr,fenceVa=(uint64_t)fence.ptr; + const uint32_t WAVES_LAUNCH=1u; // 1 wave/WG (the control) + uint32_t dims[8]={0,0,0,WAVES_LAUNCH*32,1,1,0,0}; + uint32_t pgm[6]={(uint32_t)shiftedIsa,(uint32_t)(shiftedIsa>>32)|(g_is_dgpu?0u:(1u<<8)),0,0,0,0}; + const uint32_t vgprField=15u; // static 120 VGPR (NFV~108); anti-moat control + uint32_t rsrc1=BuildPgmRsrc1(false); rsrc1=(rsrc1 & ~0x3fu)|(vgprField & 0x3fu); + uint32_t ldsBytesRaw=(uint32_t)(KCHUNK*(FMc+FNc)*256); // 6144 @ KCHUNK=4,FM=2,FN=4 + uint32_t ldsU=0,ldsA=0,ldsG=0; uint32_t ldsBits=ldsRsrc2Bits(ldsBytesRaw,&ldsU,&ldsA,&ldsG); + uint32_t rsrc2=(BuildPgmRsrc2(false) & ~0x3eu)|(15u<>32),(uint32_t)aVa,(uint32_t)(aVa>>32), + (uint32_t)bVa,(uint32_t)(bVa>>32),(uint32_t)cVa,(uint32_t)(cVa>>32), + (uint32_t)KT,(uint32_t)Ko,(uint32_t)(NT*256),TOTAL, + magic,(uint32_t)NTL,(uint32_t)(FNc*256),0u }; + uint32_t dispInit=BuildDispatchInitiator(); + const uint32_t pool=getenv("ML8_POOL")?(uint32_t)atoi(getenv("ML8_POOL")):256u; // NO 64-clamp: grind is 1 wave/WG, needs many WGs + printf(" [grind] %dx%dx%d tile=%dx%d (FM=%d FN=%d) TOTAL=%u tiles waves/WG=1 pool=%u WGs LDS=%uB(alloc %uB) VGPR=%u static RSRC2=0x%x\n", + Mo,No,Ko,TM,TN,FMc,FNc,TOTAL,pool,ldsBytesRaw,ldsA,vgprField*8,rsrc2); + uint32_t reslim[1]={0},tmpring[1]={0},restart[4]={0,0,0,0}; + memset((void*)occW,0,0x100); occW[20]=0; occW[2]=0xFFFFFFFFu; *fenceW=0; // claim base 0; min-tick sentinel + RingPlace(ring,PM4AcquireMemoryPacket(FAMILY_GFX12)); + RingPlace(ring,PM4SetShaderRegPacket(mmCOMPUTE_START_X,dims,8)); + RingPlace(ring,PM4SetShaderRegPacket(mmCOMPUTE_PGM_LO,pgm,6)); + RingPlace(ring,PM4SetShaderRegPacket(mmCOMPUTE_PGM_RSRC1,rsrc,2)); + RingPlace(ring,PM4SetShaderRegPacket(mmCOMPUTE_RESOURCE_LIMITS,reslim,1)); + RingPlace(ring,PM4SetShaderRegPacket(mmCOMPUTE_TMPRING_SIZE,tmpring,1)); + RingPlace(ring,PM4SetShaderRegPacket(mmCOMPUTE_RESTART_X,restart,4)); + RingPlace(ring,PM4SetShaderRegPacket(mmCOMPUTE_USER_DATA_0,userdata,16)); + RingPlace(ring,PM4DispatchDirectPacket(pool*WAVES_LAUNCH*32,1,1,dispInit)); + RingPlace(ring,PM4ReleaseMemoryPacket(FAMILY_GFX12,true,fenceVa,FENCE_VALUE)); + double t0=now_s(); RingSubmit(ring); + bool done=false, admitted=false; const double timeoutS=25.0; + while (true) { double now=now_s(); + if (occW[0]>0) admitted=true; + if (admitted && occW[0]==0 && *fenceW==FENCE_VALUE) { done=true; break; } + if (now-t0>timeoutS) break; } + if (!done) { fprintf(stderr,"\n*** GRIND TIMEOUT: occ0=%u fence=%s ***\n",occW[0],(*fenceW==FENCE_VALUE)?"FIRED":"--"); + fprintf(stderr," [teardown] grind did not complete -> NOT destroying queue (process-exit reclaims).\n"); return res; } + res.occ0=occW[0]; res.maxlive=occW[1]; + { uint32_t gs=occW[2],ge=occW[3]; + if (gs!=0xFFFFFFFFu && ge!=0) { res.wall=(ge>=gs)?(uint64_t)(ge-gs):((uint64_t)ge+0x100000000ull-(uint64_t)gs); + res.tf=2.0*(double)Mo*(double)No*(double)Ko*freq_hz/(double)res.wall/1e12; } } + printf(" [grind completion] occ[0]=%u (0=clean) maxlive=%u (of 2048, %.1f%%, %.2f/SIMD)\n", + res.occ0,res.maxlive,res.maxlive/2048.0*100.0,res.maxlive/128.0); + if (res.wall) printf(" [grind THROUGHPUT] %dx%dx%d TF=%.1f (%.1f%% of 307 TF fp8 peak) span=%llu ticks @ %.0f MHz\n", + Mo,No,Ko,res.tf,res.tf/307.0*100.0,(unsigned long long)res.wall,freq_hz/1e6); + else printf(" [grind THROUGHPUT] n/a (occ[2]/occ[3] unstamped)\n"); + // canary + { const uint32_t* tail=(const uint32_t*)((const char*)C.ptr+cbytes); uint64_t words=padB/4,nz=0; + for (uint64_t w=0;wres.maxRel) res.maxRel=cmp.max_rel; + } + } + printf(" [grind oracle] ok=%llu bad=%llu max_rel=%.4g (rel=%.0e abs=%.0e)\n", + (unsigned long long)res.okFrags,(unsigned long long)res.badFrags,res.maxRel,orel,oabs); + if (*fenceW==FENCE_VALUE) { CHECK(hsaKmtDestroyQueue(ring.res.QueueId)); + FreeGpu(ring.buf);FreeGpu(fence);FreeGpu(C);FreeGpu(Bd);FreeGpu(Ad);FreeGpu(occ);FreeGpu(isa); } + res.ok = (res.occ0==0) && (res.badFrags==0); + return res; +} + // --------------------------------------------------------------------------- // MAD-305 Step A phase-timer: dispatch the PROFILE build of the real BLADDER FEEDONLY kernel and read the // per-phase realtime tick-sums (occ[8..14]) + K-tile count (occ[15]) accumulated by the single profiler @@ -2667,9 +3337,39 @@ static void run_dynsmoke(uint32_t node) { FreeGpu(ring.buf); FreeGpu(fence); FreeGpu(occ); FreeGpu(isa); } +// ===== MAD-305 DSWS adaptive wave-role controller (SPEC_DSWS_CONTROLLER.md / PLAN_DSWS_CONTROLLER.md). +// v1 = static 3-role substrate {compute / A-feed / B-feed}; the controller layers on in Phases 2-4. +// Config is env-driven so build_dsws.sh / supervised runs stay parameterizable. ===== +struct DswsCfg { + uint32_t nComp, nAfeed, nBfeed; // role partition; N = nComp+nAfeed+nBfeed (wave count never changes) + uint32_t ringd; // A-ring / B-ring depth + uint32_t low, high; // watermark band: occhigh = over-served + uint32_t epochShift; // decision cadence: E = (segments_processed >> epochShift) + bool dyn; // arm s_alloc_vgpr dyn-VGPR (COMPUTE_PGM_RSRC2 bit6) + uint32_t N() const { return nComp + nAfeed + nBfeed; } +}; +static DswsCfg parse_dsws_cfg() { + DswsCfg c; + c.nComp = getenv("DSWS_NCOMP") ? (uint32_t)atoi(getenv("DSWS_NCOMP")) : 4u; + c.nAfeed = getenv("DSWS_NAFEED") ? (uint32_t)atoi(getenv("DSWS_NAFEED")) : 2u; + c.nBfeed = getenv("DSWS_NBFEED") ? (uint32_t)atoi(getenv("DSWS_NBFEED")) : 2u; + c.ringd = getenv("DSWS_RINGD") ? (uint32_t)atoi(getenv("DSWS_RINGD")) : 2u; + c.low = getenv("DSWS_LOW") ? (uint32_t)atoi(getenv("DSWS_LOW")) : 1u; + c.high = getenv("DSWS_HIGH") ? (uint32_t)atoi(getenv("DSWS_HIGH")) : (c.ringd > 1u ? c.ringd - 1u : 1u); + c.epochShift = getenv("DSWS_EPOCHSHIFT") ? (uint32_t)atoi(getenv("DSWS_EPOCHSHIFT")) : 3u; + c.dyn = getenv("DSWS_DYN") ? atoi(getenv("DSWS_DYN")) != 0 : false; + return c; +} + +// FIX 3(m): a positional non-flag argv token (e.g. "4c2a2b" after --dsws2/--dsws) used to be silently +// ignored by the argv loop below -- nothing ever read argv[i] once it failed every `--foo` strcmp. A user +// passing a role-mix that disagreed with the actual (env-derived) NCOMP/NAFEED/NBFEED got the WRONG +// config with zero warning. Captured here so the DSWS2 mode handler can validate it instead. +static char g_posMixArg[64] = {0}; + int main(int argc, char** argv) { setvbuf(stdout, NULL, _IONBF, 0); // unbuffered: if a raw-PM4 run hangs and gets SIGKILL'd, the log still shows WHERE it died - enum { CORRECT, PRONG1, PRONG2, PRONG3, COMBINED, TIMERCHECK, PROBE, MICROBATCH, MBGEMM, MBSAT, DYNFAT1, MBPROF, MERGE, WGGEMM, SGPRPROBE, WGLDS, LDSBOUND, WGGEMM2, WGPERF, WG2X2, NFUNROLL, NFOCC, NFBF, BANDSWP, FEEDPIPE, FEEDLADDER, FEEDBTR, FEEDPROF, FEEDSTAG, FEEDPB, STACK, BW, BASELINES, SUSTAIN, KWIN, KWINORACLE, TILEPROBE, BLDSPROBE, BTR128, ANOLDS, ANOLDSTR, WAVESWEEP, OCCSWEEP, REUSE82, REUSE82TW2, REUSE82KW2, VGPR82, BLDS82, BPF82, SP82, ALD82, WALL82, TW8, TW4LEAN, B128MODE, TILEORDMODE, FP8EDGE, LDSTRIMMODE, VGPRPROBE, LEAN, DECOMP, WAVESPEC, MBML8, MBML8LONG, MBML8GAUNT, MBML8DYN, MBML8GATE, MBML8NF, MBML8PROF, MBML8BATCH, MBML8MATCH, MBML8COOP, DYNSMOKE } mode = CORRECT; + enum { CORRECT, PRONG1, PRONG2, PRONG3, COMBINED, TIMERCHECK, PROBE, MICROBATCH, MBGEMM, MBSAT, DYNFAT1, MBPROF, MERGE, WGGEMM, SGPRPROBE, WGLDS, LDSBOUND, WGGEMM2, WGPERF, WG2X2, NFUNROLL, NFOCC, NFBF, BANDSWP, FEEDPIPE, FEEDLADDER, FEEDBTR, FEEDPROF, FEEDSTAG, FEEDPB, STACK, BW, BASELINES, SUSTAIN, KWIN, KWINORACLE, TILEPROBE, BLDSPROBE, BTR128, ANOLDS, ANOLDSTR, WAVESWEEP, OCCSWEEP, REUSE82, REUSE82TW2, REUSE82KW2, VGPR82, BLDS82, BPF82, SP82, ALD82, WALL82, TW8, TW4LEAN, B128MODE, TILEORDMODE, FP8EDGE, LDSTRIMMODE, VGPRPROBE, LEAN, DECOMP, WAVESPEC, MBML8, MBML8LONG, MBML8GAUNT, MBML8DYN, MBML8GATE, MBML8NF, MBML8PROF, MBML8BATCH, MBML8MATCH, MBML8COOP, DYNSMOKE, DSWS, DSWS2, GRIND } mode = CORRECT; bool fat = false; // --fat: include >128-VGPR shapes (require umr SQ_DYN_VGPR.BLOCK_SIZE=1, cap 256) for (int i = 1; i < argc; ++i) { if (!strcmp(argv[i], "--prong1")) mode = PRONG1; @@ -2690,6 +3390,9 @@ int main(int argc, char** argv) { else if (!strcmp(argv[i], "--mbml8match")) mode = MBML8MATCH; else if (!strcmp(argv[i], "--mbml8coop")) mode = MBML8COOP; else if (!strcmp(argv[i], "--dynsmoke")) mode = DYNSMOKE; + else if (!strcmp(argv[i], "--dsws")) mode = DSWS; + else if (!strcmp(argv[i], "--dsws2")) mode = DSWS2; + else if (!strcmp(argv[i], "--grind")) mode = GRIND; else if (!strcmp(argv[i], "--mbml8gate")) mode = MBML8GATE; else if (!strcmp(argv[i], "--mbsat")) mode = MBSAT; else if (!strcmp(argv[i], "--dynfat1")) mode = DYNFAT1; @@ -2746,6 +3449,11 @@ int main(int argc, char** argv) { else if (!strcmp(argv[i], "--lean")) mode = LEAN; else if (!strcmp(argv[i], "--decomp")) mode = DECOMP; else if (!strcmp(argv[i], "--fat")) fat = true; + else if (argv[i][0] != '-') { + // FIX 3(m): a non-flag positional token (e.g. a "4c2a2b" role-mix string) -- capture it for + // DSWS2's validation instead of silently dropping it. + snprintf(g_posMixArg, sizeof g_posMixArg, "%s", argv[i]); + } } // Test matrices A,B (16x16 e4m3, non-trivial) and the CPU oracle D = A.B. @@ -5253,6 +5961,249 @@ int main(int argc, char** argv) { c.name, r.maxlive, r.okMarks, expect, r.badMarks, r.missMarks, pass ? "PASS" : "FAIL"); if (!pass) rc = 3; } + } else if (mode == DSWS) { + // ===== MAD-305 DSWS adaptive wave-role controller. Phase 1 = STATIC 3-role substrate. + // T1.1 (this) = config + validation refuse-path ONLY; the actual --dsws dispatch + oracle gate + // wire in at T1.3. Mirrors the WAVESPEC / MBML8COOP brick-guard discipline: a geometry/bin + // mismatch under bit6-armed dyn is exactly what bricks gfx1201, so we REFUSE (rc=4, no dispatch) + // on any invalid config or missing bin. ===== + DswsCfg c = parse_dsws_cfg(); + printf("\n=== MAD-305 DSWS 3-role substrate (nComp=%u nAfeed=%u nBfeed=%u N=%u RINGD=%u LOW=%u HIGH=%u EPOCH_SHIFT=%u dyn=%d) ===\n", + c.nComp, c.nAfeed, c.nBfeed, c.N(), c.ringd, c.low, c.high, c.epochShift, c.dyn); + // ---- validation refuse-path (spec floors: compute>=1, A-feed>=1, B-feed>=1; band sanity) ---- + if (c.nComp < 1) { printf(" *** REFUSE: nComp>=1 required (compute floor); got %u ***\n", c.nComp); rc = 4; } + else if (c.nAfeed < 1) { printf(" *** REFUSE: nAfeed>=1 required (A-feed floor); got %u ***\n", c.nAfeed); rc = 4; } + else if (c.nBfeed < 1) { printf(" *** REFUSE: nBfeed>=1 required (B-feed floor); got %u ***\n", c.nBfeed); rc = 4; } + else if (c.ringd < 1) { printf(" *** REFUSE: RINGD>=1 required; got %u ***\n", c.ringd); rc = 4; } + else if (c.low > c.high) { printf(" *** REFUSE: LOW(%u) > HIGH(%u) — invalid watermark band ***\n", c.low, c.high); rc = 4; } + else if (c.high > c.ringd) { printf(" *** REFUSE: HIGH(%u) > RINGD(%u) — occ is clamped to [0,RINGD] ***\n", c.high, c.ringd); rc = 4; } + // ---- bin-presence brick-guard (T1.2/T1.3 build it via ./build_dsws.sh). Absent now by design. ---- + char dswsBin[160]; + snprintf(dswsBin, sizeof dswsBin, "occ_dsws_%uc%ua%ub_r%u%s_gd.bin", + c.nComp, c.nAfeed, c.nBfeed, c.ringd, c.dyn ? "_dyn" : ""); + if (rc == 0) { + FILE* fb = fopen(dswsBin, "rb"); + if (fb) fclose(fb); + else { printf(" *** DSWS kernel bin '%s' NOT BUILT — REFUSING to dispatch " + "(build it via ./build_dsws.sh; T1.2/T1.3) ***\n", dswsBin); + rc = 4; } + } + if (rc == 0) { + // ===== T1.3 static 3-role dispatch + oracle gate. Role counts are baked into the bin (defsyms), + // so the harness only sets WG threads = N*32 (totalWaves) and the bigger DSWS LDS; the C-store + // /oracle partition uses P=NCOMP (DSWS compute count). Small tile-multiple oracle shape first + // (sub-second, brick-safe); GENDIV (ml8 N are non-pow2). [SUPERVISED at T1.4.] ===== + const int FMc = 2, FNc = 4; // DSWS v1 fixed coop tile (baked into the kernel + bin name) + const int Nwaves = (int)c.N(); // launch N = NCOMP+NAFEED+NBFEED waves/WG + // Replicate the kernel's LDS_TOTAL_DSWS EXACTLY (RINGD_A defaults to RINGD): + // BRING + cons[NCOMP] + (prod,ti,epoch,initflag) + prod_b_hi[NBFEED-1] + A-ring + prod_a[NCOMP] + cons_a[NCOMP] + uint32_t BRING = (uint32_t)c.ringd * FNc * 256; + uint32_t ldsBase = BRING + 4u*c.nComp + 16u; // = LDS_TOTAL (the DSWS=0 prefix) + uint32_t aring = (uint32_t)c.ringd * c.nComp * FMc * 256; // RINGD_A * NCOMP * FM * 256 + uint32_t ldsDsws = ldsBase + 4u*(c.nBfeed - 1u) + aring + 8u*c.nComp; + const uint32_t poolD = getenv("ML8_POOL") ? (uint32_t)atoi(getenv("ML8_POOL")) : 64u; + const char* onlyShape = getenv("DSWS_ONLY"); + int oMTL = getenv("DSWS_ORACLE_MTL") ? atoi(getenv("DSWS_ORACLE_MTL")) : 4; + int oNTL = getenv("DSWS_ORACLE_NTL") ? atoi(getenv("DSWS_ORACLE_NTL")) : 8; + int TM = (int)c.nComp*FMc*16, TN = FNc*16; // WG tile + struct SH { const char* name; int M, K, N; }; + SH shapes[] = { {"down ", 2048, 9216, 2560}, {"down_pf", 512, 9216, 2560} }; + printf(" [dsws] N=%d waves/WG WGtile=%dx%d LDS=%uB bin=%s\n", Nwaves, TM, TN, ldsDsws, dswsBin); + for (auto& s : shapes) { + if (onlyShape && !strstr(s.name, onlyShape)) continue; + int Mo = TM*oMTL, No = TN*oNTL, Ko = 512; // small tile-multiple oracle shape + printf("\n #### DSWS %s oracle %dx%dx%d (STORE=1, GENDIV) ####\n", s.name, Mo, No, Ko); + CoopResult o = run_mbcoop(node, dswsBin, c.dyn, poolD < 64u ? poolD : 64u, Mo, No, Ko, + FMc, FNc, (int)c.nComp, (int)c.ringd, /*fullCheck*/true, + /*useGenDiv*/true, /*reps*/1, /*targetSecs*/0.0, + /*totalWaves*/Nwaves, /*ldsBytesOverride*/ldsDsws); + if (!o.ok) { printf(" oracle INCOMPLETE (hang/timeout) -> protocol/grow bug; STOP\n"); rc = 3; break; } + bool clean = (o.badFrags == 0 && o.okFrags > 0); + printf(" oracle %s ok=%llu bad=%llu maxlive=%u\n", + clean ? "CLEAN" : "*** BAD (race/math) ***", + (unsigned long long)o.okFrags, (unsigned long long)o.badFrags, o.maxlive); + if (!clean) { rc = 3; break; } + } + } + } else if (mode == GRIND) { + const int FMc=2, FNc=4; + const int Mo = getenv("GRIND_M") ? atoi(getenv("GRIND_M")) : 576; + const int No = getenv("GRIND_N") ? atoi(getenv("GRIND_N")) : 512; + const int Ko = getenv("GRIND_K") ? atoi(getenv("GRIND_K")) : 2048; + const float orel = getenv("GRIND_REL") ? atof(getenv("GRIND_REL")) : 5e-3f; // full-K single write -> TIGHT tier + const float oabs = getenv("GRIND_ABS") ? atof(getenv("GRIND_ABS")) : 1e-2f; + const char* gbin = getenv("GRIND_BIN") ? getenv("GRIND_BIN") : "occ_kernel_grind.bin"; + printf("\n=== GRIND control (non-split-K, one-tile-per-WG, full-K, write-once C) ===\n"); + FILE* fb=fopen(gbin,"rb"); if (fb) fclose(fb); else { printf(" *** grind bin '%s' NOT BUILT -> refuse ***\n",gbin); rc=4; } + if (rc==0) { + GrindResult o = run_grind(node,gbin,FMc,FNc,Mo,No,Ko,orel,oabs,freq_hz); + if (!o.ok && o.okFrags==0 && o.badFrags==0) { printf(" grind INCOMPLETE (hang/refuse) -> STOP\n"); rc=3; } + else if (o.badFrags>0) { printf(" grind oracle *** BAD *** -> STOP\n"); rc=3; } + else { printf(" grind oracle CLEAN\n"); rc=0; } + } + } else if (mode == DSWS2) { + // ===== MAD-305 DSWS v2 substrate (PLAN_DSWS_SUBSTRATE_V2.md, Task A8). Computes/dry-prints the + // super-tile pool params for occ_kernel_dsws.s and, when DSWS2_DRYRUN is unset, launches it via + // run_dsws2 (the v2 PM4 launch + tiered-oracle path). DSWS2_DRYRUN=1 -> print params + return + // rc=0 WITHOUT touching the GPU (gate 2 of A8: must still dry-print, never dispatch). ===== + DswsCfg c = parse_dsws_cfg(); + const int FMc = 2, FNc = 4; // v2 fixed coop tile (matches build_dsws.sh mk2) + const int Gv = getenv("DSWS2_G") ? atoi(getenv("DSWS2_G")) : 6; // M-extent (rowblks/super-tile) = NCOMP_MAX + const int SEGKv = getenv("DSWS2_SEGK") ? atoi(getenv("DSWS2_SEGK")) : 64; // split-K segment (K-elements) + // super-tile geometry (tile-multiple oracle shape, mirroring the --dsws oracle defaults: oMTL/oNTL/Ko). + const int TMsuper = Gv*16*FMc; // super-tile M rows = G*16*FM + const int TN = FNc*16; // N-panel cols = FN*16 + const int oMTL = getenv("DSWS2_ORACLE_MTL") ? atoi(getenv("DSWS2_ORACLE_MTL")) : 4; + const int oNTL = getenv("DSWS2_ORACLE_NTL") ? atoi(getenv("DSWS2_ORACLE_NTL")) : 8; + // n_kseg/Ko: DSWS2_NKSEG (when set) is the PRIMARY lever -- it derives Ko = SEGKv*n_kseg so the + // pool always covers the FULL K range in exactly n_kseg segments (SEGK is a compile-time defsym + // baked into the .bin's KSEG_STEPS-unrolled WMMA loop; Ko/SEGKv must stay exact or the resident + // A/B staging silently undercounts K). This is how the A8 command forces n_kseg=1 (TIGHT tier) + // without needing a separate DSWS2_K=64: `DSWS2_NKSEG=1` -> Ko=SEGKv*1=64 automatically. + // Without DSWS2_NKSEG, Ko comes from DSWS2_K (default 512) and n_kseg = Ko/SEGKv as before. + int n_kseg, Ko; + if (getenv("DSWS2_NKSEG")) { + n_kseg = atoi(getenv("DSWS2_NKSEG")); + Ko = (n_kseg > 0) ? SEGKv * n_kseg : 0; + } else { + Ko = getenv("DSWS2_K") ? atoi(getenv("DSWS2_K")) : 512; + n_kseg = (SEGKv > 0) ? (Ko / SEGKv) : 0; // = KT/(SEGK/16) (same K-units as the --dsws oracle) + } + const int Mo = TMsuper*oMTL, No = TN*oNTL; // tile-multiple oracle shape + const int KT = (SEGKv > 0) ? Ko/16 : 0; + const int NTL = No / TN; + const int MTLsuper = Mo / TMsuper; + const long long TOTAL_super = (long long)MTLsuper * NTL * n_kseg; // (M/(G*16*FM)) * NTL * n_kseg + const uint64_t TOTAL64 = (uint64_t)MTLsuper * (uint64_t)NTL; // coop-compat output-tile count (C sizing) + uint32_t poolSlots_h = 1u; // FIX 1: flow N-deep pool / ring D=2 / single-slot + if (getenv("DSWS2_FLOW")) poolSlots_h = getenv("FLOW_POOL_N") ? (uint32_t)atoi(getenv("FLOW_POOL_N")) : 3u; + else if (getenv("DSWS2_RING")) poolSlots_h = 2u; + // FIX 1 STAGGER: flow per-rowblk accumulator pool (ACC_N banks x FM*FN*1024B), matches kernel ACC_*. + const uint32_t accN_h = getenv("DSWS2_FLOW") ? (getenv("DSWS2_ACC_N") ? (uint32_t)atoi(getenv("DSWS2_ACC_N")) : 1u) : 0u; + const uint32_t ldsBytes = 256u + poolSlots_h * ((uint32_t)(FNc*16*SEGKv) + (uint32_t)(Gv*16*FMc*SEGKv)) + + accN_h * (uint32_t)(FMc*FNc*1024); + // A2 tiered oracle thresholds: TIGHT (proven gate) for n_kseg==1, LOOSE (split-K reassoc) for n_kseg>1. + // The A8 compare calls oracle_compare(got, ref, n, orel, oabs). + const float orel = (n_kseg == 1) ? 5e-3f : 3e-2f; + const float oabs = (n_kseg == 1) ? 1e-2f : 2e-2f; + const bool dry = getenv("DSWS2_DRYRUN") != nullptr; + printf("\n=== MAD-305 DSWS v2 substrate (A8 launch path; PLAN_DSWS_SUBSTRATE_V2.md) ===\n"); + printf(" G=%d SEGK=%d FM=%d FN=%d NCOMP=%u NAFEED=%u NBFEED=%u\n", + Gv, SEGKv, FMc, FNc, c.nComp, c.nAfeed, c.nBfeed); + printf(" oracle shape %dx%dx%d (super-tile %dx%d, KT=%d, NTL=%d, MTLsuper=%d)\n", + Mo, No, Ko, TMsuper, TN, KT, NTL, MTLsuper); + printf(" n_kseg=%d TOTAL_super=%lld LDS=%uB\n", n_kseg, TOTAL_super, ldsBytes); + printf(" oracle tier: %s (rel=%.0e abs=%.0e)\n", n_kseg == 1 ? "TIGHT" : "LOOSE", orel, oabs); + // FIX 3(m): DSWS2 input validation refuse-paths (mirror the --dsws T1.1 refuse-path discipline). + // Resolve the positional-mix-arg check to a single bool+message BEFORE the dry/refuse chain below, + // so it can sit as one `else if` link in that chain (dry-run must keep bypassing ALL of these + // checks -- including this one -- exactly like it already bypasses the degenerate-geometry and + // Gv/SEGK checks; that's an existing, load-bearing contract: DSWS2_DRYRUN never touches the GPU + // AND never refuses, it just prints whatever params were computed). + bool posMixBad = false; char posMixMsg[256] = {0}; + if (g_posMixArg[0]) { + uint32_t pc = 0, pa = 0, pb = 0; + if (sscanf(g_posMixArg, "%uc%ua%ub", &pc, &pa, &pb) != 3) { + posMixBad = true; + snprintf(posMixMsg, sizeof posMixMsg, + "unrecognized positional arg '%s' (expected a role-mix like '4c2a2b', or no " + "positional arg at all -- role counts come from DSWS_NCOMP/DSWS_NAFEED/DSWS_NBFEED)", + g_posMixArg); + } else if (pc != c.nComp || pa != c.nAfeed || pb != c.nBfeed) { + // a positional role-mix token (e.g. "4c2a2b") was given on the command line. The only built + // v2 bin is compile-time-fixed at NCOMP=4/NAFEED=2/NBFEED=2 (build_dsws.sh mk2); the bin + // filename is actually picked from c.nComp/nAfeed/nBfeed (the DSWS_NCOMP/AFEED/BFEED env + // config), NOT from this positional token -- previously the token was silently ignored, so + // a user passing a mix that disagreed with the active env config got the WRONG config with + // no warning. + posMixBad = true; + snprintf(posMixMsg, sizeof posMixMsg, + "positional role-mix arg '%s' (%uc%ua%ub) does not match the active DSWS_NCOMP/" + "DSWS_NAFEED/DSWS_NBFEED config (%uc%ua%ub) -- set the env vars to match or drop " + "the positional arg", g_posMixArg, pc, pa, pb, c.nComp, c.nAfeed, c.nBfeed); + } + } + if (dry) { printf(" [DSWS2_DRYRUN] params only -- NO GPU dispatch.\n"); rc = 0; } + else if (c.nComp < 1) { printf(" *** REFUSE: nComp>=1 required (compute floor); got %u ***\n", c.nComp); rc = 4; } + else if (c.nAfeed < 1) { printf(" *** REFUSE: nAfeed>=1 required (A-feed floor); got %u ***\n", c.nAfeed); rc = 4; } + else if (c.nBfeed < 1) { printf(" *** REFUSE: nBfeed>=1 required (B-feed floor); got %u ***\n", c.nBfeed); rc = 4; } + else if (c.N() != c.nComp + c.nAfeed + c.nBfeed) { + // role-floor/SUM check: N() (the launched wave count, WAVES_LAUNCH downstream) must equal the + // sum of the role counts actually used to size/decode the dispatch -- guards against a future + // refactor desyncing N() from its components (currently tautological by construction). + printf(" *** REFUSE: role-count sum mismatch (N()=%u != nComp+nAfeed+nBfeed=%u) ***\n", + c.N(), c.nComp + c.nAfeed + c.nBfeed); + rc = 4; + } else if (posMixBad && !getenv("DSWS2_FLOW")) { + printf(" *** REFUSE: %s ***\n", posMixMsg); + rc = 4; + } else if (Mo <= 0 || No <= 0 || Ko <= 0 || n_kseg <= 0 || TOTAL_super <= 0) { + printf(" *** REFUSE: degenerate geometry (Mo=%d No=%d Ko=%d n_kseg=%d TOTAL_super=%lld) ***\n", + Mo, No, Ko, n_kseg, TOTAL_super); + rc = 4; + } else if (TOTAL64 > 0xFFFFFFFFull || (uint64_t)TOTAL_super > 0xFFFFFFFFull) { + printf(" *** REFUSE: pool size overflows uint32_t (TOTAL=%llu TOTAL_super=%lld) -- occ[20]'s claim " + "counter and the kernel's sti are both 32-bit ***\n", (unsigned long long)TOTAL64, TOTAL_super); + rc = 4; + } else if (Gv != 6 || (SEGKv != 64 && !(getenv("DSWS2_FLOW") && SEGKv == 32))) { + // G/SEGK are compile-time defsyms baked into the kernel's instruction immediates + // (KSEG_STEPS-unrolled WMMA loop, resident-LDS strides). A host geometry that disagrees + // with the bin's compiled G/SEGK silently corrupts the resident-A/B staging/compute + // addressing (wrong strides, not a bounds violation the gate below would catch). REFUSE + // rather than guess; rebuild a matching bin before changing these envs. + // FIX 1 STAGGER: the flow bin (build_flow.sh) can now be built with SEGK=32 (halves the + // operand footprint so the g=6 write-once accumulator banks fit LDS) -- so SEGK=32 is + // allowed ONLY on the DSWS2_FLOW path, where the run must pass a matching DSWS2_SEGK=32. + printf(" *** REFUSE: DSWS2_G=%d DSWS2_SEGK=%d mismatches the built bin's compile-time geometry " + "(non-flow: G=6 SEGK=64; flow: G=6 SEGK in {32,64}) -- REFUSING geometry/bin mismatch ***\n", Gv, SEGKv); + rc = 4; + } else { + char dswsBin[160]; + // EMERGENT economy (flow): no baked mix. Derive the launch pool from FLOW_WAVES (host-set), + // cap at 30 (coordinator mailbox squat), sanity-check against the lean-fit budget. + uint32_t Wlaunch = getenv("FLOW_WAVES") ? (uint32_t)atoi(getenv("FLOW_WAVES")) : 8u; // 8 = proven-safe + if (Wlaunch < 4) Wlaunch = 4; // floor(wid0/1/2) + >=1 compute + // SAFETY: W>~8 at POOL_N=1 overcommits the SIMD dyn-VGPR pool at launch -> some waves' s_alloc_vgpr 32 + // fails -> the ONLY in-kernel exit (s_endpgm on a failed-alloc wave) corrupts the pool -> OOB page + // fault -> MODE1 brick (2026-07-05). Until that launch-starvation is root-caused, keep W_launch <= 8. + if (Wlaunch > 30) { printf(" [flow] FLOW_WAVES=%u > 30 (coord cap) -> clamping to 30\n", Wlaunch); Wlaunch = 30; } + { + const uint32_t VB = getenv("FLOW_VBUDGET") ? (uint32_t)atoi(getenv("FLOW_VBUDGET")) : 1536u; + const uint32_t leanFit = (VB - (112u - 32u)) / 32u; // (VBUDGET-(NFV-VLEAN))/VLEAN + if (Wlaunch > leanFit) + printf(" [flow] WARNING FLOW_WAVES=%u exceeds lean-fit=%u for VBUDGET=%u (bin's .error would catch a real overflow)\n", Wlaunch, leanFit, VB); + } + // FIX 1: DSWS2_FLOW -> flow bin, DSWS2_RING -> ring bin, else single-slot bin. + if (getenv("DSWS2_FLOW")) + snprintf(dswsBin, sizeof dswsBin, "occ_dsws2_w%u_flow_gd.bin", Wlaunch); // Wlaunch == built WAVES + else if (getenv("DSWS2_RING")) + snprintf(dswsBin, sizeof dswsBin, "occ_dsws2_%uc%ua%ub_ring_gd.bin", c.nComp, c.nAfeed, c.nBfeed); + else + snprintf(dswsBin, sizeof dswsBin, "occ_dsws2_%uc%ua%ub_gd.bin", c.nComp, c.nAfeed, c.nBfeed); + FILE* fb = fopen(dswsBin, "rb"); + if (fb) fclose(fb); + else { + printf(" *** DSWS2 kernel bin '%s' NOT BUILT -- REFUSING to dispatch (build it via " + "./build_dsws.sh, mk2) ***\n", dswsBin); + rc = 4; + } + if (rc == 0) { + const bool isFlow = getenv("DSWS2_FLOW"); // flow: launch Wlaunch (emergent), mix args unused + Dsws2Result o = run_dsws2(node, dswsBin, + isFlow ? Wlaunch : c.nComp, isFlow ? 0u : c.nAfeed, isFlow ? 0u : c.nBfeed, + Gv, SEGKv, FMc, FNc, Mo, No, Ko, orel, oabs, freq_hz); + if (!o.ok && o.okFrags == 0 && o.badFrags == 0) { + printf(" dsws2 INCOMPLETE (hang/timeout/refused before oracle) -> protocol/geometry bug; STOP\n"); + rc = 3; + } else if (o.badFrags > 0) { + printf(" dsws2 oracle *** BAD (race/math) *** -> STOP\n"); + rc = 3; + } else { + printf(" dsws2 oracle CLEAN\n"); + rc = 0; + } + } + } } else { // Default: dyn-VGPR cap probe. Test dyn correctness at increasing s_alloc footprints: // light NACC=8 -> s_alloc 96 (<=128, expected OK) diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_coop.s b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_coop.s index 3764964fee74..5a31a4e9dd53 100644 --- a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_coop.s +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_coop.s @@ -91,7 +91,49 @@ // (DIAG: computePhase=4, occ[10]=1 then wedge). s_sleep yields issue // cycles to the partner. Steady-state cost ~0 (loops spin ~0x when fed). -.set WAVES, (1 + P) // total waves launched per WG (harness dims must match) +// ---- DSWS (MAD-305 adaptive wave-role controller, SPEC_DSWS_CONTROLLER.md). Phase 1 = STATIC 3-role +// substrate {compute / A-feed / B-feed}, all NEW behavior gated behind `.if DSWS` so DSWS=0 stays +// BYTE-IDENTICAL to the proven 2-role coop d0 (1716B). Design (T1.2 decision, KG 4ce31886): +// * band-partitioned A: A-feed wave a owns strided bands {a, a+NAFEED, ...}; band c has ONE producer +// (prod_a[c]) and ONE consumer (compute c -> cons_a[c]). No min over consumers needed for A. +// * frag-partitioned B: B-feed wave b owns strided frags {b, b+NBFEED, ...} of the FN shared frags; +// each bumps prod_b[b]; compute gates on min(prod_b[*]). The lead B-feed (wid 0) is the SOLE tile +// claimer + ti/epoch broadcaster; A-feed + non-lead B-feed FOLLOW the broadcast like compute. +// * ZERO inter-feed rendezvous (the consumer-side min is the only cross-wave wait) -> the barrier-free +// invariant that keeps dyn-VGPR safe is preserved. ---- +.ifndef DSWS + .set DSWS, 0 +.endif +.if DSWS + .ifndef NCOMP + .set NCOMP, 4 // compute waves (fat, dyn-grow). Compute floor >= 1. + .endif + .ifndef NAFEED + .set NAFEED, 2 // A-feed waves (lean). Feed floor >= 1. + .endif + .ifndef NBFEED + .set NBFEED, 2 // B-feed waves (lean). Feed floor >= 1. + .endif + .ifndef RINGD_A + .set RINGD_A, RINGD // A-ring depth (defaults to the B-ring depth). MUST be power of two. + .endif + // ---- Controller thresholds (Phase 3 actuation; mirror occ_dispatch DSWS_LOW/HIGH/EPOCHSHIFT). ---- + .ifndef EPOCH_SHIFT + .set EPOCH_SHIFT, 3 // decision clock: epoch = segcnt >> EPOCH_SHIFT (small=reactive) + .endif + .ifndef CTRL_LOW + .set CTRL_LOW, 1 // occ_X < CTRL_LOW -> compute starved for X (shrink compute->feedX) + .endif + .ifndef CTRL_HIGH + .set CTRL_HIGH, (RINGD-1) // occ_X > CTRL_HIGH -> feed-X over-serving (grow feedX->compute) + .endif + // The existing compute body is parameterized by P = "#compute waves" (rowblk=trow*P+cid, store strides). + // Under DSWS the compute count is NCOMP -> bind P=NCOMP so all that addressing auto-targets NCOMP bands. + .set P, NCOMP + .set WAVES, (NCOMP + NAFEED + NBFEED) // total waves launched per WG (harness dims must match) +.else + .set WAVES, (1 + P) // total waves launched per WG (harness dims must match) +.endif // ---- LDS layout (bytes) -- matches run_mbcoop sizing: RINGD*FN*256 + 4*(1 + P + 3) ---- .set BRING_OFF, 0 // B ring: RINGD slots, each FN frags x 256B @@ -101,6 +143,29 @@ .set EPOCH_OFF, (TI_OFF + 4) // tile epoch (u32) -- feed bump publishes a new tile .set INITFLAG_OFF, (EPOCH_OFF + 4) // BUSYWAIT init-publish flag (the former spare u32) .set LDS_TOTAL, (EPOCH_OFF + 4 + 4) // + 1 spare u32 (now INITFLAG; sizing unchanged) +.if DSWS + // ---- DSWS extra LDS, all APPENDED after the DSWS=0 region so the proven offsets stay byte-identical ---- + // prod_b[0] reuses PROD_OFF (the frag-partition lead); prod_b[1..NBFEED-1] live just past INITFLAG. + .set PRODB_HI_OFF, LDS_TOTAL // prod_b[1..NBFEED-1] : NBFEED-1 u32 (0 bytes when NBFEED=1) + .set ARING_OFF, (PRODB_HI_OFF + 4*(NBFEED-1)) // A-ring: RINGD_A slots x NCOMP bands x FM frags x 256B + .set PROD_A_OFF, (ARING_OFF + RINGD_A*NCOMP*FM*256) // prod_a[NCOMP] (band-partitioned producers) + .set CONS_A_OFF, (PROD_A_OFF + 4*NCOMP) // cons_a[NCOMP] (compute A release; one consumer per band) + // ---- Controller state (Phase 2 sensing/slots/reservation; Phase 3 actuates). Mirrors dsws_ctrl_model.cpp. ---- + // Role slots: live count of waves currently in each role (leader inits to the launch mix; conversions adjust). + .set NCOMP_SLOT, (CONS_A_OFF + 4*NCOMP) // u32 nComp (live fat-compute wave count) + .set NAFEED_SLOT, (NCOMP_SLOT + 4) // u32 nAfeed (live A-feed wave count) + .set NBFEED_SLOT, (NAFEED_SLOT + 4) // u32 nBfeed (live B-feed wave count) + // Conversion gates: gate[dir] holds the last epoch in which direction `dir` fired (gate_try_win CAS target). + // 4 directions: 0=compute->Afeed, 1=compute->Bfeed, 2=Afeed->compute, 3=Bfeed->compute. + .set GATE_OFF, (NBFEED_SLOT + 4) // u32[4] gate[dir] + .set VRESV_OFF, (GATE_OFF + 4*4) // u32 vgpr_reserved (sum-envelope: reserve_grow target) + .set SEGCNT_OFF, (VRESV_OFF + 4) // u32 segments_processed (per-WG decision clock source) + .set LDS_TOTAL_DSWS, (SEGCNT_OFF + 4) + .set ABAND_STRIDE, (RINGD_A*FM*256) // bytes per compute band's A sub-ring + .if LDS_TOTAL_DSWS > 65536 + .error "DSWS LDS exceeds 64KB/WG (gfx1201): reduce RINGD_A / NCOMP / tile" + .endif +.endif // ---- VGPR layout ---- .set ACC, 32 // accumulators: FM*FN frags x 8 f32 (compute; above the lean block) @@ -109,7 +174,16 @@ // next_free_vgpr ROUNDED to a 16-VGPR dyn-alloc block. 2x4 -> 112 (highest index 107 needs >=108): a clean // 7-block alloc, SAFELY below the 128 cap and OFF the 128 exact-fill edge (the strategic dyn danger zone). .set NFV, ((FB + 2*FN + 15) & ~15) +.if DSWS + // Controller reservation-envelope units (hardware ALLOC footprint, not live-peak): a fat-compute wave + // reserves NFV (its s_alloc_vgpr target); a lean feed wave reserves VLEAN (its `s_alloc_vgpr 32`). + // Phase-3 grow delta = NFV - VLEAN; vgpr_reserved init (Phase 2) = NCOMP*NFV + (NAFEED+NBFEED)*VLEAN. + .set VLEAN, 32 +.endif .set BSTG, 16 // feed B staging: FN x 2 at v16.. (lean block, < 32) +.if DSWS + .set ASTG, 16 // A-feed staging: FM x 2 at v16.. (A-feed is a distinct lean wave, +.endif // so it reuses the same lean block as the B-feed's BSTG) // ---- dyn-VGPR PRE-GROW VGPR CEILING (THE deadlock + dead-marks root cause; Codex + RDNA4 ISA confirmed) ---- // A dyn-armed (RSRC2 bit6) wave LAUNCHES with exactly ONE 16-VGPR block backed (v0..v15); RSRC1.VGPRS is @@ -191,6 +265,69 @@ s_mov_b32 exec_lo, s58 .endif .endm + +.macro mark_set off, val // DIAG: lane0-of-wave PLAIN-store \val -> occ[off] (INSTANTANEOUS, overwrites). +.if DIAG // Unlike `mark` (atomic-MAX = furthest-reached), this streams the CURRENT + s_mov_b32 s58, exec_lo // value so the harness's 200ms poll sees a sensor oscillate (occ_a/occ_b). + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lms\@ + v_mov_b32 v[RM_A], 0 // self-zeroed vaddr (same robustness as mark) + v_mov_b32 v[RM_D], \val + global_store_b32 v[RM_A], v[RM_D], s[0:1] offset:\off scope:SCOPE_DEV +.Lms\@: + s_mov_b32 exec_lo, s58 +.endif +.endm + +.macro mark_inc off // DIAG: lane0-of-wave atomic-ADD 1 -> occ[off] (a free-running counter). +.if DIAG + s_mov_b32 s58, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lmi\@ + v_mov_b32 v[RM_A], 0 + v_mov_b32 v[RM_D], 1 + global_atomic_add_u32 v[RM_A], v[RM_D], s[0:1] offset:\off scope:SCOPE_DEV +.Lmi\@: + s_mov_b32 exec_lo, s58 +.endif +.endm + +// try_gate: the lock-free single-winner conversion ticket (transcribes dsws_ctrl_model.cpp gate_try_win + +// epoch_of EXACTLY). E = segcnt>>EPOCH_SHIFT. gate[dir] holds the last epoch dir fired. Among many waves +// racing the same (g MEM=(MEM==vCMP)?vNEW:MEM, vdst<-old. So vsrc0=E (new), +// vsrc1=g (compare). WIN iff returned-old == g. (T3.1 micro-check CAUGHT the swapped form: it left gate +// stuck at 0 so old==g held for all racers -> would-win ~= NCOMP*epochs instead of ~= epochs.) +.macro try_gate dir, swin + lds_get s62, SEGCNT_OFF // E = epoch_of(segcnt, EPOCH_SHIFT) + s_lshr_b32 s62, s62, EPOCH_SHIFT + lds_get s63, (GATE_OFF + (\dir)*4) // g = gate[dir] + s_mov_b32 \swin, 0 + s_cmp_ge_u32 s63, s62 // g >= E -> dir already fired this/later epoch -> lose + s_cbranch_scc1 .Ltg_done\@ + s_mov_b32 s65, exec_lo // lane0-only CAS (one ticket attempt per WAVE) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltg_restore\@ + v_mov_b32 v5, (GATE_OFF + (\dir)*4) // vaddr = &gate[dir] + v_mov_b32 v6, s63 // v6 = g (vsrc1 = CMP/expected) + v_mov_b32 v7, s62 // v7 = E (vsrc0 = NEW value to store) + ds_cmpstore_rtn_b32 v6, v5, v7, v6 // gate[dir] = (gate[dir]==g) ? E : gate[dir]; v6 <- old + s_wait_dscnt 0x0 +.Ltg_restore\@: + s_mov_b32 exec_lo, s65 + v_readfirstlane_b32 s64, v6 // s64 = old (lane0's CAS result, broadcast) + s_cmp_eq_u32 s64, s63 // WIN iff old == g (we were the swapper) + s_cbranch_scc0 .Ltg_done\@ + s_mov_b32 \swin, 1 +.Ltg_done\@: +.endm + .macro min_cons sdst // sdst = min over c in 0..P-1 of cons_count[c] lds_get \sdst, CONS_OFF .set cc, 1 @@ -200,6 +337,23 @@ .set cc, cc+1 .endr .endm +.if DSWS +.macro lds_get_r sdst, saddr // wave-uniform read LDS[saddr] -> scalar sdst (RUNTIME addr in a sreg) + v_mov_b32 v[RG_A], \saddr + ds_load_b32 v[RG_D], v[RG_A] + s_wait_dscnt 0x0 + v_readfirstlane_b32 \sdst, v[RG_D] +.endm +.macro min_prod sdst // sdst = min over b in 0..NBFEED-1 of prod_b[b] (frag-partitioned B). + lds_get \sdst, PROD_OFF // prod_b[0] is PROD_OFF; prod_b[1..] live in the DSWS hi region. + .set pw, 1 + .rept NBFEED-1 + lds_get s46, (PRODB_HI_OFF + (pw-1)*4) + s_min_u32 \sdst, \sdst, s46 + .set pw, pw+1 + .endr +.endm +.endif .text .globl occ_kernel @@ -274,6 +428,49 @@ occ_kernel: .endr v_mov_b32 v[RI_A], EPOCH_OFF ds_store_b32 v[RI_A], v[RI_D] +.if DSWS + // zero the DSWS monotonic counters: prod_b[1..NBFEED-1], prod_a[NCOMP], cons_a[NCOMP] (start at 0). + // (the A-ring storage needs no zeroing -- it is overwritten before read, gated by prod_a.) + .set zw, 0 + .rept (NBFEED-1) + v_mov_b32 v[RI_A], (PRODB_HI_OFF + zw*4) + ds_store_b32 v[RI_A], v[RI_D] + .set zw, zw+1 + .endr + .set zc, 0 + .rept NCOMP + v_mov_b32 v[RI_A], (PROD_A_OFF + zc*4) + ds_store_b32 v[RI_A], v[RI_D] + v_mov_b32 v[RI_A], (CONS_A_OFF + zc*4) + ds_store_b32 v[RI_A], v[RI_D] + .set zc, zc+1 + .endr + // ---- Controller state init (Phase 2): role slots <- launch mix, gates <- 0, vgpr_reserved <- launch + // envelope footprint, segcnt <- 0. Leader-lane writes constants to WG-shared LDS (idempotent across + // waves: every lane-0 writes identical compile-time values). v[RI_D] still holds 0 from above. ---- + v_mov_b32 v[RI_A], NCOMP_SLOT + v_mov_b32 v[RI_D], NCOMP + ds_store_b32 v[RI_A], v[RI_D] + v_mov_b32 v[RI_A], NAFEED_SLOT + v_mov_b32 v[RI_D], NAFEED + ds_store_b32 v[RI_A], v[RI_D] + v_mov_b32 v[RI_A], NBFEED_SLOT + v_mov_b32 v[RI_D], NBFEED + ds_store_b32 v[RI_A], v[RI_D] + v_mov_b32 v[RI_D], 0 + .set zg, 0 + .rept 4 + v_mov_b32 v[RI_A], (GATE_OFF + zg*4) // gate[dir] = 0 (no conversion has fired) + ds_store_b32 v[RI_A], v[RI_D] + .set zg, zg+1 + .endr + v_mov_b32 v[RI_A], VRESV_OFF // vgpr_reserved = NCOMP*NFV + (NAFEED+NBFEED)*VLEAN + v_mov_b32 v[RI_D], (NCOMP*NFV + (NAFEED+NBFEED)*VLEAN) + ds_store_b32 v[RI_A], v[RI_D] + v_mov_b32 v[RI_A], SEGCNT_OFF // segments_processed = 0 + v_mov_b32 v[RI_D], 0 + ds_store_b32 v[RI_A], v[RI_D] +.endif s_wait_dscnt 0x0 .Linit_done: s_mov_b32 exec_lo, s16 @@ -305,6 +502,19 @@ occ_kernel: mark 108, 1 // DIAGFINE occ[27] = reached post-rendezvous (passed s_barrier) .endif +.if DSWS + // ===== DSWS 3-role branch (wid uniform per wave; scalar-only -> exec stays full for every role). + // wid [0,NBFEED) -> B-feed (wid 0 = lead: sole tile-claimer + ti/epoch broadcaster) + // wid [NBFEED,NBFEED+NAFEED) -> A-feed + // wid [NBFEED+NAFEED, WAVES) -> compute, cid = wid - (NBFEED+NAFEED) ===== + v_readfirstlane_b32 s24, v1 // wid (uniform per wave) + s_cmp_lt_u32 s24, NBFEED + s_cbranch_scc1 .Lbfeed_role + s_cmp_lt_u32 s24, (NBFEED+NAFEED) + s_cbranch_scc1 .Lafeed_role + s_sub_u32 s47, s24, (NBFEED+NAFEED) // cid + s_branch .Ldsws_compute_init +.else // ============ ROLE BRANCH: wave 0 -> feed ; waves 1..P -> compute ============ v_cmp_eq_u32 vcc_lo, 0, v1 s_mov_b32 s25, exec_lo @@ -314,19 +524,44 @@ occ_kernel: s_branch .Lfeed_role .Lcompute_role: s_mov_b32 exec_lo, s25 // only compute waves reach here +.endif // ======================================================================================== - // COMPUTE BODY (waves 1..P). cid = wid-1. Owns M-band [cid*FM*16 ..). Shares the FN N-cols. + // COMPUTE BODY (waves 1..P). cid = wid-1 (DSWS=0) or wid-(NBFEED+NAFEED) (DSWS). Owns M-band + // [cid*FM*16 ..). Shares the FN N-cols. DSWS: also consumes its A-band from the A-ring. // ======================================================================================== +.if DSWS == 0 v_sub_nc_u32 v3, v1, 1 v_readfirstlane_b32 s47, v3 // cid (uniform per wave) +.endif +.Ldsws_compute_init: // DSWS path joins here with s47=cid already set // cons_count[cid] LDS address (runtime): v12 = CONS_OFF + cid*4 s_lshl_b32 s36, s47, 2 s_add_u32 s36, s36, CONS_OFF v_mov_b32 v12, s36 +.if DSWS + // per-band A-consume state: &prod_a[cid], &cons_a[cid], A-band ring base, a_step (cumulative A consume). + s_lshl_b32 s52, s47, 2 + s_add_u32 s52, s52, PROD_A_OFF // &prod_a[cid] + s_lshl_b32 s53, s47, 2 + s_add_u32 s53, s53, CONS_A_OFF // &cons_a[cid] + s_mul_i32 s51, s47, ABAND_STRIDE + s_add_u32 s51, s51, ARING_OFF // A-band ring base (this cid's sub-ring) + s_mov_b32 s54, 0 // a_step = 0 (cumulative A consume; matches producer prod_a[cid]) +.endif s_mov_b32 s35, 0 // local_epoch = 0 (matches the init epoch) s_mov_b32 s57, 0 // tiles processed (POOL=1 count-to-TOTAL terminal, see below) s_mov_b32 s56, 0 // GLOBAL cons step (cumulative consumed; MONOTONIC, matches feed s55) +.if DSWS + // ---- Phase-2 sensing: stream the live role slots (read-only; static in Phase 2) once per compute wave. + // Proves the leader's controller-state init round-trips through WG-shared LDS into the occ snapshot. + lds_get s59, NCOMP_SLOT + mark_set 136, s59 // occ[34] = nComp (live fat-compute wave count) + lds_get s59, NAFEED_SLOT + mark_set 140, s59 // occ[35] = nAfeed (live A-feed wave count) + lds_get s59, NBFEED_SLOT + mark_set 144, s59 // occ[36] = nBfeed (live B-feed wave count) +.endif .Lcompute_loop: mark 28, 1 @@ -413,10 +648,20 @@ occ_kernel: .if DYNVGPR s_sleep SLEEPN // yield to the lean feed so it can publish the next B-step .endif +.if DSWS + min_prod s44 // B ready = min over frag-partitioned producers prod_b[*] > s56 +.else lds_get s44, PROD_OFF +.endif s_cmp_le_u32 s44, s56 s_cbranch_scc1 .Lwait_prod mark 56, s56 // DIAGFINE occ[14] = max step that PASSED the prod-wait (about to ds_load B) +.if DSWS + // Phase-2 sensor: occ_b = prod_b_min - cons, sampled HERE (consume point, pre-release) where it is the + // live ring backlog and -- by the gate above (prod>cons) -- provably in [1, RINGD]. Read-only, NO action. + s_sub_u32 s59, s44, s56 + mark_set 128, s59 // occ[32] = occ_b (instantaneous B-ring backlog, [1,RINGD]) +.endif // ring slot = s56 & (RINGD-1) (GLOBAL step -> matches the feed's continuous ring) s_and_b32 s45, s56, (RINGD-1) s_mul_i32 s45, s45, (FN*256) @@ -433,6 +678,35 @@ occ_kernel: s_add_u32 s56, s56, 1 lds_put_v v12, s56 mark 40, s56 // DIAG: occ[10] = max GLOBAL cons step reached (KT*32=1024 = all consumed) +.if DSWS + // ---- consume FM own A frags from the A-ring band cid (gated on prod_a[cid] > a_step); release cons_a[cid] ---- +.Lwait_proda: + .if DYNVGPR + s_sleep SLEEPN // yield to the lean A-feed so it can publish the next A-step + .endif + lds_get_r s44, s52 // prod_a[cid] + s_cmp_le_u32 s44, s54 + s_cbranch_scc1 .Lwait_proda + // Phase-2 sensor: occ_a = prod_a[cid] - a_step, sampled HERE (A-consume point) where the gate guarantees + // prod_a > a_step -> occ_a in [1, RINGD_A]. Read-only, NO action. + s_sub_u32 s59, s44, s54 + mark_set 132, s59 // occ[33] = occ_a (instantaneous A-band backlog, [1,RINGD_A]) + // A-ring slot addr = A-band base (s51) + (a_step & (RINGD_A-1))*FM*256, per-lane lane*8 (v9) + s_and_b32 s45, s54, (RINGD_A-1) + s_mul_i32 s45, s45, (FM*256) + s_add_u32 s45, s45, s51 + v_add_nc_u32 v13, v9, s45 + .set mi, 0 + .rept FM + ds_load_b64 v[FA+mi*2:FA+mi*2+1], v13 offset:mi*256 + .set mi, mi+1 + .endr + s_wait_dscnt 0x0 + // RELEASE the A slot now (A is in our regs, before WMMA): cons_a[cid] = a_step+1 + s_add_u32 s54, s54, 1 + v_mov_b32 v13, s53 + lds_put_v v13, s54 +.else // direct-load FM own A frags; advance A saddrs +16 (next K16) .set mi, 0 .rept FM @@ -446,6 +720,7 @@ occ_kernel: .set mi, mi+1 .endr s_wait_loadcnt 0x0 +.endif mark 64, s56 // DIAGFINE occ[16] = max step with A global_load'd (about to WMMA) // FM*FN accumulating WMMA .set mi, 0 @@ -483,6 +758,28 @@ occ_kernel: // concurrent WG memory traffic -> it hung forever. Signal done + move on, per the WAR being moot here.) ---- s_add_u32 s57, s57, 1 // counted one more tile (issued its stores) mark 44, s57 // DIAG: occ[11] = tiles the compute COMPLETED (issued+retired) +.if DSWS + // ---- Phase-2 per-WG segments_processed bump (the controller's decision clock): lead compute (cid==0) + // owns it (POOL=1: one tile = one WG segment, all cids lock-step on the broadcast ti). Single writer. + // occ_a/occ_b are sampled at the per-K CONSUME points (where the ring is mid-flight), NOT here -- at + // the segment boundary the ring has fully DRAINED so occ reads ~0, which is both uninformative for + // validation and the wrong signal for the eventual controller (would always read "starved"). ---- + s_cmp_eq_u32 s47, 0 + s_cbranch_scc0 .Lseg_done + lds_get s44, SEGCNT_OFF + s_add_u32 s44, s44, 1 + lds_put SEGCNT_OFF, s44 +.Lseg_done: + // ---- T3.1 UNIT (no actuation): every compute wave races the (compute->Afeed, dir=0) ticket each segment. + // The LDS-CAS in try_gate must yield <=1 winner per (dir,epoch) across all NCOMP racers -> the would-win + // counter occ[39] should land near #epochs (= segcnt>>EPOCH_SHIFT range), NOT NCOMP*#epochs. This is the + // harness-side validation of the gate-CAS before 3.2/3.3 wire real role conversions onto \swin. ---- + try_gate 0, s59 + s_cmp_eq_u32 s59, 1 + s_cbranch_scc0 .Ltg_unit_done + mark_inc 156 // occ[39] = would-win count (compute->Afeed ticket) +.Ltg_unit_done: +.endif s_cmp_lt_u32 s57, s11 // processed < TOTAL ? s_cbranch_scc0 .Lcompute_exit // terminal -> skip store-wait; dispatch fence drains the stores s_wait_storecnt 0x0 // non-terminal: stores MUST drain before the next tile re-zeros ACC @@ -506,10 +803,259 @@ occ_kernel: .endif s_endpgm // (NOFENCE path still completes on occ[0]==0 regardless of the fence) +.if DSWS + // ======================================================================================== + // DSWS B-FEED (frag-partitioned producers). wid 0 = LEAD: sole tile-claimer + ti/epoch broadcaster. + // Every B-feed wave (b_id=wid) owns the strided frag subset {ni : ni%NBFEED==b_id} of the FN shared + // frags, fills them into the shared B-ring slot, and bumps ITS OWN prod_b[b_id]. Compute gates on + // min(prod_b[*]) so no B-feed ever waits on another B-feed (consumer-side min = barrier-free). + // ======================================================================================== +.Lbfeed_role: + mark 24, 1 // DIAG occ[6] feedPhase=1: B-feed role entered + s_mov_b32 s55, 0 // GLOBAL prod step (cumulative; monotonic, never reset) + .if DYNVGPR +.Lbfeed_lean_alloc: + s_alloc_vgpr 32 // commit this lean B-feed at 32 (dyn WG-allocator consistency) + s_cbranch_scc0 .Lbfeed_lean_alloc + .endif + // &prod_b[b_id]: lead (b_id==0) -> PROD_OFF ; else -> PRODB_HI_OFF + (b_id-1)*4 + s_sub_u32 s50, s24, 1 + s_lshl_b32 s50, s50, 2 + s_add_u32 s50, s50, PRODB_HI_OFF + s_mov_b32 s48, PROD_OFF + s_cmp_eq_u32 s24, 0 + s_cselect_b32 s50, s48, s50 + s_mov_b32 s35, 0 // local epoch (non-lead follow clock) +.Lbfeed_loop: + s_cmp_eq_u32 s24, 0 + s_cbranch_scc0 .Lbfeed_follow // non-lead: follow the lead's broadcast + // ---- LEAD: claim BATCH=1 tile (the ONE device-scope atomic), publish ti, bump epoch ---- + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lbfeed_after_grab + v_mov_b32 v3, BATCH + global_atomic_add_u32 v5, v4, v3, s[0:1] offset:20 th:TH_ATOMIC_RETURN scope:SCOPE_DEV + s_wait_loadcnt 0x0 +.Lbfeed_after_grab: + s_mov_b32 exec_lo, s16 + v_readfirstlane_b32 s17, v5 // ti + lds_put TI_OFF, s17 // publish ti FIRST... + lds_get s44, EPOCH_OFF + s_add_u32 s44, s44, 1 + s_mov_b32 s35, s44 + lds_put EPOCH_OFF, s44 // ...then bump epoch (followers see ti before the bump) + s_branch .Lbfeed_have_ti +.Lbfeed_follow: + .if DYNVGPR + s_sleep SLEEPN + .endif + lds_get s44, EPOCH_OFF + s_cmp_eq_u32 s44, s35 + s_cbranch_scc1 .Lbfeed_follow // spin until the lead bumps epoch + s_mov_b32 s35, s44 + lds_get s17, TI_OFF +.Lbfeed_have_ti: + s_cmp_ge_u32 s17, s11 // terminal tile? (broadcast already done -> all roles retire) + s_cbranch_scc1 .Lfeed_exit +.if SAFEPROBE + // SAFEPROBE: clamp the broadcast ti -> [0,TOTAL-1] before B-saddr decode (mirrors compute lines 471-472). + // Pairs with the common-prologue v9 vaddr clamp so EVERY B global address is provably in-buffer even if + // the broadcast read returns a garbage ti (defense-in-depth; terminal test above already retires ti>=TOTAL). + s_sub_u32 s36, s11, 1 // TOTAL-1 (s36 dead here; B-feed recomputes it nowhere) + s_min_u32 s17, s17, s36 +.endif + mark 24, 2 // DIAG occ[6] feedPhase=2: B-feed has a (clamped) ti + mark 32, s17 // DIAG occ[8] feedTi: the broadcast/claimed ti B is producing + // ---- B col-tile saddr (k=0): Bshuf + tcol*(FN*256) ---- + s_mul_hi_u32 s19, s17, s12 + s_mul_i32 s18, s19, s13 + s_sub_u32 s18, s17, s18 // tcol + s_mul_i32 s20, s18, s14 + s_add_u32 s20, s4, s20 + s_addc_u32 s21, s5, 0 + s_mov_b32 s26, 0 // per-tile k +.Lbprod_k: + // slot-free gate (GLOBAL step) on min compute cons: don't overwrite an undrained slot + s_cmp_lt_u32 s55, RINGD + s_cbranch_scc1 .Lbslot_ok + s_sub_u32 s45, s55, RINGD +.Lbwait_slot: + .if DYNVGPR + s_sleep SLEEPN + .endif + min_cons s44 + s_cmp_le_u32 s44, s45 + s_cbranch_scc1 .Lbwait_slot +.Lbslot_ok: + mark 72, s55 // DIAGFINE occ[18] slotok: B step that passed the slot-free gate + s_and_b32 s45, s55, (RINGD-1) + s_mul_i32 s45, s45, (FN*256) + v_add_nc_u32 v13, v9, s45 // ring vaddr base (lane*8 + slot) + // pass 1: global_load_tr owned frags into staging (compile-time owner; runtime guard on b_id) + .set ni, 0 + .rept FN + .set owner, ni % NBFEED + s_cmp_eq_u32 s24, owner + s_cbranch_scc0 1f + global_load_tr_b64 v[BSTG+ni*2:BSTG+ni*2+1], v9, s[20:21] offset:ni*256 +1: + .set ni, ni+1 + .endr + s_wait_loadcnt 0x0 + mark 76, s55 // DIAGFINE occ[19] loadtr: B step global_load_tr'd into staging + // pass 2: ds_store owned frags into the ring slot + .set ni, 0 + .rept FN + .set owner, ni % NBFEED + s_cmp_eq_u32 s24, owner + s_cbranch_scc0 2f + ds_store_b64 v13, v[BSTG+ni*2:BSTG+ni*2+1] offset:ni*256 +2: + .set ni, ni+1 + .endr + s_wait_dscnt 0x0 + mark 80, s55 // DIAGFINE occ[20] dsstore: B step ds_store'd into ring (pre-publish) + // publish prod_b[b_id] = s55+1 (this wave's owned frags for this step are now visible) + s_add_u32 s55, s55, 1 + v_mov_b32 v13, s50 + lds_put_v v13, s55 + mark 84, s55 // DIAGFINE occ[21] publish: max prod_b count PUBLISHED by B-feed + s_add_u32 s20, s20, s10 // advance B saddr += NT*256 + s_addc_u32 s21, s21, 0 + s_add_u32 s26, s26, 1 + s_cmp_lt_u32 s26, s8 + s_cbranch_scc1 .Lbprod_k + // lead drains (waits compute consumed) before next claim; non-lead just loops to wait next epoch + s_cmp_eq_u32 s24, 0 + s_cbranch_scc0 .Lbfeed_loop +.Lbdrain: + .if DYNVGPR + s_sleep SLEEPN + .endif + min_cons s44 + s_cmp_lt_u32 s44, s55 + s_cbranch_scc1 .Lbdrain + s_branch .Lbfeed_loop + + // ======================================================================================== + // DSWS A-FEED (band-partitioned producers). FOLLOWS the lead's ti/epoch broadcast (like compute). + // A-feed wave a_id owns the strided band subset {bnd : bnd%NAFEED==a_id}; band bnd has ONE producer + // (this wave) and ONE consumer (compute bnd). prod_a[bnd] doubles as the band's cumulative step. + // ======================================================================================== +.Lafeed_role: + s_sub_u32 s24, s24, NBFEED // a_id = wid - NBFEED + .if DYNVGPR +.Lafeed_lean_alloc: + s_alloc_vgpr 32 + s_cbranch_scc0 .Lafeed_lean_alloc + .endif + s_mov_b32 s35, 0 // local epoch (follow) + s_mov_b32 s60, 0 // GLOBAL A-step (gk): cumulative across tiles, mirrors compute s54 + // and the B-ring's continuous step. The per-tile k (s26) is for + // A-MATRIX addressing only; the ring protocol (publish/slot/gate) + // MUST be global or tile>=1 re-publishes prod_a<=a_step -> A-starve. +.Lafeed_loop: + .if DYNVGPR + s_sleep SLEEPN + .endif + lds_get s44, EPOCH_OFF + s_cmp_eq_u32 s44, s35 + s_cbranch_scc1 .Lafeed_loop + s_mov_b32 s35, s44 + lds_get s17, TI_OFF + s_cmp_ge_u32 s17, s11 // terminal? -> retire (role-agnostic exit) + s_cbranch_scc1 .Lfeed_exit +.if SAFEPROBE + // SAFEPROBE: clamp the broadcast ti -> [0,TOTAL-1] before A-saddr decode (mirrors compute lines 471-472). + // s36 is dead here (the per-band loop recomputes it from scratch); pairs with the common-prologue v8 + // vaddr clamp so EVERY A global address is provably in-buffer even on a garbage broadcast ti. + s_sub_u32 s36, s11, 1 // TOTAL-1 + s_min_u32 s17, s17, s36 +.endif + mark 112, 1 // DIAGFINE occ[28] Afeed: received broadcast ti, decoding bands + s_mul_hi_u32 s19, s17, s12 // trow = mul_hi(ti, magic) + // ===== K-OUTER / band-INNER (the feed-starvation fix, 2026-06-29): the OLD band-outer/K-inner loop drained + // band b's ENTIRE K-loop before touching band b+NAFEED, so when an A-feed wave owns >1 band the not-yet-fed + // compute bands starved -> min_cons stalled -> the RINGD-deep B-ring jammed -> WG wedged at ~step RINGD + // (confirmed: 2c3a3b NAFEED>=NCOMP greened; 4c2a2b/6c1a1b hung). NOW: every K-step produces ALL owned bands + // one step, so all compute bands advance lock-step with the ring (mirrors B-feed's K-outer/frag-inner). + // With lock-step production astep == k for every owned band -> use s26 (k) directly; no per-band astep state. + s_lshl_b32 s32, s9, 4 // 16*K (constant across the tile; per-frag stride) + s_mov_b32 s26, 0 // K-OUTER counter k (== astep for every owned band) +.LafeedK: + .set bnd, 0 + .rept NCOMP + .set owner, bnd % NAFEED + s_cmp_eq_u32 s24, owner + s_cbranch_scc0 3f // not owned by this A-feed wave -> skip for this k + // slot-free gate (band bnd at GLOBAL step s60): if gk>=RINGD_A wait cons_a[bnd] > gk-RINGD_A + s_cmp_lt_u32 s60, RINGD_A + s_cbranch_scc1 6f + s_sub_u32 s45, s60, RINGD_A +5: + .if DYNVGPR + s_sleep SLEEPN + .endif + lds_get s46, (CONS_A_OFF + bnd*4) + s_cmp_le_u32 s46, s45 + s_cbranch_scc1 5b +6: + // recompute A saddrs for (band bnd, step k): rowblk=trow*NCOMP+bnd ; + // saddr(mi) = A + rowblk*(16*FM)*K + mi*16*K + k*16 + s_mul_i32 s36, s19, NCOMP + s_add_u32 s36, s36, bnd + s_mul_i32 s22, s36, (16*FM) + s_mul_i32 s22, s22, s9 + s_lshl_b32 s44, s26, 4 // k*16 + s_add_u32 s22, s22, s44 + s_add_u32 s40, s2, s22 + s_addc_u32 s41, s3, 0 + .set mi2, 1 + .rept FM-1 + s_add_u32 s[40+2*mi2], s[40+2*(mi2-1)], s32 + s_addc_u32 s[41+2*mi2], s[41+2*(mi2-1)], 0 + .set mi2, mi2+1 + .endr + // global_load FM A-frags + .set mi3, 0 + .rept FM + global_load_b64 v[ASTG+mi3*2:ASTG+mi3*2+1], v8, s[40+2*mi3:41+2*mi3] + .set mi3, mi3+1 + .endr + s_wait_loadcnt 0x0 + mark 116, s26 // DIAGFINE occ[29] Afeed: step global_load'd from A + // ds_store into A-ring band bnd, slot (GLOBAL gk & (RINGD_A-1)); base folds ARING_OFF+bnd*ABAND_STRIDE + s_and_b32 s45, s60, (RINGD_A-1) + s_mul_i32 s45, s45, (FM*256) + s_add_u32 s45, s45, (ARING_OFF + bnd*ABAND_STRIDE) + v_add_nc_u32 v13, v9, s45 + .set mi3, 0 + .rept FM + ds_store_b64 v13, v[ASTG+mi3*2:ASTG+mi3*2+1] offset:mi3*256 + .set mi3, mi3+1 + .endr + s_wait_dscnt 0x0 + mark 120, s26 // DIAGFINE occ[30] Afeed: step ds_store'd into A-ring (pre-publish) + // publish prod_a[bnd] = gk+1 (GLOBAL cumulative count -> matches compute's global a_step consumer) + s_add_u32 s54, s60, 1 + lds_put (PROD_A_OFF + bnd*4), s54 + mark 124, s54 // DIAGFINE occ[31] Afeed: max prod_a count PUBLISHED +3: + .set bnd, bnd+1 + .endr + s_add_u32 s26, s26, 1 // per-tile k (A-matrix addressing) + s_add_u32 s60, s60, 1 // GLOBAL gk (ring protocol; never resets across tiles) + s_cmp_lt_u32 s26, s8 // k < KT ? + s_cbranch_scc1 .LafeedK + s_branch .Lafeed_loop +.endif // DSWS feed bodies + // ======================================================================================== // FEED BODY (wave 0). Claims a tile, publishes (ti, reset counters, epoch++), produces the FN shared // B-frags per K16-step into the ring, then drains (waits all P compute) before claiming the next tile. - // (v1 = tile-synchronous; cross-tile B-ring overlap = FUTURE ENHANCEMENT FE-1.) + // (v1 = tile-synchronous; cross-tile B-ring overlap = FUTURE ENHANCEMENT FE-1.) DSWS=1: DEAD (the DSWS + // role branch never reaches it) but kept assembled so .Lfeed_exit stays the shared retire path. // ======================================================================================== .Lfeed_role: s_mov_b32 s55, 0 // GLOBAL prod step (cumulative B-steps published; MONOTONIC, diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_dsws.s b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_dsws.s new file mode 100644 index 000000000000..2bd7c5899dfc --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_dsws.s @@ -0,0 +1,1881 @@ +// occ_kernel_dsws.s (gfx1201, wave32) -- MAD-305 DSWS v2 SUBSTRATE SCAFFOLD (PLAN_DSWS_SUBSTRATE_V2.md, Task A1). +// +// v2 re-founds the DSWS GEMM substrate so matrix work is *claimed* (not owned by compile-time wave id), with +// split-K folded in. Work = a pool of (mblk, tcol, ksi) super-tiles; a pinned claimer (wid 0) broadcasts the +// current super-tile; live waves of each role drain shared LDS atomic counters against resident-in-LDS A/B. +// +// Task A1 is SCAFFOLD-ONLY: lift the proven prologue/arming setup from occ_kernel_coop.s (NEVER modified), +// assert the v2 LDS layout fits the 32 KB group segment, and leave each role as a distinct s_endpgm stub. +// The real claimer / feed / compute loops land in A3..A7. The only gates here: (1) assembles clean, +// (2) RGA 0-spill, (3) harness --dsws2 dry-prints the computed params. +// +// ============================================================================================ +// v2 KERNARG CONTRACT (USER_SGPR=15, s0..s14; hardware-preloaded user SGPRs). The host A8 launch +// MUST set COMPUTE_PGM_RSRC2.USER_SGPR=15 and load COMPUTE_USER_DATA_0..14 to match EXACTLY: +// s0:s1 = occ buffer base (>=0x1000B; host zero-inits; see CLAIM-COUNTER / completion offsets below) +// s2:s3 = A base (fp8 e4m3, row-major, 1 byte/elem) +// s4:s5 = Bshuf base (the shuffled-B layout the coop B-feed consumes; same global_load_tr_b64 idiom) +// s6:s7 = C base (fp32; HOST MUST MEMSET C=0 before dispatch -- compute uses global_atomic_add_f32) +// s8 = KT (total K16-steps for the whole matrix = K/16) +// s9 = K(bytes/A-row) (= K, fp8 1 byte/elem) +// s10 = NT*256 (B-saddr advance per K16-step) +// s11 = TOTAL (coop-style total *tiles* = MTL*NTL; carried for addressing compat, NOT the pool size) +// s12 = magic(ceil(2^32/NTL)) (unsigned-div magic for /NTL ; tcol/mblk decode) +// s13 = NTL (number of N tile-columns) +// s14 = FN*256 (B-saddr stride per N-frag) +// (TGID_X now lands in s15 -- UNUSED; this kernel is pool-claim, not workgroup-id based.) +// NOTE: G and SEGK are COMPILE-TIME defsyms (baked into instruction immediates); they are NOT kernargs. +// FIX 1 (round-table Opus+Codex pass): v1 of this contract passed n_kseg/TOTAL_super/magic_kseg as +// s15/s16/s17, but the PM4 host only preloads COMPUTE_USER_DATA_0..15 (USER_SGPR<=16; every proven +// launch path in this tree uses 15) -- s16/s17 could NEVER actually arrive in hardware SGPRs, AND +// s16 was independently being reused per-chunk on the host as the compositor-safe chunk terminal +// (a second, unrelated collision on the same slot). This file now drops s15/s16/s17 entirely: +// n_kseg is DERIVED in-kernel from KT (s8) and the compile-time KSEG_STEPS=SEGK/16: +// n_kseg = KT >> NKSEG_SHIFT, where NKSEG_SHIFT=log2(KSEG_STEPS) is a compile-time `.set` +// (small .if ladder over KSEG_STEPS in {1,2,4,8,16}; SEGK is always a power-of-two +// multiple of 16, so KSEG_STEPS is always a power of two in that set). +// shift/mask (the sti -> (t,ksi) split) are derived ONCE in the prologue from n_kseg: +// shift = s_ff1_i32_b32(n_kseg) (bit index of n_kseg's single set bit; n_kseg=1 -> 0) +// mask = n_kseg - 1 +// DECODE_STI then does ksi = sti & mask ; t = sti >> shift -- this handles n_kseg=1 for +// free (shift=0, mask=0 -> ksi=0, t=sti), so the old magic-div n_kseg==1 special-case is +// GONE (it's no longer needed, not just hidden). +// the chunk terminal (old TOTAL_super/"chunkHi") is now MEMORY-CARRIED instead of a kernarg: the +// host writes the current chunk's terminal sti bound to occ[24] (occW[6]) once per chunk; +// the claimer reads occ[24] ONCE per dispatch (stable for the whole chunk) instead of +// receiving it as a broadcast kernarg. On sti >= occ[24] the claimer publishes a SENTINEL +// (0xFFFFFFFF) into STI_OFF instead of the raw over-claimed sti; followers (b-feed/a-feed/ +// compute) retire when STI_OFF == 0xFFFFFFFF instead of comparing against the (now +// nonexistent) TOTAL_super kernarg. +// See "CLAIM-COUNTER & completion occ offsets" by .Lclaimer for the full occ-buffer layout +// (occ[24]/occW[6] = chunk terminal bound, added by FIX 1). +// +// SCALAR REGS (derived in the prologue, before any clobber; none collide with DECODE_STI's own clobber +// list s18/s36, lds_*'s s49, the claimer's s16/s17/s35/s44, or any role body's transients, all <= s65): +// s66 = n_kseg (derived; dead after shift/mask below are computed -- kept only for that derivation) +// s67 = mask (n_kseg - 1) -- LIVE for the whole kernel; read by every DECODE_STI call, every role. +// s68 = shift (log2 n_kseg) -- LIVE for the whole kernel; read by every DECODE_STI call, every role. +// s69 = chunkHi (claimer-only; loaded once per dispatch from occ[24] right before .Lclaim_loop). +// +// Everything new is gated behind the fresh `DSWS2` build symbol (analogous to coop's `DSWS`). + +.amdgcn_target "amdgcn-amd-amdhsa--gfx1201" + +// ---- tile defsyms (lifted from occ_kernel_coop.s) ---- +.ifndef FM + .set FM, 2 // per-compute-wave M-frags (M-band = FM*16 rows) +.endif +.ifndef FN + .set FN, 4 // shared N-frags (the reuse operand) +.endif +.ifndef RGADESC + .set RGADESC, 0 // 1 = emit analysis-only AMDHSA descriptor for RGA livereg +.endif +.ifndef DIAG + .set DIAG, 0 // 1 = phase-marker instrumentation (unused in the A1 scaffold) +.endif +.ifndef SAFEPROBE + .set SAFEPROBE, 0 // 1 = clamp per-lane vector address regs into a provable in-buffer bound +.endif +.ifndef TFPROBE + .set TFPROBE, 0 // 1 = emit s_sendmsg_rtn GET_REALTIME wall-span capture (each wave stamps +.endif // occ[2]=min entry tick, occ[3]=max exit tick -> host TF readout). + // Default 0 => ZERO bytes emitted, .text byte-identical to the + // production bin (uses only already-allocated regs s30/s31/s49/v5). +.ifndef PHASEPROBE + .set PHASEPROBE, 0 // 1 = in-kernel PHASE TIMER: each compute wave stamps GET_REALTIME at every +.endif // phase boundary and atomic-adds the delta into per-phase occ accumulators + // (occ[64..69], bytes 256..276, ABOVE the per-chunk memset -> accumulate over + // the whole run). Host prints ticks + % per phase. Default 0 => byte-identical. +.ifndef NOCFLUSH + .set NOCFLUSH, 0 // PERF PROBE ONLY: 1 = skip the global_atomic_add_f32 C-flush loop (keep ALL +.endif // other bookkeeping/handshake). Isolates the device-atomic C-reduction cost + // from the coordination handshake. Result is WRONG (C never written -> oracle + // fails) -- span/TF only. Default 0 => byte-identical. +.ifndef KMAJOR + .set KMAJOR, 0 // PERF PROBE: 1 = K-MAJOR super-tile traversal. Default decode packs ksi in +.endif // the LOW bits (all n_kseg segments of a C cell claimed consecutively -> + // up to n_kseg WGs hammer one C cell at once = max atomic contention). + // KMAJOR decodes ksi = sti / TOTAL (high), t = sti % TOTAL (low) via a + // magic-div (magic_TOTAL from occ[62], loaded to s76 in prologue) -> the + // 32 segments of a cell are spread TOTAL apart in claim order (near-zero + // concurrent contention) + adjacent tiles (shared operand bands) claimed + // close in time (L2 reuse). Correctness-preserving (C add is commutative). +.ifndef CSTORE + .set CSTORE, 0 // PERF PROBE ONLY: 1 = replace the flush's global_atomic_add_f32 with an +.endif // equal-count NON-atomic global_store_b32 (same #mem-ops, same addresses, NO + // RMW/contention). Isolates atomic-contention from raw write-bandwidth: if + // TF jumps vs atomics -> contention; if flat -> bandwidth-bound. Result WRONG + // (last-writer, no accumulation) -- span/TF only. Default 0 => byte-identical. +.ifndef TRACE + .set TRACE, 0 // 1 = per-super-tile CLAIMER trace: append one row/super-tile {tick, segcnt, +.endif // epoch, nComp/nAfeed/nBfeed live role slots, ring occA/occB peak, + // convCount, vresv, sti, quiesce} to a host-provided buffer (VA in + // occ[52:53], cap in occ[54]). Time-series of the adaptive wave-role + // economy. Requires DSWS2_CONV=1 (rows written in the quiesce path). + // Default 0 => ZERO bytes; uses free high SGPRs s70..s74. +.ifndef DYNVGPR + .set DYNVGPR, 1 // 1 = compute waves s_alloc_vgpr-grow per rowblk; feeds/claimer stay lean 32 +.endif +.ifndef SLEEPN + .set SLEEPN, 2 // s_sleep arg in the busy-waits (yield issue cycles to partner waves) +.endif + +// ============================================================================================ +// DSWS v2 LDS layout (bytes from group-segment base; words u32 unless noted). Mirrors the placement +// of the coop file's LDS `.set` block. Defined unconditionally (uses only G/SEGK/FM/FN, always set). +// ============================================================================================ +.ifndef DSWS2 + .set DSWS2, 0 +.endif +.ifndef G + .set G, 6 // cooperative M-extent (rowblks per super-tile) = NCOMP_MAX +.endif +.ifndef SEGK + .set SEGK, 64 // split-K segment size in K-elements (multiple of 16) +.endif +// ---- v2 control/claim words ---- +.set STI_OFF, 0 // broadcast super-tile id +.set EPOCH_OFF, 4 +.set ROWBLK_NEXT_OFF, 8 // per-super-tile rowblk claim counter +.set ROWBLK_DONE_OFF, 12 // per-super-tile completion counter +.set BFRAG_NEXT_OFF, 16 // B-frag claim counter +.set AROW_NEXT_OFF, 20 // A-rowblk claim counter +.set NCOMP_SLOT, 24 +.set NAFEED_SLOT, 28 +.set NBFEED_SLOT, 32 +.set GATE_OFF, 36 // u32[4] -> 36,40,44,48 (conversion gates) +.set VRESV_OFF, 52 // vgpr_reserved +.set SEGCNT_OFF, 56 // controller clock +// ---- A3..A7 additions (still inside the 0..256 control region; A1 offsets 0..56 unchanged) ---- +.set BFRAG_DONE_OFF, 60 // B-frag STORE-completion counter (compute gates on this, NOT the claim ctr) +.set AROW_DONE_OFF, 64 // A-rowblk STORE-completion counter (compute gates on this) +.set INITFLAG_OFF, 68 // barrier-free LDS-init publish flag (claimer writes 0xACED LAST) +// ---- Phase-B (DSWS2_CONV) control state: role-mix snapshot slots + quiesce counter ---- +// Based at INITFLAG_OFF+4 (NOT the brief's SEGCNT_OFF+4): the brief predates the A3..A7 control +// words (BFRAG_DONE/AROW_DONE/INITFLAG at 60/64/68), so SEGCNT_OFF+4=60 would collide with them. +// Basing after the LAST control word keeps the new state inside the 0..255 control gap BELOW the +// fixed resident region (BRES_OFF=256), so NO resident-region repoint is needed -- the resident +// BRES_OFF/ARES_OFF immediates (emitted unconditionally in the kernel body) stay untouched, which +// is what keeps the DSWS2_CONV=0 binary byte-identical to the Phase-A green bin. All `.set`s here +// are inert (emit no bytes); the only new code (claimer init) is gated under `.if DSWS2_CONV`. +.ifndef DSWS2_CONV + .set DSWS2_CONV, 0 // 0 = pre-conversion static substrate (Phase A green); 1 = Phase B +.endif +.ifndef DSWS2_TICKET_SELFTEST + .set DSWS2_TICKET_SELFTEST, 0 // DIAG-only try_gate single-winner smoke (Task 4 Step 3); default 0 = no bytes +.endif +.ifndef CONV_COOLDOWN + .set CONV_COOLDOWN, 0 // Task 4: per-wave post-conversion cooldown epochs. 0 = spec-faithful (no + // cooldown, byte-identical to pre-Task-4); >0 damps thrash (skip N epochs + // of watermark decision after a wave converts role). +.endif +// Task 5: deterministic bring-up hook. DSWS2_FORCE=1 makes exactly wave DSWS2_FORCE_WID convert +// direction DSWS2_FORCE_DIR at epoch DSWS2_FORCE_EPOCH, watermarks bypassed -- a reproducible, +// single-wave/single-epoch GPU proof of role conversion. Default DSWS2_FORCE=0 emits ZERO bytes +// (byte-identical to pre-Task-5). +.ifndef DSWS2_FORCE + .set DSWS2_FORCE, 0 +.endif +.ifndef DSWS2_FORCE_WID + .set DSWS2_FORCE_WID, 0 +.endif +.ifndef DSWS2_FORCE_DIR + .set DSWS2_FORCE_DIR, 0 // 0/1 = compute->A/B ; 2/3 = A/B->compute +.endif +.ifndef DSWS2_FORCE_EPOCH + .set DSWS2_FORCE_EPOCH, 1 +.endif +// Rolling dyn-VGPR sum-envelope (2026-07-02 spec). ENVELOPE routes the per-rowblk compute burst grow +// through the shared vgpr_reserved counter so at most PEAK_CONC waves hold peak at once (the +// multi-grower collision, ISA 3.3.3.2, becomes unreachable). All default to the byte-identical value: +// ENVELOPE=0/STAGGER=0 emit ZERO new bytes and PEAK_CONC/STAGGER_PERIOD are inert unless their gate is on. +.ifndef DSWS2_ENVELOPE + .set DSWS2_ENVELOPE, 0 // 1 = route the per-rowblk compute burst grow through the vgpr_reserved +.endif // sum-envelope. 0 = HEAD (bare .Lcompute_grow) -> .text byte-identical. +.ifndef PEAK_CONC + .set PEAK_CONC, 2 // concurrent compute peaks the budget admits (R3 sweep). Used iff ENVELOPE=1. +.endif +.ifndef DSWS2_STAGGER + .set DSWS2_STAGGER, 0 // 1 = lock-free phase-token stagger (Task 9). 0 -> emergent envelope stagger. +.endif +.ifndef STAGGER_PERIOD + .set STAGGER_PERIOD, NCOMP // phase slots in the stagger ring (R3 sweep). Used iff STAGGER=1. +.endif +.set SNAP_BASE, (INITFLAG_OFF + 4) // u32[6]: [parity*3 + {0:nC,1:nA,2:nB}] role-mix snapshots +.set QUIESCE_CNT_OFF,(SNAP_BASE + 6*4) // u32 role-agnostic bail counter (LDS; DSWS2_GQUIESCE=0) +// SENSOR FIX: the claimer publishes its MID-DRAIN ring-occupancy PEAK here each super-tile; the conversion +// decisions read THESE instead of sampling occ_sample at their own quiesce (where occ_X reads ~0 post-drain +// -> always "starved" -> the 4/2/2->1/6/1 compute->feed runaway). Mid-drain peak = the true demand signal. +.set OCCA_PUB_OFF, (QUIESCE_CNT_OFF + 4) // claimer-published occ_A peak +.set OCCB_PUB_OFF, (OCCA_PUB_OFF + 4) // claimer-published occ_B peak +.set DSWS2_STATE_END,(OCCB_PUB_OFF + 4) +// DSWS2_GQUIESCE (2026-07-02 SUSPECT #2 candidate fix): route the QUIESCE handshake through a DEVICE-SCOPED +// GLOBAL atomic in the uncached occ buffer (byte QUIESCE_GOFF), mirroring the GREEN occ[20] claim/occ[0] +// live handshake, instead of the barrier-free LDS counter (whose cross-wave visibility is unguaranteed and +// is the leading SUSPECT #2 hang mechanism). occ buffer = AllocGpu 0x1000 (1024 u32, uncached); host uses +// occ[0..6] + DIAG scratch (<= byte 116); byte 200 (occ[50]) is provably free. Default 0 => LDS path, +// .text byte-identical. Requires DSWS2_CONV (QUIESCE only exists there). +.ifndef DSWS2_GQUIESCE + .set DSWS2_GQUIESCE, 0 +.endif +.set QUIESCE_GOFF, 200 // occ[] byte offset for the global QUIESCE counter (occ[50]) +// ---- TRACE (per-super-tile time-series) occ handshake words + row layout ---- +.set TRACE_PTR_OFF, 208 // occ[52:53] = trace buffer VA (host writes lo/hi per chunk) +.set TRACE_CAP_OFF, 216 // occ[54] = MAXROWS (host-provided row capacity) +.set TRACE_IDX_OFF, 220 // occ[55] = GLOBAL row-claim counter (all WGs' claimers share it) +.set TRACE_WGID_OFF, 224 // occ[56] = GLOBAL wg-id dispenser (claim-order 0..pool-1) +.set FATLIVE_OFF, 228 // occ[57] = live count of GROWN (fat NFV-VGPR) compute waves +.set FATMAX_OFF, 232 // occ[58] = PEAK concurrent fat waves -> x NFV = VGPR in flight (== B probe) +.set ALLLIVE_OFF, 240 // occ[60] = live count of ALL resident waves (++entry/--exit) + // occ[1] (byte 4) = PEAK concurrent resident waves (vs 2048 HW ceiling) +.set TRACE_ROW_BYTES, 64 // 16 u32/row +// DSWS2_BAILMARK (SUSPECT #2 localization, 2026-07-03): each follower publishes its OWN epoch (s35) to a +// PER-WAVE occ slot (BAIL_BASE + wid*4) at its _quiesce bail. One-shot per super-tile per wave -> minimal +// timing perturbation (NOT the claimer's per-spin DIAG poll stores, which are the heisenbug source and stay +// DIAG-only). After a watchdog abort the host reads occ[BAIL_BASE/4 + wid]: every follower's slot == the +// hung epoch => all reached their bail (=> a QUIESCE visibility/lost-update, gq relevant); ONE slot stale +// at the prior epoch => that exact wave is the STRAGGLER (stuck in _alloc/_init/_follow; gq irrelevant). +// Per-WAVE (not per-role): 4 compute share one role, so a role mark's last-writer-wins would hide a single +// straggler. Default 0 => no bytes, .text byte-identical. Requires DSWS2_CONV. +.ifndef DSWS2_BAILMARK + .set DSWS2_BAILMARK, 0 +.endif +.set BAIL_BASE, 160 // occ[] byte offset base for per-wave bail marks: occ[40..47] +.set CONVCNT_OFF, 192 // occ[48]: DIAG conversion-commit counter (proves waves switch role) + // (host prints occ[40..47] as BAIL[w0..w7]; clear of the + // occ[32..36]/occ[39] DSWS sensor+roles slots and occ[50] gq) +.set KSEG_STEPS, (SEGK/16) // K16-steps per split-K segment = SEGK K-elements / 16 +// FIX 1(b): NKSEG_SHIFT = log2(KSEG_STEPS), so the prologue can derive n_kseg = KT >> NKSEG_SHIFT instead +// of receiving it as a (now-dropped) kernarg. SEGK is always a power-of-two multiple of 16 in every +// config this file is built with, so KSEG_STEPS is always a power of two in {1,2,4,8,16}; a static +// ladder over that small set is simpler/safer than a general-purpose compile-time log2. +.if KSEG_STEPS == 1 + .set NKSEG_SHIFT, 0 +.elseif KSEG_STEPS == 2 + .set NKSEG_SHIFT, 1 +.elseif KSEG_STEPS == 4 + .set NKSEG_SHIFT, 2 +.elseif KSEG_STEPS == 8 + .set NKSEG_SHIFT, 3 +.elseif KSEG_STEPS == 16 + .set NKSEG_SHIFT, 4 +.else + .error "KSEG_STEPS (SEGK/16) must be a power of two in {1,2,4,8,16}" +.endif +// resident regions aligned to 256B +.set BRES_OFF, 256 // resident B for current super-tile +.set BRES_BYTES, (FN*16*SEGK) // = 4*16*64 = 4096 at the default config +.set ARES_OFF, (BRES_OFF + BRES_BYTES) // resident A for current super-tile +.set ARES_BYTES, (G*16*FM*SEGK) // = 6*16*2*64 = 12288 at the default config +.set LDS_TOTAL_DSWS2, (ARES_OFF + ARES_BYTES) +.if LDS_TOTAL_DSWS2 > 32768 + .error "DSWS2 LDS layout exceeds 32768B group segment" +.endif +// Phase-B state must fit in the control gap below the resident region (inert compile check, no bytes). +.if DSWS2_STATE_END > BRES_OFF + .error "DSWS2 Phase-B state (SNAP_BASE/QUIESCE_CNT) overlaps resident B region (BRES_OFF)" +.endif + +.if DSWS2 + // ---- role counts (lifted from coop's `.ifndef NCOMP` etc., gated under DSWS2) ---- + .ifndef NCOMP + .set NCOMP, 4 // compute waves (fat, dyn-grow). Compute floor >= 1. + .endif + .ifndef NAFEED + .set NAFEED, 2 // A-feed waves (lean). Feed floor >= 1. + .endif + .ifndef NBFEED + .set NBFEED, 2 // B-feed waves (lean). Feed floor >= 1. + .endif + .set WAVES, (NCOMP + NAFEED + NBFEED) // total waves launched per WG (harness dims must match) +.endif + +// ============================================================================================ +// VGPR layout (lifted from occ_kernel_coop.s) -- compute frags live ABOVE the lean-32 block and are +// only touched AFTER s_alloc_vgpr NFV. Feeds/claimer stay in the lean block (v0..v31). +// ============================================================================================ +.set ACC, 32 // accumulators: FM*FN frags x 8 f32 (v32..) +.set FA, (ACC + 8*FM*FN) // compute A frags (from resident LDS): FM x 2 +.set FB, (FA + 2*FM) // compute B frags (from resident LDS): FN x 2 +.set NFV, ((FB + 2*FN + 15) & ~15) // grown footprint, rounded to a 16-VGPR dyn block (=112 @ 2x4) +.set VLEAN, 32 // lean footprint (feeds, claimer, compute pre/post rowblk) +.set BSTG, 16 // staging regs (lean block, < 32): B-feed FN-frag / A-feed FM-frag + +// ---- dyn-VGPR PRE-GROW temp-reg ceiling (coop death-cert: a >v15 src pre-grow is poison under dyn). +// Gate every PRE-grow-reachable LDS/atomic temp to v11/v14 (INTERIOR to the launched 16-VGPR block). ---- +.if DYNVGPR + .set RG_A, 11 // lds_get / fetch_add address + .set RG_D, 14 // lds_get / fetch_add data+return + .set RP_A, 11 // lds_put address + .set RP_D, 14 // lds_put data +.else + .set RG_A, 27 + .set RG_D, 28 + .set RP_A, 28 + .set RP_D, 29 +.endif + +// ============================================================================================ +// LDS helper macros (s49 = exec save; v2 = lane = tid&31, set in prologue). +// ============================================================================================ +.macro lds_get sdst, off // wave-uniform read LDS[off] -> scalar sdst + v_mov_b32 v[RG_A], \off + ds_load_b32 v[RG_D], v[RG_A] + s_wait_dscnt 0x0 + v_readfirstlane_b32 \sdst, v[RG_D] +.endm +.macro lds_get_r sdst, saddr // wave-uniform read LDS[saddr] (RUNTIME addr in a sreg) -> sdst + v_mov_b32 v[RG_A], \saddr + ds_load_b32 v[RG_D], v[RG_A] + s_wait_dscnt 0x0 + v_readfirstlane_b32 \sdst, v[RG_D] +.endm +.macro lds_put off, ssrc // lane-0-of-wave writes scalar ssrc -> LDS[off] + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lput_skip\@ + v_mov_b32 v[RP_A], \off + v_mov_b32 v[RP_D], \ssrc + ds_store_b32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lput_skip\@: + s_mov_b32 exec_lo, s49 +.endm +.macro lds_fetch_add sdst, off, val // sdst <- old LDS[off]; LDS[off]+=val (lane-0 atomic, broadcast) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lfa_skip\@ + v_mov_b32 v[RP_A], \off + v_mov_b32 v[RP_D], \val + ds_add_rtn_u32 v[RP_D], v[RP_A], v[RP_D] // v[RP_D] <- old; LDS[off] += val + s_wait_dscnt 0x0 +.Lfa_skip\@: + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 \sdst, v[RP_D] // broadcast lane-0's old value +.endm +.macro lds_inc off // lane-0-of-wave LDS[off] += 1 (no return) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Linc_skip\@ + v_mov_b32 v[RP_A], \off + v_mov_b32 v[RP_D], 1 + ds_add_u32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Linc_skip\@: + s_mov_b32 exec_lo, s49 +.endm + +// ============================================================================================ +// Super-tile decode + resident A/B staging macros (A3..A6). Decode (Naming/symbols): +// ksi = sti & mask ; t = sti >> shift ; mblk = t / NTL ; tcol = t % NTL. +// FIX 1(d): n_kseg is ALWAYS a power of two (it's KT >> NKSEG_SHIFT, both compile-time-shift-derived), +// so the sti->(t,ksi) split is an exact shift/mask -- no magic-div, no n_kseg==1 special-case (shift=0, +// mask=0 falls out of the general path for free: ksi=0, t=sti). /NTL still goes via magic (s12), +// unsigned-division mul_hi (coop GENDIV idiom), since NTL is not generally a power of two. +// ============================================================================================ +.macro DECODE_STI // in: s17=sti, s67=mask, s68=shift ; out: s19=mblk s30=tcol s31=ksi ; clob: s18,s36 +.if KMAJOR + s_mul_hi_u32 s31, s17, s76 // ksi = sti / TOTAL (magic-div, magic_TOTAL in s76; K-major high bits) + s_mul_i32 s36, s31, s11 // ksi * TOTAL + s_sub_u32 s18, s17, s36 // t = sti - ksi*TOTAL (low) +.else + s_and_b32 s31, s17, s67 // ksi = sti & mask (mask-bounded -> ksi in [0,n_kseg-1]) + s_lshr_b32 s18, s17, s68 // t = sti >> shift +.endif +.if SAFEPROBE + // brick-PROOF ti clamp (the "future ti clamp" line 752 promised; COOP_STATUS.md:145 racy-garbage-ti->OOB). + // A racy/torn sti read (during the claimer's per-super-tile republish) can decode a garbage t -> garbage + // mblk/tcol -> the A/B/C SCALAR base goes out of buffer -> gfxhub page fault -> MODE1 brick. SAFEPROBE + // already pins the per-lane vaddr (v8/v9/v10); this pins the tile index too, so EVERY global address is + // provably in-buffer. s11=TOTAL is userdata, never clobbered. s36 is DECODE_STI scratch (rewritten below). + s_sub_u32 s36, s11, 1 // TOTAL-1 + s_min_u32 s18, s18, s36 // t clamped to [0,TOTAL-1] -> mblk in-bounds) +.endif + s_mul_hi_u32 s19, s18, s12 // mblk = t / NTL + s_mul_i32 s36, s19, s13 // mblk * NTL + s_sub_u32 s30, s18, s36 // tcol = t - mblk*NTL +.endm + +// RESIDENT B FRAG LAYOUT: B frag (kstep ks, frag f) at BRES_OFF + (ks*FN + f)*256 +// (each frag = the SAME 256B block coop stores per B-ring slot; lane*8 vaddr base = v9). +// Built here as: dst vbase = v9 + BRES_OFF + f*256 , ds_store offset:(ks*FN*256). +// B global addr (lift coop B-feed): Bshuf + tcol*(FN*256=s14) + (seg k0)* [ksi*KSEG_STEPS*(NT*256=s10)] +// + f*256 (frag, folded into saddr) + ks*(NT*256=s10) (k-step, folded into saddr). +.macro BSTAGE // in: s30=tcol s31=ksi ; clob: s20,s21,s23,s25,s26,s27,v13,v[BSTG..] + s_mul_i32 s20, s30, s14 // tcol * FN*256 + s_mul_i32 s21, s31, KSEG_STEPS // ksi * KSEG_STEPS + s_mul_i32 s21, s21, s10 // * NT*256 -> segment k-start byte offset + s_add_u32 s20, s20, s21 + s_add_u32 s20, s4, s20 + s_addc_u32 s21, s5, 0 // s[20:21] = B base (tcol,ksi, seg k-step 0) +.Lbcl\@: + lds_fetch_add s23, BFRAG_NEXT_OFF, 1 // claim frag f + s_cmp_ge_u32 s23, FN + s_cbranch_scc1 .Lbsd\@ // f>=FN -> all frags claimed + s_lshl_b32 s25, s23, 8 // f*256 + s_add_u32 s26, s20, s25 + s_addc_u32 s27, s21, 0 // s[26:27] = frag f base (seg k0) + v_add_nc_u32 v13, v9, BRES_OFF + v_add_nc_u32 v13, v13, s25 // resident B dst vbase for frag f + .set ks, 0 + .rept KSEG_STEPS + global_load_tr_b64 v[BSTG+ks*2:BSTG+ks*2+1], v9, s[26:27] + s_add_u32 s26, s26, s10 // next k-step (last iter over-advances; unused) + s_addc_u32 s27, s27, 0 + .set ks, ks+1 + .endr + s_wait_loadcnt 0x0 + .set ks, 0 + .rept KSEG_STEPS + ds_store_b64 v13, v[BSTG+ks*2:BSTG+ks*2+1] offset:(ks*FN*256) + .set ks, ks+1 + .endr + s_wait_dscnt 0x0 + lds_inc BFRAG_DONE_OFF // frag f STORED -> publish completion (compute gates on this) + s_branch .Lbcl\@ +.Lbsd\@: +.endm + +// RESIDENT A FRAG LAYOUT: A frag (kstep ks, rowblk r, mi) at ARES_OFF + ((ks*G + r)*FM + mi)*256 +// Built as: dst vbase = v9 + ARES_OFF + r*(FM*256) , ds_store offset:((ks*G*FM + mi)*256). +// A global addr (lift coop compute/A-feed): A + rowblk_abs*(16*FM)*K + mi*16*K + koff, rowblk_abs=mblk*G+r, +// koff = ksi*SEGK (segment K byte offset, fp8 1B/elem), k-step within segment via global offset:ks*16. +.macro ASTAGE // in: s19=mblk s31=ksi ; clob: s22,s23,s25,s32,s36,s40,s41,s44,s45,v13,v[BSTG..] + s_lshl_b32 s32, s9, 4 // rowstride16 = 16*K +.Lacl\@: + lds_fetch_add s23, AROW_NEXT_OFF, 1 // claim rowblk r + s_cmp_ge_u32 s23, G + s_cbranch_scc1 .Lasd\@ + s_mul_i32 s36, s19, G + s_add_u32 s36, s36, s23 // rowblk_abs = mblk*G + r + s_mul_i32 s22, s36, (16*FM) + s_mul_i32 s22, s22, s9 // rowblk_abs*(16*FM)*K + s_mul_i32 s25, s31, SEGK // ksi*SEGK (segment K byte offset) + s_add_u32 s22, s22, s25 + s_add_u32 s40, s2, s22 + s_addc_u32 s41, s3, 0 // s[40:41] = A base (rowblk_abs, mi0, seg k0) + s_mul_i32 s25, s23, (FM*256) // r*FM*256 + v_add_nc_u32 v13, v9, ARES_OFF + v_add_nc_u32 v13, v13, s25 // resident A dst vbase for rowblk r + .set mi, 0 + .rept FM + .if mi == 0 + s_mov_b32 s44, s40 + s_mov_b32 s45, s41 + .else + s_add_u32 s44, s44, s32 // += 16*K (next M-frag) + s_addc_u32 s45, s45, 0 + .endif + .set ks, 0 + .rept KSEG_STEPS + global_load_b64 v[BSTG:BSTG+1], v8, s[44:45] offset:(ks*16) + s_wait_loadcnt 0x0 + ds_store_b64 v13, v[BSTG:BSTG+1] offset:((ks*G*FM + mi)*256) + s_wait_dscnt 0x0 + .set ks, ks+1 + .endr + .set mi, mi+1 + .endr + lds_inc AROW_DONE_OFF // rowblk r fully STAGED -> publish completion + s_branch .Lacl\@ +.Lasd\@: +.endm + +// ============================================================================================ +// Phase-B (DSWS2_CONV) consume-point ring-occupancy sensor -- Task 3, READ-ONLY (actuation is Task 5). +// Mirrors the coop occ_a/occ_b sensor (occ = producer - consumer, sampled where the value is +// CONSUMED, not at the segment boundary). The claimer's A7 wait-done spin runs CONCURRENTLY with the +// compute drain, so it observes the ring mid-flight; at the segment boundary the resident region has +// fully drained and occ would read a stuck ~0 (permanent false-starvation) -- exactly what SPEC warns. +// +// COUNTER IDENTITIES (confirmed against the live claim/consume sites -- see report): +// producer = the STORE-completion counters the compute wave actually gates on: +// A-ring: AROW_DONE_OFF (A rowblks resident, monotonic in [0,G]; lds_inc @ ASTAGE) +// B-ring: BFRAG_DONE_OFF (B frags resident, monotonic in [0,FN]; lds_inc @ BSTAGE) +// NOT the *_NEXT claim counters: AROW_NEXT/BFRAG_NEXT overshoot the ring depth by the role +// terminal-bails (G+NAFEED / FN+NBFEED), which would break the occ <= depth bound. +// consumer = ROWBLK_NEXT_OFF, the compute rowblk-claim clock (consume progress through the super-tile: +// each claimed rowblk r consumes A(r) and re-reads all FN shared B frags). +// min-clamp: cons is clamped to prod before the subtract so the u32 result cannot underflow when the +// consume clock outruns a shallower ring (G=6 > FN=4 -> ROWBLK_NEXT can exceed BFRAG_DONE). +// INVARIANT preserved: occ_A in [0,G], occ_B in [0,FN] (nonnegative, bounded by ring depth). +// +// REGISTER DISCIPLINE (brick-critical; this path is reachable pre-grow -- a >v15 vector temp is +// OOR-poison under dyn-VGPR, SPEC S4): scalars <= s65 only (s60/s61 scratch; callers pass dst in +// [s62,s65]); the only vector temps are inside lds_get, which uses v11/v14 (INTERIOR to the launch +// 16-VGPR block) -- NO >v15 temp is introduced here. +// GATE: DSWS2_CONV || DSWS2_ENVELOPE. reserve_try + the BUDGET default are the pool-economy primitives the +// rolling envelope needs INDEPENDENTLY of role conversion (they touch only VRESV_OFF/lds_fetch_add), so the +// envelope must be able to run at CONV=0 (the isolation config). Everything in this block is macro/.set +// definition (emits ZERO bytes), so widening the gate is byte-identical at CONV=0/ENV=0 and CONV=1. +.if DSWS2_CONV || DSWS2_ENVELOPE +.macro occ_sample dst_a, dst_b // out: \dst_a=occ_A in [0,G], \dst_b=occ_B in [0,FN]; clob s60,s61 + lds_get \dst_a, AROW_DONE_OFF // prod_a: A rowblks resident (store-completion) + lds_get \dst_b, BFRAG_DONE_OFF // prod_b: B frags resident (store-completion) + lds_get s60, ROWBLK_NEXT_OFF // cons : compute rowblk-claim consume clock + s_min_u32 s61, s60, \dst_a // cons_a = min(clock, prod_a) (clamp -> no u32 underflow) + s_sub_u32 \dst_a, \dst_a, s61 // occ_A = prod_a - cons_a in [0,G] + s_min_u32 s61, s60, \dst_b // cons_b = min(clock, prod_b) + s_sub_u32 \dst_b, \dst_b, s61 // occ_B = prod_b - cons_b in [0,FN] +.endm + +// ---- DSWS2_GQUIESCE: device-scoped GLOBAL QUIESCE handshake (mirrors the green occ[20]/occ[0] pattern). +// All three ops are lane-0-masked (v2==0), exec saved/restored via s49 (the LDS-macro convention -- s49 is +// never live across a macro boundary, so it is provably free at every site these replace an lds_* op). +// vaddr = v4 (the stable occ-base per-lane offset, =0, prologue-set), data/dst = v3/v5 (occ scratch vregs, +// same as the claim/live ops). scope:SCOPE_DEV + uncached occ buffer => device-coherent visibility (the +// fix). s_wait_storecnt/loadcnt drain before proceeding so the poll observes committed bumps. +.macro gq_reset // claimer: occ[QUIESCE_GOFF] = 0 (committed before EPOCH publish) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lgqr_skip\@ + v_mov_b32 v3, 0 + global_store_b32 v4, v3, s[0:1] offset:QUIESCE_GOFF scope:SCOPE_DEV + s_wait_storecnt 0x0 +.Lgqr_skip\@: + s_mov_b32 exec_lo, s49 +.endm +.macro gq_bump // follower: occ[QUIESCE_GOFF] += 1 (one bump/wave/super-tile) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lgqb_skip\@ + v_mov_b32 v3, 1 + global_atomic_add_u32 v4, v3, s[0:1] offset:QUIESCE_GOFF scope:SCOPE_DEV + s_wait_storecnt 0x0 +.Lgqb_skip\@: + s_mov_b32 exec_lo, s49 +.endm +.macro gq_read dst // claimer: \dst = occ[QUIESCE_GOFF] (lane0 load + broadcast) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lgqrd_skip\@ + global_load_b32 v5, v4, s[0:1] offset:QUIESCE_GOFF scope:SCOPE_DEV + s_wait_loadcnt 0x0 +.Lgqrd_skip\@: + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 \dst, v5 +.endm + +// ---- Pool-T7 chunk-2 wedge localization (DIAG-only; DSWS2_CONV=0 emits nothing -> .text byte-identical). +// epoch_mark: lane-0 publishes this role's live epoch (s35) to a host-streamed occ slot so a hung dispatch +// shows how far each role advanced (stream field roles[C/A/B]). v14<=v15 (feeds/compute are lean-32 at the +// _quiesce call sites), v4=0 (occ base lane offset, prologue), s49 exec-save (LDS-macro convention). ---- +.macro epoch_mark off +.if DSWS2_CONV && DIAG + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lem_skip\@ + v_mov_b32 v14, s35 + global_store_b32 v4, v14, s[0:1] offset:\off scope:SCOPE_DEV +.Lem_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// bail_mark: PER-WAVE localization mark. Lane-0 writes this wave's epoch (s35) to occ[BAIL_BASE + wid*4] +// (runtime vaddr since the offset depends on wid=s24). s48 scratch, s49 exec-save (macro-local; free at the +// _quiesce bail sites), v13 vaddr, v14 data (both <=v15; the wave is lean-32 at every bail site). One-shot +// per super-tile -> negligible perturbation vs the DIAG per-spin claimer stores. Enabled by DIAG OR BAILMARK. +.macro bail_mark +.if DSWS2_CONV && (DIAG || DSWS2_BAILMARK) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lbmk_skip\@ + s_lshl_b32 s48, s24, 2 // wid*4 + s_add_u32 s48, s48, BAIL_BASE // occ byte offset for THIS wave + v_mov_b32 v13, s48 // vaddr = per-wave byte offset (lane0) + v_mov_b32 v14, s35 // data = this wave's current epoch + global_store_b32 v13, v14, s[0:1] scope:SCOPE_DEV +.Lbmk_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// -------------------------------------------------------------------------------------------- +// Phase-B controller thresholds + sum-envelope budget (Task 4). EPOCH_SHIFT mirrors coop / +// occ_dispatch (epoch = segcnt >> EPOCH_SHIFT). BUDGET is the per-WG VGPR sum-envelope ceiling +// the reservation counter must never exceed; default = the launch reservation, which makes the +// envelope a strict conservation law (a feed->compute grow can only fit if a compute->feed shrink +// already freed the delta). Task 5 may re-tune via `-defsym BUDGET=` if per-SIMD headroom exists. +// -------------------------------------------------------------------------------------------- +.ifndef EPOCH_SHIFT + .set EPOCH_SHIFT, 3 // decision clock: epoch = segcnt >> EPOCH_SHIFT (small = reactive) +.endif +.ifndef BUDGET +.if DSWS2_ENVELOPE + .set BUDGET, (WAVES*VLEAN + PEAK_CONC*(NFV-VLEAN)) // rolling: lean floor + concurrent-peak headroom +.else + .set BUDGET, (NCOMP*NFV + (NAFEED+NBFEED)*VLEAN) // = VRESV_OFF init (conservation ceiling) +.endif +.endif + +.if DSWS2_CONV +// compile-time no-parking invariant: every launched wave must fit lean at once +.if (WAVES * VLEAN) > BUDGET + .error "WAVES*VLEAN exceeds BUDGET — pool cannot stay all-lean (parking is out of scope)" +.endif +.endif +.if DSWS2_ENVELOPE +// forward-progress: the budget must admit at least one concurrent peak or a claimed wave can never grow +.if (WAVES*VLEAN + (NFV-VLEAN)) > BUDGET + .error "ENVELOPE: BUDGET admits < 1 concurrent peak — forward progress impossible" +.endif +.endif + +// try_gate: the lock-free single-winner conversion ticket (transcribed VERBATIM from occ_kernel_coop.s, +// which transcribes dsws_ctrl_model.cpp gate_try_win + epoch_of EXACTLY). E = segcnt>>EPOCH_SHIFT. +// gate[dir] holds the last epoch dir fired. Among many waves racing the same (gtry_gate->reserve_try point -- +// occ_sample's s62/s63 result is consumed into `dir` BEFORE this runs), v5/v6/v7 (<=v15: pre-grow / +// lean-safe). CAS operand order (gfx1201, GCN order -- NOT flipped, KG 9ed04f3c): +// ds_cmpstore_rtn_b32 vdst,vaddr,vNEW,vCMP -> MEM=(MEM==vCMP)?vNEW:MEM, vdst<-old. So vsrc0=E (new), +// vsrc1=g (compare). WIN iff returned-old == g. (Swapping them leaves gate stuck so old==g for ALL +// racers -> every racer "wins" -> would-win ~= NCOMP*epochs instead of ~= epochs.) +.macro try_gate dir, swin + lds_get s62, SEGCNT_OFF // E = epoch_of(segcnt, EPOCH_SHIFT) + s_lshr_b32 s62, s62, EPOCH_SHIFT + lds_get s63, (GATE_OFF + (\dir)*4) // g = gate[dir] + s_mov_b32 \swin, 0 + s_cmp_ge_u32 s63, s62 // g >= E -> dir already fired this/later epoch -> lose + s_cbranch_scc1 .Ltg_done\@ + s_mov_b32 s65, exec_lo // lane0-only CAS (one ticket attempt per WAVE) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltg_restore\@ + v_mov_b32 v5, (GATE_OFF + (\dir)*4) // vaddr = &gate[dir] + v_mov_b32 v6, s63 // v6 = g (vsrc1 = CMP/expected) + v_mov_b32 v7, s62 // v7 = E (vsrc0 = NEW value to store) + ds_cmpstore_rtn_b32 v6, v5, v7, v6 // gate[dir] = (gate[dir]==g) ? E : gate[dir]; v6 <- old + s_wait_dscnt 0x0 +.Ltg_restore\@: + s_mov_b32 exec_lo, s65 + v_readfirstlane_b32 s64, v6 // s64 = old (lane0's CAS result, broadcast) + s_cmp_eq_u32 s64, s63 // WIN iff old == g (we were the swapper) + s_cbranch_scc0 .Ltg_done\@ + s_mov_b32 \swin, 1 +.Ltg_done\@: +.endm + +// reserve_try: the VGPR sum-envelope reservation (transcribes reserve_grow, dsws_ctrl_model.cpp:47). +// Reserve first (atomic add of SIGNED \delta on vgpr_reserved), then validate prev+delta <= BUDGET; +// on over-budget cleanly UNDO (atomic add of -\delta) and reject. The LDS atomic serializes the <=2 +// concurrent grows an epoch permits: the second to validate sees the first's reservation and backs off. +// GROW (feed->compute): pass \delta = +(NFV-VLEAN). Over-budget -> undo, \won=0 (stay in role). +// SHRINK(compute->feed): pass \delta = -(NFV-VLEAN). new = prev+delta < prev <= BUDGET, so the +// validate branch is a proven no-op -> \won=1 ALWAYS (shrink never fails). +// One macro, one call site (Task 5 `reserve_try delta, s_ok`); direction is the sign of \delta. +// Scratch: s62/s63 (free at the bail-commit point -- try_gate's s62..s65 are long dead by then). +.macro reserve_try delta, won + lds_fetch_add s62, VRESV_OFF, (\delta) // s62 = prev reserved; vgpr_reserved += delta + s_add_u32 s63, s62, (\delta) // s63 = new reservation = prev + delta + s_mov_b32 \won, 1 + s_cmp_le_u32 s63, BUDGET // new <= BUDGET -> commit (win); shrink always passes + s_cbranch_scc1 .Lrt_done\@ + lds_fetch_add s62, VRESV_OFF, -(\delta) // over-budget: undo the reservation, reject + s_mov_b32 \won, 0 +.Lrt_done\@: +.endm + +// -------------------------------------------------------------------------------------------- +// Phase-B (Task 5) watermark thresholds + LDS put-runtime helper + bail-time commit macros. +// Watermark decision (SPEC; mirrors coop CTRL_LOW/CTRL_HIGH, occ_dispatch DSWS_LOW/HIGH): +// occ_X < CTRL_LOW -> compute STARVED for X -> shrink a compute wave into feed-X. +// occ_X > CTRL_HIGH_X -> feed-X OVER-SERVING -> grow a feed-X wave into compute. +// occ_A in [0,G], occ_B in [0,FN] (occ_sample bounds), so the HIGH marks are per-ring-depth. +// -------------------------------------------------------------------------------------------- +.ifndef CTRL_LOW + .set CTRL_LOW, 1 // occ_X < 1 (== 0, ring empty at consume) -> starved +.endif +.ifndef CTRL_HIGH_A + .set CTRL_HIGH_A, (G-1) // occ_A > G-1 -> A-ring saturated -> A-feed over-serving +.endif +.ifndef CTRL_HIGH_B + .set CTRL_HIGH_B, (FN-1) // occ_B > FN-1 -> B-ring saturated -> B-feed over-serving +.endif + +// lds_put_r: lane-0-of-wave write scalar \ssrc -> LDS[\saddr] (RUNTIME byte offset in a sreg). Mirrors +// the coop lds_put_v idiom but takes a SCALAR address (symmetry with lds_get_r). Used by the claimer's +// Step-4 snapshot write into the runtime parity half of SNAP_BASE. Temps RP_A/RP_D are v11/v14 (<=v15, +// pre-grow safe); s49 is the exec save (matches lds_put). +.macro lds_put_r saddr, ssrc + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lputr_skip\@ + v_mov_b32 v[RP_A], \saddr + v_mov_b32 v[RP_D], \ssrc + ds_store_b32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lputr_skip\@: + s_mov_b32 exec_lo, s49 +.endm + +// conv_dec_floor: floor-guarded ATOMIC decrement of a role slot -- \ok <- 1 iff it decremented \slot_off +// (only when the current value was > 1), else 0 (floor hit; source role must keep >= 1 wave). A +// ds_cmpstore_rtn_b32 CAS loop (re-reads on a lost race), so two same-source converters in one epoch +// (e.g. compute->Afeed and compute->Bfeed both dec NCOMP_SLOT) can never drive the slot below 1. +// Clob: s52 (read value), s53 (new/CAS-return), s65 (exec save); v5/v6/v7 (<=v15, pre-grow safe). +.macro conv_dec_floor slot_off, ok + s_mov_b32 \ok, 0 +.Lcdf_retry\@: + lds_get s52, \slot_off // s52 = current source-slot count + s_cmp_le_u32 s52, 1 + s_cbranch_scc1 .Lcdf_done\@ // <=1 -> at floor, cannot convert away (ok stays 0) + s_sub_u32 s53, s52, 1 // new = old - 1 + s_mov_b32 s65, exec_lo // lane0-only CAS (one attempt per WAVE) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lcdf_restore\@ + v_mov_b32 v5, \slot_off // vaddr = &slot + v_mov_b32 v6, s52 // v6 = expected old (CMP) + v_mov_b32 v7, s53 // v7 = new value (NEW) + ds_cmpstore_rtn_b32 v6, v5, v7, v6 // slot = (slot==old)? new : slot ; v6 <- prior + s_wait_dscnt 0x0 +.Lcdf_restore\@: + s_mov_b32 exec_lo, s65 + v_readfirstlane_b32 s53, v6 // s53 = prior (lane0 CAS result, broadcast) + s_cmp_eq_u32 s53, s52 // success iff prior == expected (we were the swapper) + s_cbranch_scc0 .Lcdf_retry\@ // lost the race -> re-read and retry + s_mov_b32 \ok, 1 +.Lcdf_done\@: +.endm + +// conv_apply: the bail-time role-conversion COMMIT (SPEC 3.4 Approach A). Precondition: s58 = s_win +// (1 iff this wave won the (dir,epoch) ticket). Ordered strictly BEFORE the QUIESCE_CNT bump the +// CALLER emits after this macro (the quiesce counter is the snapshot handshake). +// ORDER: (a) floor-guarded dec of \src_slot -> (b) reserve the VGPR sum-envelope \delta (shrink +// always ok; grow may abort over BUDGET) -> (c) on ok: inc \dst_slot, flip private role reg (s59), +// s_alloc_vgpr \alloc_sz (GROW=NFV feed->compute / SHRINK=32 compute->feed) with SCC-retry -> +// (d) on floor-fail or reserve-abort: cancel, remain current role (undo the source dec if a +// reservation abort happened after the dec). +// PRE-GROW OOR WINDOW (SPEC 4, #1 brick risk): the wave is lean-32 on entry; every LDS/atomic temp +// read before the s_alloc_vgpr GROW is <=v15 (occ_sample/try_gate v5/v6/v7 + v11/v14; conv_dec_floor +// v5/v6/v7; lds_fetch_add v11/v14) and every carried scalar is <=s65. NO >v15 source before GROW. +// Clob: s52,s53,s54 (+ conv_dec_floor / reserve_try scratch); s59 = new role slot id (record). +.macro conv_apply src_slot, dst_slot, delta, alloc_sz + s_cmp_eq_u32 s58, 0 + s_cbranch_scc1 .Lca_skip\@ // lost the ticket -> no conversion this bail + conv_dec_floor \src_slot, s54 // (a) floor-guarded atomic dec of source slot + s_cmp_eq_u32 s54, 0 + s_cbranch_scc1 .Lca_skip\@ // floor-fail (source at 1) -> cancel, remain current role + reserve_try (\delta), s53 // (b) reserve VGPR envelope (grow may abort; shrink ok) + s_cmp_eq_u32 s53, 0 + s_cbranch_scc0 .Lca_commit\@ + lds_fetch_add s52, \src_slot, 1 // (d) reserve aborted: UNDO the source dec, cancel + s_branch .Lca_skip\@ +.Lca_commit\@: + lds_fetch_add s52, \dst_slot, 1 // (c) inc dest slot (unbounded -> plain atomic add) + s_mov_b32 s59, \dst_slot // flip private current-role reg (records new role slot id) +.if DIAG || TRACE + // conversion-commit counter (proves a wave ACTUALLY switched role). Lean-32 pre-grow here -> v3/v4<=v15 + // OOR-safe; s49 exec-save (macro-local). (DIAG||TRACE)-gated -> DSWS2_CONV/DIAG=0/TRACE=0 byte-identical. + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lca_cm_skip\@ + v_mov_b32 v3, 1 + global_atomic_add_u32 v4, v3, s[0:1] offset:CONVCNT_OFF scope:SCOPE_DEV // occ[48] += 1 + s_wait_storecnt 0x0 +.Lca_cm_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.if CONV_COOLDOWN > 0 + s_mov_b32 s66, CONV_COOLDOWN // Task 4: committed conversion -> arm cooldown +.endif + // ---- s_alloc_vgpr resize: THE pre-grow OOR window closes here; all reads above were <=v15 ---- +.Lca_alloc\@: + s_alloc_vgpr \alloc_sz // GROW(NFV) / SHRINK(32); SCC-retry (brick-class rule) + s_cbranch_scc0 .Lca_alloc\@ +.Lca_skip\@: +.endm +.endif + +// ============================================================================================ +// TFPROBE wall-span capture (TF throughput probe). Realtime-tick min/max into occ[2]/occ[3], +// mirroring occ_kernel_coop.s's proven timer idiom. Each wave stamps lane-0 only (exec-masked +// via s49, the DSWS exec-save convention); base addr v4==0 holds kernel-wide (prologue, line +// ~765; invariant per the "v4=0 occ base lane offset" note). s[30:31] free at entry (DECODE_STI +// outputs, computed only inside role bodies) and dead at every terminal. Emits ZERO bytes at +// TFPROBE=0 -> production .text byte-identical. +// ============================================================================================ +.macro tfspan op:req, off:req // op = min (entry, occ[2]/off 8) | max (exit, occ[3]/off 12) +.if TFPROBE + s_sendmsg_rtn_b64 s[30:31], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + v_cmp_eq_u32 vcc_lo, 0, v2 // lane 0 of each wave only (v2 = tid & 31) + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltfspan_skip\@ + v_mov_b32 v5, s30 // low 32 bits of the realtime tick + global_atomic_\op\()_u32 v4, v5, s[0:1] offset:\off scope:SCOPE_DEV +.Ltfspan_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// ============================================================================================ +// PHASEPROBE: in-kernel per-phase tick timer for the COMPUTE wave (the critical path). s77 holds +// this wave's last-stamp RTC (low 32b). phase_reset seeds it; phase_stamp accumulates (now-s77) +// into occ[\off] and re-seeds. Lane-0-only atomic add; occ slots are ABOVE the per-chunk memset +// so they sum over the whole run. Scratch s62/s63 (RTC), s64 (delta) -- all free in CONV=0 compute. +// Six phases -> six occ accumulators (bytes 256..276): +.set PH_FOLLOW_OFF, 256 // occ[64]: waiting on claimer to publish next super-tile +.set PH_STAGE_OFF, 260 // occ[65]: waiting on A/B feeds to stage this super-tile +.set PH_GROW_OFF, 264 // occ[66]: claim rowblk + s_alloc_vgpr GROW 32->112 +.set PH_WMMA_OFF, 268 // occ[67]: LDS frag loads + v_wmma compute +.set PH_FLUSH_OFF, 272 // occ[68]: global_atomic_add_f32 C flush (split-K reduction) +.set PH_SHRINK_OFF, 276 // occ[69]: s_alloc_vgpr SHRINK 112->32 +// Per-wave phase accumulators live in SGPRs s78..s83 (NO per-stamp store -> zero memory perturbation, no +// s_wait_storecnt pollution). s77 = last-stamp RTC. phase_flush emits them ONCE at compute retire. +.macro phase_reset +.if PHASEPROBE + s_mov_b32 s78, 0 + s_mov_b32 s79, 0 + s_mov_b32 s80, 0 + s_mov_b32 s81, 0 + s_mov_b32 s82, 0 + s_mov_b32 s83, 0 + s_sendmsg_rtn_b64 s[62:63], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + s_mov_b32 s77, s62 +.endif +.endm +.macro phase_stamp acc:req // \acc += (now - s77); s77 = now (pure scalar, no store) +.if PHASEPROBE + s_sendmsg_rtn_b64 s[62:63], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + s_sub_u32 s64, s62, s77 // delta (mod 2^32; phase << 2^32 so wrap-safe) + s_mov_b32 s77, s62 + s_add_u32 \acc, \acc, s64 +.endif +.endm +.macro phase_flush // lane0 atomic-adds s78..s83 -> occ[64..69]; drained here (not the hot loop) +.if PHASEPROBE + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lphf_skip\@ + v_mov_b32 v5, s78 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_FOLLOW_OFF scope:SCOPE_DEV + v_mov_b32 v5, s79 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_STAGE_OFF scope:SCOPE_DEV + v_mov_b32 v5, s80 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_GROW_OFF scope:SCOPE_DEV + v_mov_b32 v5, s81 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_WMMA_OFF scope:SCOPE_DEV + v_mov_b32 v5, s82 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_FLUSH_OFF scope:SCOPE_DEV + v_mov_b32 v5, s83 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_SHRINK_OFF scope:SCOPE_DEV + s_wait_storecnt 0x0 +.Lphf_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// ============================================================================================ +// TRACE: per-super-tile time-series row (claimer, lane 0). Written once per super-tile at the +// quiesce-satisfied drain-exit (.Lqc_q_ok). Captures the adaptive wave-role economy over time: +// the LIVE role slots (do waves convert?), the per-super-tile ring-occupancy PEAK (s73/s74, +// tracked across the wait_done spins), the cumulative conversion count, and the envelope vresv. +// 16 u32/row -> buffer[segcnt*64]; bounded by MAXROWS (s72). Emits ZERO bytes at TRACE=0. +// Persistent trace regs: s70:s71 = buffer VA, s72 = MAXROWS, s73/s74 = ring occA/occB peak. +// ============================================================================================ +.macro alllive_dec // TRACE: --live on wave exit (pairs with the entry ++ for peak-concurrent) +.if TRACE + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lalld_skip\@ + v_mov_b32 v3, -1 + global_atomic_add_u32 v4, v3, s[0:1] offset:ALLLIVE_OFF scope:SCOPE_DEV +.Lalld_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +.macro trace_row +.if TRACE + // claim a GLOBALLY-unique row index (all WGs' claimers share occ[55]) -> no per-WG SEGCNT collision. + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + v_mov_b32 v14, 1 + global_atomic_add_u32 v15, v4, v14, s[0:1] offset:TRACE_IDX_OFF th:TH_ATOMIC_RETURN scope:SCOPE_DEV // v15=old idx, v4=addr(0), v14=data(1) + s_wait_loadcnt 0x0 + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 s52, v15 // s52 = unique row index (old value returned by the atomic) + s_cmp_ge_u32 s52, s72 // row >= MAXROWS -> skip (buffer bound) + s_cbranch_scc1 .Ltrow_skip\@ + s_lshl_b32 s53, s52, 6 // row * TRACE_ROW_BYTES(64) + s_add_u32 s60, s70, s53 + s_addc_u32 s61, s71, 0 // s[60:61] = row base VA + s_sendmsg_rtn_b64 s[58:59], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + lds_get s55, SEGCNT_OFF // this WG's per-WG super-tile counter (data field, not index) + v_mov_b32 v14, s58 // 0 tick_lo + v_mov_b32 v15, s55 // 1 segcnt (per-WG) + v_mov_b32 v16, s35 // 2 epoch + lds_get s54, NCOMP_SLOT + v_mov_b32 v17, s54 // 3 nComp (live role slot) + lds_get s54, NAFEED_SLOT + v_mov_b32 v18, s54 // 4 nAfeed + lds_get s54, NBFEED_SLOT + v_mov_b32 v19, s54 // 5 nBfeed + v_mov_b32 v20, s73 // 6 occA peak (across wait_done spins) + v_mov_b32 v21, s74 // 7 occB peak + global_load_b32 v22, v4, s[0:1] offset:CONVCNT_OFF scope:SCOPE_DEV // 8 convCount (cumulative) + lds_get s54, VRESV_OFF + v_mov_b32 v23, s54 // 9 vresv (envelope budget) + v_mov_b32 v24, s17 // 10 sti (claimed super-tile id) + lds_get s54, QUIESCE_CNT_OFF + v_mov_b32 v25, s54 // 11 quiesce (final) + v_mov_b32 v26, s59 // 12 tick_hi + v_mov_b32 v27, s69 // 13 chunkHi (context) + v_mov_b32 v28, s75 // 14 wg_id (which workgroup's economy this row belongs to) + v_mov_b32 v29, 0 // 15 reserved + s_wait_loadcnt 0x0 // convCount load drained before the row store + v_cmp_eq_u32 vcc_lo, 0, v2 // lane 0 of the claimer writes the row + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltrow_wskip\@ + global_store_b128 v4, v[14:17], s[60:61] offset:0 scope:SCOPE_DEV + global_store_b128 v4, v[18:21], s[60:61] offset:16 scope:SCOPE_DEV + global_store_b128 v4, v[22:25], s[60:61] offset:32 scope:SCOPE_DEV + global_store_b128 v4, v[26:29], s[60:61] offset:48 scope:SCOPE_DEV +.Ltrow_wskip\@: + s_mov_b32 exec_lo, s49 +.Ltrow_skip\@: +.endif +.endm + +// ============================================================================================ +// KERNEL +// ============================================================================================ + .text + .globl occ_kernel + .p2align 8 + .type occ_kernel,@function +occ_kernel: + // ---- FIX 1(b,c): derive n_kseg from KT (s8) + the compile-time NKSEG_SHIFT, then the shift/mask + // decode pair, into the reserved high SGPRs s66/s67/s68 BEFORE any clobber (SAFEPROBE below reuses + // s16 purely as scratch; the role bodies keep all transients <= s65). No v2 kernargs are read here + // anymore -- s15/s16/s17 are NOT hardware-preloaded under USER_SGPR=15 (see KERNARG CONTRACT above). ---- + s_lshr_b32 s66, s8, NKSEG_SHIFT // n_kseg = KT >> NKSEG_SHIFT (KT=s8) + s_ff1_i32_b32 s68, s66 // shift = log2(n_kseg) (bit index of the single set bit; n_kseg=1 -> 0) + s_sub_u32 s67, s66, 1 // mask = n_kseg - 1 + // ---- identity (lifted from coop prologue; v0=tid hardware-preloaded) ---- + v_lshrrev_b32 v1, 5, v0 // wid = tid >> 5 + v_and_b32 v2, 31, v0 // lane = tid & 31 + v_and_b32 v6, 15, v0 // lane & 15 (A vaddr) + v_mov_b32 v4, 0 +.if KMAJOR + global_load_b32 v3, v4, s[0:1] offset:248 scope:SCOPE_DEV // occ[62] = magic(TOTAL), host-written + s_wait_loadcnt 0x0 + v_readfirstlane_b32 s76, v3 // s76 = magic_TOTAL, persistent for every DECODE_STI (K-major) +.endif + // ---- per-lane address constants (mbgemm-identical; dyn-VGPR arming compatible) ---- + v_mul_lo_u32 v8, v6, s9 // (lane&15)*K + v_bfe_u32 v7, v0, 4, 1 + v_lshlrev_b32 v7, 3, v7 + v_add_nc_u32 v8, v8, v7 // v8 = A vaddr = (lane&15)*K + colhi*8 + v_lshlrev_b32 v9, 3, v2 // v9 = B/ds vaddr = lane*8 + v_lshlrev_b32 v10, 5, v2 // v10 = C store vaddr = lane*32 +.if SAFEPROBE + // brick-PROOF: clamp the per-lane VECTOR address regs to a loose upper bound (>= true max) so even a + // grow-corrupted vaddr cannot push a global access past the data+guard (pairs with the future ti clamp). + s_lshl_b32 s16, s9, 4 // 16*K (>= v8 max = (lane&15)*K + colhi*8 = 15*K+8) + v_min_u32 v8, s16, v8 // clamp A vaddr + v_min_u32 v9, 0x100, v9 // clamp B/ds vaddr (256 >= lane*8 max 248) + v_min_u32 v10, 0x400, v10 // clamp C vaddr (1024 >= lane*32 max 992) +.endif + + tfspan min, 8 // TFPROBE: every wave stamps occ[2] = min entry tick (wall-span start) +.if TRACE + // total-occupancy: every wave ++live at entry, atomic-max the peak concurrent resident count (occ[1]). + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lall_enter_skip + v_mov_b32 v3, 1 + global_atomic_add_u32 v5, v4, v3, s[0:1] offset:ALLLIVE_OFF th:TH_ATOMIC_RETURN scope:SCOPE_DEV + s_wait_loadcnt 0x0 + v_add_nc_u32 v5, v5, 1 + global_atomic_max_u32 v4, v5, s[0:1] offset:4 scope:SCOPE_DEV // occ[1] = peak concurrent resident waves +.Lall_enter_skip: + s_mov_b32 exec_lo, s49 +.endif + +.if DSWS2 + // ===== DSWS v2 role branch (wid uniform per wave; scalar-only -> exec stays full for every role). + // wid == 0 -> claimer (pinned super-tile broadcaster; A3) + // wid [0,NBFEED) -> B-feed (A4) + // wid [NBFEED,NBFEED+NAFEED) -> A-feed (A5) + // wid [NBFEED+NAFEED, WAVES) -> compute (A6) ===== + // A1: every role label is just a distinct s_endpgm stub (unique s50 tag keeps them at distinct addresses). + v_readfirstlane_b32 s24, v1 // wid (uniform per wave) + s_cmp_eq_u32 s24, 0 + s_cbranch_scc1 .Lclaimer +.if DSWS2_CONV && DIAG && DSWS2_TICKET_SELFTEST + // Task 4 Step 3 -- try_gate single-winner SMOKE (assemble-only stub; default off). Every non-claimer + // wave races the (dir=0) ticket ONCE and atomic-adds its win (0/1) into occ[28] (byte offset 112, + // clear of the 0/20/24/104/108 control+probe words). On GPU (Task 6, if enabled) the sum should land + // near #epochs, NOT NCOMP*#epochs -- the harness-side proof the LDS-CAS yields <=1 winner/(dir,epoch). + // v4=0 (set in prologue), v2=lane; try_gate temps v5/v6/v7 are <=v15 (pre-grow safe). wid (s24) + // survives -- try_gate touches only s62..s65 / s16. NOTE: pre-init-rendezvous placement -> a real run + // reads gate/segcnt before the claimer publishes them; fine for an assemble/smoke stub. + try_gate 0, s50 + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltg_selftest_skip + v_mov_b32 v5, s50 // win flag (0/1) for THIS wave + global_atomic_add_u32 v4, v5, s[0:1] offset:112 scope:SCOPE_DEV // occ[28] += win +.Ltg_selftest_skip: + s_mov_b32 exec_lo, s16 +.endif +.if DSWS2_CONV +.if CONV_COOLDOWN > 0 + s_mov_b32 s66, 0 // Task 4: init cooldown ctr (un-cooled at entry) +.endif + // FIRST-time entry seeds the role reg s59 then falls into the role's FULL entry + // (.Lcompute/.Lafeed/.Lbfeed -> _alloc -> _init -> _follow). It must NOT jump to + // .Ldispatch: that trampoline lands on _follow and is correct ONLY for RE-dispatch + // (a wave that already ran _alloc/_init once). First entry via _follow would skip the + // s_alloc_vgpr 32 allocator handshake, the INITFLAG==0xACED rendezvous, and the s35=0 + // epoch seed -> followers desync and the claimer hangs in .Lclaimer_wait_done (Pool-T7 brick). + s_cmp_lt_u32 s24, NBFEED + s_cbranch_scc1 .Lseed_bfeed + s_cmp_lt_u32 s24, (NBFEED+NAFEED) + s_cbranch_scc1 .Lseed_afeed + s_mov_b32 s59, NCOMP_SLOT + s_branch .Lcompute +.Lseed_afeed: + s_mov_b32 s59, NAFEED_SLOT + s_branch .Lafeed +.Lseed_bfeed: + s_mov_b32 s59, NBFEED_SLOT + s_branch .Lbfeed +.else + s_cmp_lt_u32 s24, NBFEED + s_cbranch_scc1 .Lbfeed + s_cmp_lt_u32 s24, (NBFEED+NAFEED) + s_cbranch_scc1 .Lafeed + s_branch .Lcompute +.endif + +// ============================================================================================ +// A3 -- .Lclaimer : pinned wid-0. Owns the super-tile claim+broadcast, the SEGCNT clock, the +// barrier-free LDS init, the completion live++/live-- (harness occ[0]==0 gate), AND -- being a +// B-feed-class wave -- stages B for the current super-tile each iteration (A4 body via BSTAGE). +// +// CLAIM-COUNTER & completion occ-buffer offsets (occ base = s0:s1; host zero-inits the whole buffer): +// occ[0] (offset 0) = live counter (claimer +1 at entry, -1 at terminal; harness polls ==0) +// occ[20] (offset 20) = GLOBAL super-tile claim counter (mirrors coop's tile-claim at offset:20) +// (offsets 4/8/12/16 stay reserved for the coop-style maxlive/timers/total bookkeeping; unused here.) +// ============================================================================================ +.Lclaimer: +.if DYNVGPR +.Lclaimer_alloc: + s_alloc_vgpr 32 // commit lean (dyn WG-allocator consistency); SCC-retry guard + s_cbranch_scc0 .Lclaimer_alloc +.endif + // live++ : lane0 occ[0] += 1 + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lclaimer_live + v_mov_b32 v3, 1 + global_atomic_add_u32 v4, v3, s[0:1] scope:SCOPE_DEV +.Lclaimer_live: + s_mov_b32 exec_lo, s16 + // --- barrier-free LDS-control init: zero all control words; INITFLAG = 0xACED LAST --- + lds_put STI_OFF, 0 + lds_put EPOCH_OFF, 0 + lds_put ROWBLK_NEXT_OFF, 0 + lds_put ROWBLK_DONE_OFF, 0 + lds_put BFRAG_NEXT_OFF, 0 + lds_put AROW_NEXT_OFF, 0 + lds_put BFRAG_DONE_OFF, 0 + lds_put AROW_DONE_OFF, 0 + lds_put NCOMP_SLOT, NCOMP + lds_put NAFEED_SLOT, NAFEED + lds_put NBFEED_SLOT, NBFEED + lds_put GATE_OFF, 0 + lds_put (GATE_OFF+4), 0 + lds_put (GATE_OFF+8), 0 + lds_put (GATE_OFF+12), 0 +.if DSWS2_ENVELOPE + lds_put VRESV_OFF, (WAVES*VLEAN) // rolling: everyone lean; counter books peaks +.else + lds_put VRESV_OFF, (NCOMP*NFV + (NAFEED+NBFEED)*VLEAN) +.endif + lds_put SEGCNT_OFF, 0 +.if DSWS2_CONV + // Phase-B: seed BOTH epoch-parity role-mix snapshots with the launch mix, zero the quiesce counter. + // Gated so DSWS2_CONV=0 emits ZERO new bytes -> byte-identical to the Phase-A green bin. +.if DSWS2_GQUIESCE + gq_reset // global QUIESCE = 0 (device-scoped; committed before INITFLAG) +.else + lds_put QUIESCE_CNT_OFF, 0 +.endif + lds_put (SNAP_BASE + 0), NCOMP // parity-0 snapshot = launch mix + lds_put (SNAP_BASE + 4), NAFEED + lds_put (SNAP_BASE + 8), NBFEED + lds_put (SNAP_BASE + 12), NCOMP // parity-1 = launch mix too (init) + lds_put (SNAP_BASE + 16), NAFEED + lds_put (SNAP_BASE + 20), NBFEED + lds_put OCCA_PUB_OFF, 2 // SENSOR FIX: neutral ring seed (not CTRL_HIGH) until 1st publish + lds_put OCCB_PUB_OFF, 2 +.endif + lds_put INITFLAG_OFF, 0xACED // LAST: publishes "LDS ready" to all follower waves + // FIX 1(e): load this dispatch's chunk terminal bound from occ[24] (host writes occW[6] per chunk; + // FIX 1j on the host side). All lanes read the same address -> no exec masking needed, just a + // plain broadcast load; stable for the WHOLE chunk, so load it ONCE here, not per-claim. + global_load_b32 v6, v4, s[0:1] offset:24 scope:SCOPE_DEV + s_wait_loadcnt 0x0 + v_readfirstlane_b32 s69, v6 // s69 = chunkHi (this dispatch's terminal sti bound) + s_mov_b32 s35, 0 // claimer local epoch +.if TRACE + global_load_b64 v[6:7], v4, s[0:1] offset:TRACE_PTR_OFF scope:SCOPE_DEV // trace buffer VA + global_load_b32 v8, v4, s[0:1] offset:TRACE_CAP_OFF scope:SCOPE_DEV // MAXROWS + s_wait_loadcnt 0x0 + v_readfirstlane_b32 s70, v6 // s[70:71] = trace VA (persistent, claimer-only) + v_readfirstlane_b32 s71, v7 + v_readfirstlane_b32 s72, v8 // s72 = MAXROWS + // wg_id = claim-order dispenser (TGID_X isn't enabled in s15 -> was all-0). One atomic per WG at entry. + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + v_mov_b32 v7, 1 + global_atomic_add_u32 v6, v4, v7, s[0:1] offset:TRACE_WGID_OFF th:TH_ATOMIC_RETURN scope:SCOPE_DEV + s_wait_loadcnt 0x0 + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 s75, v6 // s75 = this WG's claim-order id (persistent, claimer-only) +.endif +.Lclaim_loop: + // claim next sti: lane0 global_atomic_add occ[20] += 1, return old + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lclaim_grabbed + v_mov_b32 v3, 1 + global_atomic_add_u32 v5, v4, v3, s[0:1] offset:20 th:TH_ATOMIC_RETURN scope:SCOPE_DEV + s_wait_loadcnt 0x0 +.Lclaim_grabbed: + s_mov_b32 exec_lo, s16 + v_readfirstlane_b32 s17, v5 // sti + s_cmp_ge_u32 s17, s69 // sti >= chunkHi (occ[24]) -> terminal + s_cbranch_scc1 .Lclaimer_terminal + DECODE_STI // -> s19=mblk s30=tcol s31=ksi + // reset per-super-tile claim/completion counters BEFORE the epoch bump (followers see them reset) + lds_put ROWBLK_NEXT_OFF, 0 + lds_put ROWBLK_DONE_OFF, 0 + lds_put BFRAG_NEXT_OFF, 0 + lds_put AROW_NEXT_OFF, 0 + lds_put BFRAG_DONE_OFF, 0 + lds_put AROW_DONE_OFF, 0 + lds_put STI_OFF, s17 // publish STI FIRST... + lds_get s44, SEGCNT_OFF // bump SEGCNT (controller clock; +1/super-tile) + s_add_u32 s44, s44, 1 + lds_put SEGCNT_OFF, s44 + lds_get s44, EPOCH_OFF // ...then bump EPOCH LAST + s_add_u32 s44, s44, 1 + s_mov_b32 s35, s44 +.if DSWS2_CONV + // ---- Step 4 (SPEC 3.4 decision 1): snapshot the LIVE role mix into the NEXT epoch's parity half of + // SNAP_BASE, and reset QUIESCE_CNT, BEFORE the epoch bump below (which is published LAST). Followers + // and the claimer's own wait-done for THIS super-tile then size their quiesce sentinels from + // parity(newEpoch) -- so the quiesce counter is the snapshot handshake (Step 3 reads it). ---- + s_and_b32 s46, s44, 1 // parity(newEpoch) + s_mul_i32 s46, s46, 12 // parity*12 (3-word half) -- matches SNAP_BASE comment + s_add_u32 s46, s46, SNAP_BASE // s46 = byte offset of parity half + lds_get s47, NCOMP_SLOT + lds_put_r s46, s47 // snap.nC = live NCOMP_SLOT + s_add_u32 s46, s46, 4 + lds_get s47, NAFEED_SLOT + lds_put_r s46, s47 // snap.nA = live NAFEED_SLOT + s_add_u32 s46, s46, 4 + lds_get s47, NBFEED_SLOT + lds_put_r s46, s47 // snap.nB = live NBFEED_SLOT +.if DSWS2_GQUIESCE + gq_reset // global QUIESCE = 0 (device-scoped; committed before EPOCH publish) +.else + lds_put QUIESCE_CNT_OFF, 0 // reset the per-super-tile bail counter +.endif +.endif + lds_put EPOCH_OFF, s44 +.if DSWS2_CONV + s_mov_b32 s73, 0 // reset this super-tile's ring occ_A peak (SENSOR FIX + TRACE) + s_mov_b32 s74, 0 // reset ring occ_B peak (updated across wait_done spins) +.endif + BSTAGE // claimer helps stage B for this super-tile (s30,s31) + // A7 advance gate: free resident A/B only when ALL G rowblks are computed+flushed +.Lclaimer_wait_done: + s_sleep SLEEPN +.if DSWS2_CONV + occ_sample s62, s63 // sample ring mid-drain; keep the per-super-tile PEAK (SENSOR FIX + TRACE) + s_max_u32 s73, s73, s62 + s_max_u32 s74, s74, s63 +.endif +.if DSWS2_CONV +.if DIAG + // Phase-B DIAG probe (Task 3): wid 0 samples the LIVE ring occupancy (compute is mid-drain here) + // and publishes the last-sampled occ_A/occ_B so a GPU run can confirm the sensor OSCILLATES + // rather than reading a stuck 0. Written every wait-done spin -> a live poller observes it vary. + // Spare word-indexed slots occ[26]/occ[27] -> byte offsets 104/108 (well clear of the + // byte-indexed control words occ[0]/occ[20]/occ[24]). READ-ONLY sensing, NO actuation. + occ_sample s62, s63 + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Locc_diag_skip + v_mov_b32 v14, s62 // occ_A (v14 <= v15: pre-grow safe) + v_mov_b32 v15, s63 // occ_B + global_store_b32 v4, v14, s[0:1] offset:104 scope:SCOPE_DEV // occ[26] = last occ_A + global_store_b32 v4, v15, s[0:1] offset:108 scope:SCOPE_DEV // occ[27] = last occ_B +.Locc_diag_skip: + s_mov_b32 exec_lo, s16 +.endif +.endif + lds_get s44, ROWBLK_DONE_OFF // (a) all G rowblks computed + flushed + s_cmp_lt_u32 s44, G + s_cbranch_scc1 .Lclaimer_wait_done + lds_get s44, BFRAG_DONE_OFF // all B frags stored + s_cmp_lt_u32 s44, FN + s_cbranch_scc1 .Lclaimer_wait_done + lds_get s44, AROW_DONE_OFF // all A rowblks staged + s_cmp_lt_u32 s44, G + s_cbranch_scc1 .Lclaimer_wait_done + // (b) QUIESCE the CLAIM counters before reset: each role wave must have executed its terminal + // over-claim (fetch_add returns >=threshold, then bails) BEFORE we reset, else a descheduled + // straggler's next fetch_add returns 0 and claims index 0 of the NEXT super-tile against stale + // decode/resident state (round-table finding #1). Sentinels = threshold + #role-waves (each does + // exactly one terminal bail). NOTE: compile-time NCOMP/NAFEED/NBFEED is correct for STATIC roles; + // Phase-B conversion must switch these to live role counts / epoch-snapshot drained counters. +.if DSWS2_CONV + // ---- Step 3 (SPEC 3.4 decision 1): size the three claim-counter sentinels from THIS epoch's parity + // snapshot (live role mix written at broadcast, Step 4) instead of compile-time NCOMP/NAFEED/NBFEED, + // so conversions re-tune the quiesce thresholds. A SEPARATE QUIESCE_CNT >= WAVES-1 cross-check is the + // DIAG safety net; under DIAG a mismatch between the two is flagged to occ[29] (byte 116). ---- + lds_get s45, EPOCH_OFF + s_and_b32 s45, s45, 1 // parity(thisEpoch) + s_mul_i32 s45, s45, 12 // parity*12 (3-word half) + s_add_u32 s45, s45, SNAP_BASE + lds_get_r s46, s45 // snap.nC + s_add_u32 s45, s45, 4 + lds_get_r s47, s45 // snap.nA + s_add_u32 s45, s45, 4 + lds_get_r s48, s45 // snap.nB + s_add_u32 s46, s46, G // thr_rowblk = G + snap.nC + s_add_u32 s47, s47, G // thr_arow = G + snap.nA + s_add_u32 s48, s48, FN // thr_bfrag = FN + snap.nB + // sentinels_done (s50) = (ROWBLK_NEXT>=thr_rowblk) & (BFRAG_NEXT>=thr_bfrag) & (AROW_NEXT>=thr_arow) + s_mov_b32 s50, 1 + lds_get s44, ROWBLK_NEXT_OFF + s_cmp_lt_u32 s44, s46 + s_cbranch_scc0 .Lqc_rb_ok + s_mov_b32 s50, 0 +.Lqc_rb_ok: + lds_get s44, BFRAG_NEXT_OFF + s_cmp_lt_u32 s44, s48 + s_cbranch_scc0 .Lqc_bf_ok + s_mov_b32 s50, 0 +.Lqc_bf_ok: + lds_get s44, AROW_NEXT_OFF + s_cmp_lt_u32 s44, s47 + s_cbranch_scc0 .Lqc_ar_ok + s_mov_b32 s50, 0 +.Lqc_ar_ok: + // quiesce_done (s51) = QUIESCE_CNT >= WAVES-1 (each of the WAVES-1 non-claimer waves bumped once) + s_mov_b32 s51, 1 +.if DSWS2_GQUIESCE + gq_read s44 // device-scoped global read (observes committed follower bumps) +.else + lds_get s44, QUIESCE_CNT_OFF +.endif + s_cmp_lt_u32 s44, (WAVES-1) + s_cbranch_scc0 .Lqc_q_ok + s_mov_b32 s51, 0 +.Lqc_q_ok: +.if DSWS2_CONV + // SENSOR FIX: publish this super-tile's mid-drain ring PEAK so the followers' NEXT-epoch conversion + // decisions read a true demand signal instead of sampling occ_X~0 at their own post-drain quiesce. + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lpub_skip + v_mov_b32 v14, OCCA_PUB_OFF + v_mov_b32 v15, s73 + ds_store_b32 v14, v15 + v_mov_b32 v14, OCCB_PUB_OFF + v_mov_b32 v15, s74 + ds_store_b32 v14, v15 + // (no s_wait_dscnt: followers read this a full super-tile later; async LDS store drains long before) +.Lpub_skip: + s_mov_b32 exec_lo, s49 +.endif + trace_row // TRACE: append this super-tile's row (role mix, ring peak, conv, vresv) +.if DIAG + // DIAG cross-check + Pool-T7 chunk-2 WEDGE FRAME. occ[29]=s50^s51 mismatch. Plus snapshot EVERY advance-gate + // counter to host-streamed occ slots so a hung dispatch reads out exactly which sentinel is unmet: + // stream cons=ROWBLK_DONE tiles=AROW_DONE compPh=BFRAG_DONE (want G/G/FN = all work flushed) + // comp:dsB=QUIESCE_CNT (want WAVES-1 -> short => a follower never bailed; see roles[C/A/B] epochs) + // feed:tr=ROWBLK_NEXT pub=AROW_NEXT rawTi=BFRAG_NEXT (the s50 over-claim sentinels). + // Read counters at full exec, THEN publish under lane-0 mask. Scratch s53-s56/s60-s62 dead here + // (occ_sample's s60-s63 were consumed at the wait_done top). DIAG-only -> CONV=0 byte-identical. + lds_get s53, ROWBLK_DONE_OFF + lds_get s54, AROW_DONE_OFF + lds_get s55, BFRAG_DONE_OFF + lds_get s56, QUIESCE_CNT_OFF + lds_get s60, ROWBLK_NEXT_OFF + lds_get s61, AROW_NEXT_OFF + lds_get s62, BFRAG_NEXT_OFF + s_xor_b32 s52, s50, s51 + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lqc_diag_skip + v_mov_b32 v14, s52 + global_store_b32 v4, v14, s[0:1] offset:116 scope:SCOPE_DEV // occ[29] = quiesce/snapshot mismatch + v_mov_b32 v14, s53 + global_store_b32 v4, v14, s[0:1] offset:40 scope:SCOPE_DEV // occ[10] cons = ROWBLK_DONE (want G) + v_mov_b32 v14, s54 + global_store_b32 v4, v14, s[0:1] offset:44 scope:SCOPE_DEV // occ[11] tiles = AROW_DONE (want G) + v_mov_b32 v14, s55 + global_store_b32 v4, v14, s[0:1] offset:28 scope:SCOPE_DEV // occ[7] compPh = BFRAG_DONE (want FN) + v_mov_b32 v14, s56 + global_store_b32 v4, v14, s[0:1] offset:60 scope:SCOPE_DEV // occ[15] comp:dsB = QUIESCE (want WAVES-1) + v_mov_b32 v14, s60 + global_store_b32 v4, v14, s[0:1] offset:76 scope:SCOPE_DEV // occ[19] feed:tr = ROWBLK_NEXT + v_mov_b32 v14, s61 + global_store_b32 v4, v14, s[0:1] offset:84 scope:SCOPE_DEV // occ[21] pub = AROW_NEXT + v_mov_b32 v14, s62 + global_store_b32 v4, v14, s[0:1] offset:88 scope:SCOPE_DEV // occ[22] rawTi = BFRAG_NEXT +.if DSWS2_ENVELOPE + // Envelope telemetry (2026-07-02 spec 4.5): instantaneous vgpr_reserved at the wedge moment + + // PEAK_CONC config echo. If a dispatch wedges with occ[8]==BUDGET, it is permit-starvation (the + // envelope is saturated); anything below BUDGET points the wedge elsewhere. DIAG+ENVELOPE gated. + lds_get s52, VRESV_OFF + v_mov_b32 v14, s52 + global_store_b32 v4, v14, s[0:1] offset:32 scope:SCOPE_DEV // occ[8] vgpr_reserved (want < BUDGET) + v_mov_b32 v14, PEAK_CONC + global_store_b32 v4, v14, s[0:1] offset:56 scope:SCOPE_DEV // occ[14] PEAK_CONC echo (config readback) +.endif +.Lqc_diag_skip: + s_mov_b32 exec_lo, s16 +.endif + // advance only when BOTH the snapshot sentinels AND the QUIESCE cross-check agree they are done + s_and_b32 s50, s50, s51 + s_cmp_eq_u32 s50, 0 + s_cbranch_scc1 .Lclaimer_wait_done +.else + lds_get s44, ROWBLK_NEXT_OFF // G claims + NCOMP terminal bails + s_cmp_lt_u32 s44, (G + NCOMP) + s_cbranch_scc1 .Lclaimer_wait_done + lds_get s44, BFRAG_NEXT_OFF // FN claims + NBFEED terminal bails + s_cmp_lt_u32 s44, (FN + NBFEED) + s_cbranch_scc1 .Lclaimer_wait_done + lds_get s44, AROW_NEXT_OFF // G claims + NAFEED terminal bails + s_cmp_lt_u32 s44, (G + NAFEED) + s_cbranch_scc1 .Lclaimer_wait_done +.endif + s_branch .Lclaim_loop +.Lclaimer_terminal: + lds_put STI_OFF, 0xFFFFFFFF // FIX 1(e): publish SENTINEL (not the raw over-claimed sti)... + lds_get s44, EPOCH_OFF // ...bump epoch so followers wake + retire (A7 terminal) + s_add_u32 s44, s44, 1 + lds_put EPOCH_OFF, s44 + // live-- : lane0 occ[0] -= 1 (harness completion gate fires) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lclaimer_dead + v_mov_b32 v3, -1 + global_atomic_add_u32 v4, v3, s[0:1] scope:SCOPE_DEV +.Lclaimer_dead: + s_mov_b32 exec_lo, s16 + tfspan max, 12 // TFPROBE: claimer stamps occ[3] = max exit tick (wall-span end) + alllive_dec + s_endpgm + +// ============================================================================================ +// A4 -- .Lbfeed : B-feed wave. Follows EPOCH/STI, decodes (tcol,ksi), stages its claimed B frags. +// ============================================================================================ +.Lbfeed: +.if DYNVGPR +.Lbfeed_alloc: + s_alloc_vgpr 32 + s_cbranch_scc0 .Lbfeed_alloc +.endif +.Lbfeed_init: + s_sleep 1 + lds_get s44, INITFLAG_OFF + s_cmp_eq_u32 s44, 0xACED + s_cbranch_scc0 .Lbfeed_init // wait for the claimer's barrier-free LDS init + s_mov_b32 s35, 0 // local epoch +.Lbfeed_follow: + s_sleep SLEEPN + lds_get s44, EPOCH_OFF + s_cmp_eq_u32 s44, s35 + s_cbranch_scc1 .Lbfeed_follow // wait next super-tile (epoch change) + s_mov_b32 s35, s44 + lds_get s17, STI_OFF + s_cmp_eq_u32 s17, 0xFFFFFFFF // FIX 1(f): sentinel (A7) -> retire (was: STI>=TOTAL_super) + s_cbranch_scc1 .Lretire + DECODE_STI // s30=tcol s31=ksi (mblk unused) + BSTAGE +.if DSWS2_CONV + // ==== Phase-B decision (Step 1) + bail-time commit (Step 2), B-feed wave ==== + // Lean-32 -> PRE-GROW window (the feed->compute GROW closes it inside conv_apply, post all <=v15 + // reads). If the B-ring is OVER-SERVED (occ_B>CTRL_HIGH_B) grow one B-feed->compute (dir 3): + // reserve delta +(NFV-VLEAN) (may abort over BUDGET -> stay B-feed). +.if CONV_COOLDOWN > 0 + s_cmp_eq_u32 s66, 0 + s_cbranch_scc0 .Lbfeed_cooldn // Task 4: s66!=0 -> in cooldown, skip decision +.endif +.if DSWS2_FORCE +.if DSWS2_FORCE_DIR == 3 + // Task 5: deterministic forced conversion (dir 3: B-feed -> compute), watermark bypassed. + // Convert iff wid(s24)==DSWS2_FORCE_WID AND current epoch(s35)==DSWS2_FORCE_EPOCH. + s_mov_b32 s58, 0 + s_cmp_eq_u32 s24, DSWS2_FORCE_WID + s_cbranch_scc0 .Lbfeed_cooldn + s_cmp_eq_u32 s35, DSWS2_FORCE_EPOCH + s_cbranch_scc0 .Lbfeed_cooldn + s_mov_b32 s57, 3 // dir 3: B-feed -> compute (forced) + try_gate 3, s58 + conv_apply NBFEED_SLOT, NCOMP_SLOT, +(NFV-VLEAN), NFV +.endif +.else + s_mov_b32 s58, 0 + lds_get s55, OCCA_PUB_OFF // SENSOR FIX: read claimer-published mid-drain peak (not post-drain ~0) + lds_get s56, OCCB_PUB_OFF + s_cmp_gt_u32 s56, CTRL_HIGH_B + s_cbranch_scc0 .Lbfeed_cooldn + s_mov_b32 s57, 3 // dir 3: B-feed -> compute + try_gate 3, s58 + conv_apply NBFEED_SLOT, NCOMP_SLOT, +(NFV-VLEAN), NFV +.endif +.Lbfeed_cooldn: +.if CONV_COOLDOWN > 0 + s_cmp_eq_u32 s66, 0 // Task 4: decrement once per epoch (saturating) + s_cbranch_scc1 .Lbfeed_quiesce + s_sub_i32 s66, s66, 1 +.endif +.Lbfeed_quiesce: +.if DSWS2_GQUIESCE + gq_bump // device-scoped global bump (visible to the claimer poll) +.else + lds_fetch_add s61, QUIESCE_CNT_OFF, 1 // commit-before-bump ordering (SPEC 3.4 decision 2) +.endif + epoch_mark 144 // roles[2] = B-feed last epoch bumped (wedge localize) + bail_mark // per-wave localization: occ[BAIL_BASE + wid*4] = this wave's epoch +.endif +.if DSWS2_CONV + s_branch .Ldispatch +.else + s_branch .Lbfeed_follow +.endif + +// ============================================================================================ +// A5 -- .Lafeed : A-feed wave. Follows EPOCH/STI, decodes (mblk,ksi), stages its claimed A rowblks. +// ============================================================================================ +.Lafeed: +.if DYNVGPR +.Lafeed_alloc: + s_alloc_vgpr 32 + s_cbranch_scc0 .Lafeed_alloc +.endif +.Lafeed_init: + s_sleep 1 + lds_get s44, INITFLAG_OFF + s_cmp_eq_u32 s44, 0xACED + s_cbranch_scc0 .Lafeed_init + s_mov_b32 s35, 0 +.Lafeed_follow: + s_sleep SLEEPN + lds_get s44, EPOCH_OFF + s_cmp_eq_u32 s44, s35 + s_cbranch_scc1 .Lafeed_follow + s_mov_b32 s35, s44 + lds_get s17, STI_OFF + s_cmp_eq_u32 s17, 0xFFFFFFFF // FIX 1(f): sentinel (A7) -> retire (was: STI>=TOTAL_super) + s_cbranch_scc1 .Lretire + DECODE_STI // s19=mblk s31=ksi (tcol unused) + ASTAGE +.if DSWS2_CONV + // ==== Phase-B decision (Step 1) + bail-time commit (Step 2), A-feed wave ==== + // Wave is lean-32 here (feeds never grow), so this is a PRE-GROW window; the conversion GROW + // (feed->compute, s_alloc_vgpr NFV) is the ONLY grow and closes the window inside conv_apply, after + // all <=v15 LDS/atomic reads. If the A-ring is OVER-SERVED (occ_A>CTRL_HIGH_A) grow one A-feed-> + // compute (dir 2): reserve delta +(NFV-VLEAN) (may abort over BUDGET -> stay A-feed). +.if CONV_COOLDOWN > 0 + s_cmp_eq_u32 s66, 0 + s_cbranch_scc0 .Lafeed_cooldn // Task 4: s66!=0 -> in cooldown, skip decision +.endif +.if DSWS2_FORCE +.if DSWS2_FORCE_DIR == 2 + // Task 5: deterministic forced conversion (dir 2: A-feed -> compute), watermark bypassed. + // Convert iff wid(s24)==DSWS2_FORCE_WID AND current epoch(s35)==DSWS2_FORCE_EPOCH. + s_mov_b32 s58, 0 + s_cmp_eq_u32 s24, DSWS2_FORCE_WID + s_cbranch_scc0 .Lafeed_cooldn + s_cmp_eq_u32 s35, DSWS2_FORCE_EPOCH + s_cbranch_scc0 .Lafeed_cooldn + s_mov_b32 s57, 2 // dir 2: A-feed -> compute (forced) + try_gate 2, s58 + conv_apply NAFEED_SLOT, NCOMP_SLOT, +(NFV-VLEAN), NFV +.endif +.else + s_mov_b32 s58, 0 + lds_get s55, OCCA_PUB_OFF // SENSOR FIX: read claimer-published mid-drain peak (not post-drain ~0) + lds_get s56, OCCB_PUB_OFF + s_cmp_gt_u32 s55, CTRL_HIGH_A + s_cbranch_scc0 .Lafeed_cooldn + s_mov_b32 s57, 2 // dir 2: A-feed -> compute + try_gate 2, s58 + conv_apply NAFEED_SLOT, NCOMP_SLOT, +(NFV-VLEAN), NFV +.endif +.Lafeed_cooldn: +.if CONV_COOLDOWN > 0 + s_cmp_eq_u32 s66, 0 // Task 4: decrement once per epoch (saturating) + s_cbranch_scc1 .Lafeed_quiesce + s_sub_i32 s66, s66, 1 +.endif +.Lafeed_quiesce: +.if DSWS2_GQUIESCE + gq_bump // device-scoped global bump (visible to the claimer poll) +.else + lds_fetch_add s61, QUIESCE_CNT_OFF, 1 // commit-before-bump ordering (SPEC 3.4 decision 2) +.endif + epoch_mark 140 // roles[1] = A-feed last epoch bumped (wedge localize) + bail_mark // per-wave localization: occ[BAIL_BASE + wid*4] = this wave's epoch +.endif +.if DSWS2_CONV + s_branch .Ldispatch +.else + s_branch .Lafeed_follow +.endif + +// ============================================================================================ +// A6 -- .Lcompute : compute wave. Follows EPOCH/STI, decodes (mblk,tcol,ksi), waits resident A/B +// fully staged (DONE counters), then claims rowblks, runs WMMA over the SEGK segment, and flushes +// fp32 partials into C via global_atomic_add_f32 (split-K segments accumulate into the same C cell). +// ============================================================================================ +.Lcompute: +.if DYNVGPR +.Lcompute_alloc: + s_alloc_vgpr 32 // lean baseline; grow per rowblk + s_cbranch_scc0 .Lcompute_alloc +.endif +.Lcompute_init: + s_sleep 1 + lds_get s44, INITFLAG_OFF + s_cmp_eq_u32 s44, 0xACED + s_cbranch_scc0 .Lcompute_init + s_mov_b32 s35, 0 + phase_reset // PHASEPROBE: seed this compute wave's phase clock +.Lcompute_follow: + s_sleep SLEEPN + lds_get s44, EPOCH_OFF + s_cmp_eq_u32 s44, s35 + s_cbranch_scc1 .Lcompute_follow + s_mov_b32 s35, s44 + phase_stamp s78 // close FOLLOW_WAIT (spun waiting for next super-tile) + lds_get s17, STI_OFF +.if PHASEPROBE + s_cmp_eq_u32 s17, 0xFFFFFFFF // sentinel -> flush phase accumulators, then retire + s_cbranch_scc0 .Lcompute_go + phase_flush + s_branch .Lretire +.Lcompute_go: +.else + s_cmp_eq_u32 s17, 0xFFFFFFFF // FIX 1(f): sentinel (A7) -> retire (was: STI>=TOTAL_super) + s_cbranch_scc1 .Lretire +.endif + DECODE_STI // s19=mblk s30=tcol s31=ksi + // wait until resident A AND B fully STAGED (B: FN frags stored, A: G rowblks stored) +.Lcompute_staged: + s_sleep SLEEPN + lds_get s44, BFRAG_DONE_OFF + s_cmp_lt_u32 s44, FN + s_cbranch_scc1 .Lcompute_staged + lds_get s44, AROW_DONE_OFF + s_cmp_lt_u32 s44, G + s_cbranch_scc1 .Lcompute_staged + phase_stamp s79 // close STAGE_WAIT (spun waiting for A/B feeds) + // C tile-term: ti = mblk*NTL + tcol ; ti*(G*FM*FN*1024) (ksi-INDEPENDENT -> split-K accumulates) + s_mul_i32 s38, s19, s13 + s_add_u32 s38, s38, s30 + s_mul_i32 s38, s38, (G*FM*FN*1024) +.Lcompute_claim: + lds_fetch_add s33, ROWBLK_NEXT_OFF, 1 // claim rowblk r in [0,G) + s_cmp_ge_u32 s33, G + s_cbranch_scc1 .Lcompute_drained +.if DYNVGPR + s_wait_loadcnt 0x0 + s_wait_storecnt 0x0 +.if DSWS2_ENVELOPE + // Sum-envelope reserve (2026-07-02 spec 4.1): book +Δ against vgpr_reserved before growing so at most + // PEAK_CONC waves hold peak at once. Over-budget -> reserve_try already undid its add -> back off AT + // LEAN and retry (wave is still 32 here; every temp <=v15, pre-grow OOR-safe). s54=won, s62/s63 scratch + // (all dead in the claim loop -- conv only runs at .Lcompute_drained). +.Lcompute_reserve: + reserve_try +(NFV-VLEAN), s54 + s_cmp_eq_u32 s54, 0 + s_cbranch_scc0 .Lcompute_grow // won (s54!=0 -> SCC==0) -> grow + s_sleep SLEEPN + s_branch .Lcompute_reserve +.endif +.Lcompute_grow: + s_alloc_vgpr NFV // grow (SCC-retry guarded, brick-class rule) + s_cbranch_scc0 .Lcompute_grow +.if TRACE + // B-probe: this wave just went fat (NFV VGPR). ++live, track the peak concurrent fat count. + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lfatg_skip + v_mov_b32 v3, 1 + global_atomic_add_u32 v5, v4, v3, s[0:1] offset:FATLIVE_OFF th:TH_ATOMIC_RETURN scope:SCOPE_DEV // v5 = old live + s_wait_loadcnt 0x0 + v_add_nc_u32 v5, v5, 1 // new live count + global_atomic_max_u32 v4, v5, s[0:1] offset:FATMAX_OFF scope:SCOPE_DEV // peak = max(peak, new) +.Lfatg_skip: + s_mov_b32 exec_lo, s49 +.endif +.endif + phase_stamp s80 // close GROW (rowblk claim + s_alloc_vgpr 32->112) + // zero FM*FN fp32 accumulators + .set idx, 0 + .rept FM*FN + v_mov_b32 v[ACC+idx*8+0], 0 + v_mov_b32 v[ACC+idx*8+1], 0 + v_mov_b32 v[ACC+idx*8+2], 0 + v_mov_b32 v[ACC+idx*8+3], 0 + v_mov_b32 v[ACC+idx*8+4], 0 + v_mov_b32 v[ACC+idx*8+5], 0 + v_mov_b32 v[ACC+idx*8+6], 0 + v_mov_b32 v[ACC+idx*8+7], 0 + .set idx, idx+1 + .endr + // resident operand bases (lane*8 + region [+ r*FM*256 for A]) + v_add_nc_u32 v12, v9, BRES_OFF + s_mul_i32 s37, s33, (FM*256) + v_add_nc_u32 v13, v9, ARES_OFF + v_add_nc_u32 v13, v13, s37 + // WMMA over the SEGK segment (KSEG_STEPS k-steps); read resident B(ks) + A(ks,r) from LDS + .set ks, 0 + .rept KSEG_STEPS + .set ni, 0 + .rept FN + ds_load_b64 v[FB+ni*2:FB+ni*2+1], v12 offset:((ks*FN+ni)*256) + .set ni, ni+1 + .endr + .set mi, 0 + .rept FM + ds_load_b64 v[FA+mi*2:FA+mi*2+1], v13 offset:((ks*G*FM+mi)*256) + .set mi, mi+1 + .endr + s_wait_dscnt 0x0 + .set mi, 0 + .rept FM + .set ni, 0 + .rept FN + v_wmma_f32_16x16x16_fp8_fp8 v[ACC+(mi*FN+ni)*8:ACC+(mi*FN+ni)*8+7], v[FA+mi*2:FA+mi*2+1], v[FB+ni*2:FB+ni*2+1], v[ACC+(mi*FN+ni)*8:ACC+(mi*FN+ni)*8+7] + .set ni, ni+1 + .endr + .set mi, mi+1 + .endr + .set ks, ks+1 + .endr + phase_stamp s81 // close WMMA (LDS frag loads + v_wmma over the segment) + // flush: C base = C + ti-term + r*(FM*FN*1024) ; per (frag,elem) atomic-add one fp32 (vaddr v10=lane*32) + s_mul_i32 s39, s33, (FM*FN*1024) + s_add_u32 s39, s38, s39 + s_add_u32 s28, s6, s39 + s_addc_u32 s29, s7, 0 +.if NOCFLUSH == 0 + .set frag, 0 + .rept FM*FN + .set e, 0 + .rept 8 + .if CSTORE + global_store_b32 v10, v[ACC+frag*8+e], s[28:29] offset:(frag*1024 + e*4) scope:SCOPE_DEV // probe: non-atomic, same count + .else + global_atomic_add_f32 v10, v[ACC+frag*8+e], s[28:29] offset:(frag*1024 + e*4) scope:SCOPE_DEV + .endif + .set e, e+1 + .endr + .set frag, frag+1 + .endr +.endif + s_wait_storecnt 0x0 // atomic-adds READ ACC -> must drain before shrink frees ACC + phase_stamp s82 // close FLUSH (global_atomic_add_f32 C reduction + drain) +.if DYNVGPR +.Lcompute_shrink: + s_alloc_vgpr 32 // shrink (SCC-retry guarded) + s_cbranch_scc0 .Lcompute_shrink +.if TRACE + v_cmp_eq_u32 vcc_lo, 0, v2 // B-probe: this wave went lean again -> --live + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lfats_skip + v_mov_b32 v3, -1 + global_atomic_add_u32 v4, v3, s[0:1] offset:FATLIVE_OFF scope:SCOPE_DEV +.Lfats_skip: + s_mov_b32 exec_lo, s49 +.endif +.if DSWS2_ENVELOPE + lds_fetch_add s54, VRESV_OFF, -(NFV-VLEAN) // envelope release −Δ (shrink committed; wave lean, v<=v15) +.endif +.endif + phase_stamp s83 // close SHRINK (s_alloc_vgpr 112->32) + lds_inc ROWBLK_DONE_OFF // rowblk r computed + flushed (frees the A7 advance gate) + s_branch .Lcompute_claim +.Lcompute_drained: +.if DSWS2_CONV + // ==== Phase-B role-boundary decision (Step 1) + bail-time commit (Step 2), compute wave ==== + // Wave is lean-32 here (the last claimed rowblk shrank to 32; a zero-claim drain never grew), so the + // whole decision+commit runs in the PRE-GROW window -- every temp <=v15 / scalar <=s65 (see conv_apply + // OOR note). If compute is STARVED for A (occ_AA-feed (dir 0); + // else if starved for B, compute->B-feed (dir 1). Both are SHRINKs: reserve delta -(NFV-VLEAN), + // s_alloc_vgpr 32 (already lean -> no-op). Persistent: s57=dir, s58=s_win, s59=new-role (all outside + // s60..s65 so occ_sample/try_gate/reserve_try/conv_* cannot clobber them while live). +.if CONV_COOLDOWN > 0 + s_cmp_eq_u32 s66, 0 + s_cbranch_scc0 .Lcmp_cooldn // Task 4: s66!=0 -> in cooldown, skip decision +.endif +.if DSWS2_FORCE +.if DSWS2_FORCE_DIR == 0 || DSWS2_FORCE_DIR == 1 + // Task 5: deterministic forced conversion (compute is the source role for dir 0 and dir 1). + // Watermark bypassed: convert iff wid(s24)==DSWS2_FORCE_WID AND current epoch(s35)== + // DSWS2_FORCE_EPOCH. try_gate still runs (sets s58=s_win, conv_apply's precondition). + s_mov_b32 s58, 0 // s_win = 0 default (compares below may bail early) + s_cmp_eq_u32 s24, DSWS2_FORCE_WID + s_cbranch_scc0 .Lcmp_cooldn + s_cmp_eq_u32 s35, DSWS2_FORCE_EPOCH + s_cbranch_scc0 .Lcmp_cooldn +.if DSWS2_FORCE_DIR == 0 + s_mov_b32 s57, 0 // dir 0: compute -> A-feed (forced) + try_gate 0, s58 + conv_apply NCOMP_SLOT, NAFEED_SLOT, -(NFV-VLEAN), 32 +.else + s_mov_b32 s57, 1 // dir 1: compute -> B-feed (forced) + try_gate 1, s58 + conv_apply NCOMP_SLOT, NBFEED_SLOT, -(NFV-VLEAN), 32 +.endif +.endif +.else + s_mov_b32 s58, 0 // s_win = 0 (default: raced no ticket) + lds_get s55, OCCA_PUB_OFF // SENSOR FIX: read claimer-published mid-drain PEAK, not occ_sample + lds_get s56, OCCB_PUB_OFF // at post-drain quiesce (~0 -> false "starved" -> 4/2/2->1/6/1 runaway) + s_cmp_lt_u32 s55, CTRL_LOW + s_cbranch_scc0 .Lcmp_try_b + s_mov_b32 s57, 0 // dir 0: compute -> A-feed + try_gate 0, s58 // s58 = s_win (single-winner per (dir,epoch)) + conv_apply NCOMP_SLOT, NAFEED_SLOT, -(NFV-VLEAN), 32 + s_branch .Lcmp_cooldn +.Lcmp_try_b: + s_cmp_lt_u32 s56, CTRL_LOW + s_cbranch_scc0 .Lcmp_cooldn + s_mov_b32 s57, 1 // dir 1: compute -> B-feed + try_gate 1, s58 + conv_apply NCOMP_SLOT, NBFEED_SLOT, -(NFV-VLEAN), 32 +.endif +.Lcmp_cooldn: +.if CONV_COOLDOWN > 0 + s_cmp_eq_u32 s66, 0 // Task 4: decrement once per epoch (saturating) + s_cbranch_scc1 .Lcmp_quiesce + s_sub_i32 s66, s66, 1 +.endif +.Lcmp_quiesce: + // ORDERING (SPEC 3.4 decision 2): the commit above fully completed (role-slot CAS + reservation + + // s_alloc_vgpr) BEFORE this QUIESCE_CNT bump -- the bump is the snapshot handshake the claimer reads. +.if DSWS2_GQUIESCE + gq_bump // device-scoped global bump (visible to the claimer poll) +.else + lds_fetch_add s61, QUIESCE_CNT_OFF, 1 // exactly one bump per non-claimer wave / super-tile +.endif + epoch_mark 136 // roles[0] = compute last epoch bumped (wedge localize) + bail_mark // per-wave localization: occ[BAIL_BASE + wid*4] = this wave's epoch +.endif +.if DSWS2_CONV + s_branch .Ldispatch +.else + s_branch .Lcompute_follow // this super-tile's compute drained -> re-check epoch/terminal +.endif + +.if DSWS2_CONV +// ============================================================================================ +// Role RE-DISPATCH trampoline (Task 3): scalar-only. Reads the role register s59 (flipped by +// conv_apply on a role conversion, else unchanged) and branches to the matching role's +// per-epoch _follow loop. RE-DISPATCH ONLY -- reached from a per-super-tile _quiesce bail, +// never from first-time entry (the seed arms fall into the full role entry so _alloc/_init +// run once; see the seed block). Lands on _follow (NOT _alloc/_init): the wave's VGPR +// footprint is already correct (conv_apply set it) and INIT already ran -- re-entering _alloc +// would wrongly resize, and re-running _init would reset s35=0, breaking the "wait for the +// NEXT epoch" contract (the wave would re-process the current super-tile). INITFLAG is written +// once (0xACED) and never cleared, so it is the s35 reset -- not INITFLAG -- that _follow preserves. +// s35 (last-seen-epoch) is untouched here, which is what makes a re-dispatched wave wait for +// the NEXT epoch at the top of its new role's _follow loop. Scalar-only (s59 read + s_branch) -> +// adds ZERO OOR/VGPR exposure. +// ============================================================================================ +.Ldispatch: + s_cmp_eq_u32 s59, NCOMP_SLOT + s_cbranch_scc1 .Lcompute_follow + s_cmp_eq_u32 s59, NAFEED_SLOT + s_cbranch_scc1 .Lafeed_follow + s_branch .Lbfeed_follow +.endif + +// ---- A7 role-agnostic terminal (followers): retire. (Claimer retires via .Lclaimer_terminal.) ---- +.Lretire: + tfspan max, 12 // TFPROBE: every follower stamps occ[3] = max exit tick (wall-span end) + alllive_dec // TRACE: follower exit -> --live (peak-concurrent occupancy) + s_endpgm +.else + s_endpgm // DSWS2=0 has no v2 body (this file is always built DSWS2=1) +.endif + .size occ_kernel, .-occ_kernel + +// ---- RGADESC: analysis-only descriptor so `rga -s bin --co` can enumerate + livereg this kernel. +// vgpr 256 ceiling so livereg reports the true s_alloc-grown peak-live. NOT emitted for the PM4 .bin. ---- +.if RGADESC +.amdhsa_kernel occ_kernel + .amdhsa_next_free_vgpr 256 + .amdhsa_next_free_sgpr 72 // body uses up to s69 (s66=n_kseg s67=mask s68=shift s69=chunkHi, FIX 1) + .amdhsa_group_segment_fixed_size 32768 + .amdhsa_user_sgpr_count 15 // FIX 1(g): v2 contract now s0..s14 only (n_kseg/TOTAL_super/ + // magic_kseg dropped -- derived in-kernel / memory-carried) + .amdhsa_wavefront_size32 1 +.end_amdhsa_kernel +.amdgpu_metadata +--- +amdhsa.version: [ 1, 2 ] +amdhsa.kernels: + - .name: occ_kernel + .symbol: occ_kernel.kd + .kernarg_segment_size: 60 + .kernarg_segment_align: 8 + .group_segment_fixed_size: 32768 + .private_segment_fixed_size: 0 + .wavefront_size: 32 + .sgpr_count: 72 + .vgpr_count: 256 + .max_flat_workgroup_size: 256 + .args: [] +.end_amdgpu_metadata +.endif diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_dsws_flow.s b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_dsws_flow.s new file mode 100644 index 000000000000..a2c16c5c4147 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_dsws_flow.s @@ -0,0 +1,1837 @@ +// occ_kernel_dsws.s (gfx1201, wave32) -- MAD-305 DSWS v2 SUBSTRATE SCAFFOLD (PLAN_DSWS_SUBSTRATE_V2.md, Task A1). +// +// v2 re-founds the DSWS GEMM substrate so matrix work is *claimed* (not owned by compile-time wave id), with +// split-K folded in. Work = a pool of (mblk, tcol, ksi) super-tiles; a pinned claimer (wid 0) broadcasts the +// current super-tile; live waves of each role drain shared LDS atomic counters against resident-in-LDS A/B. +// +// Task A1 is SCAFFOLD-ONLY: lift the proven prologue/arming setup from occ_kernel_coop.s (NEVER modified), +// assert the v2 LDS layout fits the 32 KB group segment, and leave each role as a distinct s_endpgm stub. +// The real claimer / feed / compute loops land in A3..A7. The only gates here: (1) assembles clean, +// (2) RGA 0-spill, (3) harness --dsws2 dry-prints the computed params. +// +// ============================================================================================ +// v2 KERNARG CONTRACT (USER_SGPR=15, s0..s14; hardware-preloaded user SGPRs). The host A8 launch +// MUST set COMPUTE_PGM_RSRC2.USER_SGPR=15 and load COMPUTE_USER_DATA_0..14 to match EXACTLY: +// s0:s1 = occ buffer base (>=0x1000B; host zero-inits; see CLAIM-COUNTER / completion offsets below) +// s2:s3 = A base (fp8 e4m3, row-major, 1 byte/elem) +// s4:s5 = Bshuf base (the shuffled-B layout the coop B-feed consumes; same global_load_tr_b64 idiom) +// s6:s7 = C base (fp32; HOST MUST MEMSET C=0 before dispatch -- compute uses global_atomic_add_f32) +// s8 = KT (total K16-steps for the whole matrix = K/16) +// s9 = K(bytes/A-row) (= K, fp8 1 byte/elem) +// s10 = NT*256 (B-saddr advance per K16-step) +// s11 = TOTAL (coop-style total *tiles* = MTL*NTL; carried for addressing compat, NOT the pool size) +// s12 = magic(ceil(2^32/NTL)) (unsigned-div magic for /NTL ; tcol/mblk decode) +// s13 = NTL (number of N tile-columns) +// s14 = FN*256 (B-saddr stride per N-frag) +// (TGID_X now lands in s15 -- UNUSED; this kernel is pool-claim, not workgroup-id based.) +// NOTE: G and SEGK are COMPILE-TIME defsyms (baked into instruction immediates); they are NOT kernargs. +// FIX 1 (round-table Opus+Codex pass): v1 of this contract passed n_kseg/TOTAL_super/magic_kseg as +// s15/s16/s17, but the PM4 host only preloads COMPUTE_USER_DATA_0..15 (USER_SGPR<=16; every proven +// launch path in this tree uses 15) -- s16/s17 could NEVER actually arrive in hardware SGPRs, AND +// s16 was independently being reused per-chunk on the host as the compositor-safe chunk terminal +// (a second, unrelated collision on the same slot). This file now drops s15/s16/s17 entirely: +// n_kseg is DERIVED in-kernel from KT (s8) and the compile-time KSEG_STEPS=SEGK/16: +// n_kseg = KT >> NKSEG_SHIFT, where NKSEG_SHIFT=log2(KSEG_STEPS) is a compile-time `.set` +// (small .if ladder over KSEG_STEPS in {1,2,4,8,16}; SEGK is always a power-of-two +// multiple of 16, so KSEG_STEPS is always a power of two in that set). +// shift/mask (the sti -> (t,ksi) split) are derived ONCE in the prologue from n_kseg: +// shift = s_ff1_i32_b32(n_kseg) (bit index of n_kseg's single set bit; n_kseg=1 -> 0) +// mask = n_kseg - 1 +// DECODE_STI then does ksi = sti & mask ; t = sti >> shift -- this handles n_kseg=1 for +// free (shift=0, mask=0 -> ksi=0, t=sti), so the old magic-div n_kseg==1 special-case is +// GONE (it's no longer needed, not just hidden). +// the chunk terminal (old TOTAL_super/"chunkHi") is now MEMORY-CARRIED instead of a kernarg: the +// host writes the current chunk's terminal sti bound to occ[24] (occW[6]) once per chunk; +// the claimer reads occ[24] ONCE per dispatch (stable for the whole chunk) instead of +// receiving it as a broadcast kernarg. On sti >= occ[24] the claimer publishes a SENTINEL +// (0xFFFFFFFF) into STI_OFF instead of the raw over-claimed sti; followers (b-feed/a-feed/ +// compute) retire when STI_OFF == 0xFFFFFFFF instead of comparing against the (now +// nonexistent) TOTAL_super kernarg. +// See "CLAIM-COUNTER & completion occ offsets" by .Lclaimer for the full occ-buffer layout +// (occ[24]/occW[6] = chunk terminal bound, added by FIX 1). +// +// SCALAR REGS (derived in the prologue, before any clobber; none collide with DECODE_STI's own clobber +// list s18/s36, lds_*'s s49, the claimer's s16/s17/s35/s44, or any role body's transients, all <= s65): +// s66 = n_kseg (derived; dead after shift/mask below are computed -- kept only for that derivation) +// s67 = mask (n_kseg - 1) -- LIVE for the whole kernel; read by every DECODE_STI call, every role. +// s68 = shift (log2 n_kseg) -- LIVE for the whole kernel; read by every DECODE_STI call, every role. +// s69 = chunkHi (claimer-only; loaded once per dispatch from occ[24] right before .Lclaim_loop). +// +// Everything new is gated behind the fresh `DSWS2` build symbol (analogous to coop's `DSWS`). + +.amdgcn_target "amdgcn-amd-amdhsa--gfx1201" + +// ---- tile defsyms (lifted from occ_kernel_coop.s) ---- +.ifndef FM + .set FM, 2 // per-compute-wave M-frags (M-band = FM*16 rows) +.endif +.ifndef FN + .set FN, 4 // shared N-frags (the reuse operand) +.endif +.ifndef RGADESC + .set RGADESC, 0 // 1 = emit analysis-only AMDHSA descriptor for RGA livereg +.endif +.ifndef DIAG + .set DIAG, 0 // 1 = phase-marker instrumentation (unused in the A1 scaffold) +.endif +.ifndef SAFEPROBE + .set SAFEPROBE, 0 // 1 = clamp per-lane vector address regs into a provable in-buffer bound +.endif +.ifndef TFPROBE + .set TFPROBE, 0 // 1 = emit s_sendmsg_rtn GET_REALTIME wall-span capture (each wave stamps +.endif // occ[2]=min entry tick, occ[3]=max exit tick -> host TF readout). + // Default 0 => ZERO bytes emitted, .text byte-identical to the + // production bin (uses only already-allocated regs s30/s31/s49/v5). +.ifndef PHASEPROBE + .set PHASEPROBE, 0 // 1 = in-kernel PHASE TIMER: each compute wave stamps GET_REALTIME at every +.endif // phase boundary and atomic-adds the delta into per-phase occ accumulators + // (occ[64..69], bytes 256..276, ABOVE the per-chunk memset -> accumulate over + // the whole run). Host prints ticks + % per phase. Default 0 => byte-identical. +.ifndef WOFLUSH + .set WOFLUSH, 0 // BURST-SCOPED FLUSH (LDS-halving lever, council 2026-07-05): 1 = drop the +.endif // per-rowblk LDS accumulator banks entirely; each compute burst atomic-adds + // its fp32 ACC frags DIRECTLY to C (global_atomic_add_f32, fp32-exact, same + // addresses as the write-once completer store). Build with ACC_N=0 (host + // DSWS2_ACC_N=0) -> LDS/WG ~8KB -> ~7 WGs/CU -> per-SIMD VGPR pool BINDS -> + // the dyn-VGPR traveling-peak finally engages (grow-fail>0). Re-incurs the + // n_kseg-x C-write atomic traffic write-once removed; tunable later by burst + // K-depth J + KMAJOR. Default 0 => byte-identical to the write-once bin. +.ifndef NOCFLUSH + .set NOCFLUSH, 0 // PERF PROBE ONLY: 1 = skip the global_atomic_add_f32 C-flush loop (keep ALL +.endif // other bookkeeping/handshake). Isolates the device-atomic C-reduction cost + // from the coordination handshake. Result is WRONG (C never written -> oracle + // fails) -- span/TF only. Default 0 => byte-identical. +.ifndef KMAJOR + .set KMAJOR, 0 // PERF PROBE: 1 = K-MAJOR super-tile traversal. Default decode packs ksi in +.endif // the LOW bits (all n_kseg segments of a C cell claimed consecutively -> + // up to n_kseg WGs hammer one C cell at once = max atomic contention). + // KMAJOR decodes ksi = sti / TOTAL (high), t = sti % TOTAL (low) via a + // magic-div (magic_TOTAL from occ[62], loaded to s76 in prologue) -> the + // 32 segments of a cell are spread TOTAL apart in claim order (near-zero + // concurrent contention) + adjacent tiles (shared operand bands) claimed + // close in time (L2 reuse). Correctness-preserving (C add is commutative). +.ifndef CSTORE + .set CSTORE, 0 // PERF PROBE ONLY: 1 = replace the flush's global_atomic_add_f32 with an +.endif // equal-count NON-atomic global_store_b32 (same #mem-ops, same addresses, NO + // RMW/contention). Isolates atomic-contention from raw write-bandwidth: if + // TF jumps vs atomics -> contention; if flat -> bandwidth-bound. Result WRONG + // (last-writer, no accumulation) -- span/TF only. Default 0 => byte-identical. +.ifndef TRACE + .set TRACE, 0 // 1 = per-super-tile CLAIMER trace: append one row/super-tile {tick, segcnt, +.endif // epoch, nComp/nAfeed/nBfeed live role slots, ring occA/occB peak, + // convCount, vresv, sti, quiesce} to a host-provided buffer (VA in + // occ[52:53], cap in occ[54]). Time-series of the adaptive wave-role + // economy. Requires DSWS2_CONV=1 (rows written in the quiesce path). + // Default 0 => ZERO bytes; uses free high SGPRs s70..s74. +.ifndef DYNVGPR + .set DYNVGPR, 1 // 1 = compute waves s_alloc_vgpr-grow per rowblk; feeds/claimer stay lean 32 +.endif +.ifndef SLEEPN + .set SLEEPN, 2 // s_sleep arg in the busy-waits (yield issue cycles to partner waves) +.endif + +// ============================================================================================ +// DSWS v2 LDS layout (bytes from group-segment base; words u32 unless noted). Mirrors the placement +// of the coop file's LDS `.set` block. Defined unconditionally (uses only G/SEGK/FM/FN, always set). +// ============================================================================================ +.ifndef DSWS2 + .set DSWS2, 0 +.endif +.ifndef G + .set G, 6 // cooperative M-extent (rowblks per super-tile) = LDS accumulator-bank count (ACC_N) +.endif +.ifndef SEGK + .set SEGK, 64 // split-K segment size in K-elements (multiple of 16) +.endif +// ---- v2 control/claim words ---- +.set STI_OFF, 0 // broadcast super-tile id +.set EPOCH_OFF, 4 +.set ROWBLK_NEXT_OFF, 8 // per-super-tile rowblk claim counter +.set ROWBLK_DONE_OFF, 12 // per-super-tile completion counter +.set BFRAG_NEXT_OFF, 16 // B-frag claim counter +.set AROW_NEXT_OFF, 20 // A-rowblk claim counter +.set NCOMP_SLOT, 24 +.set NAFEED_SLOT, 28 +.set NBFEED_SLOT, 32 +.set GATE_OFF, 36 // u32[4] -> 36,40,44,48 (conversion gates) +.set VRESV_OFF, 52 // vgpr_reserved +.set SEGCNT_OFF, 56 // controller clock +// ---- A3..A7 additions (still inside the 0..256 control region; A1 offsets 0..56 unchanged) ---- +.set BFRAG_DONE_OFF, 60 // B-frag STORE-completion counter (compute gates on this, NOT the claim ctr) +.set AROW_DONE_OFF, 64 // A-rowblk STORE-completion counter (compute gates on this) +.set INITFLAG_OFF, 68 // barrier-free LDS-init publish flag (claimer writes 0xACED LAST) +// ---- Phase-B (DSWS2_CONV) control state: role-mix snapshot slots + quiesce counter ---- +// Based at INITFLAG_OFF+4 (NOT the brief's SEGCNT_OFF+4): the brief predates the A3..A7 control +// words (BFRAG_DONE/AROW_DONE/INITFLAG at 60/64/68), so SEGCNT_OFF+4=60 would collide with them. +// Basing after the LAST control word keeps the new state inside the 0..255 control gap BELOW the +// fixed resident region (BRES_OFF=256), so NO resident-region repoint is needed -- the resident +// BRES_OFF/ARES_OFF immediates (emitted unconditionally in the kernel body) stay untouched, which +// is what keeps the DSWS2_CONV=0 binary byte-identical to the Phase-A green bin. All `.set`s here +// are inert (emit no bytes); the only new code (claimer init) is gated under `.if DSWS2_CONV`. +.ifndef DSWS2_CONV + .set DSWS2_CONV, 0 // 0 = pre-conversion static substrate (Phase A green); 1 = Phase B +.endif +.ifndef DSWS2_TICKET_SELFTEST + .set DSWS2_TICKET_SELFTEST, 0 // DIAG-only try_gate single-winner smoke (Task 4 Step 3); default 0 = no bytes +.endif +.ifndef CONV_COOLDOWN + .set CONV_COOLDOWN, 0 // Task 4: per-wave post-conversion cooldown epochs. 0 = spec-faithful (no + // cooldown, byte-identical to pre-Task-4); >0 damps thrash (skip N epochs + // of watermark decision after a wave converts role). +.endif +// Task 5: deterministic bring-up hook. DSWS2_FORCE=1 makes exactly wave DSWS2_FORCE_WID convert +// direction DSWS2_FORCE_DIR at epoch DSWS2_FORCE_EPOCH, watermarks bypassed -- a reproducible, +// single-wave/single-epoch GPU proof of role conversion. Default DSWS2_FORCE=0 emits ZERO bytes +// (byte-identical to pre-Task-5). +.ifndef DSWS2_FORCE + .set DSWS2_FORCE, 0 +.endif +.ifndef DSWS2_FORCE_WID + .set DSWS2_FORCE_WID, 0 +.endif +.ifndef DSWS2_FORCE_DIR + .set DSWS2_FORCE_DIR, 0 // 0/1 = compute->A/B ; 2/3 = A/B->compute +.endif +.ifndef DSWS2_FORCE_EPOCH + .set DSWS2_FORCE_EPOCH, 1 +.endif +// Rolling dyn-VGPR sum-envelope (2026-07-02 spec). ENVELOPE routes the per-rowblk compute burst grow +// through the shared vgpr_reserved counter so at most PEAK_CONC waves hold peak at once (the +// multi-grower collision, ISA 3.3.3.2, becomes unreachable). All default to the byte-identical value: +// ENVELOPE=0/STAGGER=0 emit ZERO new bytes and PEAK_CONC/STAGGER_PERIOD are inert unless their gate is on. +.ifndef DSWS2_ENVELOPE + .set DSWS2_ENVELOPE, 0 // 1 = route the per-rowblk compute burst grow through the vgpr_reserved +.endif // sum-envelope. 0 = HEAD (bare .Lcompute_grow) -> .text byte-identical. +.ifndef PEAK_CONC + .set PEAK_CONC, 2 // concurrent compute peaks the budget admits (R3 sweep). Used iff ENVELOPE=1. +.endif +.ifndef DSWS2_STAGGER + .set DSWS2_STAGGER, 0 // 1 = lock-free phase-token stagger (Task 9). 0 -> emergent envelope stagger. +.endif +.ifndef STAGGER_PERIOD + .set STAGGER_PERIOD, 4 // phase slots in the stagger ring (R3 sweep). Used iff STAGGER=1 (inert here). +.endif +.set SNAP_BASE, (INITFLAG_OFF + 4) // u32[6]: [parity*3 + {0:nC,1:nA,2:nB}] role-mix snapshots +.set QUIESCE_CNT_OFF,(SNAP_BASE + 6*4) // u32 role-agnostic bail counter (LDS; DSWS2_GQUIESCE=0) +// SENSOR FIX: the claimer publishes its MID-DRAIN ring-occupancy PEAK here each super-tile; the conversion +// decisions read THESE instead of sampling occ_sample at their own quiesce (where occ_X reads ~0 post-drain +// -> always "starved" -> the 4/2/2->1/6/1 compute->feed runaway). Mid-drain peak = the true demand signal. +.set OCCA_PUB_OFF, (QUIESCE_CNT_OFF + 4) // claimer-published occ_A peak +.set OCCB_PUB_OFF, (OCCA_PUB_OFF + 4) // claimer-published occ_B peak +.set DSWS2_STATE_END,(OCCB_PUB_OFF + 4) +// DSWS2_GQUIESCE (2026-07-02 SUSPECT #2 candidate fix): route the QUIESCE handshake through a DEVICE-SCOPED +// GLOBAL atomic in the uncached occ buffer (byte QUIESCE_GOFF), mirroring the GREEN occ[20] claim/occ[0] +// live handshake, instead of the barrier-free LDS counter (whose cross-wave visibility is unguaranteed and +// is the leading SUSPECT #2 hang mechanism). occ buffer = AllocGpu 0x1000 (1024 u32, uncached); host uses +// occ[0..6] + DIAG scratch (<= byte 116); byte 200 (occ[50]) is provably free. Default 0 => LDS path, +// .text byte-identical. Requires DSWS2_CONV (QUIESCE only exists there). +.ifndef DSWS2_GQUIESCE + .set DSWS2_GQUIESCE, 0 +.endif +.set QUIESCE_GOFF, 200 // occ[] byte offset for the global QUIESCE counter (occ[50]) +// ---- TRACE (per-super-tile time-series) occ handshake words + row layout ---- +.set TRACE_PTR_OFF, 208 // occ[52:53] = trace buffer VA (host writes lo/hi per chunk) +.set TRACE_CAP_OFF, 216 // occ[54] = MAXROWS (host-provided row capacity) +.set TRACE_IDX_OFF, 220 // occ[55] = GLOBAL row-claim counter (all WGs' claimers share it) +.set TRACE_WGID_OFF, 224 // occ[56] = GLOBAL wg-id dispenser (claim-order 0..pool-1) +.set FATLIVE_OFF, 228 // occ[57] = live count of GROWN (fat NFV-VGPR) compute waves +.set FATMAX_OFF, 232 // occ[58] = PEAK concurrent fat waves -> x NFV = VGPR in flight (== B probe) +.set ALLLIVE_OFF, 240 // occ[60] = live count of ALL resident waves (++entry/--exit) + // occ[1] (byte 4) = PEAK concurrent resident waves (vs 2048 HW ceiling) +.set TRACE_ROW_BYTES, 64 // 16 u32/row +// DSWS2_BAILMARK (SUSPECT #2 localization, 2026-07-03): each follower publishes its OWN epoch (s35) to a +// PER-WAVE occ slot (BAIL_BASE + wid*4) at its _quiesce bail. One-shot per super-tile per wave -> minimal +// timing perturbation (NOT the claimer's per-spin DIAG poll stores, which are the heisenbug source and stay +// DIAG-only). After a watchdog abort the host reads occ[BAIL_BASE/4 + wid]: every follower's slot == the +// hung epoch => all reached their bail (=> a QUIESCE visibility/lost-update, gq relevant); ONE slot stale +// at the prior epoch => that exact wave is the STRAGGLER (stuck in _alloc/_init/_follow; gq irrelevant). +// Per-WAVE (not per-role): 4 compute share one role, so a role mark's last-writer-wins would hide a single +// straggler. Default 0 => no bytes, .text byte-identical. Requires DSWS2_CONV. +.ifndef DSWS2_BAILMARK + .set DSWS2_BAILMARK, 0 +.endif +.set BAIL_BASE, 160 // occ[] byte offset base for per-wave bail marks: occ[40..47] +.set CONVCNT_OFF, 192 // occ[48]: DIAG conversion-commit counter (proves waves switch role) + // (host prints occ[40..47] as BAIL[w0..w7]; clear of the + // occ[32..36]/occ[39] DSWS sensor+roles slots and occ[50] gq) +.set KSEG_STEPS, (SEGK/16) // K16-steps per split-K segment = SEGK K-elements / 16 +// FIX 1(b): NKSEG_SHIFT = log2(KSEG_STEPS), so the prologue can derive n_kseg = KT >> NKSEG_SHIFT instead +// of receiving it as a (now-dropped) kernarg. SEGK is always a power-of-two multiple of 16 in every +// config this file is built with, so KSEG_STEPS is always a power of two in {1,2,4,8,16}; a static +// ladder over that small set is simpler/safer than a general-purpose compile-time log2. +.if KSEG_STEPS == 1 + .set NKSEG_SHIFT, 0 +.elseif KSEG_STEPS == 2 + .set NKSEG_SHIFT, 1 +.elseif KSEG_STEPS == 4 + .set NKSEG_SHIFT, 2 +.elseif KSEG_STEPS == 8 + .set NKSEG_SHIFT, 3 +.elseif KSEG_STEPS == 16 + .set NKSEG_SHIFT, 4 +.else + .error "KSEG_STEPS (SEGK/16) must be a power of two in {1,2,4,8,16}" +.endif +// resident regions aligned to 256B +.set BRES_OFF, 256 // resident B for current super-tile +.set BRES_BYTES, (FN*16*SEGK) // = 4*16*64 = 4096 at the default config +.set ARES_OFF, (BRES_OFF + BRES_BYTES) // resident A for current super-tile +.set ARES_BYTES, (G*16*FM*SEGK) // = 6*16*2*64 = 12288 at the default config +.set LDS_TOTAL_DSWS2, (ARES_OFF + ARES_BYTES) +// (old single-slot cap check retained; 16640 < 32768 -> always passes. The RING layout below is what +// the ring role loops actually use; its own cap check follows.) +.if LDS_TOTAL_DSWS2 > 32768 + .error "DSWS2 LDS layout exceeds 32768B group segment" +.endif +// ============================================================================================ +// FIX 1 (flow) -- FLOW ECONOMY LDS layout (N-deep pool + per-wave ROLE mailbox). NEW symbols; the +// single-slot control words above stay DEFINED (no bytes; only used under .if DSWS2_CONV/DIAG/TRACE, +// all 0). The flow role loops reference ONLY the symbols below. There is NO publish flag to poll: +// compute streams rowblks from the READY pool (DRAIN_HEAD), feeds stage the next super-tile into the +// next FREE slot (STAGE_HEAD); both frontiers are super-tile indices, slot = (index mod POOL_N). Each +// wave reads its ROLE[wid] mailbox each cycle and simply IS that role (stale mailbox = last role = +// coast). Single writer (coordinator wid0) -> no CAS. See FLOW_ECONOMY_DESIGN.md. +.ifndef POOL_N + .set POOL_N, 3 // pool depth: 3 slots -> 48KB operands + ctrl < 64KB (4 = 64KB, too tight) +.endif +.ifndef ACC_N + .set ACC_N, 1 // per-rowblk fp32 reduction accumulator banks (rowblk-lifetime). CO-BUDGET with +.endif // POOL_N: OP_BASE + POOL_N*OPSTRIDE + ACC_N*ACC_STRIDE <= 65536. Defaults keep + // the POOL_N=3 bare build legal (57600); stagger build uses POOL_N=2 ACC_N=2 (49408). +.ifndef COORD_PERIOD + .set COORD_PERIOD, 64 // coordinator sense/nudge cadence (loop cycles); lazy is fine (waves coast) +.endif +// DEADMAN: per-wave wall-clock watchdog. Every wave stamps its start RTC and, at each loop head, force- +// retires if it has been alive > DEADMAN_TICKS. Converts a COORDINATION hang (a frontier that never +// advances -> waves spin the loop forever) into a CLEAN drain: all waves retire, occ[0]->0, the queue +// goes idle and the EOP fence fires -> NO wedge, NO desktop brick (the result is just incomplete, which +// the oracle flags). This is what makes scale-stress SAFE to run. Fires at 0.5s < host chunkMaxS(0.75s) +// so the host observes the clean drain before its own bail. It does NOT cover the s_alloc_vgpr grow-spin +// (grow-stagger, ISA 3.3.3.2) -- that spin never reaches the loop head; it's a separate gate (M=576 only). +// DEADMAN=0 -> zero bytes (for clean perf/byte-identity bins). +.ifndef DEADMAN + .set DEADMAN, 1 +.endif +.ifndef RETBARRIER + .set RETBARRIER, 1 // count-to-WAVES collective exit: each wave checks in at .Lflow_dead and all +.endif // s_endpgm TOGETHER once the WG's count hits WAVES -> the EOP registers a +.ifndef RETBAR_MAX // clean coordinated dispatch completion -> the fence FIRES (the staggered + .set RETBAR_MAX, 1000000 // coordinator-broadcast retire fires at 8 waves but not 16). Bounded-wait +.endif // (no message bus / no s_alloc) so it can NEVER hang the wave. +.ifndef DEADMAN_TICKS + .set DEADMAN_TICKS, 50000000 // 0.5s @ 100MHz RTC; a normal chunk is ~ms, so this fires ONLY on a wedge +.endif +// ---- shared frontiers + mailbox (single copy, at the front). 3-frontier pipeline: +// DRAIN_HEAD <= STAGE_HEAD <= ASSIGN_HEAD <= DRAIN_HEAD + POOL_N (all monotone u32) ---- +.set ASSIGN_HEAD_OFF, 0 // next local index to assign a global super-tile (SINGLE writer = coordinator wid0) +.set STAGE_HEAD_OFF, 4 // oldest assigned-but-not-fully-staged index (feeds; ds_cmpstore advance) +.set DRAIN_HEAD_OFF, 8 // oldest not-fully-drained index (compute; ds_cmpstore advance) +.set RINGINIT_OFF, 12 // barrier-free LDS-init publish flag (coordinator writes 0xACED LAST) +.set FLOWTERM_OFF, 16 // terminal flag: coordinator sets 0xDEAD once all super-tiles claimed +.set ROLE_BASE, 20 // per-wave mailbox: ROLE[wid] at ROLE_BASE + wid*4 (coordinator-written) +// role codes stored in ROLE[wid]: +.set ROLE_COMPUTE, 0 +.set ROLE_AFEED, 1 +.set ROLE_BFEED, 2 +.set ROLE_RETIRE, 3 +// ---- per-slot control block: SLOTC_BASE + slot*SLOTC_STRIDE + field (same fields as the ring) ---- +// coordinator-local TILE-CLAIM state (single writer = wid0), tucked in the reserved 32-wave mailbox +// tail (safe while WAVES<=30). occ[20] claims whole TILES; the coordinator emits a tile's n_kseg +// super-tiles (sti=(t<= G +.set SL_BFNEXT, 16 // B-frag claim counter (B-feeds) +.set SL_BFDONE, 20 // B-frags stored; slot B-ready when == FN +.set SL_ARNEXT, 24 // A-rowblk claim counter (A-feeds) +.set SL_ARDONE, 28 // A-rowblks staged; slot A-ready when == G (slot READY = B-ready && A-ready) +// ---- per-slot operand buffers: OP_BASE + slot*OPSTRIDE ; BRES at +BRES_ROFF, ARES at +ARES_ROFF ---- +.set OP_BASE, 256 // 256B-aligned; below it: frontier + mailbox + POOL_N ctrl blocks +.set OPSTRIDE, (BRES_BYTES + ARES_BYTES) // 4096 + 12288 = 16384 per slot +.set BRES_ROFF, 0 // resident B within a slot +.set ARES_ROFF, BRES_BYTES // resident A within a slot (after B) +// ---- per-rowblk reduction accumulator pool: ACC_BASE + bank*ACC_STRIDE (bank in [0,ACC_N)) ---- +// fp32, rowblk-lifetime (persists across all n_kseg K-segments of a rowblk); DISTINCT from the +// segment-lifetime operand pool above. One bank = one C-rowblk = FM*FN frags x 1024B. +.set ACC_BASE, (OP_BASE + POOL_N*OPSTRIDE) // after the operand pool +.set ACC_STRIDE, (FM*FN*1024) // = 8192 @ FM=2 FN=4 (one C-rowblk) +.set LDS_TOTAL_FLOW,(ACC_BASE + ACC_N*ACC_STRIDE) // POOL3/ACC1: 57600 ; POOL2/ACC2: 49408 +.if LDS_TOTAL_FLOW > 65536 + .error "FLOW LDS layout exceeds 65536B group segment (hardware WGP limit) -- lower POOL_N or ACC_N" +.endif +.if (SLOTC_BASE + POOL_N*SLOTC_STRIDE) > OP_BASE + .error "FLOW per-slot control blocks overlap the operand region (raise OP_BASE)" +.endif +// Phase-B state must fit in the control gap below the resident region (inert compile check, no bytes). +.if DSWS2_STATE_END > BRES_OFF + .error "DSWS2 Phase-B state (SNAP_BASE/QUIESCE_CNT) overlaps resident B region (BRES_OFF)" +.endif + +.if DSWS2 + // ---- launch wave count (EMERGENT economy: NO baked compute/feed mix; roles emerge at runtime) ---- + .ifndef WAVES + .set WAVES, 16 // waves/WG launched; host launches the SAME count. + .endif + .if WAVES > 30 + .error "WAVES>30 collides with COORD_KSI/T at ROLE[30]/ROLE[31] -- relocate coord state first" + .endif +.endif + +// ============================================================================================ +// VGPR layout (lifted from occ_kernel_coop.s) -- compute frags live ABOVE the lean-32 block and are +// only touched AFTER s_alloc_vgpr NFV. Feeds/claimer stay in the lean block (v0..v31). +// ============================================================================================ +.set ACC, 32 // accumulators: FM*FN frags x 8 f32 (v32..) +.set FA, (ACC + 8*FM*FN) // compute A frags (from resident LDS): FM x 2 +.set FB, (FA + 2*FM) // compute B frags (from resident LDS): FN x 2 +.set NFV, ((FB + 2*FN + 15) & ~15) // grown footprint, rounded to a 16-VGPR dyn block (=112 @ 2x4) +.set VLEAN, 32 // lean footprint (feeds, claimer, compute pre/post rowblk) +.set BSTG, 16 // staging regs (lean block, < 32): B-feed FN-frag / A-feed FM-frag + +// ---- dyn-VGPR PRE-GROW temp-reg ceiling (coop death-cert: a >v15 src pre-grow is poison under dyn). +// Gate every PRE-grow-reachable LDS/atomic temp to v11/v14 (INTERIOR to the launched 16-VGPR block). ---- +.if DYNVGPR + .set RG_A, 11 // lds_get / fetch_add address + .set RG_D, 14 // lds_get / fetch_add data+return + .set RP_A, 11 // lds_put address + .set RP_D, 14 // lds_put data +.else + .set RG_A, 27 + .set RG_D, 28 + .set RP_A, 28 + .set RP_D, 29 +.endif + +// ============================================================================================ +// LDS helper macros (s49 = exec save; v2 = lane = tid&31, set in prologue). +// ============================================================================================ +.macro lds_get sdst, off // wave-uniform read LDS[off] -> scalar sdst + v_mov_b32 v[RG_A], \off + ds_load_b32 v[RG_D], v[RG_A] + s_wait_dscnt 0x0 + v_readfirstlane_b32 \sdst, v[RG_D] +.endm +.macro lds_get_r sdst, saddr // wave-uniform read LDS[saddr] (RUNTIME addr in a sreg) -> sdst + v_mov_b32 v[RG_A], \saddr + ds_load_b32 v[RG_D], v[RG_A] + s_wait_dscnt 0x0 + v_readfirstlane_b32 \sdst, v[RG_D] +.endm +.macro lds_put off, ssrc // lane-0-of-wave writes scalar ssrc -> LDS[off] + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lput_skip\@ + v_mov_b32 v[RP_A], \off + v_mov_b32 v[RP_D], \ssrc + ds_store_b32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lput_skip\@: + s_mov_b32 exec_lo, s49 +.endm +.macro lds_fetch_add sdst, off, val // sdst <- old LDS[off]; LDS[off]+=val (lane-0 atomic, broadcast) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lfa_skip\@ + v_mov_b32 v[RP_A], \off + v_mov_b32 v[RP_D], \val + ds_add_rtn_u32 v[RP_D], v[RP_A], v[RP_D] // v[RP_D] <- old; LDS[off] += val + s_wait_dscnt 0x0 +.Lfa_skip\@: + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 \sdst, v[RP_D] // broadcast lane-0's old value +.endm +.macro lds_inc off // lane-0-of-wave LDS[off] += 1 (no return) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Linc_skip\@ + v_mov_b32 v[RP_A], \off + v_mov_b32 v[RP_D], 1 + ds_add_u32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Linc_skip\@: + s_mov_b32 exec_lo, s49 +.endm +// ---- FIX 1a ring: RUNTIME-address variants (slot-indexed counters live at SLOTC_BASE+slot*32+field, +// a runtime scalar). Mirror lds_fetch_add / lds_inc but take the address in a sreg. ---- +.macro lds_fetch_add_r sdst, saddr, val // sdst <- old LDS[saddr]; LDS[saddr]+=val (lane-0 atomic, bcast) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lfar_skip\@ + v_mov_b32 v[RP_A], \saddr + v_mov_b32 v[RP_D], \val + ds_add_rtn_u32 v[RP_D], v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lfar_skip\@: + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 \sdst, v[RP_D] +.endm +.macro lds_inc_r saddr // lane-0-of-wave LDS[saddr] += 1 (RUNTIME addr, no return) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lincr_skip\@ + v_mov_b32 v[RP_A], \saddr + v_mov_b32 v[RP_D], 1 + ds_add_u32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lincr_skip\@: + s_mov_b32 exec_lo, s49 +.endm +// ---- FIX 1 flow: slot-of-head (index mod POOL_N) + monotone CAS frontier advance ---- +.macro slot_of dst, head, scr // \dst = \head mod POOL_N (\scr = scratch, only used for N=3) +.if POOL_N == 1 + s_mov_b32 \dst, 0 // single tile in flight (stagger model: one tile's g banks fill LDS) +.elseif POOL_N == 2 + s_and_b32 \dst, \head, 1 +.elseif POOL_N == 4 + s_and_b32 \dst, \head, 3 +.elseif POOL_N == 3 + s_mul_hi_u32 \dst, \head, 0xAAAAAAAB // q ~ head/3 (magic-div; q = mulhi>>1) + s_lshr_b32 \dst, \dst, 1 + s_mul_i32 \scr, \dst, 3 + s_sub_u32 \dst, \head, \scr // slot = head - 3*q +.else + .error "slot_of: POOL_N must be in {2,3,4}" +.endif +.endm + +// acc_base_of: \dst = LDS byte address of accumulator bank \bank (bank in [0,ACC_N)) = ACC_BASE + bank*ACC_STRIDE. +// ACC_STRIDE is a compile-time constant, so s_mul_i32 is exact for any FM*FN (no pow2 assumption). +.macro acc_base_of dst, bank + s_mul_i32 \dst, \bank, ACC_STRIDE + s_add_u32 \dst, \dst, ACC_BASE +.endm +.macro lds_cmpstore_adv off, sexp // lane0 monotone bump: if LDS[off]==\sexp -> LDS[off]=\sexp+1 + s_mov_b32 s49, exec_lo + s_add_u32 s60, \sexp, 1 // scalar ALU ignores exec -> safe under the mask + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lcasadv_skip\@ + v_mov_b32 v11, \off // vaddr + v_mov_b32 v14, s60 // vNEW = exp+1 + v_mov_b32 v13, \sexp // vCMP -> vdst old (return unused) + ds_cmpstore_rtn_b32 v13, v11, v14, v13 // MEM=(MEM==exp)?exp+1:MEM (idempotent; losers no-op) + s_wait_dscnt 0x0 +.Lcasadv_skip\@: + s_mov_b32 exec_lo, s49 +.endm +// ---- DEADMAN watchdog: s70 = this wave's start RTC (low 32b); s71 = throttle counter (repurposed high-RTC +// reg, which is unused at TRACE=0 -- deadman_check only reads s70). The message-bus RTC read (s_sendmsg_rtn) +// is an SQ-front-end op; hundreds of idle COAST waves hitting it EVERY loop iteration spam the front-end, +// starving the compositor's SQC(inst) fetch (2026-07-05 MODE1 brick) AND destabilizing the coast wall +// (identical STAGINSTR work measured 0.32s vs 2.0s). THROTTLE: only read the RTC every DEADMAN_EVERY iters. ---- +.ifndef DEADMAN_EVERY + .set DEADMAN_EVERY, 64 // message-bus RTC-read cadence (in loop iters); force-retire slack = DEADMAN_EVERY iters +.endif +.macro deadman_stamp // stamp start RTC (low 32b in s70) once at entry +.if DEADMAN + s_sendmsg_rtn_b64 s[70:71], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + s_mov_b32 s71, 0 // repurpose the (TRACE=0-unused) high-RTC reg as the throttle counter +.endif +.endm +.macro deadman_check // if alive > DEADMAN_TICKS -> clean force-retire (no wedge) +.if DEADMAN + s_add_u32 s71, s71, 1 // THROTTLE: touch the SQ-front-end message bus only every + s_cmp_ge_u32 s71, DEADMAN_EVERY // DEADMAN_EVERY iters (idle coast waves would else spam it -> brick) + s_cbranch_scc0 .Ldm_skip\@ + s_mov_b32 s71, 0 + s_sendmsg_rtn_b64 s[62:63], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + s_sub_u32 s62, s62, s70 // elapsed = now_lo - start_lo (u32 wrap-safe; deadline << 42s) + s_cmp_ge_u32 s62, DEADMAN_TICKS + s_cbranch_scc1 .Lflow_retire +.Ldm_skip\@: +.endif +.endm +// lds_put_r (RUNTIME-addr write) is also defined inside the .if DSWS2_CONV||DSWS2_ENVELOPE block below; +// the ring needs it at CONV=0/ENV=0, so define an identical copy here, guarded to avoid a double-def +// when either gate is on (the ring is always built CONV=0 ENV=0). +.if !(DSWS2_CONV || DSWS2_ENVELOPE) +.macro lds_put_r saddr, ssrc // lane-0 write ssrc -> LDS[saddr] (RUNTIME addr) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lputr_skip\@ + v_mov_b32 v[RP_A], \saddr + v_mov_b32 v[RP_D], \ssrc + ds_store_b32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lputr_skip\@: + s_mov_b32 exec_lo, s49 +.endm +.endif + +// ============================================================================================ +// Super-tile decode + resident A/B staging macros (A3..A6). Decode (Naming/symbols): +// ksi = sti & mask ; t = sti >> shift ; mblk = t / NTL ; tcol = t % NTL. +// FIX 1(d): n_kseg is ALWAYS a power of two (it's KT >> NKSEG_SHIFT, both compile-time-shift-derived), +// so the sti->(t,ksi) split is an exact shift/mask -- no magic-div, no n_kseg==1 special-case (shift=0, +// mask=0 falls out of the general path for free: ksi=0, t=sti). /NTL still goes via magic (s12), +// unsigned-division mul_hi (coop GENDIV idiom), since NTL is not generally a power of two. +// ============================================================================================ +.macro DECODE_STI // in: s17=sti, s67=mask, s68=shift ; out: s19=mblk s30=tcol s31=ksi ; clob: s18,s36 +.if KMAJOR + s_mul_hi_u32 s31, s17, s76 // ksi = sti / TOTAL (magic-div, magic_TOTAL in s76; K-major high bits) + s_mul_i32 s36, s31, s11 // ksi * TOTAL + s_sub_u32 s18, s17, s36 // t = sti - ksi*TOTAL (low) +.else + s_and_b32 s31, s17, s67 // ksi = sti & mask (mask-bounded -> ksi in [0,n_kseg-1]) + s_lshr_b32 s18, s17, s68 // t = sti >> shift +.endif +.if SAFEPROBE + // brick-PROOF ti clamp (the "future ti clamp" line 752 promised; COOP_STATUS.md:145 racy-garbage-ti->OOB). + // A racy/torn sti read (during the claimer's per-super-tile republish) can decode a garbage t -> garbage + // mblk/tcol -> the A/B/C SCALAR base goes out of buffer -> gfxhub page fault -> MODE1 brick. SAFEPROBE + // already pins the per-lane vaddr (v8/v9/v10); this pins the tile index too, so EVERY global address is + // provably in-buffer. s11=TOTAL is userdata, never clobbered. s36 is DECODE_STI scratch (rewritten below). + s_sub_u32 s36, s11, 1 // TOTAL-1 + s_min_u32 s18, s18, s36 // t clamped to [0,TOTAL-1] -> mblk in-bounds) +.endif + s_mul_hi_u32 s19, s18, s12 // mblk = t / NTL + s_mul_i32 s36, s19, s13 // mblk * NTL + s_sub_u32 s30, s18, s36 // tcol = t - mblk*NTL +.endm + +// RESIDENT B FRAG LAYOUT: B frag (kstep ks, frag f) at BRES_OFF + (ks*FN + f)*256 +// (each frag = the SAME 256B block coop stores per B-ring slot; lane*8 vaddr base = v9). +// Built here as: dst vbase = v9 + BRES_OFF + f*256 , ds_store offset:(ks*FN*256). +// B global addr (lift coop B-feed): Bshuf + tcol*(FN*256=s14) + (seg k0)* [ksi*KSEG_STEPS*(NT*256=s10)] +// + f*256 (frag, folded into saddr) + ks*(NT*256=s10) (k-step, folded into saddr). +.macro BSTAGE // in: s30=tcol s31=ksi ; clob: s20,s21,s23,s25,s26,s27,v13,v[BSTG..] + s_mul_i32 s20, s30, s14 // tcol * FN*256 + s_mul_i32 s21, s31, KSEG_STEPS // ksi * KSEG_STEPS + s_mul_i32 s21, s21, s10 // * NT*256 -> segment k-start byte offset + s_add_u32 s20, s20, s21 + s_add_u32 s20, s4, s20 + s_addc_u32 s21, s5, 0 // s[20:21] = B base (tcol,ksi, seg k-step 0) +.Lbcl\@: + lds_fetch_add s23, BFRAG_NEXT_OFF, 1 // claim frag f + s_cmp_ge_u32 s23, FN + s_cbranch_scc1 .Lbsd\@ // f>=FN -> all frags claimed + s_lshl_b32 s25, s23, 8 // f*256 + s_add_u32 s26, s20, s25 + s_addc_u32 s27, s21, 0 // s[26:27] = frag f base (seg k0) + v_add_nc_u32 v13, v9, BRES_OFF + v_add_nc_u32 v13, v13, s25 // resident B dst vbase for frag f + .set ks, 0 + .rept KSEG_STEPS + global_load_tr_b64 v[BSTG+ks*2:BSTG+ks*2+1], v9, s[26:27] + s_add_u32 s26, s26, s10 // next k-step (last iter over-advances; unused) + s_addc_u32 s27, s27, 0 + .set ks, ks+1 + .endr + s_wait_loadcnt 0x0 + .set ks, 0 + .rept KSEG_STEPS + ds_store_b64 v13, v[BSTG+ks*2:BSTG+ks*2+1] offset:(ks*FN*256) + .set ks, ks+1 + .endr + s_wait_dscnt 0x0 + lds_inc BFRAG_DONE_OFF // frag f STORED -> publish completion (compute gates on this) + s_branch .Lbcl\@ +.Lbsd\@: +.endm + +// RESIDENT A FRAG LAYOUT: A frag (kstep ks, rowblk r, mi) at ARES_OFF + ((ks*G + r)*FM + mi)*256 +// Built as: dst vbase = v9 + ARES_OFF + r*(FM*256) , ds_store offset:((ks*G*FM + mi)*256). +// A global addr (lift coop compute/A-feed): A + rowblk_abs*(16*FM)*K + mi*16*K + koff, rowblk_abs=mblk*G+r, +// koff = ksi*SEGK (segment K byte offset, fp8 1B/elem), k-step within segment via global offset:ks*16. +.macro ASTAGE // in: s19=mblk s31=ksi ; clob: s22,s23,s25,s32,s36,s40,s41,s44,s45,v13,v[BSTG..] + s_lshl_b32 s32, s9, 4 // rowstride16 = 16*K +.Lacl\@: + lds_fetch_add s23, AROW_NEXT_OFF, 1 // claim rowblk r + s_cmp_ge_u32 s23, G + s_cbranch_scc1 .Lasd\@ + s_mul_i32 s36, s19, G + s_add_u32 s36, s36, s23 // rowblk_abs = mblk*G + r + s_mul_i32 s22, s36, (16*FM) + s_mul_i32 s22, s22, s9 // rowblk_abs*(16*FM)*K + s_mul_i32 s25, s31, SEGK // ksi*SEGK (segment K byte offset) + s_add_u32 s22, s22, s25 + s_add_u32 s40, s2, s22 + s_addc_u32 s41, s3, 0 // s[40:41] = A base (rowblk_abs, mi0, seg k0) + s_mul_i32 s25, s23, (FM*256) // r*FM*256 + v_add_nc_u32 v13, v9, ARES_OFF + v_add_nc_u32 v13, v13, s25 // resident A dst vbase for rowblk r + .set mi, 0 + .rept FM + .if mi == 0 + s_mov_b32 s44, s40 + s_mov_b32 s45, s41 + .else + s_add_u32 s44, s44, s32 // += 16*K (next M-frag) + s_addc_u32 s45, s45, 0 + .endif + .set ks, 0 + .rept KSEG_STEPS + global_load_b64 v[BSTG:BSTG+1], v8, s[44:45] offset:(ks*16) + s_wait_loadcnt 0x0 + ds_store_b64 v13, v[BSTG:BSTG+1] offset:((ks*G*FM + mi)*256) + s_wait_dscnt 0x0 + .set ks, ks+1 + .endr + .set mi, mi+1 + .endr + lds_inc AROW_DONE_OFF // rowblk r fully STAGED -> publish completion + s_branch .Lacl\@ +.Lasd\@: +.endm + +// ============================================================================================ +// FIX 1a -- RING staging macros: slot-indexed BSTAGE_R / ASTAGE_R. Identical math to BSTAGE/ASTAGE +// but claim/done counters live in the per-slot control block (\scb = SLOTC_BASE + slot*32, runtime) +// and operands land in the per-slot buffer (\sob = OP_BASE + slot*OPSTRIDE, runtime; B at +// +BRES_ROFF=0, A at +ARES_ROFF). \scb and \sob are READ-only (never clobbered). Internal address +// scratch: s46/s47 (free in the feed context). ds offset immediates are vbase-relative -> unchanged. +// ============================================================================================ +.macro BSTAGE_R scb, sob // in: s30=tcol s31=ksi ; clob: s20,s21,s23,s25,s26,s27,s46,s47,v13,v[BSTG..] + s_mul_i32 s20, s30, s14 // tcol * FN*256 + s_mul_i32 s21, s31, KSEG_STEPS // ksi * KSEG_STEPS + s_mul_i32 s21, s21, s10 // * NT*256 -> segment k-start byte offset + s_add_u32 s20, s20, s21 + s_add_u32 s20, s4, s20 + s_addc_u32 s21, s5, 0 // s[20:21] = B base (tcol,ksi, seg k-step 0) + s_add_u32 s46, \scb, SL_BFNEXT // &SL_BFNEXT[slot] +.Lbclr\@: + lds_fetch_add_r s23, s46, 1 // claim frag f + s_cmp_ge_u32 s23, FN + s_cbranch_scc1 .Lbsdr\@ // f>=FN -> all frags claimed + s_lshl_b32 s25, s23, 8 // f*256 + s_add_u32 s26, s20, s25 + s_addc_u32 s27, s21, 0 // s[26:27] = frag f base (seg k0) + v_add_nc_u32 v13, v9, \sob // + slot operand base + v_add_nc_u32 v13, v13, s25 // + f*256 (BRES_ROFF = 0) + .set ks, 0 + .rept KSEG_STEPS + global_load_tr_b64 v[BSTG+ks*2:BSTG+ks*2+1], v9, s[26:27] + s_add_u32 s26, s26, s10 + s_addc_u32 s27, s27, 0 + .set ks, ks+1 + .endr + s_wait_loadcnt 0x0 + .set ks, 0 + .rept KSEG_STEPS + ds_store_b64 v13, v[BSTG+ks*2:BSTG+ks*2+1] offset:(ks*FN*256) + .set ks, ks+1 + .endr + s_wait_dscnt 0x0 + s_add_u32 s47, \scb, SL_BFDONE + lds_inc_r s47 // frag f STORED -> compute gates on SL_BFDONE==FN + s_branch .Lbclr\@ +.Lbsdr\@: +.endm + +.macro ASTAGE_R scb, sob // in: s19=mblk s31=ksi ; clob: s22,s23,s25,s32,s36,s40,s41,s44,s45,s46,s47,v13,v[BSTG..] + s_lshl_b32 s32, s9, 4 // rowstride16 = 16*K + s_add_u32 s46, \scb, SL_ARNEXT // &SL_ARNEXT[slot] +.Laclr\@: + lds_fetch_add_r s23, s46, 1 // claim rowblk r + s_cmp_ge_u32 s23, G + s_cbranch_scc1 .Lasdr\@ + s_mul_i32 s36, s19, G + s_add_u32 s36, s36, s23 // rowblk_abs = mblk*G + r + s_mul_i32 s22, s36, (16*FM) + s_mul_i32 s22, s22, s9 // rowblk_abs*(16*FM)*K + s_mul_i32 s25, s31, SEGK // ksi*SEGK + s_add_u32 s22, s22, s25 + s_add_u32 s40, s2, s22 + s_addc_u32 s41, s3, 0 // s[40:41] = A base + s_mul_i32 s25, s23, (FM*256) // r*FM*256 + v_add_nc_u32 v13, v9, \sob // + slot operand base + v_add_nc_u32 v13, v13, ARES_ROFF // + A-within-slot offset (BRES_BYTES) + v_add_nc_u32 v13, v13, s25 // + r*FM*256 + .set mi, 0 + .rept FM + .if mi == 0 + s_mov_b32 s44, s40 + s_mov_b32 s45, s41 + .else + s_add_u32 s44, s44, s32 // += 16*K (next M-frag) + s_addc_u32 s45, s45, 0 + .endif + .set ks, 0 + .rept KSEG_STEPS + global_load_b64 v[BSTG:BSTG+1], v8, s[44:45] offset:(ks*16) + s_wait_loadcnt 0x0 + ds_store_b64 v13, v[BSTG:BSTG+1] offset:((ks*G*FM + mi)*256) + s_wait_dscnt 0x0 + .set ks, ks+1 + .endr + .set mi, mi+1 + .endr + s_add_u32 s47, \scb, SL_ARDONE + lds_inc_r s47 // rowblk r STAGED -> compute gates on SL_ARDONE==G + s_branch .Laclr\@ +.Lasdr\@: +.endm + +// ============================================================================================ +// Phase-B (DSWS2_CONV) consume-point ring-occupancy sensor -- Task 3, READ-ONLY (actuation is Task 5). +// Mirrors the coop occ_a/occ_b sensor (occ = producer - consumer, sampled where the value is +// CONSUMED, not at the segment boundary). The claimer's A7 wait-done spin runs CONCURRENTLY with the +// compute drain, so it observes the ring mid-flight; at the segment boundary the resident region has +// fully drained and occ would read a stuck ~0 (permanent false-starvation) -- exactly what SPEC warns. +// +// COUNTER IDENTITIES (confirmed against the live claim/consume sites -- see report): +// producer = the STORE-completion counters the compute wave actually gates on: +// A-ring: AROW_DONE_OFF (A rowblks resident, monotonic in [0,G]; lds_inc @ ASTAGE) +// B-ring: BFRAG_DONE_OFF (B frags resident, monotonic in [0,FN]; lds_inc @ BSTAGE) +// NOT the *_NEXT claim counters: AROW_NEXT/BFRAG_NEXT overshoot the ring depth by the role +// terminal-bails (G+NAFEED / FN+NBFEED), which would break the occ <= depth bound. +// consumer = ROWBLK_NEXT_OFF, the compute rowblk-claim clock (consume progress through the super-tile: +// each claimed rowblk r consumes A(r) and re-reads all FN shared B frags). +// min-clamp: cons is clamped to prod before the subtract so the u32 result cannot underflow when the +// consume clock outruns a shallower ring (G=6 > FN=4 -> ROWBLK_NEXT can exceed BFRAG_DONE). +// INVARIANT preserved: occ_A in [0,G], occ_B in [0,FN] (nonnegative, bounded by ring depth). +// +// REGISTER DISCIPLINE (brick-critical; this path is reachable pre-grow -- a >v15 vector temp is +// OOR-poison under dyn-VGPR, SPEC S4): scalars <= s65 only (s60/s61 scratch; callers pass dst in +// [s62,s65]); the only vector temps are inside lds_get, which uses v11/v14 (INTERIOR to the launch +// 16-VGPR block) -- NO >v15 temp is introduced here. +// GATE: DSWS2_CONV || DSWS2_ENVELOPE. reserve_try + the BUDGET default are the pool-economy primitives the +// rolling envelope needs INDEPENDENTLY of role conversion (they touch only VRESV_OFF/lds_fetch_add), so the +// envelope must be able to run at CONV=0 (the isolation config). Everything in this block is macro/.set +// definition (emits ZERO bytes), so widening the gate is byte-identical at CONV=0/ENV=0 and CONV=1. +.if DSWS2_CONV || DSWS2_ENVELOPE +.macro occ_sample dst_a, dst_b // out: \dst_a=occ_A in [0,G], \dst_b=occ_B in [0,FN]; clob s60,s61 + lds_get \dst_a, AROW_DONE_OFF // prod_a: A rowblks resident (store-completion) + lds_get \dst_b, BFRAG_DONE_OFF // prod_b: B frags resident (store-completion) + lds_get s60, ROWBLK_NEXT_OFF // cons : compute rowblk-claim consume clock + s_min_u32 s61, s60, \dst_a // cons_a = min(clock, prod_a) (clamp -> no u32 underflow) + s_sub_u32 \dst_a, \dst_a, s61 // occ_A = prod_a - cons_a in [0,G] + s_min_u32 s61, s60, \dst_b // cons_b = min(clock, prod_b) + s_sub_u32 \dst_b, \dst_b, s61 // occ_B = prod_b - cons_b in [0,FN] +.endm + +// ---- DSWS2_GQUIESCE: device-scoped GLOBAL QUIESCE handshake (mirrors the green occ[20]/occ[0] pattern). +// All three ops are lane-0-masked (v2==0), exec saved/restored via s49 (the LDS-macro convention -- s49 is +// never live across a macro boundary, so it is provably free at every site these replace an lds_* op). +// vaddr = v4 (the stable occ-base per-lane offset, =0, prologue-set), data/dst = v3/v5 (occ scratch vregs, +// same as the claim/live ops). scope:SCOPE_DEV + uncached occ buffer => device-coherent visibility (the +// fix). s_wait_storecnt/loadcnt drain before proceeding so the poll observes committed bumps. +.macro gq_reset // claimer: occ[QUIESCE_GOFF] = 0 (committed before EPOCH publish) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lgqr_skip\@ + v_mov_b32 v3, 0 + global_store_b32 v4, v3, s[0:1] offset:QUIESCE_GOFF scope:SCOPE_DEV + s_wait_storecnt 0x0 +.Lgqr_skip\@: + s_mov_b32 exec_lo, s49 +.endm +.macro gq_bump // follower: occ[QUIESCE_GOFF] += 1 (one bump/wave/super-tile) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lgqb_skip\@ + v_mov_b32 v3, 1 + global_atomic_add_u32 v4, v3, s[0:1] offset:QUIESCE_GOFF scope:SCOPE_DEV + s_wait_storecnt 0x0 +.Lgqb_skip\@: + s_mov_b32 exec_lo, s49 +.endm +.macro gq_read dst // claimer: \dst = occ[QUIESCE_GOFF] (lane0 load + broadcast) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lgqrd_skip\@ + global_load_b32 v5, v4, s[0:1] offset:QUIESCE_GOFF scope:SCOPE_DEV + s_wait_loadcnt 0x0 +.Lgqrd_skip\@: + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 \dst, v5 +.endm + +// ---- Pool-T7 chunk-2 wedge localization (DIAG-only; DSWS2_CONV=0 emits nothing -> .text byte-identical). +// epoch_mark: lane-0 publishes this role's live epoch (s35) to a host-streamed occ slot so a hung dispatch +// shows how far each role advanced (stream field roles[C/A/B]). v14<=v15 (feeds/compute are lean-32 at the +// _quiesce call sites), v4=0 (occ base lane offset, prologue), s49 exec-save (LDS-macro convention). ---- +.macro epoch_mark off +.if DSWS2_CONV && DIAG + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lem_skip\@ + v_mov_b32 v14, s35 + global_store_b32 v4, v14, s[0:1] offset:\off scope:SCOPE_DEV +.Lem_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// bail_mark: PER-WAVE localization mark. Lane-0 writes this wave's epoch (s35) to occ[BAIL_BASE + wid*4] +// (runtime vaddr since the offset depends on wid=s24). s48 scratch, s49 exec-save (macro-local; free at the +// _quiesce bail sites), v13 vaddr, v14 data (both <=v15; the wave is lean-32 at every bail site). One-shot +// per super-tile -> negligible perturbation vs the DIAG per-spin claimer stores. Enabled by DIAG OR BAILMARK. +.macro bail_mark +.if DSWS2_CONV && (DIAG || DSWS2_BAILMARK) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lbmk_skip\@ + s_lshl_b32 s48, s24, 2 // wid*4 + s_add_u32 s48, s48, BAIL_BASE // occ byte offset for THIS wave + v_mov_b32 v13, s48 // vaddr = per-wave byte offset (lane0) + v_mov_b32 v14, s35 // data = this wave's current epoch + global_store_b32 v13, v14, s[0:1] scope:SCOPE_DEV +.Lbmk_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// -------------------------------------------------------------------------------------------- +// Phase-B controller thresholds + sum-envelope budget (Task 4). EPOCH_SHIFT mirrors coop / +// occ_dispatch (epoch = segcnt >> EPOCH_SHIFT). BUDGET is the per-WG VGPR sum-envelope ceiling +// the reservation counter must never exceed; default = the launch reservation, which makes the +// envelope a strict conservation law (a feed->compute grow can only fit if a compute->feed shrink +// already freed the delta). Task 5 may re-tune via `-defsym BUDGET=` if per-SIMD headroom exists. +// -------------------------------------------------------------------------------------------- +.ifndef EPOCH_SHIFT + .set EPOCH_SHIFT, 3 // decision clock: epoch = segcnt >> EPOCH_SHIFT (small = reactive) +.endif +.ifndef VBUDGET + .set VBUDGET, 1536 // physical VGPR-file credit ceiling (R9700 wave32, per SIMD). Calibrate. +.endif // Sanity ceiling only: the hardware s_alloc_vgpr is the real concurrent-fat cap. +.ifndef BUDGET +.if DSWS2_ENVELOPE + .set BUDGET, (WAVES*VLEAN + PEAK_CONC*(NFV-VLEAN)) // rolling: lean floor + concurrent-peak headroom +.else + .set BUDGET, VBUDGET // EMERGENT: budget is PHYSICAL, not mix-derived (ledger is dormant; conv-only). +.endif +.endif + +// emergent-economy PHYSICAL sanity (always on): all waves fit lean, and >=1 can grow. +.if (WAVES * VLEAN) > BUDGET + .error "WAVES*VLEAN exceeds VBUDGET -- pool cannot stay all-lean" +.endif +.if (WAVES*VLEAN + (NFV-VLEAN)) > BUDGET + .error "VBUDGET admits < 1 concurrent grow -- compute can never make progress" +.endif + +// try_gate: the lock-free single-winner conversion ticket (transcribed VERBATIM from occ_kernel_coop.s, +// which transcribes dsws_ctrl_model.cpp gate_try_win + epoch_of EXACTLY). E = segcnt>>EPOCH_SHIFT. +// gate[dir] holds the last epoch dir fired. Among many waves racing the same (gtry_gate->reserve_try point -- +// occ_sample's s62/s63 result is consumed into `dir` BEFORE this runs), v5/v6/v7 (<=v15: pre-grow / +// lean-safe). CAS operand order (gfx1201, GCN order -- NOT flipped, KG 9ed04f3c): +// ds_cmpstore_rtn_b32 vdst,vaddr,vNEW,vCMP -> MEM=(MEM==vCMP)?vNEW:MEM, vdst<-old. So vsrc0=E (new), +// vsrc1=g (compare). WIN iff returned-old == g. (Swapping them leaves gate stuck so old==g for ALL +// racers -> every racer "wins" -> would-win ~= NCOMP*epochs instead of ~= epochs.) +.macro try_gate dir, swin + lds_get s62, SEGCNT_OFF // E = epoch_of(segcnt, EPOCH_SHIFT) + s_lshr_b32 s62, s62, EPOCH_SHIFT + lds_get s63, (GATE_OFF + (\dir)*4) // g = gate[dir] + s_mov_b32 \swin, 0 + s_cmp_ge_u32 s63, s62 // g >= E -> dir already fired this/later epoch -> lose + s_cbranch_scc1 .Ltg_done\@ + s_mov_b32 s65, exec_lo // lane0-only CAS (one ticket attempt per WAVE) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltg_restore\@ + v_mov_b32 v5, (GATE_OFF + (\dir)*4) // vaddr = &gate[dir] + v_mov_b32 v6, s63 // v6 = g (vsrc1 = CMP/expected) + v_mov_b32 v7, s62 // v7 = E (vsrc0 = NEW value to store) + ds_cmpstore_rtn_b32 v6, v5, v7, v6 // gate[dir] = (gate[dir]==g) ? E : gate[dir]; v6 <- old + s_wait_dscnt 0x0 +.Ltg_restore\@: + s_mov_b32 exec_lo, s65 + v_readfirstlane_b32 s64, v6 // s64 = old (lane0's CAS result, broadcast) + s_cmp_eq_u32 s64, s63 // WIN iff old == g (we were the swapper) + s_cbranch_scc0 .Ltg_done\@ + s_mov_b32 \swin, 1 +.Ltg_done\@: +.endm + +// reserve_try: the VGPR sum-envelope reservation (transcribes reserve_grow, dsws_ctrl_model.cpp:47). +// Reserve first (atomic add of SIGNED \delta on vgpr_reserved), then validate prev+delta <= BUDGET; +// on over-budget cleanly UNDO (atomic add of -\delta) and reject. The LDS atomic serializes the <=2 +// concurrent grows an epoch permits: the second to validate sees the first's reservation and backs off. +// GROW (feed->compute): pass \delta = +(NFV-VLEAN). Over-budget -> undo, \won=0 (stay in role). +// SHRINK(compute->feed): pass \delta = -(NFV-VLEAN). new = prev+delta < prev <= BUDGET, so the +// validate branch is a proven no-op -> \won=1 ALWAYS (shrink never fails). +// One macro, one call site (Task 5 `reserve_try delta, s_ok`); direction is the sign of \delta. +// Scratch: s62/s63 (free at the bail-commit point -- try_gate's s62..s65 are long dead by then). +.macro reserve_try delta, won + lds_fetch_add s62, VRESV_OFF, (\delta) // s62 = prev reserved; vgpr_reserved += delta + s_add_u32 s63, s62, (\delta) // s63 = new reservation = prev + delta + s_mov_b32 \won, 1 + s_cmp_le_u32 s63, BUDGET // new <= BUDGET -> commit (win); shrink always passes + s_cbranch_scc1 .Lrt_done\@ + lds_fetch_add s62, VRESV_OFF, -(\delta) // over-budget: undo the reservation, reject + s_mov_b32 \won, 0 +.Lrt_done\@: +.endm + +// -------------------------------------------------------------------------------------------- +// Phase-B (Task 5) watermark thresholds + LDS put-runtime helper + bail-time commit macros. +// Watermark decision (SPEC; mirrors coop CTRL_LOW/CTRL_HIGH, occ_dispatch DSWS_LOW/HIGH): +// occ_X < CTRL_LOW -> compute STARVED for X -> shrink a compute wave into feed-X. +// occ_X > CTRL_HIGH_X -> feed-X OVER-SERVING -> grow a feed-X wave into compute. +// occ_A in [0,G], occ_B in [0,FN] (occ_sample bounds), so the HIGH marks are per-ring-depth. +// -------------------------------------------------------------------------------------------- +.ifndef CTRL_LOW + .set CTRL_LOW, 1 // occ_X < 1 (== 0, ring empty at consume) -> starved +.endif +.ifndef CTRL_HIGH_A + .set CTRL_HIGH_A, (G-1) // occ_A > G-1 -> A-ring saturated -> A-feed over-serving +.endif +.ifndef CTRL_HIGH_B + .set CTRL_HIGH_B, (FN-1) // occ_B > FN-1 -> B-ring saturated -> B-feed over-serving +.endif + +// lds_put_r: lane-0-of-wave write scalar \ssrc -> LDS[\saddr] (RUNTIME byte offset in a sreg). Mirrors +// the coop lds_put_v idiom but takes a SCALAR address (symmetry with lds_get_r). Used by the claimer's +// Step-4 snapshot write into the runtime parity half of SNAP_BASE. Temps RP_A/RP_D are v11/v14 (<=v15, +// pre-grow safe); s49 is the exec save (matches lds_put). +.macro lds_put_r saddr, ssrc + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lputr_skip\@ + v_mov_b32 v[RP_A], \saddr + v_mov_b32 v[RP_D], \ssrc + ds_store_b32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lputr_skip\@: + s_mov_b32 exec_lo, s49 +.endm + +// conv_dec_floor: floor-guarded ATOMIC decrement of a role slot -- \ok <- 1 iff it decremented \slot_off +// (only when the current value was > 1), else 0 (floor hit; source role must keep >= 1 wave). A +// ds_cmpstore_rtn_b32 CAS loop (re-reads on a lost race), so two same-source converters in one epoch +// (e.g. compute->Afeed and compute->Bfeed both dec NCOMP_SLOT) can never drive the slot below 1. +// Clob: s52 (read value), s53 (new/CAS-return), s65 (exec save); v5/v6/v7 (<=v15, pre-grow safe). +.macro conv_dec_floor slot_off, ok + s_mov_b32 \ok, 0 +.Lcdf_retry\@: + lds_get s52, \slot_off // s52 = current source-slot count + s_cmp_le_u32 s52, 1 + s_cbranch_scc1 .Lcdf_done\@ // <=1 -> at floor, cannot convert away (ok stays 0) + s_sub_u32 s53, s52, 1 // new = old - 1 + s_mov_b32 s65, exec_lo // lane0-only CAS (one attempt per WAVE) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lcdf_restore\@ + v_mov_b32 v5, \slot_off // vaddr = &slot + v_mov_b32 v6, s52 // v6 = expected old (CMP) + v_mov_b32 v7, s53 // v7 = new value (NEW) + ds_cmpstore_rtn_b32 v6, v5, v7, v6 // slot = (slot==old)? new : slot ; v6 <- prior + s_wait_dscnt 0x0 +.Lcdf_restore\@: + s_mov_b32 exec_lo, s65 + v_readfirstlane_b32 s53, v6 // s53 = prior (lane0 CAS result, broadcast) + s_cmp_eq_u32 s53, s52 // success iff prior == expected (we were the swapper) + s_cbranch_scc0 .Lcdf_retry\@ // lost the race -> re-read and retry + s_mov_b32 \ok, 1 +.Lcdf_done\@: +.endm + +// conv_apply: the bail-time role-conversion COMMIT (SPEC 3.4 Approach A). Precondition: s58 = s_win +// (1 iff this wave won the (dir,epoch) ticket). Ordered strictly BEFORE the QUIESCE_CNT bump the +// CALLER emits after this macro (the quiesce counter is the snapshot handshake). +// ORDER: (a) floor-guarded dec of \src_slot -> (b) reserve the VGPR sum-envelope \delta (shrink +// always ok; grow may abort over BUDGET) -> (c) on ok: inc \dst_slot, flip private role reg (s59), +// s_alloc_vgpr \alloc_sz (GROW=NFV feed->compute / SHRINK=32 compute->feed) with SCC-retry -> +// (d) on floor-fail or reserve-abort: cancel, remain current role (undo the source dec if a +// reservation abort happened after the dec). +// PRE-GROW OOR WINDOW (SPEC 4, #1 brick risk): the wave is lean-32 on entry; every LDS/atomic temp +// read before the s_alloc_vgpr GROW is <=v15 (occ_sample/try_gate v5/v6/v7 + v11/v14; conv_dec_floor +// v5/v6/v7; lds_fetch_add v11/v14) and every carried scalar is <=s65. NO >v15 source before GROW. +// Clob: s52,s53,s54 (+ conv_dec_floor / reserve_try scratch); s59 = new role slot id (record). +.macro conv_apply src_slot, dst_slot, delta, alloc_sz + s_cmp_eq_u32 s58, 0 + s_cbranch_scc1 .Lca_skip\@ // lost the ticket -> no conversion this bail + conv_dec_floor \src_slot, s54 // (a) floor-guarded atomic dec of source slot + s_cmp_eq_u32 s54, 0 + s_cbranch_scc1 .Lca_skip\@ // floor-fail (source at 1) -> cancel, remain current role + reserve_try (\delta), s53 // (b) reserve VGPR envelope (grow may abort; shrink ok) + s_cmp_eq_u32 s53, 0 + s_cbranch_scc0 .Lca_commit\@ + lds_fetch_add s52, \src_slot, 1 // (d) reserve aborted: UNDO the source dec, cancel + s_branch .Lca_skip\@ +.Lca_commit\@: + lds_fetch_add s52, \dst_slot, 1 // (c) inc dest slot (unbounded -> plain atomic add) + s_mov_b32 s59, \dst_slot // flip private current-role reg (records new role slot id) +.if DIAG || TRACE + // conversion-commit counter (proves a wave ACTUALLY switched role). Lean-32 pre-grow here -> v3/v4<=v15 + // OOR-safe; s49 exec-save (macro-local). (DIAG||TRACE)-gated -> DSWS2_CONV/DIAG=0/TRACE=0 byte-identical. + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lca_cm_skip\@ + v_mov_b32 v3, 1 + global_atomic_add_u32 v4, v3, s[0:1] offset:CONVCNT_OFF scope:SCOPE_DEV // occ[48] += 1 + s_wait_storecnt 0x0 +.Lca_cm_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.if CONV_COOLDOWN > 0 + s_mov_b32 s66, CONV_COOLDOWN // Task 4: committed conversion -> arm cooldown +.endif + // ---- s_alloc_vgpr resize: THE pre-grow OOR window closes here; all reads above were <=v15 ---- +.Lca_alloc\@: + s_alloc_vgpr \alloc_sz // GROW(NFV) / SHRINK(32); SCC-retry (brick-class rule) + s_cbranch_scc0 .Lca_alloc\@ +.Lca_skip\@: +.endm +.endif + +// ============================================================================================ +// TFPROBE wall-span capture (TF throughput probe). Realtime-tick min/max into occ[2]/occ[3], +// mirroring occ_kernel_coop.s's proven timer idiom. Each wave stamps lane-0 only (exec-masked +// via s49, the DSWS exec-save convention); base addr v4==0 holds kernel-wide (prologue, line +// ~765; invariant per the "v4=0 occ base lane offset" note). s[30:31] free at entry (DECODE_STI +// outputs, computed only inside role bodies) and dead at every terminal. Emits ZERO bytes at +// TFPROBE=0 -> production .text byte-identical. +// ============================================================================================ +.macro tfspan op:req, off:req // op = min (entry, occ[2]/off 8) | max (exit, occ[3]/off 12) +.if TFPROBE + s_sendmsg_rtn_b64 s[30:31], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + v_cmp_eq_u32 vcc_lo, 0, v2 // lane 0 of each wave only (v2 = tid & 31) + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltfspan_skip\@ + v_mov_b32 v5, s30 // low 32 bits of the realtime tick + global_atomic_\op\()_u32 v4, v5, s[0:1] offset:\off scope:SCOPE_DEV +.Ltfspan_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// ============================================================================================ +// PHASEPROBE: in-kernel per-phase tick timer for the COMPUTE wave (the critical path). s77 holds +// this wave's last-stamp RTC (low 32b). phase_reset seeds it; phase_stamp accumulates (now-s77) +// into occ[\off] and re-seeds. Lane-0-only atomic add; occ slots are ABOVE the per-chunk memset +// so they sum over the whole run. Scratch s62/s63 (RTC), s64 (delta) -- all free in CONV=0 compute. +// Six phases -> six occ accumulators (bytes 256..276): +.set PH_FOLLOW_OFF, 256 // occ[64]: waiting on claimer to publish next super-tile +.set PH_STAGE_OFF, 260 // occ[65]: waiting on A/B feeds to stage this super-tile +.set PH_GROW_OFF, 264 // occ[66]: claim rowblk + s_alloc_vgpr GROW 32->112 +.set PH_WMMA_OFF, 268 // occ[67]: LDS frag loads + v_wmma compute +.set PH_FLUSH_OFF, 272 // occ[68]: global_atomic_add_f32 C flush (split-K reduction) +.set PH_SHRINK_OFF, 276 // occ[69]: s_alloc_vgpr SHRINK 112->32 +// ---- STAGINSTR: lightweight write-once diagnostic counters (is the wall feed or compute?). Lane0 +// atomic at branch points ONLY (never inside the WMMA timing region -> no s_wait_storecnt pollution). +// STAGINSTR=0 => emits nothing => byte-identical. Ratio COAST/(COAST+COMP) = compute-wave feed-starve. +.ifndef STAGINSTR + .set STAGINSTR, 0 +.endif +.set STINSTR_COAST, 280 // occ[70]: compute-wave loop iters spent coasting (feed-starved) +.set STINSTR_COMP, 284 // occ[71]: rowblk-segments actually computed + reduced +.set STINSTR_FEED, 288 // occ[72]: feed stage completions (staging throughput) +.set STINSTR_GROWFAIL, 292 // occ[73]: per-burst grow SCC0 (budget full -> coast) = stagger repulsion events +.macro instr_inc off +.if STAGINSTR + s_mov_b32 s57, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 // lane0 of the wave only + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lii_skip\@ + v_mov_b32 v3, 1 + global_atomic_add_u32 v4, v3, s[0:1] offset:\off scope:SCOPE_DEV // v4=occ base vaddr(0), v3=1 +.Lii_skip\@: + s_mov_b32 exec_lo, s57 +.endif +.endm +// Per-wave phase accumulators live in SGPRs s78..s83 (NO per-stamp store -> zero memory perturbation, no +// s_wait_storecnt pollution). s77 = last-stamp RTC. phase_flush emits them ONCE at compute retire. +.macro phase_reset +.if PHASEPROBE + s_mov_b32 s78, 0 + s_mov_b32 s79, 0 + s_mov_b32 s80, 0 + s_mov_b32 s81, 0 + s_mov_b32 s82, 0 + s_mov_b32 s83, 0 + s_sendmsg_rtn_b64 s[62:63], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + s_mov_b32 s77, s62 +.endif +.endm +.macro phase_stamp acc:req // \acc += (now - s77); s77 = now (pure scalar, no store) +.if PHASEPROBE + s_sendmsg_rtn_b64 s[62:63], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + s_sub_u32 s64, s62, s77 // delta (mod 2^32; phase << 2^32 so wrap-safe) + s_mov_b32 s77, s62 + s_add_u32 \acc, \acc, s64 +.endif +.endm +.macro phase_flush // lane0 atomic-adds s78..s83 -> occ[64..69]; drained here (not the hot loop) +.if PHASEPROBE + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lphf_skip\@ + v_mov_b32 v5, s78 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_FOLLOW_OFF scope:SCOPE_DEV + v_mov_b32 v5, s79 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_STAGE_OFF scope:SCOPE_DEV + v_mov_b32 v5, s80 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_GROW_OFF scope:SCOPE_DEV + v_mov_b32 v5, s81 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_WMMA_OFF scope:SCOPE_DEV + v_mov_b32 v5, s82 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_FLUSH_OFF scope:SCOPE_DEV + v_mov_b32 v5, s83 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_SHRINK_OFF scope:SCOPE_DEV + s_wait_storecnt 0x0 +.Lphf_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// ============================================================================================ +// TRACE: per-super-tile time-series row (claimer, lane 0). Written once per super-tile at the +// quiesce-satisfied drain-exit (.Lqc_q_ok). Captures the adaptive wave-role economy over time: +// the LIVE role slots (do waves convert?), the per-super-tile ring-occupancy PEAK (s73/s74, +// tracked across the wait_done spins), the cumulative conversion count, and the envelope vresv. +// 16 u32/row -> buffer[segcnt*64]; bounded by MAXROWS (s72). Emits ZERO bytes at TRACE=0. +// Persistent trace regs: s70:s71 = buffer VA, s72 = MAXROWS, s73/s74 = ring occA/occB peak. +// ============================================================================================ +.macro alllive_dec // TRACE: --live on wave exit (pairs with the entry ++ for peak-concurrent) +.if TRACE + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lalld_skip\@ + v_mov_b32 v3, -1 + global_atomic_add_u32 v4, v3, s[0:1] offset:ALLLIVE_OFF scope:SCOPE_DEV +.Lalld_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +.macro trace_row +.if TRACE + // claim a GLOBALLY-unique row index (all WGs' claimers share occ[55]) -> no per-WG SEGCNT collision. + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + v_mov_b32 v14, 1 + global_atomic_add_u32 v15, v4, v14, s[0:1] offset:TRACE_IDX_OFF th:TH_ATOMIC_RETURN scope:SCOPE_DEV // v15=old idx, v4=addr(0), v14=data(1) + s_wait_loadcnt 0x0 + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 s52, v15 // s52 = unique row index (old value returned by the atomic) + s_cmp_ge_u32 s52, s72 // row >= MAXROWS -> skip (buffer bound) + s_cbranch_scc1 .Ltrow_skip\@ + s_lshl_b32 s53, s52, 6 // row * TRACE_ROW_BYTES(64) + s_add_u32 s60, s70, s53 + s_addc_u32 s61, s71, 0 // s[60:61] = row base VA + s_sendmsg_rtn_b64 s[58:59], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + lds_get s55, SEGCNT_OFF // this WG's per-WG super-tile counter (data field, not index) + v_mov_b32 v14, s58 // 0 tick_lo + v_mov_b32 v15, s55 // 1 segcnt (per-WG) + v_mov_b32 v16, s35 // 2 epoch + lds_get s54, NCOMP_SLOT + v_mov_b32 v17, s54 // 3 nComp (live role slot) + lds_get s54, NAFEED_SLOT + v_mov_b32 v18, s54 // 4 nAfeed + lds_get s54, NBFEED_SLOT + v_mov_b32 v19, s54 // 5 nBfeed + v_mov_b32 v20, s73 // 6 occA peak (across wait_done spins) + v_mov_b32 v21, s74 // 7 occB peak + global_load_b32 v22, v4, s[0:1] offset:CONVCNT_OFF scope:SCOPE_DEV // 8 convCount (cumulative) + lds_get s54, VRESV_OFF + v_mov_b32 v23, s54 // 9 vresv (envelope budget) + v_mov_b32 v24, s17 // 10 sti (claimed super-tile id) + lds_get s54, QUIESCE_CNT_OFF + v_mov_b32 v25, s54 // 11 quiesce (final) + v_mov_b32 v26, s59 // 12 tick_hi + v_mov_b32 v27, s69 // 13 chunkHi (context) + v_mov_b32 v28, s75 // 14 wg_id (which workgroup's economy this row belongs to) + v_mov_b32 v29, 0 // 15 reserved + s_wait_loadcnt 0x0 // convCount load drained before the row store + v_cmp_eq_u32 vcc_lo, 0, v2 // lane 0 of the claimer writes the row + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltrow_wskip\@ + global_store_b128 v4, v[14:17], s[60:61] offset:0 scope:SCOPE_DEV + global_store_b128 v4, v[18:21], s[60:61] offset:16 scope:SCOPE_DEV + global_store_b128 v4, v[22:25], s[60:61] offset:32 scope:SCOPE_DEV + global_store_b128 v4, v[26:29], s[60:61] offset:48 scope:SCOPE_DEV +.Ltrow_wskip\@: + s_mov_b32 exec_lo, s49 +.Ltrow_skip\@: +.endif +.endm + +// ============================================================================================ +// KERNEL +// ============================================================================================ + .text + .globl occ_kernel + .p2align 8 + .type occ_kernel,@function +occ_kernel: + // ---- FIX 1(b,c): derive n_kseg from KT (s8) + the compile-time NKSEG_SHIFT, then the shift/mask + // decode pair, into the reserved high SGPRs s66/s67/s68 BEFORE any clobber (SAFEPROBE below reuses + // s16 purely as scratch; the role bodies keep all transients <= s65). No v2 kernargs are read here + // anymore -- s15/s16/s17 are NOT hardware-preloaded under USER_SGPR=15 (see KERNARG CONTRACT above). ---- + s_lshr_b32 s66, s8, NKSEG_SHIFT // n_kseg = KT >> NKSEG_SHIFT (KT=s8) + s_ff1_i32_b32 s68, s66 // shift = log2(n_kseg) (bit index of the single set bit; n_kseg=1 -> 0) + s_sub_u32 s67, s66, 1 // mask = n_kseg - 1 + // ---- identity (lifted from coop prologue; v0=tid hardware-preloaded) ---- + v_lshrrev_b32 v1, 5, v0 // wid = tid >> 5 + v_and_b32 v2, 31, v0 // lane = tid & 31 + v_and_b32 v6, 15, v0 // lane & 15 (A vaddr) + v_mov_b32 v4, 0 +.if KMAJOR + global_load_b32 v3, v4, s[0:1] offset:248 scope:SCOPE_DEV // occ[62] = magic(TOTAL), host-written + s_wait_loadcnt 0x0 + v_readfirstlane_b32 s76, v3 // s76 = magic_TOTAL, persistent for every DECODE_STI (K-major) +.endif + // ---- per-lane address constants (mbgemm-identical; dyn-VGPR arming compatible) ---- + v_mul_lo_u32 v8, v6, s9 // (lane&15)*K + v_bfe_u32 v7, v0, 4, 1 + v_lshlrev_b32 v7, 3, v7 + v_add_nc_u32 v8, v8, v7 // v8 = A vaddr = (lane&15)*K + colhi*8 + v_lshlrev_b32 v9, 3, v2 // v9 = B/ds vaddr = lane*8 + v_lshlrev_b32 v10, 5, v2 // v10 = C store vaddr = lane*32 +.if SAFEPROBE + // brick-PROOF: clamp the per-lane VECTOR address regs to a loose upper bound (>= true max) so even a + // grow-corrupted vaddr cannot push a global access past the data+guard (pairs with the future ti clamp). + s_lshl_b32 s16, s9, 4 // 16*K (>= v8 max = (lane&15)*K + colhi*8 = 15*K+8) + v_min_u32 v8, s16, v8 // clamp A vaddr + v_min_u32 v9, 0x100, v9 // clamp B/ds vaddr (256 >= lane*8 max 248) + v_min_u32 v10, 0x400, v10 // clamp C vaddr (1024 >= lane*32 max 992) +.endif + + tfspan min, 8 // TFPROBE: every wave stamps occ[2] = min entry tick (wall-span start) +.if TRACE + // total-occupancy: every wave ++live at entry, atomic-max the peak concurrent resident count (occ[1]). + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lall_enter_skip + v_mov_b32 v3, 1 + global_atomic_add_u32 v5, v4, v3, s[0:1] offset:ALLLIVE_OFF th:TH_ATOMIC_RETURN scope:SCOPE_DEV + s_wait_loadcnt 0x0 + v_add_nc_u32 v5, v5, 1 + global_atomic_max_u32 v4, v5, s[0:1] offset:4 scope:SCOPE_DEV // occ[1] = peak concurrent resident waves +.Lall_enter_skip: + s_mov_b32 exec_lo, s49 +.endif + +.if DSWS2 +// ============================================================================================ +// FIX 1 -- FLOW ECONOMY unified role section (replaces the ring's dispatcher/feed/compute). +// Every wave runs ONE loop: read ROLE[wid] mailbox -> be that role (resize on change) -> +// try_grab one atomic (work-or-empty) -> do work, or COAST (code-path flip, no resize). NO +// publish poll anywhere. wid0 = coordinator (single writer): assigns super-tiles to free slots, +// seeds/nudges mailboxes, and also does lean B-feed work. See FLOW_ECONOMY_DESIGN.md. +// Persistent regs: s24=wid s34=cur_role s50=coord period ctr s69=chunkHi +// 3-frontier pipeline (LDS): DRAIN_HEAD <= STAGE_HEAD <= ASSIGN_HEAD <= DRAIN_HEAD+POOL_N +// ============================================================================================ + v_readfirstlane_b32 s24, v1 // wid (uniform) +.if DYNVGPR +.Lflow_alloc: + s_alloc_vgpr 32 // all start lean; compute grows on adopting ROLE_COMPUTE + s_cbranch_scc0 .Lflow_alloc // (NOTE: a bounded-exit + s_endpgm here BRICKS -- s_endpgm from a +.endif // wave that failed s_alloc_vgpr corrupts the SIMD dyn-VGPR pool + // -> OOB page fault. Do NOT exit a starved wave; cap W_launch.) + // live++ : lane0 occ[0] += 1 + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lflow_live + v_mov_b32 v3, 1 + global_atomic_add_u32 v4, v3, s[0:1] scope:SCOPE_DEV +.Lflow_live: + s_mov_b32 exec_lo, s16 + s_mov_b32 s34, 0xFFFFFFFF // cur_role = none -> first .Lflow_body forces a resize + s_mov_b32 s50, 0 // coordinator period counter + deadman_stamp // s[70:71] = start RTC (watchdog baseline) + s_cmp_eq_u32 s24, 0 + s_cbranch_scc0 .Lflow_wait_init // non-coordinator waits for LDS init + // ---- coordinator (wid0) barrier-free LDS init ---- + lds_put ASSIGN_HEAD_OFF, 0 + lds_put STAGE_HEAD_OFF, 0 + lds_put DRAIN_HEAD_OFF, 0 + lds_put FLOWTERM_OFF, 0 +.if RETBARRIER + lds_put QUIESCE_CNT_OFF, 0 // count-to-WAVES collective-exit counter (reset per dispatch) +.endif + // EMERGENT economy seed: minimal liveness FLOOR + everything else COMPUTE. wid0=coordinator (runs + // lean B-feed between ASSIGN duties), wid1=dedicated A-feed, wid2=dedicated B-feed; wid>=3=COMPUTE. + // Excess compute waves self-distribute to feed via .Lflow_coast; concurrent-fat emerges from the + // hardware s_alloc_vgpr grow-fail. NO baked NCOMP/NAFEED/NBFEED. + .set w, 0 + .rept WAVES + .if w == 0 + lds_put (ROLE_BASE + w*4), ROLE_BFEED + .elseif w == 1 + lds_put (ROLE_BASE + w*4), ROLE_AFEED + .elseif w == 2 + lds_put (ROLE_BASE + w*4), ROLE_BFEED + .else + lds_put (ROLE_BASE + w*4), ROLE_COMPUTE + .endif + .set w, w+1 + .endr + // init POOL_N slot control blocks: STAMP = sentinel, all counters 0 + .set sl, 0 + .rept POOL_N + lds_put (SLOTC_BASE + sl*SLOTC_STRIDE + SL_STI), 0xFFFFFFFF + lds_put (SLOTC_BASE + sl*SLOTC_STRIDE + SL_GEN), 0 + lds_put (SLOTC_BASE + sl*SLOTC_STRIDE + SL_RBNEXT), 0 + lds_put (SLOTC_BASE + sl*SLOTC_STRIDE + SL_RBDONE), 0 + lds_put (SLOTC_BASE + sl*SLOTC_STRIDE + SL_BFNEXT), 0 + lds_put (SLOTC_BASE + sl*SLOTC_STRIDE + SL_BFDONE), 0 + lds_put (SLOTC_BASE + sl*SLOTC_STRIDE + SL_ARNEXT), 0 + lds_put (SLOTC_BASE + sl*SLOTC_STRIDE + SL_ARDONE), 0 + .set sl, sl+1 + .endr + lds_put COORD_KSI_OFF, 0xFFFFFFFF // tile-claim sentinel: first ASSIGN claims a fresh tile + lds_put RINGINIT_OFF, 0xACED // LAST: publishes "LDS ready" + global_load_b32 v6, v4, s[0:1] offset:24 scope:SCOPE_DEV + s_wait_loadcnt 0x0 + v_readfirstlane_b32 s69, v6 // chunkHi + s_branch .Lflow_loop +.Lflow_wait_init: + s_sleep 1 + deadman_check // watchdog: wid0 never published init -> clean retire, no wedge + lds_get s44, RINGINIT_OFF + s_cmp_eq_u32 s44, 0xACED + s_cbranch_scc0 .Lflow_wait_init + global_load_b32 v6, v4, s[0:1] offset:24 scope:SCOPE_DEV + s_wait_loadcnt 0x0 + v_readfirstlane_b32 s69, v6 // chunkHi + +// ======================= the unified flow loop ======================= +.Lflow_loop: + deadman_check // watchdog at every loop head: a stalled frontier -> clean drain + s_cmp_eq_u32 s24, 0 + s_cbranch_scc0 .Lflow_body // non-coordinator -> straight to role work + // ---- coordinator duty (wid0): ASSIGN + (later) sense/nudge ---- + lds_get s44, FLOWTERM_OFF + s_cmp_eq_u32 s44, 0xDEAD + s_cbranch_scc1 .Lflow_drainwait // already terminal -> wait for drain + lds_get s44, ASSIGN_HEAD_OFF // ah + lds_get s45, DRAIN_HEAD_OFF // dh + s_sub_u32 s46, s44, s45 + s_cmp_ge_u32 s46, POOL_N + s_cbranch_scc1 .Lflow_coord_period // pool full -> no assign this cycle + // TILE-CLAIM: write-once needs a WG to own a whole tile's n_kseg segments so its LDS banks sum a + // full tile. occ[20] now counts TILES; emit n_kseg super-tiles sti=(t< continue current tile + s_cbranch_scc1 .Lflow_same_tile + // ksi exhausted (or sentinel) -> claim a NEW tile: lane0 occ[20]++ (counts tiles) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lflow_claim_done + v_mov_b32 v3, 1 + global_atomic_add_u32 v5, v4, v3, s[0:1] offset:20 th:TH_ATOMIC_RETURN scope:SCOPE_DEV + s_wait_loadcnt 0x0 +.Lflow_claim_done: + s_mov_b32 exec_lo, s16 + v_readfirstlane_b32 s56, v5 // t = claimed tile index + s_cmp_ge_u32 s56, s69 // t >= chunkHi (=TOTAL tiles) -> terminal + s_cbranch_scc1 .Lflow_set_term + lds_put COORD_T_OFF, s56 // remember current tile + s_mov_b32 s55, 0 // ksi cursor = 0 + s_branch .Lflow_form_sti +.Lflow_same_tile: + lds_get s56, COORD_T_OFF // reuse current tile +.Lflow_form_sti: + s_lshl_b32 s17, s56, s68 // t << shift + s_or_b32 s17, s17, s55 // sti = (t< keep helping (coordinator feeds/coasts) + // all assigned super-tiles drained -> tell everyone to retire, then retire self + .set w, 0 + .rept WAVES + lds_put (ROLE_BASE + w*4), ROLE_RETIRE + .set w, w+1 + .endr + s_branch .Lflow_retire + +// ---- role adopt + dispatch (every wave) ---- +.Lflow_body: + s_lshl_b32 s45, s24, 2 + s_add_u32 s45, s45, ROLE_BASE + lds_get_r s35, s45 // role = ROLE[wid] (stale == last role == coast, free) + s_cmp_eq_u32 s35, ROLE_RETIRE + s_cbranch_scc1 .Lflow_retire + s_cmp_eq_u32 s35, s34 // role unchanged? + s_cbranch_scc1 .Lflow_dispatch + // role changed -> resize at this lean boundary. + // STAGGER: compute waves grow PER-BURST inside .Lflow_compute (the trapezoid), NOT once at role-adopt. + // So a role change never GROWS here; every wave sits LEAN at the loop head (a compute wave shrank + // after its last burst). A defensive shrink-to-lean preserves that invariant if ever fat crossing here. +.if DYNVGPR +.Lflow_shrink: + s_alloc_vgpr 32 + s_cbranch_scc0 .Lflow_shrink +.Lflow_resized: +.endif + s_mov_b32 s34, s35 // cur_role = role +.Lflow_dispatch: + s_cmp_eq_u32 s34, ROLE_COMPUTE + s_cbranch_scc1 .Lflow_compute + s_branch .Lflow_feed + +// ---- COMPUTE work (wave is fat): pull one rowblk from the DRAIN_HEAD slot, WMMA, flush ---- +.Lflow_compute: + lds_get s46, DRAIN_HEAD_OFF // dh + lds_get s44, STAGE_HEAD_OFF // sh + s_cmp_ge_u32 s46, s44 // DRAIN >= STAGE -> nothing fully staged -> coast to feed + s_cbranch_scc1 .Lflow_coast + slot_of s45, s46, s47 // slot = dh mod N + s_lshl_b32 s48, s45, 5 + s_add_u32 s48, s48, SLOTC_BASE // scb + s_lshl_b32 s52, s45, 14 + s_add_u32 s52, s52, OP_BASE // sob + // PER-BURST GROW: trapezoid peak starts here (fat through WMMA+ds_add, lean otherwise). COAST-ON-FAIL + // is the floodgate: if the SIMD VGPR budget is full, grow SCC0 -> coast lean, committing NO claim. +.if DYNVGPR + s_alloc_vgpr NFV + s_cbranch_scc0 .Lflow_growfail +.endif + s_add_u32 s45, s48, SL_RBNEXT + lds_fetch_add_r s33, s45, 1 // claim rowblk r (committed only AFTER grow succeeds) + s_cmp_ge_u32 s33, G + s_cbranch_scc1 .Lflow_cmp_tryadv // rowblks exhausted (we are fat) -> shrink + try advance + // read STAMP (gsti) for C addressing + s_add_u32 s45, s48, SL_STI + lds_get_r s17, s45 + DECODE_STI // s19=mblk s30=tcol s31=ksi + // zero FM*FN accumulators + .set idx, 0 + .rept FM*FN + v_mov_b32 v[ACC+idx*8+0], 0 + v_mov_b32 v[ACC+idx*8+1], 0 + v_mov_b32 v[ACC+idx*8+2], 0 + v_mov_b32 v[ACC+idx*8+3], 0 + v_mov_b32 v[ACC+idx*8+4], 0 + v_mov_b32 v[ACC+idx*8+5], 0 + v_mov_b32 v[ACC+idx*8+6], 0 + v_mov_b32 v[ACC+idx*8+7], 0 + .set idx, idx+1 + .endr + v_add_nc_u32 v12, v9, s52 // B resident base (BRES_ROFF=0) + s_mul_i32 s37, s33, (FM*256) + v_add_nc_u32 v13, v9, s52 + v_add_nc_u32 v13, v13, ARES_ROFF + v_add_nc_u32 v13, v13, s37 // A resident base for rowblk r + .set ks, 0 + .rept KSEG_STEPS + .set ni, 0 + .rept FN + ds_load_b64 v[FB+ni*2:FB+ni*2+1], v12 offset:((ks*FN+ni)*256) + .set ni, ni+1 + .endr + .set mi, 0 + .rept FM + ds_load_b64 v[FA+mi*2:FA+mi*2+1], v13 offset:((ks*G*FM+mi)*256) + .set mi, mi+1 + .endr + s_wait_dscnt 0x0 + .set mi, 0 + .rept FM + .set ni, 0 + .rept FN + v_wmma_f32_16x16x16_fp8_fp8 v[ACC+(mi*FN+ni)*8:ACC+(mi*FN+ni)*8+7], v[FA+mi*2:FA+mi*2+1], v[FB+ni*2:FB+ni*2+1], v[ACC+(mi*FN+ni)*8:ACC+(mi*FN+ni)*8+7] + .set ni, ni+1 + .endr + .set mi, mi+1 + .endr + .set ks, ks+1 + .endr +.if WOFLUSH + // BURST-SCOPED FLUSH (no LDS bank): atomic-add this segment's fp32 ACC frags STRAIGHT to C[rowblk r]. + // C is memset 0 by the host; every segment of every rowblk atomic-adds -> C = full split-K sum. + // Same addressing as the write-once completer store (v10=lane*32, offset frag*1024+e*4) so it lands + // in the identical C locations -> correct by construction. s19=mblk s30=tcol s33=rowblk r (all live). + s_mul_i32 s38, s19, s13 // mblk*NTL + s_add_u32 s38, s38, s30 // + tcol + s_mul_i32 s38, s38, (G*FM*FN*1024) // * per-tile C bytes + s_mul_i32 s40, s33, (FM*FN*1024) // + rowblk r * per-rowblk C bytes + s_add_u32 s38, s38, s40 + s_add_u32 s28, s6, s38 + s_addc_u32 s29, s7, 0 // s[28:29] = C rowblk base + .set frag, 0 + .rept FM*FN + .set e, 0 + .rept 8 + global_atomic_add_f32 v10, v[ACC+frag*8+e], s[28:29] offset:(frag*1024 + e*4) scope:SCOPE_DEV + .set e, e+1 + .endr + .set frag, frag+1 + .endr + s_wait_storecnt 0x0 // J=1 correctness baseline: drain this wave's atomics +.else + // WRITE-ONCE REDUCE: accumulate this segment's partial into LDS bank[r] (mirrors C frag layout; + // vaddr = v10=lane*32, base = ACC_BASE + r*ACC_STRIDE). ksi==0 (tile's first segment, POOL_N=1 + // guarantees it drains before any later ksi) WRITES; ksi>0 ADDS. C is stored ONCE at ksi==mask + // (last segment) in .Lflow_cmp_tryadv. s31=ksi (survives WMMA), s33=rowblk r. + acc_base_of s39, s33 // s39 = ACC_BASE + r*ACC_STRIDE + v_add_nc_u32 v12, v10, s39 // v12 = bank r ds vaddr (lane*32 + bankbase) + s_cmp_eq_u32 s31, 0 // first segment of this tile's rowblk? + s_cbranch_scc1 .Lflow_bankwr + .set frag, 0 + .rept FM*FN + .set e, 0 + .rept 8 + ds_add_f32 v12, v[ACC+frag*8+e] offset:(frag*1024 + e*4) + .set e, e+1 + .endr + .set frag, frag+1 + .endr + s_branch .Lflow_bankdn +.Lflow_bankwr: + .set frag, 0 + .rept FM*FN + .set e, 0 + .rept 8 + ds_store_b32 v12, v[ACC+frag*8+e] offset:(frag*1024 + e*4) + .set e, e+1 + .endr + .set frag, frag+1 + .endr +.Lflow_bankdn: + s_wait_dscnt 0x0 +.endif + instr_inc STINSTR_COMP // diag: a rowblk-segment was actually computed+reduced + s_add_u32 s45, s48, SL_RBDONE + lds_fetch_add_r s47, s45, 1 // s47 = old RBDONE; old==G-1 -> I am the UNIQUE completer +.if DYNVGPR +.Lflow_bshrink: + s_alloc_vgpr 32 // SHRINK -> lean (close the trapezoid burst) BEFORE any store + s_cbranch_scc0 .Lflow_bshrink +.endif + s_add_u32 s47, s47, 1 + s_cmp_ge_u32 s47, G // (old+1) >= G -> I completed this super-tile + s_cbranch_scc0 .Lflow_loop // not the completer -> done, loop + // COMPLETER (single wave -> NO race, NO redundant store, NO spinning losers): the super-tile is fully + // reduced. If it's the tile's LAST ksi (ksi==mask), store the G banks to C ONCE, s_wait_storecnt, + // THEN advance DRAIN -> the next tile's ksi=0 cannot overwrite the banks until this store is drained. + // s19/s30/s31 still hold mblk/tcol/ksi from this wave's own DECODE_STI (untouched by the reduce). + s_cmp_eq_u32 s31, s67 // ksi == mask (n_kseg-1) -> tile complete? + s_cbranch_scc0 .Lflow_drain_adv // not last ksi -> just advance DRAIN (no store) +.if !WOFLUSH + s_mul_i32 s38, s19, s13 // mblk*NTL + s_add_u32 s38, s38, s30 // + tcol + s_mul_i32 s38, s38, (G*FM*FN*1024) // * per-tile C bytes + s_add_u32 s28, s6, s38 + s_addc_u32 s29, s7, 0 // s[28:29] = C tile base (rowblk 0) + .set r, 0 + .rept G + s_mov_b32 s39, (ACC_BASE + r*(FM*FN*1024)) // bank r LDS base (compile-time) + v_add_nc_u32 v12, v10, s39 // v12 = bank r ds vaddr (lane*32) + .set frag, 0 + .rept FM*FN + .set e, 0 + .rept 8 + ds_load_b32 v13, v12 offset:(frag*1024 + e*4) + s_wait_dscnt 0x0 + global_store_b32 v10, v13, s[28:29] offset:(r*(FM*FN*1024) + frag*1024 + e*4) scope:SCOPE_DEV + .set e, e+1 + .endr + .set frag, frag+1 + .endr + .set r, r+1 + .endr + s_wait_storecnt 0x0 // store COMPLETE before DRAIN++ -> banks safe to reuse +.endif // WOFLUSH: atomics already wrote C incrementally -> no store, just DRAIN++ +.Lflow_drain_adv: + lds_get s44, DRAIN_HEAD_OFF + lds_cmpstore_adv DRAIN_HEAD_OFF, s44 // completer advances DRAIN (unique wave; store already done) + s_branch .Lflow_loop +.Lflow_cmp_tryadv: +.if DYNVGPR +.Lflow_tashrink: + s_alloc_vgpr 32 // grew but rowblks exhausted (no claim) -> shrink back lean + s_cbranch_scc0 .Lflow_tashrink +.endif + s_branch .Lflow_loop // the bank store + DRAIN advance are done by the COMPLETER + // (the unique wave whose SL_RBDONE inc hit G, in .Lflow_bankdn) + +// ---- FEED work: stage the STAGE_HEAD slot (A if ROLE_AFEED, B if ROLE_BFEED), then try-advance STAGE ---- +.Lflow_feed: + lds_get s44, STAGE_HEAD_OFF // sh + lds_get s45, ASSIGN_HEAD_OFF // ah + s_cmp_ge_u32 s44, s45 // STAGE >= ASSIGN -> nothing assigned to stage -> yield + s_cbranch_scc1 .Lflow_feed_empty + slot_of s46, s44, s47 // slot = sh mod N + s_lshl_b32 s48, s46, 5 + s_add_u32 s48, s48, SLOTC_BASE // scb + s_lshl_b32 s52, s46, 14 + s_add_u32 s52, s52, OP_BASE // sob + s_add_u32 s45, s48, SL_STI + lds_get_r s17, s45 // gsti = STAMP (assigned -> set) + DECODE_STI + s_cmp_eq_u32 s34, ROLE_BFEED + s_cbranch_scc1 .Lflow_stageB + ASTAGE_R s48, s52 + s_branch .Lflow_stage_adv +.Lflow_stageB: + BSTAGE_R s48, s52 +.Lflow_stage_adv: + // try-advance STAGE_HEAD: if the CURRENT STAGE_HEAD slot is fully staged, CAS-bump it + lds_get s44, STAGE_HEAD_OFF + lds_get s45, ASSIGN_HEAD_OFF + s_cmp_ge_u32 s44, s45 + s_cbranch_scc1 .Lflow_loop + slot_of s46, s44, s47 + s_lshl_b32 s48, s46, 5 + s_add_u32 s48, s48, SLOTC_BASE + s_add_u32 s45, s48, SL_BFDONE + lds_get_r s47, s45 + s_cmp_lt_u32 s47, FN + s_cbranch_scc1 .Lflow_loop + s_add_u32 s45, s48, SL_ARDONE + lds_get_r s47, s45 + s_cmp_lt_u32 s47, G + s_cbranch_scc1 .Lflow_loop + lds_cmpstore_adv STAGE_HEAD_OFF, s44 + s_branch .Lflow_loop +.Lflow_feed_empty: + s_sleep SLEEPN // lean feed can't compute without a grow -> yield; coordinator rebalances + s_branch .Lflow_loop + +// ---- COAST: a fat COMPUTE wave with no staged work runs feed code (FREE, no resize) to help staging ---- +.Lflow_growfail: + instr_inc STINSTR_GROWFAIL // diag: per-burst grow failed (budget full) -> coast = stagger repulsion + // fall through: the failed grow allocated nothing, so we are still lean and safe to coast +.Lflow_coast: + instr_inc STINSTR_COAST // diag: a wave coasted (no staged work, or a grow-fail) + lds_get s44, STAGE_HEAD_OFF + lds_get s45, ASSIGN_HEAD_OFF + s_cmp_ge_u32 s44, s45 + s_cbranch_scc1 .Lflow_feed_empty // nothing assigned to stage -> yield + slot_of s46, s44, s47 + s_lshl_b32 s48, s46, 5 + s_add_u32 s48, s48, SLOTC_BASE + s_lshl_b32 s52, s46, 14 + s_add_u32 s52, s52, OP_BASE + s_add_u32 s45, s48, SL_BFDONE + lds_get_r s47, s45 + s_cmp_lt_u32 s47, FN + s_cbranch_scc1 .Lflow_coastB // B behind -> help B + s_add_u32 s45, s48, SL_STI + lds_get_r s17, s45 + DECODE_STI + ASTAGE_R s48, s52 + s_branch .Lflow_stage_adv +.Lflow_coastB: + s_add_u32 s45, s48, SL_STI + lds_get_r s17, s45 + DECODE_STI + BSTAGE_R s48, s52 + s_branch .Lflow_stage_adv + +.Lflow_retire: + // live-- : lane0 occ[0] -= 1 (harness completion gate) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lflow_dead + v_mov_b32 v3, -1 + global_atomic_add_u32 v4, v3, s[0:1] scope:SCOPE_DEV +.Lflow_dead: + s_mov_b32 exec_lo, s16 + tfspan max, 12 // TFPROBE: wall-span end + alllive_dec +.if RETBARRIER + // COUNT-TO-WAVES collective exit: check in, then all s_endpgm TOGETHER once the WG's count hits WAVES. + // This coordinated exit is what the EOP needs to register a clean dispatch completion -> fence FIRES + // (the coordinator's staggered RETIRE broadcast fires the fence at 8 waves but not 16). Bounded wait + // (plain counter + s_sleep; NO RTC/message bus, NO s_alloc) -> can never hang the wave. + lds_inc QUIESCE_CNT_OFF // this wave checked in at the exit + s_mov_b32 s52, 0 +.Lflow_retbar: + lds_get s53, QUIESCE_CNT_OFF + s_cmp_ge_u32 s53, WAVES // all WAVES waves in -> exit together + s_cbranch_scc1 .Lflow_endpgm + s_add_u32 s52, s52, 1 + s_cmp_ge_u32 s52, RETBAR_MAX // safety bound -> exit anyway (never hang) + s_cbranch_scc1 .Lflow_endpgm + s_sleep SLEEPN + s_branch .Lflow_retbar +.Lflow_endpgm: +.endif + s_endpgm +.else + s_endpgm // DSWS2=0 has no v2 body (this file is always built DSWS2=1) +.endif + .size occ_kernel, .-occ_kernel + +// ---- RGADESC: analysis-only descriptor so `rga -s bin --co` can enumerate + livereg this kernel. +// vgpr 256 ceiling so livereg reports the true s_alloc-grown peak-live. NOT emitted for the PM4 .bin. ---- +.if RGADESC +.amdhsa_kernel occ_kernel + .amdhsa_next_free_vgpr 256 + .amdhsa_next_free_sgpr 72 // body uses up to s69 (s66=n_kseg s67=mask s68=shift s69=chunkHi, FIX 1) + .amdhsa_group_segment_fixed_size 65536 // FIX 1a ring: D=2 needs 33024B (RGA-analysis descriptor only) + .amdhsa_user_sgpr_count 15 // FIX 1(g): v2 contract now s0..s14 only (n_kseg/TOTAL_super/ + // magic_kseg dropped -- derived in-kernel / memory-carried) + .amdhsa_wavefront_size32 1 +.end_amdhsa_kernel +.amdgpu_metadata +--- +amdhsa.version: [ 1, 2 ] +amdhsa.kernels: + - .name: occ_kernel + .symbol: occ_kernel.kd + .kernarg_segment_size: 60 + .kernarg_segment_align: 8 + .group_segment_fixed_size: 65536 + .private_segment_fixed_size: 0 + .wavefront_size: 32 + .sgpr_count: 72 + .vgpr_count: 256 + .max_flat_workgroup_size: 256 + .args: [] +.end_amdgpu_metadata +.endif diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_dsws_ring.s b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_dsws_ring.s new file mode 100644 index 000000000000..7d6659048038 --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_dsws_ring.s @@ -0,0 +1,1696 @@ +// occ_kernel_dsws.s (gfx1201, wave32) -- MAD-305 DSWS v2 SUBSTRATE SCAFFOLD (PLAN_DSWS_SUBSTRATE_V2.md, Task A1). +// +// v2 re-founds the DSWS GEMM substrate so matrix work is *claimed* (not owned by compile-time wave id), with +// split-K folded in. Work = a pool of (mblk, tcol, ksi) super-tiles; a pinned claimer (wid 0) broadcasts the +// current super-tile; live waves of each role drain shared LDS atomic counters against resident-in-LDS A/B. +// +// Task A1 is SCAFFOLD-ONLY: lift the proven prologue/arming setup from occ_kernel_coop.s (NEVER modified), +// assert the v2 LDS layout fits the 32 KB group segment, and leave each role as a distinct s_endpgm stub. +// The real claimer / feed / compute loops land in A3..A7. The only gates here: (1) assembles clean, +// (2) RGA 0-spill, (3) harness --dsws2 dry-prints the computed params. +// +// ============================================================================================ +// v2 KERNARG CONTRACT (USER_SGPR=15, s0..s14; hardware-preloaded user SGPRs). The host A8 launch +// MUST set COMPUTE_PGM_RSRC2.USER_SGPR=15 and load COMPUTE_USER_DATA_0..14 to match EXACTLY: +// s0:s1 = occ buffer base (>=0x1000B; host zero-inits; see CLAIM-COUNTER / completion offsets below) +// s2:s3 = A base (fp8 e4m3, row-major, 1 byte/elem) +// s4:s5 = Bshuf base (the shuffled-B layout the coop B-feed consumes; same global_load_tr_b64 idiom) +// s6:s7 = C base (fp32; HOST MUST MEMSET C=0 before dispatch -- compute uses global_atomic_add_f32) +// s8 = KT (total K16-steps for the whole matrix = K/16) +// s9 = K(bytes/A-row) (= K, fp8 1 byte/elem) +// s10 = NT*256 (B-saddr advance per K16-step) +// s11 = TOTAL (coop-style total *tiles* = MTL*NTL; carried for addressing compat, NOT the pool size) +// s12 = magic(ceil(2^32/NTL)) (unsigned-div magic for /NTL ; tcol/mblk decode) +// s13 = NTL (number of N tile-columns) +// s14 = FN*256 (B-saddr stride per N-frag) +// (TGID_X now lands in s15 -- UNUSED; this kernel is pool-claim, not workgroup-id based.) +// NOTE: G and SEGK are COMPILE-TIME defsyms (baked into instruction immediates); they are NOT kernargs. +// FIX 1 (round-table Opus+Codex pass): v1 of this contract passed n_kseg/TOTAL_super/magic_kseg as +// s15/s16/s17, but the PM4 host only preloads COMPUTE_USER_DATA_0..15 (USER_SGPR<=16; every proven +// launch path in this tree uses 15) -- s16/s17 could NEVER actually arrive in hardware SGPRs, AND +// s16 was independently being reused per-chunk on the host as the compositor-safe chunk terminal +// (a second, unrelated collision on the same slot). This file now drops s15/s16/s17 entirely: +// n_kseg is DERIVED in-kernel from KT (s8) and the compile-time KSEG_STEPS=SEGK/16: +// n_kseg = KT >> NKSEG_SHIFT, where NKSEG_SHIFT=log2(KSEG_STEPS) is a compile-time `.set` +// (small .if ladder over KSEG_STEPS in {1,2,4,8,16}; SEGK is always a power-of-two +// multiple of 16, so KSEG_STEPS is always a power of two in that set). +// shift/mask (the sti -> (t,ksi) split) are derived ONCE in the prologue from n_kseg: +// shift = s_ff1_i32_b32(n_kseg) (bit index of n_kseg's single set bit; n_kseg=1 -> 0) +// mask = n_kseg - 1 +// DECODE_STI then does ksi = sti & mask ; t = sti >> shift -- this handles n_kseg=1 for +// free (shift=0, mask=0 -> ksi=0, t=sti), so the old magic-div n_kseg==1 special-case is +// GONE (it's no longer needed, not just hidden). +// the chunk terminal (old TOTAL_super/"chunkHi") is now MEMORY-CARRIED instead of a kernarg: the +// host writes the current chunk's terminal sti bound to occ[24] (occW[6]) once per chunk; +// the claimer reads occ[24] ONCE per dispatch (stable for the whole chunk) instead of +// receiving it as a broadcast kernarg. On sti >= occ[24] the claimer publishes a SENTINEL +// (0xFFFFFFFF) into STI_OFF instead of the raw over-claimed sti; followers (b-feed/a-feed/ +// compute) retire when STI_OFF == 0xFFFFFFFF instead of comparing against the (now +// nonexistent) TOTAL_super kernarg. +// See "CLAIM-COUNTER & completion occ offsets" by .Lclaimer for the full occ-buffer layout +// (occ[24]/occW[6] = chunk terminal bound, added by FIX 1). +// +// SCALAR REGS (derived in the prologue, before any clobber; none collide with DECODE_STI's own clobber +// list s18/s36, lds_*'s s49, the claimer's s16/s17/s35/s44, or any role body's transients, all <= s65): +// s66 = n_kseg (derived; dead after shift/mask below are computed -- kept only for that derivation) +// s67 = mask (n_kseg - 1) -- LIVE for the whole kernel; read by every DECODE_STI call, every role. +// s68 = shift (log2 n_kseg) -- LIVE for the whole kernel; read by every DECODE_STI call, every role. +// s69 = chunkHi (claimer-only; loaded once per dispatch from occ[24] right before .Lclaim_loop). +// +// Everything new is gated behind the fresh `DSWS2` build symbol (analogous to coop's `DSWS`). + +.amdgcn_target "amdgcn-amd-amdhsa--gfx1201" + +// ---- tile defsyms (lifted from occ_kernel_coop.s) ---- +.ifndef FM + .set FM, 2 // per-compute-wave M-frags (M-band = FM*16 rows) +.endif +.ifndef FN + .set FN, 4 // shared N-frags (the reuse operand) +.endif +.ifndef RGADESC + .set RGADESC, 0 // 1 = emit analysis-only AMDHSA descriptor for RGA livereg +.endif +.ifndef DIAG + .set DIAG, 0 // 1 = phase-marker instrumentation (unused in the A1 scaffold) +.endif +.ifndef SAFEPROBE + .set SAFEPROBE, 0 // 1 = clamp per-lane vector address regs into a provable in-buffer bound +.endif +.ifndef TFPROBE + .set TFPROBE, 0 // 1 = emit s_sendmsg_rtn GET_REALTIME wall-span capture (each wave stamps +.endif // occ[2]=min entry tick, occ[3]=max exit tick -> host TF readout). + // Default 0 => ZERO bytes emitted, .text byte-identical to the + // production bin (uses only already-allocated regs s30/s31/s49/v5). +.ifndef PHASEPROBE + .set PHASEPROBE, 0 // 1 = in-kernel PHASE TIMER: each compute wave stamps GET_REALTIME at every +.endif // phase boundary and atomic-adds the delta into per-phase occ accumulators + // (occ[64..69], bytes 256..276, ABOVE the per-chunk memset -> accumulate over + // the whole run). Host prints ticks + % per phase. Default 0 => byte-identical. +.ifndef NOCFLUSH + .set NOCFLUSH, 0 // PERF PROBE ONLY: 1 = skip the global_atomic_add_f32 C-flush loop (keep ALL +.endif // other bookkeeping/handshake). Isolates the device-atomic C-reduction cost + // from the coordination handshake. Result is WRONG (C never written -> oracle + // fails) -- span/TF only. Default 0 => byte-identical. +.ifndef KMAJOR + .set KMAJOR, 0 // PERF PROBE: 1 = K-MAJOR super-tile traversal. Default decode packs ksi in +.endif // the LOW bits (all n_kseg segments of a C cell claimed consecutively -> + // up to n_kseg WGs hammer one C cell at once = max atomic contention). + // KMAJOR decodes ksi = sti / TOTAL (high), t = sti % TOTAL (low) via a + // magic-div (magic_TOTAL from occ[62], loaded to s76 in prologue) -> the + // 32 segments of a cell are spread TOTAL apart in claim order (near-zero + // concurrent contention) + adjacent tiles (shared operand bands) claimed + // close in time (L2 reuse). Correctness-preserving (C add is commutative). +.ifndef CSTORE + .set CSTORE, 0 // PERF PROBE ONLY: 1 = replace the flush's global_atomic_add_f32 with an +.endif // equal-count NON-atomic global_store_b32 (same #mem-ops, same addresses, NO + // RMW/contention). Isolates atomic-contention from raw write-bandwidth: if + // TF jumps vs atomics -> contention; if flat -> bandwidth-bound. Result WRONG + // (last-writer, no accumulation) -- span/TF only. Default 0 => byte-identical. +.ifndef TRACE + .set TRACE, 0 // 1 = per-super-tile CLAIMER trace: append one row/super-tile {tick, segcnt, +.endif // epoch, nComp/nAfeed/nBfeed live role slots, ring occA/occB peak, + // convCount, vresv, sti, quiesce} to a host-provided buffer (VA in + // occ[52:53], cap in occ[54]). Time-series of the adaptive wave-role + // economy. Requires DSWS2_CONV=1 (rows written in the quiesce path). + // Default 0 => ZERO bytes; uses free high SGPRs s70..s74. +.ifndef DYNVGPR + .set DYNVGPR, 1 // 1 = compute waves s_alloc_vgpr-grow per rowblk; feeds/claimer stay lean 32 +.endif +.ifndef SLEEPN + .set SLEEPN, 2 // s_sleep arg in the busy-waits (yield issue cycles to partner waves) +.endif + +// ============================================================================================ +// DSWS v2 LDS layout (bytes from group-segment base; words u32 unless noted). Mirrors the placement +// of the coop file's LDS `.set` block. Defined unconditionally (uses only G/SEGK/FM/FN, always set). +// ============================================================================================ +.ifndef DSWS2 + .set DSWS2, 0 +.endif +.ifndef G + .set G, 6 // cooperative M-extent (rowblks per super-tile) = NCOMP_MAX +.endif +.ifndef SEGK + .set SEGK, 64 // split-K segment size in K-elements (multiple of 16) +.endif +// ---- v2 control/claim words ---- +.set STI_OFF, 0 // broadcast super-tile id +.set EPOCH_OFF, 4 +.set ROWBLK_NEXT_OFF, 8 // per-super-tile rowblk claim counter +.set ROWBLK_DONE_OFF, 12 // per-super-tile completion counter +.set BFRAG_NEXT_OFF, 16 // B-frag claim counter +.set AROW_NEXT_OFF, 20 // A-rowblk claim counter +.set NCOMP_SLOT, 24 +.set NAFEED_SLOT, 28 +.set NBFEED_SLOT, 32 +.set GATE_OFF, 36 // u32[4] -> 36,40,44,48 (conversion gates) +.set VRESV_OFF, 52 // vgpr_reserved +.set SEGCNT_OFF, 56 // controller clock +// ---- A3..A7 additions (still inside the 0..256 control region; A1 offsets 0..56 unchanged) ---- +.set BFRAG_DONE_OFF, 60 // B-frag STORE-completion counter (compute gates on this, NOT the claim ctr) +.set AROW_DONE_OFF, 64 // A-rowblk STORE-completion counter (compute gates on this) +.set INITFLAG_OFF, 68 // barrier-free LDS-init publish flag (claimer writes 0xACED LAST) +// ---- Phase-B (DSWS2_CONV) control state: role-mix snapshot slots + quiesce counter ---- +// Based at INITFLAG_OFF+4 (NOT the brief's SEGCNT_OFF+4): the brief predates the A3..A7 control +// words (BFRAG_DONE/AROW_DONE/INITFLAG at 60/64/68), so SEGCNT_OFF+4=60 would collide with them. +// Basing after the LAST control word keeps the new state inside the 0..255 control gap BELOW the +// fixed resident region (BRES_OFF=256), so NO resident-region repoint is needed -- the resident +// BRES_OFF/ARES_OFF immediates (emitted unconditionally in the kernel body) stay untouched, which +// is what keeps the DSWS2_CONV=0 binary byte-identical to the Phase-A green bin. All `.set`s here +// are inert (emit no bytes); the only new code (claimer init) is gated under `.if DSWS2_CONV`. +.ifndef DSWS2_CONV + .set DSWS2_CONV, 0 // 0 = pre-conversion static substrate (Phase A green); 1 = Phase B +.endif +.ifndef DSWS2_TICKET_SELFTEST + .set DSWS2_TICKET_SELFTEST, 0 // DIAG-only try_gate single-winner smoke (Task 4 Step 3); default 0 = no bytes +.endif +.ifndef CONV_COOLDOWN + .set CONV_COOLDOWN, 0 // Task 4: per-wave post-conversion cooldown epochs. 0 = spec-faithful (no + // cooldown, byte-identical to pre-Task-4); >0 damps thrash (skip N epochs + // of watermark decision after a wave converts role). +.endif +// Task 5: deterministic bring-up hook. DSWS2_FORCE=1 makes exactly wave DSWS2_FORCE_WID convert +// direction DSWS2_FORCE_DIR at epoch DSWS2_FORCE_EPOCH, watermarks bypassed -- a reproducible, +// single-wave/single-epoch GPU proof of role conversion. Default DSWS2_FORCE=0 emits ZERO bytes +// (byte-identical to pre-Task-5). +.ifndef DSWS2_FORCE + .set DSWS2_FORCE, 0 +.endif +.ifndef DSWS2_FORCE_WID + .set DSWS2_FORCE_WID, 0 +.endif +.ifndef DSWS2_FORCE_DIR + .set DSWS2_FORCE_DIR, 0 // 0/1 = compute->A/B ; 2/3 = A/B->compute +.endif +.ifndef DSWS2_FORCE_EPOCH + .set DSWS2_FORCE_EPOCH, 1 +.endif +// Rolling dyn-VGPR sum-envelope (2026-07-02 spec). ENVELOPE routes the per-rowblk compute burst grow +// through the shared vgpr_reserved counter so at most PEAK_CONC waves hold peak at once (the +// multi-grower collision, ISA 3.3.3.2, becomes unreachable). All default to the byte-identical value: +// ENVELOPE=0/STAGGER=0 emit ZERO new bytes and PEAK_CONC/STAGGER_PERIOD are inert unless their gate is on. +.ifndef DSWS2_ENVELOPE + .set DSWS2_ENVELOPE, 0 // 1 = route the per-rowblk compute burst grow through the vgpr_reserved +.endif // sum-envelope. 0 = HEAD (bare .Lcompute_grow) -> .text byte-identical. +.ifndef PEAK_CONC + .set PEAK_CONC, 2 // concurrent compute peaks the budget admits (R3 sweep). Used iff ENVELOPE=1. +.endif +.ifndef DSWS2_STAGGER + .set DSWS2_STAGGER, 0 // 1 = lock-free phase-token stagger (Task 9). 0 -> emergent envelope stagger. +.endif +.ifndef STAGGER_PERIOD + .set STAGGER_PERIOD, NCOMP // phase slots in the stagger ring (R3 sweep). Used iff STAGGER=1. +.endif +.set SNAP_BASE, (INITFLAG_OFF + 4) // u32[6]: [parity*3 + {0:nC,1:nA,2:nB}] role-mix snapshots +.set QUIESCE_CNT_OFF,(SNAP_BASE + 6*4) // u32 role-agnostic bail counter (LDS; DSWS2_GQUIESCE=0) +// SENSOR FIX: the claimer publishes its MID-DRAIN ring-occupancy PEAK here each super-tile; the conversion +// decisions read THESE instead of sampling occ_sample at their own quiesce (where occ_X reads ~0 post-drain +// -> always "starved" -> the 4/2/2->1/6/1 compute->feed runaway). Mid-drain peak = the true demand signal. +.set OCCA_PUB_OFF, (QUIESCE_CNT_OFF + 4) // claimer-published occ_A peak +.set OCCB_PUB_OFF, (OCCA_PUB_OFF + 4) // claimer-published occ_B peak +.set DSWS2_STATE_END,(OCCB_PUB_OFF + 4) +// DSWS2_GQUIESCE (2026-07-02 SUSPECT #2 candidate fix): route the QUIESCE handshake through a DEVICE-SCOPED +// GLOBAL atomic in the uncached occ buffer (byte QUIESCE_GOFF), mirroring the GREEN occ[20] claim/occ[0] +// live handshake, instead of the barrier-free LDS counter (whose cross-wave visibility is unguaranteed and +// is the leading SUSPECT #2 hang mechanism). occ buffer = AllocGpu 0x1000 (1024 u32, uncached); host uses +// occ[0..6] + DIAG scratch (<= byte 116); byte 200 (occ[50]) is provably free. Default 0 => LDS path, +// .text byte-identical. Requires DSWS2_CONV (QUIESCE only exists there). +.ifndef DSWS2_GQUIESCE + .set DSWS2_GQUIESCE, 0 +.endif +.set QUIESCE_GOFF, 200 // occ[] byte offset for the global QUIESCE counter (occ[50]) +// ---- TRACE (per-super-tile time-series) occ handshake words + row layout ---- +.set TRACE_PTR_OFF, 208 // occ[52:53] = trace buffer VA (host writes lo/hi per chunk) +.set TRACE_CAP_OFF, 216 // occ[54] = MAXROWS (host-provided row capacity) +.set TRACE_IDX_OFF, 220 // occ[55] = GLOBAL row-claim counter (all WGs' claimers share it) +.set TRACE_WGID_OFF, 224 // occ[56] = GLOBAL wg-id dispenser (claim-order 0..pool-1) +.set FATLIVE_OFF, 228 // occ[57] = live count of GROWN (fat NFV-VGPR) compute waves +.set FATMAX_OFF, 232 // occ[58] = PEAK concurrent fat waves -> x NFV = VGPR in flight (== B probe) +.set ALLLIVE_OFF, 240 // occ[60] = live count of ALL resident waves (++entry/--exit) + // occ[1] (byte 4) = PEAK concurrent resident waves (vs 2048 HW ceiling) +.set TRACE_ROW_BYTES, 64 // 16 u32/row +// DSWS2_BAILMARK (SUSPECT #2 localization, 2026-07-03): each follower publishes its OWN epoch (s35) to a +// PER-WAVE occ slot (BAIL_BASE + wid*4) at its _quiesce bail. One-shot per super-tile per wave -> minimal +// timing perturbation (NOT the claimer's per-spin DIAG poll stores, which are the heisenbug source and stay +// DIAG-only). After a watchdog abort the host reads occ[BAIL_BASE/4 + wid]: every follower's slot == the +// hung epoch => all reached their bail (=> a QUIESCE visibility/lost-update, gq relevant); ONE slot stale +// at the prior epoch => that exact wave is the STRAGGLER (stuck in _alloc/_init/_follow; gq irrelevant). +// Per-WAVE (not per-role): 4 compute share one role, so a role mark's last-writer-wins would hide a single +// straggler. Default 0 => no bytes, .text byte-identical. Requires DSWS2_CONV. +.ifndef DSWS2_BAILMARK + .set DSWS2_BAILMARK, 0 +.endif +.set BAIL_BASE, 160 // occ[] byte offset base for per-wave bail marks: occ[40..47] +.set CONVCNT_OFF, 192 // occ[48]: DIAG conversion-commit counter (proves waves switch role) + // (host prints occ[40..47] as BAIL[w0..w7]; clear of the + // occ[32..36]/occ[39] DSWS sensor+roles slots and occ[50] gq) +.set KSEG_STEPS, (SEGK/16) // K16-steps per split-K segment = SEGK K-elements / 16 +// FIX 1(b): NKSEG_SHIFT = log2(KSEG_STEPS), so the prologue can derive n_kseg = KT >> NKSEG_SHIFT instead +// of receiving it as a (now-dropped) kernarg. SEGK is always a power-of-two multiple of 16 in every +// config this file is built with, so KSEG_STEPS is always a power of two in {1,2,4,8,16}; a static +// ladder over that small set is simpler/safer than a general-purpose compile-time log2. +.if KSEG_STEPS == 1 + .set NKSEG_SHIFT, 0 +.elseif KSEG_STEPS == 2 + .set NKSEG_SHIFT, 1 +.elseif KSEG_STEPS == 4 + .set NKSEG_SHIFT, 2 +.elseif KSEG_STEPS == 8 + .set NKSEG_SHIFT, 3 +.elseif KSEG_STEPS == 16 + .set NKSEG_SHIFT, 4 +.else + .error "KSEG_STEPS (SEGK/16) must be a power of two in {1,2,4,8,16}" +.endif +// resident regions aligned to 256B +.set BRES_OFF, 256 // resident B for current super-tile +.set BRES_BYTES, (FN*16*SEGK) // = 4*16*64 = 4096 at the default config +.set ARES_OFF, (BRES_OFF + BRES_BYTES) // resident A for current super-tile +.set ARES_BYTES, (G*16*FM*SEGK) // = 6*16*2*64 = 12288 at the default config +.set LDS_TOTAL_DSWS2, (ARES_OFF + ARES_BYTES) +// (old single-slot cap check retained; 16640 < 32768 -> always passes. The RING layout below is what +// the ring role loops actually use; its own cap check follows.) +.if LDS_TOTAL_DSWS2 > 32768 + .error "DSWS2 LDS layout exceeds 32768B group segment" +.endif +// ============================================================================================ +// FIX 1a -- RING-OF-SLOTS LDS layout (D=2 double-buffered pipeline). NEW symbols; the single-slot +// control words above (STI_OFF..INITFLAG_OFF, SNAP/QUIESCE/TRACE state) stay DEFINED (they emit no +// bytes and their only *code* uses are under .if DSWS2_CONV/DIAG/TRACE, all 0 here -> not assembled). +// The ring role loops reference ONLY the symbols below. GEN is a GLOBAL-monotonic publish counter: +// the dispatcher bumps it once per publish and writes it to the published slot's SL_GEN LAST (the +// release fence); followers gate "new occupant" on SL_GEN[idx] > last-seen-gen. Slot counters reset +// only at FREE->FILLED (SL_RBDONE>=G), only by the dispatcher -> the old QUIESCE handshake is +// unnecessary (compute can't bump SL_RBDONE until SL_BFDONE==FN && SL_ARDONE==G, so no straggler is +// mid-claim at reset). See RING_SLOTS_DESIGN.md "Protocol refinements". +.set RING_D, 2 +// ---- shared frontier (single copy, at the front) ---- +.set FILL_IDX_OFF, 0 // slot the feeds are currently staging (dispatcher publishes into it) +.set DRAIN_IDX_OFF, 4 // slot compute is currently draining +.set RINGINIT_OFF, 8 // barrier-free LDS-init publish flag (dispatcher writes 0xACED LAST) +// ---- per-slot control block: SLOTC_BASE + slot*SLOTC_STRIDE + field ---- +.set SLOTC_BASE, 32 +.set SLOTC_STRIDE, 32 +.set SL_STI, 0 // super-tile id resident in this slot (0xFFFFFFFF = sentinel/retire) +.set SL_GEN, 4 // global-monotonic publish gen (release fence; consumers gate on > last-seen) +.set SL_RBNEXT, 8 // rowblk claim counter (compute) +.set SL_RBDONE, 12 // rowblks computed+flushed; dispatcher recycles this slot when >= G +.set SL_BFNEXT, 16 // B-frag claim counter (B-feeds) +.set SL_BFDONE, 20 // B-frags stored; compute gates on == FN +.set SL_ARNEXT, 24 // A-rowblk claim counter (A-feeds) +.set SL_ARDONE, 28 // A-rowblks staged; compute gates on == G +// ---- per-slot operand buffers: OP_BASE + slot*OPSTRIDE ; BRES at +BRES_ROFF, ARES at +ARES_ROFF ---- +.set OP_BASE, 256 // 256B-aligned; below it: frontier + RING_D control blocks +.set OPSTRIDE, (BRES_BYTES + ARES_BYTES) // 4096 + 12288 = 16384 per slot +.set BRES_ROFF, 0 // resident B within a slot +.set ARES_ROFF, BRES_BYTES // resident A within a slot (after B) +.set LDS_TOTAL_RING,(OP_BASE + RING_D*OPSTRIDE) // 256 + 2*16384 = 33024 +.if LDS_TOTAL_RING > 65536 + .error "RING LDS layout exceeds 65536B group segment (hardware WGP limit)" +.endif +.if (SLOTC_BASE + RING_D*SLOTC_STRIDE) > OP_BASE + .error "RING per-slot control blocks overlap the operand region (raise OP_BASE)" +.endif +// Phase-B state must fit in the control gap below the resident region (inert compile check, no bytes). +.if DSWS2_STATE_END > BRES_OFF + .error "DSWS2 Phase-B state (SNAP_BASE/QUIESCE_CNT) overlaps resident B region (BRES_OFF)" +.endif + +.if DSWS2 + // ---- role counts (lifted from coop's `.ifndef NCOMP` etc., gated under DSWS2) ---- + .ifndef NCOMP + .set NCOMP, 4 // compute waves (fat, dyn-grow). Compute floor >= 1. + .endif + .ifndef NAFEED + .set NAFEED, 2 // A-feed waves (lean). Feed floor >= 1. + .endif + .ifndef NBFEED + .set NBFEED, 2 // B-feed waves (lean). Feed floor >= 1. + .endif + .set WAVES, (NCOMP + NAFEED + NBFEED) // total waves launched per WG (harness dims must match) +.endif + +// ============================================================================================ +// VGPR layout (lifted from occ_kernel_coop.s) -- compute frags live ABOVE the lean-32 block and are +// only touched AFTER s_alloc_vgpr NFV. Feeds/claimer stay in the lean block (v0..v31). +// ============================================================================================ +.set ACC, 32 // accumulators: FM*FN frags x 8 f32 (v32..) +.set FA, (ACC + 8*FM*FN) // compute A frags (from resident LDS): FM x 2 +.set FB, (FA + 2*FM) // compute B frags (from resident LDS): FN x 2 +.set NFV, ((FB + 2*FN + 15) & ~15) // grown footprint, rounded to a 16-VGPR dyn block (=112 @ 2x4) +.set VLEAN, 32 // lean footprint (feeds, claimer, compute pre/post rowblk) +.set BSTG, 16 // staging regs (lean block, < 32): B-feed FN-frag / A-feed FM-frag + +// ---- dyn-VGPR PRE-GROW temp-reg ceiling (coop death-cert: a >v15 src pre-grow is poison under dyn). +// Gate every PRE-grow-reachable LDS/atomic temp to v11/v14 (INTERIOR to the launched 16-VGPR block). ---- +.if DYNVGPR + .set RG_A, 11 // lds_get / fetch_add address + .set RG_D, 14 // lds_get / fetch_add data+return + .set RP_A, 11 // lds_put address + .set RP_D, 14 // lds_put data +.else + .set RG_A, 27 + .set RG_D, 28 + .set RP_A, 28 + .set RP_D, 29 +.endif + +// ============================================================================================ +// LDS helper macros (s49 = exec save; v2 = lane = tid&31, set in prologue). +// ============================================================================================ +.macro lds_get sdst, off // wave-uniform read LDS[off] -> scalar sdst + v_mov_b32 v[RG_A], \off + ds_load_b32 v[RG_D], v[RG_A] + s_wait_dscnt 0x0 + v_readfirstlane_b32 \sdst, v[RG_D] +.endm +.macro lds_get_r sdst, saddr // wave-uniform read LDS[saddr] (RUNTIME addr in a sreg) -> sdst + v_mov_b32 v[RG_A], \saddr + ds_load_b32 v[RG_D], v[RG_A] + s_wait_dscnt 0x0 + v_readfirstlane_b32 \sdst, v[RG_D] +.endm +.macro lds_put off, ssrc // lane-0-of-wave writes scalar ssrc -> LDS[off] + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lput_skip\@ + v_mov_b32 v[RP_A], \off + v_mov_b32 v[RP_D], \ssrc + ds_store_b32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lput_skip\@: + s_mov_b32 exec_lo, s49 +.endm +.macro lds_fetch_add sdst, off, val // sdst <- old LDS[off]; LDS[off]+=val (lane-0 atomic, broadcast) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lfa_skip\@ + v_mov_b32 v[RP_A], \off + v_mov_b32 v[RP_D], \val + ds_add_rtn_u32 v[RP_D], v[RP_A], v[RP_D] // v[RP_D] <- old; LDS[off] += val + s_wait_dscnt 0x0 +.Lfa_skip\@: + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 \sdst, v[RP_D] // broadcast lane-0's old value +.endm +.macro lds_inc off // lane-0-of-wave LDS[off] += 1 (no return) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Linc_skip\@ + v_mov_b32 v[RP_A], \off + v_mov_b32 v[RP_D], 1 + ds_add_u32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Linc_skip\@: + s_mov_b32 exec_lo, s49 +.endm +// ---- FIX 1a ring: RUNTIME-address variants (slot-indexed counters live at SLOTC_BASE+slot*32+field, +// a runtime scalar). Mirror lds_fetch_add / lds_inc but take the address in a sreg. ---- +.macro lds_fetch_add_r sdst, saddr, val // sdst <- old LDS[saddr]; LDS[saddr]+=val (lane-0 atomic, bcast) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lfar_skip\@ + v_mov_b32 v[RP_A], \saddr + v_mov_b32 v[RP_D], \val + ds_add_rtn_u32 v[RP_D], v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lfar_skip\@: + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 \sdst, v[RP_D] +.endm +.macro lds_inc_r saddr // lane-0-of-wave LDS[saddr] += 1 (RUNTIME addr, no return) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lincr_skip\@ + v_mov_b32 v[RP_A], \saddr + v_mov_b32 v[RP_D], 1 + ds_add_u32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lincr_skip\@: + s_mov_b32 exec_lo, s49 +.endm +// lds_put_r (RUNTIME-addr write) is also defined inside the .if DSWS2_CONV||DSWS2_ENVELOPE block below; +// the ring needs it at CONV=0/ENV=0, so define an identical copy here, guarded to avoid a double-def +// when either gate is on (the ring is always built CONV=0 ENV=0). +.if !(DSWS2_CONV || DSWS2_ENVELOPE) +.macro lds_put_r saddr, ssrc // lane-0 write ssrc -> LDS[saddr] (RUNTIME addr) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lputr_skip\@ + v_mov_b32 v[RP_A], \saddr + v_mov_b32 v[RP_D], \ssrc + ds_store_b32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lputr_skip\@: + s_mov_b32 exec_lo, s49 +.endm +.endif + +// ============================================================================================ +// Super-tile decode + resident A/B staging macros (A3..A6). Decode (Naming/symbols): +// ksi = sti & mask ; t = sti >> shift ; mblk = t / NTL ; tcol = t % NTL. +// FIX 1(d): n_kseg is ALWAYS a power of two (it's KT >> NKSEG_SHIFT, both compile-time-shift-derived), +// so the sti->(t,ksi) split is an exact shift/mask -- no magic-div, no n_kseg==1 special-case (shift=0, +// mask=0 falls out of the general path for free: ksi=0, t=sti). /NTL still goes via magic (s12), +// unsigned-division mul_hi (coop GENDIV idiom), since NTL is not generally a power of two. +// ============================================================================================ +.macro DECODE_STI // in: s17=sti, s67=mask, s68=shift ; out: s19=mblk s30=tcol s31=ksi ; clob: s18,s36 +.if KMAJOR + s_mul_hi_u32 s31, s17, s76 // ksi = sti / TOTAL (magic-div, magic_TOTAL in s76; K-major high bits) + s_mul_i32 s36, s31, s11 // ksi * TOTAL + s_sub_u32 s18, s17, s36 // t = sti - ksi*TOTAL (low) +.else + s_and_b32 s31, s17, s67 // ksi = sti & mask (mask-bounded -> ksi in [0,n_kseg-1]) + s_lshr_b32 s18, s17, s68 // t = sti >> shift +.endif +.if SAFEPROBE + // brick-PROOF ti clamp (the "future ti clamp" line 752 promised; COOP_STATUS.md:145 racy-garbage-ti->OOB). + // A racy/torn sti read (during the claimer's per-super-tile republish) can decode a garbage t -> garbage + // mblk/tcol -> the A/B/C SCALAR base goes out of buffer -> gfxhub page fault -> MODE1 brick. SAFEPROBE + // already pins the per-lane vaddr (v8/v9/v10); this pins the tile index too, so EVERY global address is + // provably in-buffer. s11=TOTAL is userdata, never clobbered. s36 is DECODE_STI scratch (rewritten below). + s_sub_u32 s36, s11, 1 // TOTAL-1 + s_min_u32 s18, s18, s36 // t clamped to [0,TOTAL-1] -> mblk in-bounds) +.endif + s_mul_hi_u32 s19, s18, s12 // mblk = t / NTL + s_mul_i32 s36, s19, s13 // mblk * NTL + s_sub_u32 s30, s18, s36 // tcol = t - mblk*NTL +.endm + +// RESIDENT B FRAG LAYOUT: B frag (kstep ks, frag f) at BRES_OFF + (ks*FN + f)*256 +// (each frag = the SAME 256B block coop stores per B-ring slot; lane*8 vaddr base = v9). +// Built here as: dst vbase = v9 + BRES_OFF + f*256 , ds_store offset:(ks*FN*256). +// B global addr (lift coop B-feed): Bshuf + tcol*(FN*256=s14) + (seg k0)* [ksi*KSEG_STEPS*(NT*256=s10)] +// + f*256 (frag, folded into saddr) + ks*(NT*256=s10) (k-step, folded into saddr). +.macro BSTAGE // in: s30=tcol s31=ksi ; clob: s20,s21,s23,s25,s26,s27,v13,v[BSTG..] + s_mul_i32 s20, s30, s14 // tcol * FN*256 + s_mul_i32 s21, s31, KSEG_STEPS // ksi * KSEG_STEPS + s_mul_i32 s21, s21, s10 // * NT*256 -> segment k-start byte offset + s_add_u32 s20, s20, s21 + s_add_u32 s20, s4, s20 + s_addc_u32 s21, s5, 0 // s[20:21] = B base (tcol,ksi, seg k-step 0) +.Lbcl\@: + lds_fetch_add s23, BFRAG_NEXT_OFF, 1 // claim frag f + s_cmp_ge_u32 s23, FN + s_cbranch_scc1 .Lbsd\@ // f>=FN -> all frags claimed + s_lshl_b32 s25, s23, 8 // f*256 + s_add_u32 s26, s20, s25 + s_addc_u32 s27, s21, 0 // s[26:27] = frag f base (seg k0) + v_add_nc_u32 v13, v9, BRES_OFF + v_add_nc_u32 v13, v13, s25 // resident B dst vbase for frag f + .set ks, 0 + .rept KSEG_STEPS + global_load_tr_b64 v[BSTG+ks*2:BSTG+ks*2+1], v9, s[26:27] + s_add_u32 s26, s26, s10 // next k-step (last iter over-advances; unused) + s_addc_u32 s27, s27, 0 + .set ks, ks+1 + .endr + s_wait_loadcnt 0x0 + .set ks, 0 + .rept KSEG_STEPS + ds_store_b64 v13, v[BSTG+ks*2:BSTG+ks*2+1] offset:(ks*FN*256) + .set ks, ks+1 + .endr + s_wait_dscnt 0x0 + lds_inc BFRAG_DONE_OFF // frag f STORED -> publish completion (compute gates on this) + s_branch .Lbcl\@ +.Lbsd\@: +.endm + +// RESIDENT A FRAG LAYOUT: A frag (kstep ks, rowblk r, mi) at ARES_OFF + ((ks*G + r)*FM + mi)*256 +// Built as: dst vbase = v9 + ARES_OFF + r*(FM*256) , ds_store offset:((ks*G*FM + mi)*256). +// A global addr (lift coop compute/A-feed): A + rowblk_abs*(16*FM)*K + mi*16*K + koff, rowblk_abs=mblk*G+r, +// koff = ksi*SEGK (segment K byte offset, fp8 1B/elem), k-step within segment via global offset:ks*16. +.macro ASTAGE // in: s19=mblk s31=ksi ; clob: s22,s23,s25,s32,s36,s40,s41,s44,s45,v13,v[BSTG..] + s_lshl_b32 s32, s9, 4 // rowstride16 = 16*K +.Lacl\@: + lds_fetch_add s23, AROW_NEXT_OFF, 1 // claim rowblk r + s_cmp_ge_u32 s23, G + s_cbranch_scc1 .Lasd\@ + s_mul_i32 s36, s19, G + s_add_u32 s36, s36, s23 // rowblk_abs = mblk*G + r + s_mul_i32 s22, s36, (16*FM) + s_mul_i32 s22, s22, s9 // rowblk_abs*(16*FM)*K + s_mul_i32 s25, s31, SEGK // ksi*SEGK (segment K byte offset) + s_add_u32 s22, s22, s25 + s_add_u32 s40, s2, s22 + s_addc_u32 s41, s3, 0 // s[40:41] = A base (rowblk_abs, mi0, seg k0) + s_mul_i32 s25, s23, (FM*256) // r*FM*256 + v_add_nc_u32 v13, v9, ARES_OFF + v_add_nc_u32 v13, v13, s25 // resident A dst vbase for rowblk r + .set mi, 0 + .rept FM + .if mi == 0 + s_mov_b32 s44, s40 + s_mov_b32 s45, s41 + .else + s_add_u32 s44, s44, s32 // += 16*K (next M-frag) + s_addc_u32 s45, s45, 0 + .endif + .set ks, 0 + .rept KSEG_STEPS + global_load_b64 v[BSTG:BSTG+1], v8, s[44:45] offset:(ks*16) + s_wait_loadcnt 0x0 + ds_store_b64 v13, v[BSTG:BSTG+1] offset:((ks*G*FM + mi)*256) + s_wait_dscnt 0x0 + .set ks, ks+1 + .endr + .set mi, mi+1 + .endr + lds_inc AROW_DONE_OFF // rowblk r fully STAGED -> publish completion + s_branch .Lacl\@ +.Lasd\@: +.endm + +// ============================================================================================ +// FIX 1a -- RING staging macros: slot-indexed BSTAGE_R / ASTAGE_R. Identical math to BSTAGE/ASTAGE +// but claim/done counters live in the per-slot control block (\scb = SLOTC_BASE + slot*32, runtime) +// and operands land in the per-slot buffer (\sob = OP_BASE + slot*OPSTRIDE, runtime; B at +// +BRES_ROFF=0, A at +ARES_ROFF). \scb and \sob are READ-only (never clobbered). Internal address +// scratch: s46/s47 (free in the feed context). ds offset immediates are vbase-relative -> unchanged. +// ============================================================================================ +.macro BSTAGE_R scb, sob // in: s30=tcol s31=ksi ; clob: s20,s21,s23,s25,s26,s27,s46,s47,v13,v[BSTG..] + s_mul_i32 s20, s30, s14 // tcol * FN*256 + s_mul_i32 s21, s31, KSEG_STEPS // ksi * KSEG_STEPS + s_mul_i32 s21, s21, s10 // * NT*256 -> segment k-start byte offset + s_add_u32 s20, s20, s21 + s_add_u32 s20, s4, s20 + s_addc_u32 s21, s5, 0 // s[20:21] = B base (tcol,ksi, seg k-step 0) + s_add_u32 s46, \scb, SL_BFNEXT // &SL_BFNEXT[slot] +.Lbclr\@: + lds_fetch_add_r s23, s46, 1 // claim frag f + s_cmp_ge_u32 s23, FN + s_cbranch_scc1 .Lbsdr\@ // f>=FN -> all frags claimed + s_lshl_b32 s25, s23, 8 // f*256 + s_add_u32 s26, s20, s25 + s_addc_u32 s27, s21, 0 // s[26:27] = frag f base (seg k0) + v_add_nc_u32 v13, v9, \sob // + slot operand base + v_add_nc_u32 v13, v13, s25 // + f*256 (BRES_ROFF = 0) + .set ks, 0 + .rept KSEG_STEPS + global_load_tr_b64 v[BSTG+ks*2:BSTG+ks*2+1], v9, s[26:27] + s_add_u32 s26, s26, s10 + s_addc_u32 s27, s27, 0 + .set ks, ks+1 + .endr + s_wait_loadcnt 0x0 + .set ks, 0 + .rept KSEG_STEPS + ds_store_b64 v13, v[BSTG+ks*2:BSTG+ks*2+1] offset:(ks*FN*256) + .set ks, ks+1 + .endr + s_wait_dscnt 0x0 + s_add_u32 s47, \scb, SL_BFDONE + lds_inc_r s47 // frag f STORED -> compute gates on SL_BFDONE==FN + s_branch .Lbclr\@ +.Lbsdr\@: +.endm + +.macro ASTAGE_R scb, sob // in: s19=mblk s31=ksi ; clob: s22,s23,s25,s32,s36,s40,s41,s44,s45,s46,s47,v13,v[BSTG..] + s_lshl_b32 s32, s9, 4 // rowstride16 = 16*K + s_add_u32 s46, \scb, SL_ARNEXT // &SL_ARNEXT[slot] +.Laclr\@: + lds_fetch_add_r s23, s46, 1 // claim rowblk r + s_cmp_ge_u32 s23, G + s_cbranch_scc1 .Lasdr\@ + s_mul_i32 s36, s19, G + s_add_u32 s36, s36, s23 // rowblk_abs = mblk*G + r + s_mul_i32 s22, s36, (16*FM) + s_mul_i32 s22, s22, s9 // rowblk_abs*(16*FM)*K + s_mul_i32 s25, s31, SEGK // ksi*SEGK + s_add_u32 s22, s22, s25 + s_add_u32 s40, s2, s22 + s_addc_u32 s41, s3, 0 // s[40:41] = A base + s_mul_i32 s25, s23, (FM*256) // r*FM*256 + v_add_nc_u32 v13, v9, \sob // + slot operand base + v_add_nc_u32 v13, v13, ARES_ROFF // + A-within-slot offset (BRES_BYTES) + v_add_nc_u32 v13, v13, s25 // + r*FM*256 + .set mi, 0 + .rept FM + .if mi == 0 + s_mov_b32 s44, s40 + s_mov_b32 s45, s41 + .else + s_add_u32 s44, s44, s32 // += 16*K (next M-frag) + s_addc_u32 s45, s45, 0 + .endif + .set ks, 0 + .rept KSEG_STEPS + global_load_b64 v[BSTG:BSTG+1], v8, s[44:45] offset:(ks*16) + s_wait_loadcnt 0x0 + ds_store_b64 v13, v[BSTG:BSTG+1] offset:((ks*G*FM + mi)*256) + s_wait_dscnt 0x0 + .set ks, ks+1 + .endr + .set mi, mi+1 + .endr + s_add_u32 s47, \scb, SL_ARDONE + lds_inc_r s47 // rowblk r STAGED -> compute gates on SL_ARDONE==G + s_branch .Laclr\@ +.Lasdr\@: +.endm + +// ============================================================================================ +// Phase-B (DSWS2_CONV) consume-point ring-occupancy sensor -- Task 3, READ-ONLY (actuation is Task 5). +// Mirrors the coop occ_a/occ_b sensor (occ = producer - consumer, sampled where the value is +// CONSUMED, not at the segment boundary). The claimer's A7 wait-done spin runs CONCURRENTLY with the +// compute drain, so it observes the ring mid-flight; at the segment boundary the resident region has +// fully drained and occ would read a stuck ~0 (permanent false-starvation) -- exactly what SPEC warns. +// +// COUNTER IDENTITIES (confirmed against the live claim/consume sites -- see report): +// producer = the STORE-completion counters the compute wave actually gates on: +// A-ring: AROW_DONE_OFF (A rowblks resident, monotonic in [0,G]; lds_inc @ ASTAGE) +// B-ring: BFRAG_DONE_OFF (B frags resident, monotonic in [0,FN]; lds_inc @ BSTAGE) +// NOT the *_NEXT claim counters: AROW_NEXT/BFRAG_NEXT overshoot the ring depth by the role +// terminal-bails (G+NAFEED / FN+NBFEED), which would break the occ <= depth bound. +// consumer = ROWBLK_NEXT_OFF, the compute rowblk-claim clock (consume progress through the super-tile: +// each claimed rowblk r consumes A(r) and re-reads all FN shared B frags). +// min-clamp: cons is clamped to prod before the subtract so the u32 result cannot underflow when the +// consume clock outruns a shallower ring (G=6 > FN=4 -> ROWBLK_NEXT can exceed BFRAG_DONE). +// INVARIANT preserved: occ_A in [0,G], occ_B in [0,FN] (nonnegative, bounded by ring depth). +// +// REGISTER DISCIPLINE (brick-critical; this path is reachable pre-grow -- a >v15 vector temp is +// OOR-poison under dyn-VGPR, SPEC S4): scalars <= s65 only (s60/s61 scratch; callers pass dst in +// [s62,s65]); the only vector temps are inside lds_get, which uses v11/v14 (INTERIOR to the launch +// 16-VGPR block) -- NO >v15 temp is introduced here. +// GATE: DSWS2_CONV || DSWS2_ENVELOPE. reserve_try + the BUDGET default are the pool-economy primitives the +// rolling envelope needs INDEPENDENTLY of role conversion (they touch only VRESV_OFF/lds_fetch_add), so the +// envelope must be able to run at CONV=0 (the isolation config). Everything in this block is macro/.set +// definition (emits ZERO bytes), so widening the gate is byte-identical at CONV=0/ENV=0 and CONV=1. +.if DSWS2_CONV || DSWS2_ENVELOPE +.macro occ_sample dst_a, dst_b // out: \dst_a=occ_A in [0,G], \dst_b=occ_B in [0,FN]; clob s60,s61 + lds_get \dst_a, AROW_DONE_OFF // prod_a: A rowblks resident (store-completion) + lds_get \dst_b, BFRAG_DONE_OFF // prod_b: B frags resident (store-completion) + lds_get s60, ROWBLK_NEXT_OFF // cons : compute rowblk-claim consume clock + s_min_u32 s61, s60, \dst_a // cons_a = min(clock, prod_a) (clamp -> no u32 underflow) + s_sub_u32 \dst_a, \dst_a, s61 // occ_A = prod_a - cons_a in [0,G] + s_min_u32 s61, s60, \dst_b // cons_b = min(clock, prod_b) + s_sub_u32 \dst_b, \dst_b, s61 // occ_B = prod_b - cons_b in [0,FN] +.endm + +// ---- DSWS2_GQUIESCE: device-scoped GLOBAL QUIESCE handshake (mirrors the green occ[20]/occ[0] pattern). +// All three ops are lane-0-masked (v2==0), exec saved/restored via s49 (the LDS-macro convention -- s49 is +// never live across a macro boundary, so it is provably free at every site these replace an lds_* op). +// vaddr = v4 (the stable occ-base per-lane offset, =0, prologue-set), data/dst = v3/v5 (occ scratch vregs, +// same as the claim/live ops). scope:SCOPE_DEV + uncached occ buffer => device-coherent visibility (the +// fix). s_wait_storecnt/loadcnt drain before proceeding so the poll observes committed bumps. +.macro gq_reset // claimer: occ[QUIESCE_GOFF] = 0 (committed before EPOCH publish) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lgqr_skip\@ + v_mov_b32 v3, 0 + global_store_b32 v4, v3, s[0:1] offset:QUIESCE_GOFF scope:SCOPE_DEV + s_wait_storecnt 0x0 +.Lgqr_skip\@: + s_mov_b32 exec_lo, s49 +.endm +.macro gq_bump // follower: occ[QUIESCE_GOFF] += 1 (one bump/wave/super-tile) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lgqb_skip\@ + v_mov_b32 v3, 1 + global_atomic_add_u32 v4, v3, s[0:1] offset:QUIESCE_GOFF scope:SCOPE_DEV + s_wait_storecnt 0x0 +.Lgqb_skip\@: + s_mov_b32 exec_lo, s49 +.endm +.macro gq_read dst // claimer: \dst = occ[QUIESCE_GOFF] (lane0 load + broadcast) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lgqrd_skip\@ + global_load_b32 v5, v4, s[0:1] offset:QUIESCE_GOFF scope:SCOPE_DEV + s_wait_loadcnt 0x0 +.Lgqrd_skip\@: + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 \dst, v5 +.endm + +// ---- Pool-T7 chunk-2 wedge localization (DIAG-only; DSWS2_CONV=0 emits nothing -> .text byte-identical). +// epoch_mark: lane-0 publishes this role's live epoch (s35) to a host-streamed occ slot so a hung dispatch +// shows how far each role advanced (stream field roles[C/A/B]). v14<=v15 (feeds/compute are lean-32 at the +// _quiesce call sites), v4=0 (occ base lane offset, prologue), s49 exec-save (LDS-macro convention). ---- +.macro epoch_mark off +.if DSWS2_CONV && DIAG + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lem_skip\@ + v_mov_b32 v14, s35 + global_store_b32 v4, v14, s[0:1] offset:\off scope:SCOPE_DEV +.Lem_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// bail_mark: PER-WAVE localization mark. Lane-0 writes this wave's epoch (s35) to occ[BAIL_BASE + wid*4] +// (runtime vaddr since the offset depends on wid=s24). s48 scratch, s49 exec-save (macro-local; free at the +// _quiesce bail sites), v13 vaddr, v14 data (both <=v15; the wave is lean-32 at every bail site). One-shot +// per super-tile -> negligible perturbation vs the DIAG per-spin claimer stores. Enabled by DIAG OR BAILMARK. +.macro bail_mark +.if DSWS2_CONV && (DIAG || DSWS2_BAILMARK) + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lbmk_skip\@ + s_lshl_b32 s48, s24, 2 // wid*4 + s_add_u32 s48, s48, BAIL_BASE // occ byte offset for THIS wave + v_mov_b32 v13, s48 // vaddr = per-wave byte offset (lane0) + v_mov_b32 v14, s35 // data = this wave's current epoch + global_store_b32 v13, v14, s[0:1] scope:SCOPE_DEV +.Lbmk_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// -------------------------------------------------------------------------------------------- +// Phase-B controller thresholds + sum-envelope budget (Task 4). EPOCH_SHIFT mirrors coop / +// occ_dispatch (epoch = segcnt >> EPOCH_SHIFT). BUDGET is the per-WG VGPR sum-envelope ceiling +// the reservation counter must never exceed; default = the launch reservation, which makes the +// envelope a strict conservation law (a feed->compute grow can only fit if a compute->feed shrink +// already freed the delta). Task 5 may re-tune via `-defsym BUDGET=` if per-SIMD headroom exists. +// -------------------------------------------------------------------------------------------- +.ifndef EPOCH_SHIFT + .set EPOCH_SHIFT, 3 // decision clock: epoch = segcnt >> EPOCH_SHIFT (small = reactive) +.endif +.ifndef BUDGET +.if DSWS2_ENVELOPE + .set BUDGET, (WAVES*VLEAN + PEAK_CONC*(NFV-VLEAN)) // rolling: lean floor + concurrent-peak headroom +.else + .set BUDGET, (NCOMP*NFV + (NAFEED+NBFEED)*VLEAN) // = VRESV_OFF init (conservation ceiling) +.endif +.endif + +.if DSWS2_CONV +// compile-time no-parking invariant: every launched wave must fit lean at once +.if (WAVES * VLEAN) > BUDGET + .error "WAVES*VLEAN exceeds BUDGET — pool cannot stay all-lean (parking is out of scope)" +.endif +.endif +.if DSWS2_ENVELOPE +// forward-progress: the budget must admit at least one concurrent peak or a claimed wave can never grow +.if (WAVES*VLEAN + (NFV-VLEAN)) > BUDGET + .error "ENVELOPE: BUDGET admits < 1 concurrent peak — forward progress impossible" +.endif +.endif + +// try_gate: the lock-free single-winner conversion ticket (transcribed VERBATIM from occ_kernel_coop.s, +// which transcribes dsws_ctrl_model.cpp gate_try_win + epoch_of EXACTLY). E = segcnt>>EPOCH_SHIFT. +// gate[dir] holds the last epoch dir fired. Among many waves racing the same (gtry_gate->reserve_try point -- +// occ_sample's s62/s63 result is consumed into `dir` BEFORE this runs), v5/v6/v7 (<=v15: pre-grow / +// lean-safe). CAS operand order (gfx1201, GCN order -- NOT flipped, KG 9ed04f3c): +// ds_cmpstore_rtn_b32 vdst,vaddr,vNEW,vCMP -> MEM=(MEM==vCMP)?vNEW:MEM, vdst<-old. So vsrc0=E (new), +// vsrc1=g (compare). WIN iff returned-old == g. (Swapping them leaves gate stuck so old==g for ALL +// racers -> every racer "wins" -> would-win ~= NCOMP*epochs instead of ~= epochs.) +.macro try_gate dir, swin + lds_get s62, SEGCNT_OFF // E = epoch_of(segcnt, EPOCH_SHIFT) + s_lshr_b32 s62, s62, EPOCH_SHIFT + lds_get s63, (GATE_OFF + (\dir)*4) // g = gate[dir] + s_mov_b32 \swin, 0 + s_cmp_ge_u32 s63, s62 // g >= E -> dir already fired this/later epoch -> lose + s_cbranch_scc1 .Ltg_done\@ + s_mov_b32 s65, exec_lo // lane0-only CAS (one ticket attempt per WAVE) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltg_restore\@ + v_mov_b32 v5, (GATE_OFF + (\dir)*4) // vaddr = &gate[dir] + v_mov_b32 v6, s63 // v6 = g (vsrc1 = CMP/expected) + v_mov_b32 v7, s62 // v7 = E (vsrc0 = NEW value to store) + ds_cmpstore_rtn_b32 v6, v5, v7, v6 // gate[dir] = (gate[dir]==g) ? E : gate[dir]; v6 <- old + s_wait_dscnt 0x0 +.Ltg_restore\@: + s_mov_b32 exec_lo, s65 + v_readfirstlane_b32 s64, v6 // s64 = old (lane0's CAS result, broadcast) + s_cmp_eq_u32 s64, s63 // WIN iff old == g (we were the swapper) + s_cbranch_scc0 .Ltg_done\@ + s_mov_b32 \swin, 1 +.Ltg_done\@: +.endm + +// reserve_try: the VGPR sum-envelope reservation (transcribes reserve_grow, dsws_ctrl_model.cpp:47). +// Reserve first (atomic add of SIGNED \delta on vgpr_reserved), then validate prev+delta <= BUDGET; +// on over-budget cleanly UNDO (atomic add of -\delta) and reject. The LDS atomic serializes the <=2 +// concurrent grows an epoch permits: the second to validate sees the first's reservation and backs off. +// GROW (feed->compute): pass \delta = +(NFV-VLEAN). Over-budget -> undo, \won=0 (stay in role). +// SHRINK(compute->feed): pass \delta = -(NFV-VLEAN). new = prev+delta < prev <= BUDGET, so the +// validate branch is a proven no-op -> \won=1 ALWAYS (shrink never fails). +// One macro, one call site (Task 5 `reserve_try delta, s_ok`); direction is the sign of \delta. +// Scratch: s62/s63 (free at the bail-commit point -- try_gate's s62..s65 are long dead by then). +.macro reserve_try delta, won + lds_fetch_add s62, VRESV_OFF, (\delta) // s62 = prev reserved; vgpr_reserved += delta + s_add_u32 s63, s62, (\delta) // s63 = new reservation = prev + delta + s_mov_b32 \won, 1 + s_cmp_le_u32 s63, BUDGET // new <= BUDGET -> commit (win); shrink always passes + s_cbranch_scc1 .Lrt_done\@ + lds_fetch_add s62, VRESV_OFF, -(\delta) // over-budget: undo the reservation, reject + s_mov_b32 \won, 0 +.Lrt_done\@: +.endm + +// -------------------------------------------------------------------------------------------- +// Phase-B (Task 5) watermark thresholds + LDS put-runtime helper + bail-time commit macros. +// Watermark decision (SPEC; mirrors coop CTRL_LOW/CTRL_HIGH, occ_dispatch DSWS_LOW/HIGH): +// occ_X < CTRL_LOW -> compute STARVED for X -> shrink a compute wave into feed-X. +// occ_X > CTRL_HIGH_X -> feed-X OVER-SERVING -> grow a feed-X wave into compute. +// occ_A in [0,G], occ_B in [0,FN] (occ_sample bounds), so the HIGH marks are per-ring-depth. +// -------------------------------------------------------------------------------------------- +.ifndef CTRL_LOW + .set CTRL_LOW, 1 // occ_X < 1 (== 0, ring empty at consume) -> starved +.endif +.ifndef CTRL_HIGH_A + .set CTRL_HIGH_A, (G-1) // occ_A > G-1 -> A-ring saturated -> A-feed over-serving +.endif +.ifndef CTRL_HIGH_B + .set CTRL_HIGH_B, (FN-1) // occ_B > FN-1 -> B-ring saturated -> B-feed over-serving +.endif + +// lds_put_r: lane-0-of-wave write scalar \ssrc -> LDS[\saddr] (RUNTIME byte offset in a sreg). Mirrors +// the coop lds_put_v idiom but takes a SCALAR address (symmetry with lds_get_r). Used by the claimer's +// Step-4 snapshot write into the runtime parity half of SNAP_BASE. Temps RP_A/RP_D are v11/v14 (<=v15, +// pre-grow safe); s49 is the exec save (matches lds_put). +.macro lds_put_r saddr, ssrc + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lputr_skip\@ + v_mov_b32 v[RP_A], \saddr + v_mov_b32 v[RP_D], \ssrc + ds_store_b32 v[RP_A], v[RP_D] + s_wait_dscnt 0x0 +.Lputr_skip\@: + s_mov_b32 exec_lo, s49 +.endm + +// conv_dec_floor: floor-guarded ATOMIC decrement of a role slot -- \ok <- 1 iff it decremented \slot_off +// (only when the current value was > 1), else 0 (floor hit; source role must keep >= 1 wave). A +// ds_cmpstore_rtn_b32 CAS loop (re-reads on a lost race), so two same-source converters in one epoch +// (e.g. compute->Afeed and compute->Bfeed both dec NCOMP_SLOT) can never drive the slot below 1. +// Clob: s52 (read value), s53 (new/CAS-return), s65 (exec save); v5/v6/v7 (<=v15, pre-grow safe). +.macro conv_dec_floor slot_off, ok + s_mov_b32 \ok, 0 +.Lcdf_retry\@: + lds_get s52, \slot_off // s52 = current source-slot count + s_cmp_le_u32 s52, 1 + s_cbranch_scc1 .Lcdf_done\@ // <=1 -> at floor, cannot convert away (ok stays 0) + s_sub_u32 s53, s52, 1 // new = old - 1 + s_mov_b32 s65, exec_lo // lane0-only CAS (one attempt per WAVE) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lcdf_restore\@ + v_mov_b32 v5, \slot_off // vaddr = &slot + v_mov_b32 v6, s52 // v6 = expected old (CMP) + v_mov_b32 v7, s53 // v7 = new value (NEW) + ds_cmpstore_rtn_b32 v6, v5, v7, v6 // slot = (slot==old)? new : slot ; v6 <- prior + s_wait_dscnt 0x0 +.Lcdf_restore\@: + s_mov_b32 exec_lo, s65 + v_readfirstlane_b32 s53, v6 // s53 = prior (lane0 CAS result, broadcast) + s_cmp_eq_u32 s53, s52 // success iff prior == expected (we were the swapper) + s_cbranch_scc0 .Lcdf_retry\@ // lost the race -> re-read and retry + s_mov_b32 \ok, 1 +.Lcdf_done\@: +.endm + +// conv_apply: the bail-time role-conversion COMMIT (SPEC 3.4 Approach A). Precondition: s58 = s_win +// (1 iff this wave won the (dir,epoch) ticket). Ordered strictly BEFORE the QUIESCE_CNT bump the +// CALLER emits after this macro (the quiesce counter is the snapshot handshake). +// ORDER: (a) floor-guarded dec of \src_slot -> (b) reserve the VGPR sum-envelope \delta (shrink +// always ok; grow may abort over BUDGET) -> (c) on ok: inc \dst_slot, flip private role reg (s59), +// s_alloc_vgpr \alloc_sz (GROW=NFV feed->compute / SHRINK=32 compute->feed) with SCC-retry -> +// (d) on floor-fail or reserve-abort: cancel, remain current role (undo the source dec if a +// reservation abort happened after the dec). +// PRE-GROW OOR WINDOW (SPEC 4, #1 brick risk): the wave is lean-32 on entry; every LDS/atomic temp +// read before the s_alloc_vgpr GROW is <=v15 (occ_sample/try_gate v5/v6/v7 + v11/v14; conv_dec_floor +// v5/v6/v7; lds_fetch_add v11/v14) and every carried scalar is <=s65. NO >v15 source before GROW. +// Clob: s52,s53,s54 (+ conv_dec_floor / reserve_try scratch); s59 = new role slot id (record). +.macro conv_apply src_slot, dst_slot, delta, alloc_sz + s_cmp_eq_u32 s58, 0 + s_cbranch_scc1 .Lca_skip\@ // lost the ticket -> no conversion this bail + conv_dec_floor \src_slot, s54 // (a) floor-guarded atomic dec of source slot + s_cmp_eq_u32 s54, 0 + s_cbranch_scc1 .Lca_skip\@ // floor-fail (source at 1) -> cancel, remain current role + reserve_try (\delta), s53 // (b) reserve VGPR envelope (grow may abort; shrink ok) + s_cmp_eq_u32 s53, 0 + s_cbranch_scc0 .Lca_commit\@ + lds_fetch_add s52, \src_slot, 1 // (d) reserve aborted: UNDO the source dec, cancel + s_branch .Lca_skip\@ +.Lca_commit\@: + lds_fetch_add s52, \dst_slot, 1 // (c) inc dest slot (unbounded -> plain atomic add) + s_mov_b32 s59, \dst_slot // flip private current-role reg (records new role slot id) +.if DIAG || TRACE + // conversion-commit counter (proves a wave ACTUALLY switched role). Lean-32 pre-grow here -> v3/v4<=v15 + // OOR-safe; s49 exec-save (macro-local). (DIAG||TRACE)-gated -> DSWS2_CONV/DIAG=0/TRACE=0 byte-identical. + s_mov_b32 s49, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lca_cm_skip\@ + v_mov_b32 v3, 1 + global_atomic_add_u32 v4, v3, s[0:1] offset:CONVCNT_OFF scope:SCOPE_DEV // occ[48] += 1 + s_wait_storecnt 0x0 +.Lca_cm_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.if CONV_COOLDOWN > 0 + s_mov_b32 s66, CONV_COOLDOWN // Task 4: committed conversion -> arm cooldown +.endif + // ---- s_alloc_vgpr resize: THE pre-grow OOR window closes here; all reads above were <=v15 ---- +.Lca_alloc\@: + s_alloc_vgpr \alloc_sz // GROW(NFV) / SHRINK(32); SCC-retry (brick-class rule) + s_cbranch_scc0 .Lca_alloc\@ +.Lca_skip\@: +.endm +.endif + +// ============================================================================================ +// TFPROBE wall-span capture (TF throughput probe). Realtime-tick min/max into occ[2]/occ[3], +// mirroring occ_kernel_coop.s's proven timer idiom. Each wave stamps lane-0 only (exec-masked +// via s49, the DSWS exec-save convention); base addr v4==0 holds kernel-wide (prologue, line +// ~765; invariant per the "v4=0 occ base lane offset" note). s[30:31] free at entry (DECODE_STI +// outputs, computed only inside role bodies) and dead at every terminal. Emits ZERO bytes at +// TFPROBE=0 -> production .text byte-identical. +// ============================================================================================ +.macro tfspan op:req, off:req // op = min (entry, occ[2]/off 8) | max (exit, occ[3]/off 12) +.if TFPROBE + s_sendmsg_rtn_b64 s[30:31], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + v_cmp_eq_u32 vcc_lo, 0, v2 // lane 0 of each wave only (v2 = tid & 31) + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltfspan_skip\@ + v_mov_b32 v5, s30 // low 32 bits of the realtime tick + global_atomic_\op\()_u32 v4, v5, s[0:1] offset:\off scope:SCOPE_DEV +.Ltfspan_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// ============================================================================================ +// PHASEPROBE: in-kernel per-phase tick timer for the COMPUTE wave (the critical path). s77 holds +// this wave's last-stamp RTC (low 32b). phase_reset seeds it; phase_stamp accumulates (now-s77) +// into occ[\off] and re-seeds. Lane-0-only atomic add; occ slots are ABOVE the per-chunk memset +// so they sum over the whole run. Scratch s62/s63 (RTC), s64 (delta) -- all free in CONV=0 compute. +// Six phases -> six occ accumulators (bytes 256..276): +.set PH_FOLLOW_OFF, 256 // occ[64]: waiting on claimer to publish next super-tile +.set PH_STAGE_OFF, 260 // occ[65]: waiting on A/B feeds to stage this super-tile +.set PH_GROW_OFF, 264 // occ[66]: claim rowblk + s_alloc_vgpr GROW 32->112 +.set PH_WMMA_OFF, 268 // occ[67]: LDS frag loads + v_wmma compute +.set PH_FLUSH_OFF, 272 // occ[68]: global_atomic_add_f32 C flush (split-K reduction) +.set PH_SHRINK_OFF, 276 // occ[69]: s_alloc_vgpr SHRINK 112->32 +// Per-wave phase accumulators live in SGPRs s78..s83 (NO per-stamp store -> zero memory perturbation, no +// s_wait_storecnt pollution). s77 = last-stamp RTC. phase_flush emits them ONCE at compute retire. +.macro phase_reset +.if PHASEPROBE + s_mov_b32 s78, 0 + s_mov_b32 s79, 0 + s_mov_b32 s80, 0 + s_mov_b32 s81, 0 + s_mov_b32 s82, 0 + s_mov_b32 s83, 0 + s_sendmsg_rtn_b64 s[62:63], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + s_mov_b32 s77, s62 +.endif +.endm +.macro phase_stamp acc:req // \acc += (now - s77); s77 = now (pure scalar, no store) +.if PHASEPROBE + s_sendmsg_rtn_b64 s[62:63], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + s_sub_u32 s64, s62, s77 // delta (mod 2^32; phase << 2^32 so wrap-safe) + s_mov_b32 s77, s62 + s_add_u32 \acc, \acc, s64 +.endif +.endm +.macro phase_flush // lane0 atomic-adds s78..s83 -> occ[64..69]; drained here (not the hot loop) +.if PHASEPROBE + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lphf_skip\@ + v_mov_b32 v5, s78 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_FOLLOW_OFF scope:SCOPE_DEV + v_mov_b32 v5, s79 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_STAGE_OFF scope:SCOPE_DEV + v_mov_b32 v5, s80 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_GROW_OFF scope:SCOPE_DEV + v_mov_b32 v5, s81 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_WMMA_OFF scope:SCOPE_DEV + v_mov_b32 v5, s82 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_FLUSH_OFF scope:SCOPE_DEV + v_mov_b32 v5, s83 + global_atomic_add_u32 v4, v5, s[0:1] offset:PH_SHRINK_OFF scope:SCOPE_DEV + s_wait_storecnt 0x0 +.Lphf_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +// ============================================================================================ +// TRACE: per-super-tile time-series row (claimer, lane 0). Written once per super-tile at the +// quiesce-satisfied drain-exit (.Lqc_q_ok). Captures the adaptive wave-role economy over time: +// the LIVE role slots (do waves convert?), the per-super-tile ring-occupancy PEAK (s73/s74, +// tracked across the wait_done spins), the cumulative conversion count, and the envelope vresv. +// 16 u32/row -> buffer[segcnt*64]; bounded by MAXROWS (s72). Emits ZERO bytes at TRACE=0. +// Persistent trace regs: s70:s71 = buffer VA, s72 = MAXROWS, s73/s74 = ring occA/occB peak. +// ============================================================================================ +.macro alllive_dec // TRACE: --live on wave exit (pairs with the entry ++ for peak-concurrent) +.if TRACE + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lalld_skip\@ + v_mov_b32 v3, -1 + global_atomic_add_u32 v4, v3, s[0:1] offset:ALLLIVE_OFF scope:SCOPE_DEV +.Lalld_skip\@: + s_mov_b32 exec_lo, s49 +.endif +.endm + +.macro trace_row +.if TRACE + // claim a GLOBALLY-unique row index (all WGs' claimers share occ[55]) -> no per-WG SEGCNT collision. + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + v_mov_b32 v14, 1 + global_atomic_add_u32 v15, v4, v14, s[0:1] offset:TRACE_IDX_OFF th:TH_ATOMIC_RETURN scope:SCOPE_DEV // v15=old idx, v4=addr(0), v14=data(1) + s_wait_loadcnt 0x0 + s_mov_b32 exec_lo, s49 + v_readfirstlane_b32 s52, v15 // s52 = unique row index (old value returned by the atomic) + s_cmp_ge_u32 s52, s72 // row >= MAXROWS -> skip (buffer bound) + s_cbranch_scc1 .Ltrow_skip\@ + s_lshl_b32 s53, s52, 6 // row * TRACE_ROW_BYTES(64) + s_add_u32 s60, s70, s53 + s_addc_u32 s61, s71, 0 // s[60:61] = row base VA + s_sendmsg_rtn_b64 s[58:59], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + lds_get s55, SEGCNT_OFF // this WG's per-WG super-tile counter (data field, not index) + v_mov_b32 v14, s58 // 0 tick_lo + v_mov_b32 v15, s55 // 1 segcnt (per-WG) + v_mov_b32 v16, s35 // 2 epoch + lds_get s54, NCOMP_SLOT + v_mov_b32 v17, s54 // 3 nComp (live role slot) + lds_get s54, NAFEED_SLOT + v_mov_b32 v18, s54 // 4 nAfeed + lds_get s54, NBFEED_SLOT + v_mov_b32 v19, s54 // 5 nBfeed + v_mov_b32 v20, s73 // 6 occA peak (across wait_done spins) + v_mov_b32 v21, s74 // 7 occB peak + global_load_b32 v22, v4, s[0:1] offset:CONVCNT_OFF scope:SCOPE_DEV // 8 convCount (cumulative) + lds_get s54, VRESV_OFF + v_mov_b32 v23, s54 // 9 vresv (envelope budget) + v_mov_b32 v24, s17 // 10 sti (claimed super-tile id) + lds_get s54, QUIESCE_CNT_OFF + v_mov_b32 v25, s54 // 11 quiesce (final) + v_mov_b32 v26, s59 // 12 tick_hi + v_mov_b32 v27, s69 // 13 chunkHi (context) + v_mov_b32 v28, s75 // 14 wg_id (which workgroup's economy this row belongs to) + v_mov_b32 v29, 0 // 15 reserved + s_wait_loadcnt 0x0 // convCount load drained before the row store + v_cmp_eq_u32 vcc_lo, 0, v2 // lane 0 of the claimer writes the row + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltrow_wskip\@ + global_store_b128 v4, v[14:17], s[60:61] offset:0 scope:SCOPE_DEV + global_store_b128 v4, v[18:21], s[60:61] offset:16 scope:SCOPE_DEV + global_store_b128 v4, v[22:25], s[60:61] offset:32 scope:SCOPE_DEV + global_store_b128 v4, v[26:29], s[60:61] offset:48 scope:SCOPE_DEV +.Ltrow_wskip\@: + s_mov_b32 exec_lo, s49 +.Ltrow_skip\@: +.endif +.endm + +// ============================================================================================ +// KERNEL +// ============================================================================================ + .text + .globl occ_kernel + .p2align 8 + .type occ_kernel,@function +occ_kernel: + // ---- FIX 1(b,c): derive n_kseg from KT (s8) + the compile-time NKSEG_SHIFT, then the shift/mask + // decode pair, into the reserved high SGPRs s66/s67/s68 BEFORE any clobber (SAFEPROBE below reuses + // s16 purely as scratch; the role bodies keep all transients <= s65). No v2 kernargs are read here + // anymore -- s15/s16/s17 are NOT hardware-preloaded under USER_SGPR=15 (see KERNARG CONTRACT above). ---- + s_lshr_b32 s66, s8, NKSEG_SHIFT // n_kseg = KT >> NKSEG_SHIFT (KT=s8) + s_ff1_i32_b32 s68, s66 // shift = log2(n_kseg) (bit index of the single set bit; n_kseg=1 -> 0) + s_sub_u32 s67, s66, 1 // mask = n_kseg - 1 + // ---- identity (lifted from coop prologue; v0=tid hardware-preloaded) ---- + v_lshrrev_b32 v1, 5, v0 // wid = tid >> 5 + v_and_b32 v2, 31, v0 // lane = tid & 31 + v_and_b32 v6, 15, v0 // lane & 15 (A vaddr) + v_mov_b32 v4, 0 +.if KMAJOR + global_load_b32 v3, v4, s[0:1] offset:248 scope:SCOPE_DEV // occ[62] = magic(TOTAL), host-written + s_wait_loadcnt 0x0 + v_readfirstlane_b32 s76, v3 // s76 = magic_TOTAL, persistent for every DECODE_STI (K-major) +.endif + // ---- per-lane address constants (mbgemm-identical; dyn-VGPR arming compatible) ---- + v_mul_lo_u32 v8, v6, s9 // (lane&15)*K + v_bfe_u32 v7, v0, 4, 1 + v_lshlrev_b32 v7, 3, v7 + v_add_nc_u32 v8, v8, v7 // v8 = A vaddr = (lane&15)*K + colhi*8 + v_lshlrev_b32 v9, 3, v2 // v9 = B/ds vaddr = lane*8 + v_lshlrev_b32 v10, 5, v2 // v10 = C store vaddr = lane*32 +.if SAFEPROBE + // brick-PROOF: clamp the per-lane VECTOR address regs to a loose upper bound (>= true max) so even a + // grow-corrupted vaddr cannot push a global access past the data+guard (pairs with the future ti clamp). + s_lshl_b32 s16, s9, 4 // 16*K (>= v8 max = (lane&15)*K + colhi*8 = 15*K+8) + v_min_u32 v8, s16, v8 // clamp A vaddr + v_min_u32 v9, 0x100, v9 // clamp B/ds vaddr (256 >= lane*8 max 248) + v_min_u32 v10, 0x400, v10 // clamp C vaddr (1024 >= lane*32 max 992) +.endif + + tfspan min, 8 // TFPROBE: every wave stamps occ[2] = min entry tick (wall-span start) +.if TRACE + // total-occupancy: every wave ++live at entry, atomic-max the peak concurrent resident count (occ[1]). + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s49, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lall_enter_skip + v_mov_b32 v3, 1 + global_atomic_add_u32 v5, v4, v3, s[0:1] offset:ALLLIVE_OFF th:TH_ATOMIC_RETURN scope:SCOPE_DEV + s_wait_loadcnt 0x0 + v_add_nc_u32 v5, v5, 1 + global_atomic_max_u32 v4, v5, s[0:1] offset:4 scope:SCOPE_DEV // occ[1] = peak concurrent resident waves +.Lall_enter_skip: + s_mov_b32 exec_lo, s49 +.endif + +.if DSWS2 + // ===== DSWS v2 role branch (wid uniform per wave; scalar-only -> exec stays full for every role). + // wid == 0 -> claimer (pinned super-tile broadcaster; A3) + // wid [0,NBFEED) -> B-feed (A4) + // wid [NBFEED,NBFEED+NAFEED) -> A-feed (A5) + // wid [NBFEED+NAFEED, WAVES) -> compute (A6) ===== + // A1: every role label is just a distinct s_endpgm stub (unique s50 tag keeps them at distinct addresses). + v_readfirstlane_b32 s24, v1 // wid (uniform per wave) + s_cmp_eq_u32 s24, 0 + s_cbranch_scc1 .Ldispatcher // FIX 1a: wid0 = ring dispatcher (was .Lclaimer) +.if DSWS2_CONV && DIAG && DSWS2_TICKET_SELFTEST + // Task 4 Step 3 -- try_gate single-winner SMOKE (assemble-only stub; default off). Every non-claimer + // wave races the (dir=0) ticket ONCE and atomic-adds its win (0/1) into occ[28] (byte offset 112, + // clear of the 0/20/24/104/108 control+probe words). On GPU (Task 6, if enabled) the sum should land + // near #epochs, NOT NCOMP*#epochs -- the harness-side proof the LDS-CAS yields <=1 winner/(dir,epoch). + // v4=0 (set in prologue), v2=lane; try_gate temps v5/v6/v7 are <=v15 (pre-grow safe). wid (s24) + // survives -- try_gate touches only s62..s65 / s16. NOTE: pre-init-rendezvous placement -> a real run + // reads gate/segcnt before the claimer publishes them; fine for an assemble/smoke stub. + try_gate 0, s50 + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ltg_selftest_skip + v_mov_b32 v5, s50 // win flag (0/1) for THIS wave + global_atomic_add_u32 v4, v5, s[0:1] offset:112 scope:SCOPE_DEV // occ[28] += win +.Ltg_selftest_skip: + s_mov_b32 exec_lo, s16 +.endif +.if DSWS2_CONV +.if CONV_COOLDOWN > 0 + s_mov_b32 s66, 0 // Task 4: init cooldown ctr (un-cooled at entry) +.endif + // FIRST-time entry seeds the role reg s59 then falls into the role's FULL entry + // (.Lcompute/.Lafeed/.Lbfeed -> _alloc -> _init -> _follow). It must NOT jump to + // .Ldispatch: that trampoline lands on _follow and is correct ONLY for RE-dispatch + // (a wave that already ran _alloc/_init once). First entry via _follow would skip the + // s_alloc_vgpr 32 allocator handshake, the INITFLAG==0xACED rendezvous, and the s35=0 + // epoch seed -> followers desync and the claimer hangs in .Lclaimer_wait_done (Pool-T7 brick). + s_cmp_lt_u32 s24, NBFEED + s_cbranch_scc1 .Lseed_bfeed + s_cmp_lt_u32 s24, (NBFEED+NAFEED) + s_cbranch_scc1 .Lseed_afeed + s_mov_b32 s59, NCOMP_SLOT + s_branch .Lcompute +.Lseed_afeed: + s_mov_b32 s59, NAFEED_SLOT + s_branch .Lafeed +.Lseed_bfeed: + s_mov_b32 s59, NBFEED_SLOT + s_branch .Lbfeed +.else + s_cmp_lt_u32 s24, NBFEED + s_cbranch_scc1 .Lbfeed + s_cmp_lt_u32 s24, (NBFEED+NAFEED) + s_cbranch_scc1 .Lafeed + s_branch .Lcompute +.endif + +// ============================================================================================ +// FIX 1a -- .Ldispatcher : pinned wid-0 (was .Lclaimer). PURE COORDINATION -- NO BSTAGE (deviation +// from the old B-feed-class claimer; effective B-stager count is NBFEED-1, so pick the launch mix +// accordingly -- the exact feed:compute ratio is fix #3, not 1a). Owns: the barrier-free LDS init +// (shared frontier + BOTH slot control blocks), the completion live++/live-- (harness occ[0]==0 +// gate), the GLOBAL super-tile claim (occ[20]), and the D=2 ring fill/recycle. Registers: +// s36 = gen -- GLOBAL-monotonic publish counter (0 == "slot never published") +// s37 = fill_slot -- which slot to fill next (alternates 0,1,0,1 with D=2) +// s34 = fill_slot's control base (SLOTC_BASE + fill_slot*SLOTC_STRIDE) +// s17 = sti, s69 = chunkHi, s44/s45 = scratch, s16 = exec save +// Steady state: fill a FREE slot, publish (STI -> FILL_IDX -> SL_GEN LAST = release fence), then +// wait THAT slot READY (SL_BFDONE==FN && SL_ARDONE==G) before alternating fill_slot -- so feeds +// never split across slots and compute drains the OTHER slot throughout (STAGE/FOLLOW hidden). +// Reset of a slot's counters happens ONLY here, ONLY after the prior occupant's SL_RBDONE>=G, so +// the barrier-free reset-race is structurally impossible (see RING_SLOTS_DESIGN.md). +// occ[0]=live counter (harness polls ==0); occ[20]=global super-tile claim; occ[24]=chunkHi. +// ============================================================================================ +.Ldispatcher: +.if DYNVGPR +.Ldisp_alloc: + s_alloc_vgpr 32 // dispatcher stays lean (never grows) + s_cbranch_scc0 .Ldisp_alloc +.endif + // live++ : lane0 occ[0] += 1 + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ldisp_live + v_mov_b32 v3, 1 + global_atomic_add_u32 v4, v3, s[0:1] scope:SCOPE_DEV +.Ldisp_live: + s_mov_b32 exec_lo, s16 + // --- barrier-free LDS init: shared frontier + BOTH slot control blocks; RINGINIT = 0xACED LAST --- + lds_put FILL_IDX_OFF, 0 + lds_put DRAIN_IDX_OFF, 0 + // slot 0 control block (8 words) + lds_put (SLOTC_BASE + 0*SLOTC_STRIDE + SL_STI), 0 + lds_put (SLOTC_BASE + 0*SLOTC_STRIDE + SL_GEN), 0 + lds_put (SLOTC_BASE + 0*SLOTC_STRIDE + SL_RBNEXT), 0 + lds_put (SLOTC_BASE + 0*SLOTC_STRIDE + SL_RBDONE), 0 + lds_put (SLOTC_BASE + 0*SLOTC_STRIDE + SL_BFNEXT), 0 + lds_put (SLOTC_BASE + 0*SLOTC_STRIDE + SL_BFDONE), 0 + lds_put (SLOTC_BASE + 0*SLOTC_STRIDE + SL_ARNEXT), 0 + lds_put (SLOTC_BASE + 0*SLOTC_STRIDE + SL_ARDONE), 0 + // slot 1 control block (8 words) + lds_put (SLOTC_BASE + 1*SLOTC_STRIDE + SL_STI), 0 + lds_put (SLOTC_BASE + 1*SLOTC_STRIDE + SL_GEN), 0 + lds_put (SLOTC_BASE + 1*SLOTC_STRIDE + SL_RBNEXT), 0 + lds_put (SLOTC_BASE + 1*SLOTC_STRIDE + SL_RBDONE), 0 + lds_put (SLOTC_BASE + 1*SLOTC_STRIDE + SL_BFNEXT), 0 + lds_put (SLOTC_BASE + 1*SLOTC_STRIDE + SL_BFDONE), 0 + lds_put (SLOTC_BASE + 1*SLOTC_STRIDE + SL_ARNEXT), 0 + lds_put (SLOTC_BASE + 1*SLOTC_STRIDE + SL_ARDONE), 0 + lds_put RINGINIT_OFF, 0xACED // LAST: publishes "LDS ready" to all follower waves + // load this dispatch's chunk terminal bound (host writes occ[24] per chunk); stable -> load ONCE + global_load_b32 v6, v4, s[0:1] offset:24 scope:SCOPE_DEV + s_wait_loadcnt 0x0 + v_readfirstlane_b32 s69, v6 // s69 = chunkHi (terminal sti bound) + s_mov_b32 s36, 0 // gen = 0 (first real publish will be gen 1) + s_mov_b32 s37, 0 // fill_slot = 0 +.Ldisp_loop: + s_lshl_b32 s34, s37, 5 // fill_slot * SLOTC_STRIDE(32) + s_add_u32 s34, s34, SLOTC_BASE // s34 = fill_slot control base + // wait slot FREE: SL_GEN==0 (never published) OR SL_RBDONE>=G (prior occupant drained) +.Ldisp_wait_free: + s_sleep SLEEPN + s_add_u32 s45, s34, SL_GEN + lds_get_r s44, s45 + s_cmp_eq_u32 s44, 0 + s_cbranch_scc1 .Ldisp_free_ok + s_add_u32 s45, s34, SL_RBDONE + lds_get_r s44, s45 + s_cmp_lt_u32 s44, G + s_cbranch_scc1 .Ldisp_wait_free +.Ldisp_free_ok: + // claim next global super-tile: lane0 occ[20] += 1 (return old) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ldisp_grabbed + v_mov_b32 v3, 1 + global_atomic_add_u32 v5, v4, v3, s[0:1] offset:20 th:TH_ATOMIC_RETURN scope:SCOPE_DEV + s_wait_loadcnt 0x0 +.Ldisp_grabbed: + s_mov_b32 exec_lo, s16 + v_readfirstlane_b32 s17, v5 // sti + s_cmp_ge_u32 s17, s69 // sti >= chunkHi -> terminal + s_cbranch_scc1 .Ldisp_terminal + // reset this slot's per-super-tile counters BEFORE publishing gen (followers gate on SL_GEN) + s_add_u32 s45, s34, SL_RBNEXT + lds_put_r s45, 0 + s_add_u32 s45, s34, SL_RBDONE + lds_put_r s45, 0 + s_add_u32 s45, s34, SL_BFNEXT + lds_put_r s45, 0 + s_add_u32 s45, s34, SL_BFDONE + lds_put_r s45, 0 + s_add_u32 s45, s34, SL_ARNEXT + lds_put_r s45, 0 + s_add_u32 s45, s34, SL_ARDONE + lds_put_r s45, 0 + // publish STI first, then FILL_IDX (feeds now target this slot), then SL_GEN LAST (release fence) + s_add_u32 s45, s34, SL_STI + lds_put_r s45, s17 + lds_put FILL_IDX_OFF, s37 + s_add_u32 s36, s36, 1 // gen++ + s_add_u32 s45, s34, SL_GEN + lds_put_r s45, s36 // release: consumers observe the new occupant here + // wait THIS slot READY (fully staged) before advancing fill_slot -> feeds never split across slots +.Ldisp_wait_ready: + s_sleep SLEEPN + s_add_u32 s45, s34, SL_BFDONE + lds_get_r s44, s45 + s_cmp_lt_u32 s44, FN + s_cbranch_scc1 .Ldisp_wait_ready + s_add_u32 s45, s34, SL_ARDONE + lds_get_r s44, s45 + s_cmp_lt_u32 s44, G + s_cbranch_scc1 .Ldisp_wait_ready + s_xor_b32 s37, s37, 1 // alternate fill_slot (D=2) + s_branch .Ldisp_loop +.Ldisp_terminal: + // Publish SENTINEL into fill_slot (the confirmed-FREE slot; s34 still holds its base) + point + // FILL_IDX at it + bump its gen LAST. Feeds (via FILL_IDX) and compute (via the DRAIN_IDX + // frontier-advance that lands here after the OTHER slot's last real work drains) then wake and + // retire. The other slot, if it still holds live work, is drained normally first -- we do NOT + // sentinel it (that would retire compute before its rowblks are done). See RING_SLOTS_DESIGN.md. + s_add_u32 s36, s36, 1 // gen++ (sentinel publish) + s_mov_b32 s44, 0xFFFFFFFF + s_add_u32 s45, s34, SL_STI + lds_put_r s45, s44 // STI[fill_slot] = SENTINEL + lds_put FILL_IDX_OFF, s37 // feeds target the sentineled slot + s_add_u32 s45, s34, SL_GEN + lds_put_r s45, s36 // release LAST + // live-- : lane0 occ[0] -= 1 (harness completion gate fires) + v_cmp_eq_u32 vcc_lo, 0, v2 + s_mov_b32 s16, exec_lo + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Ldisp_dead + v_mov_b32 v3, -1 + global_atomic_add_u32 v4, v3, s[0:1] scope:SCOPE_DEV +.Ldisp_dead: + s_mov_b32 exec_lo, s16 + tfspan max, 12 // TFPROBE: dispatcher stamps occ[3] = max exit tick (wall-span end) + alllive_dec + s_endpgm + +// ============================================================================================ +// FIX 1a -- .Lbfeed : B-feed wave. Gates on SL_GEN[FILL_IDX] > last-seen-gen (new occupant), reads +// STI[FILL_IDX] (sentinel -> retire), stages its claimed B frags into the FILL_IDX slot's operand +// buffer via BSTAGE_R. A feed that skips a generation is SAFE: the dispatcher's wait-READY gate +// guarantees the skipped slot was fully staged by other feeds before FILL_IDX advanced off it, and +// compute gates on SL_BFDONE (a count), not on which feed contributed (see RING_SLOTS_DESIGN.md). +// Registers: s35 = last-seen gen, s38 = FILL_IDX, s48 = scb, s52 = sob, s44/s45 = scratch. +// ============================================================================================ +.Lbfeed: +.if DYNVGPR +.Lbfeed_alloc: + s_alloc_vgpr 32 + s_cbranch_scc0 .Lbfeed_alloc +.endif +.Lbfeed_init: + s_sleep 1 + lds_get s44, RINGINIT_OFF + s_cmp_eq_u32 s44, 0xACED + s_cbranch_scc0 .Lbfeed_init // wait the dispatcher's barrier-free LDS init + s_mov_b32 s35, 0 // last-seen gen (0 = none staged yet) +.Lbfeed_follow: + s_sleep SLEEPN + lds_get s38, FILL_IDX_OFF // current fill slot + s_lshl_b32 s48, s38, 5 + s_add_u32 s48, s48, SLOTC_BASE // s48 = scb (FILL_IDX control base) + s_add_u32 s45, s48, SL_GEN + lds_get_r s44, s45 // SL_GEN[FILL_IDX] + s_cmp_le_u32 s44, s35 // gen <= last-seen -> no new occupant yet -> spin + s_cbranch_scc1 .Lbfeed_follow + s_mov_b32 s35, s44 // adopt the new gen + s_add_u32 s45, s48, SL_STI + lds_get_r s17, s45 // STI[FILL_IDX] + s_cmp_eq_u32 s17, 0xFFFFFFFF // sentinel -> retire + s_cbranch_scc1 .Lretire + DECODE_STI // s30=tcol s31=ksi (mblk unused) + s_lshl_b32 s52, s38, 14 // FILL_IDX * OPSTRIDE(16384) + s_add_u32 s52, s52, OP_BASE // s52 = sob (FILL_IDX operand base) + BSTAGE_R s48, s52 + s_branch .Lbfeed_follow + +// ============================================================================================ +// FIX 1a -- .Lafeed : A-feed wave. Symmetric to .Lbfeed; stages A rowblks into FILL_IDX via ASTAGE_R. +// ============================================================================================ +.Lafeed: +.if DYNVGPR +.Lafeed_alloc: + s_alloc_vgpr 32 + s_cbranch_scc0 .Lafeed_alloc +.endif +.Lafeed_init: + s_sleep 1 + lds_get s44, RINGINIT_OFF + s_cmp_eq_u32 s44, 0xACED + s_cbranch_scc0 .Lafeed_init + s_mov_b32 s35, 0 +.Lafeed_follow: + s_sleep SLEEPN + lds_get s38, FILL_IDX_OFF + s_lshl_b32 s48, s38, 5 + s_add_u32 s48, s48, SLOTC_BASE + s_add_u32 s45, s48, SL_GEN + lds_get_r s44, s45 + s_cmp_le_u32 s44, s35 + s_cbranch_scc1 .Lafeed_follow + s_mov_b32 s35, s44 + s_add_u32 s45, s48, SL_STI + lds_get_r s17, s45 + s_cmp_eq_u32 s17, 0xFFFFFFFF + s_cbranch_scc1 .Lretire + DECODE_STI // s19=mblk s31=ksi (tcol unused) + s_lshl_b32 s52, s38, 14 + s_add_u32 s52, s52, OP_BASE + ASTAGE_R s48, s52 + s_branch .Lafeed_follow + +// ============================================================================================ +// FIX 1a -- .Lcompute : compute wave. Gates on SL_GEN[DRAIN_IDX] > last-seen-gen, waits the slot +// fully STAGED (SL_BFDONE==FN && SL_ARDONE==G), claims rowblks from SL_RBNEXT[DRAIN_IDX], grows, +// runs WMMA over the slot's resident operand buffer, flushes fp32 partials into C via +// global_atomic_add_f32 (split-K segments accumulate), shrinks, and bumps SL_RBDONE. The wave whose +// SL_RBDONE increment reaches G (old==G-1) is the CLOSER: it advances DRAIN_IDX to the other slot +// (lock-free frontier advance). Because the next slot is pre-published AND pre-staged (dispatcher +// keeps one ahead + waits READY), FOLLOW_WAIT/STAGE_WAIT collapse toward zero in steady state. +// Registers: s35=last-seen gen, s46=DRAIN_IDX, s48=scb (control base), s52=sob (operand base) -- +// all survive the whole rowblk loop; s44/s45/s47 scratch; s33=rowblk; s37/s38/s39/s28/s29 = C addr. +// ============================================================================================ +.Lcompute: +.if DYNVGPR +.Lcompute_alloc: + s_alloc_vgpr 32 // lean baseline; grow per rowblk + s_cbranch_scc0 .Lcompute_alloc +.endif +.Lcompute_init: + s_sleep 1 + lds_get s44, RINGINIT_OFF + s_cmp_eq_u32 s44, 0xACED + s_cbranch_scc0 .Lcompute_init + s_mov_b32 s35, 0 + phase_reset // PHASEPROBE: seed this compute wave's phase clock +.Lcompute_follow: + s_sleep SLEEPN + lds_get s46, DRAIN_IDX_OFF // current drain slot + s_lshl_b32 s48, s46, 5 + s_add_u32 s48, s48, SLOTC_BASE // s48 = scb (DRAIN control base) + s_add_u32 s45, s48, SL_GEN + lds_get_r s44, s45 // SL_GEN[DRAIN_IDX] + s_cmp_le_u32 s44, s35 // gen <= last-seen -> no new occupant -> spin + s_cbranch_scc1 .Lcompute_follow + s_mov_b32 s35, s44 + phase_stamp s78 // close FOLLOW_WAIT (spun waiting for the next super-tile) + s_add_u32 s45, s48, SL_STI + lds_get_r s17, s45 // STI[DRAIN_IDX] +.if PHASEPROBE + s_cmp_eq_u32 s17, 0xFFFFFFFF // sentinel -> flush phase accumulators, then retire + s_cbranch_scc0 .Lcompute_go + phase_flush + s_branch .Lretire +.Lcompute_go: +.else + s_cmp_eq_u32 s17, 0xFFFFFFFF // sentinel -> retire + s_cbranch_scc1 .Lretire +.endif + DECODE_STI // s19=mblk s30=tcol s31=ksi + s_lshl_b32 s52, s46, 14 // DRAIN_IDX * OPSTRIDE(16384) + s_add_u32 s52, s52, OP_BASE // s52 = sob (DRAIN operand base) + // wait until resident A AND B fully STAGED (B: FN frags stored, A: G rowblks stored) +.Lcompute_staged: + s_sleep SLEEPN + s_add_u32 s45, s48, SL_BFDONE + lds_get_r s44, s45 + s_cmp_lt_u32 s44, FN + s_cbranch_scc1 .Lcompute_staged + s_add_u32 s45, s48, SL_ARDONE + lds_get_r s44, s45 + s_cmp_lt_u32 s44, G + s_cbranch_scc1 .Lcompute_staged + phase_stamp s79 // close STAGE_WAIT (spun waiting for A/B feeds) + // C tile-term: ti = mblk*NTL + tcol ; ti*(G*FM*FN*1024) (ksi-INDEPENDENT -> split-K accumulates) + s_mul_i32 s38, s19, s13 + s_add_u32 s38, s38, s30 + s_mul_i32 s38, s38, (G*FM*FN*1024) +.Lcompute_claim: + s_add_u32 s45, s48, SL_RBNEXT + lds_fetch_add_r s33, s45, 1 // claim rowblk r in [0,G) + s_cmp_ge_u32 s33, G + s_cbranch_scc1 .Lcompute_drained +.if DYNVGPR + s_wait_loadcnt 0x0 + s_wait_storecnt 0x0 +.Lcompute_grow: + s_alloc_vgpr NFV // grow (SCC-retry guarded, brick-class rule; M=576 safe) + s_cbranch_scc0 .Lcompute_grow +.endif + phase_stamp s80 // close GROW (rowblk claim + s_alloc_vgpr 32->112) + // zero FM*FN fp32 accumulators + .set idx, 0 + .rept FM*FN + v_mov_b32 v[ACC+idx*8+0], 0 + v_mov_b32 v[ACC+idx*8+1], 0 + v_mov_b32 v[ACC+idx*8+2], 0 + v_mov_b32 v[ACC+idx*8+3], 0 + v_mov_b32 v[ACC+idx*8+4], 0 + v_mov_b32 v[ACC+idx*8+5], 0 + v_mov_b32 v[ACC+idx*8+6], 0 + v_mov_b32 v[ACC+idx*8+7], 0 + .set idx, idx+1 + .endr + // resident operand bases: v9 + sob + region [+ r*FM*256 for A] (BRES_ROFF=0, ARES_ROFF=BRES_BYTES) + v_add_nc_u32 v12, v9, s52 + s_mul_i32 s37, s33, (FM*256) + v_add_nc_u32 v13, v9, s52 + v_add_nc_u32 v13, v13, ARES_ROFF + v_add_nc_u32 v13, v13, s37 + // WMMA over the SEGK segment (KSEG_STEPS k-steps); read resident B(ks) + A(ks,r) from LDS + .set ks, 0 + .rept KSEG_STEPS + .set ni, 0 + .rept FN + ds_load_b64 v[FB+ni*2:FB+ni*2+1], v12 offset:((ks*FN+ni)*256) + .set ni, ni+1 + .endr + .set mi, 0 + .rept FM + ds_load_b64 v[FA+mi*2:FA+mi*2+1], v13 offset:((ks*G*FM+mi)*256) + .set mi, mi+1 + .endr + s_wait_dscnt 0x0 + .set mi, 0 + .rept FM + .set ni, 0 + .rept FN + v_wmma_f32_16x16x16_fp8_fp8 v[ACC+(mi*FN+ni)*8:ACC+(mi*FN+ni)*8+7], v[FA+mi*2:FA+mi*2+1], v[FB+ni*2:FB+ni*2+1], v[ACC+(mi*FN+ni)*8:ACC+(mi*FN+ni)*8+7] + .set ni, ni+1 + .endr + .set mi, mi+1 + .endr + .set ks, ks+1 + .endr + phase_stamp s81 // close WMMA (LDS frag loads + v_wmma over the segment) + // flush: C base = C + ti-term + r*(FM*FN*1024) ; per (frag,elem) atomic-add one fp32 (vaddr v10=lane*32) + s_mul_i32 s39, s33, (FM*FN*1024) + s_add_u32 s39, s38, s39 + s_add_u32 s28, s6, s39 + s_addc_u32 s29, s7, 0 +.if NOCFLUSH == 0 + .set frag, 0 + .rept FM*FN + .set e, 0 + .rept 8 + .if CSTORE + global_store_b32 v10, v[ACC+frag*8+e], s[28:29] offset:(frag*1024 + e*4) scope:SCOPE_DEV // probe: non-atomic, same count + .else + global_atomic_add_f32 v10, v[ACC+frag*8+e], s[28:29] offset:(frag*1024 + e*4) scope:SCOPE_DEV + .endif + .set e, e+1 + .endr + .set frag, frag+1 + .endr +.endif + s_wait_storecnt 0x0 // atomic-adds READ ACC -> must drain before shrink frees ACC + phase_stamp s82 // close FLUSH (global_atomic_add_f32 C reduction + drain) +.if DYNVGPR +.Lcompute_shrink: + s_alloc_vgpr 32 // shrink (SCC-retry guarded) + s_cbranch_scc0 .Lcompute_shrink +.endif + phase_stamp s83 // close SHRINK (s_alloc_vgpr 112->32) + // rowblk r computed+flushed: bump SL_RBDONE; the wave whose inc reaches G is the CLOSER and advances + // the drain frontier. old==G-1 detects the closer atomically (exactly one wave). + s_add_u32 s45, s48, SL_RBDONE + lds_fetch_add_r s44, s45, 1 // s44 = old SL_RBDONE[DRAIN_IDX] + s_cmp_eq_u32 s44, (G-1) + s_cbranch_scc0 .Lcompute_claim // not the closer -> keep claiming (next rowblk / over-claim) + s_xor_b32 s47, s46, 1 // closer: DRAIN_IDX -> other slot (lock-free frontier advance) + lds_put DRAIN_IDX_OFF, s47 + s_branch .Lcompute_claim +.Lcompute_drained: + s_branch .Lcompute_follow // this slot drained -> re-check drain frontier / terminal + +.if DSWS2_CONV +// ============================================================================================ +// Role RE-DISPATCH trampoline (Task 3): scalar-only. Reads the role register s59 (flipped by +// conv_apply on a role conversion, else unchanged) and branches to the matching role's +// per-epoch _follow loop. RE-DISPATCH ONLY -- reached from a per-super-tile _quiesce bail, +// never from first-time entry (the seed arms fall into the full role entry so _alloc/_init +// run once; see the seed block). Lands on _follow (NOT _alloc/_init): the wave's VGPR +// footprint is already correct (conv_apply set it) and INIT already ran -- re-entering _alloc +// would wrongly resize, and re-running _init would reset s35=0, breaking the "wait for the +// NEXT epoch" contract (the wave would re-process the current super-tile). INITFLAG is written +// once (0xACED) and never cleared, so it is the s35 reset -- not INITFLAG -- that _follow preserves. +// s35 (last-seen-epoch) is untouched here, which is what makes a re-dispatched wave wait for +// the NEXT epoch at the top of its new role's _follow loop. Scalar-only (s59 read + s_branch) -> +// adds ZERO OOR/VGPR exposure. +// ============================================================================================ +.Ldispatch: + s_cmp_eq_u32 s59, NCOMP_SLOT + s_cbranch_scc1 .Lcompute_follow + s_cmp_eq_u32 s59, NAFEED_SLOT + s_cbranch_scc1 .Lafeed_follow + s_branch .Lbfeed_follow +.endif + +// ---- A7 role-agnostic terminal (followers): retire. (Claimer retires via .Lclaimer_terminal.) ---- +.Lretire: + tfspan max, 12 // TFPROBE: every follower stamps occ[3] = max exit tick (wall-span end) + alllive_dec // TRACE: follower exit -> --live (peak-concurrent occupancy) + s_endpgm +.else + s_endpgm // DSWS2=0 has no v2 body (this file is always built DSWS2=1) +.endif + .size occ_kernel, .-occ_kernel + +// ---- RGADESC: analysis-only descriptor so `rga -s bin --co` can enumerate + livereg this kernel. +// vgpr 256 ceiling so livereg reports the true s_alloc-grown peak-live. NOT emitted for the PM4 .bin. ---- +.if RGADESC +.amdhsa_kernel occ_kernel + .amdhsa_next_free_vgpr 256 + .amdhsa_next_free_sgpr 72 // body uses up to s69 (s66=n_kseg s67=mask s68=shift s69=chunkHi, FIX 1) + .amdhsa_group_segment_fixed_size 65536 // FIX 1a ring: D=2 needs 33024B (RGA-analysis descriptor only) + .amdhsa_user_sgpr_count 15 // FIX 1(g): v2 contract now s0..s14 only (n_kseg/TOTAL_super/ + // magic_kseg dropped -- derived in-kernel / memory-carried) + .amdhsa_wavefront_size32 1 +.end_amdhsa_kernel +.amdgpu_metadata +--- +amdhsa.version: [ 1, 2 ] +amdhsa.kernels: + - .name: occ_kernel + .symbol: occ_kernel.kd + .kernarg_segment_size: 60 + .kernarg_segment_align: 8 + .group_segment_fixed_size: 65536 + .private_segment_fixed_size: 0 + .wavefront_size: 32 + .sgpr_count: 72 + .vgpr_count: 256 + .max_flat_workgroup_size: 256 + .args: [] +.end_amdgpu_metadata +.endif diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_grind.s b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_grind.s new file mode 100644 index 000000000000..e6c8ac69726b --- /dev/null +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_grind.s @@ -0,0 +1,341 @@ +// occ_kernel_grind.s (gfx1201, wave32) -- MAD-305 "GRIND" CONTROL fp8 WMMA GEMM. +// +// PURPOSE (control experiment, 2026-07-03): the smallest NON-split-K, ONE-OUTPUT-TILE-PER-WORKGROUP +// fp8 GEMM. It deliberately does NOT use the dyn-VGPR / split-K "moat" of occ_kernel_dsws.s. One wave +// (one workgroup) OWNS one output tile (mblk,tcol), accumulates the FULL K dimension with the fp32 +// accumulators live in VGPR across the whole K loop, and writes C EXACTLY ONCE with a plain +// global_store (NO C atomics, NO cross-WG reduction, NO ksi split). This is the baseline against which +// the split-K kernel's 2.1 TF is measured: if grind beats it, split-K may not be worth defending. +// +// STRUCTURE (reused idioms from occ_kernel_dsws.s, machinery stripped): +// - WMMA microkernel: v_wmma_f32_16x16x16_fp8_fp8 (identical fragment shapes, FM x FN frags). +// - A/B staging: global_load(_tr)_b64 -> ds_store -> ds_load -> WMMA, streaming K through LDS in +// KCHUNK-k16-step chunks (single-buffer; one wave so NO barrier -- s_wait_dscnt suffices). +// - Tile hand-out: a global atomic claim counter (occ[20]) -- a persistent grid-stride over output +// tiles (raw-PM4 DISPATCH_DIRECT does NOT deliver TGID_X on gfx1201, so a claim counter is the +// proven work-distribution mechanism; see occ_kernel_wggemm.s deprecation note). +// STRIPPED vs dsws: the (mblk,tcol,ksi) super-tile pool, the atomic-add C flush, the split-K ksi +// dimension, the claimer/feed/compute role machinery, the s_alloc_vgpr grow/shrink churn. +// +// ============================================================================================ +// KERNARG CONTRACT (USER_SGPR=15, s0..s14; hardware-preloaded user SGPRs). BYTE-IDENTICAL to the +// occ_kernel_dsws.s v2 contract EXCEPT s11=TOTAL is the *grind* tile count (finer M tiling: MTL = +// M/(FM*16), not M/(G*FM*16)) and there is no ksi. The host may reuse the dsws2 userdata packing: +// s0:s1 = occ buffer base (>=0x1000B; host zero-inits; occ[20]=claim counter, host sets to `base`) +// s2:s3 = A base (fp8 e4m3, row-major MxK, 1 byte/elem) +// s4:s5 = Bshuf base (shuffled-B layout consumed by global_load_tr_b64; same as dsws) +// s6:s7 = C base (fp32, FRAGMENT-TILED: tile ti -> C + ti*(FM*FN*1024); plain store, NO memset +// required for correctness, but zeroing is harmless) +// s8 = KT (total K16-steps for the whole matrix = K/16; MUST be a multiple of KCHUNK) +// s9 = K (bytes per A-row = K, fp8 1 byte/elem) +// s10 = NT*256 (B-saddr advance per K16-step) +// s11 = TOTAL (grind tile count = MTL*NTL, MTL=M/(FM*16); ALSO the claim terminal / chunkHi) +// s12 = magic(ceil(2^32/NTL)) (unsigned-div magic for /NTL -> mblk/tcol decode) +// s13 = NTL (number of N tile-columns = N/(FN*16)) +// s14 = FN*256 (B-saddr stride per N-frag) +// (TGID_X lands in s15 -- UNUSED.) +// occ layout (occ base = s0:s1; host zero-inits, sets occ[20]=base): +// occ[0] (byte 0) = live counter (lane0 +1 at entry, -1 at exit; host polls ==0) +// occ[1] (byte 4) = maxlive (bookkeeping) +// occ[2] (byte 8) = min start realtime clock ; occ[3] (byte 12) = max end realtime clock +// occ[20] (byte 20) = GLOBAL tile claim counter (host sets to `base`; grind atomic-adds 1/tile) +// +// G and SEGK/split-K are GONE. KCHUNK (k16-steps staged per LDS chunk) is a COMPILE-TIME defsym. +// ============================================================================================ + +.amdgcn_target "amdgcn-amd-amdhsa--gfx1201" + +// ---- tile defsyms ---- +.ifndef FM + .set FM, 2 // per-wave M-frags (M tile = FM*16 rows) +.endif +.ifndef FN + .set FN, 4 // per-wave N-frags (N tile = FN*16 cols) +.endif +.ifndef KCHUNK + .set KCHUNK, 4 // k16-steps staged through LDS per chunk (power of two) +.endif +.ifndef RGADESC + .set RGADESC, 0 // 1 = emit analysis-only AMDHSA descriptor for RGA livereg +.endif + +// log2(KCHUNK) so n_chunks = KT >> KCH_SHIFT (KCHUNK is a power of two by construction). +.if KCHUNK == 1 + .set KCH_SHIFT, 0 +.elseif KCHUNK == 2 + .set KCH_SHIFT, 1 +.elseif KCHUNK == 4 + .set KCH_SHIFT, 2 +.elseif KCHUNK == 8 + .set KCH_SHIFT, 3 +.else + .error "KCHUNK must be a power of two in {1,2,4,8}" +.endif + +// ---- LDS layout (single-buffer; A then B; frag = 256B, lane*8 base) ---- +.set ALDS_OFF, 0 // A frag (ks,mi) at ALDS_OFF + (ks*FM + mi)*256 +.set ALDS_BYTES, (KCHUNK*FM*256) +.set BLDS_OFF, ALDS_BYTES // B frag (ks,ni) at BLDS_OFF + (ks*FN + ni)*256 +.set BLDS_BYTES, (KCHUNK*FN*256) +.set LDS_TOTAL, (BLDS_OFF + BLDS_BYTES) // = KCHUNK*(FM+FN)*256 (=6144 @ KCHUNK=4,FM=2,FN=4) +.if LDS_TOTAL > 32768 + .error "grind LDS layout exceeds 32768B group segment" +.endif + +// ---- VGPR layout (STATIC alloc; full footprint live for the whole K loop -- this is the control's +// high-VGPR-duty structure by design). Accumulators live from pre-K-loop zero through the single C store. +.set STG, 16 // staging pair (v16:v17) for one in-flight A/B frag +.set ACC, 32 // FM*FN fp32 accumulators x 8 (v32..) +.set FA, (ACC + 8*FM*FN) // compute A frags (from LDS): FM x 2 (=v96 @ 2x4) +.set FB, (FA + 2*FM) // compute B frags (from LDS): FN x 2 (=v100 @ 2x4) +.set NFV, (FB + 2*FN) // next-free vgpr (=v108 @ 2x4 -> ~112 alloc, under 128 cap) + + .text + .globl occ_kernel + .p2align 8 + .type occ_kernel,@function +occ_kernel: + // ---- per-thread identity (v0=tid hardware-preloaded); one wave -> wid=0, lane=tid ---- + v_and_b32 v2, 31, v0 // lane = tid & 31 + v_and_b32 v6, 15, v0 // lane & 15 (A row within frag) + v_mov_b32 v4, 0 // vaddr-0 for atomic claim (addr = occ_base + offset) + // ---- per-lane address constants (dsws-identical fragment maps) ---- + v_mul_lo_u32 v8, v6, s9 // (lane&15)*K + v_bfe_u32 v7, v0, 4, 1 // colhi = (tid>>4)&1 + v_lshlrev_b32 v7, 3, v7 // colhi*8 + v_add_nc_u32 v8, v8, v7 // v8 = A vaddr = (lane&15)*K + colhi*8 + v_lshlrev_b32 v9, 3, v2 // v9 = B/LDS vaddr = lane*8 + v_lshlrev_b32 v10, 5, v2 // v10 = C store vaddr = lane*32 + + // ---- live++ : lane0 occ[0] += 1 ; maxlive book ---- + s_mov_b32 s16, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lafter_live + v_mov_b32 v5, 1 + global_atomic_add_u32 v11, v4, v5, s[0:1] th:TH_ATOMIC_RETURN scope:SCOPE_DEV // v11 = old live + s_wait_loadcnt 0x0 + v_add_nc_u32 v11, v11, 1 + global_atomic_max_u32 v4, v11, s[0:1] offset:4 scope:SCOPE_DEV // maxlive +.Lafter_live: + s_mov_b32 exec_lo, s16 + + // ---- timer t0 (min start) ---- + s_sendmsg_rtn_b64 s[36:37], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + s_mov_b32 s16, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lafter_t0 + v_mov_b32 v5, s36 + global_atomic_min_u32 v4, v5, s[0:1] offset:8 scope:SCOPE_DEV +.Lafter_t0: + s_mov_b32 exec_lo, s16 + + // n_chunks = KT >> KCH_SHIFT (KT = s8) + s_lshr_b32 s33, s8, KCH_SHIFT + // A mi-stride = 16*K bytes + s_lshl_b32 s30, s9, 4 + +// ============================================================================================ +// CLAIM LOOP: ti = atomicAdd(occ[20], 1) ; if ti >= TOTAL(s11) done ; compute tile ti full-K ; store C. +// ============================================================================================ +.Lclaim: + s_mov_b32 s16, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lclaim_bcast + v_mov_b32 v5, 1 + global_atomic_add_u32 v11, v4, v5, s[0:1] offset:20 th:TH_ATOMIC_RETURN scope:SCOPE_DEV // v11=old ti + s_wait_loadcnt 0x0 +.Lclaim_bcast: + s_mov_b32 exec_lo, s16 + v_readfirstlane_b32 s20, v11 // ti (uniform) + s_cmp_ge_u32 s20, s11 // ti >= TOTAL -> drained + s_cbranch_scc1 .Ldone + + // ---- decode: mblk = ti/NTL (magic) ; tcol = ti - mblk*NTL ---- + s_mul_hi_u32 s21, s20, s12 // mblk + s_mul_i32 s22, s21, s13 // mblk*NTL + s_sub_u32 s22, s20, s22 // tcol = ti - mblk*NTL + + // ---- tile scalar bases (k=0) ---- + // A: A + (mblk*FM*16)*K + s_mul_i32 s38, s21, (FM*16) // row_start = mblk*FM*16 + s_mul_i32 s34, s38, s9 // low(row_start*K) + s_mul_hi_u32 s35, s38, s9 // high + s_add_u32 s34, s2, s34 + s_addc_u32 s35, s3, s35 // s[34:35] = A tile base (k=0), advances per chunk + // B: Bshuf + tcol*(FN*256) + s_mul_i32 s36, s22, s14 // low(tcol*FN*256) + s_mul_hi_u32 s37, s22, s14 // high + s_add_u32 s36, s4, s36 + s_addc_u32 s37, s5, s37 // s[36:37] = B tile base (k=0), advances per chunk + // C: C + ti*(FM*FN*1024) + s_mul_i32 s28, s20, (FM*FN*1024) // low + s_mul_hi_u32 s29, s20, (FM*FN*1024) // high + s_add_u32 s28, s6, s28 + s_addc_u32 s29, s7, s29 // s[28:29] = C tile base + + // ---- zero the FM*FN fp32 accumulators (live across the whole K loop) ---- + .set idx, 0 + .rept FM*FN + v_mov_b32 v[ACC+idx*8+0], 0 + v_mov_b32 v[ACC+idx*8+1], 0 + v_mov_b32 v[ACC+idx*8+2], 0 + v_mov_b32 v[ACC+idx*8+3], 0 + v_mov_b32 v[ACC+idx*8+4], 0 + v_mov_b32 v[ACC+idx*8+5], 0 + v_mov_b32 v[ACC+idx*8+6], 0 + v_mov_b32 v[ACC+idx*8+7], 0 + .set idx, idx+1 + .endr + + s_mov_b32 s31, 0 // chunk = 0 +// ---- K-CHUNK LOOP: stage KCHUNK k16-steps of A/B into LDS, then WMMA-accumulate ---- +.Lkloop: + s_cmp_ge_u32 s31, s33 + s_cbranch_scc1 .Lkdone + + // ===== STAGE A into LDS: A frag (ks,mi) ; global_load_b64(v8, base + ks*16) -> ds_store ===== + .set mi, 0 + .rept FM + // sAmi = A_chunk_base + mi*(16*K) + .if mi == 0 + s_mov_b32 s44, s34 + s_mov_b32 s45, s35 + .else + s_add_u32 s44, s44, s30 + s_addc_u32 s45, s45, 0 + .endif + .set ks, 0 + .rept KCHUNK + global_load_b64 v[STG:STG+1], v8, s[44:45] offset:(ks*16) + s_wait_loadcnt 0x0 + ds_store_b64 v9, v[STG:STG+1] offset:(ALDS_OFF + (ks*FM + mi)*256) + s_wait_dscnt 0x0 + .set ks, ks+1 + .endr + .set mi, mi+1 + .endr + + // ===== STAGE B into LDS: B frag (ks,ni) ; global_load_tr_b64(v9, base + ks*s10 + ni*256) -> ds_store + s_mov_b32 s42, s36 // sBc = B_chunk_base (ks=0) + s_mov_b32 s43, s37 + .set ks, 0 + .rept KCHUNK + s_mov_b32 s40, s42 // sf = sBc + s_mov_b32 s41, s43 + .set ni, 0 + .rept FN + global_load_tr_b64 v[STG:STG+1], v9, s[40:41] + s_wait_loadcnt 0x0 + ds_store_b64 v9, v[STG:STG+1] offset:(BLDS_OFF + (ks*FN + ni)*256) + s_wait_dscnt 0x0 + s_add_u32 s40, s40, 256 // += frag stride (FN dir) + s_addc_u32 s41, s41, 0 + .set ni, ni+1 + .endr + s_add_u32 s42, s42, s10 // += k16-step stride (NT*256) + s_addc_u32 s43, s43, 0 + .set ks, ks+1 + .endr + + // ===== COMPUTE: read resident A/B from LDS, WMMA-accumulate into live ACC ===== + .set ks, 0 + .rept KCHUNK + .set ni, 0 + .rept FN + ds_load_b64 v[FB+ni*2:FB+ni*2+1], v9 offset:(BLDS_OFF + (ks*FN + ni)*256) + .set ni, ni+1 + .endr + .set mi, 0 + .rept FM + ds_load_b64 v[FA+mi*2:FA+mi*2+1], v9 offset:(ALDS_OFF + (ks*FM + mi)*256) + .set mi, mi+1 + .endr + s_wait_dscnt 0x0 + .set mi, 0 + .rept FM + .set ni, 0 + .rept FN + v_wmma_f32_16x16x16_fp8_fp8 v[ACC+(mi*FN+ni)*8:ACC+(mi*FN+ni)*8+7], v[FA+mi*2:FA+mi*2+1], v[FB+ni*2:FB+ni*2+1], v[ACC+(mi*FN+ni)*8:ACC+(mi*FN+ni)*8+7] + .set ni, ni+1 + .endr + .set mi, mi+1 + .endr + .set ks, ks+1 + .endr + + // advance chunk bases: A += KCHUNK*16 bytes ; B += KCHUNK*(NT*256) + s_add_u32 s34, s34, (KCHUNK*16) + s_addc_u32 s35, s35, 0 + s_mul_i32 s46, s10, KCHUNK + s_add_u32 s36, s36, s46 + s_addc_u32 s37, s37, 0 + s_add_u32 s31, s31, 1 // chunk++ + s_branch .Lkloop +.Lkdone: + + // ===== C STORE (once, plain non-atomic): C_tile + frag*1024 + {0,16} ; vaddr v10=lane*32 ===== + .set frag, 0 + .rept FM*FN + global_store_b128 v10, v[ACC+frag*8+0:ACC+frag*8+3], s[28:29] offset:(frag*1024 + 0) + global_store_b128 v10, v[ACC+frag*8+4:ACC+frag*8+7], s[28:29] offset:(frag*1024 + 16) + .set frag, frag+1 + .endr + s_wait_storecnt 0x0 // stores must be issued before we reuse ACC next claim + s_branch .Lclaim + +.Ldone: + s_wait_storecnt 0x0 + // ---- timer t1 (max end) ---- + s_sendmsg_rtn_b64 s[36:37], sendmsg(MSG_RTN_GET_REALTIME) + s_wait_kmcnt 0x0 + s_mov_b32 s16, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lafter_t1 + v_mov_b32 v5, s36 + global_atomic_max_u32 v4, v5, s[0:1] offset:12 scope:SCOPE_DEV +.Lafter_t1: + s_mov_b32 exec_lo, s16 + // ---- live-- : lane0 occ[0] -= 1 ---- + s_mov_b32 s16, exec_lo + v_cmp_eq_u32 vcc_lo, 0, v2 + s_and_b32 exec_lo, exec_lo, vcc_lo + s_cbranch_execz .Lafter_dec + v_mov_b32 v5, -1 + global_atomic_add_u32 v4, v5, s[0:1] scope:SCOPE_DEV +.Lafter_dec: + s_mov_b32 exec_lo, s16 + s_endpgm + .size occ_kernel, .-occ_kernel + +// ---- RGADESC: analysis-only AMDHSA descriptor so `rga -s bin --co` can enumerate + livereg. NOT +// emitted for the PM4 .bin (the host provides RSRC1/RSRC2 directly). ---- +.if RGADESC +.amdhsa_kernel occ_kernel + .amdhsa_next_free_vgpr NFV + .amdhsa_next_free_sgpr 50 + .amdhsa_group_segment_fixed_size LDS_TOTAL + .amdhsa_user_sgpr_count 15 + .amdhsa_wavefront_size32 1 +.end_amdhsa_kernel +.amdgpu_metadata +--- +amdhsa.version: [ 1, 2 ] +amdhsa.kernels: + - .name: occ_kernel + .symbol: occ_kernel.kd + .kernarg_segment_size: 60 + .kernarg_segment_align: 8 + .group_segment_fixed_size: 6144 + .private_segment_fixed_size: 0 + .wavefront_size: 32 + .sgpr_count: 50 + .vgpr_count: 112 + .max_flat_workgroup_size: 32 + .args: [] +.end_amdgpu_metadata +.endif diff --git a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/rga_check.sh b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/rga_check.sh index cdb81ed0ce19..585b2a152d51 100755 --- a/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/rga_check.sh +++ b/ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/rga_check.sh @@ -14,7 +14,7 @@ set -euo pipefail ROCM=/opt/rocm L="$ROCM/llvm/bin" RGA=/home/kmbandy/Downloads/rdts/RadeonDeveloperToolSuite-2026-05-28-1806/rga -KSRC=occ_kernel_wggemm2.s +KSRC="${KSRC:-occ_kernel_wggemm2.s}" # override to gate a different kernel, e.g. KSRC=occ_kernel_coop.s LABEL="${1:?usage: rga_check.sh
(superseded) T1.3 in-progress notes + +Approach: extend `run_mbcoop` with two **defaulted** params so the proven 2-role coop callers stay +byte-identical, then `--dsws` calls it with `P=NCOMP`, `totalWaves=N`, and the DSWS LDS size. + +- **DONE:** `run_mbcoop` signature + `WAVES_LAUNCH = totalWaves>0 ? totalWaves : (1+P)` (occ_dispatch.cpp). + Harness recompiles clean (niced build, 2026-06-27 23:08). +- **REMAINING (3 items, all offline):** + 1. `ldsBytes` override line in `run_mbcoop` (`ldsBytesOverride>0 ? ldsBytesOverride : coop-formula`). + 2. Replace the T1.1 `--dsws` stub (`[T1.1] ... dispatch wiring lands in T1.3`) with the real call: + compute `N`, `ldsDsws = BRING + 4*nComp+16 + 4*(nBfeed-1) + RINGD*nComp*FM*256 + 8*nComp` + (FM=2,FN=4 fixed v1 tile), loop `{down, down_pf}`, small tile-multiple oracle first, + `run_mbcoop(dswsBin, dyn, pool, Mo,No,Ko, 2,4, nComp, ringd, /*fullCheck*/true, /*GENDIV*/true, + 1,0.0, /*totalWaves*/N, /*ldsOverride*/ldsDsws)`, gate on `bad==0`. + 3. `build_dsws.sh` — emits `occ_dsws_cab_r[_dyn]_gd.bin` (DSWS=1 FM=2 FN=4 POOLTERM=1) + for cells {4c2a2b, 6c1a1b, 2c3a3b} × {dyn,static}; + RGA gate. +- **Gate:** `./build_dsws.sh static` (all cells assemble + RGA 0-spill) + harness links. No GPU. +