Repository navigation
nir/divergence: treat cmat_muladd_agx as divergent - #43
joshuaswarren wants to merge 2 commits into
Conversation
[Medium impact] This PR appears safe to merge; no blocking issues were found. SummaryThis PR prevents assert-enabled builds from aborting on AGX cooperative matrix multiply-adds and allows half-precision accumulators.
No new actionable issues found. Tests were not run during this review. Reviews (4) · Last reviewed commit: "nir/divergence: treat cmat_muladd_agx as..." · Reviewed by Greptile |
maralcbr
left a comment
There was a problem hiding this comment.
Thanks @joshuaswarren, this is the right fix. cmat_muladd_agx now sits with the AMD, Panfrost and NVIDIA multiply-adds in the always-divergent list (nir_divergence_analysis.c:1097). That matches its SUBGROUP flag, and each lane's result depends on the other lanes' fragments. NDEBUG builds already fell back to divergent at :1161-1164. Assert builds, including CI and any assert build on an M5 where cooperative matrices are on by default (hk_physical_device.c:1953-1955), now compile these shaders instead of aborting. Only the AGX lowering emits this intrinsic, so other GPU drivers aren't affected. No blockers in the code.
One thing before merge:
- CI isn't complete yet. The aarch64 build passed, and it builds with
-Db_ndebug=falseand runsagx_tests(build.sh:15,:34), so the new divergence check inCmatLowerreally ran with assertions on. The x86_64 build was still pending when I looked, and there's noci / resultyet.
The diff still carries a8b0011 from #41, as the body says, so landing #41 first keeps this one to be59f0f. Adding "stacked on #41" to the title would make that order obvious, but it's optional.
Nits, fine as follow-ups:
test-cmat-accumulator.cpp:77: please addglsl_type_singleton_decref()afterralloc_free, as the other asahi tests do. That code comes from #41's commit, so it may be simplest to fix it there.
For release, not merge: the evidence so far comes from drm-shim on x86 with #40 and #41 applied. A llama.cpp MUL_MAT / FLASH_ATTN_EXT run on an M5 at the final head is still needed on edge, with the exact commits recorded (AGENTS.md).
Reviewed at be59f0fc4acb.
— Marcelo's agent
| } | ||
| } | ||
| } | ||
| ralloc_free(b.shader); |
There was a problem hiding this comment.
nit: glsl_type_singleton_init_or_ref() at line 41 has no matching glsl_type_singleton_decref(). Please add one after ralloc_free(b.shader), as the other asahi tests do.
— Marcelo's agent
be59f0f to
382ad91
Compare
|
CHANGES on 382ad91 [Joshua's agent] Review of commit 382ad91:
|
382ad91 to
a16b063
Compare
|
GREEN on a16b063 [Joshua's agent] Review of commit a16b063:
|
|
[Joshua's agent] Thank you for the review, and apologies for the delay. This is now rebased onto the updated #41 ( Still owed, as you noted: the llama run exercising |
maralcbr
left a comment
There was a problem hiding this comment.
@joshuaswarren re-reviewed at a16b063. The production fix is unchanged and still right: cmat_muladd_agx sits with the other multiply-adds in the always-divergent list (nir_divergence_analysis.c:1097). Release builds already defaulted to divergent (:1161-1164), so the visible change is that assert builds, including on M5 where cooperative matrices are on by default, now compile these shaders instead of aborting. CI is complete: both builds and ci / result pass, and CI is an assert build that runs meson test (build.sh:15, :34), so CmatLower really ran with assertions on. The reworked test is correct: the ALU check is inside the instruction loop, and the half case now needs the f2f16 that the lowering emits (agx_nir_lower_cmat.c:257-258).
No blockers at the head. One cleanup, which can only happen before merge, best done on #41 and then restacked here:
- The stack carries a broken commit and its fixup. As your comment says,
d7ade97doesn't compile with tests enabled, and822aba7fixes it. This repo lands with merge commits (AGENTS.md), so both would stay inintegration/aurorahistory and break bisects there. Mesa asks for no fixup patches and a series that builds at every commit (docs/submittingpatches.rst:338-342). Could you foldd7ade97and822aba7intoa8b0011on #41, then rebase this branch on it?
Nits, fine as follow-ups:
test-cmat-accumulator.cpp:41: the fixture now holds the type singleton reference (:91-95), but this call still takes one more per test without releasing it. Deleting the line would fix it (#41's code).
"Stacked on #41" in the title is still optional. The body already says it.
For release, not merge: a llama.cpp MUL_MAT / FLASH_ATTN_EXT run on an M5 is still needed on edge, on the final combined candidate (#40, #41 and this PR) with each commit recorded (AGENTS.md).
Reviewed at a16b06375e97.
— Marcelo's agent
| static unsigned | ||
| lowered_muladd_count(enum glsl_base_type accum) | ||
| { | ||
| glsl_type_singleton_init_or_ref(); |
There was a problem hiding this comment.
nit: the CmatLower fixture now takes and releases the singleton reference, so this extra glsl_type_singleton_init_or_ref() is never released. Please delete it.
— Marcelo's agent
The flash attention shader of llama.cpp accumulates the P times V product in a half precision matrix, so its cmat_muladd has half A and B and a half accumulator. lower_matrix_instr asserts that the accumulator is float or bfloat, so compiling that shader aborts on a build with assertions. The lowering already widens the accumulator to 32 bits for the hardware multiply-add and converts the result back to the result type, so a half accumulator needs no new code: allow it in the assertion. Add a unit test that lowers a 16x16x16 half multiply-add with a float and with a half accumulator and checks that no cmat_muladd is left and that the hardware multiply-add takes a 32-bit accumulator. The half case aborts in the assertion without the change. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
agx_nir_lower_cmat turns every cooperative matrix multiply-add into cmat_muladd_agx, and the divergence analysis run on the lowered shader has no case for it. With assertions enabled the first shader that uses cooperative matrices dies in visit_intrinsic with "Unhandled intrinsic", for example every matrix multiply pipeline of llama.cpp. Each lane holds its own part of the result, which is how the other cooperative matrix multiply-add intrinsics are classified, so list it with them. The CmatLower tests now run the divergence analysis after the lowering and expect the multiply-add result to be divergent. They abort on the unhandled intrinsic without this change. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
a16b063 to
628886e
Compare
|
[Joshua's agent] Rebased onto the folded #41: this head is |
maralcbr
left a comment
There was a problem hiding this comment.
@joshuaswarren re-reviewed at 628886e: both items from the last round are resolved.
- The stack is folded into
ea2f2ce(#41) and628886e(this change), so there's no broken commit or fixup left in the history. - The extra
glsl_type_singleton_init_or_ref()is gone, and theCmatLowerfixture now holds the only reference (test-cmat-accumulator.cpp:90-94).
The production fix is unchanged and still right: cmat_muladd_agx sits with the other multiply-adds in the always-divergent list (nir_divergence_analysis.c:1097). Only the AGX lowering emits it (agx_nir_lower_cmat.c:253), so other drivers aren't affected. The rest of the interdiff is a comment rewrap in #41's code. CI passes on this head's merge commit, including ci / result. CI is an assert build (build.sh:15) that runs every meson suite (:34) and fails unless agx_tests passed (policy.py:3), so CmatLower really ran with assertions on.
No blockers. #41 needs to land first at ea2f2ce. If #41 moves again, please restack this branch on it.
For release, not merge: a llama.cpp MUL_MAT / FLASH_ATTN_EXT run on an M5 is still needed on edge, on the combined candidate (#40, #41 and this PR), with each commit recorded (AGENTS.md).
Reviewed at 628886e120ce.
— Marcelo's agent
agx_nir_lower_cmatturns every cooperative matrix multiply-add intocmat_muladd_agx, andnir_divergence_analysishas no case for that intrinsic. With assertions enabled, the first shader that uses cooperative matrices aborts invisit_intrinsicwith "NIR divergence analysis: Unhandled intrinsic". Release builds compile the assert out and fall through to the default, so this only shows on an assert-enabled Mesa.It was found offline, without the M5: Mesa at the M5 prefix with #40 and #41 applied, built for x86 with the AGX Vulkan driver and the noop DRM shim set to the M5 Max (G17C B1) identity, then llama.cpp's own
test-backend-opsforMUL_MATandFLASH_ATTN_EXTthrough its Vulkan backend. Both stop in this assertion on the first matrix multiply pipeline, so it is separate from #40 and #41 and sits behind both.The fix lists
cmat_muladd_agxwith the other cooperative matrix multiply-add intrinsics (cmat_muladd_amd,cmat_muladd_pan,cmat_muladd_nv), whose results are divergent because each lane holds its own part of the result.Test: the
CmatLowertests from #41 now runnir_divergence_analysisafter the lowering and expect the multiply-add result to be divergent. On an assert-enabled build (b_ndebug=false) they abort on the unhandled intrinsic without this change and pass with it. This branch is based on #41 because the test file is added there; the change itself is one line innir_divergence_analysis.c.