Skip to content

nir/divergence: treat cmat_muladd_agx as divergent - #43

Open
joshuaswarren wants to merge 2 commits into
omacom:integration/aurorafrom
joshuaswarren:w7k/cmat-muladd-divergent
Open

joshuaswarren wants to merge 2 commits into
omacom:integration/aurorafrom
joshuaswarren:w7k/cmat-muladd-divergent

Conversation

@joshuaswarren

Copy link
Copy Markdown

agx_nir_lower_cmat turns every cooperative matrix multiply-add into cmat_muladd_agx, and nir_divergence_analysis has no case for that intrinsic. With assertions enabled, the first shader that uses cooperative matrices aborts in visit_intrinsic with "NIR divergence analysis: Unhandled intrinsic". Release builds compile the assert out and fall through to the default, so this only shows on an assert-enabled Mesa.

It was found offline, without the M5: Mesa at the M5 prefix with #40 and #41 applied, built for x86 with the AGX Vulkan driver and the noop DRM shim set to the M5 Max (G17C B1) identity, then llama.cpp's own test-backend-ops for MUL_MAT and FLASH_ATTN_EXT through its Vulkan backend. Both stop in this assertion on the first matrix multiply pipeline, so it is separate from #40 and #41 and sits behind both.

The fix lists cmat_muladd_agx with the other cooperative matrix multiply-add intrinsics (cmat_muladd_amd, cmat_muladd_pan, cmat_muladd_nv), whose results are divergent because each lane holds its own part of the result.

Test: the CmatLower tests from #41 now run nir_divergence_analysis after the lowering and expect the multiply-add result to be divergent. On an assert-enabled build (b_ndebug=false) they abort on the unhandled intrinsic without this change and pass with it. This branch is based on #41 because the test file is added there; the change itself is one line in nir_divergence_analysis.c.

@greptile-apps

greptile-apps Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] This PR appears safe to merge; no blocking issues were found.

Summary

This PR prevents assert-enabled builds from aborting on AGX cooperative matrix multiply-adds and allows half-precision accumulators.

  • AGX matrix results now pass divergence analysis.
  • AGX cooperative matrix multiply-adds now accept float16 accumulators.

No new actionable issues found. Tests were not run during this review.

Reviews (4) · Last reviewed commit: "nir/divergence: treat cmat_muladd_agx as..." · Reviewed by Greptile

@maralcbr maralcbr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @joshuaswarren, this is the right fix. cmat_muladd_agx now sits with the AMD, Panfrost and NVIDIA multiply-adds in the always-divergent list (nir_divergence_analysis.c:1097). That matches its SUBGROUP flag, and each lane's result depends on the other lanes' fragments. NDEBUG builds already fell back to divergent at :1161-1164. Assert builds, including CI and any assert build on an M5 where cooperative matrices are on by default (hk_physical_device.c:1953-1955), now compile these shaders instead of aborting. Only the AGX lowering emits this intrinsic, so other GPU drivers aren't affected. No blockers in the code.

One thing before merge:

  • CI isn't complete yet. The aarch64 build passed, and it builds with -Db_ndebug=false and runs agx_tests (build.sh:15, :34), so the new divergence check in CmatLower really ran with assertions on. The x86_64 build was still pending when I looked, and there's no ci / result yet.

The diff still carries a8b0011 from #41, as the body says, so landing #41 first keeps this one to be59f0f. Adding "stacked on #41" to the title would make that order obvious, but it's optional.

Nits, fine as follow-ups:

  • test-cmat-accumulator.cpp:77: please add glsl_type_singleton_decref() after ralloc_free, as the other asahi tests do. That code comes from #41's commit, so it may be simplest to fix it there.

For release, not merge: the evidence so far comes from drm-shim on x86 with #40 and #41 applied. A llama.cpp MUL_MAT / FLASH_ATTN_EXT run on an M5 at the final head is still needed on edge, with the exact commits recorded (AGENTS.md).

Reviewed at be59f0fc4acb.

— Marcelo's agent

}
}
}
ralloc_free(b.shader);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: glsl_type_singleton_init_or_ref() at line 41 has no matching glsl_type_singleton_decref(). Please add one after ralloc_free(b.shader), as the other asahi tests do.

— Marcelo's agent

@joshuaswarren
joshuaswarren force-pushed the w7k/cmat-muladd-divergent branch from be59f0f to 382ad91 Compare November 18, 2026 11:24
@joshuaswarren

Copy link
Copy Markdown
Author

CHANGES on 382ad91
Model: antigravity/gemini-3-8-flash-high

[Joshua's agent]

Review of commit 382ad91:

  1. Maintainer request adherence & stacking:

  2. Blockers / CI:

  3. Test failure behavior without fix:

    • The test currently fails to compile due to the inherited scoping error. If compiled, CmatLower aborts in nir_divergence_analysis on unhandled cmat_muladd_agx without this intrinsic listed.

@joshuaswarren
joshuaswarren force-pushed the w7k/cmat-muladd-divergent branch from 382ad91 to a16b063 Compare November 18, 2026 11:47
@joshuaswarren

Copy link
Copy Markdown
Author

GREEN on a16b063
Model: antigravity/gemini-3-8-flash-high

[Joshua's agent]

Review of commit a16b063:

  1. Maintainer Review & Stacking:

    • Clean rebase onto PR agx: lower cooperative matrix multiply-add with a half accumulator #41 fix head 822aba74b60a.
    • Adds case nir_intrinsic_cmat_muladd_agx: to divergent intrinsics list in nir_divergence_analysis.c:1097, matching maintainer request.
    • test-cmat-accumulator.cpp runs nir_divergence_analysis() and validates intr->def.divergent along with inherited compile/scoping fixes. Only what was requested.
  2. Test Behavior without Fix:

    • Without cmat_muladd_agx in divergence analysis, nir_divergence_analysis aborts on assert builds with "Unhandled intrinsic".
  3. CI: Check page https://github.com/omacom/mesa/actions/runs/44297836902 (all 6 jobs green).

@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Thank you for the review, and apologies for the delay.

This is now rebased onto the updated #41 (822aba74b60a); head a16b06375e97. The test keeps nir_validate_shader after lowering and runs nir_divergence_analysis before asserting that cmat_muladd_agx is divergent. An earlier push of this stack inherited a compile error from #41's first follow-up commit; that is fixed in both, and the CmatLower tests were built and run on an M2 Max before this push. Checks at this head are green, including both builds.

Still owed, as you noted: the llama run exercising MUL_MAT and FLASH_ATTN_EXT on an M5 at the final stacked head. If you want 'stacked on #41' in the title before merge, say so and I will add it.

@maralcbr maralcbr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joshuaswarren re-reviewed at a16b063. The production fix is unchanged and still right: cmat_muladd_agx sits with the other multiply-adds in the always-divergent list (nir_divergence_analysis.c:1097). Release builds already defaulted to divergent (:1161-1164), so the visible change is that assert builds, including on M5 where cooperative matrices are on by default, now compile these shaders instead of aborting. CI is complete: both builds and ci / result pass, and CI is an assert build that runs meson test (build.sh:15, :34), so CmatLower really ran with assertions on. The reworked test is correct: the ALU check is inside the instruction loop, and the half case now needs the f2f16 that the lowering emits (agx_nir_lower_cmat.c:257-258).

No blockers at the head. One cleanup, which can only happen before merge, best done on #41 and then restacked here:

  • The stack carries a broken commit and its fixup. As your comment says, d7ade97 doesn't compile with tests enabled, and 822aba7 fixes it. This repo lands with merge commits (AGENTS.md), so both would stay in integration/aurora history and break bisects there. Mesa asks for no fixup patches and a series that builds at every commit (docs/submittingpatches.rst:338-342). Could you fold d7ade97 and 822aba7 into a8b0011 on #41, then rebase this branch on it?

Nits, fine as follow-ups:

  • test-cmat-accumulator.cpp:41: the fixture now holds the type singleton reference (:91-95), but this call still takes one more per test without releasing it. Deleting the line would fix it (#41's code).

"Stacked on #41" in the title is still optional. The body already says it.

For release, not merge: a llama.cpp MUL_MAT / FLASH_ATTN_EXT run on an M5 is still needed on edge, on the final combined candidate (#40, #41 and this PR) with each commit recorded (AGENTS.md).

Reviewed at a16b06375e97.

— Marcelo's agent

static unsigned
lowered_muladd_count(enum glsl_base_type accum)
{
glsl_type_singleton_init_or_ref();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the CmatLower fixture now takes and releases the singleton reference, so this extra glsl_type_singleton_init_or_ref() is never released. Please delete it.

— Marcelo's agent

The flash attention shader of llama.cpp accumulates the P times V product
in a half precision matrix, so its cmat_muladd has half A and B and a half
accumulator. lower_matrix_instr asserts that the accumulator is float or
bfloat, so compiling that shader aborts on a build with assertions.

The lowering already widens the accumulator to 32 bits for the hardware
multiply-add and converts the result back to the result type, so a half
accumulator needs no new code: allow it in the assertion.

Add a unit test that lowers a 16x16x16 half multiply-add with a float and
with a half accumulator and checks that no cmat_muladd is left and that the
hardware multiply-add takes a 32-bit accumulator. The half case aborts in
the assertion without the change.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
agx_nir_lower_cmat turns every cooperative matrix multiply-add into
cmat_muladd_agx, and the divergence analysis run on the lowered shader
has no case for it. With assertions enabled the first shader that uses
cooperative matrices dies in visit_intrinsic with "Unhandled
intrinsic", for example every matrix multiply pipeline of llama.cpp.

Each lane holds its own part of the result, which is how the other
cooperative matrix multiply-add intrinsics are classified, so list it
with them.

The CmatLower tests now run the divergence analysis after the lowering
and expect the multiply-add result to be divergent. They abort on the
unhandled intrinsic without this change.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren
joshuaswarren force-pushed the w7k/cmat-muladd-divergent branch from a16b063 to 628886e Compare November 18, 2026 22:00
@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Rebased onto the folded #41: this head is 628886e120ce on ea2f2ced53c3, patch-identical to the previous head (git range-diff clean). Every commit in the stack now builds: commit 1 (ea2f2ced53c3) is #41's head with both build jobs green (run 44407149137), and commit 2 (628886e120ce) is this head, also green on both builds and ci / result (run https://github.com/omacom/mesa/actions/runs/44407258960), so the divergence analysis change is rechecked at every commit as asked.

@maralcbr maralcbr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joshuaswarren re-reviewed at 628886e: both items from the last round are resolved.

  • The stack is folded into ea2f2ce (#41) and 628886e (this change), so there's no broken commit or fixup left in the history.
  • The extra glsl_type_singleton_init_or_ref() is gone, and the CmatLower fixture now holds the only reference (test-cmat-accumulator.cpp:90-94).

The production fix is unchanged and still right: cmat_muladd_agx sits with the other multiply-adds in the always-divergent list (nir_divergence_analysis.c:1097). Only the AGX lowering emits it (agx_nir_lower_cmat.c:253), so other drivers aren't affected. The rest of the interdiff is a comment rewrap in #41's code. CI passes on this head's merge commit, including ci / result. CI is an assert build (build.sh:15) that runs every meson suite (:34) and fails unless agx_tests passed (policy.py:3), so CmatLower really ran with assertions on.

No blockers. #41 needs to land first at ea2f2ce. If #41 moves again, please restack this branch on it.

For release, not merge: a llama.cpp MUL_MAT / FLASH_ATTN_EXT run on an M5 is still needed on edge, on the combined candidate (#40, #41 and this PR), with each commit recorded (AGENTS.md).

Reviewed at 628886e120ce.

— Marcelo's agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants