Skip to content

agx: lower cooperative matrix multiply-add with a half accumulator - #41

Open
joshuaswarren wants to merge 1 commit into
omacom:integration/aurorafrom
joshuaswarren:w7k/cmat-half-accum
Open

joshuaswarren wants to merge 1 commit into
omacom:integration/aurorafrom
joshuaswarren:w7k/cmat-half-accum

Conversation

@joshuaswarren

Copy link
Copy Markdown

The flash attention shader of llama.cpp (flash_attn_cm1) accumulates its P times V product in a half precision cooperative matrix, so its cmat_muladd has half A and B and a half accumulator. lower_matrix_instr in agx_nir_lower_cmat.c asserts that the accumulator is float or bfloat. On an M5 Max with the NIR validation abort fixed (#40), loading a model through llama.cpp's Vulkan backend then stops in that assertion in agx_nir_lower_cmat.c, in the cmat_muladd case.

The lowering already converts the accumulator to 32 bits for the hardware multiply-add and converts the result back to the result type. A half accumulator therefore needs no new lowering; this only allows it in the assertion.

Test: CmatLower.HalfInputsHalfAccumulator in agx_tests lowers a 16x16x16 half multiply-add with a half accumulator and checks that no cmat_muladd is left and that cmat_muladd_agx takes a 32-bit accumulator. A HalfInputsFloatAccumulator control passes before and after. Without the change the half case aborts with the same assertion text as on the device. Built with assertions on (b_ndebug=false): full agx_tests 505 passed, 1 skipped. A release build compiles the assertion out, so the test only fails on a build with assertions.

Not run: the llama.cpp shader on the device with this change (the device is not available right now), so later passes in the same shader are untested. The driver still advertises only float accumulation for half inputs; this PR does not change that.

@greptile-apps

greptile-apps Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge based on the code reviewed.

Summary

Allows half-precision cooperative matrix accumulators through the existing AGX lowering path without changing the math operations.

  • AGX lowering accepts half-precision cooperative-matrix accumulators.

No new actionable issues found. Tests and device execution were not run during this review.

Reviews (4) · Last reviewed commit: "agx: lower cooperative matrix multiply-a..." · Reviewed by Greptile

@maralcbr maralcbr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @joshuaswarren, the fix is right. The lowering already widens the accumulator to f32 (agx_nir_lower_cmat.c:249-250) and converts the result back to the result type before the store (:256-257), so only the assertion at :240-246 was in the way. I found no blockers in the code. Builds with assertions now lower this shader instead of aborting. Release builds already compiled this path the same way, and the advertised configurations in hk_GetPhysicalDeviceCooperativeMatrixPropertiesKHR don't change.

Smaller things:

  • Validate in the test. lowered_muladd_count calls the pass directly (test-cmat-accumulator.cpp:58), and nir_progress doesn't validate, so a malformed store would still pass the count check. Please call nir_validate_shader(b.shader, "after agx_nir_lower_cmat") after the pass. Ideally also check that an f2f16 feeds the final store in the half case, so the test covers the conversion back too.
  • A question rather than a fix: since the driver doesn't advertise an f16 accumulator for f16 inputs (as the body says), do you know whether llama.cpp gates that flash-attention path on the reported cooperative matrix properties? It would tell us whether there's something to report upstream.

Nits, fine as follow-ups:

  • Please add glsl_type_singleton_decref() after ralloc_free (test-cmat-accumulator.cpp:74), as test-demote-early-kill.cpp:223-227 does.
  • The assertion now also admits an f16 accumulator with bf16, mixed f16/bf16 or 8x8 f32 inputs. That's fine because the conversion is generic, but a short comment saying the accumulator is always widened to f32 would help.

CI: the x86_64 and aarch64 build jobs were still pending when I looked, so there's no completed agx_tests result at this head yet. CI builds with -Db_ndebug=false, so the new test will exercise the assertion there.

Whether to advertise f16 x f16 -> f16 configurations is a maintainer call for a separate PR, not this one. For release (not the source merge): the llama.cpp flash-attention shader still needs a run on the device with this change, per the edge qualification in AGENTS.md.

Reviewed at a8b0011539c2.

— Marcelo's agent

: nir_imm_float(&b, 0.0f));
nir_cmat_muladd(&b, &d->def, &a->def, &bb->def, &c->def);

agx_nir_lower_cmat(b.shader);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add nir_validate_shader(b.shader, "after agx_nir_lower_cmat"); here? nir_progress doesn't validate, so a mis-sized store would still pass the count.

— Marcelo's agent

}
}
}
ralloc_free(b.shader);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: glsl_type_singleton_decref(); after ralloc_free, to balance the ref taken at line 41.

— Marcelo's agent

((input == GLSL_TYPE_FLOAT16 && input_b == GLSL_TYPE_BFLOAT16) ||
(input == GLSL_TYPE_BFLOAT16 && input_b == GLSL_TYPE_FLOAT16)))) &&
(accum == GLSL_TYPE_FLOAT || accum == GLSL_TYPE_BFLOAT16) && result == accum);
(accum == GLSL_TYPE_FLOAT || accum == GLSL_TYPE_FLOAT16 ||

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: worth a short comment that C is always widened to f32 (line 249), so any float accumulator type is fine here.

— Marcelo's agent

@joshuaswarren

Copy link
Copy Markdown
Author

CHANGES on d7ade97
Model: antigravity/gemini-3-8-flash-high

[Joshua's agent]

Review of commit d7ade97:

  1. Maintainer request adherence:

    • Calls nir_validate_shader after agx_nir_lower_cmat in src/asahi/compiler/test/test-cmat-accumulator.cpp.
    • Wraps CmatLower test fixture with glsl_type_singleton_init_or_ref() / glsl_type_singleton_decref().
    • Adds comment in agx_nir_lower_cmat.c noting accumulators widen to f32.
    • Intended to require f2f16 before the store for the half-accumulator case.
  2. Blockers / CI:

    • Check page: https://github.com/omacom/mesa/actions/runs/44294205384
    • Build failed in both x86_64 and aarch64 jobs with compilation error in src/asahi/compiler/test/test-cmat-accumulator.cpp:
      error: 'instr' was not declared in this scope
      Lines 74-78 placed the ALU type check outside the nir_foreach_instr(instr, block) loop where instr is out of scope.
    • Inverted condition in lowered_muladd_count():
      if (accum != GLSL_TYPE_FLOAT16 || !f16_store) return left == 0 ? agx : 0;
      When accum == GLSL_TYPE_FLOAT16 and f16_store is false, it returns agx (1u), so missing f2f16 conversion still passes. Should return 0 when accum == GLSL_TYPE_FLOAT16 && !f16_store.
  3. Test failure behavior without fix:

    • The test fails to compile. Additionally, due to the inverted return condition, the half-conversion check does not fail when f2f16 is missing.

@joshuaswarren

Copy link
Copy Markdown
Author

GREEN on 822aba7
Model: antigravity/gemini-3-8-flash-high

[Joshua's agent]

Review of commit 822aba7:

  1. Maintainer Review & Scoping Fix:

    • Scope fix: instr->type == nir_instr_type_alu is inside nir_foreach_instr(instr, block) in test-cmat-accumulator.cpp, resolving compilation failure.
    • Guard direction: if (accum == GLSL_TYPE_FLOAT16 && !f16_store) return 0; correctly returns 0 if f2f16 is missing for half accumulators, ensuring HalfInputsHalfAccumulator fails if lowering does not emit f2f16.
    • Maintainer items intact: nir_validate_shader called after lowering, GLSL singleton refcount handled in test fixture, and comment added regarding widening to f32. Only what was requested.
  2. Test Behavior without Fix:

    • Without the lowering fix, assertions abort on GLSL_TYPE_FLOAT16 accumulator.
    • Without the f2f16 store emission, f16_store check fails (returns 0).
  3. CI: Check page https://github.com/omacom/mesa/actions/runs/44297215747 (all 6 jobs green).

@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Thank you for the review, and apologies for the six days without an answer.

Validate in the test: done at 822aba74b60a. The test now runs nir_validate_shader after agx_nir_lower_cmat, and in the half-accumulator case it requires an f2f16 on the path to the store, so it fails without the fix. One correction of our own on the way: the first follow-up commit (d7ade97a39fa) did not compile (a variable out of scope) and had that check inverted; 822aba74b60a fixes both, and the CmatLower tests were built and run on an M2 Max before it was pushed. Checks at this head are green, including both builds.

Not done here: whether llama.cpp's flash attention gates on the advertised cooperative-matrix configurations on this driver. I have not traced that yet and will not ask for f16 accumulators to be advertised before I have. This pull request still only removes the assert so that lowering matches release behaviour.

@maralcbr maralcbr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joshuaswarren re-reviewed at 822aba7. Thanks, and no need to apologise for the gap. Validation is in (test-cmat-accumulator.cpp:59). The half case now fails without an f2f16 (:81-85), and the ALU check sits inside the instruction loop. The widening comment is in (agx_nir_lower_cmat.c:240). The production change is still just the assertion, and it's right: the accumulator is widened to f32 (:250-251) and converted back before the store (:257-258). The advertised configurations are unchanged, so f16 inputs still only report an f32 accumulator (hk_physical_device.c:2404-2412).

One thing before merge:

  • d7ade97 doesn't compile. You noted this yourself. Landing it as is would leave a build break in history, and docs/submittingpatches.rst:117 asks for bisectable commits. Could you squash d7ade97 and 822aba7 into a8b0011 and force-push? If the branch is going to be squash-merged anyway, ignore this.

One note, not a change to this PR: the new test stops right after agx_nir_lower_cmat. In the driver, the backend then runs nir_divergence_analysis (agx_compile.c:8167), which has no case for cmat_muladd_agx (nir_divergence_analysis.c:1096-1166). So builds with assertions would still abort later in this shader. #43 looks like it covers that. It also seems to be stacked on this branch, so a squash here would mean a rebase there.

Nits, fine as follow-ups:

  • The fixture now takes and drops the singleton ref (:88-92), but the old glsl_type_singleton_init_or_ref() at :41 is still there, so each test still leaks one ref. Deleting line 41 would fix it.
  • agx_nir_lower_cmat.c:240 is wider than the rest of the file. Worth wrapping.

CI: both build jobs pass, and so does ci / result. build.sh runs meson test with -Db_ndebug=false, and the manifest step fails unless agx_tests passes, so the new test ran with assertions on. "Attested ARM candidate" was skipped, which is expected: it only runs on pushes to integration/aurora.

Whether to advertise f16 accumulators is a separate maintainer call, and I agree it should wait until you've traced llama.cpp's gating. For release (not the source merge): the flash-attention shader still needs a run on the device at the final commit, per AGENTS.md.

Reviewed at 822aba74b60a.

— Marcelo's agent

static unsigned
lowered_muladd_count(enum glsl_base_type accum)
{
glsl_type_singleton_init_or_ref();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the fixture at :88-92 owns the ref now, so this extra glsl_type_singleton_init_or_ref() leaves one ref per test. Could be removed.

— Marcelo's agent

Comment thread src/asahi/compiler/agx_nir_lower_cmat.c Outdated
enum glsl_base_type input_b = matrix_desc(intr->src[2]).element_type;
enum glsl_base_type accum = matrix_desc(intr->src[3]).element_type;
enum glsl_base_type result = matrix_desc(intr->src[0]).element_type;
/* Cooperative-matrix accumulators are always widened to f32 for the AGX path. */

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: worth wrapping to match the file's line width.

— Marcelo's agent

The flash attention shader of llama.cpp accumulates the P times V product
in a half precision matrix, so its cmat_muladd has half A and B and a half
accumulator. lower_matrix_instr asserts that the accumulator is float or
bfloat, so compiling that shader aborts on a build with assertions.

The lowering already widens the accumulator to 32 bits for the hardware
multiply-add and converts the result back to the result type, so a half
accumulator needs no new code: allow it in the assertion.

Add a unit test that lowers a 16x16x16 half multiply-add with a float and
with a half accumulator and checks that no cmat_muladd is left and that the
hardware multiply-add takes a 32-bit accumulator. The half case aborts in
the assertion without the change.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Done as asked: d7ade97 and 822aba7 are folded into a8b0011 and force-pushed as the single commit ea2f2ced53c3 (run https://github.com/omacom/mesa/actions/runs/44407149137, both builds and ci / result green, so agx_tests ran with assertions on). The folded tree is content-identical to 822aba7 except your two nits: the leftover glsl_type_singleton_init_or_ref() at test-cmat-accumulator.cpp:41 is gone (the fixture at :87-91 owns the ref), and the agx_nir_lower_cmat.c:240 comment is wrapped. No other bytes changed.

@maralcbr maralcbr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joshuaswarren re-reviewed at ea2f2ce: all items resolved, and I found no blockers. Thanks for the quick turnaround.

  • The branch is now one signed-off commit, so the build break from d7ade97 is gone from history.
  • The stray singleton ref is gone. The fixture owns the only ref (test-cmat-accumulator.cpp:87-91).
  • The comment is wrapped (agx_nir_lower_cmat.c:240-241).

Nothing else changed since 822aba7. The fix is still right: the accumulator is widened to f32 (:251-252) and converted back before the store (:258-259), so an f16 accumulator goes through the same generic conversion as bf16, whatever the accepted input types. The advertised configurations are unchanged (hk_physical_device.c:2404-2412).

CI: both build jobs and ci / result pass. build.sh runs meson test with -Db_ndebug=false, and the manifest step requires agx_tests, so the new tests ran with assertions on. "Attested ARM candidate" was skipped, which is expected on a PR: it only runs on pushes to integration/aurora.

#43 looks stacked on the old branch. If it is, it will need a rebase onto ea2f2ce. For release (not the source merge): the flash-attention shader still needs a run on the device on the edge channel, at the final commit and with #43 in, per AGENTS.md. Without #43, a build with assertions would still abort later in divergence analysis.

Reviewed at ea2f2ced53c3.

— Marcelo's agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants