Annotate W4A16 profiler scopes with group size and symmetry - #1067
Merged
Conversation
The W4A16 record_function labels carried only MxNxK, which is not enough
to turn a trace into a bandwidth number. A grouped W4A16 GEMM also streams
a per-group scale, and for asymmetric layers a zero-point, each
N * (K/group_size) elements at 2 bytes. Neither is derivable from the
shape, so the roofline tooling counted N*K/2 weight bytes and nothing
else.
On gemma-4-31B-it-AWQ-4bit (g=32, asymmetric) that omission is 28.9 MB
against 115.6 MB of weights -- a 25% undercount. It made the decode GEMMs
read as 163 GiB/s next to 216 GiB/s for the fp16 lm_head in the same
trace, i.e. an apparent 25% kernel deficit that does not exist: counting
all the bytes actually moved puts every shape at 187-204 GiB/s.
Labels now carry g<group_size> and sym/asym:
wvsplitk_int4 1x43008x5376 g32 asym
fused_moe_wvsplitk_int4 1x1536x2048 E=128 top_k=8 g=32 sym
Changes:
- Covers both W4A16 GEMM paths (wvsplitk_int4, hybrid_triton_w4a16) and
the dense-prefill hybrid_dequant_w4a16, plus the MoE scopes
(fused_moe_wvsplitk_int4 GEMM1/GEMM2, moe_gemm_w4a16, hybrid_triton_moe).
- MoE GEMM1 and GEMM2 read w1_zp and w2_zp independently rather than
assuming both halves share a symmetry.
- hybrid_triton_moe is hardcoded "sym" because that wrapper is
symmetric-only by construction (b_zp_ptr=None, has_zp=False, zero-point
bias folded into the constant 8). Emitting the marker anyway lets the
tooling account the scale bytes.
- moe_gemm_w4a16 takes an optional asym flag defaulting to None, which
emits nothing, so any caller that cannot supply it keeps the old label
rather than getting a wrong "sym".
- Consumer side is a separate change in the roofline script; labels
without the markers keep parsing, so old traces are unaffected.
Validated on gfx1151: gemma-4-31B (dense, g32 asym) and
Qwen3-30B-A3B-Instruct-2507-AWQ-4bit (MoE, g32 sym); byte counts
hand-checked against the emitted labels.
Signed-off-by: Matthias Gehre <matthias.gehre@amd.com>
roberteg16
approved these changes
Jul 29, 2026
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The W4A16
record_functionlabels carried onlyMxNxK. That is not enough to turn atrace into a bandwidth number, because a grouped W4A16 GEMM does not only stream weights:
N * (K/group_size)elements at 2 bytesNeither is derivable from the shape, so the roofline tooling counted
N*K/2weight bytesand nothing else.
On
cyankiwi/gemma-4-31B-it-AWQ-4bit(g=32, asymmetric) that is 28.9 MB unaccountedagainst 115.6 MB of weights — a 25% undercount. The consequence was not cosmetic: the
decode GEMMs read as ~163 GiB/s. Counting the bytes actually moved puts
every one of those shapes at 187-204 GiB/s
Change
Labels now carry
g<group_size>andsym/asym: