Widen w4a16 perf-bench weight rotation to match in-model timing - #1065
Merged
Conversation
The rotation in test_hybrid_w4a16_perf.py existed to make each captured call read cold weights, sized at 48 MiB on the rationale that this exceeds the gfx1151 32 MiB MALL. Checking the bench against per-call times from a vLLM torch-profiler trace (gemma-4-31B AWQ, bs=1) showed that rationale is insufficient: the bench was 19% optimistic on the 44-58 MB weights while matching within 2% on a 116 MB one. 48 MiB clamps n_buf to 2, so those shapes cycle an ~88 MB working set. That already exceeds MALL, yet still gets locality the model never has, since the model streams weights scattered across ~19.5 GB. Raising the target to 512 MiB brings all three shapes to within 1.1% of the trace. Also adds the three gemma-4-31B shapes that exposed this, and makes both rotation constants env-overridable so the value can be re-validated. Changes: - 512 MiB, not larger: 512 and 2048 MiB measure identically, so this is past the knee. _ROTATE_MAX stays 32; small weights converge at the ~84 MB that 32 buffers already give, and raising the cap changed nothing for them. - Goldens are fully regenerated because the wider rotation shifts every shape whose weight exceeds ~24 MB. - Comparisons were made with both sides unpinned. Pinning the clock via a fixed shader frequency is actively misleading here: it slowed the in-model kernel by 44% while barely moving the bench, because the serving stack's CPU load shares the package power budget and forcing gfxclk high makes the SMU cut memory/fabric clocks instead. Known issue, why this is a draft: the regenerated goldens are not reproducible at the current +/-8% tolerance. Two verification runs against them failed 18/140 and 19/140, always in the "improvement" direction, and two independent recording runs disagree by more than +/-8% on 17.5% of measurements. The unmodified 48 MiB setting verifies at 1/128. The wider footprint appears to import the same physical-page / TLB variability that makes it accurate, so tolerance handling needs to be resolved before this can merge. Signed-off-by: Matthias Gehre <matthias.gehre@amd.com>
eble-amd
approved these changes
Jul 28, 2026
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
tests/kernels/quantization/test_hybrid_w4a16_perf.pyrotates the weight operand throughseveral buffers so each captured call reads cold weights. The size was 48 MiB, justified in
the comment as "exceeds the gfx1151 32 MiB MALL".
Checking the bench against real per-call times from a vLLM torch-profiler trace
(
cyankiwi/gemma-4-31B-it-AWQ-4bit, bs=1 decode, gfx1151) showed that rationale does nothold. Standalone vs in-model
wvsplitk_int4average kernel time:1x43008x5376gate_up1x5376x21504down1x16384x5376qkv48 MiB clamps
n_bufto 2 for weights of 44–116 MB, so the two smaller shapes cycle an~88 MB working set. That already exceeds MALL, yet still gets locality the model never
has — the model streams weights scattered across ~19.5 GB. The 116 MB shape matched only
because 2 buffers (231 MB) was already near-converged.
Fix
Raise
_ROTATE_TARGET_BYTESto 512 MiB. All three shapes then land within 1.1%:1x43008x53761x5376x215041x16384x5376Also included:
VLLM_BENCH_ROTATE_MIB/VLLM_BENCH_ROTATE_MAXenv overrides (defaults unchanged inbehaviour) so the constant can be re-validated without editing the file.
~24 MB.
separate benchmarks; none touch the rotation constant or the goldens. [ROCm][Kernel] W4A16 skinny GEMM: hand-asm kernel for gfx1151 #1045 and Tune hybrid_triton_w4a16 prefill kernel for gfx1151 #879 will
need a golden refresh when they land, but that is a rebase concern, not overlapping work.
AI assistance
AI assistance (Claude Code) was used for this change: running the benchmarks, correlating
against the profiler trace, and drafting this description. All measurements above are real
runs on the target hardware, reproducible with the commands listed.