Repository navigation
perf(xla): load single-sequence graphs with transposed weights (#2859) - #2861
Merged
Merged
Conversation
The single-sequence decode graph contracted each projection's last axis against the checkpoint's `[out, in]` layout, and IREE lowered that to a full-weight transpose plus a vecmat on every decode step. Profiling on gfx1151 showed 112 weight transposes per step at 96.5 ms/step (57 ms of transposes, plus a 9.65 ms f32 LM-head transpose), against 4.1 ms/step for the MLX ROCm path. The graph now declares projection weights `[in, out]` and contracts the shared axis, and the loader transposes each `Proj` / `Rows` weight once at upload (`transpose_projection`, 32x32 blocked). Graph emission picks the layout through `Config::projection_layout` and the `linear_proj` / `linear_seq_proj` helpers, so a graph's weight declarations and its contractions cannot disagree. The layout is per engine, not global, because the best lowering depends on the graph shape. The ragged batch decode already lowered the old `[out, in]` contraction to one transposed-B `linalg.matmul` (fast at its row count), while the canonical matmul over `[in, out]` was about 8x slower per call; the single-row case was the opposite. `IreeLlama` therefore emits `InOut` and `IreeRaggedLlama` keeps the checkpoint's `OutIn`, and every committed golden stays byte-identical on the `OutIn` path. Phi4MM LoRA adapter tensors and the LM head stay untransposed because the loader uploads them `Whole`. Validation on gfx1151 (hip, llama-3.2-1b-instruct-4bit, release build): B=1 CLI at capacity 256, n=64 10.5-11.4 -> 18.6-21.4 tok/s and n=128 9.0-10.0 -> 16.5-18.3 tok/s; batch B=4 N=8 63.9-69.2 -> 66.8-67.7 tok/s and B=8 N=16 80.1-80.5 -> 74.8-84.1 tok/s (unchanged path, within run-to-run spread); REFERENCE-EXACT PASS on both batch sizes. Greedy output is token-identical to the previous binary on llama-3.2-1b, qwen3-1.7b, qwen2.5-0.5b and gemma-3-4b. mlxcel-xla lib tests 385 pass with unchanged goldens; clippy clean at default and `iree` features. Refs #2859
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The single-sequence XLA/IREE decode graph contracted each projection's last axis against the checkpoint's
[out, in]weight layout. IREE lowered that to a full-weight transpose plus a vecmat on every decode step: profiling on gfx1151 showed 112 projection transposes (57 ms) and a 9.65 ms f32 LM-head transpose inside a 96.5 ms step, against 4.1 ms/step for the MLX ROCm path (issue #2859).This PR makes the graph declare projection weights
[in, out]and contract the shared axis, so IREE emits one vecmat per projection, and moves the transpose to load time where it runs once per weight.Changes
ProjectionLayout(OutIn|InOut) onConfigandBuilder, withlinear_proj/linear_seq_projselecting the contraction to match the declared weight shape. Dense projection call sites moved to the layout-aware helpers; Phi4MM LoRA adapter tensors (modal,[rank, in]/[out, rank]) and the LM head (Whole) keep the untransposed helpers.transpose_projectioninweights.rs, a 32x32 blocked[out, in] -> [in, out]transpose applied once perProj/Rowsweight at upload. The packed 4-bit path keeps its in-graph dequant plus transpose, since it already trades decode throughput for upload bytes.IreeLlama(single sequence) emitsInOut;IreeRaggedLlama(continuous batching) keepsOutIn; Gemma3n's own emitters keepOutIn.Why the layout is per engine
The best lowering depends on the graph shape, not the model. The ragged batch decode already lowered the old
[out, in]contraction to a single transposed-Blinalg.matmul(about 0.3 ms per down projection at its row count), while the canonical matmul over[in, out]was about 8x slower per call (2.38 ms); a single-layout branch measured B=4 N=8 at 28.0-28.5 tok/s against 59.3-65.5 before. The single-row case is the opposite, where the old layout cost the transpose plus vecmat. KeepingOutInfor the batching engine also keeps every committed golden byte-identical, which is what the 385-testmlxcel-xlalib suite checks.Validation (gfx1151, hip, llama-3.2-1b-instruct-4bit, release build)
REFERENCE-EXACT PASSon both batch sizes.cargo test -p mlxcel-xla --lib385 passed, committed goldens unchanged.cargo clippy -p mlxcel --lib --tests -- -D warningsandcargo clippy -p mlxcel-xla --all-targets --features iree -- -D warningsclean;cargo fmt --checkclean.cargo check --features xla-iree --bin mlxcel --example xla_batch_benchclean.Closes #2859