Skip to content

perf(xla): load single-sequence graphs with transposed weights (#2859) - #2861

Merged
inureyes merged 1 commit into
mainfrom
perf/xla-transposed-weights
Nov 5, 2026
Merged

inureyes merged 1 commit into
mainfrom
perf/xla-transposed-weights

Conversation

@inureyes

@inureyes inureyes commented Nov 5, 2026

Copy link
Copy Markdown
Member

Summary

The single-sequence XLA/IREE decode graph contracted each projection's last axis against the checkpoint's [out, in] weight layout. IREE lowered that to a full-weight transpose plus a vecmat on every decode step: profiling on gfx1151 showed 112 projection transposes (57 ms) and a 9.65 ms f32 LM-head transpose inside a 96.5 ms step, against 4.1 ms/step for the MLX ROCm path (issue #2859).

This PR makes the graph declare projection weights [in, out] and contract the shared axis, so IREE emits one vecmat per projection, and moves the transpose to load time where it runs once per weight.

Changes

  • ProjectionLayout (OutIn | InOut) on Config and Builder, with linear_proj / linear_seq_proj selecting the contraction to match the declared weight shape. Dense projection call sites moved to the layout-aware helpers; Phi4MM LoRA adapter tensors (modal, [rank, in] / [out, rank]) and the LM head (Whole) keep the untransposed helpers.
  • transpose_projection in weights.rs, a 32x32 blocked [out, in] -> [in, out] transpose applied once per Proj / Rows weight at upload. The packed 4-bit path keeps its in-graph dequant plus transpose, since it already trades decode throughput for upload bytes.
  • The layout is chosen per engine, not globally: IreeLlama (single sequence) emits InOut; IreeRaggedLlama (continuous batching) keeps OutIn; Gemma3n's own emitters keep OutIn.

Why the layout is per engine

The best lowering depends on the graph shape, not the model. The ragged batch decode already lowered the old [out, in] contraction to a single transposed-B linalg.matmul (about 0.3 ms per down projection at its row count), while the canonical matmul over [in, out] was about 8x slower per call (2.38 ms); a single-layout branch measured B=4 N=8 at 28.0-28.5 tok/s against 59.3-65.5 before. The single-row case is the opposite, where the old layout cost the transpose plus vecmat. Keeping OutIn for the batching engine also keeps every committed golden byte-identical, which is what the 385-test mlxcel-xla lib suite checks.

Validation (gfx1151, hip, llama-3.2-1b-instruct-4bit, release build)

case before after
B=1 CLI, n=64, capacity 256 10.5-11.4 tok/s 18.6-21.4 tok/s
B=1 CLI, n=128, capacity 256 9.0-10.0 tok/s 16.5-18.3 tok/s
batch B=4 N=8 63.9-69.2 tok/s 66.8-67.7 tok/s
batch B=8 N=16 80.1-80.5 tok/s 74.8-84.1 tok/s
  • REFERENCE-EXACT PASS on both batch sizes.
  • Greedy output token-identical to the previous binary on llama-3.2-1b, qwen3-1.7b, qwen2.5-0.5b and gemma-3-4b.
  • cargo test -p mlxcel-xla --lib 385 passed, committed goldens unchanged.
  • cargo clippy -p mlxcel --lib --tests -- -D warnings and cargo clippy -p mlxcel-xla --all-targets --features iree -- -D warnings clean; cargo fmt --check clean.
  • cargo check --features xla-iree --bin mlxcel --example xla_batch_bench clean.

Closes #2859

The single-sequence decode graph contracted each projection's last axis against the checkpoint's `[out, in]` layout, and IREE lowered that to a full-weight transpose plus a vecmat on every decode step. Profiling on gfx1151 showed 112 weight transposes per step at 96.5 ms/step (57 ms of transposes, plus a 9.65 ms f32 LM-head transpose), against 4.1 ms/step for the MLX ROCm path.

The graph now declares projection weights `[in, out]` and contracts the shared axis, and the loader transposes each `Proj` / `Rows` weight once at upload (`transpose_projection`, 32x32 blocked). Graph emission picks the layout through `Config::projection_layout` and the `linear_proj` / `linear_seq_proj` helpers, so a graph's weight declarations and its contractions cannot disagree.

The layout is per engine, not global, because the best lowering depends on the graph shape. The ragged batch decode already lowered the old `[out, in]` contraction to one transposed-B `linalg.matmul` (fast at its row count), while the canonical matmul over `[in, out]` was about 8x slower per call; the single-row case was the opposite. `IreeLlama` therefore emits `InOut` and `IreeRaggedLlama` keeps the checkpoint's `OutIn`, and every committed golden stays byte-identical on the `OutIn` path. Phi4MM LoRA adapter tensors and the LM head stay untransposed because the loader uploads them `Whole`.

Validation on gfx1151 (hip, llama-3.2-1b-instruct-4bit, release build): B=1 CLI at capacity 256, n=64 10.5-11.4 -> 18.6-21.4 tok/s and n=128 9.0-10.0 -> 16.5-18.3 tok/s; batch B=4 N=8 63.9-69.2 -> 66.8-67.7 tok/s and B=8 N=16 80.1-80.5 -> 74.8-84.1 tok/s (unchanged path, within run-to-run spread); REFERENCE-EXACT PASS on both batch sizes. Greedy output is token-identical to the previous binary on llama-3.2-1b, qwen3-1.7b, qwen2.5-0.5b and gemma-3-4b. mlxcel-xla lib tests 385 pass with unchanged goldens; clippy clean at default and `iree` features.

Refs #2859
@inureyes inureyes added type:performance Performance improvements status:review Under review labels Nov 5, 2026
@inureyes
inureyes merged commit bb1072d into main Nov 5, 2026
31 checks passed
@inureyes
inureyes deleted the perf/xla-transposed-weights branch November 5, 2026 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:review Under review type:performance Performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(xla): every decode step transposes all projection weights because dot_general is emitted with the checkpoint's [out, in] layout

1 participant