Repository navigation
perf: 5-bit nibble operands in the Apple7/8 register decode tile - #18
Merged
paperniuk merged 1 commit intoOct 21, 2026
Merged
Conversation
The F32 register decode tile ran Q5_K, Q5_1 and Q5_0 (as ZeroAsMin) through operand_f32: the fifth bit shifted below the nibble, or-ed into half 1024 and the offset subtracted, about two more operations per pair than the 4-bit formats' nibble_operand. Q5_K is the largest format of the Unsloth UD-Q4 and UD-Q5 files (29% of UD-Q4_K_M's bytes, 45% of UD-Q4_K_XL's and UD-Q5_K_M's) and was the costliest per byte in the tile. Their F32 chains per 32 inputs now take nibble_operand with the fifth bit at bit 8 of each halfword, 1 + code / 64, exact for codes to 31, and close with NibbleChunk's epilogue as the 4-bit formats do. MLX affine 5-bit in groups of 32 stays on operand_f32: it took 1.3% more at four lanes. The Apple9 instances are unchanged. M2 Max, ms at one to four lanes: q5k 17408x5120 gate/up 0.544 0.924 1.208 1.640 -> 0.520 0.905 1.196 1.611 q5k 5120x17408 down 0.288 0.488 0.628 0.853 -> 0.273 0.480 0.625 0.836 q5k 5120x6144 out 0.109 0.200 0.259 0.350 -> 0.105 0.198 0.257 0.342 q50 17408x5120 gate/up 0.550 0.931 1.206 1.646 -> 0.523 0.918 1.205 1.645 q51 17408x5120 gate/up 0.556 0.955 1.228 1.690 -> 0.534 0.945 1.229 1.694 q4k, af4g64 (unchanged code) within 0.4% 27B decode at one, two and four lanes, tok/s: UD-Q4_K_XL 74.4 90.2 102.5 -> 76.0 90.8 104.5 UD-Q5_K_M 70.3 85.0 97.7 -> 71.5 85.7 98.7 UD-Q5_K_XL 69.3 85.6 98.2 -> 70.5 86.3 99.2 UD-Q4_K_M 77.8 93.7 106.8 -> 78.9 94.2 106.6 Q4_0, Q4_1 (their ssm_out is Q5_K), UD-IQ4_XS and UD-Q6_K_M gain up to 1.1%; prefill and the MLX 4-bit, Q8_0, UD-Q6_K_XL and UD-Q8_K_XL decodes are within 0.6%.
Author
|
@paperniuk Heads-up: I have a follow-up for Q6_K, the same idea applied to the 6-bit format. The code goes into the half as
It’s stacked on this PR. I’ll open it once the end-to-end runs and make check are done. Most likely tomorrow morning. |
This was referenced Oct 21, 2026
Owner
|
Merged, thanks! I ran the whole stack (#18 to #21) on M1 Max (Apple7, 32 cores). gguf-projection passes natively, with and without MTL_SHADER_VALIDATION. Q5_K got faster here too: 2560x6144 by 1.7/3.5/0.2/1.6% at 1/2/3/4 lanes, and the 248320x2560 output by 5.0/3.0/3.0/4.4%. Numbers for the rest are coming in #21. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After PR #15, Q4_0 decodes within 3–7% of MLX 4-bit on my M2 Max, but the Unsloth UD quants were still 10–20% behind.
I ran explored most of the quants in Qwen3.8-27B GGUF. Q5_K is the largest format in the UD-Q4/Q5 files: 29% of
UD-Q4_K_M's bytes and 45% of UD-Q4_K_XL's and UD-Q5_K_M's. It was also the costliest per byte in the Apple7/8
register decode tile. This change gives the 5-bit formats the exact nibble operand that PR #15 gave the 4-bit ones. As
before, I used Claude Code to author it and tested it by hand.
What changed
Only the Apple7/8 F32 instances of the register decode tile (
linear_gguf_sgmatrix.metal) change. Apple9 and hostcode are untouched.
with its zero point as a min. The nibble goes to mantissa bits 4–7 and the fifth bit to bit 8, so their per-32 chains
close with the same epilogue as the 4-bit nibble formats.
operand_f32: the fifth bit was shifted below the nibble,| 0x6400, then a half2subtract.
gguf-projectionalready runs every format through the F32 instances: natively on Apple7/8, and onApple9+ through PR perf: zero points as mins and nibble operands in the Apple7/8 register decode tile #15's pass.
Results (M2 Max, 38-core GPU)
Kernel time,
gguf-projection-benchmark, at 1 / 2 / 3 / 4 lanes, ABBA order; Δ: − is faster. Two runs of the same build agree within0.4%.
All models on the M2 Max, after the change
make test-performance-real, Qwen3.8-27B, ANE share 0.35. Times are in seconds; decode is in tok/s at 1 / 2 / 4lanes. The % against MLX 4-bit is in brackets (time: + is slower; decode: − is slower). The two MLX 4-bit runs, before
and after the GGUF runs, agree within 0.1% on each build.
Decode before and after
The same benchmark on
apple7-m1-1.3.2, run right after the branch on the same machine, in tok/s at 1 / 2 / 4 lanes.The Q5_K share is of each file's tensor bytes.
two and four lanes. UD-Q4_K_M, at 29%, is 1.4% faster at one lane. A first two-model run gave it +1.5 / +1.7 /
−0.1%, so the two-lane gain is between +0.5% and +1.7%.
kernel is 4–5% faster at one lane on all three of its shapes, and Q5_K is 30–45% of those files' bytes.
outputin Q6_K andssm_out(the out projection of the 48 GDN layers) in Q5_K, 1.04 GB each. Q5_K is 6–6.5% of their bytes, so they pick up a
small share of the gain: up to +0.6% at one lane and +1.0% (Q4_1) at four.
staged MMA tile.
UD-Q8_K_XL at four lanes (−0.6%). Its code path didn't change, and earlier runs of the same build put it at
108.8–109.1 tok/s, so that's noise.
doesn't touch. Q6_K is also every file's
outputlayer, except the 8-bit ones.Next
Follow-up PRs, opened at the same time so they can be tested and merged in order:
so the chains need no seeds. The UD-Q5/Q6 files decode 1.2–1.5% faster at one lane and 3.6–5.7% at two and four.
Testing
make checkpasses on the M2 Max.gguf-projectionpasses on the M2 Max (all 43 formats, F32 instances natively).make test-performance-realon every model above, before and after.