Repository navigation
perf: zero points as mins and nibble operands in the Apple7/8 register decode tile - #15
Merged
paperniuk merged 1 commit intoOct 20, 2026
Conversation
…r decode tile The F32 register decode tile ran Q4_0 and Q5_0 as zero-point formats: each 16-input half of a group closed its chain against the table's seed, with a seed load and an epilogue per half. On an M2 Max the 27B in Q4_0, at MLX 4-bit's size (16.1 GB), decoded 10-14% below MLX 4-bit at one to four lanes, while its prefill matched it. The F32 instances now decode Q4_0 and Q5_0 through ZeroAsMin, which takes the zero point as a min (m = -zero d): their chains close once per 32 inputs against the table's sums, as Q4_1's and Q5_1's do. The 4-bit formats with a min per 32 inputs (Q4_K, Q4_1, Q4_0 as ZeroAsMin, MLX affine 4-bit in groups of 32) take nibble_operand in their per-32 chains, as MLX 4-bit's span chains do. The Apple9 instances are unchanged. M2 Max, 17408x5120 gate/up, ms at one to four lanes: q40 0.473 0.853 1.199 1.651 -> 0.424 0.766 1.101 1.416 q50 0.560 1.006 1.300 1.774 -> 0.547 0.929 1.200 1.628 q41 0.446 0.800 1.136 1.459 -> 0.427 0.774 1.120 1.434 q4k 0.456 0.790 1.101 1.432 -> 0.433 0.773 1.089 1.403 af4g64 (unchanged code) 0.416 0.733 1.012 1.380 -> 0.416 0.732 1.012 1.383 27B Q4_0 decode at one, two and four lanes: 77.7, 89.0, 97.1 -> 82.6, 95.6, 108.3 tok/s (MLX 4-bit 86.9, 103.0, 111.8). Prefill and the MLX 4-bit, UD-Q4_K_M, UD-Q4_K_XL, UD-Q6_K_XL and UD-Q8_K_XL decodes are within 0.6%. gguf-projection also runs the register decode and gate/up suites on the Apple7/8 F32 instances on later GPUs.
Owner
|
Thanks! I checked it on my M1 Max: gguf-projection passes, and the 17408x5120 gate/up kernel at 1/2/3/4 lanes went from 0.582/1.065/1.478/2.001 ms to 0.516/0.960/1.359/1.722 for Q4_0, so 8 to 14% faster. Q5_0 is 2 to 7% faster, Q4_1 and Q4_K up to 5%, MLX 4-bit unchanged. Merging, it will be in the next release. |
Author
|
@paperniuk Thanks for testing it on M1 Max and merging! Nice to see similar numbers there. BTW, UD quants (UD-Q4_K_M etc.) are still ~10% behind MLX 4-bit on decode on my M2 Max. Most of their weights are Q5_K/Q6_K/IQ4_XS, which this change doesn’t touch. I can look into the Q5_K/Q6_K decode next, if that sounds useful. |
This was referenced Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A while ago I compared GGUF with MLX 4-bit on my M2 Max, and GGUF decoded slower. I hoped 1.3.2 would close the gap,
but it was still slow, even with Q4_0, which is the same size as MLX 4-bit (16.1 GB for the 27B). So I looked into
whether anything could be done, and found a way to bring Q4_0 close to MLX 4-bit. I have some C++ and shader
experience, but from a long time ago, so I used Claude Code to author this change and tested it by hand.
What changed
Only the Apple7/8 F32 instances of the register decode tile (
linear_gguf_sgmatrix.metal); Apple9 and host code areuntouched.
m = -zero * d), so their chains close once per 32 inputs against thetable's sums, like Q4_1 and Q5_1. Before, they closed once per 16 against the seeds.
nibble_operandintheir per-32 chains, like MLX 4-bit's span chains.
gguf-projectionalso runs the register decode and gate/up suites on the Apple7/8 F32 instances on Apple9+ GPUs.Results (M2 Max, 38-core GPU)
Kernel time,
gguf-projection-benchmark, 17408×5120 gate/up, ms at 1 / 2 / 3 / 4 lanes:All models on the M2 Max, after the change
make test-performance-real, Qwen3.8-27B, ANE share 0.35. Times are in seconds and decode is in tok/s at 1 / 2 / 4lanes, with the % against MLX 4-bit in brackets (time: + is slower; decode: − is slower). An MLX 4-bit run before and
after the GGUF runs agreed within 0.1%.
The UD quants changed by less than 0.6% with this PR. Their remaining decode gap comes from their Q5_K/Q6_K tensors,
which this change doesn't touch.
Testing
make checkpasses on the M2 Max.gguf-projectionpasses on an M4, including the new Apple7/8 F32 pass (all 42 formats).make test-performance-realon every model above, before and after.