Repository navigation
perf: Q6_K's zero point in the operands of the Apple7/8 register decode tile - #19
Merged
paperniuk merged 2 commits intoOct 21, 2026
Merged
Conversation
The F32 register decode tile ran Q5_K, Q5_1 and Q5_0 (as ZeroAsMin) through operand_f32: the fifth bit shifted below the nibble, or-ed into half 1024 and the offset subtracted, about two more operations per pair than the 4-bit formats' nibble_operand. Q5_K is the largest format of the Unsloth UD-Q4 and UD-Q5 files (29% of UD-Q4_K_M's bytes, 45% of UD-Q4_K_XL's and UD-Q5_K_M's) and was the costliest per byte in the tile. Their F32 chains per 32 inputs now take nibble_operand with the fifth bit at bit 8 of each halfword, 1 + code / 64, exact for codes to 31, and close with NibbleChunk's epilogue as the 4-bit formats do. MLX affine 5-bit in groups of 32 stays on operand_f32: it took 1.3% more at four lanes. The Apple9 instances are unchanged. M2 Max, ms at one to four lanes: q5k 17408x5120 gate/up 0.544 0.924 1.208 1.640 -> 0.520 0.905 1.196 1.611 q5k 5120x17408 down 0.288 0.488 0.628 0.853 -> 0.273 0.480 0.625 0.836 q5k 5120x6144 out 0.109 0.200 0.259 0.350 -> 0.105 0.198 0.257 0.342 q50 17408x5120 gate/up 0.550 0.931 1.206 1.646 -> 0.523 0.918 1.205 1.645 q51 17408x5120 gate/up 0.556 0.955 1.228 1.690 -> 0.534 0.945 1.229 1.694 q4k, af4g64 (unchanged code) within 0.4% 27B decode at one, two and four lanes, tok/s: UD-Q4_K_XL 74.4 90.2 102.5 -> 76.0 90.8 104.5 UD-Q5_K_M 70.3 85.0 97.7 -> 71.5 85.7 98.7 UD-Q5_K_XL 69.3 85.6 98.2 -> 70.5 86.3 99.2 UD-Q4_K_M 77.8 93.7 106.8 -> 78.9 94.2 106.6 Q4_0, Q4_1 (their ssm_out is Q5_K), UD-IQ4_XS and UD-Q6_K_M gain up to 1.1%; prefill and the MLX 4-bit, Q8_0, UD-Q6_K_XL and UD-Q8_K_XL decodes are within 0.6%.
…de tile The F32 register decode tile ran Q6_K through operand_f32 (128 + code), each 16-input chain seeded from the table with -160 times the sum of its inputs: a seed load per chain and the operand's half subtraction. Q6_K is 43-60% of the Unsloth UD-Q5 and UD-Q6 files' bytes and the output layer of every 4- to 6-bit Qwen3.8-27B GGUF. Its F32 chains now take nibble6_operand: the nibble at bits 4-7 and the high bits at 8-9 of a half, 1 + code / 64, less 1.5, the exact (code - 32) / 64. The zero point is in the operand, so the chains start from zero, read no seed and close with fma(dot, 64 s). The Apple9 instances are unchanged. M2 Max, ms at one to four lanes: q6k 17408x5120 gate/up 0.549 0.965 1.258 1.717 -> 0.533 0.862 1.159 1.530 q6k 5120x17408 down 0.289 0.513 0.659 0.894 -> 0.280 0.456 0.609 0.810 q6k 5120x6144 out 0.112 0.211 0.271 0.370 -> 0.109 0.172 0.255 0.335 q6k 248320x5120 output 3.471 6.066 8.170 10.965 -> 3.335 5.621 7.518 9.572 q5k, q3k, af4g64 (unchanged code) within 0.2% The staged tile no longer beats the register tile on Q6_K at two and four lanes (17408x5120: 0.881 and 1.599 ms). 27B decode at one, two and four lanes, tok/s: UD-Q6_K_M 69.3 86.3 98.5 -> 70.3 90.8 104.1 UD-Q6_K_XL 68.5 89.7 102.4 -> 69.6 93.1 106.9 UD-Q5_K_M 71.5 85.7 98.7 -> 72.5 89.6 103.0 UD-Q5_K_XL 70.5 86.3 99.2 -> 71.4 89.5 102.7 UD-Q4_K_XL 76.0 90.8 104.5 -> 76.5 92.0 106.0 Q4_0, Q4_1 and UD-Q4_K_M (their output is Q6_K) gain up to 0.9-2.2% at four lanes; prefill and the MLX 4-bit, Q8_0, UD-IQ4_XS and UD-Q8_K_XL decodes are within 0.5%.
This was referenced Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This follows the 5-bit nibble operand PR (#18). Q6_K is the other large format in the Unsloth UD files: 43–60% of
UD-Q5_K_M's, UD-Q5_K_XL's and UD-Q6_K_M's bytes. It's also the
outputlayer of every 4- to 6-bit Qwen3.8-27B GGUF,1.04 GB read for every generated token. As before, I used Claude Code to author it and tested it by hand.
What changed
Only the Apple7/8 F32 instances of the register decode tile (
linear_gguf_sgmatrix.metal) change. Apple9 and hostcode are untouched.
high bits to bits 8–9, which gives 1 + code/64; subtracting 1.5 then leaves (code − 32)/64.
(−160 × the sum of 16 inputs). That saves a seed load per chain, and the epilogue is a single
fma(dot, 64 s).operand_f32(128 + code), with each chain seeded from the table.gguf-projectionalready runs every format through the F32 instances: natively on Apple7/8, and onApple9+ through PR perf: zero points as mins and nibble operands in the Apple7/8 register decode tile #15's pass.
Results (M2 Max, 38-core GPU)
Kernel time,
gguf-projection-benchmark, at 1 / 2 / 3 / 4 lanes, ABBA order; Δ: − is faster. Before is the 5-bitnibble operand branch.
Before this change, the staged tile beat the register tile, which the policy picks, for Q6_K at two and four lanes.
On the 17408×5120 gate/up, the best staged split took 0.881 ms at two lanes and 1.599 ms at four, against the register
tile's 0.965 and 1.717. Now the register tile takes 0.862 and 1.530 ms, so it's the fastest at every lane count and no
policy change is needed.
All models on the M2 Max, after the change
make test-performance-real, Qwen3.8-27B, ANE share 0.35. Times are in seconds; decode is in tok/s at 1 / 2 / 4lanes. The % against MLX 4-bit is in brackets (time: + is slower; decode: − is slower).
Q8_0 was rerun on its own: the first all-model run caught a system stall in two of its three 14K cold samples.
Decode before and after
The same benchmark on the 5-bit nibble operand branch, run earlier the same evening, in tok/s at 1 / 2 / 4 lanes;
Δ: + is faster. The MLX 4-bit runs of the two builds agree within 0.1%. The Q6_K share is of each file's tensor bytes.
10–18% faster against 3–4% at one lane. The files with 42–60% Q6_K decode 1.2–1.5% faster at one lane and 3.6–5.7%
at two and four.
UD-Q4_K_M gain up to 0.9–2.2% at four lanes.
apple7-m1-1.3.2: UD-Q6_K_M decodes at 70.3 / 90.8 / 104.1 tok/s, against 68.6 /86.0 / 98.4 (+2.5 / +5.6 / +5.8%). UD-Q5_K_M decodes at 72.5 / 89.6 / 103.0, against 70.3 / 85.0 / 97.7 (+3.1 / +5.4
/ +5.4%).
Testing
gguf-projectionpasses on the M2 Max (all 43 formats, F32 instances natively).make checkpasses on the M2 Max.make test-performance-realon every model above, after; before is the 5-bit nibble operand branch.