Repository navigation
perf: Q3_K's zero point in its operands, and epilogue factors once per coefficient, in the Apple7/8 register decode tile - #21
Conversation
…de tile The F32 register decode tile ran Q3_K through operand_f32, each 16-input chain seeded from the table, as Q6_K was. Its 3-bit codes now enter the MMA as the exact half (code - 4) / 8: the two low bits at bits 7-8 and the hmask bit at 9, 1 + code / 8, less 1.5. The chains start from zero, read no seed and close with fma(dot, 8 s). CenteredChunk now holds each centered format's operand and scale. The Apple9 instances are unchanged. M2 Max, ms at one to four lanes: q3k 17408x5120 gate/up 0.586 1.005 1.300 1.767 -> 0.575 0.975 1.218 1.595 q3k 5120x17408 down 0.308 0.529 0.677 0.920 -> 0.300 0.515 0.630 0.842 q6k (unchanged) within 0.2% Q3_K is 5.6% of UD-IQ4_XS's bytes and 0.7-1.6% of the UD-Q4 files'.
…nt on Apple7/8 The F32 per-32 nibble chains (Q4_K, Q4_1, Q4_0 and Q5_0 as ZeroAsMin, Q5_K, Q5_1, MLX affine 4-bit in groups of 32) closed each chain with fma(dot, 64 s) and fma(sum, 64 s + (m - 128 s)), two operations per chain and column fragment that every chain of a coefficient repeated. The coefficients of those formats now hold 64 s and m - 64 s, computed once when they are decoded, with the same rounding, and the centered chains' (Q6_K, Q3_K) hold Scale s rather than multiplying each chain's s. The span chains (MLX affine in groups of 64 or 128) and the Apple9 instances are unchanged. M2 Max, ms at one to four lanes, two ABBA passes: q4k 17408x5120 gate/up 0.434 0.771 1.089 1.404 -> 0.429 0.758 1.067 1.408 q4k 5120x17408 down 0.228 0.411 0.577 0.729 -> 0.225 0.401 0.564 0.732 q5k 17408x5120 gate/up 0.520 0.907 1.193 1.606 -> 0.517 0.897 1.180 1.581 q5k 5120x17408 down 0.275 0.478 0.622 0.835 -> 0.273 0.471 0.615 0.831 q40 17408x5120 gate/up 0.423 0.767 1.103 1.416 -> 0.420 0.756 1.079 1.421 q6k 17408x5120 gate/up 0.533 0.863 1.164 1.531 -> 0.531 0.862 1.151 1.530 q6k 5120x17408 down 0.280 0.457 0.608 0.808 -> 0.278 0.457 0.605 0.803 q3k 17408x5120 gate/up 0.574 0.975 1.220 1.593 -> 0.571 0.963 1.223 1.588 af4g64 (unchanged) within 0.1%
d81a227 to
86a2c7f
Compare
|
@paperniuk Done, rebased onto apple7-m1-1.3.2. Only the Q3_K (463b6ee) and epilogue (86a2c7f) commits are left. The resulting code is identical to what I tested before (make check, gguf-projection and the all-model runs in the description), so those numbers still apply. Thanks for merging #18–#20! |
|
Merged, thanks for the rebase! Here are the M1 Max numbers I promised in #18. End to end on
Kernel time on M1 Max (32-core GPU),
|
… 1.4 port # Conflicts: # dev/benchmarks/attention_sweep.mm
|
@paperniuk Thank you for merging and for running it on M1 Max! It's nice to see it holds there too, even a bit better than on my M2 Max at 1 lane. I have two more PRs for decode on M1/M2 coming, both bit exact. One is for GDN verify kernel (−35% kernel time), another one for the decode norm (−60%). Here are preliminary numbers from my M2 Max, decode tok/s at 1 / 2 / 4 lanes, on apple7-m1-1.4 (A = apple7-m1-1.4, B = A + GDN verify, C = A + decode norm):
Prefill is the same. I didn't measure both together yet. Will send PRs to 1.3.2 once I finish end to end runs there with more quants. Also, while testing on apple7-m1-1.4, I saw some tests failing on base too: q8_chunked_prefill_metal_test.mm does not build (#24 has the fix), and on my M2 Max gguf-moe, qwen4-kernels, paged-attention-plan and metal-backend fail with base metallib. Not sure if it's M2 specific, just letting you know. |
Two small follow-ups to the 5-bit (#18) and Q6_K PRs (#19), in the same Apple7/8 F32 register decode tile, as two commits. As
before, I used Claude Code to author them and tested them by hand.
What changed
Only the Apple7/8 F32 instances of the register decode tile (
linear_gguf_sgmatrix.metal) change. Apple9 and hostcode are untouched.
(code − 4)/8: the two low bits go to bits 7–8 and the hmask bit to bit 9, which gives 1 + code/8, less 1.5. The
per-16 chains start from zero, read no seed and close with one fma.
CenteredChunknow holds each centeredformat's operand and scale.
4-bit in groups of 32) closed every chain with
fma(dot, 64 s)andfma(sum, 64 s + (m − 128 s)), recomputing thetwo factors per chain. They're now computed once when the coefficient is decoded, with the same rounding. The
centered chains (Q6_K, Q3_K) likewise hold Scale·s. MLX 4-bit's span chains keep their epilogue: precomputing there
was 0.3–0.4% slower at two and four lanes.
gguf-projectionalready runs every format through the F32 instances.Results (M2 Max, 38-core GPU)
Kernel time,
gguf-projection-benchmark, at 1 / 2 / 3 / 4 lanes, ABBA order; Δ: − is faster. Each commit is measuredagainst the one before it: the (1) rows compare #19 without and with the Q3_K commit, and the (2) rows compare the
Q3_K commit without and with the epilogue commit.
Q4_K and Q4_0 take 0.3% more at four lanes, consistently across two ABBA passes. One to three lanes gain 1–2%, so I
kept it for them.
End to end (M2 Max), before and after
make test-performance-real, ANE share 0.35, decode in tok/s at 1 / 2 / 4 lanes; Δ: + is faster. Before is the Q6_Kbranch, or for UD-Q4_K_M the Q6_K branch plus the MLX PR and the Q3_K commit. After is this branch. Prefill is
unchanged (14K cold within 0.02 s on every model). MLX 4-bit runs before and after the GGUF runs agree within 0.3%.
These are small, and UD-Q4_K_M's are within run-to-run noise. The gains show mostly where Q4_0 or Q5_K dominate. Q3_K
alone, for UD-IQ4_XS (5.6% Q3_K), measured +0.1 / +0.1 / +0.4%.
All four PRs together, against
apple7-m1-1.3.2The same benchmark on every model, decode in tok/s at 1 / 2 / 4 lanes (Δ: + is faster); prefill is unchanged:
Testing
gguf-projectionpasses on the M2 Max after each commit (all 43 formats, F32 instances natively).make checkpasses on the M2 Max.