Repository navigation
Conversation
…ter tile Apple9's register decode tile ran Q4_0 and Q5_0 as zero-point formats: each 16-input half of a group was a chain of its own, seeded from the table with -160 times the sum of its inputs, with a seed load per lane and a coefficient per half. Apple9 now decodes them in a view of their own (Apple9View, ZeroAsMin) that takes the zero point as a min, m = -zero d: a group enters the MMA as 128 + code in one chain, closed against the table's sum of its 32 inputs with b = -(128 + zero) d, exact, as Q4_1's and Q5_1's groups are, with half the coefficients. Its kernels are the gguf_decode_sg9_ instances, and a fused segment in the view carries GGUF_DECODE_APPLE9_VIEW in its format; gguf_apple9_view names the formats on both sides. Apple10, whose narrow grids run the register tile, and the MoE expert kernels decode the formats as before. The outputs move within the fp64 bound gguf-projection checks: the chain's sums round in another order. 10-core M4, ms at one to four lanes (kernel-ab, three ABBA passes; an A/A pass is within 0.2%, 0.5% at one lane): q40 17408x5120 gate/up 1.154 2.068 3.040 3.945 -> 1.112 1.979 2.812 3.818 q40 5120x17408 down 0.580 1.032 1.521 1.973 -> 0.560 0.983 1.398 1.908 q50 17408x5120 gate/up 1.354 2.240 3.201 4.214 -> 1.306 2.155 3.078 4.111 q50 5120x17408 down 0.684 1.121 1.602 2.120 -> 0.656 1.080 1.540 2.054 q4k, q41, af4g64 (unchanged code) within 0.3% gguf-projection also runs the register decode suite in the other family's view of these formats, so that an Apple9 GPU tests Apple10's seeded chains and an Apple10 GPU Apple9's.
…s register tile Apple9's register decode tile entered Q6_K's and Q3_K's codes into the MMA as 160 + code - zero, each 16-input chain seeded from the table with -160 times the sum of its inputs: a seed load per chain and lane. Apple9 now decodes them in a view (Centered) whose operand is code - zero, -32..31 and -4..3: 128 + code in one integer add, less 128 + zero in bf16, exact. The chains start from zero and read no seed. The operand costs one subtraction a pair of weights, which every lane of a threadgroup shares, and saves a seed load per chain for each lane, so the gain grows with the lanes. Apple10 and the MoE expert kernels decode the formats as before. The outputs move within gguf-projection's fp64 bound. 10-core M4, ms at one to four lanes (kernel-ab, ABBA passes free of other GPU work; an A/A pass is within 0.2%, 0.5% at one lane): q6k 17408x5120 gate/up 1.495 2.186 3.178 4.210 -> 1.496 2.143 3.085 3.988 q6k 5120x6144 out 0.284 0.399 0.579 0.760 -> 0.285 0.392 0.557 0.722 q6k 248320x5120 output 10.21 17.08 24.86 32.59 -> 10.34 17.05 23.81 30.83 q3k 17408x5120 gate/up 1.329 2.229 3.220 4.241 -> 1.328 2.228 3.147 4.107 q4k (unchanged code) within 0.1% The output layer's one-lane difference (+1.2%) is within that case's spread from run to run. With both commits the 27B in Q4_0 (its output layer Q6_K) takes 3.6, 1.7, 3.3 and 3.0% less GPU time per decode step at one to four lanes on the 10-core M4 than main (backend-benchmark's decode scenario, Neural Engine off, ABBA x 3); its output tokens are the same.
PQ2_0, Prism ML's ternary format (zero point 1, one scale per 128 inputs), ran on Apple9's register decode tile as Q4_0 did before ZeroAsMin: two 16-input chains per group seeded from the table. Ternary-Bonsai-2-27B is 99% PQ2_0. Apple9 now decodes it in ZeroAsMin's view too: one 32-input chain closed against the table's sum with b = -129 d, exact. Its coefficient unit is unchanged, so the host's K splits are too. Apple10 and the MoE expert kernels decode it as before. The outputs move within gguf-projection's fp64 bound. 10-core M4, ms at one to four lanes (kernel-ab, ABBA): pq20 17408x5120 gate/up 1.139 2.028 2.987 3.843 -> 1.102 1.941 2.790 3.753 pq20 5120x17408 down 0.571 1.025 1.491 1.926 -> 0.552 0.991 1.420 1.915 Ternary-Bonsai-2-27B takes 2.3, 2.4, 2.3 and 2.2% less GPU time per decode step at one to four lanes than main (backend-benchmark's decode scenario, Neural Engine off, ABBA x 2); its output tokens are the same.
Collaborator
|
Thanks @ghisguth, both ideas were worth measuring. I couldn't push to your branch, so #602 carries your three commits with your authorship, stacked on #600 (the register tile decodes each coefficient once per coefficient group and adds a min's term once per group; it merges first), plus one commit that replaces the Apple9-only views with a rule every family's register tile shares. I'm leaving this PR open for now. What your two rules measured as shared rules, on a 10-core M4 and a 32-core M5 Max:
The rule that won keeps your idea of taking the zero point like a min: where one scale covers 32 inputs or more (Q4_0, Q5_0, IQ1_S, PQ2_0, Q2_0), chains cover 32 inputs, and the zero point's term, −160 times the table's 32-input sums, starts the last chain of each coefficient group, at most once per 64-input span, the way a min's term is added. The views, the sg9 kernels, the ABI bit and the family branch are gone, and the table is unchanged. Over #600 on the M4 (one to four lanes): Q4_0 −0.8/−3.5/−3.9/−4.4%, Q5_0 −0.4/−3.1/−6.5/−4.5%, Q2_0 −2.8/−2.8/−2.3/−4.6%, PQ2_0 −2.6/−3.1/−2.4/−4.7%, IQ1_S −2.1/−3.1/−4.9/−4.0%, with no shape slower. On the M5 Max's narrow grids no row is slower than main. Tests on the M5 Max: gguf-dequant, gguf-projection, linear-plan, gguf-moe, gguf-moe-e512 and gguf-moe-sigmoid under shader validation, |
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base:
incoai/splashmain(3135ab4). Measured on 1.4.0 (1482f6b) before rebasing. None of upstream's changes since then touch the files this PR changes.Branch:
ghisguth/apple9-zero-points(2c12b35)I have been working on the Apple7/8 decode tile in paperniuk's M1/M2 fork for a while. Most of what we changed there is for the F32 tile, and the Apple9 bf16 tile doesn't have those problems. But two ideas looked like they could carry over (paperniuk#15, paperniuk#19 and paperniuk#21 there, tested on an M1 Max too), so I tried them on my MacBook Air M4 (10-core GPU, 32 GB), which runs upstream's Apple9 register tile for decode. Both help. The first one helps at every lane count, the second mostly with three and four lanes. I used Claude Code to author it and tested it by hand.
What changed
Three commits. They change only what Apple9 runs. Apple10 runs this tile on narrow grids, and it decodes these formats as before. So do the MoE expert kernels.
ZeroAsMin<F>, a view with m = −zero·d. This is exact, the zero points are powers of two. A group of 32 inputs then enters the MMA as 128 + code in one chain, closed against the table's sum with b = −(128 + zero)·d, the way Q4_1 and Q5_1 already work. Before, each half of the group was a chain of its own, seeded from the table, with a seed load per lane and a coefficient per half. The operand is upstream's existing one-add128 + code, so there is no new operand code.Centered<F>, whose operand is code − zero (−32..31 and −4..3): upstream's 128 + code, then one bf16 subtraction, exact. The 16-input chains start from zero and read no seeds. The subtraction is per weight pair and shared by the lanes, the saved seed load is per chain and lane, so the gain grows with lanes.Apple9View<F>names the type Apple9 decodes F as, andgguf_apple9_view()inmetal/abi/Gguf.hnames the same formats for the host. A static_assert keeps the two in sync. Apple9 getsgguf_decode_sg9_<fmt>_*instances for these five formats. A fused segment in such a format carriesGGUF_DECODE_APPLE9_VIEWin itsfmt.Linear::addGgufRegisterpicks both whenfamily_ == Apple9.gguf-projectionalso runs the register decode suite through aLinearof the other family for these formats. So an Apple9 GPU still tests the seeded chains Apple10 runs, and an Apple10 GPU tests Apple9's views.gguf-projectionchecks. Apple10's outputs don't change.I kept it to Apple9 on purpose, since that is what I can measure. M5 runs the same tile on narrow grids and I have no M5, so this PR doesn't touch what it runs. The first idea may help there too, and then it would be simpler: a few lines in
Shape<F>instead of the views, covering the fused and MoE kernels as well. If someone with an M5 can measure it, I am happy to do that version as a follow-up.Results (MacBook Air M4, 10-core GPU)
Kernel time,
gguf-projection-benchmark, at 1 / 2 / 3 / 4 lanes, the policy's config, ABBA order. The Air is fanless and throttles, so each run waits for a cooldown. Δ: − is faster. An A/A run of the same build is within 0.2%, 0.5% at one lane.The q6k output layer is 1 GB and drifts 2–5% within a session on the Air, so its one-lane +1.3% is noise.
End to end, decode only:
backend-benchmark --scenario decode --samples 3 --ane-ffn-share 0, main against the branch, each build's own binary, ABBA. GPU time per decode step in ms at 1 / 2 / 3 / 4 lanes. Output tokens and acceptance are the same on both builds.Testing
make checkpasses on the M4 Air on the rebased branch.gguf-projectionpasses on the M4 Air after each commit, and with and withoutMTL_SHADER_VALIDATIONon the rebased branch: all 43 formats on Apple9, and the five formats through Apple10'sLineartoo.make test-performance-realon main and the branch with MLX 4-bit, Q4_0, PQ2_0 (Bonsai) and UD-IQ3_XXS: the same output tokens and draft acceptance on every model.Need help on this PR? Tag
@codesmith-botwith what you need.