Skip to content

Decode zero-point formats with fewer chains on every family's register tile - #602

Merged
jianc99 merged 6 commits into
incoai:mainfrom
jianc99:perf/register-tile-zero-points
Oct 27, 2026
Merged

Decode zero-point formats with fewer chains on every family's register tile#602
jianc99 merged 6 commits into
incoai:mainfrom
jianc99:perf/register-tile-zero-points

Conversation

@jianc99

@jianc99 jianc99 commented Oct 27, 2026 •
edited by blacksmith-sh Bot
Loading

Copy link
Copy Markdown
Collaborator

Carries #569 by @ghisguth, his three commits with his authorship, stacked on #600, which merges first; until then this diff also shows #600's two commits.

The register tile chained zero-point codes 16 inputs at a time, each chain seeded from the table, also where one scale covers 32 inputs or more. #569 decoded Q4_0, Q5_0 and PQ2_0 with their zero point as a min and Q6_K and Q3_K with it in the operand, through Apple9-only views of those formats.

Measured as rules shared by every family, on a 10-core M4 and a 32-core M5 Max:

  • The zero point in the operand (Q6_K, Q3_K, IQ1_M) was slower than main at one and two lanes on both: Q3_K +3.5% at one lane on the M4, IQ1_M +3.8% on the M5 Max. Those formats keep main's chains.
  • The zero point as a min was faster than main, but slower than 32-input seeded chains on both: Q4_0 at two lanes on the M4, −2.9% against −5.6%.

The last commit keeps the zero point as a term per coefficient group, like a min. Where one scale covers 32 inputs or more (Q4_0, Q5_0, IQ1_S, PQ2_0, Q2_0), chains cover 32 inputs, and the zero point's term, −160 times the table's 32-input sums, starts the last chain of each coefficient group, at most once per 64-input span. It removes the Apple9 views, the gguf_decode_sg9_* kernels, the GGUF_DECODE_APPLE9_VIEW bit, gguf_apple9_view and the family branch; the table is unchanged.

Over #600 on the M4:

L1 L2 L3 L4
Q4_0 −0.8% −3.5% −3.9% −4.4%
Q5_0 −0.4% −3.1% −6.5% −4.5%
Q2_0 −2.8% −2.8% −2.3% −4.6%
PQ2_0 −2.6% −3.1% −2.4% −4.7%
IQ1_S −2.1% −3.1% −4.9% −4.0%

No shape is slower, and on the M5 Max's narrow grids no row is slower than main.

Tests. On the M5 Max, make test-engine (Metal under shader validation), gguf-dequant, gguf-projection, linear-plan, gguf-moe, gguf-moe-e512 and gguf-moe-sigmoid under validation, and the Qwen3.6-35B-A3B MLX runtime oracle pass.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

jianc99 and others added 6 commits October 26, 2026 13:13
@jianc99
Decode the register tile's coefficients once per coefficient group
dbe16c1 
The register tile (Apple9, and Apple10's narrow grids) decoded a column's
scale (and min) into threadgroup memory once per MMA chain, every 32 inputs
or 16 for zero-point codes, also where one coefficient covers several
chains: MLX affine's groups of 64 and 128, PQ2_0's 128, Q2_0's 64, FP8's
and BF16's 256, and the 32-input scales of Q4_0, Q5_0 and IQ1_S. Half to
seven eighths of those decodes, with their meta loads and threadgroup
stores, repeated a value.

Each format now names the inputs one coefficient covers as its Group (16,
32, or 32 * MetaGroups where the groups of a meta unit share one), and the
register tile decodes one coefficient per column and group, which each of
the group's chains reads (the x halves of a QuantCoef, equal for such a
coefficient). Chains, epilogues and outputs are unchanged bit for bit; the
kernels of formats whose coefficient covers one chain (the K-quants, the
IQ formats but IQ1_S, Q8_0, Q4_1, Q5_1, MXFP4, NVFP4, MLX affine g32)
compile to the same AIR as before.

On the 10-core M4 (Apple9), MLX 4-bit g64 at eight 27B and 35B projection
shapes (GDN in and out, q|k|v, draft q|k|v and down, the 27B's down, the
LM head), the policy's splits, ABBA against main: 1.5% less time at one
lane, 1.6% at two, 0.9% at three, 0.4% at four.
@jianc99
Add a min's term once per coefficient group on the register tile
e30bd16 
Each MMA chain of the register tile covers 32 inputs and closed with
s * chain + b * sum, so MLX affine's groups of 64 and 128 added their min
term, b times the sum of the group's inputs, two and four times. Apple9 runs
the MMA on the FP32 pipe that this epilogue shares, and from two lanes the
tile is bound by that pipe.

A format with a min now adds b times the sum of those inputs once per
coefficient group, but at least once per 64-input span of the table, with
the last chain of either, from the span's 32-input sums; every chain still
closes with its scale. Shape asserts that a min term's chains are 32-input
groups lying in one span, which keeps its sum reads inside the current
span. Only the MLX affine g64 and g128 kernels change, their outputs by
reassociation; every other format's compile to the same AIR as before.

Measured on the 10-core M4 (Apple9), the policy's splits, ABBA:
- Chains keep their 32 inputs: one chain over a 64-input group leaves a
  lane two dependent chains per simdgroup where it had four, and took 1-6%
  more time at one to three lanes.
- A group of 128 adding its term once, from four sums gathered across two
  spans, took 4.5-8.8% more time at four lanes than a term per span (8.8%
  in MLX 8-bit g128 2048 x 4096).
- Written over per-unit table pointers, the change also moved the GGUF
  formats' code and they measured 0.2-1.2% slower; their kernels are now
  unchanged.

MLX 4-bit g64 at the 19 projection shapes of the 27B and 35B and their
drafts, with the previous commit, against main: up to 2.7% less time per
shape at one lane (none on the 35B's GDN in-projection, most on the LM
heads and the 35B draft's down projection), 1.7-3.7% at two and three and
3.3-4.6% at four; over a B1 DFlash2 cycle's calls of those shapes, 16.67
instead of 16.89 ms on the 35B and 165.80 instead of 168.06 ms on the 27B.
MLX 4-bit and 8-bit g128: 1.7-1.9% less in total at one to three lanes and
3.6% at four; MLX 8-bit g64 4-5% less at four lanes.

The tile also runs Apple10's narrow grids (apple10Registers): on the
32-core M5 Max the 35B's 2048-column MLX 4-bit g64 projections, which take
it at one lane, take 1.6-2.2% less time at their splits, and the
320 x 10240 hyper-connection projections 0.4-1.8% less at one to three
lanes and 4.4-5.2% at four. Partitions of under 512 inputs at sixteen
splits take 2-10% more, which the tiers never pick.
@ghisguth @jianc99
Decode Q4_0 and Q5_0 with their zero point as a min on Apple9's regis…
86542b4 
…ter tile

Apple9's register decode tile ran Q4_0 and Q5_0 as zero-point formats:
each 16-input half of a group was a chain of its own, seeded from the
table with -160 times the sum of its inputs, with a seed load per lane and
a coefficient per half.

Apple9 now decodes them in a view of their own (Apple9View, ZeroAsMin)
that takes the zero point as a min, m = -zero d: a group enters the MMA as
128 + code in one chain, closed against the table's sum of its 32 inputs
with b = -(128 + zero) d, exact, as Q4_1's and Q5_1's groups are, with
half the coefficients. Its kernels are the gguf_decode_sg9_ instances, and
a fused segment in the view carries GGUF_DECODE_APPLE9_VIEW in its format;
gguf_apple9_view names the formats on both sides. Apple10, whose narrow
grids run the register tile, and the MoE expert kernels decode the formats
as before. The outputs move within the fp64 bound gguf-projection checks:
the chain's sums round in another order.

10-core M4, ms at one to four lanes (kernel-ab, three ABBA passes; an A/A
pass is within 0.2%, 0.5% at one lane):
  q40 17408x5120 gate/up  1.154 2.068 3.040 3.945 -> 1.112 1.979 2.812 3.818
  q40 5120x17408 down     0.580 1.032 1.521 1.973 -> 0.560 0.983 1.398 1.908
  q50 17408x5120 gate/up  1.354 2.240 3.201 4.214 -> 1.306 2.155 3.078 4.111
  q50 5120x17408 down     0.684 1.121 1.602 2.120 -> 0.656 1.080 1.540 2.054
  q4k, q41, af4g64 (unchanged code) within 0.3%

gguf-projection also runs the register decode suite in the other family's
view of these formats, so that an Apple9 GPU tests Apple10's seeded
chains and an Apple10 GPU Apple9's.
@ghisguth @jianc99
Decode Q6_K and Q3_K with their zero point in the operands on Apple9'…
c3a4486 
…s register tile

Apple9's register decode tile entered Q6_K's and Q3_K's codes into the MMA
as 160 + code - zero, each 16-input chain seeded from the table with -160
times the sum of its inputs: a seed load per chain and lane.

Apple9 now decodes them in a view (Centered) whose operand is code - zero,
-32..31 and -4..3: 128 + code in one integer add, less 128 + zero in bf16,
exact. The chains start from zero and read no seed. The operand costs one
subtraction a pair of weights, which every lane of a threadgroup shares,
and saves a seed load per chain for each lane, so the gain grows with the
lanes. Apple10 and the MoE expert kernels decode the formats as before.
The outputs move within gguf-projection's fp64 bound.

10-core M4, ms at one to four lanes (kernel-ab, ABBA passes free of other
GPU work; an A/A pass is within 0.2%, 0.5% at one lane):
  q6k 17408x5120 gate/up    1.495 2.186 3.178  4.210 -> 1.496 2.143 3.085  3.988
  q6k 5120x6144 out         0.284 0.399 0.579  0.760 -> 0.285 0.392 0.557  0.722
  q6k 248320x5120 output   10.21 17.08 24.86  32.59  -> 10.34 17.05 23.81  30.83
  q3k 17408x5120 gate/up    1.329 2.229 3.220  4.241 -> 1.328 2.228 3.147  4.107
  q4k (unchanged code) within 0.1%
The output layer's one-lane difference (+1.2%) is within that case's
spread from run to run.

With both commits the 27B in Q4_0 (its output layer Q6_K) takes 3.6, 1.7,
3.3 and 3.0% less GPU time per decode step at one to four lanes on the
10-core M4 than main (backend-benchmark's decode scenario, Neural Engine
off, ABBA x 3); its output tokens are the same.
@ghisguth @jianc99
Decode PQ2_0 with its zero point as a min on Apple9's register tile
fd7ea25 
PQ2_0, Prism ML's ternary format (zero point 1, one scale per 128 inputs),
ran on Apple9's register decode tile as Q4_0 did before ZeroAsMin: two
16-input chains per group seeded from the table. Ternary-Bonsai-2-27B is
99% PQ2_0.

Apple9 now decodes it in ZeroAsMin's view too: one 32-input chain closed
against the table's sum with b = -129 d, exact. Its coefficient unit is
unchanged, so the host's K splits are too. Apple10 and the MoE expert
kernels decode it as before. The outputs move within gguf-projection's
fp64 bound.

10-core M4, ms at one to four lanes (kernel-ab, ABBA):
  pq20 17408x5120 gate/up  1.139 2.028 2.987 3.843 -> 1.102 1.941 2.790 3.753
  pq20 5120x17408 down     0.571 1.025 1.491 1.926 -> 0.552 0.991 1.420 1.915
Ternary-Bonsai-2-27B takes 2.3, 2.4, 2.3 and 2.2% less GPU time per decode
step at one to four lanes than main (backend-benchmark's decode scenario,
Neural Engine off, ABBA x 2); its output tokens are the same.
@jianc99
Add a zero point's term once per coefficient group on every family's …
870f07d 
…register tile

Apple9's register tile took Q4_0's, Q5_0's and PQ2_0's zero point as a min
and Q6_K's and Q3_K's into the operand through views of their own
(ZeroAsMin, Centered, Apple9View), with gguf_decode_sg9_ kernels, a
GGUF_DECODE_APPLE9_VIEW bit in the fused segments' formats and
gguf_apple9_view on both sides, chosen by a family branch in
Linear::addGgufRegister. Apple10's narrow grids and the MoE experts kept
chains of 16 inputs, each seeded from the table.

One rule now serves every family: a zero point is a term of its
coefficient group, as a min is, entered where it costs no epilogue. Where
a scale covers 32 inputs or more (Q4_0, Q5_0, IQ1_S, PQ2_0, Q2_0), chains
cover 32 inputs and the last chain of each group, at most of each 64-input
span, starts from -160 times the sum of the group's inputs, from the
table's sums of 32; the operand stays 160 + code - zero. Scales of 16
inputs (Q6_K, Q3_K, IQ1_M) and Q2_K keep their chains seeded from the
table per 16 inputs. The views, the sg9 kernels, the fused-format bit,
gguf_apple9_view, the family branch and the test of the other family's
view are gone; the table, the host's splits and every other format's
kernels are unchanged. Outputs change by the seed's rounding, as the
table's seeds round.

Neither of the views' rules held for both families (ABBA with
gguf-projection-benchmark at the policy's register splits):
- code - zero as the operand, chains from zero: slower than seeded chains
  at one and two lanes on both. On the 10-core M4 Q3_K +3.5% at one lane,
  Q6_K +1.4% at two, IQ1_M +0.9% at one (2.0-5.9% less at three and
  four); on the 32-core M5 Max's narrow grids IQ1_M +3.8-3.9% at one
  lane, Q6_K +2.2%, Q3_K +1.7%. Apple9's Q6_K and Q3_K give back that
  three- and four-lane gain.
- the zero point as a min, an epilogue term b * sum: faster than main but
  slower than seeded chains of 32 on both, Q4_0 at two lanes on the M4
  -2.9% against this rule's -5.6% (-4.7% with chains of 32 seeded by the
  sum of two 16-input seeds), and up to 3.3 points behind this rule
  (IQ1_S at one lane).

With the previous two commits, against main, totals over the shapes, one
to four lanes:
10-core M4, 2048x4096, 12288x5120, 17408x5120 gate/up and 5120x17408:
  q40  -1.4 -5.6 -5.0 -5.8    q50  -2.3 -5.7 -6.1 -5.8
  q20  -6.4 -5.3 -3.8 -6.2    pq20 -5.4 -5.6 -3.5 -5.9
  iq1s -7.1 -7.4 -8.0 -7.5
  (Q6_K, Q3_K, IQ1_M, Q2_K and Q4_K within 0.5%)
32-core M5 Max, 320 to 2048 columns at the lanes that run the tile:
  q40  -4.2 -2.1 -2.8 -2.7    q50  -2.8 -2.5 -5.9 -2.6
  q20  -3.9 -4.3 -3.2 -3.7    pq20 -2.7 -3.0 -2.5 -3.3
  iq1s -1.0 -2.2 -3.4 -3.0
No format, shape and lane count is slower than main beyond noise on
either machine. This commit alone, over the coefficient-group commits
(e30bd16), on the M4:
  q40  -0.8 -3.5 -3.9 -4.4    q50  -0.4 -3.1 -6.5 -4.5
  q20  -2.8 -2.8 -2.3 -4.6    pq20 -2.6 -3.1 -2.4 -4.7
  iq1s -2.1 -3.1 -4.9 -4.0
@jianc99 jianc99 mentioned this pull request Oct 27, 2026
Closed
@jianc99
jianc99 merged commit c0d00e8 into incoai:main Oct 27, 2026
5 checks passed
@ghisguth ghisguth mentioned this pull request Oct 30, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jianc99 @ghisguth