[ghisguth]

@paperniuk this one is a small one for Flash-Next decode on M1/M2. In the profile of a verify cycle on my M2 Max the
injecting hyper-connection norm is ~4.5% of a one-lane cycle. Its four quarters each read all four streams of x and of
the norm weights and work out the same normalized values again. This works out each stream's values once and keeps
them in threadgroup memory for the other quarters. Outputs are bitwise the same. The norm is ~20% faster (0.5–0.6 ms
less per cycle), so the cycle is 1.1–1.3% faster at one lane and ~0.6% at two and three. Small, but it is in every
Flash-Next cycle. As before, I used Claude Code to author it and tested it by hand.

What changed

• kernels/shared/hyper_connection.metal: hyper_connection_norm_staged. Same threadgroup of 1024 threads per row as
  hyper_connection_norm_quarters, one quarter per stream. Each stream's quarter works out v = x * scale * norm once,
  writes the bf16 output and leaves v (fp32) in threadgroup memory, two streams at a time (20 KB, four would be 40 KB).
  Then every quarter's thread runs its dot over the staged values at its own positions. The dots run in the same order
  as before, so the output and the weights are bit for bit the quarters kernel's. The two kernels share one
  hyper_norm_quarter_row template.
• ExecutionPlans::hyperNormStaged(rows): true on Apple7/8 while rows * 4 <= gpuCores * 3. Qwen4ExpTarget
  passes it to addNorm for the prefill and verify norms. Other families, and more rows, keep the quarters kernel.
• qwen4_kernels_test.mm: runs the staged kernel beside the quarters one and compares output and weights byte for
  byte. execution_plans_test.cc: the row edge per family and core count.

Why the row limit: with many rows the staged kernel gets slower than the quarters one. On my M2 Max (38 cores) it is
faster up to 33 rows and 6–9% slower from 34 rows. Staging one stream at a time (10 KB) is slower
everywhere and jumps at the same place, so I don't think it is the threadgroup memory. I don't know the cause. Three
quarters of the cores keeps a margin below that edge. I measured the edge only on the M2 Max, so on M1 the limit is a
guess from the core count. I know you prefer not to pick by core count. Here both kernels give the same bits, so the
limit only decides which one is faster, not any output.

Results (M2 Max, 38-core GPU)

The norm alone, all 96 norms of a Flash-Next Q2_0 step, ms, quarters → staged, A B B A:

Rows	16	32	33	34	40	64
Quarters	2.73	2.80	2.82	2.80	2.81	2.90
Staged	2.10	2.27	2.24	3.01	3.04	3.13

decode-profile, Flash-Next, 2048-token prompt, int8 KV, fused GPU ms per cycle at 1 / 2 / 3 / 4 lanes, A B B A,
eight verify rows per lane (Q2_0 four runs per side, the others two). 1–3 lanes are 8–24 rows and take the staged
kernel. 4 lanes are 32 rows, over the limit for 38 cores (28 rows), so they keep the quarters kernel and are the same on
both sides:

Model	Before	After	Δ (%)
Q2_0	60.43 / 90.86 / 117.51 / 144.89	59.63 / 90.45 / 116.92 / 145.03	−1.3 / −0.5 / −0.5 / +0.1
Coder (IQ1_M)	71.16 / 103.72 / 134.74 / 162.64	70.40 / 102.75 / 133.92 / 162.47	−1.1 / −0.9 / −0.6 / −0.1
UD-IQ3_XXS	70.53 / 104.76 / 133.64 / 162.31	69.63 / 104.00 / 132.57 / 162.59	−1.3 / −0.7 / −0.8 / +0.2
Q2_0, 32K prompt	62.39 / 94.48 / 121.98 / 149.75	62.01 / 93.86 / 121.24 / 150.87	−0.6 / −0.7 / −0.6 / +0.7

The norm in the same runs, ms per cycle: 2.68–2.71 → 2.10–2.14 at one lane, 0.50–0.61 less at one to three lanes on
every model and context, the same at four. So the cycle saves about what the norm saves. The four-lane ±0.7% is the
run-to-run noise, it is the same kernels on both sides.

With the verify budget's own rows (noisier, the rows differ between runs):

Model	Before	After	Δ (%)
Q2_0	53.71 / 79.33 / 98.09 / 118.97	52.68 / 79.01 / 97.55 / 119.73	−1.9 / −0.4 / −0.5 / +0.6
Coder (IQ1_M)	61.18 / 87.68 / 109.98 / 136.72	60.45 / 87.33 / 109.69 / 134.38	−1.2 / −0.4 / −0.3 / −1.7
UD-IQ3_XXS	59.05 / 87.28 / 111.56 / 134.31	57.96 / 87.55 / 111.47 / 134.21	−1.8 / +0.3 / −0.1 / −0.1
Q2_0, 32K prompt	54.92 / 83.30 / 104.07 / 127.32	54.30 / 82.46 / 103.61 / 126.87	−1.1 / −1.0 / −0.4 / −0.4

Qwen3.8-27B Q4_0 (dense, no hyper connection) as a control: 81.36 / 143.25 / 195.30 / 255.81 → 81.66 / 143.26 /
195.47 / 255.89, the same.

Server, Flash-Next Q2_0 with my usual flags, 5 greedy 400-token requests per run, two runs each, A B B A. Per
request in cycles per second (decode tok/s over tok/cycle from the server log, which doesn't depend on how many draft
tokens get accepted): 18.62 → 18.82 on average (+1.1%). 3 of the 5 requests are faster in both runs, the other two
about the same. The log rounds tok/cycle to one decimal, so a single request is only good to ~2%. Median decode tok/s
54.81 / 54.82 before and 55.22 / 57.30 after, but that moves with acceptance more than with this.

Long agent chat replay in one of my tests (78 turns up to 125K context, each sent twice), server log, base → this
PR. Cycles per second is the decode speed without the effect of how many draft tokens get accepted:

Model	Context	Tok/cycle	Cycles/s	Δ cycles/s (%)
Q2_0	32K	3.16 → 3.84	17.11 → 16.65	−2.7
Q2_0	64K	4.33 → 4.31	16.20 → 16.36	+1.0
Q2_0	96K	4.54 → 4.46	15.91 → 16.10	+1.2
Q2_0	128K	5.00 → 5.07	15.60 → 15.65	+0.3
Coder	32K	3.42 → 3.41	14.86 → 15.24	+2.5
Coder	64K	4.23 → 4.24	14.40 → 14.48	+0.5
Coder	96K	4.65 → 4.67	14.05 → 14.11	+0.5
Coder	128K	5.02 → 5.04	13.59 → 13.68	+0.7

The Q2_0 32K row is only 4 requests, and at 30K the two builds wrote different answers (590 tokens at 3.9 tok/cycle
vs 210–228 at 3.1), so the budget verified more rows there and the cycles are not the same work. The one request
with the same answer is 17.09 → 16.82, inside the log's rounding of tok/cycle. Above 32K, where most of the requests
are, it is +0.3…+1.2%, about what the norm saves. Prefill is the same (paired requests −0.3% and +0.0%).

Testing

• qwen4-kernels passes with and without MTL_SHADER_VALIDATION, with the new staged vs quarters check (bit for bit
  without validation and in make check's uninstrumented qwen4-kernels bitwise run. Under validation the test
  compares the instrumented kernels within its tolerance, like its other checks).
• gguf-moe, gguf-moe-e512, gguf-moe-sigmoid and linear-plan pass with and without validation.
  linear-plan --cpu, execution-plans and model-execution-plans pass.
• make check on the M2 Max: check-source and check-python pass. In check-native-engine two tests fail here
  on the base too, because my disk is nearly full: disk-space and runtime-resources (it wants SSD cache room). So
  I ran the CPU tests after disk-space one by one, all pass, and all the Metal tests with validation (as make check runs them) and again without. All pass except runtime-resources. check-app hung downloading Sparkle
  again, as before. It doesn't touch this code.
• Not tested on M1. The kernel uses only what the quarters kernel uses, plus 20 KB of threadgroup memory.

[paperniuk]

Thanks, merged. Measured together with #45 above. The core count limit is fine with me here, since both kernels give the same bits.