Skip to content

fix: time the device's attention tile in attention-sweep - #23

Merged
paperniuk merged 1 commit into
paperniuk:apple7-m1-1.4from
sady4850:apple7-attention-sweep-tile
Oct 22, 2026
Merged

paperniuk merged 1 commit into
paperniuk:apple7-m1-1.4from
sady4850:apple7-attention-sweep-tile

Conversation

@sady4850

Copy link
Copy Markdown

attention-sweep builds its graph with PagedAttention::prefillPlan / verifyPlan called without the tile argument, so it always takes the default AttentionTile::Mpp. On Apple7/8 the engine runs the register tile (ExecutionPlans picks AttentionTile::Register, the *_sgf pipelines), so on M1/M2 the sweep was timing the stock MPP kernels, not the ones that serve. Found it while looking at long-context verify on M1 Pro: decode-profile shows verify_attention_q8_split_sgf, the sweep printed verify_attention_q8_split.

The fix builds the case graph from ops::ExecutionPlans(backend.capabilities()), same as the runtime does, one per backend (so --compare-metallib also uses it). Apple9+ unchanged, its plans give Mpp as before. apple7-m1-1.3.2 has the same bug.

M1 Pro (16-core GPU), qwen3.8-27b shape, Page32 int8 KV, verify, 1 lane, median of 15 fused graphs:

history before (times MPP verify_attention_q8_split) after (times verify_attention_q8_split_sgf)
2048 0.449 ms 0.280 ms
8192 1.419 ms 0.803 ms
24576 3.843 ms 2.145 ms
49152 7.667 ms 4.280 ms

Prefill phase picks prefill_attention_q8_split_sgf / _kv2_g8 now too. dev/tests/attention_sweep_cli.sh passes, build clean with -Werror. Output digests change on Apple7/8, as expected (other kernel, other summation order).

Why I care: at agent contexts verify attention becomes a real part of the M1 decode cycle (decode-profile, 27B, B1: 3% of the cycle at 2K, 22% at 24K, ~50 ms of 226 ms), so I want to try some things in the register tile and need the sweep to A/B it.

@paperniuk
paperniuk merged commit bc94a25 into paperniuk:apple7-m1-1.4 Oct 22, 2026
@paperniuk

Copy link
Copy Markdown
Owner

Thanks, good catch! Confirmed on M1 Max: the sweep now times the _sgf kernels in both phases. 27B verify at 1 lane goes from 0.239 / 0.742 / 1.919 / 3.758 ms (MPP) to 0.162 / 0.451 / 1.100 / 2.147 ms at 2K / 8K / 24K / 48K history. CLI test passes, build is clean. Merging.

paperniuk pushed a commit that referenced this pull request Oct 22, 2026
attention-sweep built its graph with PagedAttention::prefillPlan and verifyPlan without the tile argument, so on Apple7/8 it timed the MPP kernels instead of the register tile the engine runs. Build the case graph from ops::ExecutionPlans, as the runtime does.

Ported from 173a457 (#23) on apple7-m1-1.4.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants