Repository navigation
perf(speculative): commit per-row accepts in batched DFlash - #2865
Merged
Merged
Conversation
inureyes
added
status:done
6 tasks done
inureyes
added a commit
that referenced
this pull request
Nov 6, 2026
Add the bilingual technical report for PR #2865, which commits per-row accepts in the batched DFlash round loop. Records the B = 4 measurement (83.06 to 116.05 tok/s, 1.397x), the accept and round counts, the parity gate, the measurement caveat, and the unvalidated surface.
The batched Qwen 3.5 DFlash round loop clamped every round to the shortest live row because the attention KV cache was rectangular. Committed accepts sat at 0.323 per row-round against a 2.938 walk, so batching bought only 1.06x over B = 1. Each row now commits its own walk count. The target compacts its attention KV back to per-row contiguous valid prefixes, rotates and masks the verify block at each row's logical positions with a per-query stale-gap mask, and replays each GDN layer's conv and recurrent state over that row's own accepted prefix. Dense Fp16 caches take the per-row path; a quantized, paged or head-trimmed cache falls back to the clamp, and MLXCEL_DFLASH_PER_ROW_ACCEPTS=0 restores it explicitly. Measured on an M3 Ultra, qwen3.5-4b-4bit target with qwen3.5-4b-dflash, block 16, 128 tokens per row, three alternating runs per arm: B = 4 aggregate 116.05 tok/s against the clamp's 83.06 (1.397x) and B = 1's 78.09; committed accepts 1.036 against 0.323 per row-round. The drafter context stays a shared rectangular prefix, so the walk itself falls from 2.938 to 1.036; per-row drafter context is a separate follow-up. Validation: the qwen35_dflash_batched parity gate and its identical-prompts control pass with per-row accepts as the default under the #2851 near-tie margin, the attention-compaction unit tests pass, and cargo fmt plus cargo clippy --lib --tests -D warnings are clean. Numbers and method: docs/benchmark_results/dflash-per-row-accepts-m3ultra-2026-11-06.md. Refs #2442
Add the bilingual technical report for PR #2865, which commits per-row accepts in the batched DFlash round loop. Records the B = 4 measurement (83.06 to 116.05 tok/s, 1.397x), the accept and round counts, the parity gate, the measurement caveat, and the unvalidated surface.
inureyes
force-pushed
the
feature/issue-2442-dflash-per-row-accepts
branch
from
November 6, 2026 01:01
d87e5f0 to
0be2d26
Compare
inureyes
marked this pull request as ready for review
November 6, 2026 01:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The batched Qwen 3.5 DFlash round loop now commits a different accept count in every row instead of clamping the round to the shortest live row (issue #2353). The attention KV caches are compacted back to per-row contiguous valid prefixes, the verify block rotates and attends at each row's logical positions with its stale gap masked, and each GDN layer replays its state over that row's accepted prefix. The clamp stays reachable with
MLXCEL_DFLASH_PER_ROW_ACCEPTS=0and is selected automatically for quantized, paged or head-trimmed caches.Measured at B = 4
M3 Ultra,
qwen3.5-4b-4bitwithqwen3.5-4b-dflash, block 16, 128 tokens per row, three alternating runs per arm.Per-row is 1.397x the clamp; B = 1 measured 78.09 tok/s. The drafter context is still a shared rectangular prefix, so the walk itself is lower than the clamp arm's. That limitation and the follow-up are recorded in
docs/benchmark_results/dflash-per-row-accepts-m3ultra-2026-11-06.md.Test plan
cargo test --profile test-fast --features metal,accelerate --test speculative_parity qwen35_dflash_batched -- --ignored --test-threads=1(2 passed with per-row accepts as the default)throughput_dflash_qwen35_batched_clamp_vs_per_row, ignored harness, three alternating runs per armcargo test --profile test-fast --lib models::qwen3_5_testsand the mlxcel-core round-loop unit testscargo clippy --lib --tests --features metal,accelerate -- -D warningsandcargo fmt --all -- --checkCloses #2442