Repository navigation
Window-cost tables are measured once per binary: one start's timing noise decides every later start's 4-stream speed (Qwen3.6-35B-A3B + MTP: -6 to -16 %) #1173
Description
Summary
The qwen3_5 CUDA engine times its windows at a binary's first start. It caches that table under a key that includes the executable's SHA-256 (lanes/cost_cache.zig, ~/.cache/tensorfold/window-costs/). Every later start of that binary plans its rounds from that one measurement.
The round planner turns small price differences into whole drafts (#1059's mechanism). So a single noisy first start makes a build look slower on every run that follows, with identical replies. In A/Bs between builds this compares two cost tables rather than two codes.
Related: #1059 (shared rounds keep depths at their priors, so a table change moves 4-stream depth by one draft). That issue is about how the planner reacts to a table. This one is about where the table comes from: a single first-start measurement, persisted per binary.
Evidence
Setup:
- GB10, Qwen3.6-35B-A3B MLX-4bit with its MTP layer as drafter (
--drafter <same dir>),--parallel 4 --context 32768, greedy. - Builds: main d1018b3, and a branch whose decode path is identical to main's here. Pages were off; the branch only adds code for them.
- Replies were byte-identical across every start below (3/3 chat, 3/3 8k prefill).
bench_concurrent.py"chat" prompt at 4 streams, 256 tokens. Each line is one server start.
| Build / cost table | c=4 chat tok/s by start |
|---|---|
| main, its cached table | 223.7, 224.9, 224.1, 223.8 |
| branch B, its cached table (measured at its first start) | 211.5, 208.2, 187.4, 211.4 |
branch B, a fresh table each start (HOME pointed at an empty cache) |
225.4, 222.3 |
| branch A (same decode path), its own first measurement | 229.5 |
| branch A, a fresh table each start | 224.6, 190.5 |
Not all of the spread is the cached table: the drafter's own step time (MTP: the quickest of 16 chains) is timed at every start, which is why one binary's starts on one table still differ (187.4 vs 211.4). But branch B's table kept all four of its starts below every one of main's, and fresh tables brought it back to main's level.
Per request, the planner's choices differ while the tokens do not. On main, every one of the 4 concurrent chat streams takes 111 rounds with 144/244 drafts accepted, on every start. On branch B's cached table, the same streams take 130-136 rounds (121-125 accepted of 211-227 offered) on one start and 105-107 rounds on another.
One-stream decode moves less: about 1 % (main 106.7-107.3 code tok/s, branch B 105.3-106.0).
Why it matters
- Users: whatever a binary's first start measured, that is how it runs. A box that was busy during that start (a download, another model warming up) keeps paying for it.
- Benchmarks and A/Bs: a build-vs-build comparison at c>1 with drafts can differ by 6-16 % with no code difference. The usual guard of alternating starts does not help, because each binary reuses its own table on every start.
- The fresh-table runs above show the spread one measurement can have: 190.5 vs 224.6 for the same binary.
Possible directions (not a fix proposal yet)
- Take each width's median of several timed passes, possibly over more than one start. Today the table is whatever the first start measured.
- Re-measure (or blend) when live round timings disagree with the table by more than a margin. The lane host already sees every round's wall time.
- Log the table's identity and age in the startup line, and offer a switch to re-measure. Benchmark guidance could then say "fresh table per start" or "same table for both arms".
- For A/Bs, key the table by the kernels' images rather than the whole executable, so builds that do not change kernels share one table.
Reproduce
# same binary, two cost-table regimes; replies identical, c=4 chat tok/s differs
HOME=$(mktemp -d) tensorfold-native serve <Qwen3.6-35B-A3B-MLX-4bit-MTP> --drafter <same> --parallel 4 --context 32768
# ... bench_concurrent.py --levels 4 --tokens 256, then restart with the default HOME (cached table)
Activity
akol1 commentedon Oct 21, 2026
Checking this against the DeepSeek V4.1 Metal line we have open (#826, #827): it does not inherit the per-binary cached table, because it never reads or writes the window-cost cache at all.
Three things I checked on our rebased branches:
zig/src/core/lanes/cost_cache.zigis byte-identical tomainon every branch we have open. The executable-SHA-256 keying you describe is upstream's design, so nothing we carry changes it either way.cost_cache.keyare the CUDA adapters andzig/src/native/qwen27_lanes.zig. Our DeepSeek V4.1 lane host reaches lanes through the shared planner and takes no cost table from the cache.qwen27_lanes.zig:188already re-times a start whose fresh measurement drifts from the cached reference viacost_rule.drifted, so at least the one path that does cache has a re-measure step. That does not address your report: the guard compares against the cached table and only fires on drift, so a first start that lands inside the drift band still fixes every later start's table.On the report itself I think it is right, and the mechanism is the one you name:
key()hashes the executable's bytes so a rebuild invalidates the table, but a rebuild is exactly the thing that would produce a new first-start measurement. Two builds with identical decode paths therefore compare two tables rather than two codes, which is what your A/B table shows (branch B at 187.4 to 211.5 against main's 223.7 to 224.9 on a cached table, and both branches recovering whenHOMEpoints at an empty cache).If it helps, the fix that would make the A/B meaningful is separating "same code" from "same binary": key on the decode-relevant parts rather than the whole executable, or re-time on a schedule instead of once, so a single noisy start cannot fix every later one. The reference table
referenceKeyalready keys without the executable, so the pieces are close.ashhart commentedon Oct 24, 2026
Fixed on main by 14ae735, thank you @jschmied. Each width's price in the Qwen3.5 CUDA window-cost table is now the fastest of seven timed runs, so one slow run can't set it. A stored table is also spot-checked at start and measured again when it no longer holds. A host test injects an outlier and checks that the stored table keeps the fast price. Nemotron already measured this way; GLM and qwen27 re-measure on drift.
- Synthema, Ash's AI Assistant.