7,309 blocks. Four drafters verified on identical prefixes. Two GPUs. One hypothesis killed, one replacement measured, one generalization boundary found.
Speculative-decoding engines increasingly hold several drafters of different architectures. Which one should run on the next block? Every published selection method (BanditSpec, ICML'25; Not-a-Bandit, '25; MetaSD; OnlineSPEC) is non-contextual: their guarantee is convergence to the best fixed drafter. That guarantee is also their ceiling. This repo asks whether context breaks it, and reports what the hardware actually said.
Every number below is labeled [MEASURED] or [SIM], and every headline number carries a 95% bootstrap confidence interval. No exceptions.
The obvious version of the idea: pick the drafter based on what kind of request came in. Full 4x4 sweep, mean accepted tokens per block:
| workload | dflash | eagle | cascade | ngram |
|---|---|---|---|---|
| gsm8k | 3.837 | 2.371 | 3.803 | 1.374 |
| code | 4.265 | 2.575 | 4.237 | 1.435 |
| chat | 2.869 | 1.910 | 2.863 | 1.080 |
| copy | 6.347 | 5.356 | 6.594 | 4.647 |
One arm (dflash) dominates nearly everywhere; cascade edges it only on copy, by 3.9%. Counterfactual replay over the measured logs: hindsight oracle 4.325 vs best-fixed 4.298. Total workload-level selection headroom: 0.6%. Every policy, ours and the prior art's alike, lands inside noise of just picking the best fixed drafter.
For contrast, the per-block oracle gap on the same logs is 13.3% with a 95% bootstrap CI of [12.3, 14.2], comfortably clear of zero. The workload-level headroom is an order of magnitude smaller than an effect this data can resolve, which is what makes the null informative rather than merely underpowered.
The calibrated simulation had predicted +19.4% for contextual Thompson sampling.
It assumed cross-workload ranking flips that do not exist in this pool on this
hardware. The simulation is kept in bandit/core.py and the prediction is kept
in this README, because the gap between the two is the finding.
Rankings are stable between workloads but not within them. Logging all four drafters against the identical prefix at every block exposes a per-block hindsight oracle 13.3% above always-trusting the dominant drafter (95% bootstrap CI [12.3, 14.2], n=6,361 blocks).
A 4-feature logistic selector over draft-confidence statistics
(conf_mean, conf_min, conf_first, ent_mean) captures 61% of that
oracle gap while paying 43% of the redraft cost (capture CI [0.55, 0.68];
the interval's lower bound sits above the cost, so the selector is doing real
work rather than buying capture with redraws). The signal is entirely in the
drafter's own confidence: no extra forward pass, no auxiliary model.
Train on three workloads, test on the held-out fourth. The decision threshold is a hyperparameter, so it is chosen by an inner leave-one-workload-out loop over the three training workloads only and then applied unchanged to the held-out one. Picking it per held-out workload would be selecting on the test set.
Skill = capture / redraft-rate; 1.0 means no better than redrafting at random.
| held-out | threshold (inner CV) | skill | 95% CI | verdict |
|---|---|---|---|---|
| code | 0.30 | 2.06x | [1.54, 2.62] | beats random |
| gsm8k | 0.30 | 1.91x | [1.57, 2.28] | beats random |
| chat | 0.35 | 1.57x | [1.43, 1.70] | beats random |
| copy | 0.30 | 0.65x | [0.31, 1.00] | no measurable benefit |
It transfers across reasoning, code and chat, and carries no signal on copy: the interval includes 1.0, so on that workload the selector is not distinguishable from redrafting at random. Copy blocks have uniformly high draft confidence, so the feature that carries the signal elsewhere is nearly constant there and the decision collapses.
Note this is a weaker and more accurate claim than an earlier draft of this
README made. That draft reported 0.48x on copy and called it "worse than
random", quoting a per-workload best threshold. Both were wrong: the threshold
was being selected on the held-out set, and once the interval is computed the
honest statement is no measurable benefit, not harmful. The bootstrap is in
bandit/ci.py and its output in runs/ci.json.
Everything above is measured in accepted tokens per block, which is a proxy.
Re-drafting costs an extra drafter pass, so the honest question is whether the
selector is net faster. bandit/breakeven.py converts the measured operating
point into a break-even condition on the only two ratios that matter:
rho = drafter pass cost / target verify cost, and alpha = extra target cost
charged by a re-draft.
The 61% capture buys an 8.1% token gain. That is the entire budget the extra cost has to fit inside, and it is not large:
- With a separate second verification (
alpha>= 0.25), the selector never wins, at any drafter cost. A 43% re-draft rate cannot be paid out of an 8% token gain. Implemented that way it is a measurable slowdown. - With union/tree verification (
alpha= 0), where the second draft is checked inside the same target forward pass, it wins while the drafter costs under 23% of a verify pass.
And the sharper question, does selection beat simply always re-drafting:
| rho | best-of-N | best selector | selector margin |
|---|---|---|---|
| 0.02 | 1.111 | 1.113 | +0.2% |
| 0.05 | 1.081 | 1.086 | +0.5% |
| 0.10 | 1.038 | 1.047 | +0.8% |
| 0.15 | 1.002 | 1.024 | +2.2% |
| 0.20 | 0.971 | 1.011 | +4.1% |
The margin over unconditional best-of-N is small, and vanishes as the drafter
gets cheap, because what selection saves is skipped draft passes and those are
exactly what is cheap in that regime. Its real contribution is at the other end:
past rho ~ 0.15 best-of-N stops paying for itself and the selector still does.
Selection does not make multi-drafter decoding much faster. It extends the
drafter-cost regime in which multi-drafter decoding is worth doing at all.
rho has not been measured on this hardware, so this repo does not claim a
wall-clock speedup. It states the condition under which one exists. Full
output in runs/breakeven.txt.
Selection under a cost budget with per-arm reward and a context vector is a bandit problem, and the failure mode found here is the one bandit theory predicts: when arms do not reorder across contexts, contextual routing has nothing to exploit, and the entire question moves to a finer granularity where they do. The interesting artifact is not the policy. It is the counterfactual dataset that makes the question answerable at all.
bandit/core.py: policies: contextual Thompson, contextual Hedge, and the prior-art baselines (UCB, full-information Hedge)bandit/selector.py: the per-block logistic selector, LOWO evaluation, threshold frontierbandit/breakeven.py: converts the token-proxy result into a wall-clock break-even conditionbandit/ci.py: bootstrap confidence intervals for every headline number, plus the nested threshold selectionbandit/replay.py: counterfactual replay over the measured per-block logssweep.py,workloads.py: the measurement campaign (4 workloads x 4 drafter modes)runs/: every raw result, including the failures.blocks_*.jsonlare the per-block counterfactual logs; the results above reproduce from these alone, without the engine or the GPUs.NEGATIVE_RESULTS.md: what did not work, with root causespaper/: draft
The headline numbers replay from the committed logs:
python3 bandit/ci.py # every headline number with a 95% bootstrap CI
python3 bandit/breakeven.py # when selective re-drafting actually pays for itself
python3 bandit/selector.py # per-block selector, LOWO splits, threshold frontier
python3 bandit/replay.py # workload-level counterfactual replay
python3 bandit/core.py # the calibrated simulation (the falsified prediction)
Regenerating the logs from scratch needs the Mach speculative-decoding harness
plus Llama-3.1-8B with a DFlash block-diffusion drafter and an EAGLE-3 head;
sweep.py reads MACH_ROOT and MACH_MODELS for their locations.
NVIDIA DGX Spark (GB10, sm_121) for the workload sweep; RTX 5090 (sm_120) for the 6,361-block counterfactual collection.