Train and evaluate DSpark speculative decoding draft models for Llama-3.2-3B-Instruct (rasyosef/Llama-3.2-3B-Instruct-DSpark) and Llama-3.2-1B-Instruct (rasyosef/Llama-3.2-1B-Instruct-DSpark), using the speculators library and vLLM.
- Drafters — 3B: 5 layers, ~0.7B params; 1B: 3 layers, ~0.3B params. Both propose 8 tokens per cycle.
- Verifiers — the full
Llama-3.2-3B-Instruct/Llama-3.2-1B-Instruct, checking each block in one forward pass, so output is identical to running the verifier alone. - Data — 100,000 Open PerfectBlend prompts, regenerated on-policy by each verifier itself.
- Result — 3B: 3.412 tokens accepted per verification round, 5.529 on math_reasoning, for a 2.32× average single-stream throughput gain (up to 3.62× on HumanEval). 1B: 3.148, 4.573 on HumanEval.
- Scope — regenerate the data, train online against a live vLLM verifier, then serve and benchmark: one notebook for 3B, two for 1B.
.
├── notebooks/
│ ├── [3B][A100] Train-Llama-3.2-3B-DSpark-Online.ipynb
│ ├── llama-1b-dspark/
│ │ ├── data-preparation.ipynb
│ │ ├── [A100] train-dspark-llama-3.2-1B-instruct-online.ipynb
│ │ └── [2xT4 kaggle] train-dspark-llama-3.2-1B-instruct-online.ipynb
│ ├── llama-1b-magpie-kaggle/
│ └── qwen-3-0.6b/
├── LICENSE
└── README.md
Everything runs inside the notebooks. Nothing to install; speculators is cloned at run time.
The 3B notebook is self-contained: it regenerates the data, prepares it, trains online against the verifier, pushes the drafter, then serves and evaluates it, all on a single A100.
The 1B notebooks live in notebooks/llama-1b-dspark/. Both 1B training notebooks use the same Open PerfectBlend data and differ in hardware and configuration: the A100 notebook produced the published drafter, while the 2× T4 Kaggle notebook is a smaller free-tier variant (2 layers, block 4, 80,000 samples, --total-seq-len 4096).
notebooks/llama-1b-magpie-kaggle/ holds the earlier Magpie-trained drafter, and notebooks/qwen-3-0.6b/ an earlier drafter for Qwen/Qwen3-0.6B in both offline and online modes — see its own README.
Speculative decoding: the drafter proposes a block of tokens and the target (verifier) model checks them in a single pass.
Speculative decoding speeds up inference by letting a small drafter propose several tokens at once, then having the full model verify the whole block in a single forward pass and keep the longest prefix it would have produced itself. Output is identical to running the verifier alone — the win is fewer verifier forward passes per token, not a different distribution. The metric that matters is acceptance length, the mean number of tokens kept per verification round.
DSpark builds on DFlash: instead of predicting the block autoregressively the way EAGLE-3 does, it predicts the entire block in one forward pass using anchored block diffusion, conditioned on hidden states read from selected verifier layers (--target-layer-ids 2 8 14 20 26 for 3B, 2 8 14 for 1B). Pure block-parallel drafting leaves no dependency between tokens inside a block, so acceptance decays toward the block's end; DSpark restores that dependency with a Markov head — a low-rank logit bias conditioned on the previous token — and adds a confidence head that estimates per-position acceptance probability. The pos_0…pos_7 decay in the Evaluation tables below is exactly this within-block effect.
- Drafter —
rasyosef/Llama-3.2-3B-Instruct-DSpark. Proposes 8 tokens per cycle for the verifier to check. - Verifier —
unsloth/Llama-3.2-3B-Instruct. Validates each proposed block in a single forward pass, so output is identical to running the verifier alone.
Architecture: 5 Qwen3 layers (hidden size 3072, intermediate size 8192, 24 attention heads over 8 KV heads, sliding-window attention with a 2048-token window), ~0.7B params, bfloat16. Block size 8, draft vocabulary reduced to 50,000 tokens, aux hidden-state layers 2/8/14/20/26, confidence head with Markov (rank 256).
- Drafter —
rasyosef/Llama-3.2-1B-Instruct-DSpark. Proposes 8 tokens per cycle for the verifier to check. - Verifier —
unsloth/Llama-3.2-1B-Instruct. Validates each proposed block in a single forward pass, so output is identical to running the verifier alone.
Architecture: 3 Qwen3 layers (hidden size 2048, intermediate size 8192, 32 attention heads over 8 KV heads, sliding-window attention with a 2048-token window), ~0.3B params, bfloat16. Block size 8, draft vocabulary reduced to 32,000 tokens, aux hidden-state layers 2/8/14, confidence head with Markov (rank 256).
Both drafters share the same recipe, each trained on its own verifier's regenerations: 100,000 Open PerfectBlend prompts, 3 epochs, 96/4 train/validation split, AdamW at 3e-4 with a cosine schedule and 4% warmup, loss weights {"ce": 0.1, "tv": 0.9}. Prompts prepared at 1024 tokens and packed into an 8192-token training window, up to 1024 anchors per sample. Each was trained online on a single A100, with the verifier sharing the same card at --gpu-memory-utilization 0.25.
# 3B
vllm serve rasyosef/Llama-3.2-3B-Instruct-DSpark \
--port 8000 \
--gpu-memory-utilization 0.8# 1B
vllm serve rasyosef/Llama-3.2-1B-Instruct-DSpark \
--port 8000 \
--gpu-memory-utilization 0.8Query the OpenAI-compatible endpoint at http://localhost:8000/v1. vLLM reads the verifier and the speculators_config block out of the drafter's own config, so neither the verifier nor --speculative-config is passed separately.
A drafter is not a standalone model — it only works paired with its verifier.
| Notebook | What it does |
|---|---|
[3B][A100] Train-Llama-3.2-3B-DSpark-Online.ipynb |
End to end for 3B: regenerates Open PerfectBlend with Llama-3.2-3B-Instruct (pushed to yosefw/open-perfectblend-llama-3.2-3b-instruct), trains online, pushes checkpoint_best, then serves and evaluates it |
llama-1b-dspark/data-preparation.ipynb |
Regenerates Open PerfectBlend prompts with the verifier and pushes the JSONL to yosefw/magpie-llama-3.2-1b-instruct |
llama-1b-dspark/[A100] train-dspark-llama-3.2-1B-instruct-online.ipynb |
Filters and tokenizes that data, trains the drafter online against a live vLLM verifier, pushes checkpoint_best, then serves and evaluates it |
llama-1b-dspark/[2xT4 kaggle] train-dspark-llama-3.2-1B-instruct-online.ipynb |
Free-tier variant of the A100 notebook on Kaggle's 2× T4 |
Online training keeps the verifier resident and fetches hidden states per batch instead of caching them to disk — near-zero disk, but the verifier has to stay up for the whole run. In the A100 notebooks (3B and 1B) both jobs share one card: vLLM takes 25% for the verifier and its KV cache, training takes the rest. The 2× T4 notebook instead gives the verifier GPU 0 and trains on GPU 1.
The data step is a one-off; its output is cached on the Hub, so retraining never re-runs generation.
Prerequisites: vllm>=0.22.0, a Hugging Face write token, and an A100 (for 1B, two smaller GPUs also work).
The A100 notebooks read the token via google.colab.userdata, the Kaggle ones via kaggle_secrets. Elsewhere those imports fail; replace those cells with a direct os.environ["HF_TOKEN"] = ....
Three things to keep in sync:
- Sample count —
--limiton response regeneration and--max-samplesonprepare_data.py,100000in both A100 runs. - Prompt length — the prompt filter (≤768 tokens) and
--seq-length 1024onprepare_data.py, which leaves at least 256 response tokens per retained sample. This is deliberately not--total-seq-len 8192: that is the packing window several conversations share, not a per-conversation cap. --target-layer-ids—2 8 14 20 26for 3B,2 8 14for 1B. It must match in the vLLM launch and training cells, or the drafter trains against hidden states the server isn't exporting; nothing cross-checks it.
Budget for a long session: each A100 run takes several hours on a single card, 3B longer than 1B, and the 1B 2× T4 variant roughly 10–11 hours on Kaggle's free accelerators.
Measured with evaluate.py throughput in the training notebooks, against the drafter served in vLLM, across the nine RedHatAI/speculator_benchmarks subsets.
acceptance_length is the mean tokens committed per verification round, including the bonus token — floor 1.0, ceiling 9.0 at block size 8. pos_N is the percentage of blocks whose slot N survives verification, decaying across the block as intended. The two use different denominators, so pos_N does not sum to acceptance_length. The weighted acceptance length is an upper bound on single-stream speedup, since it does not charge for the drafter's own forward pass; measured throughput for 3B is reported below.
| subset | acceptance_length | pos_0 | pos_1 | pos_2 | pos_3 | pos_4 | pos_5 | pos_6 | pos_7 |
|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 5.529 | 89.6% | 78.4% | 67.9% | 58.8% | 50.5% | 43.3% | 36.0% | 28.2% |
| HumanEval | 5.058 | 85.6% | 72.6% | 61.6% | 52.3% | 43.7% | 36.3% | 29.9% | 23.8% |
| tool_call | 3.492 | 73.6% | 54.2% | 39.5% | 28.6% | 20.8% | 14.8% | 10.5% | 7.2% |
| question | 2.952 | 67.7% | 43.8% | 28.8% | 19.7% | 13.8% | 10.0% | 6.8% | 4.6% |
| rag | 2.917 | 67.6% | 45.8% | 30.8% | 20.4% | 12.6% | 7.6% | 4.4% | 2.4% |
| writing | 2.908 | 67.5% | 43.3% | 28.0% | 18.7% | 13.0% | 9.3% | 6.4% | 4.5% |
| translation | 2.653 | 67.0% | 42.7% | 25.2% | 14.2% | 7.9% | 4.4% | 2.6% | 1.3% |
| summarization | 2.597 | 65.6% | 40.5% | 23.9% | 13.9% | 8.0% | 4.3% | 2.2% | 1.2% |
| qa | 2.241 | 55.0% | 31.6% | 17.9% | 9.4% | 5.0% | 2.7% | 1.6% | 0.8% |
Weighted over 69,061 verification steps, acceptance length is 3.412.
Measured wall-clock decode throughput in tokens/s on a single A100 at max concurrency 1, against the verifier served alone in vLLM as the baseline:
| subset | baseline | DSpark | speedup |
|---|---|---|---|
| math_reasoning | 150.8 | 524.9 | 3.48× |
| HumanEval | 150.6 | 545.2 | 3.62× |
| tool_call | 147.8 | 337.7 | 2.28× |
| question | 149.9 | 335.9 | 2.24× |
| rag | 142.6 | 272.6 | 1.91× |
| writing | 150.8 | 273.5 | 1.81× |
| translation | 150.0 | 289.3 | 1.93× |
| summarization | 147.7 | 279.5 | 1.89× |
| qa | 150.3 | 259.1 | 1.72× |
Averaged across the nine subsets (unweighted), DSpark runs 2.32× faster than the verifier alone (up to 3.62× on HumanEval).
| subset | acceptance_length | pos_0 | pos_1 | pos_2 | pos_3 | pos_4 | pos_5 | pos_6 | pos_7 |
|---|---|---|---|---|---|---|---|---|---|
| HumanEval | 4.573 | 83.7% | 68.5% | 56.2% | 46.0% | 36.8% | 28.3% | 21.7% | 16.1% |
| math_reasoning | 4.436 | 83.6% | 68.2% | 55.0% | 43.8% | 34.5% | 25.8% | 19.2% | 13.4% |
| tool_call | 3.462 | 73.3% | 55.5% | 41.3% | 30.1% | 20.2% | 13.3% | 8.1% | 4.3% |
| question | 2.646 | 63.0% | 38.9% | 24.3% | 15.8% | 9.7% | 6.4% | 4.0% | 2.5% |
| writing | 2.636 | 61.7% | 38.8% | 24.4% | 15.8% | 9.9% | 6.2% | 4.1% | 2.6% |
| rag | 2.523 | 64.9% | 39.4% | 23.9% | 13.3% | 6.4% | 2.8% | 1.1% | 0.5% |
| qa | 2.258 | 55.4% | 32.1% | 18.2% | 9.6% | 5.2% | 2.8% | 1.6% | 0.9% |
| summarization | 2.182 | 58.5% | 31.0% | 16.3% | 7.6% | 3.2% | 1.1% | 0.4% | 0.1% |
| translation | 2.021 | 54.6% | 28.6% | 12.6% | 4.3% | 1.3% | 0.5% | 0.2% | 0.0% |
Weighted over 89,102 verification steps, acceptance length is 3.148.
Acceptance is highest where the verifier's next token is most predictable — code, math, structured tool calls. HumanEval and math_reasoning are far ahead of everything else and hold their lead deep into the block: HumanEval's pos_4 (36.8%) is above summarization's pos_1 (31.0%), and both still accept better than one token in six at pos_7. The prose-like subsets cluster tightly at 2.0–2.6 and fall off sharply after pos_3, where the longer block buys little — translation is under 2% accepted from pos_4 onward. A block of 8 pays for itself on code, math, and tool calls, and mostly idles on prose traffic.
Built on
speculators— the DSpark trainer, converter, andevaluate.pyused throughout, plus the official training and performance evaluation tutorials these notebooks follow.- vLLM — serves the verifier during online training and runs the speculative-decoding benchmark.
- Open PerfectBlend — the seed prompts, regenerated with each verifier (in the 3B notebook and
llama-1b-dspark/data-preparation.ipynb). The earlier runs used Magpie-Align prompts instead. unsloth/Llama-3.2-3B-Instructandunsloth/Llama-3.2-1B-Instruct— the verifiers, mirrors of Meta'sLlama-3.2-3B-InstructandLlama-3.2-1B-Instruct.
Both published drafters were trained on a single Colab A100; the earlier drafters on Kaggle's free 2× T4 accelerators.
License
- This repository — Apache-2.0, see
LICENSE, matchingspeculators. - The published drafter weights — subject to the Llama 3.2 Community License, inherited from the verifiers they are trained against and can only run with. Meta's Acceptable Use Policy applies, as does the "Built with Llama" attribution requirement.
- The regenerated datasets — Llama 3.2 outputs, so the same Llama 3.2 terms apply on top of the source dataset's own license.