Author: Joao Felipe De Souza Hardware: NVIDIA RTX 2070 (8.6 GB) · WSL2 · Ubuntu 22.04
Three-phase empirical study of speculative decoding using real transformer models:
- Draft: Qwen2-0.5B (494M parameters, 0.99 GB fp16)
- Target: Qwen2-1.5B (1544M parameters, 3.09 GB fp16)
- GPU: RTX 2070 (8.6 GB VRAM, 448 GB/s memory bandwidth)
The study measures token acceptance rate, real wall-clock speedup, and validates the Leviathan et al. (2023) analytical model against empirical data.
The central question:
Does speculative decoding help on a consumer GPU with a 3x parameter ratio? If not — why exactly, and what hardware conditions would make it viable?
For methodology details see DESIGN.md.
speculative-decoding-real/
|-- speculative_decoding.py # Phase 1: benchmark across 8 prompts x 4 gamma
|-- analysis_deep.py # Phase 2: root cause analysis + simulation
|-- phase3_profiling.py # Phase 3: per-phase time breakdown + alpha dynamics
|-- final_report.py # Consolidated report across all phases
|-- results/
| |-- speculative_results.csv
| |-- time_breakdown.csv
| |-- alpha_vs_position.csv
| |-- alpha_vs_temperature.csv
| |-- timing_isolation.csv
| |-- simulation_results.csv
| |-- regime_map.csv
|-- plots/
|-- DESIGN.md
|-- LICENSE
|-- requirements.txt
8 prompt types across 4 gamma values (draft tokens per step: 1, 2, 4, 8). 3 repetitions each, median reported. Metrics: autoregressive baseline, draft-only time, speculative time, token acceptance rate, real speedup, analytical model prediction.
Explains why speculative decoding failed via three quantified mechanisms. Simulates results at realistic cost ratios using real alpha measurements. Produces regime map (alpha x cost_ratio x gamma).
Per-phase time breakdown: draft passes, target pass, KV sync overhead. Alpha vs token position (does acceptance rate change as sequence grows?). Alpha vs temperature (does sampling help?). Isolated single-token timing (N=50) for precise cost_ratio measurement.
Leviathan et al. (2023) expected speedup:
E[speedup] = (1 - alpha^(gamma+1)) / ((1 - alpha) x (1 + gamma/c))
Where:
- alpha = per-token acceptance rate
- gamma = draft tokens per speculation step
- c = cost_ratio = target_time / draft_time
All 32 configurations (8 prompts x 4 gamma values) had speedup below 1.0.
Best speedup: 0.737x (code_python, gamma=8)
Mean speedup: 0.559x
Helpful: 0 of 32 configurations
Root cause 1 — Memory bandwidth equalization
Roofline prediction: cost_ratio = 3.09/0.99 = 3.12x
Measured (N=50): cost_ratio = 1.178x (38% of expected)
Draft: 23.278 ms/token
Target: 27.424 ms/token
Both models are memory-bandwidth-bound at batch_size=1. The 3x parameter difference is diluted by shared per-layer overhead (KV cache attention, layer norm, CUDA kernel launch).
Root cause 2 — KV cache synchronization overhead
gamma=1: KV sync = 22.82ms (31.8% of total step time)
gamma=2: KV sync = 24.30ms (24.3%)
gamma=4: KV sync = 24.44ms (16.1%)
gamma=8: KV sync = 21.95ms (9.9%)
The analytical model assumes KV sync = 0ms. Python implementation requires a full forward pass to re-sync (~23ms constant). Production systems (vLLM C++) use tensor slicing (~0.1ms).
Root cause 3 — Temperature incompatibility
Temperature Alpha vs greedy
greedy (T=0) 0.767 1.000x
T=0.3 0.458 0.598x
T=0.5 0.508 0.663x
T=0.7 0.242 0.315x
T=1.0 0.083 0.109x
T=1.5 0.008 0.011x
Greedy acceptance requires exact token match. Temperature=1.0 reduces alpha by 89% (0.767 to 0.083). Speculative decoding with greedy acceptance requires greedy draft generation.
code_python: first 3 steps alpha=0.000, last 3 steps alpha=1.000 (rising)
creative: first 3 steps alpha=0.167, last 3 steps alpha=0.750 (rising)
Early in generation, draft guesses wrong structures. Later, draft has seen the generated pattern and converges to target predictions. This "context alignment" effect means alpha is not constant — it grows.
gamma=1: alpha=0.350
gamma=2: alpha=0.250
gamma=4: alpha=0.588
gamma=8: alpha=0.750 (highest)
When the draft is in predictable regions, it maintains agreement for many tokens consecutively. gamma=8 captures these "flow" states. However, gamma=8 still has lowest speedup (0.737x best) because 8 sequential draft passes cost more than the target savings.
Correlation with measured: 0.246
Mean absolute error: 0.184
The model correctly predicted speedup below 1.0 for all 32 configurations. Zero false positives. It never incorrectly recommended speculative decoding.
At gamma=4 and gamma=8, measured speedup EXCEEDS predicted:
gamma Predicted Measured Ratio
1 0.827x 0.560x 0.677x (sync dominates)
2 0.657x 0.541x 0.823x (sync dominates)
4 0.515x 0.570x 1.107x (sync amortized)
8 0.285x 0.567x 1.990x (sync amortized)
At high gamma, the sync overhead is amortized over more accepted tokens, making measured speedup higher than the overhead-free analytical prediction.
Minimum cost_ratio needed for speedup above 1.0:
alpha gamma=1 gamma=2 gamma=4 gamma=8
0.5 2.00x 2.67x 4.27x 8.03x
0.6 1.67x 2.08x 3.06x 5.42x
0.7 1.43x 1.68x 2.26x 3.64x
0.8 1.25x 1.39x 1.69x 2.40x
0.9 1.11x 1.17x 1.29x 1.56x
Simulated speedup using real alpha values at projected cost ratios:
Scenario cost_ratio Helpful Mean speedup
Measured (RTX 2070, bs=1) 1.18x 0% 0.82x
Batch=8 on target 2.50x 100% 1.10x
7B target / 0.5B draft 4.00x 100% 1.24x
70B target / 0.5B draft 8.00x 100% 1.47x
CPU draft / GPU target 10.00x 100% 1.54x
factual_science: 0.606 (reference-style continuations)
conversational: 0.598 (common patterns)
code_python: 0.560 (structural regularity)
technical_systems: 0.559
code_cpp: 0.556
creative_story: 0.496 (more diverse vocabulary)
mathematical: 0.494
technical_ai: 0.446 (specialized terminology)
Scenario Recommendation
Single GPU, models same size class Do not use speculative decoding
Single GPU, 7B+ target only Try CPU draft (different bus)
Multi-GPU A100, 7B+70B pair Viable, expect 1.2-1.5x speedup
High batch size (bs >= 8) Increases cost_ratio to ~2.5x
Production serving (vLLM, TGI) Use built-in — C++ eliminates sync overhead
Any temperature > 0 Use probabilistic acceptance, not greedy
plots/master_summary.png 6-panel consolidated summary
plots/corrected_model.png Original vs corrected analytical model
plots/regime_map_final.png Continuous regime map (alpha x cost_ratio)
plots/time_breakdown.png Stacked time per phase per gamma
plots/alpha_vs_position.png Alpha dynamics over sequence
plots/alpha_vs_temperature.png Temperature vs acceptance rate
plots/timing_analysis.png Latency distributions + sync overhead
plots/phase_distribution.png Phase percentage per gamma
plots/speedup_vs_cost_ratio.png Simulated speedup by scenario
plots/regime_heatmap.png Discrete regime map table
plots/analytical_curves.png Analytical model at 3 cost ratios
plots/min_alpha_needed.png Required alpha vs cost_ratio curve
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python3 speculative_decoding.py # Phase 1: benchmark
python3 analysis_deep.py # Phase 2: root cause analysis
python3 phase3_profiling.py # Phase 3: granular profiling
python3 final_report.py # Consolidated report
- Single GPU only
- Greedy acceptance only in Phases 1 and 3
- Python KV sync adds ~23ms overhead vs ~0.1ms in production C++
- Only one draft/target pair tested
- 64 new tokens per run (short sequences)
- Qwen2-0.5B and Qwen2-1.5B share training data — alpha may differ for other pairs
- Leviathan et al., Fast Inference from Transformers via Speculative Decoding, ICML 2023
- Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling, 2023
- Xia et al., Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation, EMNLP 2023
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention, 2023
MIT -- see LICENSE.