Event-driven benchmark of speculative prefill during user reading time in multi-turn LLM conversations, measuring net TTFT benefit, contention penalty, and the conditions under which speculation becomes net-negative.
Author: Joao Felipe De Souza Year: 2026
When a language model finishes generating a response, the GPU enters an idle period while the user reads and formulates a reply. If the system predicts the next user turn during this window and executes its prefill speculatively, it can reduce TTFT to near zero for correctly predicted turns.
This benchmark asks:
- When does speculative prefill deliver positive net benefit?
- When does it become net-negative?
- Which gating policy is most robust across workloads?
- What is the tipping point between helpful and harmful speculation?
Central metric:
net_benefit_per_turn = saved_ttft - contention_penalty - wasted_ms
In chat_realtime at 20% prediction accuracy, eager speculation produces -3.6ms net benefit per turn. In interruptive_voice at 20% accuracy, -0.13ms per turn.
This is the first identified regime where speculation is reliably harmful.
The read_time_aware policy maintains spec_too_slow_frac at 0.0% across all workloads, while eager reaches 3.2% in 1.5B interruptive workloads. Read_time_aware delivers positive net benefit in every tested regime, including interruptive_voice where eager fails.
The breakeven between helpful and harmful speculation is a product of: prediction accuracy, speculative capacity fraction, and idle-to-cost ratio. No fixed accuracy threshold applies across workloads.
When speculative capacity fraction is high (0.50+), confidence-gated policies reduce speculation rate aggressively, limiting contention at the cost of some hit rate. This produces higher net benefit than read_time_aware in ultra-fast workloads with severe contention.
- voice-like: 3 to 15 ms per turn
- chat_interactive: 14 to 25 ms per turn
- chat_thoughtful: 60 to 100 ms per turn
- document_review: 135 to 220 ms per turn
Qwen2-1.5B reaches spec_too_slow_frac of 3.2% in interruptive workloads at accuracy 0.80, because higher prefill cost per token shrinks the idle-to-cost ratio. Speculation becomes structurally difficult in ultra-fast workloads for larger models.
no_spec: No speculation. Baseline.
eager_spec: Always speculate after decode. Maximum hit rate, maximum contention. Net-negative in adversarial regimes.
confidence_gated: Speculate when predictor confidence exceeds threshold. Best under high contention when predictor is well-calibrated.
read_time_aware: Speculate when idle window exceeds prefill cost. Most robust across all workloads. Never produces spec-too-slow events.
Slow readers: chat_interactive, chat_thoughtful, document_review, mixed_sessions
Fast readers: chat_fast_reader, chat_ultra_fast, chat_realtime, chat_voice_like
Adversarial: interruptive_voice (long prompts, very short idle windows)
- net_benefit_per_turn_ms
- spec_too_slow_frac
- avg_idle_vs_cost_ratio
- ttft_reduction_frac
- avg_contention_ms
- zero_ttft_frac
- roi_negative_frac_per_spec
Full benchmark:
cd ~/dev/speculative-prefill-bench
source venv/bin/activate
python -u run.py
Stress / adversarial workloads only:
python -u run_stress.py
python -u analyze_stress.py
results/summary_v12.csv
results/summary_stress.csv
plots/
speculative-prefill-bench/
|-- src/
| |-- __init__.py
| |-- config.py
| |-- workload.py
| |-- predictor.py
| |-- scheduler.py
| |-- simulator.py
| |-- bench.py
| |-- analysis.py
|-- results/
|-- plots/
|-- run.py
|-- run_stress.py
|-- analyze_stress.py
|-- SUMMARY.txt
|-- DESIGN.md
|-- LICENSE
|-- README.md
|-- requirements.txt
|-- .gitignore
Speculative prefill is highly effective when the user idle window dominates prefill cost, but becomes net-negative in interruptive fast-turn workloads under eager speculation. Read-time-aware gating is the most robust policy, maintaining positive net benefit in every tested regime including adversarial voice-like workloads. For systems with high speculative capacity contention, confidence gating is preferred. Prediction accuracy matters but the idle-to- cost ratio and capacity fraction jointly determine whether speculation pays.
MIT License. See LICENSE.