Weeks 9-11: speculative decoding + persistent KV cache

[ai-hpc]

Goal

Speculative decoding (draft model accelerating a target model) and persistent KV cache across multi-turn conversations. These are the two biggest end-user latency wins after the basic decode loop is fast.

Tasks

Speculative decoding

• Add --draft-model <gguf> flag. Use a small model (e.g., TinyLlama Q4) to draft and the production model (Phi-4-mini Q4) to verify.
• Implement n-gram draft acceptance + reject-and-recompute path in src/engine/.
• Measure tok/s with vs without on the Week 2 corpus.

Persistent KV cache

• Disk-backed cache for the last N conversations (/opt/jllm/data/kv-cache/<conv-id>.bin).
• On a multi-turn call, hydrate the cache for the matching conversation id before decode.
• Size cap + LRU eviction.

Acceptance

• Speculative decoding delivers ≥1.5× tok/s on the Week 2 corpus with the same final-text output (no quality regression).
• Persistent KV cache reduces second-turn prompt-eval time by ≥80%.
• All Week 1 coherence + Week 3 kernel + Week 6 multi-model tests still pass.

[ai-hpc]

Path H feasibility audit — speculative decoding for Qwen3-4B-Q4_K_M on Jetson

The persistent-KV half of this issue shipped as alpha.9 + alpha.10 (Path F + F5, see #45). This comment is the feasibility audit for the speculative-decoding half — three gates that have to pass before we commit to a multi-week Path H implementation.

Why we're considering it

Path G's profile (#58) showed alpha.10 decode is mostly already-optimized Q4_K (~70 % of decode wall in Path-C kernels). Remaining Path-G levers project to +10–20 % decode at most. Path H is the only lever in sight with +50–150 % potential for serving-style chat workloads where the draft-target alignment is decent.

---

Gate 1: Tokenizer compatibility

Hard requirement. The draft must produce tokens that the target model already accepts — same vocabulary, same BPE merges, same special-token IDs. Cross-family drafting (TinyLlama for Qwen3, etc.) is dead before it starts because the token streams aren't comparable.

For Qwen3-4B target, the Qwen3 family has the natural candidates:

Model	Params	Q4_K_M size	Same tokenizer as Qwen3-4B?
Qwen3-0.6B	0.6 B	~400 MB	yes (shared Qwen tokenizer, vocab 151 936)
Qwen3-1.7B	1.7 B	~1.1 GB	yes
Qwen3-4B (target)	4 B	2.4 GB	—
Qwen3-8B+	larger than target	n/a	irrelevant as draft

Verdict: PASS. Qwen3-0.6B exists and shares the tokenizer. (Cross-check by inspecting GGUF metadata once we have one downloaded.)

---

Gate 2: Memory budget

On the 8 GB Jetson Orin Nano Super, both models + KV pools must fit alongside whatever GenieClaw runs (whisper.cpp, Piper TTS, denoise, Home Assistant container).

Item	Bytes
OS + L4T services + CMA	~1.5–1.8 GB
CUDA context	~300 MB
Target model Qwen3-4B Q4_K_M weights	2.4 GB
Target model KV pool (36 layers × 8 KV heads × 128 head_dim × FP16 × 1024 ctx)	144 MB
Draft model Qwen3-0.6B Q4_K_M weights	~400 MB
Draft model KV pool (~28 layers × 8 KV heads × 64 head_dim × FP16 × 1024 ctx)	~28 MB
Scratch (largest target intermediate × 2 for parallel verify)	~150 MB
Safety margin	256 MB
Total LLM budget	~5.3 GB
Free for GenieClaw voice stack	~2.5 GB

whisper.cpp small.en + Piper en_US-medium + denoise + HA container historically fits in ~1.8 GB → fits.

Verdict: PASS, tight. ~700 MB of headroom. Tighter if we ever move the target to Q5_K_M or context > 1024. F5's 1 GB KV cache budget would have to share that headroom or move to NVMe; flag as a follow-up.

---

Gate 3: Realistic acceptance rate

For same-family draft/target (Qwen3-0.6B → Qwen3-4B) on chat workloads, literature estimates and llama.cpp's tracked benchmarks land around 55–70 % per-token acceptance, depending on prompt entropy. Math-heavy / code-heavy prompts trend lower; conversational text trends higher.

With draft depth k = 4 and acceptance α:

E[accepted tokens per verify] = (1 - α^(k+1)) / (1 - α)
α=0.55 → 1.62
α=0.65 → 2.10
α=0.70 → 2.31

Per verify step the target does one batched forward pass over k+1 = 5 tokens. Empirically on Qwen3-4B on alpha.10's MMQ prefill at 38 tok/s, that 5-token batch costs ~150 ms — roughly 1.4× one decode step (110 ms) rather than 5×, because MMQ amortizes weight reads across the batch.

End-to-end realistic decode speedup:

real_throughput = E[accepted] / (verify_cost_ratio)
α=0.55 → 1.62 / 1.4 = 1.16×
α=0.65 → 2.10 / 1.4 = 1.50×
α=0.70 → 2.31 / 1.4 = 1.65×

So alpha.10's 9.1 tok/s would land at 10.5 – 15 tok/s depending on acceptance. Worth doing if α turns out ≥ 0.6; marginal if < 0.55.

The actual acceptance rate has to be measured on a representative corpus — it's the dominant variable. Literature is a guide, not a guarantee.

Verdict: CONDITIONAL PASS on the literature estimate; must measure α with a minimal end-to-end implementation before declaring victory.

---

Recommended phasing if we proceed

Mirroring Path E / Path F:

Phase	Scope
H1	Acquire Qwen3-0.6B Q4_K_M, confirm tokenizer GGUF-metadata equals Qwen3-4B's (md5 of vocab section). Half-day.
H2	Load both models simultaneously in a standalone test. Verify memory budget on Jetson matches the audit. No inference yet.
H3	Draft-only loop: generate k tokens from Qwen3-0.6B given a prompt. No verification yet — sanity-check the draft model gives reasonable output.
H4	Verify-only loop: prefill the k drafted tokens through Qwen3-4B as a batched prefill, sample at each position, compute acceptance. First real α measurement on a chat corpus. Pivot point: if α < 0.55, close as negative result.
H5	Integrate the accept-or-reject decode loop end-to-end behind a flag (JLLM_SPEC_DECODE=1).
H6	Default flip + alpha.11 release if α + perf both hit.

Risks worth naming up front

1. Acceptance below 0.55 — likely close-as-negative if it does. The literature is encouraging but not guaranteed for our specific workload.
2. Memory pressure under load. Tight at idle (2.5 GB headroom); a GenieClaw concurrent voice burst could tip it. Adds an OOM-guard surface to the engine.
3. Two models = two model fingerprints. Path F's KV cache files become draft-or-target specific. Need to extend the header or namespace cache files by (target_hash, draft_hash). Small addition; design before H3.
4. Increased latency variance. When the draft is wrong, decode is slower (wasted verify work). Average throughput up; p99 latency may worsen.

Verdict for the whole audit

Gate	Status
Tokenizer compatibility	PASS (Qwen3-0.6B is the draft)
Memory budget	PASS (tight: ~700 MB headroom on Jetson 8 GB)
Acceptance ≥ 0.55	CONDITIONAL — needs H4 measurement

Recommendation: proceed to H1 + H2 (download model, verify memory). If both pass, commit to H3–H4 to actually measure α. If α ≥ 0.55, then H5 / H6 ship as alpha.11. If α < 0.55, close Path H without integration and report the negative result.

Open question for you: greenlight Path H now, or want to think about it overnight?