prefix-cache-real

Measures real prefix cache costs on GPU across 3 experimental iterations: prefill savings, multi-turn speedup, hit rate simulation, and batch sharing breakeven.

Validates prefix-cache-sim (project 2).

For design and methodology see DESIGN.md.

---

Key Findings

1. Prefix cache delivers up to 2.41x speedup (v3, stable)

prefix=512 suffix=128: 2.41x  savings=8816us
prefix=512 suffix=64:  2.08x  savings=6675us
prefix=256 suffix=64:  1.26x  savings=1679us

Larger prefix fraction = higher speedup.
Speedup = cost(prefix+suffix) / cost(suffix_only).

2. Multi-turn speedup grows monotonically to 2.06x

turn=4  (256 tokens total): 1.41x speedup, 3.5ms cumulative saved
turn=8  (512 tokens total): 1.59x speedup, 13.5ms cumulative saved
turn=10 (640 tokens total): 2.06x speedup, 25.9ms cumulative saved

3. Hit rate: cache size and Zipf alpha jointly matter

32 system prompts, 200 requests, LFU eviction:

Zipf alpha=2.0  cache=4:  82% hit rate  (concentrated access)
Zipf alpha=1.0  cache=4:  39% hit rate
Zipf alpha=0.5  cache=4:  14% hit rate  (uniform access, cache too small)

With small cache (4 slots), Zipf alpha dominates.
With unlimited cache, hit rate converges to 84-91% for all alphas.

4. Batch sharing breakeven: longer prefix pays off sooner

prefix=128: breakeven at n=12 requests
prefix=256: breakeven at n=3  requests
prefix=512: breakeven at n=2  requests

For a single request, separate prefix+suffix prefill is always slower
than full prefill (CUDA kernel launch overhead for the prefix costs extra).
Prefix sharing only wins when N >= breakeven(prefix_len).

---

Experimental Versions

profile_prefix_cache.py     v1: basic measurements
profile_prefix_cache_v2.py  v2: direct + corrected hit rate + breakeven
profile_prefix_cache_v3.py  v3: stable (15 repeats, 10 warmup, median)

---

Quick Start

python3 -m venv venv
source venv/bin/activate
pip install torch transformers

python3 profile_prefix_cache_v3.py   # final version

---

Results

results/prefill_curve.csv         v1 cost curve
results/prefix_savings.csv        v1 savings
results/direct_savings_v2.csv     v2 direct measurements
results/multi_turn_v2.csv         v2 multi-turn
results/hit_rate_v2.csv           v2 hit rate
results/batch_sharing_v2.csv      v2 batch sharing
results/savings_v3.csv            v3 stable savings
results/multi_turn_v3.csv         v3 stable multi-turn
results/hit_rate_v3.csv           v3 hit rate (32 prompts + LFU)
results/breakeven_v3.csv          v3 batch sharing breakeven

---

Portfolio Context

Project 19 in a series on LLM inference infrastructure. Validates prefix-cache-sim (project 2) with real GPU measurements. Full series: https://github.com/JohnScheuer