Measures real KV cache costs on GPU across 3 experimental iterations: memory per token, allocation time, decode extension, fragmentation, compaction bandwidth, concurrent pressure, and interference.
Validates kv-cache-compaction-lab (project 1) and calibrates models used in kv-cache-disaggregation-sim (project 9).
For full design and methodology see DESIGN.md.
KV bytes = 2 * n_layers * n_heads * d_head * dtype_bytes
GPT-2: 2 * 12 * 12 * 64 * 2 = 36 KB/token
Measured: 36.000 KB/token for ALL sequence lengths (match=YES)
Step cost variation across prefix 64-512: 3.0%
KV growth per step: exactly 36 KB (formula match)
Confirms: decode bottleneck is DRAM bandwidth, not attention compute
Small workloads (<5MB): 19.2 GB/s (kernel-launch-bound)
Large workloads (>10MB): 52.5 GB/s (2.7x better)
Best case (4x512 tok): 66.9 GB/s (14.9% of RTX 2070 theoretical)
After evicting 25MB of KV caches, allocator still reserves 131.7MB.
The CUDA caching allocator pools memory for reuse.
Model weights: 237 MB. Available for KV: 7.5 GB.
seq=512: 49+ requests (882MB KV, 3.7x model size) without OOM.
Theoretical max: 414 req x 512 tok, 828 req x 256 tok.
Compact alone: 1289 us (51.2 GB/s)
Compact + decode: 903 us (73.2 GB/s)
The CUDA scheduler overlaps memory copy and compute streams,
achieving higher GPU utilization when both run simultaneously.
GPT-2 (237MB): KV > model at 16 requests x 512 tokens
Crossover token count: 6,751 tokens for one request
For 7B models (~14GB weights): KV crossover at ~388 tokens x batch_size.
This is why KV cache management dominates production LLM memory.
profile_kv_cache.py v1: basic measurements (gpu_mb before/after)
profile_kv_cache_v2.py v2: direct tensor measurement + per-layer
profile_kv_cache_v3.py v3: pressure, interference, crossover
python3 -m venv venv
source venv/bin/activate
pip install torch transformers
python3 profile_kv_cache_v2.py # most accurate
python3 profile_kv_cache_v3.py # deep dives
results/kv_memory.csv v1 memory measurements
results/kv_memory_v2.csv v2 direct tensor measurement
results/kv_per_layer.csv per-layer breakdown
results/kv_decode_cost_v2.csv decode step costs
results/kv_frag_v2.csv realistic fragmentation
results/kv_compact_v2.csv compaction bandwidth scaling
results/kv_pressure_v3.csv concurrent request pressure
results/kv_interference_v3.csv compaction interference
results/kv_crossover_v3.csv KV vs model weight crossover
Project 18 in a series on LLM inference infrastructure. Full series: https://github.com/JohnScheuer