Measures actual GPU costs for LLM prefill and decode to validate and calibrate simulation parameters.
Uses subprocess isolation to prevent CUDA context corruption. Each measurement runs in a fresh Python process.
For full design, measurement protocol, and calibration discussion see DESIGN.md.
Project 15 in a series on LLM inference infrastructure. Validates the cost models used in projects 12-14.
Model seq=128 seq=256 seq=512 Sim default
gpt2 56.0 39.4 30.3 50 us/tok
gpt2-medium 105.6 71.0 70.2 50 us/tok
At short sequences, CUDA kernel launch overhead dominates. At long sequences, cost converges to 30-70 us/token per model size.
Decode cost is constant regardless of prefix length:
gpt2: 5277-5391 us/token (variation < 2%)
gpt2-medium: 10513-10793 us/token (variation < 3%)
The bottleneck is loading model weights from GPU DRAM, not attention.
Single-request (bs=1): 5300-10800 us/token
Server amortized (bs~32+): ~200 us/token
The gap closes when the server processes many requests in parallel, amortizing the memory bandwidth cost across the batch.
gpt2 seq=512 bs=4: 276 MB
gpt2-medium seq=512 bs=4: 396 MB
Validates the KV pressure modeled in kv-cache-compaction-lab, kv-cache-disaggregation-sim, and admission-control-sim.
Regime prefill_us decode_us
Simulation default 50 200
GPT-2 real (long seq, bs=1) 30-56 5300
GPT-2 Medium real 70-105 10700
Server amortized 40-50 ~200
python3 -m venv venv
source venv/bin/activate
pip install transformers torch accelerate
python3 profile_model.py
results/prefill_profile.csv 35 rows: per_token_us vs seq_len and batch_size
results/decode_profile.csv 15 rows: per_token_us vs prefix_len and batch_size
results/memory_profile.csv 21 rows: KV cache and peak memory by config