continuous-batching-profiler

Real PyTorch inference server comparing FCFS, ContinuousBatching, and ChunkedPrefill with actual GPU execution — including a controlled interference benchmark that reveals the true purpose of chunked prefill.

Validates and refines the simulation results from llm-inference-scheduler.

For design details see DESIGN.md.

---

The Refined Insight

The simulation (project 3) concluded: "ChunkedPrefill eliminates starvation; FCFS collapses under load."

Real profiling reveals a more nuanced truth:

FCFS is bad. Eager continuous batching is best on average. Chunked prefill is valuable not because it minimizes mean latency, but because it limits worst-case decode stalls for active requests under long-prompt interference.

---

Three Core Findings

1. FCFS collapses under any load

arrival_rate=10, n=30:
FCFS:              TTFT=1870ms  tput=5.06rps
ContinuousBatching: TTFT=  39ms  tput=12.19rps  <- 48x TTFT, 2.4x tput

Mixed workload, rate=5:
FCFS short requests: TTFT=806ms  (wait MORE than long requests at 553ms!)

FCFS inverts priority: short requests wait longer than long requests because they arrive after a long request that monopolizes the GPU.

2. Eager continuous batching wins on average metrics

Interference benchmark:
                  short_early   long     short_late   throughput
EagerContBatch:    11.6ms      34.8ms     16.3ms       7.77 rps
ChunkedPrefill_c64: 13.1ms   217.6ms     33.3ms       7.96 rps

Eager batching does full prefill immediately on admission. This gives the best TTFT for ALL request types and high throughput.

3. ChunkedPrefill is a fairness mechanism, not a throughput optimizer

Short-early TBT max (worst-case decode stall):
EagerContBatch:     58.6ms  (2 stalls > 50ms)
ChunkedPrefill_c32: 43.8ms  (0 stalls)
ChunkedPrefill_c16: 34.1ms  (0 stalls)

But the cost:
Long TTFT:          34.8ms -> 682.0ms (c16)
Throughput:         7.77 -> 5.70 rps  (c16)

Smaller chunks reduce worst-case decode stalls for already-active requests at the cost of worse TTFT for newly arriving long prompts and lower throughput.

ChunkedPrefill exists to protect decode streaming quality, not to maximize aggregate metrics.

---

Experiments

v1: Basic comparison (server.py)

FCFS vs ContinuousBatching vs ChunkedPrefill
16 runs: main comparison + arrival rate sweep + chunk size sweep

Key: ContBatch is 13x better TTFT than FCFS
Key: Chunk size is the primary TTFT control knob

v2: High load + mixed workload (server_v2.py)

26 runs across 4 experiments
- Uniform high load (n=30, rate=10)
- Mixed workload (50% short + 50% long)
- Arrival rate sweep: 2-40 req/s (uniform + mixed)

Key: FCFS saturates at 5.3 rps; ChunkedPrefill scales to 13.5 rps
Key: FCFS inverts priority in mixed workloads

Interference benchmark (interference_benchmark.py)

Controlled scenario:
t=0.0: 2 short requests (32 tok, 64 output)
t=0.1: 2 long requests (512 tok, 32 output)
t=0.3: 2 more short requests

5 strategies compared: FCFS, EagerContBatch, ChunkedPrefill c64/c32/c16

Key: ChunkedPrefill reduces worst-case TBT from 58.6ms to 34.1ms
Key: Price paid: long TTFT 34.8ms -> 682.0ms, throughput 7.77 -> 5.70 rps

---

Quick Start

python3 -m venv venv
source venv/bin/activate
pip install torch transformers

python3 server.py                    # v1: basic comparison
python3 server_v2.py                 # v2: high load + mixed
python3 interference_benchmark.py    # interference test

---

Results

results/batching_results.csv         v1 (16 rows)
results/batching_results_v2.csv      v2 (26 rows)
results/interference_benchmark.csv   interference test (5 rows)

---

Portfolio Context

Project 17 in a series on LLM inference infrastructure. Validates and refines the simulation from llm-inference-scheduler (project 3) using real GPT-2 inference on RTX 2070.

Full series: https://github.com/JohnScheuer