Skip to content

DESIGN.md

Latest commit

 

History

History
97 lines (70 loc) · 3.39 KB

DESIGN.md

File metadata and controls

97 lines (70 loc) · 3.39 KB
Raw

Design Document -- continuous-batching-profiler

1. Motivation

The llm-inference-scheduler (project 3) simulated continuous batching with an analytical cost model. This project replaces the simulation with real PyTorch inference on GPU, measuring actual TTFT, TBT, and throughput.

The interference benchmark goes further: it creates a controlled scenario where long-prompt prefill interferes with active short-request decode, revealing the true purpose of chunked prefill.

2. Serving Strategies

FCFS

One request at a time. Full prefill, then full decode. No concurrency. Baseline for comparison.

EagerContBatch (Eager Continuous Batching)

Multiple requests active simultaneously. New requests are fully prefilled immediately on admission. All active requests share decode steps. Best average TTFT and throughput, but prefill of new requests blocks decode for existing ones.

ChunkedPrefill

Prefill split into chunks of configurable size. After each chunk, decode steps run for all active requests. This limits the maximum decode stall caused by a long prefill. Trades worse mean TTFT for better worst-case TBT.

3. Interference Benchmark Design

Fixed scenario to isolate the interference effect:

t=0.0: 2 short requests arrive (32 tok prompt, 64 tok output)
t=0.1: 2 long requests arrive (512 tok prompt, 32 tok output)
t=0.3: 2 more short requests arrive

Key metrics:

  • short_early TBT: time between tokens for shorts already in decode
  • short_early TBT max: worst-case decode stall
  • long TTFT: how fast the long prompts get their first token
  • throughput: overall tokens per second

4. Key Results

Interference comparison (5 strategies)

Strategy          short_early    short_early    long       throughput
                  TBT max (ms)   stalls>50ms   TTFT (ms)   (rps)
FCFS                97.5           1            1011.2      2.75
EagerContBatch      58.6           2              34.8      7.77
ChunkedPrefill_c64  57.9           2             217.6      7.96
ChunkedPrefill_c32  43.8           0             379.9      6.96
ChunkedPrefill_c16  34.1           0             682.0      5.70

The tradeoff

Smaller chunks:

  • BETTER: reduce worst-case TBT stalls (58.6 -> 34.1ms)
  • WORSE: increase long-request TTFT (34.8 -> 682.0ms)
  • WORSE: decrease throughput (7.77 -> 5.70 rps)

ChunkedPrefill is a fairness/SLA mechanism, not a throughput optimizer.

5. Validation of Simulation

The simulation (project 3) predicted:

  1. ChunkedPrefill eliminates decode starvation -> CONFIRMED
  2. FCFS collapses under load -> CONFIRMED (5.06 rps max vs 12.19 rps)
  3. Scheduling does not change throughput -> PARTIALLY CONFIRMED At low load: similar throughput At high load: scheduling strategy DOES affect throughput (FCFS saturates)

Refinement from real profiling:

  • ChunkedPrefill does not maximize mean TTFT — EagerContBatch does
  • ChunkedPrefill's value is in TBT tail control, not aggregate metrics
  • This distinction is invisible in the simulation's analytical model

6. Connections to Prior Projects

Project                      Validation
llm-inference-scheduler      ChunkedPrefill vs FCFS confirmed on real GPU
latency-breakdown-simulator  Prefill dominance at high load confirmed
real-model-profiler          Cost parameters validated
admission-control-sim        FCFS throughput saturation matches prediction