Skip to content

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

inference-observability-bench

Python License Status Methodology Domain Focus

Simulation benchmark for LLM serving observability design.

Answers the question that 76 prior benchmarks in this portfolio left open:

What is the minimum set of signals needed to detect all major serving failure modes before SLO violation — and what does each signal miss?


Why This Exists

76 prior benchmarks identified many failure modes but never asked:

What metrics do you need to monitor to detect all of them?

Five key benchmarks connect here:

  • admission-pressure-predictor: detects KV pressure before the cliff
  • continuous-batching-frag: fragmentation as a throughput signal
  • capacity-planning-sim: headroom_ms as operational safety signal
  • slo-aware-autoscaling-sim: SLO violation as reactive signal
  • inference-observability-bench: minimum signal set for full coverage

Key Results

No individual signal covers all failure modes with useful lead time

Every signal has at least one blind spot:

Signal Modes with useful lead time Blind spot
ttft_p99_ms fragmentation (13s), kv_cliff (5s), prefill (5s) decode_spike (4s — too late)
tpot_p95_ms decode (9s), fragmentation (18s), prefill (6s) kv_cliff (3s — too late)
queue_depth_growth kv_cliff (9s), prefill (21s) decode, fragmentation
headroom_ms kv_cliff (24s) decode, fragmentation, prefill
fragmentation_pct fragmentation (38s) decode, kv_cliff, prefill
prefill_share_of_step prefill (16s) decode, fragmentation, kv_cliff
kv_utilization_rate none all 4 modes

kv_utilization_rate provides zero useful early warning

Despite being commonly monitored, it fires only 1 second before breach. It confirms incidents but does not prevent SLO violations.

A minimum of 2 signals covers all failure modes with useful lead time

  • tpot_p95_ms covers decode_spike, fragmentation_drift, prefill_interference
  • queue_depth_growth or headroom_ms closes the kv_cliff blind spot

Latency signals are most efficient per unit cost

Signal Modes with useful lead time Coverage efficiency
ttft_p99_ms 3 3.00
tpot_p95_ms 3 3.00
queue_depth_growth 2 1.33
headroom_ms 1 0.50
fragmentation_pct 1 0.40
kv_utilization_rate 0 0.00

Structural signals are complementary, not redundant

Each structural signal covers a different failure mode with minimal overlap. The minimum effective bundle requires both latency and structural signals because their blind spots are different.


Failure Modes

Failure mode Description Earliest signal Lead time
kv_cliff KV cache fills up, queue grows, latency spikes headroom_ms 24s
fragmentation_drift Batching efficiency degrades gradually fragmentation_pct 38s
decode_spike Per-token decode latency increases abruptly tpot_p95_ms 9s
prefill_interference Long prefills crowd out decode traffic queue_depth_growth 21s

Observability Bundles

Bundle Kind Signals Coverage Useful lead time
latency_only baseline 3 100% 91.7%
queue_only structural 3 50% 50%
resource_only structural 3 25% 25%
latency_plus_resource composite 4 100% 100%
structural_bundle composite 5 75% 75%
minimal_set_cover optimized 1 100% 58%
minimal_leadtime_cover optimized 2 100% 100%
full_signal_set oracle 20 100% 100%

Quick Start

cd ~/dev/inference-observability-bench

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

python -u run.py

Runtime: approximately 10 seconds. No GPU required.


Output Files

results/
  summary.csv
  per_signal_lead_time.csv
  per_incident_lead_time.csv
  blind_spots.csv
  decision_map.csv

plots/
  01_coverage_vs_cost.png
  02_coverage_and_leadtime.png
  03_obs_score.png
  04_signal_count_vs_coverage.png
  06_lead_time_heatmap.png
  07_lead_time_comparison.png
  08_blind_spots.png

Project Structure

inference-observability-bench/
+-- src/
|   +-- config.py
|   +-- incidents.py
|   +-- signals.py
|   +-- detector.py
|   +-- optimizer.py
|   +-- bench.py
|   +-- analysis.py
+-- results/
+-- plots/
+-- run.py
+-- README.md
+-- summary.txt
+-- design.md
+-- LICENSE
+-- requirements.txt

Methodology

This project uses synthetic incident traces to model each failure mode as a time series of metric values progressing toward SLO breach.

Each signal is evaluated independently against each incident trace. A detection is registered when the signal crosses its threshold. Lead time is measured as breach time minus detection time. Useful lead time is defined as >= 5 seconds.

The minimum signal set is found by a greedy set-cover optimizer that finds the smallest subset covering all failure modes.

For full design details, see design.md.


Requirements

  • Python 3.10+
  • NumPy >= 1.26.0
  • Pandas >= 2.0.0
  • Matplotlib >= 3.8.0

No GPU required.


Practical Recommendation

If you need a minimum observability bundle for LLM serving:

  • always include tpot_p95_ms as primary latency signal
  • add queue_depth_growth to close the kv_cliff blind spot
  • add fragmentation_pct if throughput degradation is a concern
  • do not rely on kv_utilization_rate as an early warning signal
  • monitor headroom_ms as the primary operational safety indicator

Documentation

  • design.md -- detailed design rationale and signal definitions
  • summary.txt -- concise high-level summary of findings
  • LICENSE -- MIT License

License

MIT License -- Copyright (c) 2026 Joao Felipe De Souza


Author

Joao Felipe De Souza 2026

About

Simulation benchmark for LLM serving observability design: which minimum set of signals detects all major failure modes before SLO violation, and what does each signal miss?

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages