Simulation benchmark for LLM serving observability design.
Answers the question that 76 prior benchmarks in this portfolio left open:
What is the minimum set of signals needed to detect all major serving failure modes before SLO violation — and what does each signal miss?
76 prior benchmarks identified many failure modes but never asked:
What metrics do you need to monitor to detect all of them?
Five key benchmarks connect here:
- admission-pressure-predictor: detects KV pressure before the cliff
- continuous-batching-frag: fragmentation as a throughput signal
- capacity-planning-sim: headroom_ms as operational safety signal
- slo-aware-autoscaling-sim: SLO violation as reactive signal
- inference-observability-bench: minimum signal set for full coverage
Every signal has at least one blind spot:
| Signal | Modes with useful lead time | Blind spot |
|---|---|---|
| ttft_p99_ms | fragmentation (13s), kv_cliff (5s), prefill (5s) | decode_spike (4s — too late) |
| tpot_p95_ms | decode (9s), fragmentation (18s), prefill (6s) | kv_cliff (3s — too late) |
| queue_depth_growth | kv_cliff (9s), prefill (21s) | decode, fragmentation |
| headroom_ms | kv_cliff (24s) | decode, fragmentation, prefill |
| fragmentation_pct | fragmentation (38s) | decode, kv_cliff, prefill |
| prefill_share_of_step | prefill (16s) | decode, fragmentation, kv_cliff |
| kv_utilization_rate | none | all 4 modes |
Despite being commonly monitored, it fires only 1 second before breach. It confirms incidents but does not prevent SLO violations.
tpot_p95_mscovers decode_spike, fragmentation_drift, prefill_interferencequeue_depth_growthorheadroom_mscloses the kv_cliff blind spot
| Signal | Modes with useful lead time | Coverage efficiency |
|---|---|---|
| ttft_p99_ms | 3 | 3.00 |
| tpot_p95_ms | 3 | 3.00 |
| queue_depth_growth | 2 | 1.33 |
| headroom_ms | 1 | 0.50 |
| fragmentation_pct | 1 | 0.40 |
| kv_utilization_rate | 0 | 0.00 |
Each structural signal covers a different failure mode with minimal overlap. The minimum effective bundle requires both latency and structural signals because their blind spots are different.
| Failure mode | Description | Earliest signal | Lead time |
|---|---|---|---|
| kv_cliff | KV cache fills up, queue grows, latency spikes | headroom_ms | 24s |
| fragmentation_drift | Batching efficiency degrades gradually | fragmentation_pct | 38s |
| decode_spike | Per-token decode latency increases abruptly | tpot_p95_ms | 9s |
| prefill_interference | Long prefills crowd out decode traffic | queue_depth_growth | 21s |
| Bundle | Kind | Signals | Coverage | Useful lead time |
|---|---|---|---|---|
| latency_only | baseline | 3 | 100% | 91.7% |
| queue_only | structural | 3 | 50% | 50% |
| resource_only | structural | 3 | 25% | 25% |
| latency_plus_resource | composite | 4 | 100% | 100% |
| structural_bundle | composite | 5 | 75% | 75% |
| minimal_set_cover | optimized | 1 | 100% | 58% |
| minimal_leadtime_cover | optimized | 2 | 100% | 100% |
| full_signal_set | oracle | 20 | 100% | 100% |
cd ~/dev/inference-observability-bench
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -u run.py
Runtime: approximately 10 seconds. No GPU required.
results/
summary.csv
per_signal_lead_time.csv
per_incident_lead_time.csv
blind_spots.csv
decision_map.csv
plots/
01_coverage_vs_cost.png
02_coverage_and_leadtime.png
03_obs_score.png
04_signal_count_vs_coverage.png
06_lead_time_heatmap.png
07_lead_time_comparison.png
08_blind_spots.png
inference-observability-bench/
+-- src/
| +-- config.py
| +-- incidents.py
| +-- signals.py
| +-- detector.py
| +-- optimizer.py
| +-- bench.py
| +-- analysis.py
+-- results/
+-- plots/
+-- run.py
+-- README.md
+-- summary.txt
+-- design.md
+-- LICENSE
+-- requirements.txt
This project uses synthetic incident traces to model each failure mode as a time series of metric values progressing toward SLO breach.
Each signal is evaluated independently against each incident trace. A detection is registered when the signal crosses its threshold. Lead time is measured as breach time minus detection time. Useful lead time is defined as >= 5 seconds.
The minimum signal set is found by a greedy set-cover optimizer that finds the smallest subset covering all failure modes.
For full design details, see design.md.
- Python 3.10+
- NumPy >= 1.26.0
- Pandas >= 2.0.0
- Matplotlib >= 3.8.0
No GPU required.
If you need a minimum observability bundle for LLM serving:
- always include tpot_p95_ms as primary latency signal
- add queue_depth_growth to close the kv_cliff blind spot
- add fragmentation_pct if throughput degradation is a concern
- do not rely on kv_utilization_rate as an early warning signal
- monitor headroom_ms as the primary operational safety indicator
- design.md -- detailed design rationale and signal definitions
- summary.txt -- concise high-level summary of findings
- LICENSE -- MIT License
MIT License -- Copyright (c) 2026 Joao Felipe De Souza
Joao Felipe De Souza 2026