mlsystems

Analytical benchmark for speculative decoding at batch sizes 1-64. Finds the gamma crossover where speculative decoding becomes slower than greedy, and shows that batch size does not degrade performance when continuous refill is active.

Latency, diversity, and quality benchmark for autoregressive decoding strategies on RTX 2070: greedy, top-k, top-p, min-p, and beam search across GPT-2 models.

Discrete-event simulation of LLM request routing across multiple serving instances: round-robin, least-load, prefix-aware, and hybrid cache-aware routing with queue-depth threshold sweep.

Phase-by-phase benchmark of LLM instance cold start on RTX 2070: weight loading, PCIe transfer, CUDA init, KV cache allocation, and JIT warmup — with pre-warming savings and extrapolation to 7B+ models.

Discrete-event simulation of SLO-aware autoscaling for multi-instance LLM serving — comparing reactive, predictive, and conservative policies across workload patterns and startup delays.

Simulating KV cache reuse across multi-turn LLM conversations using real ShareGPT traces. Compares sticky, least-load, hybrid, and cost-aware routing strategies for cache locality vs load balance.

Discrete-event simulation of multi-LoRA adapter serving strategies on a single GPU — comparing naive swap, hot-set preloading, and batch-by-adapter under variable VRAM pressure and arrival rates.

Comparing LLM serving under real ShareGPT traces vs synthetic Poisson workloads. Real distributions produce 42x worse P99 TTFT and earlier SLO violations — scheduling policies only differentiate under heavy-tailed real traffic.

A C++ Autograd Engine written without ANY frameworks that can train a pretty big GPT locally. Built to maximize performance.

Benchmarking output length prediction quality and calibration for KV-cache-aware LLM serving scheduling. A noisy predictor with 1.5x safety margin captures 99.7% of oracle gains — calibration matters more than accuracy.

Comparing KV-cache-aware scheduling policies for LLM serving: greedy preempts 49% of requests under memory pressure while memory-first achieves 43% higher effective throughput with zero preemptions.

Benchmark comparing Flash-Decoding-style chunked KV-parallel attention against standard SDPA at decode time (q_len=1) on RTX 2070, measuring the crossover point where KV-parallelism outperforms fused attention kernels.

Step-level simulation of chunked prefill under KV cache pressure. Block policy wastes 89.6% of prefill chunks while achieving the same throughput as upfront rejection. Chunk waste rate is the metric standard throughput metrics miss.