This is a production‑style LLM inference platform built on vLLM and deployed on AWS/EKS. It’s designed to run under real load, where latency, throughput, and GPU behavior actually matter. The goal isn’t to show off model performance—it’s to understand how an LLM system behaves end‑to‑end when it’s doing real work.
LLM inference systems fail in ways that are hard to see. Latency jumps without warning. GPUs report high utilization but produce no work. Routing logic becomes opaque under load. A single backend issue can cascade through the entire stack.
This platform exists to make those behaviors visible, explainable, and contained. It’s a layered system where each layer owns one invariant, isolates its own failures, and exposes the signals you need to understand how the system is actually behaving in production.
vLLM Inference Architecture
Four layers, cleanly separated. Traffic admission doesn't know about routing. Routing doesn't touch GPU execution. Observability ties it all together.
|
|
This is the front door. It accepts traffic, validates it, and forwards good requests downstream. What it does:
Checks that requests are well-formed Rejects bad traffic immediately Scales horizontally Doesn't make routing decisions Doesn't know anything about models or GPUs
Why: If you don't control traffic at the edge, every downstream component has to defend itself. That adds latency and couples failures across the system.
This decides where each request should go. What it does:
Looks at request metadata, backend health, and policy Makes a routing decision and logs it Handles retries with clear limits Supports session affinity if needed Everything is observable
Why: When routing logic lives inside gateways or inference code, it becomes impossible to test or understand. Pulling it into its own layer means you can change routing without touching GPU execution.
This runs the models on GPUs and returns results. What it does:
Executes inference Stays GPU-saturated Warms up to avoid cold starts Reports per-request latency and health Doesn't know or care about routing
Why: GPU execution is expensive and fragile. Isolating it makes performance predictable and cost visible. You can experiment here without risking the control plane.
This makes the system understandable.
- Tracks latency, throughput, and GPU utilization
- Correlates signals across all layers
- Produces structured logs
- Alerts on things you can act on
Why: Without observability, you're guessing. You can't explain latency, plan capacity, or debug failures. This layer closes the loop between what the system does and why it does it.
What happens under load :
Traffic hits the edge gateway, which validates and forwards it. The router decides where it should go and logs that decision. Inference backends execute on GPUs. Observability captures how everything interacts.
You can answer real questions:
- Why did time-to-first-token increase?
- Which backend handled this request and why?
- Are the GPUs actually saturated or just stuck?
- Where's the bottleneck—routing, execution, or capacity?
End-to-End LLM Inference Under Sustained Load
DCGM GPU Metrics
Prefill vs Decode Behavior
- Each layer owns a single responsibility
- System behavior is deterministic, not heuristic
- Failures are contained and never silent
- Observe before optimizing
- Configure the system — don’t rewrite it
This is a working system you can run, inspect, and reason about.
It’s been tested under sustained load and structured for long‑term maintainability.
This is not a benchmark, a framework comparison, a demo, or a pile of scripts.
It’s an actual platform built to expose the real engineering challenges of LLM inference.
- services/edge-gateway/ — Layer 1
- services/router-service/ — Layer 2
- services/inference-backend/ — Layer 3
- services/observability/ — Layer 4
- infrastructure/ — Terraform and cluster setup
- load-testing/ — Load generation
- docs/ — Architecture details
Each layer has its own README explaining the internal design.
Apache 2.0 - See LICENSE