A single Streamlit chat UI that runs against any OpenAI-compatible LLM inference engine — vLLM, TGI, Ollama, or Triton fronting a TensorRT-LLM engine — with zero code changes between them. Built and tested on a RunPod A100 SXM GPU pod.
The point of this repo: prove that a client-facing API contract can stay stable while the serving engine underneath it changes — the same principle a real inference platform's API gateway relies on.
Multi-Inference-Engine-In-GPU-A100SXM/
├── README.md # you are here
├── requirements.txt # Python deps for the demo app
├── .env.example # template for local config
│
├── app/
│ └── streamlit_app.py # unified chat UI — engine-agnostic
│
├── engines/
│ ├── vllm/
│ │ └── setup.sh # install + launch vLLM
│ ├── tgi/
│ │ └── setup.sh # install + launch TGI (Docker)
│ ├── ollama/
│ │ └── setup.sh # install + launch Ollama (fastest path)
│ ├── tensorrt-llm/
│ │ └── SETUP.md # compile + prep a TensorRT-LLM engine
│ └── triton/
│ └── SETUP.md # serve a compiled engine via Triton
│
├── scripts/
│ ├── runpod_bootstrap.sh # one-time pod setup, run this first
│ ├── benchmark.py # TTFT + throughput benchmark, any engine
│ └── health_check.sh # quick liveness probe
│
└── docs/
└── ARCHITECTURE.md # diagrams + design rationale
This is the full sequence, in order, from an empty RunPod account to a working chat UI talking to a real GPU. Follow it top to bottom the first time; use the Quickstart section below once you've done it once.
Go to runpod.io → sign up → Billing → add credit (RunPod is pay-as-you-go per second of pod uptime; A100 SXM pods run roughly $1.50–$2.50/hr depending on region and availability at time of writing — check current pricing on the console before deploying).
- Console → Pods → Deploy
- GPU filter → select A100 SXM (40GB or 80GB — 80GB gives you more headroom for larger models / longer context, pick 40GB if you just need this demo working and want to save cost)
- Template: choose a template with CUDA pre-installed —
RunPod PyTorch 2.xis the safe default; it ships with CUDA, cuDNN, and Python already configured, so you skip driver-install headaches - Container Disk: bump to at least 50GB (model weights for an 8B model in FP16 are ~16GB; you'll want room for more than one model or engine's cache)
- Expose HTTP Ports: add
8501(Streamlit),8000(vLLM),8080(TGI),11434(Ollama) — RunPod proxies these through its own public URL per pod, you don't need to configure a firewall yourself - Click Deploy
Once the pod status shows Running:
- Click Connect → Start Web Terminal (fastest, no local SSH setup), or
- Click Connect → copy the SSH command if you prefer your own terminal:
ssh root@<pod-ip> -p <ssh-port> -i ~/.ssh/id_rsa
This is the single most important sanity check — do it before anything else:
nvidia-smiYou should see an A100-SXM4-40GB or A100-SXM4-80GB listed, a driver
version, and a CUDA version (12.x). If this fails or shows no GPU,
stop here — something's wrong with the pod allocation, don't proceed
to install anything until this is fixed (check the RunPod console for
the pod's GPU assignment, or terminate and redeploy).
git clone https://github.com/rpaut03l/Multi-Inference-Engine-In-GPU-A100SXM.git
cd Multi-Inference-Engine-In-GPU-A100SXMbash scripts/runpod_bootstrap.shThis installs system packages, confirms Docker/NVIDIA runtime status, and installs the Python requirements for the demo app.
pip install huggingface_hub
huggingface-cli login --token YOUR_HF_TOKENGet a token from huggingface.co/settings/tokens,
and make sure you've accepted the model's license on its HF page first
(e.g. meta-llama/Llama-3.1-8B-Instruct) — the download will fail with
a 403 otherwise, not an obvious error to debug if you don't know this
in advance.
Pick one (see the engine comparison table below for trade-offs):
# Fastest, lowest-risk option:
bash engines/ollama/setup.sh
# OR, production-representative option:
bash engines/vllm/setup.shLeave this running in its own terminal/tmux pane — it's a long-running server process, not a one-shot command.
In a second terminal (open another Web Terminal tab, or tmux):
bash scripts/health_check.sh http://localhost:11434/v1 # if using Ollama
# or
bash scripts/health_check.sh http://localhost:8000/v1 # if using vLLMConfirm this returns a healthy model list before debugging the UI — if the engine itself isn't up, the UI will just show a generic connection error that wastes time to trace back.
Still in that second terminal:
cd Multi-Inference-Engine-In-GPU-A100SXM
pip install -r requirements.txt
streamlit run app/streamlit_app.py --server.port 8501 --server.address 0.0.0.0--server.address 0.0.0.0 is required — Streamlit binds to localhost
only by default, which is unreachable from outside the pod's container.
RunPod console → your pod → Connect → under HTTP Services, click
the link next to port 8501. This opens Streamlit through RunPod's
proxy at a public URL like https://<pod-id>-8501.proxy.runpod.net.
In the Streamlit sidebar, pick the engine you launched in Step 8 from the dropdown, hit Ping engine to confirm connectivity, then chat.
python scripts/benchmark.py \
--base-url http://localhost:11434/v1 \
--model gemma3:12b \
--runs 5RunPod bills per second while a pod is Running — stop or terminate it from the console once you're finished, or you'll keep paying for an idle GPU overnight.
# 1. On your RunPod A100 SXM pod, one-time setup
bash scripts/runpod_bootstrap.sh
# 2. Launch Ollama
bash engines/ollama/setup.sh
# 3. In a separate terminal, run the demo UI
pip install -r requirements.txt
streamlit run app/streamlit_app.py --server.port 8501 --server.address 0.0.0.0
# 4. Open the pod's port-8501 link from the RunPod console,
# select "Ollama" in the sidebar, and chat| Engine | Setup effort | Best for | Key mechanism |
|---|---|---|---|
| Ollama | ⭐ Lowest | Fast demo, local iteration | llama.cpp-based, GGUF quantization |
| vLLM | ⭐⭐ Low | General-purpose production serving | PagedAttention, continuous batching |
| TGI | ⭐⭐ Low-Medium | HF ecosystem integration | Continuous batching, Docker-native |
| Triton + TensorRT-LLM | ⭐⭐⭐⭐ High | Max throughput on pinned models at scale | Ahead-of-time compiled kernels |
bash engines/ollama/setup.sh
# Exposes OpenAI-compatible API at http://localhost:11434/v1bash engines/vllm/setup.sh
# Exposes OpenAI-compatible API at http://localhost:8000/v1bash engines/tgi/setup.sh
# Exposes OpenAI-compatible API at http://localhost:8080/v1# See engines/tensorrt-llm/SETUP.md to compile the engine first,
# then engines/triton/SETUP.md to serve it.
# Exposes OpenAI-compatible API at http://localhost:8003/v1 (once configured)
⚠️ Only run one engine at a time on a single A100 — each one aggressively reserves GPU memory for weights + KV-cache. Stop one before starting the next. Seedocs/ARCHITECTURE.mdfor details.
app/streamlit_app.py is a chat interface with:
- Engine switcher (sidebar dropdown) — instantly retarget the UI at a different base URL/port, no restart needed
- Streaming responses via Server-Sent Events (SSE), token-by-token
- Live TTFT + throughput metrics shown under each response
- Health check button to confirm the selected engine is reachable before you try to chat with it
Run it with:
pip install -r requirements.txt
streamlit run app/streamlit_app.pypython scripts/benchmark.py \
--base-url http://localhost:8000/v1 \
--model meta-llama/Llama-3.1-8B-Instruct \
--runs 5Reports average Time-to-First-Token (TTFT) and tokens/sec across N runs — useful for genuinely comparing engines on the same hardware rather than quoting numbers from memory.
- Why one UI works for every engine: vLLM, TGI, Ollama, and Triton's
OpenAI frontend all implement the same
/v1/chat/completionswire format. The app only depends on that shared contract. - Why streaming matters: TTFT is dominated by the prefill phase (processing the whole prompt); inter-token latency after that is dominated by the memory-bandwidth-bound decode phase. Streaming lets the UI show progress instead of blocking on the full response.
- Why this maps to platform engineering, not just a demo: a real inference API gateway does exactly what this app's engine switcher does at a larger scale — routes requests to whichever backend serves a given model, while presenting one stable contract to clients.
See docs/ARCHITECTURE.md for the full diagram and rationale.
Two phases run for every request, and they stress the GPU in opposite ways. Understanding this split is the single most important mental model for reasoning about inference performance, cost, and tuning.
┌──────────────────────────────────────────────────────────────────┐
│ ONE INFERENCE REQUEST │
└──────────────────────────────────────────────────────────────────┘
PHASE 1: PREFILL PHASE 2: DECODE
───────────────── ────────────────
Process the ENTIRE input prompt Generate ONE new token at a time,
in a single parallel pass autoregressively, in a loop
[tok1][tok2][tok3]...[tokN] [tokN+1] → read KV-cache → [tokN+2]
│ │
▼ ▼
All N tokens' attention Each step reads the FULL
computed simultaneously KV-cache built so far —
→ GPU compute units are cache grows every step
saturated, this is where │
FLOPs matter most ▼
│ Only 1 token computed per step
▼ → GPU compute sits mostly idle,
COMPUTE-BOUND waiting on memory reads
│ │
▼ ▼
Determines: TTFT MEMORY-BANDWIDTH-BOUND
(Time To First Token) │
▼
Determines: ITL / TPOT
(Inter-Token Latency /
Time Per Output Token)
Long prompt ───► more parallel work ───► higher TTFT
─────────────────────────────────────────────────────────
What to watch: GPU utilization % during the FIRST moment
of a request (should spike near 100%)
What to tune: batch multiple prefills together if
several requests arrive close together
(continuous batching engines do this
automatically — vLLM, TGI)
Failure mode: a single very long prompt (e.g. 32K tokens)
can monopolize compute and delay TTFT for
every OTHER concurrent request behind it
Watch it live:
# Run this WHILE a request is in flight — watch utilization spike
# to ~100% during prefill, then drop during decode
nvidia-smi dmon -s u -c 20 Every step reads: full KV-cache (grows with sequence length)
+ all model weights
─────────────────────────────────────────────────────────
What to watch: GPU memory bandwidth utilization, NOT
compute utilization — compute sits idle
here, bandwidth is the real bottleneck
What to tune: KV-cache memory allocation (vLLM's
--gpu-memory-utilization), quantization
to shrink both weights AND cache reads,
continuous batching to keep multiple
sequences' decode steps overlapping
Failure mode: KV-cache grows unbounded with long
conversations/long generations → eventually
OOMs or forces eviction of other sequences
Watch it live:
# Memory throughput during decode -- this is the number that
# actually explains decode-phase latency, not GPU compute %
dcgmi dmon -e 1005,1009 -d 1000
# 1005 = GPU utilization, 1009 = memory bandwidth utilization| If your bottleneck is... | Symptom | Fix to try |
|---|---|---|
| Prefill (compute) | Slow TTFT, especially with long prompts | Reduce prompt length, or shard across GPUs (tensor parallel) so more compute is available in parallel |
| Decode (memory bandwidth) | Slow token-by-token streaming, TTFT is fine | Quantize the model (fewer bytes to read per step), use GQA-based models (smaller KV-cache), increase KV-cache page efficiency |
- A RunPod pod with an A100 SXM GPU (40GB or 80GB)
- CUDA drivers pre-installed (RunPod's PyTorch templates ship with these)
- Python 3.10+
- Docker + NVIDIA Container Toolkit (only needed for TGI and Triton — not required for vLLM or Ollama)
| Symptom | Likely cause | Fix |
|---|---|---|
nvidia-smi shows no GPU |
Pod didn't get a GPU allocated, or wrong template | Check pod GPU assignment in RunPod console; redeploy if needed |
Model download fails with 403 |
Gated model license not accepted, or bad/missing HF token | Accept the license on the model's HF page; re-run huggingface-cli login |
| Engine starts then immediately OOMs | --gpu-memory-utilization / context length too high for available VRAM |
Lower MAX_MODEL_LEN (vLLM) or reduce --max-total-tokens (TGI) |
| Streamlit UI loads but shows "Unreachable" on Ping | Engine not actually running, or wrong port in sidebar | Run scripts/health_check.sh against the engine's URL directly first |
| Streamlit page won't load from the RunPod proxy link | Forgot --server.address 0.0.0.0 |
Restart Streamlit with that flag — binding to localhost only isn't reachable externally |
| TGI container exits immediately | Docker shared memory too small (default 64MB) | Confirm --shm-size 1g is in the docker run command |
| Two engines both fail to start | Both trying to claim GPU memory simultaneously | Only run one engine at a time on a single A100 — stop one before starting the next |
This is the layer below the app — use these when the symptom is "something's wrong with the GPU itself" rather than "the app won't connect."
nvidia-smi+--------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 |
|-------------------------------+----------------------+-------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M|
|===============================+======================+===================|
| 0 A100-SXM4-80GB On | 00000000:00:1E.0 Off | 0 |
| N/A 42C P0 68W / 400W | 22145MiB / 81920MiB | 97% Default |
+-------------------------------+----------------------+-------------------+
What to read: Pwr:Usage/Cap (68W/400W = not power-throttled),
Temp (42C = healthy, A100s throttle around 85-90C), Memory-Usage
(22GB/80GB used — plenty of headroom), GPU-Util (97% — compute is
busy, expected during prefill or a batched decode step).
nvidia-smi dmon -s pucvmet -c 30Flags decoded: power, utilization, clock, violation
(throttle reasons), memory, ecc errors, temperature — -c 30
limits to 30 samples so it doesn't run forever.
# gpu pwr gtemp mtemp sm mem enc dec mclk pclk
# Idx W C C % % % % MHz MHz
0 68 42 - 97 64 - - 1512 1410
Watch the sm (streaming multiprocessor / compute) column spike during
prefill and drop during decode, while mem (memory controller
utilization) stays elevated through decode — this is the prefill/decode
distinction from the section above, observed directly.
# Install if not present (RunPod PyTorch images often have it already)
dcgmi discovery -l
# Live dashboard of key fields
dcgmi dmon -e 203,204,1005,1009 -d 1000Field IDs: 203=GPU temp, 204=power usage, 1005=SM utilization,
1009=memory bandwidth utilization. DCGM is the tool a real fleet
uses for programmatic, Prometheus-scrapeable GPU telemetry — nvidia-smi
is fine for one box, DCGM is what you'd point Prometheus at across a
whole cluster.
nvidia-smi -q -d ECCLook for Volatile Uncorrected under both SRAM and DRAM sections —
any non-zero uncorrected count is a real hardware concern, not noise.
# Faster one-line check
nvidia-smi --query-gpu=ecc.errors.uncorrected.volatile.total --format=csvnvidia-smi -q -d PERFORMANCELook specifically for the Clocks Throttle Reasons block:
Clocks Throttle Reasons
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Active <-- this is your problem
If HW Thermal Slowdown shows Active, the GPU is capping clocks to
protect itself — check cooling/airflow (on your own hardware) or, on a
cloud pod, this usually means noisy-neighbor thermal pressure or a
genuinely under-provisioned instance — worth escalating to RunPod
support with this exact output attached.
nvidia-smi topo -m GPU0 GPU1 CPU Affinity
GPU0 X NV12 0-63
GPU1 NV12 X 0-63
NV12 means NVLink-connected (fast); PHB/SYS would mean routed
through the PCIe host bridge (much slower) — this tells you whether
tensor-parallel sharding across these GPUs will actually be fast or
will bottleneck on interconnect.
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csvpid, process_name, used_memory [MiB]
14832, python3, 21504 MiB
Use this when memory usage looks wrong and you need to know which process is actually holding it — common after a crashed engine leaves orphaned memory allocated.
# 1. Confirm it's actually a GPU OOM, not host RAM
dmesg | tail -50 | grep -i "out of memory"
# 2. Check what's currently allocated
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
# 3. If an engine crashed and didn't release memory cleanly:
nvidia-smi --query-compute-apps=pid --format=csv,noheader | xargs -I{} kill -9 {}Then re-launch the engine with a lower --gpu-memory-utilization
(vLLM) or shorter --max-total-tokens (TGI) — OOM during decode
specifically usually means the KV-cache grew past what you budgeted for.
# Confirm the driver's kernel module is actually loaded
lsmod | grep nvidia
# Should show: nvidia, nvidia_uvm, nvidia_drm, nvidia_modeset
# If missing entirely, the driver isn't loaded -- rare on a RunPod
# template but worth ruling out first on bare-metal setupsThis expands each branch of the quick tree into its own investigation flow, so you're not just told what the problem category is, but exactly which command to run next and what each possible answer means.
┌─────────────────────┐
│ Something's wrong │
│ with GPU inference │
└───────────┬─────────┘
│
┌─────────────▼──────────────┐
│ Run: nvidia-smi │
└─────────────┬──────────────┘
│
┌────────────────────────┼────────────────────────┐
▼ ▼ ▼
┌───────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Shows NOTHING │ │ Shows GPU, but │ │ Shows GPU, looks │
│ / command not │ │ engine won't │ │ healthy, but │
│ found │ │ start / crashes │ │ something's slow │
└──────┬────────┘ └─────────┬────────┘ └─────────┬────────┘
│ │ │
▼ ▼ ▼
[Branch A] [Branch B] [Branch C]
Driver/kernel OOM / crash Performance
problem diagnosis bottleneck
│
┌─────────────┼─────────────┐
▼ ▼ ▼
[Branch C1] [Branch C2] [Branch C3]
Slow TTFT Slow token Thermal/ECC
(prefill) streaming (hardware)
(decode)
nvidia-smi
│
▼
"command not found"? ────► Yes ──► Driver not installed at all
│ → redeploy pod with a CUDA-enabled
│ template (RunPod PyTorch 2.x)
│ No — command exists but errors
▼
"No devices were found" or similar?
│
▼
Run: lsmod | grep nvidia
│
├── Shows nvidia, nvidia_uvm, nvidia_drm ──► Module loaded but GPU
│ not attached to this
│ container → check RunPod
│ console: is a GPU actually
│ assigned to this pod?
│
└── Shows NOTHING ──► Kernel module not loaded
→ Run: dmesg | grep -i nvidia
→ Look for a version-mismatch error between
driver and kernel — if found, this pod's
image is broken, don't debug further,
terminate and redeploy fresh
Engine process exits immediately
│
▼
Check exit signal: was it OOM-killed?
│
▼
Run: dmesg | tail -50 | grep -i "out of memory"
│
├── Match found ─────────────────────────────┐
│ ▼
│ ┌──────────────────────┐
│ │ CONFIRMED: GPU OOM │
│ └──────────┬───────────┘
│ │
│ ┌────────────────────┼────────────────────┐
│ ▼ ▼ ▼
│ Model too big for KV-cache config Orphaned memory
│ available VRAM too aggressive from a previous
│ → quantize (INT8/ → lower crashed process
│ AWQ) or use a --gpu-memory- → nvidia-smi
│ smaller model utilization --query-compute-
│ or --max-model-len apps=pid --format=
│ csv,noheader
│ → kill -9 <pid>
│
└── No OOM match ─────► Check engine's own logs directly
(not dmesg) — likely a config error:
wrong model path, missing HF auth (403),
or a port already in use
→ Run: docker logs <container> (TGI/Triton)
or check the terminal output directly (vLLM/Ollama)
Slow TTFT
│
▼
Run WHILE sending a request: nvidia-smi dmon -s u -c 20
│
▼
Does "sm" (compute) utilization spike near 100% during
the delay before the first token appears?
│
├── YES, sm spikes to ~100% ──► This IS prefill working as expected —
│ the slowness is proportional to PROMPT
│ LENGTH, not a bug
│ │
│ ▼
│ Is the prompt genuinely long (>2-4K tokens)?
│ │
│ ┌───────────┴───────────┐
│ ▼ ▼
│ YES NO
│ │ │
│ ▼ ▼
│ Expected behavior — Something else is adding
│ reduce prompt length latency before compute even
│ or shard across GPUs starts — check network path
│ with tensor parallelism (gateway → engine hop),
│ if consistently long not the GPU itself
│
└── NO, sm stays low ──► Compute isn't even being reached —
this usually means requests are QUEUED,
not actively processing
→ check the engine's queue depth metric
(vLLM/TGI both expose this on /metrics)
→ if queue is deep, you're at capacity —
this is a scaling problem, not a bug
Slow inter-token latency (ITL/TPOT)
│
▼
Run WHILE streaming: dcgmi dmon -e 1005,1009 -d 1000
│
▼
Is "mem" (memory bandwidth utilization) consistently high (>80%)
while "sm" (compute) stays comparatively low?
│
├── YES ──► CONFIRMED decode is memory-bandwidth-bound (expected
│ behavior of the decode phase itself) — the question is
│ whether it's SLOWER than it should be
│ │
│ ▼
│ Compare against the benchmark script's baseline numbers
│ (scripts/benchmark.py) for this same model/engine
│ │
│ ┌───────┴───────┐
│ ▼ ▼
│ Matches Meaningfully worse than
│ baseline expected baseline
│ │ │
│ ▼ ▼
│ This is just Investigate: is the KV-cache
│ the model's fragmented / near its memory
│ real speed on ceiling? Longer conversation
│ this hardware — history = larger cache = more
│ consider bytes read per step = naturally
│ quantization slower over a long session
│ to speed up
│
└── NO, mem is low too ──► Bottleneck isn't the GPU at all —
check network between client and
engine (SSE chunking, proxy buffering)
or CPU-side Python overhead in the
serving loop
Numbers look "off" but not clearly OOM or slow
│
▼
Run: nvidia-smi -q -d PERFORMANCE,ECC
│
├── "Clocks Throttle Reasons: HW Thermal Slowdown = Active"
│ │
│ ▼
│ GPU is self-protecting from overheating
│ │
│ ▼
│ On a cloud pod, this is almost never something YOU fix —
│ it's the provider's cooling/placement
│ │
│ ▼
│ Action: capture this exact output, open a support ticket
│ with RunPod referencing the throttle reason directly
│
└── ECC "Volatile Uncorrected" count > 0
│
▼
Possible hardware memory fault on this specific GPU
│
▼
Action: do NOT keep running production workloads on this
GPU — terminate the pod and redeploy onto a fresh one;
uncorrected ECC errors can silently corrupt computation
results, not just crash — this is the one failure mode
here that can produce WRONG answers without any error at all
┌────────────────────────┬──────────────────┬───────────────────────────┐
│ Symptom │ Can you fix it? │ If not, escalate to... │
├────────────────────────┼──────────────────┼───────────────────────────┤
│ Driver/module missing │ Yes — redeploy │ — │
│ OOM (config too big) │ Yes — retune │ — │
│ Orphaned process memory│ Yes — kill+retry │ — │
│ Slow prefill (long │ Yes — shorten │ — │
│ prompt, as expected) │ prompt / shard │ │
│ Slow decode (expected │ Yes — quantize │ — │
│ for model/hardware) │ │ │
│ Thermal throttling │ No: provider-side│ RunPod support │
│ Uncorrected ECC errors │ No: hardware │ RunPod support (terminate │
│ │ fault │ pod first, don't keep │
│ │ │ using it) │
└────────────────────────┴──────────────────┴───────────────────────────┘
MIT — use freely for demos, learning, or as a starting point for production serving infrastructure.