vLLM on a T4 — Continuous Batching Benchmark

Ran vLLM on a free Colab T4 GPU and measured how throughput scales with batch size — the continuous-batching behavior a 4 GB local card can't demonstrate.

Headline finding

vLLM's continuous batching scales throughput with load, reaching ~790 tok/s at batch 64. Contrast with my GGUF serving benchmark (serving-benchmark), where Ollama peaked at ~113 tok/s and dropped past concurrency 4. vLLM does ~7× the throughput and keeps climbing where the GGUF backend fell off — this is why vLLM dominates high-throughput LLM serving.

Results (Qwen2.5-1.5B-Instruct, T4)

batch size	throughput
8	~290 t/s
32	~710 t/s
64	~790 t/s

Honest note — limits of this measurement

The lowest batch sizes read artificially low because vLLM compiles CUDA graphs on the first runs, so the early points are polluted by one-time warmup cost (same lesson as Project 2's warmup pass). A rigorous version fires warmup batches first and discards them. The trend — throughput scaling with batch size — is the real, reproducible finding.

Install gotcha (worth documenting)

vLLM's current wheel targets CUDA 13, but Colab's T4 runtime is CUDA 12 — so a plain pip install vllm fails with libcudart.so.13: cannot open shared object file. The working install:

!pip install "vllm<0.11"          # last CUDA-12 release line
!pip install "transformers<4.55"  # matching tokenizer API for that vLLM
# Runtime -> Restart session, then run

Run it

Open vllm_benchmark.ipynb in Colab → Runtime → change to T4 GPU → Run all.

Stack

vLLM · Qwen2.5-1.5B-Instruct · Colab T4 · matplotlib

Résumé line

Benchmarked vLLM on a T4 GPU — measured continuous-batching throughput scaling to ~790 tok/s at batch 64 (~7× a GGUF backend's peak) and documented the CUDA-12/13 + transformers version constraints for a clean install.