Serving video vision-language models is limited by KV cache capacity, not by memory bandwidth and not by compute. This repository contains the analytical model behind that claim, parameterised so anyone can check it against their own model and workload.
GPUs amortise weight bandwidth across a batch. They cannot amortise KV cache bandwidth, because the KV cache is private to each sequence. In text serving this only bites at large batch sizes. In video serving it bites immediately, because a single request carries an enormous visual prefix.
The consequence is not that the hardware runs slower. It is that the same memory holds far less useful work.
Qwen2.5-VL-72B geometry (80 layers, 8 KV heads, head_dim 128 → 320 KB of KV per token), 72 GB of int8 weights, visual tokens at 28×28 pixels each with 2-frame temporal merging.
| Workload | Visual tokens | KV (fp16) | Batch where KV > weights |
|---|---|---|---|
| Text, 4K context | 4,096 | 1.25 GB | 57.6 |
| High-res image (4 tiles + thumbnail) | 1,280 | 0.39 GB | 184.3 |
| Video, 1 min @ 1 fps | 7,680 | 2.34 GB | 30.7 |
| Video, 10 min @ 1 fps | 76,800 | 23.44 GB | 3.1 |
| Video, 1 h @ 1 fps | 460,800 | 140.62 GB | 0.5 |
| Video, 1 h @ 2 fps, 720p | 4,050,000 | 1,235.96 GB | — |
On an 8×H100 node (640 GB, minus 72 GB of weights):
| Workload | Concurrent users | Aggregate tok/s | Per-user tok/s |
|---|---|---|---|
| Text, 4K context | 454 | 17,719 | 39.0 |
| Video, 10 min @ 1 fps | 24 | 944 | 39.3 |
| Video, 1 h @ 1 fps | 4 | 157 | 39.3 |
The step time is 25.6 ms in every row, and per-user throughput is ~39 tok/s in every row. Only aggregate throughput collapses, by 113×. The node is equally busy moving bytes; it simply holds 454 conversations' worth of work in one case and four in the other. This is what makes it a capacity limit rather than a bandwidth limit, and it is why adding bandwidth does not help while adding capacity multiplies useful work directly.
Arithmetic intensity makes the same point. An H100 balances at roughly 300 FLOP/byte:
| Batch | Weights (FLOP/byte) | Attention (FLOP/byte) |
|---|---|---|
| 1 | 2 | 2 |
| 32 | 64 | 2 |
| 128 | 256 | 2 |
| 512 | 1,024 | 2 |
Weights get there by batching. Attention never moves.
No dependencies beyond a Python 3 interpreter.
python vlm_kv_calc.py # KV per token, per scenario, capacity and latency
python vlm_kv_batch.py # crossover batch, concurrent users, throughput ceiling
Model geometry, resolution, frame rate and duration are constants at the top of each file.
- This is an analytical model, not a measurement. It computes KV cache size from model geometry and token counts. It has not yet been validated against instrumented inference.
- KV compression is not modelled by default. The scripts report an illustrative int8 + 90 % figure, but modern VLM KV compression (VL-Cache, KVCapsule, and others) is a serious counterargument. Under aggressive compression a one-hour video at 1 fps falls to roughly 7 GB and fits comfortably. The capacity argument survives in a narrower regime: high resolution, real frame rates, continuous streaming, and multi-tenant serving, where compression reduces less than concurrency multiplies.
- Prefill is out of scope. Encoding hundreds of thousands of visual tokens is compute-bound and stays on the GPU. This work concerns decode.
- Visual token counts follow the Qwen2-VL/Qwen2.5-VL convention. Other models tile differently; the geometry constants need adjusting per model.
The visual KV cache is largely a static prefix: its size is known once prefill completes, it does not grow, and it is reusable across requests about the same image or video. That property removes most of the dynamic-mapping problem that makes text LLMs awkward for processing-in-memory, which suggests video VLM serving is a better fit for a dedicated near-memory KV tier than text serving is.
Jesús Pérez Bazarot — Universidad de Sevilla. Corrections and counterarguments are genuinely welcome; open an issue.
MIT