Skip to content

Commit 5940ef3

Browse files
cursoragentFluffyAIcode
cursoragent
and
FluffyAIcode
committed
bench(h200): Kakeya restored-S5 vs vLLM multi-tenant, same H200, gemma-4-26B ctx1238
Apples-to-apples on one H200 (same model bf16, identical NIAH ctx1238 token-ids, greedy, N=1,2,4,8). Both recall 1.0. vLLM ~8-14x faster absolute decode tok/s (optimized paged-attn/CUDA-graph/fused-MoE kernels vs Kakeya eager HF decode loop); Kakeya scales more linearly (8.15x@N=8 vs vLLM 4.66x). Memory: Kakeya bounded S5 57.7-77.6GB (reproduces quoted table), vLLM static 126GB pool (71GiB KV = 337k tokens). Adds vllm bench + report + 4 evidence JSONs. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
1 parent 84067f3 commit 5940ef3

5 files changed

Lines changed: 361 additions & 0 deletions

‎docs/reports/kakeya-vs-vllm-multitenant-h200.md‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Kakeya restored-S5 vs vLLM — multi-tenant parallel decode (same H200, gemma-4-26B-A4B)
2+
3+
Apples-to-apples comparison of Kakeya's recall-preserving **restored-S5** batched
4+
multi-tenant decode against **vLLM** (PagedAttention, the production baseline),
5+
run back-to-back on the **same H200** so there is zero cross-host variance.
6+
7+
## Methodology
8+
9+
| Axis | Setting (identical for both engines) |
10+
| --- | --- |
11+
| GPU | 1× NVIDIA H200 (~140 GB), `vastgpu4` |
12+
| Model | `google/gemma-4-26B-A4B-it`, **bf16** (the precision the Kakeya bench loads) |
13+
| Workload | NIAH, `make_niah_dataset(haystack_lines=60, seed=0)` → modal prompt **1238 tokens** (= the quoted ctx≈1238) |
14+
| Prompts | the modal-length bucket tiled to N, fed as **identical token-ids** to both engines |
15+
| Concurrency | N = 1, 2, 4, 8 sessions decoded in parallel |
16+
| Decode | greedy (temperature 0), **gen=128** (primary; stable steady-state rate) and gen=24 (matches the original table) |
17+
| Recall | answer needle substring in the decoded text |
18+
| Kakeya | `scripts/research/k3_cuda_multitenant_parallel_bench.py` — batched restored-S5 (per-session KV row), eager HF transformers decode loop |
19+
| vLLM | `scripts/research/vllm_multitenant_parallel_bench.py` — `vllm==0.23.0`, continuous batching; decode tok/s from per-request metrics (prefill excluded), matching the Kakeya decode-loop timing |
20+
21+
## Results (gen=128, ctx 1238, recall 1.0 throughout)
22+
23+
| N | Kakeya restored-S5 tok/s | (×N=1) | vLLM tok/s | (×N=1) | **vLLM / Kakeya** | Kakeya recall | vLLM recall |
24+
| --- | --- | --- | --- | --- | --- | --- | --- |
25+
| 1 | 16.2 | 1.00× | 224.8 | 1.00× | **13.9×** | 1.0 | 1.0 |
26+
| 2 | 32.0 | 1.98× | 364.8 | 1.62× | **11.4×** | 1.0 | 1.0 |
27+
| 4 | 65.9 | 4.08× | 579.9 | 2.58× | **8.8×** | 1.0 | 1.0 |
28+
| 8 | 131.7 | **8.15×** | 1048.0 | 4.66× | **8.0×** | 1.0 | 1.0 |
29+
30+
(gen=24 reproduces the same picture: Kakeya 17.9/35.6/71.0/110.7 tok/s, peaks
31+
57.7/60.5/66.2/77.5 GB — see below; vLLM's gen=24 decode rate is noisy because
32+
24 tokens is too short to measure a steady decode window, hence gen=128 is the
33+
primary comparison.)
34+
35+
## Memory
36+
37+
| N | Kakeya restored-S5 peak GPU | vLLM |
38+
| --- | --- | --- |
39+
| 1 | 57.7 GB | model ~52 GB + **static KV pool** |
40+
| 2 | 60.6 GB | reserves `gpu_memory_utilization=0.9` ≈ **126.6 GB** |
41+
| 4 | 66.2 GB | (KV pool = **71.1 GiB → 337,590 tokens**, 82× concurrency for 4096-len) |
42+
| 8 | 77.6 GB | independent of N (pool reserved up front) |
43+
44+
- **Kakeya restored-S5** allocates per batch and grows modestly with N
45+
(57.7 → 77.6 GB). Per-session KV is **bounded**: sink+window on the sliding
46+
layers + 5 exact full-attention layers (S5). This reproduces the quoted table
47+
near-exactly (57.6/60.4/66.0/77.3 GB), confirming identical model/precision/ctx.
48+
- **vLLM** is full-KV PagedAttention: it reserves a **static KV pool** (default
49+
90% util ≈ 126.6 GB measured peak) sized for 337k tokens, and per-token KV
50+
covers **all layers/positions** — so its KV footprint grows with total tokens
51+
and is **unbounded in context length**, the opposite of Kakeya's bounded S5.
52+
53+
## Findings (honest)
54+
55+
1. **vLLM is ~8–14× faster in absolute decode throughput.** This is a
56+
**kernel/implementation** gap — vLLM ships optimized paged-attention, CUDA
57+
graphs, and fused-MoE kernels; the Kakeya restored bench runs an **eager HF
58+
transformers decode loop** (`attn_implementation="eager"`, per-step Python
59+
`model()` calls). It is **not** an algorithmic property of restored-S5.
60+
2. **Kakeya scales more linearly** (8.15× at N=8 vs vLLM's 4.66×). The metric
61+
favors Kakeya only because its per-request rate is low (lots of batching
62+
headroom), while vLLM's per-request rate is already high (less headroom) — so
63+
parallel **speedup** and absolute **throughput** point in opposite
64+
directions, and absolute throughput is what ships.
65+
3. **Both preserve recall 1.0** at ctx 1238.
66+
4. **Different value axes.** vLLM wins raw throughput; Kakeya restored-S5's claim
67+
is **bounded per-session KV + recall restoration**, whose payoff is memory at
68+
long context / many tenants, not raw speed in this eager implementation.
69+
Closing the absolute-throughput gap would require porting restored-S5 onto
70+
optimized kernels — the bounded-KV + recall contribution is orthogonal to
71+
kernel optimization.
72+
73+
## Cross-host note
74+
75+
This H200 is slower (and likely more contended) than the host that produced the
76+
originally-quoted table: Kakeya N=1 here is 16.2 tok/s vs the quoted 27.4, while
77+
**peak memory reproduces the quoted table almost exactly**. Because both engines
78+
were measured on the **same** host, the transferable result is the **ratio**
79+
(vLLM ≈ 8–14× the Kakeya restored-S5 decode throughput at equal recall), not the
80+
absolute tok/s.
81+
82+
## Evidence
83+
84+
- `results/research/k3_cuda_multitenant_parallel_h200nvl_ctx1238_gen128.json`
85+
- `results/research/vllm_multitenant_parallel_h200nvl_ctx1238_gen128.json`
86+
- `results/research/k3_cuda_multitenant_parallel_h200nvl_ctx1238.json` (gen=24)
87+
- `results/research/vllm_multitenant_parallel_h200nvl_ctx1238.json` (gen=24)

‎results/research/k3_cuda_multitenant_parallel_h200nvl_ctx1238.json‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
{
2+
"kind": "k3_cuda_multitenant_parallel",
3+
"schema_version": 1,
4+
"config": {
5+
"verifier_id": "google/gemma-4-26B-A4B-it",
6+
"drafter_id": "z-lab/gemma-4-26B-A4B-it-DFlash",
7+
"haystack_lines": 60,
8+
"modal_prompt_len": 1238,
9+
"gen_tokens": 24,
10+
"sink": 4,
11+
"window": 64,
12+
"batch_sizes": [
13+
1,
14+
2,
15+
4,
16+
8
17+
],
18+
"exact_layers": [
19+
5,
20+
11,
21+
17,
22+
23,
23+
29
24+
],
25+
"note": "per-session binding via batched decode (each row = a session with its own KV-cache row); recall-preserving S5 only \u2014 non-recall configs out of scope."
26+
},
27+
"env": {
28+
"gpu": "NVIDIA H200",
29+
"torch": "2.12.0+cu130"
30+
},
31+
"results": [
32+
{
33+
"agents": 1,
34+
"ar_aggregate_tps": 17.77,
35+
"restored_aggregate_tps": 17.89,
36+
"ar_recall": 1.0,
37+
"restored_recall": 1.0,
38+
"ar_parallel_speedup_vs_n1": 1.0,
39+
"restored_parallel_speedup_vs_n1": 1.0,
40+
"peak_gpu_gb": 57.73
41+
},
42+
{
43+
"agents": 2,
44+
"ar_aggregate_tps": 35.26,
45+
"restored_aggregate_tps": 35.55,
46+
"ar_recall": 1.0,
47+
"restored_recall": 1.0,
48+
"ar_parallel_speedup_vs_n1": 1.98,
49+
"restored_parallel_speedup_vs_n1": 1.99,
50+
"peak_gpu_gb": 60.52
51+
},
52+
{
53+
"agents": 4,
54+
"ar_aggregate_tps": 70.55,
55+
"restored_aggregate_tps": 71.02,
56+
"ar_recall": 1.0,
57+
"restored_recall": 1.0,
58+
"ar_parallel_speedup_vs_n1": 3.97,
59+
"restored_parallel_speedup_vs_n1": 3.97,
60+
"peak_gpu_gb": 66.17
61+
},
62+
{
63+
"agents": 8,
64+
"ar_aggregate_tps": 142.04,
65+
"restored_aggregate_tps": 110.69,
66+
"ar_recall": 1.0,
67+
"restored_recall": 1.0,
68+
"ar_parallel_speedup_vs_n1": 7.99,
69+
"restored_parallel_speedup_vs_n1": 6.19,
70+
"peak_gpu_gb": 77.46
71+
}
72+
]
73+
}

‎results/research/k3_cuda_multitenant_parallel_h200nvl_ctx1238_gen128.json‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
{
2+
"kind": "k3_cuda_multitenant_parallel",
3+
"schema_version": 1,
4+
"config": {
5+
"verifier_id": "google/gemma-4-26B-A4B-it",
6+
"drafter_id": "z-lab/gemma-4-26B-A4B-it-DFlash",
7+
"haystack_lines": 60,
8+
"modal_prompt_len": 1238,
9+
"gen_tokens": 128,
10+
"sink": 4,
11+
"window": 64,
12+
"batch_sizes": [
13+
1,
14+
2,
15+
4,
16+
8
17+
],
18+
"exact_layers": [
19+
5,
20+
11,
21+
17,
22+
23,
23+
29
24+
],
25+
"note": "per-session binding via batched decode (each row = a session with its own KV-cache row); recall-preserving S5 only \u2014 non-recall configs out of scope."
26+
},
27+
"env": {
28+
"gpu": "NVIDIA H200",
29+
"torch": "2.12.0+cu130"
30+
},
31+
"results": [
32+
{
33+
"agents": 1,
34+
"ar_aggregate_tps": 15.84,
35+
"restored_aggregate_tps": 16.16,
36+
"ar_recall": 1.0,
37+
"restored_recall": 1.0,
38+
"ar_parallel_speedup_vs_n1": 1.0,
39+
"restored_parallel_speedup_vs_n1": 1.0,
40+
"peak_gpu_gb": 57.73
41+
},
42+
{
43+
"agents": 2,
44+
"ar_aggregate_tps": 33.14,
45+
"restored_aggregate_tps": 32.01,
46+
"ar_recall": 1.0,
47+
"restored_recall": 1.0,
48+
"ar_parallel_speedup_vs_n1": 2.09,
49+
"restored_parallel_speedup_vs_n1": 1.98,
50+
"peak_gpu_gb": 60.55
51+
},
52+
{
53+
"agents": 4,
54+
"ar_aggregate_tps": 65.48,
55+
"restored_aggregate_tps": 65.93,
56+
"ar_recall": 1.0,
57+
"restored_recall": 1.0,
58+
"ar_parallel_speedup_vs_n1": 4.13,
59+
"restored_parallel_speedup_vs_n1": 4.08,
60+
"peak_gpu_gb": 66.21
61+
},
62+
{
63+
"agents": 8,
64+
"ar_aggregate_tps": 131.73,
65+
"restored_aggregate_tps": 131.69,
66+
"ar_recall": 1.0,
67+
"restored_recall": 1.0,
68+
"ar_parallel_speedup_vs_n1": 8.31,
69+
"restored_parallel_speedup_vs_n1": 8.15,
70+
"peak_gpu_gb": 77.55
71+
}
72+
]
73+
}

‎results/research/vllm_multitenant_parallel_h200nvl_ctx1238.json‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
{
2+
"kind": "vllm_multitenant_parallel",
3+
"schema_version": 1,
4+
"config": {
5+
"verifier_id": "google/gemma-4-26B-A4B-it",
6+
"dtype": "bfloat16",
7+
"quantization": null,
8+
"haystack_lines": 60,
9+
"modal_prompt_len": 1238,
10+
"gen_tokens": 24,
11+
"batch_sizes": [
12+
1,
13+
2,
14+
4,
15+
8
16+
],
17+
"gpu_memory_utilization": 0.9,
18+
"max_model_len": 4096,
19+
"note": "vLLM PagedAttention (full KV) baseline; continuous batching is the parallel path. decode tok/s excludes prefill via per-request metrics, matching the Kakeya decode-loop timing. vLLM pre-reserves a KV pool (gpu_memory_utilization), so torch_peak reflects the reserved pool, not per-request growth \u2014 see kv_pool."
20+
},
21+
"env": {
22+
"gpu": "NVIDIA H200",
23+
"torch": "2.11.0+cu130"
24+
},
25+
"kv_pool": null,
26+
"results": [
27+
{
28+
"agents": 1,
29+
"decode_aggregate_tps": 196.97,
30+
"e2e_aggregate_tps": 205.54,
31+
"recall": 1.0,
32+
"decode_parallel_speedup_vs_n1": 1.0,
33+
"e2e_parallel_speedup_vs_n1": 1.0,
34+
"torch_peak_alloc_gb": 0.0
35+
},
36+
{
37+
"agents": 2,
38+
"decode_aggregate_tps": 281.7,
39+
"e2e_aggregate_tps": 293.95,
40+
"recall": 1.0,
41+
"decode_parallel_speedup_vs_n1": 1.43,
42+
"e2e_parallel_speedup_vs_n1": 1.43,
43+
"torch_peak_alloc_gb": 0.0
44+
},
45+
{
46+
"agents": 4,
47+
"decode_aggregate_tps": 107.14,
48+
"e2e_aggregate_tps": 111.8,
49+
"recall": 1.0,
50+
"decode_parallel_speedup_vs_n1": 0.54,
51+
"e2e_parallel_speedup_vs_n1": 0.54,
52+
"torch_peak_alloc_gb": 0.0
53+
},
54+
{
55+
"agents": 8,
56+
"decode_aggregate_tps": 794.18,
57+
"e2e_aggregate_tps": 828.71,
58+
"recall": 1.0,
59+
"decode_parallel_speedup_vs_n1": 4.03,
60+
"e2e_parallel_speedup_vs_n1": 4.03,
61+
"torch_peak_alloc_gb": 0.0
62+
}
63+
]
64+
}

‎results/research/vllm_multitenant_parallel_h200nvl_ctx1238_gen128.json‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
{
2+
"kind": "vllm_multitenant_parallel",
3+
"schema_version": 1,
4+
"config": {
5+
"verifier_id": "google/gemma-4-26B-A4B-it",
6+
"dtype": "bfloat16",
7+
"quantization": null,
8+
"haystack_lines": 60,
9+
"modal_prompt_len": 1238,
10+
"gen_tokens": 128,
11+
"batch_sizes": [
12+
1,
13+
2,
14+
4,
15+
8
16+
],
17+
"gpu_memory_utilization": 0.9,
18+
"max_model_len": 4096,
19+
"note": "vLLM PagedAttention (full KV) baseline; continuous batching is the parallel path. decode tok/s excludes prefill via per-request metrics, matching the Kakeya decode-loop timing. vLLM pre-reserves a KV pool (gpu_memory_utilization), so torch_peak reflects the reserved pool, not per-request growth \u2014 see kv_pool."
20+
},
21+
"env": {
22+
"gpu": "NVIDIA H200",
23+
"torch": "2.11.0+cu130"
24+
},
25+
"kv_pool": null,
26+
"results": [
27+
{
28+
"agents": 1,
29+
"decode_aggregate_tps": 224.82,
30+
"e2e_aggregate_tps": 226.59,
31+
"recall": 1.0,
32+
"decode_parallel_speedup_vs_n1": 1.0,
33+
"e2e_parallel_speedup_vs_n1": 1.0,
34+
"torch_peak_alloc_gb": 0.0
35+
},
36+
{
37+
"agents": 2,
38+
"decode_aggregate_tps": 364.81,
39+
"e2e_aggregate_tps": 367.69,
40+
"recall": 1.0,
41+
"decode_parallel_speedup_vs_n1": 1.62,
42+
"e2e_parallel_speedup_vs_n1": 1.62,
43+
"torch_peak_alloc_gb": 0.0
44+
},
45+
{
46+
"agents": 4,
47+
"decode_aggregate_tps": 579.93,
48+
"e2e_aggregate_tps": 584.49,
49+
"recall": 1.0,
50+
"decode_parallel_speedup_vs_n1": 2.58,
51+
"e2e_parallel_speedup_vs_n1": 2.58,
52+
"torch_peak_alloc_gb": 0.0
53+
},
54+
{
55+
"agents": 8,
56+
"decode_aggregate_tps": 1048.02,
57+
"e2e_aggregate_tps": 1056.27,
58+
"recall": 1.0,
59+
"decode_parallel_speedup_vs_n1": 4.66,
60+
"e2e_parallel_speedup_vs_n1": 4.66,
61+
"torch_peak_alloc_gb": 0.0
62+
}
63+
]
64+
}

0 commit comments

Comments
 (0)