DSv4-Flash-spark

Serve DeepSeek-V4-Flash with DSpark speculative decoding on a pair of NVIDIA DGX Sparks (TP=2 over 200G RoCE, no Ray; add pairs to scale out) — with a tuned scheduler/serving config and two env-gated vLLM overlays that together cut multi-turn TTFT in half and add 3–5% single-stream decode.

Sibling repo to GLM-spark; same skeleton (launcher + recipes/ + mods/ + scripts/ + experimental/ for negative results).

Topology

The unit of deployment is one pair of DGX Sparks — that's all you need:

┌─────────────┐  2x200G RoCE  ┌─────────────┐
│ dgx-spark-1 ├───────────────┤ dgx-spark-2 │
│   rank 0    │  CX-7, f0↔f1  │   rank 1    │
└─────────────┘               └─────────────┘
   TP=2, PyTorch-distributed (--no-ray),
 one OpenAI endpoint at rank 0 port :8000

Scaling out (optional): more capacity = more independent pairs, each with its own launcher copy, master address, and endpoint (we run two pairs behind a client-side round-robin). Pairs share nothing at runtime — no cross-pair config exists. If you clone a launcher for a second pair, adapt SPARK2/master first (see Troubleshooting).

• Model: DeepSeek-V4-Flash — 43 layers, 256 experts top-6+1 shared (MXFP4), DSA sparse attention (index_topk 512), MLA, per-layer compress ratios 0/4/128.
• Spec decode: DSpark, γ=5 (3 MTP layers + Markov head). Decode = ~14.5 verify steps/s; tok/s = steps/s × (accepted+1), so 45–65+ tok/s depending on how predictable the content is.
• Fork: rafaelcaricio/vllm branch codex/dspark-harness-integration (DSpark harness integration over upstream vLLM).

Quickstart

1. Copy .env.example → .env, set your pair's addresses.
2. Put mods/dsv4-flash-overlays/*.py at /home/bird/vllm-overlays/ on both nodes (the launcher bind-mounts them; missing files are skipped safely).
3. cd launch && ./dspark_launch.sh on the rank-0 node. It cleans stale containers on both nodes, starts the worker over ssh, then the head. First boot takes ~10 min (graph capture); later boots are faster thanks to the persistent TileLang cache.
4. Health: curl localhost:8000/v1/models. Quality gate: python3 bench/quality_ref_dspark.py.

Before running a launcher copied between pairs, verify SPARK2 and the master address. An unadapted copy on pair B once pointed at pair A's worker and took the endpoint down; only an ssh host-key failure stopped it from killing the other pair too.

What's tuned, and what it bought (all A/B-measured on this hardware)

Change	Where	Effect
Incremental tokenizer cache	mods/.../deepseek_v4_tok.py, VLLM_INCREMENTAL_ENCODE_CACHE=1	median multi-turn TTFT 0.92s → 0.47s; saving grows linearly with context (35× faster encode, token-exact)
fp8 draft head	mods/.../dspark_nvidia.py, VLLM_DSPARK_FP8_DRAFT_HEAD=1	+3–5% single-stream decode (draft lm_head GEMM 2.73→1.45 ms, argmax-identical on eval set)
--long-prefill-token-threshold 4096	launcher	+53% decode tok/s while another request prefills (decode no longer stalls behind 8k-token chunks)
--max-num-seqs 24	launcher	aggregate decode ceiling 108 → 131 tok/s (single-stream unchanged)
Persistent TileLang cache	launcher mount, TILELANG_CACHE_DIR	removes 6–10 s JIT stall on first requests after restart
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=600	launcher	survives very long prefill steps that killed the engine at the 300 s default

The tokenizer cache is model-agnostic and is being proposed upstream to vllm-project/vllm; once merged there, drop the overlay. The fp8 draft head is DSpark-specific (fork territory).

Performance model (why decode is what it is)

Single-stream decode is weight-bandwidth-bound at ~14.5 verify steps/s (~67 ms/step on GB10's ~235 GB/s LPDDR5X): MoE experts ~30 ms (≈87% achieved BW), dense fp8 GEMMs ~10.5 ms, weight-only GEMMs ~7 ms, draft+verify heads ~4 ms, NCCL ~4.5 ms, misc ~6 ms. Steps/s is nearly flat in context depth and across every config dial we measured — perceived "slowdown with context" was tokenization cost + prefill interference, both fixed above. Raising tok/s further requires raising acceptance (drafter quality — a training problem), not serving config: wider speculation loses on this architecture because verifying M tokens pages in 256·(1−(249/256)^M) distinct experts, so a width-2 tree costs 1.85× MoE bandwidth and nets roughly zero. See experimental/negative-results.md for everything that didn't work.

Benchmarks

bench/ is contamination-guarded (samples num_requests_running in-flight and retries if foreign traffic overlaps — shared clusters lie to naive benchmarks):

• bench_ctx.py — context ladder: prefill tok/s, decode tok/s, acceptance per depth
• chat_ttft_probe.py — multi-turn TTFT (the tokenizer-cache metric)
• bench_interfere.py — decode-under-prefill (the LPT metric); solo phases are unguarded
• bench_conc.py — concurrency sweep (use per_req_decode_tok_s, not the ramp-polluted aggregate)
• quality_ref_dspark.py — 5 greedy prompts with exact-answer checks; run after any change

Troubleshooting

Symptom	Cause / fix
TP=2 collapses onto one GPU (local_world_size=2)	launched with Ray — always --no-ray (PyTorch-distributed); see launcher
Engine dead after huge prefill	step exceeded execute-model timeout; keep VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=600
First requests after restart take +6–10 s	TileLang JIT — keep the cache mount
ibv_modify_qp ... 110 Connection timed out on 4-rail	cables cross f0↔f1 but NCCL name-pairs identical dev names; use one port per card (2-rail; zero perf loss, allreduce is overlapped)
Host wedged, ssh banner timeout	compressed-MLA toggle hang wedged unified memory once; wait for ssh, docker rm -f, relaunch. Never set VLLM_DSV4_B12X_COMPRESSED_MLA
Benchmark numbers look impossible	foreign traffic on the endpoint — use bench/ guards, never trust a solo number without checking num_requests_running

License

MIT for this repo's own content (launchers, benchmarks, docs). The Python files under mods/ and experimental/ are modified copies of vLLM source and remain Apache-2.0 (SPDX headers + change statements in each file).

Acknowledgments

• DSpark integration we run: rafaelcaricio/vllm, lineage of local-inference-lab/vllm (community DSpark/SM120 hub). DSpark also merged into upstream vLLM on 2026-07-01 (vllm#46995, independent implementation) — a future rebase onto ≥v0.24.1 can retire much of this fork stack.
• NVFP4 KV / 1.5M-context variant explored (not deployed here): drowzeys/DeepSeek-V4-Flash-DSpark-NVFP4-KV-1.5M-CTX-2xDGX-Spark — a concurrency/long-context play (aggregate 318 tok/s @ c=16, single-stream lower than fp8 KV)