dgx-spark-ling

Serve inclusionAI's official Ling-3.0-flash INT4 on a single NVIDIA DGX Spark (GB10), where it benches faster than the community GGUF everyone runs today.

Ling-3.0-flash is a 124B MoE (5.1B active) with hybrid KDA + MLA attention. The official quants look 2-box-or-datacenter on paper: the fp4 serve config wants --tp-size 2, the fp8 does not fit 128 GB, and the int4 loads on one Spark but the obvious serving paths fail in ways that waste days. This repository is the working single-box recipe, the exact reason each wall exists, and the numbers.

Target hardware: NVIDIA DGX Spark. GB10 Grace-Blackwell, compute sm_121a, 128 GB unified LPDDR5X (about 119 GB usable), DGX OS (Ubuntu 24.04), CUDA 13.0, driver 580.x, aarch64. Single box, tensor-parallel size 1.

TL;DR, the numbers

Same box, same prompt, same measurement, back to back, single stream, 512-token generations, warmup discarded, mean of 3:

path	speed	notes
official INT4, vllm-ling-v3 fork, cudagraphs + MTP	38.7 tok/s	this repo's recipe. re-verified 40.9 on Aug 22
community Q5_K_M GGUF, llama.cpp fork	35.2 tok/s	the popular drop-in, full 128K native ctx
official INT4, same fork, cudagraphs, no MTP	22.9 tok/s	the drafter is worth 1.8x on its own
official INT4, same fork, --enforce-eager, no MTP	20.8 tok/s	the naive config

The official int4 went from "does not run on this box" to the fastest single-Spark path. The speed was in inclusionAI's code the whole time, their fork, their KDA kernels, their MTP draft layer. This repo is the ignition sequence.

Read the middle two rows together. Cudagraphs alone buy 10%; the MTP draft layer buys 78%. An earlier version of this README credited the whole gap to launch overhead, which was wrong, see BENCHMARKS.md.

Quickstart

# 1. get the weights (about 77 GB)
hf download inclusionAI/Ling-3.0-flash-int4 --local-dir "$HOME/models/ling-3.0-flash-int4"

# 2. install the vendor fork (python-level overlay, no CUDA build)
./install.sh

# 3. serve, watchdog-wrapped (see the load-stall wall below)
./serve/watchdog.sh ./serve/serve-int4-mtp.sh

# 4. bench it yourself
python3 bench/bench.py http://127.0.0.1:30000 ling-int4

The serve config is agent-ready as shipped: --tool-call-parser ling3 --enable-auto-tool-choice are on, so tool_choice: auto works. Without the tool parser an agent's call leaks into content and comes back as a 400 or an empty turn.

# override the defaults if you want. leave UTIL alone unless you are dropping
# context too, see wall 4.
CTX=32768 PORT=8000 HOST=0.0.0.0 ./serve/watchdog.sh ./serve/serve-int4-mtp.sh

The four walls, and why they exist

1. Released vLLM silently produces garbage. No released vLLM through v0.27.1 has BailingMoeV3ForCausalLM. If you force the config onto the older BailingMoeV2_5ForCausalLM (the trick most people will try, it loads and serves), you are running KDA weights through lightning-attention math. Wrong equations, fluent-looking token soup. The only correct path on a Spark today is the vendor fork inclusionAI/vllm-ling-v3, branch ling_3_0. Updated Aug 22: V3 and its MTP model are now merged into vLLM main, just not into a release, and there is no aarch64 build of main to install on a Spark. The fork step has an expiry date. Details in FINDINGS.md.

2. Shard loading freezes about half the time. Cold loads randomly stall around shard 7-8 of 24: the engine core pins one CPU at 100%, the log stops moving, and it never recovers. A retry usually loads clean. serve/watchdog.sh detects the frozen log and relaunches automatically, up to 3 cycles. Root cause unfound, suspected weight-processing race on aarch64 unified memory, upstream eyes welcome.

3. The naive config leaves half the speed on the table, and the MTP layer is most of it. --enforce-eager (the usual "safe on new hardware" reflex) plus no speculative decoding gives 20.8 tok/s. Dropping eager buys 10% (22.9). Turning on the MTP draft layer that already ships in the checkpoint (num_nextn_predict_layers: 1) buys another 78% (40.9). Keep num_speculative_tokens: 1, the higher values were tested and are worse. Flags in serve/serve-int4-mtp.sh.

4. A single deep prefill can hard-hang the whole box. At --gpu-memory-utilization 0.80 with long context, the KV pool is sized so aggressively that one large prefill has no allocator scratch left, throws NV_ERR_NO_MEMORY at the driver, and takes the machine down to a hardware-watchdog reboot. The shipped config uses util 0.72 plus chunked prefill (--max-num-batched-tokens 8192) and does not do this. Do not raise util to 0.80 at long context, see FINDINGS.md #6.

What is still on the table

• A GB10 MoE tuning config. The fork ships exactly one tuned MoE kernel config, for the NVIDIA H20-3e, so the Spark runs generic fallback tiles. Now quantified: this model reads 8.40 GB per decode token, and at the 22.9 tok/s no-spec baseline that is 192 GB/s against the box's 273 GB/s, about 70% efficiency. Across five 4-bit builds measured on this box the band is 70-85% and Ling sits at the bottom of it. That gap is what a device_name=NVIDIA_GB10 autotune should recover.
• num_speculative_tokens: 2. Untested. Closed Aug 22: tested, and n=1 was already right. n=2 and n=3 raise acceptance length and lower throughput, freeform drops 13%. See BENCHMARKS.md.
• Long context. 16K tested here. Closed Aug 22: 131,072 verified serving (KV pool 1,128,214 tokens, 8.61x concurrency) at util 0.72 with chunked prefill. Not at util 0.80, which hard-hangs the box, see wall 4.
• DSpark. inclusionAI shipped a DSpark drafter on Aug 22 with an SGLang recipe. It cannot be run on Ling-3.0-flash from any public branch today and the blocker is named in FINDINGS.md #9. When that lands it should beat this recipe.
• fp4 and fp8 stay off the table on one box. fp4 is --tp-size 2 by design, fp8 does not fit. int4 is the single-Spark story.

Box protection

Serving 100+ GB models on unified memory means a memory spike can hard-hang the whole machine, not just the process. This repo assumes the protection layer from the sibling repo dgx-spark-laguna: swap, earlyoom, cgroup caps, and a memory-capped launcher. Set that up once, then serve anything.

Credits

• inclusionAI / AntLing for Ling-3.0-flash, the official int4, and the vllm-ling-v3 fork that does the actual work here.
• bloomer010 for the community GGUF ladder and aetherbird for the bailingmoe3-support llama.cpp fork, the comparison path in the benchmarks.

Apache-2.0. Numbers measured Aug 8 2026, re-verified and extended Aug 22 2026. Single DGX Spark, driver 580.x, CUDA 13.0.