data-angel · Oct 10, 2026 · Oct 10, 2026 · Oct 10, 2026
diff --git a/.env.example b/.env.example
@@ -10,21 +10,23 @@ WORKER=you@192.0.2.2
 #MODEL_ROOT=/srv/models/sparkglm
 
 # 4x1m (default): 4 requests x 1M context, the model's full window; the
-# requests share one KV pool (~2.62M tokens with the shard and the carveout at
-# 0.91; a 925K-token needle prompt is found). 4x512k: 4 x 512K. 8x128k: 8 x 128K.
+# requests share one KV pool (~2.61M tokens with the shard and the carveout at
+# 0.91, ~2.36M with the disk prefix cache; a 925K-token needle prompt is found). 4x512k: 4 x 512K. 8x128k: 8 x 128K.
 #PROFILE=4x1m
 
 # Share of each Spark's unified memory for the engine (0.80-0.95, default 0.88).
 # On Sparks that run nothing else, 0.91 roughly doubles the KV pool that holds
-# long contexts and the prefix cache (about 600K to 1.13M tokens). Above 0.91,
+# long contexts and the prefix cache (about 600K to 1.13M tokens unsplit; the
+# KV shard below holds about 1.8x that). Above 0.91,
 # real traffic took our rank 0 below 1 GiB free; keep at least ~4 GB free.
 #GPU_MEMORY_UTILIZATION=0.91
 
 # Keep conversations that fall out of the prefix cache on each Spark's own disk
 # (ext4/xfs, not tmpfs), so a returning conversation is read back instead of
 # prefilled again. Off unless the directory is set. It is created at the same
 # path on both Sparks. The size is in GiB per Spark (16-100, default 48) and
-# must be free there; the default takes about 60K tokens from the KV pool.
+# must be free there; the default takes about 60K tokens from an unsplit KV
+# pool, about 250K from a split one (2.61M to 2.36M at 0.91 with the carveout).
 #PREFIX_CACHE_DIR=/srv/sparkglm/prefix-cache
 #PREFIX_CACHE_GB=48
 
@@ -36,8 +38,8 @@ WORKER=you@192.0.2.2
 
 # The KV cache is split between the two Sparks: each stores the latents of half
 # the blocks, so the shared pool holds about 1.8x the tokens. With the display
-# carveout at 0.91: ~2.62M tokens (1.36M unsplit), lowest free memory 4.5 GB in
-# a four-session 204K fill. On by default, with or without the disk prefix
+# carveout at 0.91: ~2.61M tokens (1.38M unsplit), lowest free memory 4.5 GB in
+# a four-session 204K fill; ~2.36M with the disk prefix cache (4.8 GB). On by default, with or without the disk prefix
 # cache (each Spark spills only its own half). 0 turns it off.
 #KV_SHARD=0
 

diff --git a/README.md b/README.md
@@ -8,7 +8,9 @@ GLM-5.3-Flash on two NVIDIA DGX Sparks, served by the
 [Atlas](https://github.com/Enntity/atlas) inference engine: tensor and expert
 parallelism across both boxes over one ConnectX-7 cable, NVIDIA's NVFP4
 checkpoint, DFlash2 speculative decoding, prefix caching, up to four
-concurrent requests with 512K context each, and an OpenAI-compatible API.
+concurrent requests with the model's full 1M context each, and an
+OpenAI-compatible API. Greedy output is the same whether a request runs alone
+or beside others.
 Multi-turn agents get a cached conversation back: a turn of a 45K-token
 conversation starts in under a second instead of re-reading the whole
 transcript.
@@ -24,9 +26,10 @@ cp .env.example .env      # set WORKER to the other Spark's ssh destination
 
 Measured on two DGX Sparks joined by one 200G cable, with images built from
 a fresh clone. Receipts and caveats:
-[the display carveout's KV placement](results/2026-10-05-carveout-placement/RESULT.md) (this release),
+[this release: KV shard, deterministic verify, faster MoE decode](results/2026-10-09-rc2/RESULT.md),
+[the display carveout's KV placement](results/2026-10-05-carveout-placement/RESULT.md),
 [a faster verify step and cheaper warm turns](results/2026-09-30-decode-step/RESULT.md),
-[RigMark decode, prefill and staggered arrivals on this release](results/2026-10-05-rigmark/RESULT.md),
+[RigMark decode, prefill and staggered arrivals on 2026-10-05](results/2026-10-05-rigmark/RESULT.md),
 [RigMark short-code concurrency](results/2026-09-30-rigmark/RESULT.md),
 [prefix cache on disk](results/2026-09-30-nvme-tier/RESULT.md),
 [exact kernels and the prefix-cache policy](results/2026-09-30-exact-speedups/RESULT.md),
@@ -39,24 +42,28 @@ and [the first Atlas release](results/2026-09-29-atlas-merged/RESULT.md)
 |---|---|---|
 | Multi-turn conversation, 30–48K tokens: time to first token on turns 2+ (median) | **0.67 s** | 17.6 s with caching off |
 | Four concurrent ~204K-token sessions, cached follow-up turns | 12/12 exact answers | |
+| KV pool shared by all requests (0.91, display carveout, disk prefix cache on) | **2.36M tokens**, up to 1M per request | 1.38M, up to 512K, on 2026-10-05 |
+| Same greedy output at 1 and 4 concurrent requests | **4/4 prompts** | 0/4 on 2026-10-09 morning |
 | Replaying a 35K-token prompt | **0.80 s** (14.4 s cold) | |
 | New session sharing a 24K system prompt with earlier ones: time to first token | **15.7 s** | 36.2 s with the cache policy off |
 | Idle session resuming after another session's 17 turns (median) | **2.5 s** | 49.6 s with the cache policy off |
 | Cold 61K / 125K prompt: time to first token | **25.0 / 49.1 s** | 29.2 / 62.5 s two engines ago |
-| Matrix: C1/C2 at 16K and 32K, C4 at 16K, 400 tokens each, cold (sum of walls) | **136.7 s** | vLLM SparkGLM 206.6 s · Mia TensorFold 165.6 s |
-| Staggered C4: four ~16K requests arriving 1 s apart, cold (median of 5) | **49.3 s** | vLLM SparkGLM 70.5 s · Mia TensorFold 58.3 s · Mia EXL3 113.2 s |
-| Single-stream prose decode, 5 × 384 tokens | **40.2–44.2 tok/s** | 32.8–34.5 tok/s on 2026-09-30, before the decode-step work |
-| RigMark decode, code / prose / structured (thinking on, low effort) | **65.7 / 36.3 / 91.6 tok/s** | [RiNGSiDE](https://github.com/othexmr/GLM-5.3-Flash-NVFP4-2x-4x-DGX-Sparks-RiNGSiDE) vLLM TP2 (published) 56.5 / 33.0 / 83.4 |
-| RigMark cold prefill 8K / 32K / 64K: time to first token | **3.28 / 12.35 / 23.71 s** | RiNGSiDE 3.56 / 12.94 / 25.66 s |
-| RigMark staggered arrivals, prefill first: newcomer time to first token at 2 / 4 | **2.86 / 3.51 s** | RiNGSiDE 4.70 / 5.15 s |
-| RigMark short code, 1 / 4 streams, aggregate (2026-09-30 release) | 45.8 / 67.0 tok/s | RiNGSiDE 44.0 at 1 stream; 97.3 at 6 streams (our 4 × 512K profile serves 4 at a time) |
+| Matrix: C1/C2 at 16K and 32K, C4 at 16K, 400 tokens each, cold (sum of walls) | **136.3 s** | vLLM SparkGLM 206.6 s · Mia TensorFold v1.10 166.9 s |
+| Staggered C4: four ~16K requests arriving 1 s apart, cold (median of 5) | **49.9 s** | vLLM SparkGLM 70.5 s · Mia TensorFold v1.10 60.2 s · Mia EXL3 113.2 s |
+| Staggered C4: four ~32K requests arriving 1 s apart, cold (median of 3) | **77.3 s** | Mia TensorFold v1.10 100.1 s |
+| Single-stream decode, 12 prose / 12 code prompts × 256 tokens | **42.9 / 61.7 tok/s** | 41.2 / 61.8 on 2026-10-09 morning; Mia TensorFold v1.10 is faster here ([release notes](results/2026-10-09-rc2/RESULT.md)) |
+| RigMark decode, code / prose / structured (thinking on, low effort) | **65.5 / 36.5 / 88.8 tok/s** | [RiNGSiDE](https://github.com/othexmr/GLM-5.3-Flash-NVFP4-2x-4x-DGX-Sparks-RiNGSiDE) vLLM TP2 (published) 56.5 / 33.0 / 83.4 |
+| RigMark cold prefill 8K / 32K / 64K: time to first token | **3.29 / 12.47 / 23.72 s** | RiNGSiDE 3.56 / 12.94 / 25.66 s |
+| RigMark staggered arrivals, prefill first: newcomer time to first token at 2 / 4 | **2.80 / 3.54 s** | RiNGSiDE 4.70 / 5.15 s |
+| RigMark short code, 1 / 4 streams, aggregate | **47.7 / 69.5 tok/s** | RiNGSiDE 44.0 at 1 stream; 97.3 at 6 streams (our 4 × 1M profile serves 4 at a time) |
 | Returning to a ~209K-token conversation evicted to disk (prefix cache on disk, 48 GB) | **1.2–1.5 s** | about 82 s without it |
 | Strict JSON (`response_format` json_schema), ~2,000-token structured answer | **30–50 s**, schema-valid | 135–155 s with masked serial decode |
 | Strict JSON object with 96 required keys, temperature 0 | **39 s**, 59 tok/s, valid | engine killed (out of memory) two releases ago |
-| Quality probe: arithmetic / two-hop 24K needle | 40/40 · 11/12 | |
+| Quality probe: arithmetic / two-hop 24K needle | 39/40 · 12/12; 491/500 · 24/24 on the 500-item set | every engine configuration we ran scored 484–495/500 |
 
-These are measurements on our pair, not a guarantee for yours. Known gaps and
-open questions are in [docs/LIMITATIONS.md](docs/LIMITATIONS.md).
+These are measurements on our pair, not a guarantee for yours. Rows not re-run
+for this release keep the receipts linked above. Known gaps and open questions
+are in [docs/LIMITATIONS.md](docs/LIMITATIONS.md).
 
 ## Requirements
 
@@ -113,17 +120,23 @@ videos are supported.
 
 **Profiles** (`PROFILE` in `.env`):
 
-- `4x512k` (default): up to four requests with up to 512K context each.
+- `4x1m` (default): up to four requests with up to 1,048,576 tokens each, the
+  model's full window.
+- `4x512k`: up to four requests with up to 512K each.
 - `8x128k`: up to eight requests with up to 128K each, for more concurrent
   short work.
 
-All requests share one FP8-latent KV pool that also holds the prefix cache:
-about 600K tokens at the default `GPU_MEMORY_UTILIZATION` of 0.88, and about
-1.13M at 0.91, which we run on Sparks that run nothing else (about 1.37M with
-the display carveout, below). Above 0.91, real traffic took our rank 0 below
-1 GiB free ([docs/LIMITATIONS.md](docs/LIMITATIONS.md)). Requests that don't
-fit wait for room. Prefix caching keeps the 16 most recent conversations'
-recurrent state warm.
+All requests share one FP8-latent KV pool that also holds the prefix cache.
+The KV cache is split between the two Sparks (`KV_SHARD`, on by default): each
+stores the latents of half the blocks, so the pool holds about 1.8x the tokens,
+and the output is bit-identical to the unsplit pool's. On Sparks that run
+nothing else, at `GPU_MEMORY_UTILIZATION=0.91` with the display carveout and
+the disk prefix cache (both below), the pool is about 2.36M tokens. Four
+concurrent 204K-token sessions left rank 0 at 4.8 GB free at worst. Above
+0.91, real traffic took our rank 0 below 1 GiB free
+([docs/LIMITATIONS.md](docs/LIMITATIONS.md)). Requests that don't fit wait for
+room. Prefix caching keeps the 16 most recent conversations' recurrent state
+warm.
 
 ## Prefix cache on disk (optional)
 
@@ -133,7 +146,11 @@ With `PREFIX_CACHE_DIR` set, each Spark writes what it drops to its own disk
 instead and reads it back when the conversation returns. In one pass on
 2026-09-30, with 25K-token conversations pushed out of the cache, the time to
 first token of turn 2 went from 10.7 s to 0.91 s and the answers were
-byte-identical. Each restore read its blocks in about 75 ms (7.8 GB/s).
+byte-identical. Each restore read its blocks in about 75 ms (7.8 GB/s). With
+the KV shard (the default) each Spark spills and restores only the latents it
+owns: on 2026-10-09 a 25K-token conversation came back in about 40 ms per
+restore (5.2 GB/s), and turns 2-4 after three restores were byte-identical to a
+run that never evicted, token logprobs included.
 
 To turn it on, add to `.env` and rerun `./start.sh`:
 
@@ -144,8 +161,8 @@ PREFIX_CACHE_GB=48                             # GiB per Spark, 16-100 (default
 
 `start.sh` creates `kv/` and `ssm/` in the directory on both Sparks, refuses a
 directory on tmpfs or without that much space free, and mounts it into each
-rank. Half of the size holds KV blocks (6.66 KB per token; 24 GiB holds 3.9M
-tokens, over twice the pool at 0.93). The other half holds the 78 MB
+rank. Half of the size holds KV blocks (6.66 KB per token unsplit, 3.7 KB per
+Spark with the KV shard; 24 GiB holds 3.9M or 6.9M tokens). The other half holds the 78 MB
 recurrent-state snapshots a restore starts from (24 GiB holds 330). A restore
 needs both, so at 25K tokens the even split keeps about 150 conversations with
 two snapshots each.
@@ -157,7 +174,10 @@ What it costs, per Spark:
   slots), plus 640 B for each 16-token block the KV half can hold, which is
   6.2 MiB per GiB. At the default that is 383.8 MiB. On our pair at 0.93 the
   pool lost about 4,000 of its 104,000 blocks, about 60K tokens (4%); at the
-  default 0.88 the same 60K tokens are about 10% of the pool. Each GiB added
+  default 0.88 the same 60K tokens are about 10% of the pool. With the KV
+shard each block costs each Spark about half as much memory, so the same
+reservation takes about twice the tokens: at 0.91 with the carveout the pool
+goes from 163,177 to 147,480 blocks (2.61M to 2.36M tokens). Each GiB added
   to `PREFIX_CACHE_GB` costs about 500 more tokens. The fixed part is most of
   the cost, and 48 is the size we measured.
 - **Disk.** The KV half is reserved when the engine starts; the snapshot half
@@ -234,14 +254,15 @@ Atlas here is [`Enntity/atlas`](https://github.com/Enntity/atlas) branch
 layers:
 
 1. Atlas-Inf `main`.
-2. GLM-5.3-Flash support and optimizations (branch `upstream/glm53-flash`),
-   which we intend to propose to Atlas-Inf after review. It started as
+2. GLM-5.3-Flash support and optimizations (this release:
+   `upstream/glm53-flash-20261009-rc2`), which we intend to propose to
+   Atlas-Inf after review. It started as
    Reiner Schmidt's port
    ([Mango-kid/atlas](https://github.com/Mango-kid/atlas/tree/feat/glm53-dual-spark));
    the engine's `docs/porting/GLM_5_3_FLASH.md` has the history.
 3. SparkGLM-only commits: FlashKDA and native sparse-MLA prefill bridges,
    which rely on libraries built outside the Atlas tree, and the GB10
-   display carveout.
+   display carveout (including where the split KV pools are placed in it).
 
 ## History
 
@@ -269,6 +290,7 @@ code is ours unless [docs/LICENSING.md](docs/LICENSING.md) says otherwise.
 | `ATLAS_GLM_PC_EVICT` | Reederey87's prefix-cache eviction policy ([glm53-flash-exl3-2x-dgx-spark](https://github.com/Reederey87/glm53-flash-exl3-2x-dgx-spark)) |
 | `ATLAS_GLM_PC_BRANCH` | Marconi's branch-point admission (Pan et al., MLSys 2025, [arXiv:2411.19379](https://arxiv.org/abs/2411.19379)) |
 | `ATLAS_GLM_STRICT_SPEC` | vLLM's speculative decoding with structured outputs ([#14702](https://github.com/vllm-project/vllm/pull/14702)) and its reasoning-boundary fix ([#44297](https://github.com/vllm-project/vllm/pull/44297)): per-position grammar masks over the draft window, the bonus row included, with the matcher rolled back after rejected drafts. We added per-rank masking for the vocab-split verify head. |
+| `ATLAS_GLM_CANONICAL_VERIFY` | Batch-invariant kernels, from Thinking Machines Lab's "Defeating Nondeterminism in LLM Inference" (Horace He, 2025): a row's result must not depend on the rows computed beside it. We apply it to every op a DFlash verify row touches, at every verify width. |
 | `DISPLAY_CARVEOUT` | kindling-spark-os's `dispram` ([kindlingai/kindling-spark-os](https://github.com/kindlingai/kindling-spark-os), by mmastrac, coffee-the-dev and adapt-ai-systems; they credit an earlier NVIDIA developer forum post by emihuang). Our implementation was re-derived from the open GPU kernel modules and needs no daemon. |
 
 Measurement: [RigMark](https://github.com/alexellis/rigmark) by Alex Ellis

diff --git a/docs/INSTALL-MANUAL.md b/docs/INSTALL-MANUAL.md
@@ -9,7 +9,7 @@ time. Run steps 1–4 on **both** Sparks with the same `MODEL_ROOT`.
 | Model | [`nvidia/GLM-5.3-Flash-NVFP4`](https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4) @ `423acf37583782c51c142d145aef733d72943d93` (MIT) |
 | Drafter | [`incoai/GLM-5.3-Flash-DFlash2`](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2) @ `7d74cdd881ed7e32c31175984a67823127b66cfe` (CC BY-NC-ND 4.0) |
 | Native dependencies | FlashInfer `8eccd0c1`, CUTLASS `cf064d2e`, FlashKDA ([`install/flash_kda/`](../install/flash_kda/)), Rust 1.93.1, CUDA 13.0 |
-| Profiles | [`install/profiles/`](../install/profiles/): `4x512k` (default) and `8x128k` |
+| Profiles | [`install/profiles/`](../install/profiles/): `4x1m` (what `./start.sh` runs), `4x512k` (the `start-node.sh` default) and `8x128k` |
 
 ## 1. Get the recipe
 
@@ -88,20 +88,30 @@ until curl -sf http://127.0.0.1:8893/health; do sleep 10; done
 the CUDA kernel cache (`~/.cache/atlas-cuda`), and so do the first requests of
 each shape; later starts reuse it. Run one warmup pass before measuring.
 
-**Profiles.** `--profile 8x128k` selects the other shipped profile. A path to a
-JSON file runs your own profile. Use the same profile on both ranks.
+**Profiles.** `start-node.sh` defaults to `4x512k`. `--profile 4x1m` runs what
+`./start.sh` runs (four requests of up to 1,048,576 tokens), and
+`--profile 8x128k` selects the third shipped profile. A path to a JSON file
+runs your own profile. Use the same profile on both ranks.
 
-**Memory share.** `--gpu-memory-utilization 0.91` on both ranks gives a KV
-pool of about 1.13M tokens on Sparks that run nothing else; without it the
-profile's 0.88 gives about 600K. We run 0.91: above it, real traffic took our
-rank 0 below 1 GiB free.
+**Memory share.** `--gpu-memory-utilization 0.91` on both ranks gives an
+unsplit KV pool of about 1.13M tokens on Sparks that run nothing else; without
+it the profile's 0.88 gives about 600K. We run 0.91: above it, real traffic
+took our rank 0 below 1 GiB free.
+
+**KV shard.** `--kv-shard` on both ranks splits the KV cache between the two
+Sparks: each stores the latents of half the blocks, so the pool holds about
+1.8x the tokens, and the output is bit-identical to the unsplit pool's.
+`./start.sh` turns it on by default; `start-node.sh` leaves it off unless you
+pass the flag. At 0.91 with the display carveout the pool is about 2.61M
+tokens, or 2.36M with the disk prefix cache.
 
 **Prefix cache on disk.** `--prefix-cache-dir DIR` (and optionally
 `--prefix-cache-gb 48`) on both ranks keeps evicted prefix-cache entries on
 each node's disk; see the README for what it costs.
 
 **Display memory as KV cache.** `--display-carveout` on both ranks lends each
-Spark's 2 GiB display carveout to the KV cache (about 1.37M tokens at 0.91).
+Spark's 2 GiB display carveout to the KV cache (about 1.38M tokens at 0.91
+unsplit, 2.61M with `--kv-shard`).
 The container starts with `CAP_SYS_ADMIN` to export it, which the server
 drops before it loads anything, and shares the lock directory
 `/run/lock/sparkglm`. It needs a validated driver (580.173.02 or 580.178.04)

diff --git a/docs/LIMITATIONS.md b/docs/LIMITATIONS.md
@@ -25,19 +25,24 @@ What to know before relying on SparkGLM.
 
 ## Capacity and prefix caching
 
-- All requests share one KV pool, which also holds the prefix cache: about
-  600K tokens at the default 0.88 memory utilization, and about 1.13M at 0.91
-  on dedicated Sparks. A request needs pool room for its whole context, so four
-  full 512K requests do not fit at once; later requests wait for room.
+- All requests share one KV pool, which also holds the prefix cache. Unsplit,
+  it is about 600K tokens at the default 0.88 memory utilization and about
+  1.13M at 0.91 on dedicated Sparks. With the KV shard (`./start.sh`'s default)
+  it holds about 1.8x that: 2.61M tokens at 0.91 with the display carveout, or
+  2.36M with the disk prefix cache. A request needs pool room for its whole
+  context, so four full 1M requests do not fit at once; later requests wait
+  for room.
 - At 0.93 on dedicated Sparks the pool is about 1.63M tokens, and rank 0 went
   down to 1.35 GB of available memory at the end of a four-session 204K fill.
 - At 0.92, a Spark sharing the host with other services went down to 3.1 GB of
   free memory. GB10 hosts can hang when memory runs out, so raise the setting
   only on Sparks that run nothing else.
 - Under real mixed traffic at 0.93, rank 0 dropped below our 1 GiB memory guard
   about hourly (2026-10-02 to 10-04), so our production setting is now 0.91.
-  At 0.91 the pool is about 1.13M tokens, or about 1.38M with
-  `DISPLAY_CARVEOUT=1`, which adds KV without using system memory. The GPU
+  At 0.91 the unsplit pool is about 1.13M tokens, or about 1.38M with
+  `DISPLAY_CARVEOUT=1`, which adds KV without using system memory. With the
+  KV shard, the carveout and the disk prefix cache (our production setup),
+  a four-session 204K fill left rank 0 at 4.8 GB free at worst. The GPU
   does not cache the carveout in L2, so only the latent KV pools go there;
   a cacheable mapping that could also take the index buffers is tracked in
   [#32](https://github.com/Enntity/sparkglm/issues/32).