Skip to content

Living Weights for GLM-5.3-Flash on NVIDIA GB10 (DGX Spark) CUDA — port request + test hardware offer #566

Closed as duplicate

Description

@GraithSecurity

First — Living Weights looks like the most interesting serving-side work this year. The claims in the launch thread (weight updates gated behind /learn, shard growth with a size cap, persistence across unload and machine moves, cracked catastrophic forgetting) are exactly what personal-model serving needs.

We run GLM-5.3-Flash as our primary pair-served model on two NVIDIA DGX Spark (GB10, sm_121) boxes via TensorFold (stock upstream, currently the 0.6.x Python line at v0.6.6 in Docker — the operator-mandated primary lane since 2026-10-01). LW ships with Nemotron Lightning, you mentioned Qwen3.6-35B-A3B works but is gated for a few days. Questions:

  1. GLM-5.3 on the native CUDA server: 1.0.x qualifies GLM-5.3 on two M5 Ultras (Metal). Is GLM-5.3 on GB10 CUDA on the LW/native roadmap? If it needs a port, what does the model gate look like (which measurements must pass)?
  2. Living Weights on quantized checkpoints: our serving checkpoint is a 4bpw TR3/EXL3-class mixed-bit GLM quant. Your thread says 4-bit recall "seems really good" — does LW's shard-growth mechanism operate on quantized tensors in place, or does it need a BF16/native-format base? What happens to a mixed-bit checkpoint?
  3. Speculative decoding interaction: our GLM lane serves with DFlash2+MTP drafts and verifies reply-equality against plain decoding. Does LW updating live weights break draft/verify equality guarantees, and is draft acceptance re-measured after learned updates?
  4. Rollback: is learned state separable from base weights (drop the grown shard / revert to the frozen checkpoint)? Is there a per-/learn-call audit trail (what was written, where)?
  5. Cost: what's the per-/learn update cost on CUDA GB10 for a ~30B-class MoE at 4bpw (wall time, memory), and does serving pause during an update?

Happy to give the GLM-on-GB10 native port a real test target: the pair is on my desk, we have py-spy/nccl/pytest harnesses and a round-trip verification lane already built. If a CUDA GLM branch exists behind any door, point me at it.

Activity

  1. ashhart commented on Oct 10, 2026

    @ashhart
    Owner

    Thank you, @GraithSecurity, for the careful questions and for offering your GB10 pair. Here is where 1.0.4 stands.

    1. GLM-5.3-Flash is not on the native CUDA server yet. Native CUDA serves Nemotron 3.5 Lightning only. A GLM-5.3-Flash port to the native CUDA engine is in progress, and its gate is the one every family passes: drafted equal to "draft": false, resumed equal to fresh, concurrent equal to solo, and prefill and decode measured against the current release. We will post here when a build is ready for your pair.
    2. Living Weights learns into one model today, Nemotron 3.5 Lightning in MLX 4-bit, and only on Apple-silicon Macs. Each kept fact changes the output projection, written into the checkpoint's safetensors shards in bf16. It has not been built for EXL3 or other mixed-bit checkpoints.
    3. A learned folder is an ordinary checkpoint to the server, so drafts are verified against the same weights they came from and the exactness contract holds on them. We have not published draft acceptance before and after learning.
    4. There is no per-fact undo. The guide says to teach a copy and keep the original, and a reset is a fresh copy. --slide-graph FILE keeps the list of learned facts, and GET /v1/slide/graph returns it.
    5. On a Mac a fact takes two to four minutes, and learning runs only when no reply is being generated, so the server keeps answering. NVIDIA GPUs cannot learn yet, so there is no GB10 cost to quote.

    docs/living-weights.md has the details. We will keep this issue as the request for Living Weights on CUDA and on GLM.

  2. ashhart commented on Oct 10, 2026

    @ashhart
    Owner

    Thank you, @GraithSecurity. GLM-5.3-Flash on native CUDA is now tracked in #593, and Living Weights on CUDA is coming through #591, for Nemotron on a GB10 first. Living Weights for GLM-5.3-Flash on CUDA is listed in #593 as a later piece, after both. Your GB10 pair offer is welcome there when a build needs a second gate. Closing in favour of #593.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions