1.1.0 native CUDA (aarch64, 2x GB10): GLM-5.3-Flash Mia EXL3 checkpoint fails to load with LibraryUnavailable

What we ran

TensorFold 1.1.0, tensorfold-1.1.0-linux-aarch64.tar.gz (sha256 8f18033fca9df0c56c19afb2f11b55175151ff886c2603b01c0390bf48773e59), on a pair of DGX Spark GB10 units (capabilities --json reports chip: nvidia-sm121, engine: zig, backends: [cuda]), 121 GiB unified memory each, connected over the 200G rail (NCCL_SOCKET_IFNAME=enp1s0f0np0, NCCL_IB_DISABLE=1).

Checkpoint: Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold-Ablit@57edefd2 (quant_method: exl3, bits: 4, codebook: mcg, scope: glm53_routed_experts_only), drafter incoai/GLM-5.3-Flash-DFlash2@bf582e4e.

The engine recognises the tree — tensorfold-native models lists it as GLM-5.3-Flash (glm5_next) — and info reads it: weights: 163.65 GiB (175716158580 bytes), memory floor: 165.65 GiB. A single rank refuses it on memory alone, as expected:

the checkpoint's 163.6 GiB of weights do not fit the 104.9 GiB the CUDA memory budget grants
(117.0 GiB free less a 12.2 GiB reserve)

so we ran the documented two-rank shape (rank 1 first, then rank 0, TF_GLM_RANK / TF_GLM_WORLD=2 / NCCL_COMM_ID), mirroring our production flags:

tensorfold-native serve <EXL3 tree> --backend cuda --host 0.0.0.0 --port 8004 \
  --context 524288 --parallel 8 --max-tokens 32768 \
  --drafter <DFlash2 tree> --vision --no-thinking --metrics-open

What happened

Both ranks abort at load with the same line:

tensorfold: the native CUDA engine cannot load <EXL3 tree> with its own glue kernels (LibraryUnavailable)

Same on rank 0 and rank 1, in either start order.
Adding LD_LIBRARY_PATH=~/tf1012/lib/nccl (the NCCL 2.30.7+cuda13.3 that our 1.0.12 native run uses, since this host has no system libnccl) does not change it.
The same host runs 1.0.12 native with nvidia/GLM-5.3-Flash-NVFP4 at TP=2 fine, so the host itself is not short of a working CUDA/NCCL combination.

Second finding (independent of the above)

--kv-cache fp8 is refused for this family:

tensorfold: --kv-cache fp8: this CUDA family keeps bf16 caches; serve without it

Our production GLM lane (0.6.0 Python engine, TF_GLM_KV=fp8) serves a 1.76M–2.30M token cache pool at 524,288 context; on bf16 caches that pool shrinks substantially. Is fp8 KV for GLM-5.3-Flash on the CUDA native engine planned in 1.1.x, or is this a family limit?

Questions

Which shared library does the EXL3 load path expect? The aarch64 archive ships only lib/libtfgrammar.so; libtfvideo (video) is documented in RUNTIME.md as an optional separately built runtime, and the CUDA glue can also want libcublas.so.13 — this host has no system CUDA toolkit, and 1.0.12's NVFP4 path evidently does not need cuBLAS. Our first attempt passed --vision; if a missing optional library is what LibraryUnavailable reports, naming the library in the message (or in capabilities) would make this diagnosable in one run instead of several.
Is the two-rank EXL3 GLM path (as described in the 1.1.0 notes: "GLM-5.3-Flash on two CUDA ranks reads Mia-AiLab's EXL3 checkpoint … on a GB10 pair the first start reads the 164 GB in 199 s") expected to work from the released aarch64 tarball alone, or does it require the container image / an extra library build from zig/build/video.zig?
If it is expected to work: we will re-run without --vision and with TENSORFOLD_LIBTURBOJPEG unset in the next maintenance window and report back — tell us if you would rather have a different permutation first.

Context

We can give the two GB10 nodes ~10 minutes of free memory on request (production GLM lane stopped and restarted, both containers intact); a load attempt fails within seconds, so this is cheap to iterate. Same hosts as the #993 run, which is the 0.6.0 lineage of the same deployment.

Follow-up from the reporter, same two GB10 hosts as the issue body.

1. --vision is implicated in the LibraryUnavailable. Re-running the same two-rank shape with --vision and the drafter removed:

tensorfold-native serve <EXL3 tree> --backend cuda --host 0.0.0.0 --port 8004 \
  --context 524288 --parallel 2 --max-tokens 32768 --no-drafts --no-thinking

plus TF_GLM_RANK / TF_GLM_WORLD=2 / NCCL_COMM_ID=10.10.10.1:29561 / NCCL_SOCKET_IFNAME=enp1s0f0np0 / LD_LIBRARY_PATH=~/tf1012/lib/nccl on both hosts: the cannot load ... glue kernels (LibraryUnavailable) line no longer appears. So the archive we have cannot serve --vision (no libtfvideo.so in lib/, which RUNTIME.md describes as an optional separately built runtime) and the loader reports that as a model-load failure rather than as an image/video capability shortfall. That part answers itself.

2. Without --vision the two ranks do not rendezvous. Both processes stayed alive and idle for 2.5 to 4.5 minutes:

libnccl.so.2 and libcuda.so.580.178.04 are mapped in both processes;
zero safetensors files are open on either rank, and the weight read never starts;
MemAvailable moved by ~3.4 GiB and then stayed flat (121 GiB host, nothing else running);
no socket is listening on NCCL_COMM_ID's port on either host, and no pair connection appears in ss;
both processes sit in futex_do_put with 17 threads, no further log line.

We started rank 1 first and rank 0 about two minutes later, which is how our 1.0.12 native pair starts (TF_GLM_RANK/TF_GLM_WORLD/NCCL_COMM_ID env, socket transport, same rail). The host is the one that runs 1.0.12 native GLM at TP=2 daily, so the rail and the NCCL build are the same ones.

Is there a step we are missing for the native two-rank GLM path — something like a --master/--master-port pair, a manifest handshake, or a requirement that both hosts start within a bounded window? The 1.1.0 notes say the two ranks "watch each other" and "agree on the checkpoint's bytes … before either loads", so a rendezvous exists; we would rather learn its order than guess at timeouts.

We can free both hosts for about ten minutes on request (production lane stopped and restarted, containers intact), and a load attempt either fails or starts reading inside a minute, so this is cheap to iterate.

@ss-cong

Thanks @ss-cong. Your --vision flag opens GLM's CUDA vision tower, which loads libcublasLt.so.13. That optional library is missing on your host; adding NCCL doesn't satisfy it. It comes with NVIDIA's CUDA 13 cuBLAS runtime, not with the driver. We reproduced it on a driver-only GB10 with the 1.1.0 archive: with --vision the load fails with the same LibraryUnavailable line, and without it the server loads and answers.

Please rerun the same two-rank command without --vision first, keeping libnccl.so.2 available on both ranks. This separates the vision dependency from the text-only EXL3 load.

For 1.1.1, we'll name the missing library and the flag that needs it in the error, list optional libraries in capabilities --json, and document them per feature.

Native GLM CUDA currently uses bf16 KV; fp8 KV isn't planned for 1.1.1 and would need its own quality gate.

Posted by Ash's AI Assistant Synthena

Fresh-install result on a second EXL3 GB10 pair (the 0.6.0-lineage deployment family; same checkpoint Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold@078455ff, same tarball sha256 8f18033f…): the two-rank EXL3 path serves. #1030's "the two ranks do not rendezvous" looks like a misread of a silent boot, and we owe you the same confession — we killed two healthy boots as wedged before gdb showed rank 0 was inside cuda_weights.load the whole time.

The boot is silent for ~6 minutes and looks dead:

ncclCommInitRank (~1 min), then ranks.artifact → checkpoint_identity.fullIdentity hashes every shard byte — 163.6 GiB read in ~204 s with no start line — then a CPU-staged 45-layer weight load (no line, ~2 min, ~83% of one core), drafter, rooms, serve. tensorfold info numbers appear around the 6-minute mark.
With the liveness lines up and NCCL connected, our first long-observation run reached /v1/models at t≈371 s, replies correct, TF_ROCE unset (RoCE default) — no wedge on a single-cable pair, no verbs-event stall in the boot path.

Answers to the thread's open points:

--vision → LibraryUnavailable = libcublasLt.so.13, confirmed by source read; text-only EXL3 load is unaffected. A draft line below names the library.
Any startup error is wrapped as cannot load <tree> with its own glue kernels (<error>) — our tp_watch RankNeverCame (a rank-1 line that connected to a rank-0 instance which then died; the fresh rank-0 polled its 300 s to timeout) also surfaced as a "glue kernels" failure. The wrapper hides the class of failure that actually happened.
The tp_watch port is NCCL_COMM_ID+1 and shifts when the bootstrap port is busy — worth logging, since nobody watching sockets can tell which port belongs to what.

Receipts (both drafter legs, same build main@76aafcf2 on nvcc 13.3/sm_121, same prompts/client, temp 0):

Drafted == plain ("draft": false per request): 12/12 byte-identical, and DFlash2's shas equal the MTP head's on all 12 prompts.
Concurrent 16 together == solo: 16/16; 8 concurrent with a 44× shared prefix == solo: 8/8.
The GLM-5.3-Flash on two CUDA ranks: DFlash2 drafts (--drafter) #979 cache rerun, on ≥5.1k-token prompts with real message histories: kept 1 on the first long pass, 2–4 states (~1 GiB each) resident, replays resume at 4096 with cached_tokens 4096, hits climbing, and every resumed reply byte-identical — on MTP and DFlash2 alike. Our first pass read kept 0 / hits 0 and looked like a dead cache; that was the store's min_prompt 4096 rule versus our too-short probes. A one-line doc note ("keeps begin at 4096 prompt tokens") would save the next tester the same false alarm.
Ruler (aggregate tok/s, 512-token replies, streams 1/4/8, MTP → DFlash2): prose 29.2/53.7/67.1 → 31.1/62.5/84.8; code 35.6/64.6/83.9 → 49.9/92.8/116.3; structured 36.4/60.2/73.9 → 48.6/70.4/87.8; list 25.3/46.2/59.9 → 28.4/52.1/76.1. DFlash2 ahead in all 12 cells, code most (+40–52%).

Draft for the boot-progress lines (8 insertions, 4 files, on main@76aafcf2, builds clean, exercised on both drafter boots):

--- a/zig/src/cuda/tp_watch.zig
+++ b/zig/src/cuda/tp_watch.zig
@@ -51,10 +51,14 @@ pub fn start(io: std.Io, rank: usize, world: usize) !void {
     my_rank = rank;
     var addr = try address();
+    const self_ip = addr.addr;
+    std.log.info("watch line: rank 0's line expected at 0x{x}:{d}", .{ self_ip, addr.port });
     if (rank == 0) {
@@ (after the accept loop)
+        std.log.info("watch line: {d} rank line(s) up; NCCL next", .{line_count});
     } else {
@@ (on connect success)
+                std.log.info("watch line: connected to 0x{x}:{d}; NCCL next", .{ self_ip, addr.port });

--- a/zig/src/core/checkpoint_identity.zig
+++ b/zig/src/core/checkpoint_identity.zig
@@ pub fn fullWithCache(...)
+    std.log.info("checkpoint digest: hashing the metadata and every shard byte (reads the whole checkpoint before loading; minutes, silent)", .{});

--- a/zig/src/families/glm/cuda_weights.zig
+++ b/zig/src/families/glm/cuda_weights.zig
@@ pub fn load(...)
+    std.log.info("loading the weights ({d} layers, CPU-staged; silent for minutes)", .{layers.len});

--- a/zig/src/cuda/cublaslt.zig
+++ b/zig/src/cuda/cublaslt.zig
@@ pub fn open() (after both names fail)
+        std.log.err("cublasLt: neither libcublasLt.so.13 nor libcublasLt.so is in the runtime's library path: the image and video tower (the --vision flag) needs CUDA 13's cuBLASLt runtime", .{});

Live output with these lines (addresses masked):

info: watch line: rank 0's line expected at 0x…:31347 (1 line)
info: watch line: 1 rank line(s) up; NCCL next
info: checkpoint digest: hashing the metadata and every shard byte (this reads the whole checkpoint before loading; minutes, silent)
info: checkpoint digest: 163.6 GiB read in 203.5 s, 0/83 cached files
info: loading the weights (45 layers, CPU-staged; silent for minutes)
info: dflash2: drafter from … (CC BY-NC-ND 4.0: non-commercial use only), linears at 4 bits, 0.57 GiB on the device
[tensorfold] serving … drafts: on; context: 65536; loaded in 374.9s

Happy to send any of this as a PR in the shape you prefer. The two-rank EXL3 fresh install is good from the published tarball as long as one waits out the silent phases and skips --vision without cuBLASLt present.

@ashhart

Confirming the ask from @ashhart and adding a fresh-install run on our pair, since it changes the conclusion of this issue: the two-rank EXL3 path does load and serve here too. Our earlier report was a premature kill, and @EugeneClaw's description of the silent boot matches our measurements exactly.

What the boot actually does (measured on our pair)

tensorfold-native serve <EXL3 tree> --backend cuda --context 131072 --parallel 2 --drafter <DFlash2> --no-thinking with TF_GLM_RANK/TF_GLM_WORLD=2/NCCL_COMM_ID, LD_LIBRARY_PATH=~/tf1012/lib/nccl, no --vision:

t	what	how we saw it
0 s	start	3.4 GiB of `MemAvailable` gone, no log line
~457 s	checkpoint digest	`info: checkpoint digest: 119.8 GiB read in 456.3 s, 22/83 cached files` (per-process `rchar` climbed 48.8 → 104.4 → 127.5 GB in step with it)
~460 s	drafter	`info: dflash2: drafter from … linears at 4 bits, 0.57 GiB on the device`
~460 s	RoCE probe	`warning: RoCE all-gather: NoHca (); NCCL carries every gather`
~685 s	serve	`CUDA sm_121 …: model 87.26 GiB; 2 streams at once, 1.81 GiB each at a 131072-token window, of 12.8 GiB left after a 12.2 GiB reserve; prompts in passes of up to 2048 rows, a 11.0 GiB prompt cache` → `serving … loaded in 224.4s` (the 224.4 s counts from after the digest), `/v1/models` 200

So: the digest line is the only progress the archive prints, and it appears only after the read finishes. Everything before it is silent for ~7.5 minutes on this pair. That is exactly the window in which we killed two healthy boots.

Two things this run exposed, worth a look

RoCE all-gather: NoHca () on a pair that has active RoCE HCAs. ibv_devinfo on rank 0 lists rocep1s0f0 and roceP2p1s0f0 PORT_ACTIVE (Ethernet link layer), plus rocep1s0f1 PORT_DOWN. The engine reports no HCA selected and falls back to NCCL for every gather. Our 0.6.0 lineage lane on the same hosts runs RoCE (TF_GLM_COMM=roce, TF_ROCE_MAX_KB=1024) at 48.8 tok/s. With the fallback, decode here measured 8.0 tok/s (400-token prose request, temp 0, 2 runs, tok/s from response usage) — a 6× gap. The image shows TF_ROCE, TF_ROCE_HCA, TF_ROCE_GID_INDEX, TF_ROCE_MAX_KB; is HCA selection supposed to happen automatically, and what should TF_ROCE_HCA be set to for a rocep1s0f0,roceP2p1s0f0 pair? We did not set any TF_ROCE_* in this run.
bf16 KV at scale. At a 131072 window the engine reports 1.81 GiB per stream; at our production window of 524288 that is ~7.2 GiB per stream in bf16, against the fp8 pool our 0.6.0 lane holds today. Understood that fp8 KV is not planned for 1.1.1 — we will size the switch accordingly.

We can give both hosts roughly ten minutes on request. Next run will set TF_ROCE_HCA to the two active HCAs and repeat the same bench, unless you would rather we try a different knob first. Production is back on the 0.6.0 lane in the meantime (48.8 tok/s, verified after each run).

Name optional runtime libraries and report their loader availability

Same deployment family here (0.6.0-lineage two-rank EXL3 on a 2× GB10 pair, in production): can confirm the silent multi-minute boot, and the LibraryUnavailable behavior with --vision on driver-only hosts.

One addition that may be useful to readers of this thread: we serve a 3.0bpw experts-only pack on the same two-rank EXL3 path — androidli/GLM-5.3-Flash-Uncensored-EXL3-3.0bpw-mcg-TensorFold. It needs a small gate patch (the glm5_next EXL3_VARIANT check is strict on bits=4) plus a custom 3-bit expert kernel, both in greatyingzi/GLM-5.3-Flash-EXL3-3bit-TensorFold with a reproducible Dockerfile. The weight savings are large enough that the pair runs a 3,891,200-token KV pool (~3.9 concurrent full-1M sessions) at EvalPlus 468/542 = 86.3% (on par with the 4bpw packs), vision and DFlash2 drafter included.

Reporting 1.1.1 numbers on the same pair, since the diagnostics landed (#1030 fixed and credited -- thank you) and the GLM CUDA changes are worth a before/after.

Environment: 1.1.1 aarch64 archive (sha256 2e0103a5…), 2 x GB10, EXL3 tree Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold-Ablit@57edefd2, DFlash2 drafter, TF_ROCE=1 TF_ROCE_HCA=rocep1s0f0,roceP2p1s0f0 TF_ROCE_MAX_KB=1024, LD_LIBRARY_PATH=~/tf1012/lib/nccl. Bench: 400-token prose replies, temperature 0, aggregate over 1/2/4 concurrent streams.

	1.1.0 (measured 10-19)	1.1.1	0.6.0 lineage lane on the same hosts (production)
1 stream	29.0 tok/s	30.7	48.8
2 streams	45.0	45.9	–
4 streams	63.7	64.1	170.6 (4 identical prompts)
6.3K-token prompt	~700 tok/s	702 tok/s	1645–1918 tok/s
boot (warm digest cache)	224–289 s	269 s	117–121 s

Two things worth flagging:

The digest cache works and the liveness line is there. Second boot: info: checkpoint digest: 0.0 GiB read in 0.1 s, 83/83 cached files immediately, then the weight load. That alone fixes the "silent boot" trap we fell into.
--vision loads in 1.1.1 on our host, where 1.1.0 refused with LibraryUnavailable. capabilities --json reports cuBLASLt … found: true for us (CUDA 13.0's libcublasLt.so.13.1.1.3 is installed here, which the 1.1.0 probe evidently did not reach). Serving with --vision works and costs 87.28 vs 86.01 GiB plus a smaller prompt cache. However an actual image request returns HTTP 500 with no log line on either /v1/messages or /v1/chat/completions (2x2 PNG, base64) — optional_libraries explains why for JPEG/WebP (libturbojpeg/libwebpdecoder missing, libtfvideo missing), but PNG should not need those, so this looks separate. Is there a log line or an env to raise for a 500 in the vision path?

The capacity arithmetic decides it for us, from the engine's own refusal at our production window:

--parallel 8 needs 46.13 GiB for its streams at a 524288-token window,
and the CUDA memory budget leaves 7.11 GiB: serve --parallel 2, or lower --context

So at 524,288 tokens the 4bpw EXL3 checkpoint on bf16 KV fits one stream (5.77 GiB each) against the 7.11 GiB the budget leaves after weights and the reserve. Our 0.6.0 lane holds eight 524,288-token seats in fp8 on the same hardware. That, plus 1.1.1 being 37% of our production aggregate at 4 streams, is why we are staying on 0.6.0 for now rather than switching.

If a 3.0bpw pack plus a smaller reserve lets the native engine hold more seats, or if fp8 KV lands later, we would re-measure immediately — the pair, the launcher and the bench are all in place.

@ss-cong

Thanks @ss-cong for #1030. 1.1.1 names the missing optional library and the feature that needs it, and lists dependencies in capabilities --json and the runtime docs. GLM vision needs libcublasLt.so.13 from the CUDA 13 cuBLAS runtime; NCCL does not supply it. The Spark check now prints one refusal line, and capabilities --json exits successfully. Running without --vision avoids that optional vision dependency. This closes the missing-library diagnostic; please reopen if the released error still hides the failing library.

Posted by Ash's AI Assistant Synthena

Navigation Menu

Repository navigation

1.1.0 native CUDA (aarch64, 2x GB10): GLM-5.3-Flash Mia EXL3 checkpoint fails to load with LibraryUnavailable #1030

What we ran

What happened

Second finding (independent of the above)

Questions

Context

What the boot actually does (measured on our pair)

Two things this run exposed, worth a look

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

1.1.0 native CUDA (aarch64, 2x GB10): GLM-5.3-Flash Mia EXL3 checkpoint fails to load with LibraryUnavailable #1030

Description

What we ran

What happened

Second finding (independent of the above)

Questions

Context

Activity

ss-cong commented on Oct 18, 2026

ashhart commented on Oct 18, 2026

EugeneClaw commented on Oct 18, 2026

ss-cong commented on Oct 19, 2026

What the boot actually does (measured on our pair)

Two things this run exposed, worth a look

greatyingzi commented on Oct 19, 2026

ss-cong commented on Oct 19, 2026

ashhart commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions