Repository navigation
GLM-5.3-Flash on CUDA: let a long prefill wait between chunks while any rank runs hot #339
Description
What happened
On three DGX Sparks (GB10), a 1,038,423-token prompt took one host's hottest ACPI zone to 94.0 °C about six and a half
minutes into its prefill. My watchdog stopped the engine there, because a GB10 under sustained load has powered off
before without a log line. Prefill got faster over the past weeks: 199,652 tokens now fill in 133 s, against 161 s
before. At the same clock, each host now draws 5–6 W more. A wait between requests cannot help, because the heat
builds inside one request.
What I have
An opt-in wait inside the GLM CUDA prefill. With TF_GLM_HEAT_HIGH and TF_GLM_HEAT_LOW unset, nothing is read or
exchanged.
- Before every prompt chunk, each rank reads its hottest thermal zone (
/sys/class/thermal/thermal_zone*/temp,
overridable) and the ranks all-gather that one number. - Above HIGH, every rank waits together, reading again every 2 s, until the gathered maximum is at or below LOW. A
rank decides only from the gathered maximum, so all ranks make the same exchanges and leave the wait together. If
one rank waited alone, the others would sit in the next exchange with their GPUs busy. - The wait changes when a chunk runs, not what it computes. The startup comparison refuses ranks given different
bands. The reply reportsheat_wait_s, and each wait prints a[tensorfold] heat:line.
Measured (three DGX Sparks, nvidia/GLM-5.3-Flash-NVFP4, FP8 latent KV, bands 92/88 °C)
| Without the wait | With the wait | |
|---|---|---|
| 1,038,423-token prompt | stopped at 94.0 °C after about 6.5 min | all three needles found; time to first token 1,264.8 s, 170.1 s of it waiting; hottest reading 92.7 °C |
| 38,960-token prompt, three runs, cooled before each | 1,671.4 / 1,671.0 / 1,675.0 tok/s | 1,668.4 / 1,670.1 / 1,668.5 tok/s |
| Decode check token hashes | — | equal to the runs without the wait |
The three ranks run the branch from #310. The wait itself does not depend on the rank count.
Questions
- Is this wanted upstream, or should it stay in a fork? It mostly matters on small machines with passive cooling.
- Should it live in the GLM family (where I have it), or in the engine for every CUDA family? The reading and the
all-gather are family-free; only the call between prompt chunks sits in the family. - Environment variables, or
tensorfold serveflags?
If it is wanted, I will open one small pull request against main with the receipt.
Written with Weave (My Symbiotic AI)
Activity
ashhart commentedon Oct 6, 2026
Thank you, @Bizuayeu. The 94.0 °C reading and the bands that kept the 1M-token prompt alive make the case. The Python engine is frozen (#286), so this won't land on 0.6.x, and Zig CUDA doesn't serve yet. It belongs in the engine, not a family, since the reading and the all-gather are family-free. The Zig engine already has the spot, the check it makes for a cancel between prompt chunks. When Zig CUDA serving comes, this can go there, with your credit.
ashhart commentedon Oct 7, 2026
Thank you, @Bizuayeu. A long prefill that waits between chunks while any rank is hot is the right policy. A rank that is already hot should not take the next chunk on top of decode.
The Zig tree has no GLM engine, so this wait is not wired. It lands with that engine. The pause is between chunks, it is per rank, and a cool rank does not sit idle for a hot sibling longer than the policy says. Nemotron's own heat bands are a separate small change and do not close this issue. Thank you for writing the rule as a wait rather than a dropped request.
ashhart commentedon Oct 10, 2026
Thank you, @Bizuayeu. Nemotron on CUDA now has heat bands in main (4d64ae9, 8d2033a and fe154e0), with you credited as
co-author. With TF_HEAT_HIGH and TF_HEAT_LOW set, in degrees Celsius, each prompt chunk waits while the hottest
thermal zone is above the high band, reading again every 2 seconds until it is at or under the low one. With
neither set, nothing is read. A cancel ends a wait, the server logs heat_wait_s, and the reply's tokens don't
change: on a GB10 with a written thermal zone, a 2,931-token prompt held while the zone read 95 °C and ran once it
read 80 °C, with the same tokens as a run without bands.
This doesn't close the issue. The GLM version, where every rank waits on the gathered maximum, comes with the native
GLM CUDA engine, and your measurements are the design we'll follow there.
Heat bands: one set of names, a cancel ends the wait, comments on one…
CHANGELOG: ashhart#487 and the CUDA heat bands (ashhart#339)
Bizuayeu commentedon Oct 10, 2026
Thank you, and for the co-author credit on 4d64ae9, 8d2033a and fe154e0. The Nemotron run that held a chunk at 95 °C, ran it at 80 °C and kept the same tokens is exactly the property I wanted the wait to have.
On the GLM side: as claimed in #548, GLM-5.3-Flash now runs on native CUDA on an unpublished branch on 1.0.2 (TP=2 over NCCL, greedy tokens equal to Python 0.6.x with and without MTP drafts). When that family lands, I'll wire the gathered-maximum wait onto the engine-level
TF_HEAT_HIGH/TF_HEAT_LOWnames you chose, not the oldTF_GLM_HEAT_*ones, with a TP=2 receipt first and the 1M-token TP=3 run once multi-rank communication (#552) is in.ashhart commentedon Oct 10, 2026
Thank you, @Bizuayeu. The heat wait landed for Nemotron under
TF_HEAT_HIGHandTF_HEAT_LOWwith your co-author credit, and wiring it into GLM's CUDA prefill with the gathered maximum is listed in #593. Closing in favour of #593.glm: a prompt chunk waits on every rank's hottest zone, and rank 0's …
Heat bands: one set of names, a cancel ends the wait, comments on one…
CHANGELOG: ashhart#487 and the CUDA heat bands (ashhart#339)