Repository navigation
GLM on CUDA, GB10: the native engine spins two CPU cores per rank, and long runs of replies overheat the CPU #1033
Description
On DGX Spark-class GB10 hosts serving GLM-5.3-Flash (NVFP4) at TP=2 and TP=3 with the native engine (1.0.12 plus #1016-#1018), the hottest thermal zone during long runs of replies is a CPU zone, not the GPU, and it comes from two engine threads that spin on every rank. The prefill heat wait cannot hold it, since it acts between prompt chunks. Before proposing a change, here are the measurements of three remedies; which direction would you prefer upstream?
What spins. Every rank has two threads at about 100% each: rank 0 while it decodes, the other ranks while they wait for rank 0, between requests too.
- One is mostly user time and stops spinning with
CU_CTX_SCHED_BLOCKING_SYNCon the primary context (it then sits infutexat about 30%): the CUDA synchronization spin. The context sets no scheduling flag today. - The other is mostly system time, starts with the communicator and keeps spinning with blocking sync; by that, most likely NCCL's progress thread.
On a GB10, five of the seven ACPI thermal zones are the CPU's. On the pair, rank 1's host idled at about 70 °C against rank 0's 53 °C, from the waiting spin alone.
The load. Forty requests back to back, each a fresh nonce and about 6,000 prompt tokens of generated Python with a request for a detailed review, up to 1,024 reply tokens, temperature 0, at TP=2. A thermal watch stops the engine at two readings in a row at or above 94 °C.
| TP=2, heat wait 92/88 °C | Every core (as shipped) | Container on the efficiency cores (0-4, 10-14) | CU_CTX_SCHED_BLOCKING_SYNC |
|---|---|---|---|
| The forty replies | stopped at the 7th, 94.7 °C | 40 of 40 | 40 of 40 |
| Hottest zone, rank 0's host | 94.7 °C | 85.1 °C | 93.8 °C |
| Rank 1's host waiting | about 69-70 °C | 56.9-63.2 °C | 68.6-74.0 °C |
| Decode, 512 tokens after a short prompt | 42.45 tok/s | 41.98 (-1.1%) | 37.26 (-12.2%) |
| Prefill, 38,960 tokens | 1,481.6 tok/s | 1,442.0 (-2.7%) | not measured |
| 200K-token prompt | 148.7 s | 152.4 s (+2.5%) | not measured |
| Start | 244 s | 269 s | 244 s |
The same binary without the flag decoded at 42.25 tok/s, so the -12% is the flag, not the build.
The same holds for other loads. tool-eval-bench's 69 scenarios back to back at TP=2 took rank 0's host to 96.1 °C on every core, where the watch stopped the engine, and to 85.7 °C on the efficiency cores. At TP=3 on the efficiency cores, the forty replies peaked at 79.7 °C, and a 1M-token prompt at 88.9 °C, with no heat wait (TP=3 on every core not measured with these loads).
Lower heat bands do not reach it: 86/82 °C instead of 92/88 °C made the prompts that waited 26 to 89% slower (a 1M-token prompt at TP=3 took 1,910.6 s instead of 1,059.8 s) without lowering that prompt's peak, and decode is unchanged by any band.
Possible directions, each as small as we can make it:
- Documentation only: recommend pinning the engine to the efficiency cores on GB10 (
docker run --cpuset-cpus, ortaskset), with the numbers above. Nothing changes in the engine. This is what we ship downstream. - An opt-in sleeping wait (
CU_CTX_SCHED_BLOCKING_SYNCby an environment variable, about 10 lines in the CUDA context): a 12% decode cost, and the NCCL thread still spins. - The engine picks its cores on hosts with two kinds of core: the same effect as 1 without the operator's step, at the cost of a policy in the engine.
- A heat check between decode rounds, as the prefill has between chunks. Not measured.
These were measured on 1.0.12 plus #1016-#1018. 1.1.0's CUDA context still sets no scheduling flag; we have not measured 1.1.0, whose NCCL communicators changed. We can measure 1.1.x, TP=3 or another remedy the same way before any PR, and will follow the direction you choose.
Activity
ashhart commentedon Oct 18, 2026
Thanks @Bizuayeu. In your 1.0.12 plus #1016-#1018 TP=2 runs, efficiency-core pinning completed 40/40 replies at 85.1 °C, while the unrestricted run stopped at request seven at 94.7 °C; decode changed from 42.45 to 41.98 tok/s. Blocking sync reached 37.26 tok/s, a 12.2% loss, while the second thread kept spinning.
We reproduced it on a 1.1.0 GLM pair. Only the waiting rank spins while idle: its main thread in a CUDA wait on the receive it has posted for rank 0's next step, and NCCL's progress thread yielding in a loop. Rank 0 stays quiet, and rank 1 idles about 10 °C warmer.
Our direction keeps decode speed: while no round is in flight, the waiting rank blocks on a host message from rank 0 and posts its receive only when a step is coming, so both threads go quiet; inside rounds nothing changes. We'd welcome a PR on main for that, with your heat runs and a decode A/B. Meanwhile we'll document efficiency-core pinning on GB10, and blocking sync stays off by default.
Posted by Ash's AI Assistant Synthena
Document efficiency-core pinning for long CUDA runs
ashhart commentedon Oct 19, 2026
Thanks @Bizuayeu. 1.1.1's runbook now documents GB10 efficiency-core pinning: prefix each rank's existing launch with
taskset -c 0-4,10-14, or add--cpuset-cpus=0-4,10-14to its Docker launch. Check the CPU layout on each host and keep the existing model, GPU, network and rank options. The documented heat and speed results are your 1.0.12 plus #1016-#1018 measurements. This is the documented workaround; the engine change to stop idle spinning is still separate. Keeping this issue open for that PR.Posted by Ash's AI Assistant Synthena
glm: the waiting ranks sleep on their lines while rank 0 has no stream
Document efficiency-core pinning for long CUDA runs
glm: the waiting ranks sleep on their lines while rank 0 has no stream
ashhart commentedon Oct 21, 2026
Thank you for the report. Version 1.1.2 lets GLM CUDA waiting ranks sleep while rank 0 has no stream (#1053). Please retest idle CPU use and wake-up behavior on the same setup; the documented CPU-pinning results retain their earlier release scope.
- Synthema, Ash's AI Assistant.
Bizuayeu commentedon Oct 22, 2026
#1033 (comment)
Retested on 1.1.2, on the pair of #1053's receipt and at TP=3: while no request runs, no rank spins, and requests after the idle windows were answered, at TP=2 with v1.1.1's tokens.
The tree is v1.1.2 (
4ff78441) with #1050 rebased onto it and one local commit for our ring's NCCL id, as in the #1067 retest; GLM-5.3-Flash NVFP4 (423acf37), drafts on,TF_HEAT_HIGH=92andTF_HEAT_LOW=88,--parallel 1.TP=2, the pair of #1053's receipt (two GB10s over a direct ConnectX-7 link),
--context 245760, every core (no pinning), the idle windows in that receipt's order. The engine's threads above 5% CPU, 60 s in and 10 s before each window's end, and the hottest ACPI zone's mean over the window's last 60 s:The two columns were measured on different days, so the temperatures are each day's; the thread counts are the point.
Waking: after the 120 s window, drafted output equalled
"draft": falseon four prompts, one with an image. After the 600 s window, the decode check (a counting, a prose and a code prompt, three greedy replies each, the second and third resumed from the prompt cache) gave v1.1.1's token ids, and its first request (2,066 prompt tokens, cold) had its first token in 1.548 s; a launch on the efficiency cores whose requests came between short waits took 1.565 s. The resumed requests took 0.18 s to their first token in both.TP=3, three GB10s over a switchless ring,
--context 1048576, the efficiency cores (0-4, 10-14), 1.1.2's default path (noTF_GLM_MM). The engine's threads were sampled for 10 s every 2 minutes through our whole acceptance window, 5.5 hours and 167 samples a host. In the 132 samples taken between requests, no host's engine used more than 0.8% CPU in all. The 35 taken during requests show each rank's two threads at about 100% in the rounds, as in the report. Before each stage the ranks idled while rank 0's host cooled (up to about 15 minutes), and each stage's first request after it was answered: the decode check, drafted against"draft": false(one prompt with an image), prompts of up to 1,035,295 tokens, and 40 requests back to back. A second launch gave the same token ids.Thank you for carrying #1053 into 1.1.2; closing this issue as completed.