From 99787af73e3f8d67520d2a43ea8df5f6fcaa862b Mon Sep 17 00:00:00 2001
From: Bizuayeu <30091157+Bizuayeu@users.noreply.github.com>
Date: Tue, 20 Oct 2026 18:11:40 +0900
Subject: [PATCH 1/2] RUNBOOK: a GPU clock cap for GB10s, with what it cost and
 lowered for GLM-5.3-Flash NVFP4 at TP=2

GB10 hosts are reported to power off under sustained GPU load, and a 2,200 MHz clock cap is the reported workaround.
The page had the heat bands and the CPU pinning but not the cap: how to set it at each boot, what it does not do, and
what it cost and lowered in four serves on v1.1.1.
---
 RUNBOOK.md | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/RUNBOOK.md b/RUNBOOK.md
index 32d9565274..ffaa0b436a 100644
--- a/RUNBOOK.md
+++ b/RUNBOOK.md
@@ -184,6 +184,43 @@ On GLM-5.3-Flash NVFP4, version 1.0.12 plus #1016-#1018, TP=2, with heat bands o
 completed 40/40 replies, peaking at 85.1 C; the unrestricted run stopped at the seventh reply at 94.7 C.
 Decode throughput fell 1.1% and prefill throughput 2.7%. These are the contributor's measurements, not 1.1.0 results.
 
+### GB10 GPU clock cap for long CUDA runs
+
+GB10 hosts are reported to power off under sustained GPU load, leaving no log
+([NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/hard-power-off-under-sustained-gpu-load-at-90w-persists-after-full-platform-firmware-update/378315)).
+The reported workaround is a GPU clock cap at 2,200 MHz
+([tonyd2wild/dgx-spark-hard-poweroff-fix](https://github.com/tonyd2wild/dgx-spark-hard-poweroff-fix)).
+`sudo nvidia-smi -lgc 300,2200` sets the cap and `sudo nvidia-smi -rgc` removes it. To set it at each boot, save this
+unit as `/etc/systemd/system/gb10-clock-cap.service` and run `sudo systemctl enable --now gb10-clock-cap`:
+
+```ini
+[Unit]
+Description=GB10 GPU clock cap
+After=nvidia-persistenced.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/bin/nvidia-smi -lgc 300,2200
+ExecStop=/usr/bin/nvidia-smi -rgc
+
+[Install]
+WantedBy=multi-user.target
+```
+
+Cap every rank's host. Under load the clock then reads 2,200 MHz or under
+(`nvidia-smi --query-gpu=clocks.gr --format=csv`). In the serves below, the hottest thermal zone (`/sys/class/thermal`)
+peaked 8 to 11 C above the GPU's own peak. The cap holds the GPU only: on a capped host the hottest zone reached 94 C
+with the GPU idle (312 MHz, 6.7 W) while the kernel contract's checks ran and nvcc built its fatbins. The cap does not
+replace the bands: without them, the 1,035,295-token prompt above (ranks not pinned) reached 97.7 C under the cap, the
+hottest host's clock held at 2,177 to 2,190 MHz.
+
+On GLM-5.3-Flash NVFP4, version 1.1.1, TP=2, each rank on the efficiency cores, @Bizuayeu's serves with the cap,
+twice without it and with it again: the cap made drafted decode about 1% slower, `"draft": false` decode about 1.5%
+slower and a cold prompt of 9,600 to 38,400 tokens 2 to 5% slower, with the same token ids; each GPU peaked at about
+60 W instead of 89 to 92 W, and the hottest thermal zone at 88.6 to 90.7 C instead of 96.8 to 98.1 C. These are the
+contributor's measurements.
+
 ## Flash Next and paired Metal serving
 
 Flash Next reads its MLX 6-bit/group-32 checkpoint directly and creates its native layouts locally.

From 3bcd9af745b406ec4626f46a1810fd5db584ed4e Mon Sep 17 00:00:00 2001
From: Cursor Agent <cursoragent@cursor.com>
Date: Thu, 22 Oct 2026 00:47:09 +0000
Subject: [PATCH 2/2] =?UTF-8?q?docs:=20GB10=20=E3=81=AE=E3=82=AF=E3=83=AD?=
 =?UTF-8?q?=E3=83=83=E3=82=AF=E4=B8=8A=E9=99=90=E3=81=AF=E8=B2=A0=E8=8D=B7?=
 =?UTF-8?q?=E4=B8=AD=E3=81=AE=E5=AE=9F=E3=82=AF=E3=83=AD=E3=83=83=E3=82=AF?=
 =?UTF-8?q?=E3=81=A7=E7=A2=BA=E8=AA=8D=E3=81=99=E3=82=8B?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

-lgc のあとも Applications Clocks は元の値のままなので、decode 中の clocks.gr を見る。
oneshot の unit はドライバの再読み込み後に掛け直さない。1,800 MHz の報告も添える。

Co-authored-by: GarupanOjisan <GarupanOjisan@users.noreply.github.com>
---
 RUNBOOK.md | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/RUNBOOK.md b/RUNBOOK.md
index ffaa0b436a..1789eb157a 100644
--- a/RUNBOOK.md
+++ b/RUNBOOK.md
@@ -221,6 +221,16 @@ slower and a cold prompt of 9,600 to 38,400 tokens 2 to 5% slower, with the same
 60 W instead of 89 to 92 W, and the hottest thermal zone at 88.6 to 90.7 C instead of 96.8 to 98.1 C. These are the
 contributor's measurements.
 
+`nvidia-smi -q` still shows the old Applications Clocks and Max Clocks after `-lgc`. While a reply decodes, read the
+live clock with `nvidia-smi --query-gpu=clocks.gr,power.draw,temperature.gpu --format=csv -l 2` (about 2,190 MHz in
+the serves above). The oneshot unit does not apply the cap again after a driver reload or a GPU reset; check that
+live clock after either, or reboot so the unit runs.
+
+A lower cap is also reported. On five GB10s serving a model of 125B parameters or larger, an uncapped 2,455 MHz clock
+read 56.0 W, 84 C and 73.4 tok/s; `sudo nvidia-smi -lgc 0,2200` read 40.1 W, 78 C and 72.5 tok/s; and
+`sudo nvidia-smi -lgc 0,1800` read 23.5 W, 68 C and 70.5 tok/s. These are that report's measurements. The unit above
+keeps 2,200 MHz.
+
 ## Flash Next and paired Metal serving
 
 Flash Next reads its MLX 6-bit/group-32 checkpoint directly and creates its native layouts locally.