[Bizuayeu]

What this changes

A subsection in RUNBOOK.md's GB10 notes, after the CPU pinning one: how to cap the GPU clock at 2,200 MHz at each boot,
why (GB10 hosts are reported to power off under sustained load without a log, and the cap is the reported workaround),
and what the cap did for GLM-5.3-Flash NVFP4 at TP=2 on v1.1.1: drafted decode about 1% slower and a cold prompt 2 to 5%
slower, with the same token ids, while each GPU peaked at about 60 W instead of 89 to 92 W and the hottest thermal zone
at 88.6 to 90.7 C instead of 96.8 to 98.1 C. It also says what the cap does not do: it holds the GPU only, and it does
not replace the heat bands. The page has the heat bands and the pinning, but not the cap.

Receipt

Documentation only; the subsection rests on these four serves.

• Environment: plain v1.1.1 (b10231b9) built from source with Zig 0.17.0 and nvcc 13.0 for sm_121. Two MSI EdgeXpert
  (GB10, DGX Spark-compatible) at TP=2 over a direct RoCE link, each rank's container on the efficiency cores
  (--cpuset-cpus=0-4,10-14). nvidia/GLM-5.3-Flash-NVFP4 at 423acf37, served with --parallel 4 --context 40960,
  drafts on, no heat bands; each host's page cache of the checkpoint dropped before each serve.

• The runs: cap, no cap, no cap, cap, in one afternoon on the same binary and hosts, with at least 3 minutes of rest
  after each serve. The cap was switched between runs with systemctl stop and start of gb10-clock-cap on both
  hosts. Each serve ran tools/bench_openai.py (256 tokens, 5 reps, medians) with drafts; the same requests with
  "draft": false added to each body, greedy only (the script has no such option, so a wrapper adds it:
  openai_aligned.py nodraft in glm/cuda: load and serve nvidia/GLM-5.3-Flash-FP4-Local as stored #1050's receipt); and tools/bench_cache.py --require-cold on an 18,789-token prompt
  and on 9,600 and 38,400 token ids (the first cold request's ttft_s). Telemetry every 2 s on both hosts:
  nvidia-smi's GPU clock, power and temperature, and the hottest /sys/class/thermal zone; the busy median is over the
  samples at 50% utilization or more. Two values are rank 0's host and rank 1's.

  	cap	no cap	no cap	cap
  drafted decode, fibonacci-raw, greedy / sampled (tok/s)	55.9 / 35.9	56.5 / 36.1	56.4 / 36.2	56.1 / 35.9
  drafted decode, gpu-chat-no-think, greedy / sampled (tok/s)	30.5 / 28.2	30.8 / 28.5	30.8 / 28.5	30.6 / 28.3
  "draft": false decode, both prompts (tok/s)	18.3	18.6	18.6	18.3
  cold prompt, 9,600 / 18,789 / 38,400 tokens (s)	7.22 / 14.13 / 28.42	6.90 / 13.72 / 27.80	6.90 / 13.71 / 27.75	7.21 / 14.13 / 28.35
  GPU clock, top / busy median (MHz)	2,190 / 2,190	2,509, 2,541 / 2,489, 2,496	2,509, 2,541 / 2,489, 2,496	2,190 / 2,190
  GPU power, top (W)	59.7, 60.4	88.8, 92.0	89.2, 92.1	59.7, 60.7
  GPU temperature, top (C)	81, 81	87, 87	87, 87	82, 81
  hottest thermal zone, top (C)	90.7, 88.9	97.9, 96.8	98.1, 97.3	90.7, 88.6

  The replies were the same in all four serves: the greedy replies of each prompt with drafts and with "draft": false
  (f1f736f680e3 and 0c5820aa8286), the cold prompts' replies (20fff53e1cb7, 6cbdb0dff1df and 954ab9447bbd),
  and the sampled replies, rep by rep. The cold prompt row is each serve's first cold request; over the next three
  requests of the 9,600 and 38,400 token ids, the serves without the cap took up to 7.04 s and 28.21 s, and the serves
  with it stayed within 0.5% of their first. The 94 C on a capped host with the GPU idle is from the same telemetry that
  morning, while the kernel contract's checks ran and nvcc built its fatbins on that host. The hottest host's clock in
  the 1,035,295-token run is from RUNBOOK: what the heat bands did on three GB10s, GLM-5.3-Flash NVFP4 at TP=3 with a 1M-token prompt #988; the cap was on all three hosts.

• Not run, and why: other families, TP=3 and longer prompts (the subsection names the case measured); other cap values
  (2,200 MHz is the reported workaround); serves longer than about 15 minutes (the peaks above are of 15 minutes each);
  a serve with the cap on one host only (the subsection caps every host); a power-off itself (the subsection says what
  the cap lowers, not that it prevents one).

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off (documentation only).
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (documentation only; the replies were the same with and without the cap).
• No precision traded for speed. If the bits change, the description says which sums change and why (documentation only).
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt (no kernel changes: documentation only).
• Prompt processing is no slower than the last release (documentation only; the cap is opt-in, and the subsection gives its cost).
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not (CUDA on GB10s).
• New tests fail before the change, pass after it, and skip cleanly without their dependency (no tests: documentation only).
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[SvangenStudios]

@Bizuayeu thanks for documenting this and including the measured performance cost. Some complementary GB10 observations from our deployment:

On one node serving DeepSeek with vLLM in August, busy-sample telemetry (GPU utilisation >50%, averaging about 95% in each window) showed:

	Uncapped (~2,400 MHz)	2,100 MHz cap	1,700 MHz cap
Mean GPU power	54.2 W	32.3 W	19.5 W
Mean GPU temperature (max)	77.9 °C (90)	69.9 °C (80)	64.5 °C (68)
Mean hottest thermal-zone temperature (max)	90.6 °C (98)	86.7 °C (94)	86.8 °C (95)

These were different live-serving windows with mixed workloads, not a controlled cap benchmark. They do not establish the throughput or TTFT cost of either cap. The thermal-zone readings also do not identify the sensor locations.

Two operational details may be useful additions:

• In our setup, Applications Clocks and Max Clocks in nvidia-smi -q retained their stock values after -lgc. Those fields did not confirm the lock; the under-load clock check already in your subsection did. Separately, with a 2,200 MHz cap on a GB10 pair serving GLM-5.3-Flash, we observed 2,190 MHz at about 33 W during decode.
• A Type=oneshot unit with RemainAfterExit=yes applies the cap at startup but does not monitor whether it remains applied. Its active state alone therefore does not confirm the cap after a driver reload or GPU reset.

[ashhart]

Thank you. This is on main as 6774777, with your authorship: RUNBOOK.md's GB10 notes now cover the 2,200 MHz GPU clock cap, how to set it at boot, and your measurements of what it cost for GLM-5.3-Flash NVFP4 at TP=2, labelled as yours. It ships with the next release. Closing as landed.

- Synthema, Ash's AI Assistant.

[Bizuayeu]

This is on main as 6774777, with your authorship: RUNBOOK.md's GB10 notes now cover the 2,200 MHz GPU clock cap, how to set it at boot, and your measurements of what it cost for GLM-5.3-Flash NVFP4 at TP=2, labelled as yours.

#1110 (comment)

Thank you for landing it.