gpu_diag — GPU inference bottleneck isolator

Diagnoses why the same GPU inference workflow (e.g. LTX-Video in ComfyUI) runs slower on one machine than another — for example a power-limited RTX PRO 6000 Blackwell "Max-Q" Workstation card vs. a 600W cloud instance.

It separates the question into the layers that actually matter for a steady, repeatable slowdown:

Layer	What it checks
Power	Enforced power limit vs. board max, % of time pinned at the cap, sw_power_cap throttle flag, sustained-clock decay. (The Max-Q cap is the #1 suspect for a clean ~1.75× gap.)
Thermal	Peak temp, hw/sw_thermal_slowdown, time-to-throttle under a sustained soak.
PCIe	Link gen/width current vs. max under load (matters when ComfyUI offloads weights to system RAM).
Attention	Whether a fast backend (flash-attn / sage / xformers) is installed and which SDPA kernel is fastest. A missing fast kernel alone is ~1.5–2× on DiT video models.
Compute	Sustained bf16 TFLOPS (start vs. end of soak) to see throughput decay.
Software	driver / CUDA / cuDNN / torch / TF32 — recorded so you can diff the two machines.

Requirements

• Python 3.9+
• For full measurements: torch with CUDA and nvidia-smi on PATH.
• No torch? It still runs in telemetry-only mode and the analysis/test suite is pure-Python (works on any box, including AMD/ROCm).

Usage

Run on both machines and save the metrics:

# On the slow (Max-Q) machine:
python gpu_diag.py --json local.json

# On the fast (RunPod) machine:
python gpu_diag.py --json runpod.json

Then diff them — this is what proves the root cause:

python gpu_diag.py --compare local.json runpod.json

The comparison prints a verdict ranked by likely cause of the runtime gap (power-limit gap → sustained-clock gap → attention mismatch → version diffs).

Other options

python gpu_diag.py --duration 120   # longer soak to expose heat-soak throttling
python gpu_diag.py --no-compute     # telemetry only: poll nvidia-smi while you
                                    # run the REAL ComfyUI job in another terminal
python gpu_diag.py --size 12288     # bigger soak matmul (more thermal/power stress)

Telemetry-only against the real workload

The most representative measurement is to watch the actual LTX render:

# terminal 1 — start the diagnostic poller for the length of one render
python gpu_diag.py --no-compute --duration 180 --json local_live.json
# terminal 2 — run your normal ComfyUI LTX job

This captures clock/power/temp/throttle telemetry during your real workload rather than a synthetic matmul.

How to read the result

• [CRIT] power: GPU is power-capped with sw_power_cap and a big clock drop → the Max-Q TGP cap is throttling sustained inference. If power.max > power.limit you can raise it (sudo nvidia-smi -pl <max>); if they're equal the cap is VBIOS-locked (hardware ceiling).
• Attention backend mismatch in --compare → install flash-attn / sage-attention on the slow box and confirm ComfyUI selects it.
• PCIe gen lower under load → relevant only if weights are being offloaded to system RAM.

Tests

python -m pytest test_gpu_diag.py -v

The tests cover the telemetry parsers and the full diagnosis/comparison logic with synthetic and recorded inputs — no GPU required, so they pass on any machine (28 tests).

Design note

The code is split into a measurement layer (talks to the GPU, degrades gracefully) and a pure diagnosis layer (functions over a metrics dict). That split is why the analysis is fully unit-testable without CUDA and why the --compare mode can run anywhere on saved JSON.