Killing a cuda-checkpoint restored process hard-hangs the whole host (bare metal, open 595.71.05) — suspending first avoids it

[imlach]

We run vLLM workers that get checkpointed with cuda-checkpoint + CRIU and
restored later (scale-to-zero with warm restores). The restores themselves are
fine - a restored process serves inference for as long as we leave it running.
The problem is killing it: terminate a live restored process and ~35 seconds
later the whole machine is gone. Nothing on the console, nothing in pstore, no
Xid or NVRM output beforehand, and the BMC (iDRAC) only logs an OS-side reset
with no hardware fault. SSH drops, the console freezes, and only a power cycle
brings it back. It takes an unrelated CUDA context on the second GPU down with
it. 4 out of 4 attempts so far, so it's deterministic for us.

The dump side never does this - after a checkpoint the process is left
suspended and you can kill it freely. It's specifically teardown of a live
restored CUDA context.

Workaround

Suspending the CUDA state again before the kill makes teardown completely safe:

cuda-checkpoint --action lock --pid <pid> --timeout 30000
cuda-checkpoint --action checkpoint --pid <pid>
# verify with nvidia-smi: the pid's device memory should drop to ~0
# (don't trust the exit code alone)

then SIGTERM/SIGKILL as normal. We've productionized this as a Kubernetes
preStop hook and haven't lost a node since - but it only covers orderly
terminations. A segfault or OOM-kill of a live restored process would
presumably still take the host, which is why we're reporting rather than just
living with the hook.

Environment

• Driver: open kernel modules 595.71.05 (Talos system extension nvidia-open-gpu-kernel-modules-production 595.71.05-v1.13.3)
• Container toolkit: 595.71.05 / v1.19.0
• Kernel: 6.18.33 (Talos Linux v1.13.3; KSPP-hardened: init_on_alloc=1, slab_nomerge, pti=on)
• GPUs: RTX A6000 (GA102, 48 GB) + RTX PRO 4000 (Blackwell) in one host; the restored process ran on the PRO 4000
• Host: Dell R740, bare metal (no hypervisor)
• Runtime: containerd 2.2.4 / Kubernetes v1.36.1; restored workload is a vLLM EngineCore process in a pod
• Checkpoint stack: CRIU + cuda-checkpoint (build with --action lock|checkpoint|restore|unlock)

Repro

1. Start a vLLM server (CUDA process) in a container; lock + checkpoint the
   engine pid with cuda-checkpoint, CRIU-dump the tree.
2. CRIU-restore on the same host, cuda-checkpoint restore + unlock. Process
   resumes and serves normally.
3. Leave it running, seconds or 10+ minutes, duration doesn't matter.
4. Kill the restored process (SIGTERM via container teardown, or SIGKILL
   directly).
5. ~35 s later the host is frozen. Power cycle required.

Insert the lock + checkpoint between 3 and 4 and it's clean every time.

Why we don't think it's our setup

The identical checkpoint/restore/teardown flow ran for months on the same
physical cards under VFIO passthrough VMs on an earlier driver line without a
single hang but a guest never sees real PCIe reset/AER semantics, the VMM
virtualizes all of that away. Bare metal + open 595 + kernel 6.18 is the new
combination. The total silence of the failure (no oops, no MCE, not even an
AER line) looks like the restored context's cleanup path wedging the bus
rather than faulting cleanly.

Possibly relevant: all four repros had the restored process on the Blackwell
card (RTX PRO 4000). We haven't sacrificed a host to find out whether Ampere
does it too, but the signature lines up with NVIDIA/open-gpu-kernel-modules#1111 — silent GSP halt on a
Blackwell RTX Pro 6000, zero diagnostics, power-cycle only, where the reporter
also notes the same workload is stable on Ampere. Different trigger (sustained
inference, no checkpointing), same smell.

Closest existing reports we could find (nothing covers this trigger):

• GSP firmware halt on sm_120 (RTX Pro 6000 Blackwell) under sustained zero-gap llama.cpp inference — silent hard hang open-gpu-kernel-modules#1111 - silent GSP firmware halt, Blackwell,
  open modules; sustained inference, no checkpoint/restore involved
• cuda-checkpoint cuda-checkpoint toggle program does not respond when restoring #17 / Restore permanently fails after initial OOM — process stuck in CHECKPOINTED state #44 - restore-path problems, but process-level
  only; nothing takes the host down

One caveat: we're on the open kernel modules and haven't tried the
proprietary 595 build on this box, so we can't yet say whether it's
open-module specific. Filing here rather than the kernel-modules tracker for
that reason.

Happy to grab nvidia-bug-report output from the healthy state (the freeze
itself logs nothing, so a post-incident report only shows a clean reboot),
share the exact CRIU/cuda-checkpoint invocations, or test instrumented
builds, the repro is quick and deterministic on our side.

[imlach]

Recurred on the same host about 5 hours after filing, and it's the failure mode we flagged at the end of the report (the one the suspend-before-kill workaround can't cover).

This time a restored worker came up, passed validation, served /health 200, and started a real inference request then approx 4 seconds in, while live on the Blackwell card (RTX PRO 4000), its CUDA context took a fatal error mid-request (vLLM surfaced it as an EngineCore … encountered a fatal error, with the usual CUDA_LAUNCH_BLOCKING=1 hint and no Xid/NVRM line beforehand). 20–30 s later the host was gone, same silent signature as before: SSH drops, console freezes, nothing in pstore, iDRAC logs only an OS-side reset. Because the process died on its own rather than via an orderly pod termination, our preStop lock+checkpoint hook never executed - exactly the "segfault or OOM-kill of a live restored process would presumably still take the host" case we called out. So the hook protects orderly scale-down, but a live restored context that faults under load still wedges the box.

Timeline (UTC):

00:23:05 CRIU restore completes, process live, serving /health 200
00:23:09 live restored context takes a fatal CUDA error mid-inference (real request in flight); NCCL HeartbeatMonitor had started logging TCPStore … Broken pipe ~4 s earlier
~00:23:30–00:24:00 host goes silent (unreachable)
00:25:13 kernel boots — same kernel/driver, no upgrade

One new detail vs our manual repros: recovery was automatic and fast (~110 s wedge-to-boot) rather than needing a hand-driven power cycle, consistent with iDRAC resetting the box on the OS hang, which lines up with the "BMC logs an OS-side reset" we noted.

Additional context: another worker restored on an RTX 3090 (Ampere) in the same cluster at the same minute, from a byte-identical image and the same CRIU/cuda-checkpoint path, served the equivalent request fine and never wedged. Same software on both but only the Blackwell card goes down. That continues to point at the Blackwell + open-595 + bare-metal-reset combination rather than the workload or the checkpoint tooling.

Environment unchanged from the original report (open 595.71.05, toolkit v1.19.0, kernel 6.18.33 / Talos v1.13.3, CRIU + cuda-checkpoint, vLLM EngineCore in a pod on a Dell R740, bare metal). Still happy to run instrumented builds or capture an nvidia-bug-report from the healthy state. Though as before, the freeze itself logs nothing, so a post-incident report only shows the clean reboot.