cuda-checkpoint default restore preserves stale cuDeviceGetPCIBusId cache across cross-node gpu migration; NVML lookups fail as NVML look ups include cudaDeviceGetPciBusId() which returns old id and causes failure post restore

[shubh3794]

Summary

After a restore of a CUDA process onto a different node with different gpu uuids, but same pcie layout, cuDeviceGetPCIBusId(ord) returns the dump-side (source node) bus IDs, while cuDeviceGetUuid_v2(ord) and nvidia-smi both correctly return the restore-side (destination node) physical identities.

This mixed/inconsistent libcuda state breaks downstream consumers that resolve identity by bus ID. Specifically, NCCL's nvmlDeviceGetHandleByPciBusId() fails with Not Found during communicator initialization post restore, because NVML on the restore node has the destination's physical bus IDs, not the dumped ones.

This appears upstream of the issues tracked in #35 and #47 (--device-map) — the failure occurs on the default cuda-checkpoint restore path, with no gpuPairs involved.

Environment:

• Hardware: 8× NVIDIA B200 (180 GB HBM3e), same-SKU on both source and destination nodes (identical PCI layouts 03/13/63/73/83/93/E3/F3:00.0)
• NVIDIA driver: 580.105.08 on both nodes
• cuda-checkpoint tool: 570.86.10 (5976-byte stub binary forwarding via cuGetExportTable)
• CUDA toolkit in container: 12.8
• Framework: PyTorch 2.11, torchrun --nproc_per_node=4, NCCL 2.27.5
• Kubernetes: container partial-GPU allocation (4 of 8 GPUs per pod), device-plugin–driven

Expected

Either:

• (a) cuDeviceGetPCIBusId(ord) returns the restore-side physical bus ID (63/83/E3/F3), consistent with cuDeviceGetUuid_v2 and nvidia-smi; or
• (b) Documentation clearly states that bus IDs are preserved from dump-side (for ordinal-identity semantics), so NCCL / downstream consumers can reconcile.

Per the maintainer comment in #46 ("by default, it should restore GPU 0 to GPU 0"), behavior (a) appears intended. Current behavior is neither (a) nor documented.

Actual

Mixed/inconsistent libcuda state observed. Exact probe output from the restored process:

Ordinal	cuDeviceGetPCIBusId (libcuda)	cuDeviceGetUuid_v2 (libcuda)	nvidia-smi bus_id	nvidia-smi UUID
0	0000:03:00.0 ← OLD	GPU-0aaf5892-... ← NEW	00000000:63:00.0 ← NEW	GPU-0aaf5892-... ← NEW
1	0000:13:00.0 ← OLD	GPU-2e4b37d6-... ← NEW	00000000:83:00.0 ← NEW	GPU-2e4b37d6-... ← NEW
2	0000:63:00.0 ← OLD	GPU-1a91801a-... ← NEW	00000000:E3:00.0 ← NEW	GPU-1a91801a-... ← NEW
3	0000:73:00.0 ← OLD	GPU-cc420f7a-... ← NEW	00000000:F3:00.0 ← NEW	GPU-cc420f7a-... ← NEW

UUID is correctly re-resolved against the kernel; bus ID is not.

NCCL init failure (immediately downstream): FAIL: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:94,
unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.27.5
ncclSystemError: System call (e.g. socket, malloc) or external library call failed
or device error.
Last error:
nvmlDeviceGetHandleByPciBusId() failed: Not Found

Mechanism: NCCL reads the ordinal→bus_id mapping from libcuda (gets 03/13/63/73), then calls nvmlDeviceGetHandleByPciBusId("0000:03:00.0") against the restore-side NVML (which sees 63/83/E3/F3) → Not Found → NCCL abort.

Questions

1. Is this cuDeviceGetPCIBusId behavior intentional (ordinal-preserving restore, surface as OLD bus IDs by design)?
2. If intentional, how should a downstream consumer like NCCL know the libcuda bus-ID view is no longer authoritative?
3. If unintentional: what path refreshes libcuda's bus-ID table post-restore? Is there a public API, or is this a cuda-checkpoint restore-path gap?
4. Does the --device-map fix planned in Cannot get GPU migration to work with CUDA 13 and driver 580 #35/Could not restore on process ID 3406143: "invalid argument" #47 also address this default-path cache staleness, or are they independent issues?

What this means

libcuda has a stale bus-ID cache that dump preserved. For each ordinal, it returns Node A's dump-time bus IDs (03/13/63/73) while simultaneously returning Node B's fresh UUIDs.

This is a mixed/inconsistent libcuda state:

• cuDeviceGetPCIBusId(ord) → process-memory cache from Node A (OLD)
• cuDeviceGetUuid_v2(ord) → kernel query (NEW)
• nvidia-smi (independent process) → kernel query (NEW)

[shubh3794]

@sgurfinkel can you please advice

[sgurfinkel]

@sgurfinkel can you please advice

Yes, let me take a look!

[shubh3794]

Any updates here?

[sgurfinkel]

Any updates here?

I can confirm that NVML does patch PCI IDs upon restore but that CUDA doesn't.
I think the best fix would be for CUDA to patch PCI IDs upon restore as well (at least some of the time) - though this would be a new fix!

[shubh3794]

Yes, I can contribute if needed. I have multiple findings at different layers here. For example: I got around the PCIe bus id issue by using a shim (LD_PRELOAD) to intercept and return correct values but at the end there are multiple fixes needed below that layer. One of those issues is the following:

P2P transport uses cuMemImportFromShareableHandle to import IPC memory handles between ranks. That call internally resolves which physical device the memory belongs to using libcuda's stale bus ID cache. Stale bus IDs → handle can't be mapped to a real device → error 101. This happens deep inside the CUDA driver, below any API you can intercept.
SHM transport has the same class of problem — shared memory regions are tied to specific device identities that go through the same stale cache.

I have concrete evidence and the causal chain that verifies this. Do you know who to loop in to get this fixed or if the fix can be from within this repo?