Fails with "out of memory" on MIG-enabled GPUs

[ayushr2]

When MIG is enabled on a GPU, running the basic counter demo also fails.

Repro:

• I tested on a GCE VM with A100-40GB
• nvidia-smi -i 0 --query-gpu=pci.bus_id,mig.mode.current --format=csv to check if MIG is enabled
• Enable it with sudo nvidia-smi -i 0 -mig 1
• sudo nvidia-smi --gpu-reset
• Reboot VM
• Create MIG slice and compute instance: sudo nvidia-smi mig -cgi 14 -C
• Run the counter demo: https://github.com/NVIDIA/cuda-checkpoint/blob/main/src/example.sh
• It fails for me with the following logs:

$ ./counter &
[3] 2026

$ echo hello | nc -u localhost 10000 -w 1
101

$ echo hello | nc -u localhost 10000 -w 1
102

$ /usr/local/cuda/bin/cuda-checkpoint --action lock --pid 2026

$ /usr/local/cuda/bin/cuda-checkpoint --action checkpoint --pid 2026
Could not checkpoint on process ID 2026: "out of memory"

[sgurfinkel]

Hi ayushr2

Yes, that makes sense. Unfortunately, MIG isn't supported right now.
How much of an issue is that for your use case?

[ayushr2]

We'd like to support MIG-enabled GPUs and this is fairly high priority for us. How feasible is this? I have opened an NvOnline bug for this as well.

[lehuannhatrang]

Does anyone have any updates regarding this issue? I am aiming to support the live migration of MIG slices within my system.
Thank you.

[rst0git]

@lehuannhatrang AFAIK the response above is the latest update - checkpoint/restore with MIG is not supported yet. It would be great to learn more about your use case. Please join one of the upcoming Kubernetes Checkpoint/Restore working group meetings (https://criu.org/Kubernetes).