cuda-checkpoint fails to throw error when performing CRIU on non-CUDA processes

[xuhongqiang-kernel]

Problem Description

cuda-checkpoint is a CRIU extension tool designed to manage checkpoint/restore (CRIU) operations for CUDA processes. In a recent test, critical issues were observed:

1. The tool successfully executes CRIU operations on an invalid/non-CUDA PID (2716) (no error/warning is thrown, even though 2716 is not a CUDA process and not the target CUDA process);

2. Worse, after executing cuda-checkpoint --toggle --pid 2716, the actual running CUDA process (PID 2715, a counter CUDA application) unexpectedly terminates and disappears (confirmed via nvidia-smi and ps commands).

[sgurfinkel]

Hi xuhongqiang-kernel

Thank your reporting the issue. It is strange that specifying another PID is having an impact on the counter process.
This is happening because cuda-checkpoint uses /proc to establish an IPC channel with the CUDA process.

In your example, 2716 is a thread created in 2715, so it also exists in /proc and can be used to checkpoint the process too.
I didn't realize that threads were listed in /proc like that, but there you go.

I was able to reproduce your issue except for counter terminating after being toggled by cuda-checkpoint. When I tried, everything seemed successful and it could be restored later:

[rst0git]

In the CUDA plugin for CRIU we use the --get-restore-tid option to obtain the PID value for checkpoint/restore.