590 Features

[rst0git]

The cuda-checkpoint tool exposes a new --launch-job option with driver version 590.44.01. However, it is not clear how to use this functionality.

--launch-job|-j <program> [<arg1> ... [<argn>]]
	Initialize a memfd to become a job file, set CUDA_CHECKPOINT_JOB_FILE to point to that memfd
	and launch the specified command.

@sgurfinkel Would it be possible to add an example?

[sgurfinkel]

Yes, I will do a '590 Features' update to the repo soon!

[ZeroExistence]

I recently tested this with Pytorch that has data-loader workers and criu checkpointing works fine now. It works with single GPU SGLang workload as well.

cuda-checkpoint --launch-job python pytorch-training.py

It seems that this flag is creating new memfd file descriptors to any CUDA processes that utilize GPU, I assume.

Hello @sgurfinkel , can you also indicate some side-effect on running GPU processes with cuda-checkpoint --launch-job on your documentation update? It looks to me that it does not really affect any performance and only exposes memfds within the processes specially for multi-process workload like Pytorch with dataloader or SGLang, but I might be missing something.

Thank you very much for releasing this feature with cuda-checkpoint.