plugins/cuda: switch to libcuda checkpoint API - #3076
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## criu-dev #3076 +/- ##
============================================
+ Coverage 57.72% 57.94% +0.22%
============================================
Files 161 162 +1
Lines 43920 44247 +327
Branches 9637 9692 +55
============================================
+ Hits 25351 25641 +290
- Misses 18330 18367 +37
Partials 239 239 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Forwarded this over to Steven as well to take a look as well as some of the other cuda checkpoint devs. 4th of July is a holiday so might not get responses there for a couple days just as a heads up. The value redefinitions of the structs may be a bit brittle in that you may have to carry forward some backwards compatibility that the cuda-checkpoint utility hides away by doing some manual cuda version checks for what features are available in which versions. So instead of checking whether cuda-checkpoint reports a certain flag is supported you'd have to check the cuda version and see which fields of the structs you can use as new features are added and the reserved area carved away. If you don't mind doing that handling here as new features are added that's fine, the API will return CUDA_ERROR_INVALID_VALUE if the reserved fields in that version of CUDA are non-zero if for example you try to use a newly added feature via CRIU in a slightly older driver that doesn't have the feature. One thing to point out though is that while CUDA does have minor version compatibility a major CUDA version number may have revised definitions for the API's or structs that could throw a wrench in things so that's something to keep in mind. I'll take a more thorough pass on the patch though and see if anything else sticks out and let Steven and others weigh in too. |
|
I have a question around the license. CRIU is GPLv2, libcriu LGPL and anything in images/ is MIT licensed. Using Also the libcuda mock library looks like nothing CRIU has in the code base. Where was that taken from? |
|
@jesus-ramos Thank you so much Jesus! Steven actually suggested that we do this some time ago. These changes also make it easier to deploy the CUDA plugin for CRIU in Kubernetes clusters. We presented some preliminary results for these changes, along with a PoC improvements for memory page compression and incremental GPU checkpointing at KubeCon earlier this year:
The following is a link to the CUDA documentation: I will add a comment that we intentionally avoid including NVIDIA's proprietary headers and the plugin resolves all symbols dynamically with
It is a minimal CUDA Driver API declarations reconstructed from the published CUDA Driver API specification. |
How is that licensed? Can it be included in GPLv2 code? |
My understanding is that using Slurm uses Blender's CUEW loader dynamically loads CUDA ( DPDK's CUDA GPU driver loads |
|
Sounds like the examples you have given should allow CRIU to adopt it. Although I am not the right person for this decision. Slurm is interesting as it is owned by nvidia now. What about the tests you added, can that be redistributed as GPLv2? |
As far as you don't distribute nvidia binary code, you should be fine. |
|
@rst0git maybe we need to introduce a new plugin, release it and then deprecate the old one in the next release? If something doesn't work, users will be able to switch back to the old plugin? wdyt? |
| unsigned char newUuid[16]; | ||
| } CUcheckpointGpuPair; | ||
|
|
||
| typedef struct { |
There was a problem hiding this comment.
Why do we redefine the structs here? I'm sure there's a reason I'm just curious.
|
We already know that Nvidia and Amd are here and will not go anywhere for inference and training for a while. Can we instead have this build-in in CRIU instead of dynamic loading? Dynamic loading is terrible for static compilation and does not help with distribution. Now is very hard to release criu for each single cloud provider with different versions of kernel, glibc and OS. |
|
@fals not sure what you are asking. Can you try to elaborate. |
|
@fals Not sure if I understood your comment correctly, but I think what you are asking for is not possible because of licensing reasons. |
My ask is simple, embedded this plugins in the binary instead of doing dlopen, keeping it as a external plugins makes no sense anymore and does not bring to much in my option but difficulties on building criu statically. Ideal world we should embedded everything that is possible in criu itself to help to release a static version for any architecture, that is my ask. |
if is not possible I will try to come up with something, because main issue at the moment is this packing and distribution to multi cloud and architectures, is quite not viable building for all OS versions right now. |
|
@fals I have patches that build static CRIU binary. It makes it easier to install and update CRIU in Kubernetes clusters. I will open a pull request for this soon. |
6fa4895 to
6037957
Compare
We do build static but you will see the problem when you mix this dynamic load plugins for gpu, unless you do something I couldn't figure out, but doesn't work well on cloud with different versions of glibc. |
6037957 to
b5b5b4c
Compare
@avagin I've updated the pull request to preserve the old plugin as v1 and introduce the new plugin as v2.
@dfeigin-nv I've updated the patch to fix this. |
b9efa2d to
95f21a2
Compare
26ba0ae to
85c4be7
Compare
85c4be7 to
2378d96
Compare
2378d96 to
8af83de
Compare
please don't do that. We don't want to choose a plugin version dynamically. Users have to install just one version. |
NVIDIA doesn't provide static libraries as far as I know. The issue here isn't the plugin mechanism, but the NVIDIA driver libraries. Even with static libraries, we'd run into compatibility problems whenever CRIU is linked against one version while containers use another. |
@avagin This patch aims to address the earlier comment here: #3076 (comment) The plugin that uses the The intended behavior is for CRIU to automatically detect whether the new plugin is supported by the installed driver. If it is supported, CRIU uses the new plugin, otherwise, it falls back to the Users typically do not install CRIU or either plugin themselves. They are deployed automatically to each Kubernetes node, for example, through a DaemonSet. The deployment is an OCI image that contains the CRIU binary and the required libraries and plugins. |
check_and_remove_inventory_plugin() compares only the number of bytes passed by its caller. A plugin can therefore remove an inventory entry for a longer name with the same prefix. CRIU then fails to report the required plugin as missing. Compare full names and remove the length argument from the helper. Add has_inventory_plugin() to query the inventory without removing an entry, and cover exact and non-destructive lookups in the plugin tests. Fixes: 5335b35 ("images/inventory: add field for enabled plugins") Assisted-by: Codex:GPT-5 Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
The late device hook may return negative device errors or positive worker errors, but restore currently logs only some failures and resumes the process with incomplete external state. Treat zero and -ENOTSUP as the only nonfatal results. Abort restore for every other return value so a failed GPU restore cannot release a partially restored task. Assisted-by: Codex:GPT-5 Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Plugins are loaded after CRIU parses command-line and RPC options. A plugin therefore cannot add its own parser without changing CRIU for every new setting. Keep namespaced PLUGIN.NAME=VALUE strings in order and expose them as an argv array. This lets plugins use getopt_long() and ignore options they do not recognize. Append RPC and libcriu values instead of replacing matching names, and clear request values between service requests. Assisted-by: Codex:GPT-5 Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Plugin-specific settings are otherwise discoverable only from external documentation. CRIU also has no way to inspect plugins in the selected library directory. Let plugins publish versioned declarative option metadata. Add a plugins command that lists descriptor-backed plugins or their supported options without calling plugin initialization. Keep generic help static because loading shared objects may run constructors and destructors. Cover both actions with descriptor-backed, optionless, and legacy test plugins. Assisted-by: Codex:GPT-5 Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Older supported NVIDIA drivers use cuda-checkpoint, while r580 and newer drivers expose the CUDA 13 checkpoint Driver API. Shipping these as separate CRIU plugin implementations would make plugin selection part of the generic loader and the deployment interface. Build one cuda_plugin.so containing both backends. Prefer the Driver API when all required symbols are present and the reported API version is at least 13000; fall back to cuda-checkpoint only when that probe reports unsupported. Let operators force driver-api or cuda-checkpoint through cuda_plugin.backend, with auto as the default and no fallback for an explicit selection. Keep the selected backend fixed for the operation, propagate probe and action errors, avoid probing either backend for CPU-only restore, and record the stable plugin inventory identity in the coordinator after either backend handles a CUDA task. Assisted-by: Codex:GPT-5 Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
CUDA backend choice must depend on driver capability, remain fixed after initialization, and honor an explicit operator selection without probing or falling back to the other backend. Add hardware-independent mock coverage for direct-backend preference, older-driver CLI fallback, forced selection, invalid values, probe isolation, CPU-only restore, unavailable backends, cross-backend restore, and operation rollback. Exercise the matrix in native x86-64 CI without requiring GPU hardware. Assisted-by: Codex:GPT-5 Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Add ZDTM workloads covering CUDA device memory, streams, events,
mapped host memory, stream-ordered allocations, graphs, cuBLAS handles,
and multiple GPUs. Build and run these only when the CUDA toolkit,
driver, and required device features are available.
Example:
// We run tests serially to avoid multiple CUDA workloads competing for the same GPU.
sudo env \
PATH=/usr/local/cuda/bin:$PATH \
NVCC=/usr/local/cuda/bin/nvcc \
CUDA_HOME=/usr/local/cuda \
CUDA_LIBDIR=/usr/local/cuda/targets/x86_64-linux/lib \
python3 ./test/zdtm.py run \
-t zdtm/static/cuda00 \
-t zdtm/static/cuda_streams00 \
-t zdtm/static/cuda_zerocopy00 \
-t zdtm/static/cuda_mempool00 \
-t zdtm/static/cuda_graph00 \
-t zdtm/static/cuda_cublas00 \
-t zdtm/static/cuda_multigpu00 \
--cuda-checkpoint \
-p 1 \
--keep-going
Assisted-by: Codex:GPT-5
Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
DUMP_DEVICES_LATE and RESTORE_INIT coordinate global state for each device plugin. Stopping after the first handler prevents other installed plugins from completing their cleanup or restore preparation. Add a dispatcher that calls every registered handler, skips unsupported handlers, and returns the first real error after all handlers have run. Use it for both global device hooks, document the broadcast semantics, and cover success, unsupported, and multiple-error ordering in the unit suite. Assisted-by: Codex:GPT-5 Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
CRIU exposes plugin descriptors, hooks, image helpers, device lifecycle callbacks, and namespaced options without a single description of how these interfaces fit together. Document plugin discovery, callback and broadcast return values, option ownership, inventory requirements, and GPU lifecycle ordering. Explain that filenames are opaque and that one installed plugin may select an internal backend during initialization without changing the generic loader ABI. Assisted-by: Codex:GPT-5 Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
I don't like this over-complicated logic in the criu plugin engine. Maybe it will be easier if we will have one nvidia plugin that will support both modes and the plugin will decide what mode to use. |
8af83de to
5ef1f46
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| return -1; | ||
| } | ||
|
|
||
| handle = dlopen(path, RTLD_LAZY | RTLD_LOCAL); |
| int ret; | ||
|
|
||
| size = cuda_gpu_inventory__get_packed_size(inventory); | ||
| if (!size || size > CUDA_GPU_INVENTORY_MAX_SIZE || size > UINT32_MAX) |
| return ret; | ||
| } | ||
|
|
||
| int cuda_device_map_resolve(const char *value, struct cuda_device_map *map) |
Save a plugin-private ordinal and UUID inventory after any CUDA backend handles a dump. On restore, accept complete UUID, ordinal, or automatic maps through cuda_plugin.device-map and preserve UUID-only compatibility with images that predate the auxiliary inventory. Resolve mappings once into backend-neutral UUID pairs. Pass them through CUcheckpointRestoreArgs for the Driver API or as a canonical --device-map value for a capable cuda-checkpoint utility. Keep restore mappings out of dump rollback and require mapping support during backend selection. Validate image framing, schema versions, GPU entries, map coverage, and duplicate endpoints. Enumerate only CRIU's configured CUDA-visible view and document the shared-view and plugin-private image constraints. Assisted-by: Codex:GPT-5 Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
Add a hardware-independent helper test for inventory framing and validation, UUID/ordinal/automatic resolution, old-image behavior, visibility preservation, canonical formatting, and the sequential image-streamer path. Extend the CUDA mocks to capture restore mappings and maintain restore state across CLI invocations. Verify that both forced backends receive the same resolved map and that a cuda-checkpoint utility without --device-map support is rejected before restore. Run the helper and backend integration coverage in the native CUDA CI block. Assisted-by: Codex:GPT-5 Signed-off-by: Radostin Stoyanov <rstoyanov@fedoraproject.org>
5ef1f46 to
7fc44dc
Compare
| if (nr_gpus > MAX_GPUS) { | ||
| fail("too many CUDA devices: %d > %d", nr_gpus, MAX_GPUS); | ||
| return -1; |
| return -1; | ||
| } | ||
|
|
||
| if (child_pid == 0) { // child |
|
|
||
| close(fd[WRITE]); | ||
| buf_off = 0; | ||
| /* Reserve one byte for the null charracter. */ |
This series switches the CUDA plugin from invoking
cuda-checkpointto using the CUDA Driver API fromlibcuda.so.1, and adds ZDTM tests with CUDA workloads.