criu: add an external-memory provider for restore - #3119
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
99e03f3 to
28819da
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## criu-dev #3119 +/- ##
============================================
- Coverage 57.77% 57.36% -0.42%
============================================
Files 161 163 +2
Lines 43886 44567 +681
Branches 9626 9720 +94
============================================
+ Hits 25357 25566 +209
- Misses 18290 18762 +472
Partials 239 239 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
c3f5114 to
3ebcde6
Compare
|
please format commits properly. |
| (vma->e->madv & (1ul << MADV_WIPEONFORK))) | ||
| continue; | ||
|
|
||
| vma->e->flags &= ~MAP_ANONYMOUS; |
There was a problem hiding this comment.
Why do we need to do that here? What if a memory provider restores only a few specified VMAs or only shared VMAs?
There was a problem hiding this comment.
Yeah, that’s fair. A provider may only support some VMAs, or only shared memory. We’ll treat -ENOTSUP from GET_VMA as “this VMA isn’t handled” and let CRIU restore it normally. Other errors will still fail the restore.
| if (kdat.has_memfd && (!is_hugetlb || kdat.has_memfd_hugetlb)) { | ||
| provider_ret = extmem_init(); | ||
| if (provider_ret == 0) { | ||
| provider_ret = extmem_get_shared(si->shmid, si->size, &f); |
There was a problem hiding this comment.
why extmem_get_shared can't call extmem_init and extmem_validate_memfd_mapping_fd?
There was a problem hiding this comment.
Good point. I’ll make extmem_get_shared() handle all that.
| return provider_request(&req, fd); | ||
| } | ||
|
|
||
| int extmem_validate_mapping_fd(int fd, unsigned long length) |
There was a problem hiding this comment.
why do we need validating fd? In real life, a mapping can be wider than its backing store.
There was a problem hiding this comment.
Hmm, I think you're right. I was trying to be defensive but that's shouldn't be the case here. A mapping can be wider than its backing file, so I'll remove the FD validation from the provider paths.
|
I think the first patch should be split into three patches: handling images, handling memfd/shmem files, and handling anon VMAs. |
|
@dfeigin-nv Adding a few notes from the meeting earlier today: it would be good to have a reference implementation of an external-memory provider. For example, this could follow the precedent of criu-image-streamer - a separate repository with a more permissive license, independent code reviews, and its own release cycle; or use tighter integration, similar to the |
|
@rst0git as discussed I'll open it in https://github.com/ai-dynamo/snapshot in a subdirectory so that I can start using it and you can take a look at it and see how it can look like. If you will deem it appropriate we can move it to its own repo under checkpoint-restore. |
You're right. I think a fourth patch for the protocol itself is also good to make it more readable. I'll restructure it. |
a99da14 to
217b7fd
Compare
Add the CRIU-side protocol for external memory providers. Connect through the inherited provider socket and request image, shared-memory, and anonymous-VMA objects through one restore-side interface. Initialize the provider early enough to serve restore metadata and wait for it to be ready before applying memfd seals. Keep restore setup compatible with CG_MODE_IGNORE. Assisted-by: Codex:GPT-5 Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Use provider-supplied FDs for restore images while keeping dump-time and locally required inventory images on CRIU's existing path. Treat -ENOTSUP as a per-image fallback so providers can add image types incrementally. Keep returned image FDs in CRIU's existing buffered reader. Non-page images are small compared with the memory payload, so a mmap-based reader can be considered separately if it proves worthwhile. Assisted-by: Codex:GPT-5 Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Restore checkpointed memfd and shmem mappings from provider-supplied file descriptors. Fall back to CRIU's normal path only when the provider returns ENOTSUP; propagate other provider errors. Assisted-by: Codex:GPT-5 Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Restore provider-backed anonymous mappings from provider-supplied file descriptors. Fall back per VMA on ENOTSUP, preserving normal CRIU handling for unsupported regions. Tolerate ESRCH when detaching a thread that has already exited. Assisted-by: Codex:GPT-5 Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Add unit tests for the provider protocol and a ZDTM mock that restores private and shared memory through the provider. Exercise a regular checkpoint and restore without the provider, as well as per-object ENOTSUP fallback. Keep the existing IPv4/IPv6 address expression scoped to its case-insensitive match so the spelling check accepts the expression. Assisted-by: Codex:GPT-5 Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
Describe the inherited provider FD, request protocol, fallback behavior, memory-object requirements, and restore ordering. Assisted-by: Codex:GPT-5 Signed-off-by: Dan Feigin <dfeigin@nvidia.com>
217b7fd to
78ef07f
Compare
5 participants
This adds a generic external-memory provider interface for restore.
CRIU can use an inherited
extmem-providerUnixSOCK_SEQPACKETFD to obtain:The provider receives relative image names and object identities. It can decline unsupported image files with
-ENOTSUP; CRIU then uses its normal local image path. Private and shared-memory requests must succeed.The protocol uses protobuf request/response envelopes and
SCM_RIGHTSfor one-FD replies. CRIU validates received FDs, prepares provider-backed VMAs early, waits for provider readiness before pre-resume, and closes the provider session withCOMMITorABORT.This is restore-only. Checkpoint output remains unchanged (upcoming commit/PR)
Tests cover the request operations, VMA ID zero, scoped provider FDs, FD validation, image fallback, provider failures, readiness, commit/abort, and a focused static restore with a mock provider.
120B restore measurements
Workload: GPT-OSS-120B
Checkpoint data: ~140 GB
“Provider I/O” is the time spent reading checkpoint data and preparing memfds before CRIU starts.
Will add a few more benchmarks in place.
S3 restore measurements
Workload: Qwen3-14B
Checkpoint data: 45.7 GiB
Each row is the median of five successful restores.
More information can be found in:
https://docs.google.com/document/d/1i6xlonOpnY-HkUPmsLnuXzB0RqsiM9_YjwFVS-mVQ1M/edit?tab=t.0#heading=h.qkpu5w2ghwv9