refactor(blaze)!: backend-agnostic checkpoint payload and hibernate rebuild - #2678
Conversation
github-actions
Bot
added
component:blaze
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: be9f32d9ae
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
be9f32d to
8704ee6
Compare
|
@codex review both findings are addressed: the hibernate payload walk now validates every artifact path before the backend is stopped (failure compensates and the sandbox stays running), and the README shutdown paragraph now matches the implemented behavior (startup reconciliation instead of shutdown cleanup). |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8704ee6a4c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
8704ee6 to
1e8efd5
Compare
|
@codex review round-2 findings addressed: rejected payload trees stay removable via the dedicated removal depth bound, and resume now rejects restored owners that omit the guest socket the hibernation image requires. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1e8efd5752
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
1e8efd5 to
7bfb8b4
Compare
|
@codex review round-3 findings addressed: resume revalidates the identity of every consumed payload artifact against descriptors retained since verification before publishing running, and hibernation rejects payload files with foreign ownership or extra hard links before the backend is stopped. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7bfb8b41f2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
7bfb8b4 to
591311a
Compare
|
@codex review R5: hibernate directory parent linkage verification in require_payload_identity, three-layer supervisor pattern for both hibernate and resume (cancel-safe) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 591311aa2b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
591311a to
48ae5ca
Compare
WeissonHan
changed the title
|
@codex review R6: combined 3-commit series (payload subtrees + quiesce hook + hibernate rebuild); non-UTF-8 removal fix, empty-dir rejection in hibernate, threat-model explanation for sandbox-dir linkage and in-place mutation |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 48ae5ca3a2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
48ae5ca to
20765a6
Compare
|
@codex review R7: require at least one artifact from the backend subtree before publishing (empty backend + rootfs only is rejected) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 20765a642b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
8c3b23b to
87a637b
Compare
|
@codex review R8: rebased onto current main now that #2473 (Firecracker capture and restore) has landed, so this series carries the Firecracker adapter migration onto the payload-subtree contract. Since R7 the changes are: the Firecracker adapter names its own payload layout on both capture and restore; hibernation freezes the captured host shape (network slot, console recording) into its manifest because a resume has no live owner to probe and may run after a daemon restart; hibernate and resume pin the backend executable once and reuse that pin for both the capability answer and the launch; and the R5/R6 review fixes were moved from commit 2 into commit 1 where the publication and removal rules they change were introduced, which also let the duplicated non-UTF-8 removal walker collapse into the single bounded remover. Please pay particular attention to the payload-path trust argument (a payload directory is handed out as a configured pathname, with integrity resting on retained-descriptor hashing and revalidation rather than on the path) and to whether hibernation records everything a post-restart resume needs to rebuild the host shape. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 87a637b5dc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
ba795e0 to
496327d
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 496327db78
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
496327d to
67d0285
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 67d0285fc8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
67d0285 to
d55707a
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d55707a294
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
d55707a to
2832e86
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2832e86df6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
2832e86 to
d03135b
Compare
|
@codex review This PR adds sandbox hibernate and resume endpoints, restart-safe lifecycle recovery, and a manifest-verified payload contract consumed by Firecracker's existing snapshot and restore implementation. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d03135b6b2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The checkpoint format froze one backend's shape into the daemon: capture handed every backend exactly two named VM files (vmstate.snap, memory.snap), and both the store and the restore transaction validated committed directories against a fixed three-file whitelist. A backend whose native checkpoint is a directory tree - runsc writes an image directory plus the OCI spec that restore must revalidate byte-for-byte - cannot express its payload in that contract at all, so every non-VM backend would have had to fake VM file names or fork the format. Split the payload by producer instead. A checkpoint now carries two owned subtrees: backend/ belongs to the backend adapter and its internal layout is private to that backend; storage/ belongs to the storage provider and holds the rootfs capture. SnapshotRequest passes one payload directory rather than two file paths, RestoreRequest hands the adapter back the same subtree from the verified checkpoint, and version-1 checkpoints map both subtrees onto the checkpoint root so everything captured before the split stays restorable in place. With backend-private layouts a name whitelist can no longer exist, so integrity moves to the manifest: publication walks both subtrees, rejects symlinks and non-regular files, hashes every regular file under its slash-separated relative path, and commits a canonically sorted inventory. Load and every revalidation boundary then require the directory and the manifest to match exactly - a missing, altered, or unregistered file fails closed, which is the same guarantee the whitelist gave, now derived from the record each checkpoint carries. Artifact paths are validated as pure relative paths (no traversal, bounded depth and length) before the daemon ever resolves them, and removal recurses with the same bound. Publication also refuses a payload that carries nothing: an empty capture subtree means the adapter produced no artifact, and committing it would create a checkpoint that only fails when someone restores it. Removal, by contrast, has to stay more permissive than publication, because it is the exit from every rejected capture - including the daemon's own destroy path. It therefore enumerates names as raw bytes, recursively removes directories, and unlinks unrecognized non-directory entries regardless of their encoding or file type, so escaped capture output cannot strand the sandbox in RecoveryRequired. Recognized manifest-backed entries retain descriptor identity checks before they are removed. A payload directory is named by its configured pathname rather than by a descriptor name, because an adapter may hand that directory to a backend process which cannot resolve this daemon's /proc/self/fd entries. Integrity does not travel with the path: publication reopens and hashes the capture subtree through the retained stage descriptors, and restore revalidates every retained artifact identity through held descriptors before HEAD can move. A directory redirected by path replacement therefore fails the operation closed - an empty capture subtree is rejected rather than published - instead of letting foreign bytes be committed. The metadata format moves to version 2; version-1 manifests keep their frozen artifact-set validation and stay read-only. The guest mock now produces a directory-shaped payload (image/ plus bundle/config.json, mirroring a runsc checkpoint) and restores from it, proving the contract carries a container-backend layout end to end, while the plain mock keeps the flat VM pair so both shapes stay covered. The Firecracker adapter moves onto the contract without changing what it writes to disk: capture transfers its snapshot pair into the payload subtree under two names that are now private to that adapter, and its restore reads the same two names back out of the subtree it was handed. BREAKING: checkpoints written by this daemon use format version 2 and a subtree layout that older daemons cannot read. Existing version-1 checkpoints remain restorable but are never re-captured in place. Assisted-by: Qoder Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Checkpoint capture unconditionally paused the backend before invoking its snapshot and resumed it afterwards. That sequence encodes a VM assumption: Firecracker-style backends expose no consistent state until the VM is paused from outside. Backends built on a self-freezing capture primitive break under it - runsc checkpoint stops the container itself, and runsc rejects state operations against a container that is already paused, so the orchestration's own pause would poison the capture it prepares. Route the sequence through a pair of orchestration hooks on BackendInstance instead. quiesce_for_capture brings the workload to a consistent stop before the snapshot and defaults to pause; unquiesce_after_capture returns it to execution on both the publication and the compensation path and defaults to resume. Existing backends keep their exact behaviour through the defaults, while a self-freezing backend overrides both hooks as no-ops and leaves pause/resume unimplemented. The capture flow, its HEAD-update resume, and the compensation resume all call the hooks now; the failpoint names covering these boundaries are unchanged, so existing crash-injection coverage carries over as is. BREAKING: backend implementations that overrode pause/resume purely to participate in capture keep working through the defaults, but the documented integration contract for new backends is now the hook pair. Assisted-by: Qoder Signed-off-by: Weisson <Weisson@linux.alibaba.com>
This lets API users release the host resources a running sandbox holds — its backend process and memory — while keeping guest-visible state, then resume the sandbox later, including across a daemon restart. Hibernation is refused before the lifecycle journal changes unless the backend supports checkpoint capture and its adapter can restore the same backend version, so an unsupported combination leaves the sandbox running. A successful hibernate quiesces the workload through the capture hook, hands the backend one payload subtree to capture into, inventories whatever regular files the adapter wrote — a VM backend saves two named files, a container-shaped backend a whole image directory — records their sizes and SHA-256 digests in a manifest, synchronizes the image, and only then publishes it and commits the hibernated state. Resume verifies manifest identity, that the payload subtree and the inventory account for each other exactly, and every digest, before starting a replacement backend from that subtree and owning it through readiness checks. Every hibernation object is resolved relative to the retained sandbox directory descriptor, matching the checkpoint catalog, so a replaced or symlinked instance directory cannot redirect the image. Payload walks refuse symbolic links and bound their depth, and image removal recurses with the same bound. The host shape a resume has to rebuild is frozen into the manifest while the captured runtime is still alive. The generic restore transaction can probe the owner it is about to replace, but a hibernated sandbox has no owner left, and a resume may run in a daemon that never saw the original: whether the runtime held a per-sandbox network slot and whether it recorded guest console output would otherwise be rebuilt from defaults, silently returning a sandbox without its network device or without the console history its operator asked for. The backend executable is pinned once per operation and the same pinned file serves both the adapter's capability answer and the launch that follows. Reading the version from the configured path and then re-resolving that path to start would let a replacement land in between, and on the resume path that gap is only observable after the sandbox can no longer simply be left hibernated. Durable phases separate retryable failures from ownership that needs explicit cleanup: a failure before the stop begins resumes the original runtime, a clean resume failure returns to hibernated, and unconfirmed cleanup is retained as recovery-required. Startup keeps a completed hibernation resumable and retains an interrupted one for explicit destroy. Tests cover restart continuity with a directory-shaped guest-mock payload, unsupported adapters, corrupt artifacts, and failures around snapshot, backend stop, publication, resume start, readiness, and cleanup. Assisted-by: Qoder Signed-off-by: Weisson <Weisson@linux.alibaba.com>
d03135b to
e5aa655
Compare
|
@codex review This PR adds sandbox hibernate and resume endpoints, restart-safe lifecycle recovery, and a manifest-verified payload contract consumed by Firecracker’s existing snapshot and restore implementation. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e5aa6550b4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review This PR adds sandbox hibernate and resume endpoints, restart-safe lifecycle recovery, and a manifest-verified payload contract consumed by Firecracker’s existing snapshot and restore implementation. |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
单测实验设计思路实验从三个观察位置同时取证：
休眠完成后，实验还会停止并重新启动 Blaze，再调用恢复接口。这一步用于证明已保存的信息 接口响应、宿主机进程、串口输出和客户机内存四类证据必须同时成立。任何一类证据不符合 单测结果
可以确认以下能力已经通过本次验证：
这些证据共同排除了“只修改状态名称”和“恢复时重新冷启动客户机”两种虚假实现。因此， |
casparant
left a comment
casparant
left a comment
There was a problem hiding this comment.
Reviewed against the baseline we have been applying since #1830.
Review baseline — all met:
- Unified motivation: clear Why / Before / After — the checkpoint payload was locked to one backend's shape; this PR reworks it into a backend-agnostic, producer-owned subtree protocol (v2) with a SHA-256 manifest inventory, and rebuilds hibernate/resume on top of it, including survival across a daemon restart.
- No caller-less code: every new pub item (
begin_hibernate_operation/advance_hibernate_phase,begin_resume_operation/advance_resume_phase,backend_payload_dir/storage_payload_dir,validate_artifact_path,HibernateSandbox/ResumeSandbox,cleanup_owned_instances) has a production call site; test helpers stay behind#[cfg(test)]. - Docs: bilingual user guides and READMEs updated; the existing storage-artifact-synchronization design doc is updated in place to match the v2 protocol.
- Boundary: deferred items are tracked — payload-walk offload (#2697) and startup reconciliation reporting (#2721).
- Reserved annotation: the quiesce hook default (delegate to pause/resume), the v1 read-compatibility path, and the legacy Reset/Warm states are all annotated.
- Naming:
hibernate/resumeon the data plane, journal/phase vocabulary on the control plane; no reserved-word misuse.
End-to-end validation: the complete hibernate → daemon restart → resume sequence is confirmed on real KVM + Firecracker at head e5aa6550: the original Firecracker process stays gone for the full hibernation interval, the daemon restarts, resume starts a fresh Firecracker process from the saved snapshot, and the guest PID / in-memory nonce / counter are preserved across the cycle with the counter advancing again after resume. That is the feature working end to end on real hardware.
One housekeeping note: #2476 is the draft this supersedes — worth closing it once this lands.
LGTM.
2 participants
Summary
Three-commit series that makes the blaze checkpoint payload contract backend-agnostic and rebuilds hibernate/resume on top of it. Rebased onto main after #2473 landed, so the Firecracker adapter is migrated onto the new contract in the same series.
Commits
1. refactor(blaze)!: hand capture payloads to producers as owned subtrees
Motivation: The checkpoint payload contract was frozen around one backend's shape — capture handed every backend exactly two named VM files (
vmstate.snap,memory.snap) and validation used a fixed three-file whitelist. A backend whose native checkpoint is a directory tree (runsc writes an image directory plus the OCI spec) cannot express its payload in that contract at all.Changes:
CHECKPOINT_FORMAT_VERSION1 → 2; payload layout becomesbackend/(backend-owned subtree) +storage/rootfs.snapREQUIRED_ARTIFACTSwhitelist; integrity moves to the manifest (traverse + sort + SHA-256 per relative path; manifest and directory must account for each other exactly)SnapshotRequest{payload_dir, kind}/RestoreRequest{payload_dir, ...}replace fixed artifact pathsRecoveryRequiredimage/+bundle/config.json) and restores from it, proving the contract carries a container-backend layout end to endNote on path handling: a payload directory is named by its configured pathname rather than a
/proc/self/fdname, because an adapter may hand it to a backend process that cannot resolve the daemon's descriptors. Integrity does not travel with the path — publication hashes the subtree through retained stage descriptors and restore revalidates artifact identity through held descriptors before HEAD moves, so a redirected directory fails closed.2. refactor(blaze)!: quiesce backends through a capture hook
Motivation: Capture unconditionally paused the backend before
snapshot()and resumed it afterwards, which encodes a VM assumption. A self-freezing backend breaks under it: runsc checkpoint stops the container itself and rejects state operations against an already-paused container, so the orchestration's own pause would poison the capture it prepares.Changes:
BackendInstance::quiesce_for_capture()/unquiesce_after_capture(), defaulting to pause/resume so existing backends are unchanged3. feat(blaze): hibernate and resume sandboxes
Motivation: Release the host resources a running sandbox holds (backend process and memory) while keeping guest-visible state, then resume later — including across a daemon restart. The original draft #2476 was Firecracker-specific; on the new contract this works for any backend that supports checkpoint capture.
Changes:
hibernate()/resume()on the new contract: one payload subtree, manifest inventory, quiesce hookrequire_payload_identityrevalidates the hibernate directory's parent linkage and every retained artifact's identity after the adapter consumes the payloadDesign documentation
docs/design/storage-artifact-synchronization.md(+_zh): checkpoint protocol rewritten for producer-owned subtrees, manifest-inventory validation, quiesce hook contractdocs/user-guide/en|zh/runtime/blaze.md: checkpoint examples updated to the v2 subtree layout; hibernate/resume documentedsrc/blaze/README.md/README_zh.md: hibernate/resume behaviorsrc/blaze/AGENTS.md: trait contract updated with the hook pairTesting
cargo fmt --all -- --check,cargo clippy --workspace --all-targets -- -D warnings, and the 34 portable checkpoint-store tests pass locally.Test blaze, documentation lint, commit-message lint, PR checks, website build, component-version checks, runtime-dependency checks, and CLA all pass.storage/rootfs.snap, and every manifest entry matches the artifact's recorded size and SHA-256 digest.e5aa6550b4bd2aa8aeb7fbcd0bfb0216aa64e577, the completehibernate → daemon restart → resumesequence passes on Linux x86_64 with KVM and Firecracker 1.16.0 when Firecracker networking is disabled.Tracking