feat(blaze): hibernate and resume sandboxes - #2476
Conversation
github-actions
Bot
added
component:blaze
This lets API users release the host resources a running sandbox holds — its backend process and memory — while keeping guest-visible state, then resume the sandbox later, including across a daemon restart. Hibernation is refused before the lifecycle journal changes unless the backend supports pause and full snapshot capture and its adapter can restore the same backend version, so an unsupported combination leaves the sandbox running. A successful hibernate stages VM state and memory privately, records artifact sizes and SHA-256 digests in a manifest, synchronizes the image, and only then publishes it and commits the hibernated state. Resume verifies manifest identity, the exact file set, and every digest before starting a replacement backend, and owns that backend before readiness checks. Every hibernation object is resolved relative to the retained sandbox directory descriptor, matching the checkpoint catalog, so a replaced or symlinked instance directory cannot redirect the image. Durable phases separate retryable failures from ownership that needs explicit cleanup: a failure before the stop begins resumes the original runtime, a clean resume failure returns to hibernated, and unconfirmed cleanup is retained as recovery-required. Startup keeps a completed hibernation resumable and retains an interrupted one for explicit destroy. Tests cover restart continuity, unsupported adapters, corrupt artifacts, and failures around snapshot, backend stop, publication, resume start, readiness, and cleanup. Signed-off-by: Weisson <Weisson@linux.alibaba.com>
5539bd8 to
f528766
Compare
|
@codex review Rebuilt as a single commit on current main (includes merged #2472 and #2475): adds POST /v1/sandboxes/{id}/hibernate and /resume, resolves every hibernation object relative to the retained sandbox directory descriptor, and keeps the restore failure boundary. Stays a draft until #2473 provides the Firecracker restore adapter. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f528766465
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| snapshot_path: hibernate_dir.configured_path().join(VMSTATE_ARTIFACT), | ||
| mem_path: hibernate_dir.configured_path().join(MEMORY_ARTIFACT), |
There was a problem hiding this comment.
When the configured <state_dir>/<id> path is renamed or replaced after manifest verification, these PathBufs resolve through the replacement path rather than the retained hibernate_dir descriptor whose contents were hashed. The restore adapters reopen these paths directly, so a replacement containing matching sandbox metadata can substitute unverified VM state or memory, defeating the descriptor-relative guarantee and potentially restoring corrupted guest state.
Useful? React with 👍 / 👎.
| During graceful shutdown, the daemon first stops accepting work and drains | ||
| accepted connections. It then attempts bounded cleanup for every persisted | ||
| record and retained backend owner. One cleanup failure does not skip the | ||
| remaining sandboxes, and all unresolved records are reported. |
There was a problem hiding this comment.
When SIGTERM or SIGINT reaches daemon.rs::serve, the daemon only exits the accept loop and shuts down the storage-sync and template-import workers; it neither drains the detached connection tasks nor invokes sandbox cleanup. cleanup_owned_instances is currently reached only from startup reconciliation, so this new paragraph promises bounded shutdown cleanup and reporting that operators do not receive.
AGENTS.md reference: AGENTS.md:L337-L343
Useful? React with 👍 / 👎.
1 participant
Why
A running sandbox holds host resources — a live backend process and its memory —
even during periods when nobody is using it. Today the only way to release those
resources is
DELETE /v1/sandboxes/{id}, which discards guest-visible statealong with them. Checkpoint capture (#2472) records state but deliberately keeps
the backend running, so it does not free anything either.
This PR closes that gap: hibernation releases the live backend while keeping the
sandbox identity, its storage, and its guest-visible state, so the same sandbox
can be resumed later — including after a daemon restart.
What changed
Two new routes in the sandbox namespace:
/v1/sandboxes/{id}/hibernate/v1/sandboxes/{id}/resumeCapability gating happens before any state changes. Hibernation requires a
backend that supports pause and full snapshot capture whose configured adapter
can restore the same backend version. An unsupported combination returns HTTP
501 with the sandbox still running; a sandbox in the wrong state returns 409.
Publication is ordered so an interruption cannot corrupt the image. A
successful hibernate records intent, pauses the backend, writes VM state and
memory into a private staging directory, flushes the retained storage slot,
records artifact sizes and SHA-256 digests in a manifest, synchronizes the
complete image, and only then publishes it and commits
Hibernated.Resume verifies before it switches. It checks manifest identity, the exact
file set, and every artifact digest before starting a replacement backend, takes
ownership of that backend before waiting for optional guest readiness, and
commits
Runningonly after a final liveness check.Every hibernation object is descriptor-relative. Objects are resolved
through the retained sandbox directory descriptor rather than a configured
pathname, matching the checkpoint catalog, so a replaced or symlinked instance
directory cannot redirect the image. This required exposing the sandbox
directory descriptor on
OwnedRunDir(+13 lines instate_store.rs).Failure boundaries follow the restore contract from #2475. A failure before
the stop begins resumes the original runtime and leaves the sandbox
Running. Aresume failure whose cleanup can be confirmed returns to
Hibernatedfor retry;unconfirmed cleanup retains the replacement owner and journal through
RecoveryRequired. Startup keeps a completed hibernation resumable and retainsan interrupted one for explicit destroy.
Lifecycle grows from 10 to 13 states (
Hibernating,Hibernated,Resuming),and
begin_hibernate_operation/begin_resume_operationreject a concurrentoperation instead of overwriting its journal, matching
begin_checkpoint_operation.Scope
db0e4654e14808d2398c813719e4742567e88954(main, includes merged feat(blaze): capture sandbox checkpoints #2472 and feat(blaze): restore sandbox checkpoints #2475)f528766465f18e2716149855b0e9126a4448fb807e69d56d99f427c45db3753914846aa8d9e46a7b/v1/instancesroutesThis PR does not re-carry the accumulated history of the previous draft branch.
The routes, pool endpoints, and template endpoints that branch also touched are
out of scope and were dropped:
/v1/instances/*was removed by #2585, and pooland template behaviour is unchanged from main.
Validation
An uninterrupted Linux x86_64 run on Rust 1.88.0 completed for the tree above.
cargo fmt --all -- --checkcargo metadata(default / all-features)cargo build --workspace --all-targets(default / all-features)cargo clippy --workspace --all-targets -- -D warnings(default / all-features)cargo test --workspace -- --test-threads=1cargo test --workspace --all-features -- --test-threads=1cargo doc --workspace --no-deps(default / all-features)-D warningsscripts/docs-lint.shscripts/docs-link-check.pygit diff --check14/14 gate stages pass. 15 hibernation and resume tests are green, covering
restart continuity, unsupported adapters, corrupt artifacts, and failures around
snapshot, backend stop, publication, resume start, readiness, and cleanup.
Documentation
docs/user-guide/{en,zh}/runtime/blaze.md— new hibernation and resumesection: status-code contract, artifact verification, failure ownership, and
the disk-space tradeoff of retaining the latest image.
src/blaze/README.md/README_zh.md— route table, feature list, the 13lifecycle states, and startup/shutdown behaviour for hibernated sandboxes.
Remaining dependency
Per the phase-3 plan, this PR stays a draft until #2473 lands: its public scope
includes real Firecracker hibernate–restart–resume validation, which needs the
Firecracker restore adapter. The mock guest backend exercises the full flow in
CI today.
Rollback
Revert the single commit. The routes, the three lifecycle states, and the
hibernation image format are all introduced here, so no persisted state written
by earlier releases changes format.