feat(archive): shared layer store with reference-counted sweep - #1101
Closed
G4614 wants to merge 36 commits into
Closed
feat(archive): shared layer store with reference-counted sweep#1101G4614 wants to merge 36 commits into
G4614 wants to merge 36 commits into
Conversation
The guest rootfs disk is a thin COW overlay over the host-global guest rootfs cache, keyed by the bootstrap image plus guest binary version. It holds no user state, and clone and snapshot-restore already treat it as disposable: when it is absent, the next start recreates the overlay from the local cache. Export was the exception. It flattened the overlay into the archive, which both bloated every archive with a host-independent blob and stripped the backing reference, so an imported box would boot from the archived copy instead of the importing host's correctly-versioned cache. Export now omits it and import never installs one, even from an older archive. No archive version bump: the disk was already optional, since a never-started box exported without it, so old importers handle its absence and new importers ignore its presence. guest_disk_checksum stays on the manifest, always empty, so older importers still parse it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Export flattened the container disk's qcow2 chain into one image, so every archive carried a full copy of the image layer even though every box on a host is a COW child of the same one. v6 archives instead carry the chain as `layers/` blobs keyed by sha256, ordered base first, and an importer that already holds a layer skips its transfer entirely. Measured on a real box, a layered archive is the same size as a flattened one (12,578,096 vs 12,564,806 bytes): a short chain has almost no superseded data, and zstd erases the sparse image disk's holes. Export also no longer pays the flatten pass, and base digests are cached in the store so repeat exports do not re-hash immutable layers. Import resolves each layer against the local base store by digest, materializes only what is missing, and relinks children to paths it chose itself. The manifest carries digests and never paths, every blob is verified against its declared digest before anything points at it, and each relink is read back and checked — so a crafted archive still cannot aim a backing file at a host path of its choosing. Imported bases are ref-counted against the new box so the GC does not drop a layer the box reads through. Qcow2Helper::flatten keeps no caller but is retained: MAX_BACKING_CHAIN_DEPTH caps a chain at 8, and collapsing a chain is the compaction step that keeps clone-heavy lineages under that cap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Inserting LayerFormat and ArchiveLayer split the manifest's version-history doc comment away from the struct, leaving it dangling — clippy's empty_line_after_doc_comments. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unrelated to this PR's change: boxlite-ai#1084 landed an unformatted println! on main, so main's own Lint run is red and every PR merged against it inherits the failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five defects found reviewing the layered archive path, two of them exploitable. The deepest layer was never relinked nor validated, so its header's backing path — attacker-controlled data — survived verbatim into bases/. The chain is granted to the sandbox at start, so an archive could name any host file and have its bytes handed to the guest. That layer now goes through validate_no_backing_references. A layer already held locally was relinked to satisfy the incoming archive, rewriting a file other boxes and snapshots are backed by and silently re-pointing them at archive-supplied content. Reuse now requires that the local copy already sit on the parent the archive describes; otherwise a private copy is installed. Only freshly installed layers are ever relinked. A layer's digest covered its qcow2 header, which holds its parent's absolute local path. Layers therefore hashed differently on every host — cross-host dedup could never match — and the recorded digest went stale the moment import relinked the file, so re-exporting a box imported with a 3+ layer chain produced an archive only that host could read. Digests now name the canonical form, with backing_file_size and the path string blanked; backing_file_offset is kept, because it locates a reservation within the file that an importer needs in order to write the parent it picked. set_backing_file_path accepts that blanked reservation. Layers installed before a mid-way failure leaked permanently, since nothing collects a base with no dependents, and a layer sat unreferenced between installation and provisioning where a concurrent box rm would GC it. Each layer is now pinned to an import token as it lands; the token transfers to the box on success and collects on failure. A layer's declared format was written but never read, so a mislabelled layer reached relink and surfaced as a rebase error. verify_layer_format checks it against the blob. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The image disk is the deepest layer of every chain and usually the largest, and it has no base_disk record, so digest_of returned None and export hashed it in full every single time. It is immutable and its path is derived from its image digest, so a sidecar file next to it is enough. Registering it as a base disk instead would have pulled it into try_gc_base's reach, and the image cache has its own lifecycle. This does not make the image layer dedup across hosts, and it cannot: mke2fs embeds a random filesystem UUID and creation timestamps, so two hosts building the ext4 for the same OCI image produce different bytes. Verified by building twice from one source tree with identical arguments — ceeecb83… vs fe5397e5…. Content addressing can only ever match the image layer within a single host. Skipping that layer entirely, by naming it with its image reference and letting the importer rebuild it the way the guest rootfs already works, is the only thing that would help across hosts — and it trades away the archive being self-contained. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An archive is only worth having if it restores into a working box, so a failed guest freeze now abandons the export instead of quietly producing a lesser one. SIGSTOP pauses the vCPUs but leaves the guest's page cache unwritten, so without FIFREEZE the disk is crash-consistent — the equivalent of pulling the power cord. That archive looks exactly like a good one, and nothing in the manifest distinguishes them, which makes it worse than no archive: the failure surfaces at restore time, on data someone was relying on. The freeze was already attempted, but both an RPC error and the timeout only logged a warning and carried on; the `frozen` flag decided nothing beyond whether to thaw. Export now passes QuiescePolicy::RequireFrozen and the bracket refuses before SIGSTOP, so a doomed export costs neither a paused VM nor a copied disk. Clone and snapshot keep BestEffort — their output is a COW fork the caller boots immediately, not an artifact restored months later. The timeout goes from 5s to 30s. FIFREEZE does not fail under write load, it blocks until the filesystem flushes, so 5s turned a merely busy guest into a refusal. Verified with test_export_under_write_pressure, which exports while a background loop writes random 4KiB blocks: it passes with the freeze succeeding, and the refusal path is never reached. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bottom of every chain is the image's ext4, usually the largest layer, and it is the one layer content addressing can never reuse across hosts: mke2fs writes a random filesystem UUID and creation timestamps, so two hosts building the same image produce different bytes. Measured — two builds from one source tree with identical arguments hash differently. The image digest does match everywhere, being a hash of the OCI layer digests rather than of the built filesystem. Export now records it for whichever layer lives in the image cache, reading it back from the cache filename so export never has to reach a registry. An importer that already holds that image's disk uses its own copy and leaves the archived blob untouched. The archive still carries the blob, so it stays self-contained and an offline import keeps working. Dropping the blob entirely would save the transfer too, but only by making import depend on the image being pullable — a trade to make deliberately, and separately. The reused disk is returned without a base disk id: the image cache owns that file and manages its own lifecycle, so it must not be drawn into base-disk GC. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An `ExportOptions { as_directory: true }` export writes `manifest.json`
beside `layers/{hex}.zst` — one compressed object per layer, named by its
content — instead of one `.boxlite` file. The single file cannot be
backed up incrementally: it is opaque and changes completely between
exports. The directory can, and needs no protocol to do it: a layer two
exports share lands under the same name, so any mirror tool's existence
check (`aws s3 sync`, `mc mirror`, rsync) already skips everything the
destination holds. The sync tool is the negotiation.
Ordering makes an interrupted mirror safe: objects are written under a
temporary name and renamed, the manifest is written last, and a
re-export into the same directory leaves existing objects untouched —
verified by mtime in a_reexport_into_the_same_directory_skips_existing_objects.
Import reads the directory in place: no up-front extraction, each object
unpacked only when the host actually wants that layer. A layer already
held locally is never even opened — proven by handing the importer a
mirror whose already-held object is garbage bytes, which must not and
does not fail (a_layer_the_host_already_holds_is_never_read_from_the_directory).
Python (`ExportOptions(as_directory=True)`) and Node
(`{ asDirectory: true }`) expose the flag. REST refuses it: the wire
format is one HTTP body, and refusing beats silently handing back a
single file to a caller who asked for a mirrorable directory.
Real-VM round trip: export as directory, import, boot — passes as the
suite's tenth test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It verifies an existing capability (an archive is a file any S3 client can move) against a live MinIO, which does not belong in this change's scope. The file stays local; whether it becomes its own PR is a separate decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A store is `archives/<name>.json` manifests over one `layers/` pool —
the layout the user-facing sketch always wanted: N archives sharing
every layer they have in common, so the pool grows by what is new, not
by what is exported. Publishing is `ExportOptions { as_directory: true,
archive_name: Some(..) }` with the store root as destination; importing
is handing the manifest's path to the ordinary import. The manifests are
the references: deleting an archive is deleting its manifest, and
`ArchiveStore::gc` sweeps what no manifest names.
The sweep fails closed. A manifest that cannot be parsed aborts it,
because its references are unknown and anything deleted might be them.
Only content-named `.zst` objects and stale `.partial` staging files are
candidates — a README in the pool directory is not the sweeper's to
take, and a single-archive directory (root `manifest.json`) refuses to
open as a store at all, since its layers would all look orphaned.
Publish races sweep, and the classic failure is a manifest committed
over objects a concurrent gc just deleted — silently, discovered at
restore time. Two mechanisms close it: unreferenced objects younger than
the grace period are never swept (a publish writes objects before its
manifest, so in-flight work looks orphaned), and after its manifest
lands the publisher re-checks every object it named and rewrites any
that a sweep took in the window.
Real-VM round trip: publish into a store, import from the manifest path,
boot, then remove + gc empties the pool — the suite's eleventh test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Main migrated digest formatting to hex::encode while this branch was in
flight — sha2 0.11's output type no longer implements LowerHex, so the
merge left CanonicalLayer::digest as the one remaining `{:x}` and every
CI clippy job red. Aligned it, gave main's new base-disk test the digest
field this branch added, and satisfied the two lints the workspace-wide
clippy adds over the package-level run this branch had been validated
with: do_export_finalize's dest/as_directory pair becomes an ExportDest
enum (too_many_arguments), and extract_layer_object moves above the test
module (items_after_test_module).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # src/boxlite/src/litebox/clone_export.rs
Thirty seconds was enough for a busy guest but not for a busy host: with four VMs booting in parallel, the running-box export test flaked on a freeze timeout turned refusal, while a lone run passed. An export is not latency-sensitive — the timeout exists to bound a wedged or agentless guest, not to keep a busy one on schedule — so the ceiling doubles. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Thirty seconds was enough for a busy guest but not for a busy host: with four VMs booting in parallel, the running-box export test flaked on a freeze timeout turned refusal, while a lone run passed. An export is not latency-sensitive — the timeout exists to bound a wedged or agentless guest, not to keep a busy one on schedule — so the ceiling doubles. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The guest rootfs disk is a thin COW overlay over the host-global guest rootfs cache, keyed by the bootstrap image plus guest binary version. It holds no user state, and clone and snapshot-restore already treat it as disposable: when it is absent, the next start recreates the overlay from the local cache. Export was the exception. It flattened the overlay into the archive, which both bloated every archive with a host-independent blob and stripped the backing reference, so an imported box would boot from the archived copy instead of the importing host's correctly-versioned cache. Export now omits it and import never installs one, even from an older archive. No archive version bump: the disk was already optional, since a never-started box exported without it, so old importers handle its absence and new importers ignore its presence. guest_disk_checksum stays on the manifest, always empty, so older importers still parse it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Export flattened the container disk's qcow2 chain into one image, so every archive carried a full copy of the image layer even though every box on a host is a COW child of the same one. v6 archives instead carry the chain as `layers/` blobs keyed by sha256, ordered base first, and an importer that already holds a layer skips its transfer entirely. Measured on a real box, a layered archive is the same size as a flattened one (12,578,096 vs 12,564,806 bytes): a short chain has almost no superseded data, and zstd erases the sparse image disk's holes. Export also no longer pays the flatten pass, and base digests are cached in the store so repeat exports do not re-hash immutable layers. Import resolves each layer against the local base store by digest, materializes only what is missing, and relinks children to paths it chose itself. The manifest carries digests and never paths, every blob is verified against its declared digest before anything points at it, and each relink is read back and checked — so a crafted archive still cannot aim a backing file at a host path of its choosing. Imported bases are ref-counted against the new box so the GC does not drop a layer the box reads through. Qcow2Helper::flatten keeps no caller but is retained: MAX_BACKING_CHAIN_DEPTH caps a chain at 8, and collapsing a chain is the compaction step that keeps clone-heavy lineages under that cap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Inserting LayerFormat and ArchiveLayer split the manifest's version-history doc comment away from the struct, leaving it dangling — clippy's empty_line_after_doc_comments. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five defects found reviewing the layered archive path, two of them exploitable. The deepest layer was never relinked nor validated, so its header's backing path — attacker-controlled data — survived verbatim into bases/. The chain is granted to the sandbox at start, so an archive could name any host file and have its bytes handed to the guest. That layer now goes through validate_no_backing_references. A layer already held locally was relinked to satisfy the incoming archive, rewriting a file other boxes and snapshots are backed by and silently re-pointing them at archive-supplied content. Reuse now requires that the local copy already sit on the parent the archive describes; otherwise a private copy is installed. Only freshly installed layers are ever relinked. A layer's digest covered its qcow2 header, which holds its parent's absolute local path. Layers therefore hashed differently on every host — cross-host dedup could never match — and the recorded digest went stale the moment import relinked the file, so re-exporting a box imported with a 3+ layer chain produced an archive only that host could read. Digests now name the canonical form, with backing_file_size and the path string blanked; backing_file_offset is kept, because it locates a reservation within the file that an importer needs in order to write the parent it picked. set_backing_file_path accepts that blanked reservation. Layers installed before a mid-way failure leaked permanently, since nothing collects a base with no dependents, and a layer sat unreferenced between installation and provisioning where a concurrent box rm would GC it. Each layer is now pinned to an import token as it lands; the token transfers to the box on success and collects on failure. A layer's declared format was written but never read, so a mislabelled layer reached relink and surfaced as a rebase error. verify_layer_format checks it against the blob. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The image disk is the deepest layer of every chain and usually the largest, and it has no base_disk record, so digest_of returned None and export hashed it in full every single time. It is immutable and its path is derived from its image digest, so a sidecar file next to it is enough. Registering it as a base disk instead would have pulled it into try_gc_base's reach, and the image cache has its own lifecycle. This does not make the image layer dedup across hosts, and it cannot: mke2fs embeds a random filesystem UUID and creation timestamps, so two hosts building the ext4 for the same OCI image produce different bytes. Verified by building twice from one source tree with identical arguments — ceeecb83… vs fe5397e5…. Content addressing can only ever match the image layer within a single host. Skipping that layer entirely, by naming it with its image reference and letting the importer rebuild it the way the guest rootfs already works, is the only thing that would help across hosts — and it trades away the archive being self-contained. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An archive is only worth having if it restores into a working box, so a failed guest freeze now abandons the export instead of quietly producing a lesser one. SIGSTOP pauses the vCPUs but leaves the guest's page cache unwritten, so without FIFREEZE the disk is crash-consistent — the equivalent of pulling the power cord. That archive looks exactly like a good one, and nothing in the manifest distinguishes them, which makes it worse than no archive: the failure surfaces at restore time, on data someone was relying on. The freeze was already attempted, but both an RPC error and the timeout only logged a warning and carried on; the `frozen` flag decided nothing beyond whether to thaw. Export now passes QuiescePolicy::RequireFrozen and the bracket refuses before SIGSTOP, so a doomed export costs neither a paused VM nor a copied disk. Clone and snapshot keep BestEffort — their output is a COW fork the caller boots immediately, not an artifact restored months later. The timeout goes from 5s to 30s. FIFREEZE does not fail under write load, it blocks until the filesystem flushes, so 5s turned a merely busy guest into a refusal. Verified with test_export_under_write_pressure, which exports while a background loop writes random 4KiB blocks: it passes with the freeze succeeding, and the refusal path is never reached. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bottom of every chain is the image's ext4, usually the largest layer, and it is the one layer content addressing can never reuse across hosts: mke2fs writes a random filesystem UUID and creation timestamps, so two hosts building the same image produce different bytes. Measured — two builds from one source tree with identical arguments hash differently. The image digest does match everywhere, being a hash of the OCI layer digests rather than of the built filesystem. Export now records it for whichever layer lives in the image cache, reading it back from the cache filename so export never has to reach a registry. An importer that already holds that image's disk uses its own copy and leaves the archived blob untouched. The archive still carries the blob, so it stays self-contained and an offline import keeps working. Dropping the blob entirely would save the transfer too, but only by making import depend on the image being pullable — a trade to make deliberately, and separately. The reused disk is returned without a base disk id: the image cache owns that file and manages its own lifecycle, so it must not be drawn into base-disk GC. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An `ExportOptions { as_directory: true }` export writes `manifest.json`
beside `layers/{hex}.zst` — one compressed object per layer, named by its
content — instead of one `.boxlite` file. The single file cannot be
backed up incrementally: it is opaque and changes completely between
exports. The directory can, and needs no protocol to do it: a layer two
exports share lands under the same name, so any mirror tool's existence
check (`aws s3 sync`, `mc mirror`, rsync) already skips everything the
destination holds. The sync tool is the negotiation.
Ordering makes an interrupted mirror safe: objects are written under a
temporary name and renamed, the manifest is written last, and a
re-export into the same directory leaves existing objects untouched —
verified by mtime in a_reexport_into_the_same_directory_skips_existing_objects.
Import reads the directory in place: no up-front extraction, each object
unpacked only when the host actually wants that layer. A layer already
held locally is never even opened — proven by handing the importer a
mirror whose already-held object is garbage bytes, which must not and
does not fail (a_layer_the_host_already_holds_is_never_read_from_the_directory).
Python (`ExportOptions(as_directory=True)`) and Node
(`{ asDirectory: true }`) expose the flag. REST refuses it: the wire
format is one HTTP body, and refusing beats silently handing back a
single file to a caller who asked for a mirrorable directory.
Real-VM round trip: export as directory, import, boot — passes as the
suite's tenth test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It verifies an existing capability (an archive is a file any S3 client can move) against a live MinIO, which does not belong in this change's scope. The file stays local; whether it becomes its own PR is a separate decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Main migrated digest formatting to hex::encode while this branch was in
flight — sha2 0.11's output type no longer implements LowerHex, so the
merge left CanonicalLayer::digest as the one remaining `{:x}` and every
CI clippy job red. Aligned it, gave main's new base-disk test the digest
field this branch added, and satisfied the two lints the workspace-wide
clippy adds over the package-level run this branch had been validated
with: do_export_finalize's dest/as_directory pair becomes an ExportDest
enum (too_many_arguments), and extract_layer_object moves above the test
module (items_after_test_module).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Thirty seconds was enough for a busy guest but not for a busy host: with four VMs booting in parallel, the running-box export test flaked on a freeze timeout turned refusal, while a lone run passed. An export is not latency-sensitive — the timeout exists to bound a wedged or agentless guest, not to keep a busy one on schedule — so the ceiling doubles. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # sdks/node/lib/native-contracts.ts # sdks/node/src/snapshot_options.rs # sdks/python/src/snapshot_options.rs # src/boxlite/src/disk/base_disk.rs # src/boxlite/src/disk/qcow2.rs # src/boxlite/src/litebox/archive.rs # src/boxlite/src/litebox/box_impl.rs # src/boxlite/src/litebox/clone_export.rs # src/boxlite/src/rest/litebox.rs # src/boxlite/src/runtime/import.rs # src/boxlite/src/runtime/options.rs # src/boxlite/tests/clone_export_import.rs
9 tasks
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a shared layer store —
archives/<name>.jsonmanifests over one content-addressedlayers/pool — where archives of different boxes share their common layers, manifests are the references, andArchiveStore::gcsweeps unreferenced objects with a grace period plus publish-side repair to close the publish/sweep race.Stacked on #1080: the first 10 commits are that PR; only the last commit is new here.
Test plan:
cargo nextest run -p boxlite --features krun --test clone_export_import --profile vm— 11/11 on real VMs, includingtest_store_export_import_roundtrip(publish into a store → import from the manifest path → boot → remove + gc empties the pool)cargo nextest run -p boxlite --no-default-features --lib— 976/977; the one failure (seatbeltsandbox) reproduces on a pristine treea_layer_stays_while_any_archive_references_it— two manifests sharing a layer: removing one frees only its unique layer; the pool empties only after the last reference is gonea_store_frees_a_layer_only_with_its_last_reference— same property through the real export path: two publishes of one chain share every object (verified by mtime), gc frees nothing until both names are removeda_corrupt_manifest_aborts_the_sweep— unparseable manifest ⇒ gc refuses and deletes nothing (fail closed)an_unreferenced_object_inside_the_grace_period_is_kept— young orphans survive the sweep;gc(0)takes themforeign_files_in_the_pool_are_left_alone— only content-named.zstand stale.partialfiles are sweep candidatesa_single_archive_directory_is_refused— a rootmanifest.jsoncannot be opened as a storea_path_escaping_archive_name_is_refused—../evil,a/b,.hidden, emptycargo fmt --all -- --check;cargo clippy -p boxlite --no-default-features --all-targets -- -D warnings;cargo check -p boxlite -p boxlite-python -p boxlite-node