Skip to content

volume: fix the 3x duplication and failure modes - #18

Merged
Daviduche03 merged 1 commit into
masterfrom
fix/volume-storage
Nov 6, 2026
Merged

Daviduche03 merged 1 commit into
masterfrom
fix/volume-storage

Conversation

@Daviduche03

Copy link
Copy Markdown
Owner

What

Single-host setups stored every volume byte 3x (live disk.img + volumes/chunks cache copy + volume-remote copy): 21G consumed for 7G of data, on a 38G disk. This fixes that and the failure modes found on that host.

The six fixes

# Problem Fix
1 3x duplication with local remote Reads go straight through to the remote dir; the legacy cache copy is removed on first use. Footprint: 3x → ~2x. Real remotes get a bounded cache (4 GiB, oldest-first).
2 EnsureLocal rewrote the whole image every create Per-chunk hash compare, identical ranges skipped — second create on an unchanged image performs zero chunk writes. A .applied.json sidecar (manifest digest + size/mtime) skips even the comparison when fully up to date.
3 Cache reads unverified, writes non-atomic Hash-verify on every cache hit (poisoned entry → delete + refetch), atomic cache writes (temp + rename), verify-before-write into disk.img. Kill -9 mid-apply heals on next create.
4 Failed commits/prepares leak chunks forever warmbox volume gc [--dry-run] (POST /api/volumes/gc): collects hashes from all volume + snapshot + base manifests, deletes the rest. 1h grace window spares in-flight commits. Safe with a desktop attached (only unreferenced chunks die).
5 Stale lease needs a daemon restart Lease conflict against an owner with no live desktop is reclaimed + retried on every create path, not just at boot.
6 Destroy exceeds the API timeout CLI destroy timeout 30s → 150s. The server commits before answering; the CLI no longer reports failure for an op that then succeeds.

Test plan

  • go build ./..., go vet, full go test ./... pass.
  • New tests in internal/volume + internal/cloudstore: zero-write second EnsureLocal (+ sidecar fast path), crash-heal (poisoned cache + corrupt disk → manifest-exact), GC dry-run lists exactly the orphan / real run reclaims / snapshot-pinned chunks survive / grace window holds, local-backend never populates the cache.
  • Acceptance mapping: (1) read-through + legacy cleanup, (2) LastApply stats + tests, (3) corrupt-cache test, (4) dry-run + delete tests, (5) reconcile on conflict, (6) 150s timeout + reconcile as backstop.

Notes

  • No manifest format change (sidecar is a local additive file); chunk size untouched; cloud backends untouched.
  • Out of scope as agreed: guest agent, images, Tailscale.
  • POST /api/volumes/gc returns the report as JSON (orphans, reclaimed_bytes, deleted, dry_run).

Single-host setups stored every volume byte three times (live image +
volumes/chunks cache copy + volume-remote copy): 21G for 7G of data. This
also fixes the failure modes found on that host.

- Local backend reads straight through to the remote dir instead of copying
  into the cache; the legacy cache copy is removed on first use. Footprint
  drops from 3x to ~2x (live + one backend copy). Real remotes get a bounded
  cache (4 GiB, oldest-first) instead of an unbounded one.
- EnsureLocal compares every manifest chunk against the disk first and skips
  identical ranges: create on an up-to-date image performs zero chunk writes.
  A sidecar (.applied.json) records the last materialized manifest plus
  size/mtime, so the fully-up-to-date case skips even the comparison.
- Cache hits are hash-verified and cache writes are atomic (temp + rename),
  so a torn file from a killed run is re-fetched, never served. Fetched
  bytes are verified before reaching the disk image.
- `warmbox volume gc [--dry-run]` (POST /api/volumes/gc) deletes remote
  chunks no volume, snapshot, or base manifest references. A 1h grace window
  spares in-flight commits; only unreferenced chunks are ever deleted, so it
  is safe with a desktop attached.
- A lease conflict against an owner with no live desktop is reclaimed and
  retried on every create path, not just at daemon startup.
- `warmbox destroy` gets a 150s timeout (was 30s): the server commits the
  volume before answering, and a slow commit used to make the CLI report
  failure for an operation that then succeeded.
@Daviduche03
Daviduche03 merged commit b3ca1b9 into master Nov 6, 2026
6 checks passed
@Daviduche03
Daviduche03 deleted the fix/volume-storage branch November 6, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant