Skip to content

offload-v1: end-to-end shakedown on real backends before trusting offload with real data #129

Description

@mbertschler

Why

offload-v1 is merged. The code is sound and the test suite (unit + integration with fakes) is green, but nothing has run the full chain against the real backends from the NAS design (Hetzner Storage Box over SFTP, S3 Glacier Deep Archive, local kopia mirror). Because squirrel offload deletes the only local copy based on recorded durability, the first real use should be a deliberate, dry-run-first shakedown on a throwaway volume — not pointed at anything irreplaceable.

This issue is the acceptance checklist for that shakedown. It gates trusting offload with real data; it does not block the merge (already done).

Depends on / coordinate with:

Checklist (throwaway volume, real backends)

  • Seed a small throwaway volume with a mix of file sizes (incl. at least one object large enough to trigger a multipart upload on S3).
  • Index it; confirm the contents/files rows look right (squirrel query).
  • Sync to every target in offload_requires — NAS (peer), Hetzner (crypt content-addressed), S3 Glacier (crypt content-addressed), kopia mirror. Note the kopia first-use --init requirement (see below).
  • Fingerprints: confirm squirrel verify <dest> records/matches a checksum for each object on the content-addressed offsites — and specifically that a multipart object gets a usable fingerprint (this is the [medium] Confirm/repair S3 multipart ETag capture for scan-back fingerprints #118 dependency; record what lsjson --hash actually returns for the large object).
  • Durability pull: from the laptop, squirrel peer-sync pull-durability and confirm the relayed offsite components land for the targets only the NAS pushes to.
  • offload --dry-run: confirm the per-file gate decisions match expectations — every file OutcomeOffloaded, no surprise NotDurable, and the reasons read sensibly for any held-back file.
  • Real offload: run it; confirm local bytes are unlinked, rows flip present → offloaded, and a kind='offload' run is recorded.
  • Recover: restore at least one offloaded file from each offsite (incl. the kopia path via the kopia CLI) and confirm it round-trips to the original BLAKE3.
  • Negative check: deliberately make one target not durable (e.g. skip its sync) and confirm offload refuses that file with a clear per-target reason rather than deleting.

Operational notes to confirm during the run

  • kopia first use requires squirrel sync --init once (the agent/scheduler never sets it) — confirm the bootstrap step and that the README/flag help match the actual behavior.
  • offload deletes for real by default (no confirmation prompt; --dry-run is opt-in) — confirm this is the intended ergonomics for the agent/scripted path.
  • A destination's layout (mirror vs content-addressed) is fixed at first use and switching is refused — confirm the refusal fires as documented.

Done when

The checklist passes against the real backends and the multipart-fingerprint question (#118) is resolved, at which point offload can be pointed at real volumes with confidence.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions