offload-v1: NAS offsite + offload, security-audited

AGENTS.md

@@ -3,11 +3,15 @@
 Squirrel indexes **content** (BLAKE3 hashes), not paths. A hash ever observed
 must stay retrievable. Paths are observations of content; content is the entity.
 
-So `Upsert` never rewrites a row's `blake3` in place: when content at a path
-changes it marks the prior row `superseded` and inserts a new one, keeping at
-most one live (non-`superseded`) row per path. The schema enforces this on
-`files` — the `files_blake3_immutable` trigger and the `uniq_files_live_per_path`
-partial unique index (`store/migrations.go`).
+The schema makes this literal: `contents` is the append-only content entity
+(one row per BLAKE3, with size and origin), and `files` rows are path↔content
+observations referencing it. `Upsert` never rewrites a row's `content_id` in
+place: when content at a path changes it marks the prior row `superseded` and
+inserts a new one, keeping at most one live (non-`superseded`) row per path —
+enforced by the `uniq_files_live_per_path` partial unique index
+(`store/migrations.go`); the id↔hash binding itself is immutable by
+construction (`contents.blake3` is UNIQUE and contents rows are never
+updated).
 
 The `runs` table follows the same no-loss spirit by policy, not schema: squirrel
 never auto-prunes runs — they're an audit trail, and any retention is explicit

SAFETY-AUDIT.md

@@ -1022,6 +1022,143 @@ when restoreFromNode lands.
 
 ---
 
+## Durability evidence & offsite verification (offload-v1)
+
+Findings specific to the offload-v1 feature set — the durability version
+vectors that gate `offload`, the peer durability pull that carries
+evidence between nodes, and the content-addressed offsite push. These
+are framed for the intended deployment: a **single operator** whose
+nodes (laptop, NAS) are all machines they control and hold the only
+credentials to. Under that model the adversary is overwhelmingly
+*entropy and bugs*, not a hostile peer; the findings are sized
+accordingly.
+
+### D1: Durability-pull trust boundary (relayed offsite evidence)
+
+**Severity:** Medium (defence-in-depth) • **Likelihood:** N/A
+(documented assumption, not a live defect).
+
+**Where**
+
+- `sync/durability.go` — `PullDurability` / `pullDurability`,
+  `validateComponent`, `validateFreshness`.
+- `offload/gate.go` — `check`, `methodVerified`, `freshnessFailure`.
+- `store/nodes.go` — `GetOrCreateOriginNode`.
+
+**The boundary**
+
+A durability component is recorded one of two ways, and they differ in
+what they trust:
+
+- **Direct, self-verified.** When this node pushes to a target itself —
+  the NAS via peer sync (`sync/node.go`, tagged `peer-blake3` after the
+  receiver re-hashes every path) or a bucket via rclone (`sync/sync.go`)
+  — it writes the component into its **own** store from its **own**
+  confirmed transfer (`AdvanceDestinationVectorTo`). No peer is trusted.
+- **Relayed, peer-asserted.** For a target this node never pushes to (an
+  offsite only the NAS reaches), the only evidence is what the NAS
+  reports over the durability pull (`UpsertDestinationRunIDVerified`,
+  reached only from `pullDurability`). Putting such a target in a
+  volume's `offload_requires` means the local delete decision trusts the
+  NAS's recorded `(origin, run, method)` assertion. The pull validates
+  shape (positive run, valid origin name, recognised method) and is
+  monotonic, but carries **no proof of possession** — a peer that
+  asserts an inflated run for a destination in the accepted set would be
+  believed.
+
+**Decision (intended):** the relayed-evidence trust is **accepted**. The
+NAS is in the same trust domain as the laptop; a NAS that lies about
+durability is a compromised-or-broken NAS, in which case the archive it
+holds is already in question and `offload` is a footnote. The gate fails
+*closed* on absent evidence, so a peer can only ever *withhold*
+offload-eligibility, and the redundancy decision (gate on **all** copies
+via `offload_requires`, not the fewest-trusted subset) is what protects
+against data loss — see the offload section of `README.md`.
+
+**Defence-in-depth implemented in this branch** (cheap; turns *bugs*
+into loud failures, not a security control):
+
+- **Verify-method allow-list at the pull boundary** — `validateComponent`
+  refuses a non-empty `verify_method` that isn't one this build defines
+  (`store.KnownVerifyMethod`). Previously an unknown method was stored
+  and then silently treated as unverified by the gate; now a peer bug or
+  version-skew string is rejected at receipt. Empty (legitimately
+  "unverified") still passes.
+- **Origin-node creation cap** — `pullDurability` refuses a pull that
+  names more than `maxOriginNodesPerPull` (256) distinct origins, so a
+  runaway peer cannot grow the local `nodes` table without bound via
+  `GetOrCreateOriginNode`. A real volume references a handful of origins;
+  the cap only converts a flood into an observable refusal.
+
+**Not done (deliberately):** no proof-of-possession protocol, no
+laptop-side independent verification of relayed offsites. Those defend
+against a malicious NAS, which is out of model. The random per-file
+nonce in the rclone crypt overlay also makes "the NAS proves the stored
+ciphertext decrypts to the right content" impractical without either a
+content-derived nonce or the laptop downloading and decrypting the
+object — neither warranted here.
+
+**Issue:** `durability: document the relayed-evidence trust boundary; add verify-method allow-list and origin-node cap as defence-in-depth` (implemented in this branch)
+
+### D2: Content-addressed offsite push proves presence+size, not decrypt-correctness
+
+**Severity:** Medium • **Likelihood:** Low (requires a transfer-time
+corruption that preserves decrypted size, or the documented
+re-hash→read TOCTOU window to fire).
+
+**Where**
+
+- `sync/content_addressed.go` — `uploadOneObject` (re-hash → `copyTo` →
+  `statRemote` size check), `captureFingerprints`.
+- `sync/verify_remote.go` — `VerifyRemote` (scan-back re-check).
+
+**What it does / doesn't establish**
+
+At upload the push: (1) re-hashes the **local plaintext** and refuses on
+drift, so the encryption input is the right content; (2) confirms the
+object is **present** and its **decrypted size** (stat is through the
+crypt overlay) matches the index; (3) records the provider's checksum of
+the ciphertext as the scan-back baseline. The underlying backend's own
+transfer integrity (e.g. S3 Content-MD5 on PUT) covers "the ciphertext
+rclone sent is the ciphertext stored."
+
+It does **not** confirm that the stored ciphertext *decrypts back* to the
+indexed hash — there is no post-upload decrypt-and-rehash. The
+unguarded slivers are the documented fork/exec window between the
+re-hash and rclone's open (`uploadOneObject` "Residual:" comment) and a
+hypothetical crypt bug that produces a right-decrypted-size, wrong
+content object. Ongoing bitrot is caught by the scan-back re-verify; a
+*wrong-at-upload* object is the gap.
+
+**Proposed mitigation (opt-in, NAS-local — sketch, not built):**
+
+- Add a `--verify` mode to the content-addressed push that, after an
+  object lands, downloads it back **through the crypt overlay**,
+  BLAKE3s the plaintext, and compares to the indexed hash. This is the
+  only check that closes decrypt-correctness, and it lives entirely on
+  the pushing node (which holds the plaintext) with no protocol or
+  laptop change.
+- Scope it to the **initial upload** of each content hash (or a sampled
+  subset), not every run — the object is append-only and immutable, so
+  one read-back per object is sufficient. Cost is one download per
+  verified object (egress), so it must be opt-in and never run against
+  cold-tier targets (e.g. Glacier Deep Archive, where a read needs a
+  restore).
+- Tightening the re-hash→read TOCTOU window further (snapshot/lock the
+  source) is **not** recommended — the window is one fork/exec, the
+  indexer and scrub already surface drift, and chasing it is
+  disproportionate.
+
+**Acceptance**
+
+- With `--verify`, a seeded object whose stored ciphertext decrypts to
+  the wrong content is caught and the run fails before the durability
+  vector advances; without it, behaviour is unchanged.
+
+**Issue:** `sync: optional read-back-decrypt-rehash verification for content-addressed uploads`
+
+---
+
 ## Cross-cutting recommendations
 
 ### Tests we should add now

agent/durability.go

@@ -0,0 +1,102 @@
+package agent
+
+import (
+	"context"
+	"fmt"
+	"net/http"
+
+	"github.com/mbertschler/squirrel/store"
+	"github.com/mbertschler/squirrel/syncproto"
+)
+
+// handleDurability implements POST /v1/sync/durability: a session-less,
+// read-only listing of this node's recorded destination durability
+// vectors for one volume. Peers pull it (after a sync, or standalone)
+// to hold offline evidence about destinations only this node can see.
+// Node identity travels as names — local node ids mean nothing to the
+// caller.
+func (r *peerSyncRouter) handleDurability(w http.ResponseWriter, req *http.Request) {
+	var body syncproto.DurabilityRequest
+	if err := decodeJSON(w, req, &body); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	if body.Volume == "" {
+		writeError(w, http.StatusBadRequest, "volume is required")
+		return
+	}
+	if _, ok := r.volumes[body.Volume]; !ok {
+		writeError(w, http.StatusNotFound, fmt.Sprintf("volume %q is not declared on this node", body.Volume))
+		return
+	}
+	resp, err := r.durabilityResponse(req.Context(), body.Volume)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, resp)
+}
+
+// durabilityResponse assembles the wire components for one volume. A
+// declared volume with no store row (never indexed or synced) yields an
+// empty component list rather than an error — "no recorded durability"
+// is a valid answer.
+func (r *peerSyncRouter) durabilityResponse(ctx context.Context, volumeName string) (syncproto.DurabilityResponse, error) {
+	v, err := r.srv.store.GetVolumeByName(ctx, volumeName)
+	if store.IsNotFound(err) {
+		return syncproto.DurabilityResponse{}, nil
+	}
+	if err != nil {
+		return syncproto.DurabilityResponse{}, fmt.Errorf("lookup volume: %w", err)
+	}
+	rows, err := r.srv.store.ListVolumeDestinationRunIDs(ctx, v.ID)
+	if err != nil {
+		return syncproto.DurabilityResponse{}, fmt.Errorf("list destination vectors: %w", err)
+	}
+	fresh, err := r.srv.store.ListVolumeDestinationPushFreshness(ctx, v.ID)
+	if err != nil {
+		return syncproto.DurabilityResponse{}, fmt.Errorf("list push freshness: %w", err)
+	}
+	names := make(map[int64]string, 4)
+	resolve := func(nodeID int64) (string, error) {
+		if name, ok := names[nodeID]; ok {
+			return name, nil
+		}
+		node, err := r.srv.store.GetNodeByID(ctx, nodeID)
+		if err != nil {
+			return "", fmt.Errorf("resolve origin node %d: %w", nodeID, err)
+		}
+		names[nodeID] = node.Name
+		return node.Name, nil
+	}
+	resp := syncproto.DurabilityResponse{
+		Components: make([]syncproto.DurabilityComponent, 0, len(rows)),
+		Freshness:  make([]syncproto.DurabilityFreshness, 0, len(fresh)),
+	}
+	for _, row := range rows {
+		name, err := resolve(row.OriginNodeID)
+		if err != nil {
+			return syncproto.DurabilityResponse{}, err
+		}
+		resp.Components = append(resp.Components, syncproto.DurabilityComponent{
+			Destination:  row.Destination,
+			OriginNode:   name,
+			OriginRun:    row.OriginRunID,
+			UpdatedAtNs:  row.UpdatedAtNs,
+			VerifyMethod: row.VerifyMethod,
+		})
+	}
+	for _, row := range fresh {
+		name, err := resolve(row.OriginNodeID)
+		if err != nil {
+			return syncproto.DurabilityResponse{}, err
+		}
+		resp.Freshness = append(resp.Freshness, syncproto.DurabilityFreshness{
+			Destination: row.Destination,
+			OriginNode:  name,
+			OriginRun:   row.OriginRunID,
+			UpdatedAtNs: row.UpdatedAtNs,
+		})
+	}
+	return resp, nil
+}

agent/durability_test.go

@@ -0,0 +1,116 @@
+package agent
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/mbertschler/squirrel/config"
+	"github.com/mbertschler/squirrel/syncproto"
+)
+
+// postDurability drives POST /v1/sync/durability against the server's
+// handler and decodes the response into out. Returns the HTTP status.
+func postDurability(t *testing.T, srv *Server, body syncproto.DurabilityRequest, out any) int {
+	t.Helper()
+	encoded, err := json.Marshal(body)
+	if err != nil {
+		t.Fatalf("marshal request: %v", err)
+	}
+	req := httptest.NewRequest(http.MethodPost, "/v1/sync/durability", bytes.NewReader(encoded))
+	req.Header.Set("Authorization", "Bearer test-token")
+	req.Header.Set("Content-Type", "application/json")
+	rec := httptest.NewRecorder()
+	srv.Handler().ServeHTTP(rec, req)
+	if out != nil && rec.Code == http.StatusOK {
+		if err := json.Unmarshal(rec.Body.Bytes(), out); err != nil {
+			t.Fatalf("decode response: %v (%s)", err, rec.Body.String())
+		}
+	}
+	return rec.Code
+}
+
+// TestDurabilityEndpointListsComponents: the endpoint returns every
+// recorded vector component for the volume with origin nodes resolved
+// to names — the cross-node identity the caller can map locally.
+func TestDurabilityEndpointListsComponents(t *testing.T) {
+	ctx := context.Background()
+	vol := &config.Volume{Name: "pics", Path: t.TempDir()}
+	srv := newTestServer(t, Config{Volumes: map[string]*config.Volume{vol.Name: vol}})
+
+	v, err := srv.store.CreateVolume(ctx, vol.Name, vol.Path)
+	if err != nil {
+		t.Fatalf("CreateVolume: %v", err)
+	}
+	self, err := srv.store.GetSelfNode(ctx)
+	if err != nil {
+		t.Fatalf("GetSelfNode: %v", err)
+	}
+	ext, err := srv.store.CreateNode(ctx, "ext", "peer://ext")
+	if err != nil {
+		t.Fatalf("CreateNode: %v", err)
+	}
+	for _, seed := range []struct {
+		dest   string
+		nodeID int64
+		run    int64
+	}{
+		{"offsite-a", self.ID, 12},
+		{"offsite-a", ext.ID, 4},
+		{"mirror", self.ID, 9},
+	} {
+		if err := srv.store.UpsertDestinationRunID(ctx, v.ID, seed.dest, seed.nodeID, seed.run, false); err != nil {
+			t.Fatalf("seed %+v: %v", seed, err)
+		}
+	}
+
+	var resp syncproto.DurabilityResponse
+	if code := postDurability(t, srv, syncproto.DurabilityRequest{Volume: "pics"}, &resp); code != http.StatusOK {
+		t.Fatalf("status = %d, want 200", code)
+	}
+	if len(resp.Components) != 3 {
+		t.Fatalf("components = %d, want 3: %+v", len(resp.Components), resp.Components)
+	}
+	got := map[string]int64{}
+	for _, c := range resp.Components {
+		if c.UpdatedAtNs == 0 {
+			t.Fatalf("component %+v has zero updated_at_ns", c)
+		}
+		got[c.Destination+"/"+c.OriginNode] = c.OriginRun
+	}
+	want := map[string]int64{
+		"offsite-a/" + self.Name: 12,
+		"offsite-a/ext":          4,
+		"mirror/" + self.Name:    9,
+	}
+	for k, w := range want {
+		if got[k] != w {
+			t.Fatalf("component %s = %d, want %d (full: %+v)", k, got[k], w, got)
+		}
+	}
+}
+
+// TestDurabilityEndpointGuards: an undeclared volume 404s, a missing
+// volume name 400s, and a declared volume with no store row answers
+// with an empty component list (a valid "nothing recorded yet").
+func TestDurabilityEndpointGuards(t *testing.T) {
+	vol := &config.Volume{Name: "pics", Path: t.TempDir()}
+	srv := newTestServer(t, Config{Volumes: map[string]*config.Volume{vol.Name: vol}})
+
+	if code := postDurability(t, srv, syncproto.DurabilityRequest{Volume: "ghost"}, nil); code != http.StatusNotFound {
+		t.Fatalf("undeclared volume status = %d, want 404", code)
+	}
+	if code := postDurability(t, srv, syncproto.DurabilityRequest{}, nil); code != http.StatusBadRequest {
+		t.Fatalf("missing volume status = %d, want 400", code)
+	}
+	var resp syncproto.DurabilityResponse
+	if code := postDurability(t, srv, syncproto.DurabilityRequest{Volume: "pics"}, &resp); code != http.StatusOK {
+		t.Fatalf("declared-but-unmaterialised volume status = %d, want 200", code)
+	}
+	if len(resp.Components) != 0 {
+		t.Fatalf("components = %+v, want empty", resp.Components)
+	}
+}