offload-v1: NAS offsite + offload, security-audited#126
Merged
Conversation
An optional [destinations.<name>.crypt] table declares client-side
encryption via rclone's crypt overlay: password (required) and
password2 (salt, optional), both rclone-obscured and accepting the
same literal / { env = "VAR" } forms as destination credentials.
A crypt destination renders two rclone.conf sections — the underlying
remote unchanged plus a crypt remote wrapping it at the destination
root, with filename encryption fixed off (contents only).
type=local is rejected (no rclone remote to wrap), unknown crypt
fields fail loudly like everywhere else in the config, and a sibling
destination occupying the <name>-crypt section name is refused.
Transfers, backup-dir, and the index ride-along for a crypt destination go through <name>-crypt:, whose remote line already carries the destination root; the three URI builders now share remoteSubpathURI. rclone crypt remotes expose no content hashes, so the default --checksum --hash blake3 verification cannot pass through the overlay. effectiveShallow downgrades such transfers to rclone's size+mtime comparison (the same path --shallow takes), the runs row records them as shallow so the audit trail stays honest, and a warning on the report surfaces the fallback unless the operator already asked for --shallow.
A crypt destination forces shallow comparison, so requiring the blake3- capable rclone floor for it blocks crypt-only setups on older rclone for a flag the run never passes. EnsureMinVersion callers now derive the preflight from the actual targets: ShallowForPairs over the invocation's pairs (sync, agent scheduler, desktop trigger) or EffectiveShallow for the single restore destination.
…rypt doc The section renderer always ends with a newline (tests pin it); the doc claimed the opposite. The resolveCrypt local-rejection rationale already lives in its error message.
config/sync: optional crypt encryption overlay for destinations
The core reshape behind the offload feature: content becomes the entity, paths become observations of it. - contents (v14): one append-only row per BLAKE3 digest carrying size_bytes and the content-level origin (origin_node_id NULL = introduced locally; origin_run_id is in the origin node's run space, so deliberately not a local FK). The files_blake3_immutable trigger is dropped - the id<->hash binding is immutable by construction. - files (v14): rekeyed to PK (folder_id, name, content_id); blake3 and size_bytes move to contents; the status CHECK gains 'offloaded' (intentional sibling of 'missing': current content whose local bytes were deliberately removed). The one-live-row-per-path partial unique index survives unchanged. - runs (v15): kind CHECK gains 'offload', joining index/audit in the destination-NULL branch. - destination_run_ids (+history) and remote_objects (v16): the per-destination durability version vector and the per-upload offsite fingerprint table; accessors land separately. Backfill maps each distinct hash to one contents row, taking size and origin from the files row with the earliest first_seen_run_id. The old source_* columns were per-observation sender attribution while origin_* is first-introduction provenance, so the earliest observation is the closest available approximation. Consumers follow the new shape: FileRow gains ContentID and renames Source* to Origin*, Upsert resolves content rows (recording prov as origin only for never-seen hashes), and ListPresentBySource becomes ListPresentByOrigin. Peer-sync keeps its current negotiation semantics, now reading content origin instead of row attribution; tests that relied on relabeling an existing row's attribution are reworked to introduce peer content the way a receiver's /close does.
destination_run_ids is the durability version vector: per (volume, destination, origin node), the highest origin-space run id known durable on that destination. Content with origin (N, r) is durable on a destination iff the vector's component for N is >= r. The upsert is monotonic - a backwards move is refused with a DestinationRewindError unless allowRewind is passed - and every advance appends a history row in the same transaction, mirroring the peer-sync watermark store. The rewind sentinel is shared with peer-sync (its message loses the peer-sync prefix), so errors.Is(err, ErrWatermarkRewind) matches both. remote_objects records the provider checksum captured at upload time for destinations whose bytes can't be cheaply re-read; verification re-fetches the provider value and compares strings, then stamps verified_at_ns. Inserts refuse duplicates so the recorded fingerprint can't be silently replaced. No callers yet - the offload and offsite upload flows wire these up in later PRs.
Drives a populated v13 fixture (duplicate content with mixed attribution, a peer-sourced live row, a superseded predecessor, a missing row) through the v14-v16 chain and pins the reshape: row counts, the hash<->content mapping with its earliest-observation size/origin backfill, preserved statuses and run stamps, duplicate detection across the shared content row, the surviving partial unique index, the dropped immutability trigger, the widened runs kind CHECK, and the new watermark store's rewind refusal against the migrated database.
An offloaded row's on-disk absence is intentional: MarkMissing only flips 'present' rows, so a re-index leaves the row offloaded and reports zero missing. When the file reappears with its recorded content, the next index run re-hashes it (the shallow shortcut only applies to 'present' rows) and the upsert flips offloaded back to present, preserving first_seen_run_id. Both behaviours fall out of the existing indexer machinery; these tests pin them ahead of the offload command landing.
…onotonicity getOrCreateContentTx now errors when a known digest is observed with a different size (corruption or a mis-hashing caller), and inserts with ON CONFLICT DO NOTHING + re-lookup so a concurrent writer of the same digest cannot fail the transaction. UpsertDestinationRunID enforces monotonicity inside the conditional DO UPDATE itself, so a racing advance can never be regressed by a stale writer; the rewind error is derived from the zero-rows outcome. MarkRemoteObjectVerified wraps its RowsAffected error like the surrounding methods.
store: split files into contents + files observations (schema v14–v16)
A kopia destination is a local-filesystem repository driven by the
kopia binary rather than an rclone remote: root is the repository
path and the only parameter is the required repository password
(literal or { env = "VAR" }, resolved like every other secret).
It renders no rclone.conf section, so the password stays out of
that file, and a crypt block is rejected because kopia encrypts
its repository itself.
GetOrCreateOriginNode maps a wire-carried origin node name to a local nodes row (names are the cross-node identity; ids differ per node), creating placeholder-endpoint rows for origins this host has never peered with. ValidNodeName exposes the name rule so the protocol layer can refuse bad names up front. ContentIntroductionRunID returns the earliest first_seen_run_id of a content in a volume — the origin-run coordinate the sender materialises for locally-introduced content.
Computes the per-origin-node maximum origin run over a volume's present files (NULL content origin maps to the self node at the observation's first_seen_run_id; reserved sync subtrees are excluded because they never travel to a destination) and advances each component through the monotonic UpsertDestinationRunID. A component already recorded higher stays — componentwise max, the version-vector join. Also adds ListVolumeDestinationRunIDs for the peer durability endpoint.
A destination's behavior is now owned by a curated, type-determined Handler: the rclone handler wraps the existing bucket Sync and the peer handler wraps the SyncNode handshake, both unchanged in behavior. RunPair resolves the handler from a Tools bundle instead of dispatching on the Pair slots directly. Each push now also reports a typed VerifyResult on the Report: which comparison backed the transfer (blake3, size+mtime, or the receiver-side peer re-hash) and the tool's counts. The verified flag is unexported and the Handler interface is sealed, so a positive durability claim can only be minted by the curated handlers in this package — the hook mechanism stays exit-code-only by construction, pinned by an external-package test.
IndexEntry gains (origin_node, origin_run): the content's global origin coordinate — origin node NAME plus that node's run id at introduction — carried verbatim across every hop, never relabelled to the immediate sender. The supersede/conflict disposition docs now describe delivery- based classification (who materialised the receiver's row) since origin no longer identifies the sender. The new session-less POST /v1/sync/durability exchange exposes a node's recorded destination durability vectors for a volume, so a peer can pull and cache cross-node durability evidence for offline decisions.
The receiver no longer attributes received content to the immediate sender: /close records each entry's declared (origin node name, origin run) on the contents row, resolving the name to a local nodes row (get-or-create — a forwarded origin may name a node this host never peered with). Entries without the pair (pre-origin-exchange initiators) fall back to the initiator at its declared sync run, which is the same initiator-run-space coordinate. Since origin now answers "where did the bytes first enter the system" rather than "who sent them", the supersede-vs-conflict classifier switches to delivery evidence: the receiver row's first_seen run and its peer linkage + correlated initiator id. A forwarded origin still supersedes through its forwarder; receiver-local writes and other peers' deliveries still conflict. The conflict pre-stage's drift relabel now clears the prior content's origin: drifted bytes are a fresh local introduction, and inheriting the old coordinate would make the durability vector vouch for content no destination holds. POST /v1/sync/durability (session-less, read-only) serves the node's recorded destination vectors for a volume with origin nodes resolved to names, backing the peer durability pull.
A node first met by name only (forwarded origin, durability pull before any sync) gets the peer://<name> placeholder endpoint; the first real handshake presenting an actual URL now upgrades the row in place. Genuine real-endpoint collisions stay refused.
The kopia handler drives the kopia binary as an opaque child process, mirroring the rclone wrapper's philosophy: squirrel owns the argv, every invocation runs against a squirrel-managed, destination-scoped config file next to rclone.conf, and the repository password reaches the child only via KOPIA_PASSWORD in its environment — never argv, logs, or error strings. A push connects the repository at the destination root (falling back to create on first use; connect runs every time so a moved root is re-pointed instead of silently snapshotting into the old repository), snapshots the volume path, and runs kopia's own `snapshot verify` scoped to the new manifest. The typed VerifyResult carries the manifest id plus the file/byte counts from the manifest's root summary, and the runs row matches other sync targets: kind='sync', destination=name, shallow=false since kopia verifies its own hashes. Restore refuses kopia destinations (the repository speaks kopia's CLI), dry-run is refused (kopia has no equivalent), kopia pairs are exempt from the rclone blake3 version preflight, and the index snapshot ride-along stays out of the repository (local tier only, like peer syncs). Tests cover argv/env construction and status mapping through a PATH-shim fake kopia, and a full connect→create→snapshot→verify cycle against the real binary that skips when kopia is not installed (verified locally against kopia 0.23).
ToolsFor resolves the kopia wrapper exactly when the matched pairs include a kopia destination, in the CLI sync flow, per kick in the agent's scheduler runner, and in the desktop's sync trigger — a host that never targets a kopia destination runs without the binary installed. The per-pair sync output renders the snapshot's own numbers (files, bytes, manifest id, verified) since kopia pushes have no rclone counters.
Covers config shape, the connect/create → snapshot → verify flow, the password-via-environment and per-destination config-file handling, and how kopia destinations differ from rclone ones (self-verified, no shallow/dry-run, no crypt block, restore via the kopia CLI). Also notes in the hooks section that verification results come only from built-in destination types.
A dry run transfers and checks nothing, so stamping the derived verification result would have reported a default bucket dry-run as verified. Both flows now stamp only real runs, matching the field's documented contract, with the dry-run path pinned by a test.
Every /plan entry now carries its content's origin coordinate. Content with a recorded origin forwards it verbatim — local node id resolved to the node NAME (the cross-node identity), origin run untranslated. Locally-introduced content is materialised as (self name, the content's introduction run in the volume — its earliest first_seen_run_id, so a later duplicate path doesn't shift the coordinate). At a verified successful close (never dry-run) the initiator records the durability consequence: AdvanceDestinationVector moves the peer's vector over the volume's present set (peers are destinations in the flat target namespace), then the new PullDurability fetches the peer's own destination components and merges them monotonically into the local destination_run_ids — metadata-only, refused rewinds surfaced as warnings with an allow-rewind override reserved for the standalone command. A failed vector advance fails the run; the next sync re-plans everything already-correct and re-advances cheaply. Existing tests that asserted immediate-sender attribution now assert verbatim origins, and rows seeded as receiver-local writes are seeded with genuinely local first-seen runs to match the delivery-based classifier. The new 3-node chain test pins the acceptance scenario: content introduced on alpha and forwarded bravo→charlie lands on charlie with alpha's origin (and a nodes row for alpha, a peer charlie never spoke to), while the forwarding hop still classifies cleanly.
Standalone entry point for the durability metadata pull that also runs automatically after a successful node sync: fetch the peer's destination vectors for a volume and merge them into the local index. Refused rewinds are printed and fail the command with a pointer at --allow-rewind, the explicit recovery override. The sync report now also prints how many peer components the automatic pull applied.
An id-less manifest would have scoped `snapshot verify` to an empty argument; surfacing the malformed manifest is the honest failure.
…ding - Strip any parent-shell KOPIA_PASSWORD before appending the configured one, so the child sees exactly one value (pinned by a stale-parent-export case in the happy-path test). - Stamp Verification.Method at the start of a kopia push so output renderers use the kopia format even when the push fails before a snapshot exists. - README: the runs row is never shallow, but verified is kopia's call (clean snapshot + passing verify), so say that instead of 'always fully verified'.
The self component now aggregates one coordinate per present content — its introduction run, the same MIN-first-seen coordinate the peer-sync sender materialises on the wire — instead of each observation's first_seen_run_id. A duplicate path observed later no longer advances the component past the coordinates actually in circulation, keeping the vector consistent with the origin space the gate compares against (Copilot review on PR 99). Also stop ignoring a GetSelfNode error in the transfer test.
The indexer captured size and mtime at directory-walk time, then hashed later from a fresh open with no re-stat. A file appended-to between the walk and the hash bound the new BLAKE3 digest to the stale walk size in an immutable contents row; every later observation of those true bytes then failed the size cross-check in lookupContentTx and aborted the whole ApplyIndexBatch repeatedly. hashFile now Stats the open handle after hashing and returns the size and mtime alongside the digest, so the row's metadata describes the same inode state as the bytes that produced the hash. Re-opening by path would reintroduce the race, so the live handle is used. Refs #107
uploadOneObject stat-guarded the source on size+mtime before rclone read it (a stat->copy race) and confirmed only size after upload, so a size+mtime-preserving in-place edit could upload bytes that did not match the recorded hash; the recorded remote_objects row then suppressed any future re-upload of the honest content forever. The pre-transfer guard now re-hashes the source with BLAKE3 immediately before copyto and refuses (errContentDrift) when the digest no longer matches the indexed hash. A refusal is surfaced as a warning and fails the run without recording an object or writing the manifest segment, so the watermark holds and the next run re-offers the object — the honest bytes land once restored, and the drifted bytes never bind to the hash. Residual: rclone reads the file in a child process after the re-hash, so an edit in that fork/exec window could still upload drifted bytes; the window is one rclone invocation rather than the whole walk-to-push span, and the scan-back fingerprint pass re-reads the landed object before it counts as content-verified. Refs #107
Address Copilot review on #123: - Scope-check before full validation. An out-of-scope destination is dropped before validateComponent/validateFreshness runs, so a junk entry for an unconfigured destination with a malformed origin can no longer abort the whole pull — restoring the intended fail-safe. - Reconcile the counts. Fetched now counts components and freshness together and a merged freshness coordinate counts as applied, so every fetched entry lands in exactly one of applied/rewind/dropped and Dropped can never exceed Fetched. - Bound the drop detail. Dropped stays exact, but the sampled Drops slice is capped (maxDurabilityDropSamples) and the sync warning is a single summary line, so a peer flooding out-of-scope destinations can't blow up the report or the output that renders it. The CLI printer prints the sample then a '… N more' tail. Tests: freshness-drop path, the sample cap under a 50-destination flood, and the count reconciliation.
config: add optional s3 storage_class and sftp host-key validation params
sync: scope durability pull to the volume's accepted destinations
index, sync: stop binding wrong bytes to a content hash (#107)
The v13->v14 reshape dropped files_blake3_immutable without installing an equivalent guard on the new contents table. v21 re-asserts the append-only contract the AGENTS.md guarantee implies: BEFORE UPDATE and BEFORE DELETE triggers on contents that RAISE(ABORT). Defense in depth -- there is no UPDATE/DELETE path on contents today. Regenerated store/schema.sql via -update-schema; golden test green. Extended the minimal v18 fixture with the contents table a real v18 DB carries, so the chain to v21 can attach the triggers. Addresses #113.
Adds a populated v18 fixture (contents, files, remote_objects, destination_run_ids) and drives it through the v19-v21 chain, asserting the offload-substrate rows survive intact (verify_method NULL-backfilled, remote_objects fingerprint preserved). Asserts the new v21 contents triggers abort an in-place UPDATE and a DELETE while the row is unchanged. Addresses #113 (#12c migration-shape coverage).
Offload defers to every run kind, but index and sync did not block on an in-flight offload, so a sync could enumerate (or an index could observe-and-flip) a tree mid-unlink. Add 'offload' to both begin-gate blocking sets to make the exclusion symmetric. Tests both directions: offload blocks a new sync and a new index/audit, and admits them again once it finishes. Addresses #114 (run-gate asymmetry).
hook_runs was the only runs-like table without FinishRun's terminal-state guard: a double finish silently overwrote the first terminal record. Read the status inside the finishing transaction and refuse with ErrAlreadyFinished when it is already terminal, leaving the row untouched. Test a double-finish is refused and the recorded outcome survives. Addresses #114 (FinishHookRun terminal guard).
The shared requireIndexedVolume gate looked the volume up by name only. A stale volumes.path would let a handler enumerate the config path's tree while the durability advance covered the DB volume's rows — a wholesale false durability claim. Refuse on mismatch, the same cross-check offload and restore already make. Covers every push handler (bucket, kopia, content-addressed, peer) since they share the gate. Test the mismatch is refused before rclone is invoked. Addresses #114 (config-path vs DB-volume-path agreement).
ensureRepository created a fresh repository on any connect failure. On a transient outage or a mistyped path that mints an empty repository while the destination's monotonic durability vector keeps claiming coverage the new repository cannot honour — and there is no CLI to rewind the vector. Require opt-in (the existing --init flag, which already gates the local marker bootstrap): without it, a connect failure is an error, not a silent create. Test connect-fail without --init refuses and never creates. Addresses #114 (kopia repo re-create guard).
rcloneVerification set verified purely from flags + exit-0. When a backend shares no hash with the source, rclone emits a NOTICE and silently falls back to a size comparison; parseJSONLog dropped NOTICE-level events, so the run was still recorded as blake3-verified. Detect the stable 'no hashes in common' phrase at any log level, flag it on RunResult, and downgrade the verification to size+mtime (unverified) so the durability vector does not advance on a copy rclone never content-checked. The guard only ever downgrades, so a false match cannot wrongly mark a run verified. Addresses #114 (--checksum blake3 silent degrade).
Schema & robustness hardening: contents immutability triggers (v21) + #114 cluster
rclone's sftp backend only autodetects md5sum/sha1sum, never a BLAKE3 command, so every non-shallow sync — which runs `rclone copy --checksum --hash blake3` — aborted with "hash type not supported", even when the remote has b3sum installed. Render `blake3sum_command = b3sum` in every sftp section so rclone can compute server-side BLAKE3.
config: emit blake3sum_command for sftp so `--hash blake3` syncs work
…loods Two cheap defence-in-depth guards on the peer durability pull. Neither is a trust control — the peer is trusted to assert its own durability — they turn a peer bug or version skew into a loud refusal instead of a silent local effect. - Add store.KnownVerifyMethod and refuse a non-empty verify_method the build does not recognise in validateComponent. Previously an unknown method was stored and then silently treated as unverified by the offload gate; now it is rejected at receipt. Empty (legitimately "unverified") still passes. - Cap distinct origins one pull will resolve (maxOriginNodesPerPull) so a runaway peer cannot grow the local nodes table without bound via GetOrCreateOriginNode. A real volume references a handful of origins. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KE9A1VTdw7k1GQq87fQyfx
Add an offload-v1 section to SAFETY-AUDIT.md framed for the single-operator deployment (laptop + NAS, all owned, adversary is entropy not a hostile peer): - D1 documents the relayed-evidence trust boundary in the durability pull (direct self-verified vs peer-asserted components), states the accepted decision to trust an in-domain NAS, and records the two defence-in-depth guards added alongside. - D2 documents that the content-addressed offsite push proves presence+size and decrypt-input correctness but not decrypt-output correctness, and sketches an opt-in, NAS-local read-back-decrypt-rehash verification as the proportionate close. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KE9A1VTdw7k1GQq87fQyfx
…ening durability: document the relayed-evidence trust boundary + defence-in-depth guards
This was referenced Jun 20, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overnight build of the squirrel offload-v1 feature set (NAS offsite + offload), followed by a four-auditor security audit focused on the append-only / never-delete-indexed-data guarantee, and fixing PRs for every finding. Nothing here has touched
main— this PR is for your review. All work landed onoffload-v1via--mergecommits (no squash/rebase).How to review
Each sub-PR below was merged into
offload-v1only after: CI green (build/vet/test + golangci-lint) → GitHub Copilot review addressed → acode-reviewerstandards pass (comment policy, public-repo genericness, conventions) addressed. The four deletion-gating PRs additionally got dedicated adversarial soundness re-audits.Feature PRs (in dependency order)
--no-persist-credentialsverifycommandAudit-fix PRs
--initgateAudit outcome
The append-only guarantee holds on the local/bucket/store side (zero
DELETE FROM; copy-only rclone verbs;--backup-diron every overwriting push; every prior SAFETY-AUDIT finding genuinely fixed). The risk was concentrated in durability-claim soundness (the thing that gatesoffloaddeletion) and the peer-receiver write path. All four criticals (#103, #105, #106, #115) and both highs (#107, #108) are fixed and merged.Decisions to review (your call)
squirrel sync --init(no more silent auto-create on connect-fail). The agent/scheduler never sets--init, so the first kopia sync must be bootstrapped interactively once. Intended (err-toward-refusing), documented in flag help.Deferred (open issues, tracked for you — not fixed tonight)
offload_requirestarget can never satisfy the gate (e.g. crypt path-mirrored).Infra side (separate repo)
Draft
mbertschler/infra#18shows the end-statenas.toml/squirrel.tomlfor kreisel against this CLI surface — blocked on this PR, no live changes to the NAS or any bucket. It surfaced the two config gaps fixed in #122 (s3 storage_class, sftp host-key validation).Schema
Migrations v14 → v21 (all additive after v13;
schema.sqlregenerated, golden test green). Since you're the only user and OK re-migrating, these can be collapsed before a public release if desired.Note
GitHub Copilot auto-review appeared rate-limited after ~16 PRs; #125 (final) merged on a clean standards review + green CI without a Copilot pass. Everything else got both.