Skip to content

feat(manifest): cover an archive on a NAS — streamed hashes, excludes, links - #20

Merged
adammharris merged 2 commits into
mainfrom
manifest-nas
Oct 28, 2026
Merged

adammharris merged 2 commits into
mainfrom
manifest-nas

Conversation

@adammharris

Copy link
Copy Markdown
Member

Covering a real photo share (a Synology: 80,857 photographs and videos, 1.66 TB, about 2 GB of memory) turned up three things a manifest needed.

  • Streamed hashes. Every file was read whole before hashing, so a 6.4 GB video needed 6.4 GB of memory. Files over 64 MiB are now hashed 8 MiB at a time through ReadStorage::read_chunks (fs-transaction 0.4.1), for every payload checksum: manifest rows, sidecars, inline rows, check. On the share: 11 GB including that video, 41 MB peak.
  • exclude: on a manifest (attach --manifest --exclude PATTERN), a gitignore subset. Excluded directories are not walked; their files are neither rows nor drift. Synology's @eaDir thumbnails had been 82% of the rows.
  • Links. The covered walk follows symbolic links, so an archive on another share can be linked in as the root: a dangling link is not listed, more than eight linked directories on one path is refused as a loop, and prov ignore rules a linked root out as claimed (/photo, no trailing slash).

Whole share, unhashed, through a linked root with the Synology excludes: 80,857 rows in 3.6 s; check clean in 1.5 s.

Behavioural changes are recorded on the commit. docs/manifests.md §3.2–3.4 describe all three.

Checked locally: fmt, clippy -D warnings, cargo test --workspace, both guide transcripts, prov check, package isolation. MSRV not run locally.

…, links

Three things stood between a manifest and a real photo share, found by
covering one: a Synology with 80,857 photographs and videos, 1.66 TB, and
about 2 GB of memory.

- Hashing read every file whole, so a 6.4 GB video needed 6.4 GB of
  memory. A file over 64 MiB is now hashed 8 MiB at a time through the
  storage port's new `read_chunks` (`Graph::digest_file`,
  `push_file_digest`, `fixity::Hasher`, `Digests::push_digest`), for
  every payload checksum prov records: manifest rows, sidecars, inline
  rows, and `check`. On the share, 11 GB with that video hashed in 41 MB.
- A manifest can say what it leaves out: `exclude:`, gitignore-style
  patterns written by `attach --manifest --exclude`. An excluded
  directory is not walked, and its files are neither rows nor drift. On
  the share, Synology's `@eaDir` thumbnails had been 82% of the rows.
- The covered walk follows symbolic links, so an archive kept on another
  share can be linked in as the manifest's root: a link to a file is that
  file, a link to a directory is walked, a dangling link is not listed,
  and more than eight linked directories on one path is refused as a
  loop. `prov ignore` rules a linked root out as claimed, without the
  trailing `/` gitignore would not match a link by. Before, a linked
  directory below the root was listed as a file and its hash failed.

Needs fs-transaction with `ReadStorage::read_chunks`, not yet published.

Behavioural-change: a symbolic link to a directory under a manifest's root is walked rather than listed as a file, and a dangling one is no longer listed.
Behavioural-change: `prov ignore` lists a manifest root that is a symbolic link as claimed.
@adammharris
adammharris merged commit 6342218 into main Oct 28, 2026
8 checks passed
@adammharris
adammharris deleted the manifest-nas branch October 28, 2026 01:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant