Repository navigation
feat(manifest): cover an archive on a NAS — streamed hashes, excludes, links - #20
Merged
Merged
Conversation
…, links Three things stood between a manifest and a real photo share, found by covering one: a Synology with 80,857 photographs and videos, 1.66 TB, and about 2 GB of memory. - Hashing read every file whole, so a 6.4 GB video needed 6.4 GB of memory. A file over 64 MiB is now hashed 8 MiB at a time through the storage port's new `read_chunks` (`Graph::digest_file`, `push_file_digest`, `fixity::Hasher`, `Digests::push_digest`), for every payload checksum prov records: manifest rows, sidecars, inline rows, and `check`. On the share, 11 GB with that video hashed in 41 MB. - A manifest can say what it leaves out: `exclude:`, gitignore-style patterns written by `attach --manifest --exclude`. An excluded directory is not walked, and its files are neither rows nor drift. On the share, Synology's `@eaDir` thumbnails had been 82% of the rows. - The covered walk follows symbolic links, so an archive kept on another share can be linked in as the manifest's root: a link to a file is that file, a link to a directory is walked, a dangling link is not listed, and more than eight linked directories on one path is refused as a loop. `prov ignore` rules a linked root out as claimed, without the trailing `/` gitignore would not match a link by. Before, a linked directory below the root was listed as a file and its hash failed. Needs fs-transaction with `ReadStorage::read_chunks`, not yet published. Behavioural-change: a symbolic link to a directory under a manifest's root is walked rather than listed as a file, and a dangling one is no longer listed. Behavioural-change: `prov ignore` lists a manifest root that is a symbolic link as claimed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Covering a real photo share (a Synology: 80,857 photographs and videos, 1.66 TB, about 2 GB of memory) turned up three things a manifest needed.
ReadStorage::read_chunks(fs-transaction 0.4.1), for every payload checksum: manifest rows, sidecars, inline rows,check. On the share: 11 GB including that video, 41 MB peak.exclude:on a manifest (attach --manifest --exclude PATTERN), a gitignore subset. Excluded directories are not walked; their files are neither rows nor drift. Synology's@eaDirthumbnails had been 82% of the rows.prov ignorerules a linked root out as claimed (/photo, no trailing slash).Whole share, unhashed, through a linked root with the Synology excludes: 80,857 rows in 3.6 s;
checkclean in 1.5 s.Behavioural changes are recorded on the commit.
docs/manifests.md§3.2–3.4 describe all three.Checked locally: fmt, clippy
-D warnings,cargo test --workspace, both guide transcripts,prov check, package isolation. MSRV not run locally.