Skip to content

fix(fixity): size the hashing budget to the threads that drain it - #21

Merged
adammharris merged 1 commit into
mainfrom
fixity-budget-by-threads
Oct 28, 2026
Merged

adammharris merged 1 commit into
mainfrom
fixity-budget-by-threads

Conversation

@adammharris

Copy link
Copy Markdown
Member

A fixed 256 MB of read-but-unhashed bytes is right for a laptop and too
much for a two-core NAS, whose hash threads also lose the CPU to the
filesystem and everything else it runs. Covering an 80,857-file photo
share there peaked at 407 MB on 2 GB of RAM.

The budget is now 32 MiB per rayon thread, still capped at 256 MiB. With
two threads and 60 MB files read from cache, peak memory falls from
294 MB to 122 MB at the same speed; eight threads are unchanged.

A fixed 256 MB of read-but-unhashed bytes is right for a laptop and too
much for a two-core NAS, whose hash threads also lose the CPU to the
filesystem and everything else it runs. Covering an 80,857-file photo
share there peaked at 407 MB on 2 GB of RAM.

The budget is now 32 MiB per rayon thread, still capped at 256 MiB. With
two threads and 60 MB files read from cache, peak memory falls from
294 MB to 122 MB at the same speed; eight threads are unchanged.
@adammharris
adammharris merged commit 1c10a28 into main Oct 28, 2026
8 checks passed
@adammharris
adammharris deleted the fixity-budget-by-threads branch October 30, 2026 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant