Skip to content

fix(seeder/nfx-store): one size per sha256, crash-safe puts and removals, a load that never stops the seeder - #37

Closed
SovereignTechnology wants to merge 2 commits into
mainfrom
fix/nfx-store-review
Closed

SovereignTechnology wants to merge 2 commits into
mainfrom
fix/nfx-store-review

Conversation

@SovereignTechnology

Copy link
Copy Markdown
Owner

#35 merged its own NFX store while #36 built the same slice. #36's store went through an independent adversarial review, and the same flaws turned up in #35's. This ports the review's fixes into the store on main, keeping its API, which the Studio wiring uses. #36 is closed as superseded.

Fixes (packages/seeder/src/nfx-store/store.ts)

  • One size per sha256.
    • Before: a list naming a stored file, or a stored hash list, at another size was recorded at the false size (:286, :311). That skewed totalBytes, file() sizes and later cap frees. A list giving one sha256 two sizes was first-wins (:271). Once seeders store what peers send, a peer could do this at will.
    • Now: both are refused.
  • Crash leftovers. The fs adapter can't list a directory, so nothing may be left on disk that no record names.
    • A put is saved as a pending record before its first file is written, and a load removes a pending put's files and .part files.
    • A removal deletes files before it saves the record list. Before, it saved first, so a crash left files nothing counted.
  • The disk decides at load. A record naming a missing file, or a file of another size, is dropped with its video. Files only it named are removed.
  • Damaged index.
    • A malformed record is dropped on its own; before, one bad record emptied the store.
    • A videos.json that isn't one, or of another version, is set aside, not overwritten.
  • Load never throws. A store that can't load is off (loadError, puts return unavailable), and the seeder still starts. Before, it took the legacy seeder down too.
  • Limits. Hash lists over 16 MiB are refused, and buffers are copied before they're checked.
  • Logging. The seeder now logs a load failure and any videos dropped at load.

Tests (nfx-store.test.ts)

Each finding gets a case:

  • a stored file named at another size, including the stored hash list itself;
  • one sha256 listed at two sizes;
  • a hash list over 16 MiB;
  • an interrupted put with a .part file;
  • the disk deciding between two records;
  • a malformed record, and a newer-version index;
  • a store that can't load.

Review

This applies findings from #36's adversarial review, so it was not reviewed again separately. The new paths are built only from validated hex, as before.

Merge

Squash, locally as sovtech.

…als, a load that never stops the seeder

From an independent adversarial review of the same design (#36), applied to the store #35
merged:
- A list naming a stored file (or a stored hash list) at another size, or one sha256 at two
  sizes, is refused. Before, the record took the false size: totalBytes, file() sizes and
  later cap frees went wrong, and a peer-supplied list could have that effect at will.
- A put is saved as a `pending` record before its first file is written; a load removes a
  pending put's files and .part files. A removal deletes files before it saves the records.
  Without a directory listing, nothing is left on disk that no record names.
- At load the size on disk decides: a record naming a missing file, or one of another size,
  is dropped with its video, and the files only it named are removed.
- A malformed record is dropped on its own; a videos.json that is not one, or of another
  version, is set aside instead of written over.
- load() never throws: a store that cannot load is off (`loadError`, puts `unavailable`).
- Hash lists over 16 MiB are refused; buffers are copied before they are checked.
SovereignTechnology added a commit that referenced this pull request Oct 25, 2026
…als, a load that never stops the seeder (#37)

The fixes from #36's adversarial review, ported to the store #35 merged:
- A list naming a stored file (or the stored hash list) at another size, or one sha256 at two
  sizes, is refused; before, the record took the false size and skewed totalBytes, file()
  sizes and later cap frees.
- A put is saved as a pending record before its first write, and a load removes a pending
  put's files and .part files; a removal deletes files before it saves the records.
- At load the size on disk decides which records are true; a malformed record is dropped on
  its own; a videos.json that is not one, or of another version, is set aside.
- load() never throws: a store that cannot load is off (loadError, puts 'unavailable').
- Hash lists over 16 MiB are refused; buffers are copied before they are checked.
@SovereignTechnology

Copy link
Copy Markdown
Owner Author

Squash-merged as 625ca50 (main, as sovtech).

@SovereignTechnology
SovereignTechnology deleted the fix/nfx-store-review branch October 25, 2026 03:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant