Repository navigation
feat(seeder, worker/studio): the NFX file store, and a free upload packaged into it - #36
Closed
SovereignTechnology wants to merge 3 commits into
Closed
SovereignTechnology wants to merge 3 commits into
SovereignTechnology wants to merge 3 commits into
Conversation
NfxFileStore keeps NFX videos' files (ADR 0020, NFX-05) under their sha256 in <dataDir>/nfx/files, apart from every Hypercore, so none of them is served on the legacy swarm. A put verifies the hash list against root, address and segs, then each file's size and sha256 against the list, and keeps nothing on a mismatch. A file two videos list is stored and counted once. The index records a put as pending before its first file is written, so a load removes what an interrupted put left; a load also drops a video whose files are missing or of the wrong size. The seeder sizes its one DiskCap over the Hyperblobs payload and these files.
…egacy publish storeNfxUpload remuxes the renditions the studio already encoded into NFX-05 content (media.packageNfx, no second encode) and stores it in the seeder's NFX file store. runWorkerUpload does this only for a free upload whose renditions are all H.264 in MP4 (ADR 0020: Phase 1 is free videos only, so a free NFX copy never undercuts a paid video), and only when it is given WorkerNfxDeps; the host passes none yet, so uploads stay legacy only until NFX has its settings (slice 5). NFX never fails an upload: a failure is logged and the upload goes on, and a failed legacy publish removes the NFX video again.
…afe order, never fail the seeder
From the pre-push adversarial review of the NFX file store:
- put() declared `listed` twice (a compile error); the verified list is now `verified`.
- A list naming a stored file (or a stored hash list) at another size, or one sha256
at two sizes, is refused: before, it made the index disagree with the disk, and the
next load dropped every video sharing that file, or the cap counted a false size.
- At load the size on disk decides: only the records it contradicts are dropped.
- Removals and rollbacks delete files before saving the index, and temporary files
have one fixed name per sha256 that a load removes, so a crash never leaves a file
no record names.
- A malformed index entry is dropped on its own; an index that is not one, or of
another version, is set aside instead of written over.
- load() never throws: a failure leaves NFX off ('unavailable'), not the seeder down.
- Hash lists over 16 MiB are refused, and buffers are copied before they are checked.
- storeNfxUpload catches a throwing video-id source, and returns (and so rolls back)
only a video it stored itself.
- putVideo documents that a peer's video and root must come from a verified manifest.
Owner
Author
|
Superseded: #35 merged its own store first; the review fixes from this PR are ported to it in the follow-up PR (fix/nfx-store-review). |
SovereignTechnology
added a commit
that referenced
this pull request
Oct 25, 2026
…als, a load that never stops the seeder (#37) The fixes from #36's adversarial review, ported to the store #35 merged: - A list naming a stored file (or the stored hash list) at another size, or one sha256 at two sizes, is refused; before, the record took the false size and skewed totalBytes, file() sizes and later cap frees. - A put is saved as a pending record before its first write, and a load removes a pending put's files and .part files; a removal deletes files before it saves the records. - At load the size on disk decides which records are true; a malformed record is dropped on its own; a videos.json that is not one, or of another version, is set aside. - load() never throws: a store that cannot load is off (loadError, puts 'unavailable'). - Hash lists over 16 MiB are refused; buffers are copied before they are checked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 slice 2b of the NFX interop plan (ADR 0020): where NFX files live on a seeder, and the studio step that puts a free upload there. Stacked work from #32 and #33, which are now on
main.What's in it
NfxFileStore(packages/seeder/src/store/nfx-files.ts):<dataDir>/nfx/files/, apart from every Hypercore, so none of them is served on the legacy swarm.root, the address andsegs, then each file's size and sha256 against the list, and keeps nothing on a mismatch.DiskCapover the Hyperblobs payload plus these files.pendingbefore its first write, and a load removes its files.packages/app-desktop/src/worker/studio/nfx.ts):media.packageNfx, stream copy) and stored before the legacy publish.Off until slice 5
runWorkerUploadpackages only when givenWorkerNfxDeps(the network and a video-id source), and the host doesn't pass them yet. Uploads stay legacy only.The free switch is decided: an option for uploaders to publish free and for seeders to seed for free. It comes with the slice 5 settings, since the Studio today requires at least 1 sat.
Review before push
An independent adversarial review of the first two commits found 14 items. The third commit fixes every one that applies to this slice, with tests:
Left for slice 5, and noted in the code:
video/rootmust come from a manifest the caller verified, because the first put for an address wins.Load checks sizes, not content. A same-size corruption is caught by the receiver's sha256 check (NFX-05 §4).
CI
Not yet run. This PR is the first run.
Merge
Squash, locally as sovtech.