Repository navigation
feat(studio): the NFX file store and "Also publish on NFX (free)" — contracts v8 - #35
Closed
SovereignTechnology wants to merge 10 commits into
Closed
SovereignTechnology wants to merge 10 commits into
SovereignTechnology wants to merge 10 commits into
Conversation
…d the NFX packager A move, not a change: the same argv-only run and the same MediaError mapping (aborted, ffmpeg-not-found, process-failed with stderr). pipeline.ts imports it; app-desktop's ipc/errors.ts comment points at its new home.
… esds A port of nfx-media's mp4.rs: avc1/avc3.PPCCLL from avcC, mp4a.OO.A from esds (with the object-type escape), video first. ffprobe reports no profile for a bare init segment, so the bytes the decoder is configured with are read, never a profile name. Bounds-checked box and descriptor walks; malformed input is a MediaError (process-failed), a codec NFX does not package is unsupported-input. The init-segment builders live in testing/fakes.ts for the packager's tests too.
…muxed into CMAF by stream copy `media.packageNfx` (a port of nfx-media's package.rs, ADR 0020, Phase 1 slice 2a): the renditions `transcode` already encodes (H.264 + AAC, keyframes pinned every GOP_SECONDS) are remuxed with `-c copy` into CMAF, so one encode serves the legacy MP4 and NFX. `cmafRemuxArgv` is nfx-media's CMAF tail (HLS VOD, fMP4, independent segments, init.mp4 + seg_%05d.m4s), with the input read through `file:` behind a protocol (file) and format (mov) whitelist. Then, as nfx-media does: every file named by its sha256, each rendition playlist rewritten to content names, a master playlist (BANDWIDTH = peak segment bitrate, RESOLUTION, CODECS from the init segment), and the hash list rendered with the `nfx` codec. One GOP grid is required across renditions; the result is checked as a consumer would (hash list against root, every file against its sha256 with the codec's own hash, every playlist against the content-name rule) before it is returned. Nothing is stored: each file says where its bytes are, for slice 2b's store. Read caps on ffmpeg's output and the receivers' 16 MiB hash-list limit; work directories are removed on failure. Tests: the packager against a fake ffmpeg (layout, names, bandwidth, master text, every refusal), the argv, and an end-to-end run with real ffmpeg (skipped without a binary, as node-real-ffmpeg.test.ts is).
core/media's real-binary suites (node-real-ffmpeg: the transcode pipeline; node-real-ffmpeg-nfx: the NFX CMAF remux by stream copy) skip when no ffmpeg is found, and only the e2e job installed one, so neither ever ran in CI. The test job now installs Ubuntu's ffmpeg the same way e2e does.
…r's disk cap, in no swarm ADR 0020, Phase 1 slice 2b. `nfx-store/store.ts` keeps the files of NFX videos by sha256 under `<dataDir>/nfx/`, beside the legacy Hyperblobs store and never inside it (the shared `blobs` core is priced and swarm-joined, so NFX files there would be served to the legacy swarm). - A put verifies the hash list against its root first (`nfx.verifyHashList`), takes the file list from that verified list, and checks each file's size and sha256 in memory before writing it through a `.part` file and an atomic rename. A refused put removes what it wrote and releases its reservation. - `file(video, sha)` serves only a file that video's hash list lists (or the hash list itself), with bounded, offset-checked reads; paths come only from validated lowercase sha256. `source()` is the transport's `NfxFileSource` shape, without importing it. - Files are refcounted across videos; removing a video deletes what no other video lists. - One disk pool: `Seeder.create` loads the store beside the CAS index, the `DiskCap` starts from both stores' bytes, and puts reserve against it. `seeder.nfx` exposes the store. - `SeederFs.readRange?` (Node and Bare, ranged `createReadStream`); without it the store falls back to the stream.
…oth hops, and upload.nfx CONTRACTS_VERSION 7 -> 8, additive (every v7 consumer compiles): `UploadInput.nfx?` asks for the upload to be packaged for NFX as well (ADR 0020). Free videos only (`satsPerBlock` 0, issue #15), so the free NFX path never undercuts a paid video. - ipc/guards.ts: `nfx` is an optional boolean of the upload meta, and `nfxOnlyIfFree` refuses it on a priced upload at the renderer → host boundary. - ipc/worker-guards.ts: the same rule on the host → worker hop (`studio.upload` and the `studio.publish` draft's meta), and the new worker event `upload.nfx`: `stored` (NFX-01 video address, root, segs, renditions, bytes) or `failed` with one of four reasons, nothing else. - worker-protocol.ts: `NfxUploadStatusWire`, `NfxUploadFailure`, the event. - host/adapter.ts logs `upload.nfx` (publishing it is slice 4, showing it slice 5). - preload/bridge.ts forwards `nfx: true` only. - Guard samples: a free NFX upload passes; `nfx` on a priced video, a non-boolean `nfx`, a malformed status (another token, uppercase, a leading zero in segs, no renditions, an unknown reason, an extra field) are refused.
…the NFX store ADR 0020, Phase 1 slice 2b. With `meta.nfx`, once the legacy upload has published, `studio/nfx.ts` packages the H.264 renditions it encoded as NFX-05 CMAF by stream copy (`media.packageNfx`), stores every file and the hash list in `seeder.nfx` (verified there, counted in the shared disk cap), removes ffmpeg's work directories, and reports `upload.nfx`. - Detached from the legacy result: the remux and the store's writes never delay or fail the legacy upload; every failure is a typed status (`unsupported`, `packaging-failed`, `disk-cap`, `store-failed`), never a throw. - Free videos only, checked again here: a priced upload is `unsupported`, before ffmpeg runs. - The NFX address: `nfx:testnet:1` (one constant, `NFX_DEFAULT_NAMESPACE`, until Settings choose a network in slice 5) and the video id `nutflix-<first 32 hex of the legacy event id>`: deterministic, inside NFX-02's grammar, and the link slice 4 needs to tie the two together. - Tests: the derivation and its refusals; priced, VP9 and failed-remux uploads report their status and store nothing; with the system ffmpeg, a free upload with `nfx` still returns its legacy manifest, then `upload.nfx` is `stored`, the hash list in the store verifies against its root, every file it lists is served for that video, and the disk cap counts them.
…e video ADR 0020, contracts v8. `StudioDraft.nfx?` (default off) and `nfxEligible`: a price of 0 only, so the free NFX path never undercuts a paid video. The choice sits under the price, as a card like the format choice, and `toUploadInput` sends `nfx` only for an eligible draft (a stale choice on a priced draft is dropped). Free videos cannot be uploaded yet (issue #15: the price must be 1 or more), so the choice is not shown until #15 lands; the worker path is reachable over IPC and tested end to end. CHANGELOG [Unreleased] "Added".
no-unnecessary-condition: `output` is set inside the transcode callback, but TypeScript narrows it from its `null` initializer, so `done !== null` read as always false; it is now read through a function. unbound-method: `UploadDeps.nfx` is declared with `this: void`.
…not the upload step The previous commit's `this: void` on UploadDeps.nfx answered an error that was really nfx-store.test.ts:232 (destructuring readRange off the fs adapter), and itself tripped no-invalid-void-type. It is reverted; the test now copies the adapter and deletes the optional readRange.
SovereignTechnology
added a commit
that referenced
this pull request
Oct 25, 2026
…ontracts v8 (#35) Phase 1 slice 2b (ADR 0020). A free upload can also be packaged for NFX and stored, ready for slice 4 to publish. - NFX file store (packages/seeder/src/nfx-store): sha256-named files written via .part + rename after their size and sha256 are checked against the verified hash list, refcounted across videos, served only for files a video's hash list names, joined to no swarm. - One disk cap shared with the legacy store: a put reserves its new bytes first and fails before writing anything that would exceed it. - Contracts v8 (additive): UploadInput.nfx and the upload.nfx status event; free videos only, refused at the renderer→host guard, the host→worker guard and in the worker. - Worker: packageNfx runs detached after the legacy upload publishes, so it never delays or fails it. Video address nfx:testnet:1:nutflix-<first 32 hex of the legacy event id>. - Studio: the toggle, shown only for a free video (it appears once 0-sat uploads, #15, land).
Owner
Author
|
Squash-merged as 600195a (main, as sovtech). |
SovereignTechnology
added a commit
that referenced
this pull request
Oct 25, 2026
…als, a load that never stops the seeder (#37) The fixes from #36's adversarial review, ported to the store #35 merged: - A list naming a stored file (or the stored hash list) at another size, or one sha256 at two sizes, is refused; before, the record took the false size and skewed totalBytes, file() sizes and later cap frees. - A put is saved as a pending record before its first write, and a load removes a pending put's files and .part files; a removal deletes files before it saves the records. - At load the size on disk decides which records are true; a malformed record is dropped on its own; a videos.json that is not one, or of another version, is set aside. - load() never throws: a store that cannot load is off (loadError, puts 'unavailable'). - Hash lists over 16 MiB are refused; buffers are copied before they are checked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 slice 2b of the NFX interop plan (ADR 0020): the NFX file store and the Studio wiring. A free upload can now also be packaged for NFX and stored, ready for slice 4 to publish.
Stacked on #33 (the packager): until #33 is merged, this diff includes #33's commits.
verifyalso needs #34 (the undici fix). Merge order: #34, #33, then this.Decisions taken (Cameron: "continue building"); easy to change
What's in it
packages/seeder/src/nfx-store/):<dataDir>/nfx/files/<aa>/<sha>. Each is written to a.partfile and then renamed, after its size and sha256 have been checked against the verified hash list.file(video, sha)serves only files that video's hash list names, with bounded reads.blobscore would leak NFX files into the legacy swarm.source()has the shape of the transport'sNfxFileSource(feat(seeder/nfx): the NFX-12 HyperDHT transport — dial, listen, files/1 both sides, pay/1 lane #32).DiskCapstarts from both stores. A put reserves its new bytes first, so a put that would exceed the cap fails before anything is written, and a removal frees bytes no other video lists.UploadInput.nfx?, and theupload.nfxevent with a typed status (stored, orfailed:unsupported/packaging-failed/disk-cap/store-failed).packageNfx(feat(core/media): NFX-05 packaging — the studio's H.264 renditions remuxed into CMAF by stream copy #33) runs detached, so it can never delay or fail the legacy result. The files and the hash list go into the store, and the work directories are removed.nfx:testnet:1:nutflix-<first 32 hex of the legacy NIP-71 event id>.alink needs.Known limit
The toggle doesn't appear yet. Studio doesn't accept a 0-sat price until zero-price videos (issue #15) are implemented. The store, IPC and worker path are all in place and tested end to end: a new real-ffmpeg test runs a 0-sat upload through the worker, remuxes and stores it, and ends with
upload.nfxstored.Residual risks
file()scans the video's file list on each call.Review and CI
verify,test,e2e,package) on this branch with fix(deps): undici 7.30.1 and 6.29.1 — clear the high npm audit advisory that fails verify #34 merged in, from astage-check/push that's since deleted. It took two fixes along the way:b687294: a narrowing false positive on the transcode output.8e7e651: anunbound-methodfinding in the store test.Merge
Squash, after #34 and #33.