Skip to content

Live REPL, extra effect folders, opt-in hardware video decode - #299

Closed
BuddhiLW wants to merge 9 commits into
kevinraymond:mainfrom
BuddhiLW:feat/live-repl
Closed

BuddhiLW wants to merge 9 commits into
kevinraymond:mainfrom
BuddhiLW:feat/live-repl

Conversation

@BuddhiLW

@BuddhiLW BuddhiLW commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Three opt-in features, each off by default and absent from the release downloads.

Live REPL (--features repl). An embedded clojurust nREPL: start with FOSFORA_NREPL=<port> and connect CIDER, Calva or Conjure. (require '[fosfora :as f]) is preloaded; f/media!, f/ls-media and f/tessera open images and videos as media layers. Local paths only come from the in-process REPL, never from a WebSocket or LAN client. The cljrs crates are git dependencies pinned to a rev of BuddhiLW/clojurust staging (upstream csm/clojurust plus two fixes under review there, csm#412 and csm#413), so a clean clone and CI build without a sibling checkout. Once those land upstream the pin can move to csm/clojurust.

Extra effect folders. FOSFORA_TRAMA_DIRS and FOSFORA_EFFECT_DIRS list directories whose trama .wgsl and .pfx effects load after the built-ins (built-in indices do not move) and hot-reload when an external generator writes to them. A relative shader in an extra .pfx resolves against that file's own directory.

Hardware video decode. FOSFORA_HWACCEL=cuda|vaapi|qsv|vulkan|auto adds -hwaccel and a cheaper chroma scaler to the media-layer ffmpeg command; the value must be a plain identifier. On a 16 s 1080p30 H.264 clip, CPU time went from 17.0 s to 6.5 s.

Checked locally: cargo check for default, repl, and repl,video; cargo clippy -p fosfora-app --features repl -- -D warnings; cargo fmt --check; scripts/check_changelog.py. CHANGELOG entries are under Unreleased.

FOSFORA_NREPL=<port> (0 = any free port) starts a clojurust interpreter on a
`cljrs-repl` thread with an nREPL server on 127.0.0.1, writing .nrepl-port so
CIDER/Calva/Conjure auto-connect. `(require '[fosfora :as f])` is preloaded.

Commands reuse the web remote's vocabulary: the `fosfora` namespace pr-strs a
message map, `fosfora.native/send-edn` reads it with cljrs-reader, converts it
to JSON and runs web::client::parse_client_message, so the REPL and the
WebSocket remote cannot drift apart. Messages go through a WebSystem queue that
outlives server restarts and is drained with the network queue in update().
Reads come from the snapshots the web system already builds (full state and
audio at 10 Hz), kept fresh while the REPL is attached even with no client.
`f/put-trama!` writes a trama effect file for the hot-reload watcher.

clojurust values are !Send, so the runtime lives on its own thread and is
driven by cljrs_nrepl::Poller; its per-thread GC never pauses rendering.
Interpreter only: no JIT and no git-deps tree.

SPIKE: cljrs crates are relative path deps resolved from this worktree, because
the clojurust HEAD in use is unpushed. Switch to a pinned git rev before merge.

Verified live: load-effect!, set-param! (read back from f/state), error on an
unknown message, and CIDER eval + completion over the embedded server.
FOSFORA_HWACCEL=<method> (cuda, vaapi, qsv, vulkan, auto) adds -hwaccel
<method> before the clip's input and -sws_flags fast_bilinear to the output.
Unset, empty or `none` leaves the ffmpeg command unchanged. The method must
be a plain identifier, so the variable cannot smuggle extra arguments.

Frames are still downloaded for the RGBA pipe, so decode alone moving to the
GPU saves little: the YUV to RGBA conversion dominates. The clip is never
resized, so fast_bilinear only affects chroma upsampling. Measured on a 16 s
1080p30 H.264 clip (RTX 4070 Laptop, ffmpeg 8.0.1), CPU time:
  stock 17.0 s, -hwaccel cuda 13.2 s, fast_bilinear 8.3 s, both 6.5 s.
scale_cuda cannot convert nv12 to rgb0 in this ffmpeg, so the conversion stays
on the CPU; uploading NV12 and converting in a shader is the larger follow-up.
(f/media! path) decodes an image or video into a new media layer through
App::start_media_layer, the same background decode the "add media layer"
dialog uses; the layer becomes active when ready. (f/ls-media dir) lists the
playable files in a folder, and (f/tessera [pick]) lists or opens the loops
in ~/PP/mentat-collective/tessera/out/hd by file-name substring.

The command is WsInMessage::AddMedia, which parse_client_message never
produces: only the in-process REPL can name a local path, never a WebSocket
or LAN client. app.rs collects the paths inside the active-layer borrow and
starts them right after it.

The pre-decode path (clips up to 60 s) now takes FOSFORA_HWACCEL too. It does
not shorten the load (16 s 1080p: 3.3 s before, 3.6 s after; the 3.8 GB RGBA
copy dominates), but keeps decode off the CPU.
FOSFORA_TRAMA_DIRS lists directories (split like PATH, leading ~ expanded)
whose *.wgsl files are trama effects exactly like assets/trama/effects/:
loaded at startup, a new file registers live, a change hot-swaps, a broken
save keeps the last good version. An external generator can write straight
into one; the first user is tessera, whose desargues emitter writes field
loops to tessera/out/layers/.

Load order is the built-in directory, then each extra directory; an id
defined twice takes the last definition, which is also what a hot reload
does (the changed file wins). The watcher recognises the built-in directory
by name (trama/effects) and an extra one by being listed, watched
non-recursively and seeded like the built-in one so startup reads are not
reported as changes. A shader-library change reloads every directory.

Tested end to end through the real watcher: a file written into an extra
directory arrives on the trama route, not the layer-shader one.
FOSFORA_EFFECT_DIRS lists directories (same rules as FOSFORA_TRAMA_DIRS)
whose *.pfx files are layer effects like assets/effects/: they appear in the
catalog (under "Yours" unless authored "Fosfora"), load onto layers, save in
presets by name, and hot-reload when the .pfx or one of its shaders changes.
Each directory is watched recursively, so a .pfx and its .wgsl can sit side by
side. The first user is tessera, whose loops get a catalog entry each.

A relative `shader` in an extra directory's .pfx resolves against that .pfx's
own directory (effect::dirs::localize), at scan and again on every hot reload,
so the reload path cannot point it back at assets/shaders. Built-in effects
and absolute paths are untouched. Extra effects are appended after the
built-ins, so built-in indices do not move.

The built-in scan now reads each file through the same read_pfx helper, which
keeps the interaction_3d fixup in one place.
…ath deps

Cargo resolves optional path deps even with the feature off, so any clone
without a sibling clojurust checkout (CI included) could not build at all.
The cljrs crates now come from the fork's staging line (upstream
csm/clojurust plus fixes still under review there). Local development can
point back at a checkout with a [patch."https://github.com/BuddhiLW/clojurust"]
table in a .cargo/config.toml above the repo.

Builds that the old path setup hid:
- cljrs_nrepl::Config gained a `limits` field upstream; take its default.
- repl/media.rs used VIDEO_EXTENSIONS unconditionally, which only exists
  with the `video` feature, so `--features repl` alone did not compile.
- archery moved to 1.2.2 in Cargo.lock to satisfy cljrs-value's exact pin.
@kevinraymond

Copy link
Copy Markdown
Owner

Howdy Pedro,

Thanks for this! The commit messages are clear, and I like that you measured the CPU savings on the hardware decode.

I can't take it as one PR, though. Three unrelated features in one change is too much to review, so please split it into two PRs: one for the effect folders and one for the hardware decode. Going forward, please keep each PR to a single feature.

I'm going to pass on the embedded REPL, for three reasons:

  1. It adds 47 packages to Cargo.lock. They're git dependencies, so Cargo fetches them to resolve the graph even with the repl feature off. Every build of Fosfora would then need your fork reachable at that rev. With default features, cargo tree --offline fails on cljrs-gc.
  2. The cljrs crates are EPL-1.0. Fosfora is MIT or Apache-2.0, and I don't want EPL in the dependency tree, even behind a feature. The dependency audit (cargo deny) fails on it too.
  3. The WebSocket already covers it. Every REPL command becomes a web remote message through parse_client_message, and the reads come from the same snapshots the remote gets. A REPL outside Fosfora can do all of that over the WebSocket without adding anything to Fosfora. That's how I'd like to see it built. Opening a local file is the one thing the remote can't do, on purpose. If you need that from a script, open an issue and we can work out a safe way to do it.

The effect folders PR (ffeb397 and 47e9ac3) has one bug to fix first. read_pfx canonicalizes the .pfx path before it calls localize (effect/dirs.rs:75), but the listed folders are only made absolute (trama/dirs.rs:53). So when a folder in FOSFORA_EFFECT_DIRS goes through a symlink, is_under fails at startup and the shader path stays relative to assets/shaders. A hot reload uses the watcher's path and works, so the bug looks intermittent. Localizing before canonicalizing fixes it, and a test with a symlinked folder would keep it fixed. Also, Fosfora uses US spelling, so recognised in trama/dirs.rs should be recognized.

The hardware decode PR (a19cc26) is ready as it is. The two lines that add FOSFORA_HWACCEL to the pre-decode path in video.rs are in the REPL media commit (1de29ed), so bring those along.

Each PR can take its CHANGELOG entry from this one. They read well.

Thanks again!

@BuddhiLW

Copy link
Copy Markdown
Contributor Author

Thanks Kevin. I've split this as you asked:

Each one takes its CHANGELOG entry from here. I've dropped the embedded REPL. If I need to open local files from a script, I'll open an issue first, as you suggested. Closing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants