Skip to content

preview.md

Latest commit

 

History

History
442 lines (394 loc) · 26.8 KB

preview.md

File metadata and controls

442 lines (394 loc) · 26.8 KB
Raw

Preview

The preview is the editor's frame-at-the-playhead surface. It is composited by the same Rust + Direct3D 11 native crate that drives MP4 export — crates/compositor/ reached via the compositor_view.node napi-rs addon — and pulled into the renderer as an RGBA8 bitmap that paints onto an HTML <canvas>. The DOM around the canvas hosts the interactive-only layers (zoom gimbal, annotation selection, PiP webcam drag) that need real hitboxes. Everything visible in the preview comes out of the same compositor that writes the export; parity is the property of one renderer, not a discipline across two.

This document describes the live composition path. The export pipe (architecture/export-pipeline.md) shares the same compositor and scene contract; the GPU-resident path it builds on is documented in architecture/native-compositor.md. Performance numbers (preview fluidity, bench methodology, the trade-offs that drove this design) live in engineering/rendering-performance.md.

The compositor path

One compositor exists in the source tree, and it is the live one. A Pixi/WebGL single-canvas screen compositor was tried alongside it and removed once it was established that nothing mounted it.

Native D3D compositor overlay (live)

Mounted as the first child of .previewFrame by NativeCompositorOverlay.tsx. It owns the <canvas>, drives a useNativeCompositorView hook that allocates an offscreen compositor_view view sized to the canvas's device-pixel rect (useNativeCompositorView.ts:73), and pushes a SceneDescription JSON every time the document or the editor settings change. Wallpaper, screen video, webcam, cursor, zoom regions, annotations — every visible pixel comes from this view. The DOM neighbours it (the .screenStage wrapper, the screen <video>, the WebcamOverlay <video>, the AnnotationLayer, the ZoomFocusOverlay, the webcam drag hitbox) are interactive overlays: their pixels are hidden in CSS, only their pointer-event geometry counts.

The two <video> elements decode no picture either. The screen <video> is the playback clock, and an <audio> on the same file plays its sound; both deselect their video track once their metadata is in (dropVideoTrack, under the AudioVideoTracks Blink feature the editor window enables). Left alone, each decoded the whole recording a second and a third time beside the compositor: ~6.7 % of an RTX 4070 Ti's decode engine apiece on a 1080p60 take, measured, and zero once the track is dropped, with the clock, playbackRate and seeks unchanged. A recording without sound drops its picture too, but with neither track selected the element has nothing left to keep time with and jumps to its end on play, so it keeps time by the wall clock instead (keepTimeByWallClock, wallClock.ts): it never plays for real, and answers play, pause, currentTime and playbackRate from a clock that runs playbackRate times the wall clock. Before that, a soundless take was decoded twice by Chromium at the speed region's rate, against the compositor's own decode: at 4K and 3× the view fell 3.5 s of playback behind the playhead (#994). The WebcamOverlay <video> is never played at all — it is read for the camera's size (loadedmetadata), which shapes the PiP box in the scene.

The path is enabled by the presence of the native addon — there is no flag, no capability probe, no per-document switch. The compositing service loads compositor_view.node at startup via compositorViewService.ts (ensureAddon); when the binary is missing the service logs once ([compositor-view] native addon not present; running as no-op, compositorViewService.ts:288) and returns synthetic negative view ids whose readFrame always returns null, so the whole overlay stays inert. In that mode the renderer's own <video> element keeps playing (decode is the responsibility of the DOM, not the compositor), but nothing composites a frame — the canvas stays empty. The editor ships like this on platforms where the addon isn't built; the dev build brings the addon in.

The renderer→addon IPC goes through native-bridge:invoke with a single compositor domain (compositorViewClient.ts:24). The service in the main process loads the addon from electron/native/bin/<platform>-<arch>/compositor_view.node (packaged) or electron/native/compositor-view/build/compositor_view.node (dev), with an OPENSCREEN_COMPOSITOR_VIEW_NODE env override for the standalone builds. The ffmpeg shared-DLL directory is prepended to PATH before the require so the addon's LoadLibrary("avcodec-NN.dll") resolves against the same pinned build the crate links against (compositorViewService.ts:206).

There is no fallback to a CPU/Canvas2D legacy compositor: before the native view ships a frame, only the wallpaper painted by CSS is visible, and the cursor / zoom DOM layers depend on the layout math (not the compositor) to know where to land. That is a product gap, not a fallback path. The native preview is the only compositor in service.

Scene description

document → SceneDescription → JSON is the contract the app hands the native compositor so it can compute the composed frame itself; the renderer does no per-frame math.

buildSceneDescription (src/native/sceneDescription.ts) is a pure data mapping from an AxcutDocument plus the current editor settings to a SceneDescription JSON string. It resolves:

  • Visible clips. resolveVisibleClips is the single shared clip list: resolvePlaybackSegments(document.timeline.clips, document.timeline.trimRanges) (trim-narrowed, so word-level cuts from the transcript editor actually reach the compositor instead of only affecting the transcript panel's own strikethrough) sorted by timelineStartSec and filtered to clips whose asset has a resolvable originalPath. The same call backs buildSceneDescription, the native-export clip list in ExportDialog, and the active-clip lookup in NativeCompositorOverlay — three previously-divergent sites that are now the same expression.
  • Scene regions. Zoom, Full Camera, speed, annotations, and captions are stored in RAW document time (trims still occupy their place) but applied at each frame's source time (the compositor matches each decoded frame's PTS, not a timeline counter). projectRegionsToSource bridges the two reference frames by resolving each region against every visible segment's own raw extent and emitting one entry per source-time span it covers, tagged with the segment's clipIndex. Captions piggyback on annotations (captionCuesToTextRegions produces text annotations at the same zIndex layer) so the compositor draws them with the same path; deliberately no separate caption layer in the native code.
  • Layout. The webcam rect is resolved by the same computeCompositeLayout call the legacy frameRenderer runs and shipped to the addon as layout.webcamRect / layout.screenRect in fractions of the output frame. The native side consumes those verbatim and applies the padding-slider and reactive-zoom adjustments on top, so preview/export never disagree on placement. Per-clip screen resolutions and crops mean a multi-clip document that mixes recording shapes (e.g. a 16:9 screen crop clipped to 9:16 beside another native 9:16 clip) lays out exactly like a single recorded ratio; layout.layoutByClip is index-aligned with clips and cropByClip and the Rust for_clip_window selects the entry for the clip being composed.
  • Lengths as fractions. Every length that crosses the contract is a fraction of its own reference box — roundnessFrac of the output frame's short side, screenRadiusFrac of the screen box's short side, annotation x/y/w/h of the screen rect, padding 0..1. There are no render-target pixels in the payload: the native compositor rasterises the preview into a small contain-fitted frame and the export at full output size, so a pixel meant two different things on the two sides of the boundary; a fraction has no unit to get wrong. The slider itself stays in pixels for the user — the division happens once, here.

The descriptor mirrors the Rust struct in crates/compositor/src/scene.rs; field rename is camelCase on both sides. The Rust consumer (compositor.rs::compose_frame, crates/compositor/src/compositor.rs:1421) reads the JSON per frame, derives the per-clip and per-frame values it needs (zoom state from regions.rs::zoom_state_at, camera-fullscreen progress from regions.rs::camera_fullscreen_progress_at, screen crop from SceneCrop::belongs), and only then issues GPU draw calls. Region visibility is expressed as [startSec, endSec) intervals and matched against t = source_time; a region straddling a clip boundary emits one entry per covered clip via projectRegionsToSource.

Frame delivery

The native compositor runs off-screen (no OS window, no HWND ever crosses IPC), and the renderer pulls composed frames out of it. The protocol is the load-bearing contract between the two:

readFrame(id: number, sinceGen: number): { gen, width, height, data } | null

(addons.d.ts declares it as CompositorViewAddon.readFrame, electron/native/compositor-view/addon.d.ts:93; the renderer-facing type is CompositorFramePacket, contracts.ts:120.)

The contract, with the invariant on the consumer side:

  1. Self-describing packet. A returned object carries its own width/height/data. The drawing buffer is resized to those values before putImageData runs (useNativeCompositorView.ts:195), so pixels and canvas can never drift apart: there is no separate source of truth for "how big the bitmap is right now" — every packet carries it.
  2. Generation-gated. gen is a monotonic per-frame generation (≥ 1). The renderer holds the last one it painted and passes it back as sinceGen; the addon returns null whenever gen <= sinceGen. An idle preview pays nothing: no buffer clone, no IPC crossing, no canvas copy, no putImageData. The null return is the dominant case while the preview sits still (paused editing). One in-flight readFrame at a time (useNativeCompositorView.ts:155) so two responses can't land out of order and rewind lastGen.
  3. data is RGBA8, width * height * 4 bytes. The renderer wraps it via new Uint8ClampedArray(data.buffer, data.byteOffset, data.byteLength) — ImageData rejects SharedArrayBuffer-backed arrays, and the IPC binary is never shared, so the cast is safe. createImageBitmap decodes off the main thread (UI stays at 60/120 Hz) and snapshots the view, then putImageData is the synchronous fallback if bitmap creation is unavailable.
  4. Verify before trusting. data.byteLength !== width * height * 4 || width === 0 || height === 0 is a hard bail (useNativeCompositorView.ts:190): a mismatch would corrupt the image silently. The consumer never assumes a size — every draw is preceded by a resize of the canvas's drawing buffer to the packet's declared width/height.
  5. Pull loop cadence. Read-back frames are pulled on every other rAF tick (PULL_LOOP_TICK_DIVISOR = 2), shared-texture frames (below) on every tick. Each read-back frame is a GPU readback, a structured clone across IPC and a canvas upload, and the tick counter does not advance while a read is in flight: an 8 MB frame whose round trip passes 16.7 ms is pulled one tick in three, about 20 fps.

The renderer-side wrapper mirrors this verbatim in the Electron main process (compositorViewService.ts), so when the addon is absent the IPC layer returns null too — the renderer never has to special-case "addon missing".

Shared textures (Windows)

On Windows' hardware backend the pixels never leave the GPU. Copying them through RAM was the preview's bottleneck, not the compositor: at the same ~57 composed frames per second, read-back reached the canvas at 20-26 fps and kept 54-57 % of the renderer's main thread busy (measurements in engineering/rendering-performance.md).

  • Native side. The render thread copies each composed frame into one of four shared D3D11 textures (shared_frames.rs, NT handles, no keyed mutex), waits for the GPU to finish the copy, and publishes { gen, slot, handle }. SlotBook tracks which slot holds the ready frame and which ones Chromium still holds; a frame nobody took gives its slot back. A held slot is never rewritten, since Chromium may still read it: when every slot is held and one has waited 1 s for its release, the view falls back to read-back.
  • Main process. readFrame takes the frame (readSharedFrame), imports it with sharedTexture.importSharedTexture, sends it with sharedTexture.sendSharedTexture to the frame that asked, drops its own reference and answers with a receipt ({ …meta, shared: true }, no pixels). allReferencesReleased returns the slot (releaseSharedFrame(id, slot, gen)) once Chromium is done with it in every process.
  • Renderer. The preload's setSharedTextureReceiver hands the frame to electronAPI.onCompositorFrame, ahead of the receipt; the hook draws the VideoFrame with drawImage and closes it. The pixels are byte-identical to read-back.
  • Fallbacks, all to read-back. Not Windows, the software backend, Chromium without GPU compositing, or OPENSCREEN_PREVIEW_READBACK=1 never enable it. A failed import or send turns it off for the view, and so does a first frame that lands transparent (a texture Chromium could not open, e.g. on another adapter): the composed frame is cleared opaque, so a transparent pixel can only be that. The render thread republishes the current frame on every switch, so the canvas never waits for something to move.

Playback sync

The native view runs its own clock while playing, so the renderer only pushes a seek when the user actually moves the playhead (scrub, step, or wrap-around to a new clip). The mapping sits in useNativePlaybackSync.ts.

  • Play / pause → free-run. setNativePlaying toggles the addon's free-run decoder. While playing, currentTimeSec ticks every rAF in the renderer; pushing that per tick would force a rewind+seek seek each frame and fight the addon's free-run (the render thread prioritises app-requested frames over free-run). So useNativePlaybackSync only pushes setNativeTime while the transport is paused.
  • RAW → native source. The playhead currentTimeSec is in RAW virtual time (trims still occupy their space on the ruler — same reference the V4 timeline and the webcam overlay use), but the native stream plays compressed segments (resolveVisibleClips, trims removed). resolveNativePosition bridges the two: it maps the RAW playhead via document.timeline.clips (the raw layout) to find the segment that contains it, then returns the segment's clipIndex + source time. Without this bridge a RAW playhead against a compressed clip list pointed at the wrong clip after a trim — wrong camera, misaligned screen.
  • Drift, measured rather than guessed. Every frame carries where the view was when it composed it (clipIndex, sourceTimeSec, both transports), and NativeCompositorOverlay compares that with the playhead on the one timeline both share: programme time, the trim-compressed one, where a cut is no jump (nativeSync.ts). The gap is counted in seconds of playback, so a speed region divides it by its speed, and the last frame is aged by the time since it arrived, at most 250 ms, past which the view counts as stalled. A gap over 150 ms that holds for 100 ms re-anchors the view with setActiveClip, at most every 500 ms. That covers a stall of the render thread and a jump by the user while playing. It replaced a guess from the wall clock at 1× speed, which inside a 2× speed region re-seeked the view ten times a second; useNativePlaybackSync still makes that guess for an addon that reports no position.
  • Free-run decodes every source frame. Inside a speed region the render thread adopts speed × the source's frame rate, so the decoder sets the ceiling: 4K at 2× needs 60 decoded frames a second, at 4× 120. A clip without a camera used to decode its screen file a second time for a stand-in webcam decoder, halving that budget; the stand-in is now never stepped or sought, and compose takes the screen frame in its place (Player::webcam_frame in crates/compositor/src/live.rs). On an M1, 4K at 2× went from 1.2 s of source behind on average in the editor to no lag the counter could measure, and from 8 to 57 frames published a second in the bench. cargo run --release --example live_free_run_bench_macos -- <file> <speed> replays the loop and prints the lag.

The overlay's rect is kept aligned with the DOM via the same primitives used elsewhere in the renderer:

  • computeDeviceRect (src/native/nativeViewRect.ts) turns getBoundingClientRect() into a device-pixel CompositorViewRect, rounding every axis to dodge the truncated / off-by-one windows the addon produces on non-integer inputs.
  • rectsEqual skips the setRect push when nothing has changed (nativeViewRect.ts:33) — the the ResizeObserver (useNativeCompositorView.ts:259) and a coalesced rAF schedule steady-state scrolling and resizes without re-pushing the rect.
  • The overlay reuses the canvas's CSS width: 100%; height: 100% for its display box (NativeCompositorOverlay.tsx:217); the addon's rect.x/rect.y are vestigial (ignored native-side per addon.d.ts:18 and kept on the wire for source compatibility).

Clip changes across the playhead boundary are atomic at the setActiveClip(viewId, screenPath, webcamPath, webcamOffsetSec, clipIndex, sourceTimeSec) RPC (compositorViewClient.ts), and who sends it depends on the transport:

  • Paused (scrub, step): NativeCompositorOverlay sends it whenever the playhead enters another clip.
  • Playing: the render thread crosses into the next clip by itself, preloading it ahead of the cut. The overlay used to send the clip again at every cut, which made the view seek back to a place it had just left, or drop the clip it had preloaded: a hitch at every cut of an edited take. It now only sends it when the view is elsewhere — a jump (a click on the timeline, followed at once) or a gap the drift watch above catches. On a pause it also brings back a view left on another clip, before setNativeTime seeks in it.
  • An addon that reports no position is driven as before: native is paused across the decoder swap, and the live transport is re-read now (not from a captured isPlaying) before resuming, so a user pause that lands mid-transition is honoured.

When the decode clock fails

The hidden <video> is a clock and an audio source, not the picture — the pixels come from the native canvas. So its failing is not the preview failing, and the invariant since #395 is: a media error can never render EditorEmptyState. That component now answers exactly one question — is there anything in this project to show? — and Preview.tsx's render condition (hasProject && hasAsset && previewSources.length > 0) takes no failure input at all.

What replaced it, in two layers:

  • Classify, then reload (mediaError.ts, applied in VirtualPreview's onError). MEDIA_ERR_ABORTED is ignored outright and never counted: the <video> is keyed on activeSource.id, so every cross-asset clip boundary remounts it mid-load and Chromium reports exactly that. MEDIA_ERR_SRC_NOT_SUPPORTED gets one retry (a recording the capture process is still writing reports as unsupported); everything else — including an error event carrying no MediaError — rides RETRY_DELAYS_MS. The reload is a plain load(): re-assigning src first would start a second load that aborts the first and emit a spurious abort. The budget is re-armed by playing past the position the failure happened at (tracked in the rAF tick). Both obvious alternatives are wrong: never re-arming makes the third transient failure of a long session terminal — #395 with a longer fuse — while re-arming when the reload completes is worse still, because loadedmetadata/canplay prove the header parsed and the decoder is willing, not that the bytes that killed us are readable. A truncated recording re-fires both on every reload, so that version loops at 400 ms forever and never surfaces the card at all.
  • Resume through the existing path. The reload queues pendingSeekRef and lets onLoadedMetadata restore position and playback — the same path a cross-asset seek uses, so the component has one resume path rather than two. The position is resolved at reload time, not when the error fired, because the user can scrub during the backoff: it comes from locateVirtualPosition against the live playhead, guarded by an assetId check (that function answers for whatever clip the playhead is on, which after a boundary advance can belong to a different asset), and falls back to a source time sampled while the decoder was known good. video.currentTime is not usable here — it reads 0 after load().

While a reload is in flight the rAF tick returns early (recoveringRef), publishing the clock but taking no decision. This cannot ride on the v.paused gate the rest of the tick uses: an error does not fire pause, so a failure during playback leaves paused false, and the tick would go on making trim/boundary decisions against a frozen clock and clobber the queued resume.

Only once the budget is spent does Preview hear about it, and it renders PreviewErrorCard over the still-mounted canvas — the last composed frame stays visible, which is the evidence the project is intact. The card carries the MediaError code, because a user's report is otherwise unactionable: the reason #395 could only ever be described as "the preview disappeared" is that the code was thrown away.

Known gaps

  • Live preview is video-only. crates/compositor/src/live.rs (1628 lines) handles only the video packet stream; audio is decoded and mixed in audio.rs::decode_clip_audio for the export path and not for the live view. Editing playback is therefore silent against the exported file; users hear audio only when the export runs. There is no flag in this branch that re-routes live audio.
  • Speeds the decoder cannot follow fall behind. VideoToolbox decodes 4K at about 97 frames a second on an M1, short of the 120 a 30 fps source needs at 4×. The render thread caps a tick at max_steps frames and counts at most 0.1 s of clock per tick, so the view loses time and the drift watch re-anchors it every 500 ms; skipping ahead to a keyframe instead is not implemented.
  • Drift under 150 ms is left alone. The view and the app's clock run independently inside the drift watch's tolerance, and a correction is a seek, not a change of pace: a view drifting slowly is re-anchored with a visible step rather than eased back.
  • Shared textures are Windows-only. macOS (an IOSurface-backed Metal texture) and Linux (a dmabuf exported from Vulkan) still read back: sharedTexture imports both, the native halves are not written.
  • Add-on absent = blank frame. When compositor_view.node is missing (development with the addon not yet built, or a packaged build for an unsupported architecture) the overlay renders no pixels: only the DOM/CSS wallpaper and the interactive layers show. There is no CPU/Canvas2D fallback path; the live preview is gated on the addon being present.

A single frame, end-to-end

The labelled arrows are the IPC + state edges; everything inside one box is in-process.

sequenceDiagram
    participant Doc as AxcutDocument
    participant TSD as src/native/sceneDescription.ts
    participant Overlay as NativeCompositorOverlay.tsx
    participant Hook as useNativeCompositorView.ts
    participant IPC as native-bridge:invoke<br/>(compositor domain)
    participant Svc as compositorViewService.ts
    participant Addon as compositor_view.node
    participant Canvas as renderer <canvas>

    Doc->>TSD: buildSceneDescription(doc, settings)
    TSD-->>Overlay: SceneDescription JSON
    Overlay->>IPC: setScene(viewId, json)
    IPC->>Svc: dispatch
    Svc->>Addon: setScene(id, json)

    Overlay->>Hook: ResizeObserver on canvas
    Hook->>IPC: setRect(viewId, deviceRect)
    IPC->>Svc: dispatch
    Svc->>Addon: setRect(id, rect)
    Addon-->>Svc: ok

    Note over Addon,Canvas: compositing thread runs off-screen<br/>(D3D11VA decode → compositor → RGBA8 staging)

    loop every other rAF tick (~30 fps)
        Hook->>IPC: readFrame(viewId, lastGen)
        IPC->>Svc: dispatch
        Svc->>Addon: readFrame(id, sinceGen)
        alt same generation or no frame
            Addon-->>Svc: null
            Svc-->>Hook: null
            Hook-->>Canvas: (canvas untouched, idle path costs nothing)
        else new generation
            Addon-->>Svc: { gen, width, height, data }
            Svc-->>Hook: { gen, width, height, data }
            Hook->>Hook: validate byteLength === width*height*4
            Hook->>Canvas: canvas.width = width; canvas.height = height<br/>createImageBitmap → drawImage
            Hook->>Hook: lastGen = gen
        end
    end

    Overlay->>IPC: setActiveClip(viewId, screenPath, webcamPath,<br/>webcamOffsetSec, clipIndex, sourceTimeSec)
    IPC->>Svc: dispatch
    Svc->>Addon: setActiveClip(...)
    Overlay->>IPC: setPlaying(viewId, playing) (per play/pause)
    IPC->>Svc: dispatch
Loading