Design: define a shared video capture session contract

[marcusschiesser]

Motivation

Native SDK has an audio-capture design in #262, with an implementation in #264. That work establishes useful capture conventions:

• Explicit permission checks; starting capture never prompts.
• Keyed capture sessions.
• Negotiated output formats.
• Bounded delivery with observable backpressure.
• Clear start, stop, failure and rejection behavior.
• Stop-and-drain versus immediate discard.
• Zig and TypeScript command/event parity.
• Honest capability discovery and null-platform behavior.

Screen and camera capture should reuse those conventions, while acknowledging that raw video cannot travel through the TypeScript model like short PCM chunks.

This issue proposes the shared video-capture contract needed before implementing actual ScreenCaptureKit, AVFoundation, Windows or Linux backends.

Screen/window source discovery is also requested in #240. That issue can build on the contract defined here.

Proposed scope

• Shared video-capture source, format, timing, lifecycle and failure types.
• A keyed video-capture session API usable by display, window and camera sources.
• A common process-monotonic presentation clock compatible with the audio-capture timeline from Design: expose reliable paired system and microphone audio streams #262/feat: add macOS audio capture streams #264.
• TypeScript lifecycle commands and fixed event records.
• Zig types for native captured video frames.
• Optional binding of a capture session to a media-surface for preview.
• Capability flags for display, window and camera capture.
• Null-platform implementations that report unsupported honestly.
• Runtime, fake-executor, journal/replay and lifecycle tests.
• No real platform capture backend in the initial contract PR.

Relationship to audio capture

Audio and video should share control-plane concepts but not their data plane.

Audio capture can return short borrowed PCM chunks through audioCaptureRead. A video frame may contain several megabytes and can be GPU-backed, so pixels must not enter the TypeScript Model, Msg, command wire format or session journal.

The proposed common timing vocabulary is:

pub const CaptureTiming = struct {
    presentation_time_micros: u64,
    duration_micros: u64,
};

The matching TypeScript fields are:

readonly presentationTimeMicros: number;
readonly durationMicros: number;

presentationTimeMicros uses one process-monotonic clock shared by audio, display, window and camera capture. It describes when media should be presented, not when the application happens to receive it.

This lets applications synchronize a future CapturedVideoFrame with the CapturedAudioChunk proposed in #264.

Proposed video types

export type VideoCaptureSource =
  | {
      readonly kind: "display";
      readonly id: Uint8Array;
      readonly includeCursor?: boolean;
      readonly excludeCurrentProcessWindows?: boolean;
    }
  | {
      readonly kind: "window";
      readonly id: Uint8Array;
      readonly includeCursor?: boolean;
    }
  | {
      readonly kind: "camera";
      readonly device: "default" | Uint8Array;
    };

export type VideoPixelFormat =
  | "bgra8"
  | "nv12";

export interface VideoCaptureOptions {
  readonly source: VideoCaptureSource;

  // Preferred constraints. The negotiated values are reported by started.
  readonly preferredWidth?: number;
  readonly preferredHeight?: number;
  readonly preferredFrameRate?: number;
  readonly preferredPixelFormat?: VideoPixelFormat;
}

Display/window and camera enumeration remain source-specific APIs. Both produce selections that can be passed to the shared video-capture start command.

Lifecycle contract

export type VideoCaptureState =
  | "started"
  | "format_changed"
  | "stopped"
  | "failed"
  | "rejected";

export type VideoCaptureReason =
  | "none"
  | "invalid_options"
  | "permission_missing"
  | "permission_required"
  | "already_recording"
  | "source_not_found"
  | "source_ended"
  | "device_disconnected"
  | "format_unsupported"
  | "consumer_too_slow"
  | "capture_failed"
  | "discarded"
  | "unsupported";

export type VideoCaptureEventArm = {
  readonly key: string;
  readonly state: VideoCaptureState;
  readonly reason: VideoCaptureReason;

  // Negotiated format.
  readonly width: number;
  readonly height: number;
  readonly frameRate: number;
  readonly pixelFormat: VideoPixelFormat;

  // Diagnostics, not synchronization coordinates.
  readonly framesProduced: number;
  readonly framesDropped: number;
};

Proposed commands:

Cmd.videoCaptureStart(
  key,
  options,
  { event: "video_capture" },
);

Cmd.videoCaptureStop(key);
Cmd.videoCaptureDiscard(key);

// Attach or replace a latest-wins preview surface.
// Surface 0 detaches the preview.
Cmd.videoCaptureSetSurface(key, surface);

Unlike the initial audio contract, the video contract must allow multiple concurrent sessions. Screen and camera capture commonly run at the same time.

State behavior

• started reports the negotiated format after capture is operational.
• format_changed reports a source-driven resolution or format change.
• stopped indicates an orderly requested stop.
• failed terminates a session after an asynchronous capture failure.
• rejected means the start command was not accepted.
• stop seals native consumer queues so previously accepted frames can drain.
• discard releases the session and queued frames immediately.
• Stopping or discarding an unknown key is a no-op.
• Late events from a previous generation of the same key must be ignored.

No generic discontinuity Boolean is proposed. Source loss, dropped frames, format changes and backend failures have explicit representations.

Native frame representation

The Zig tier needs an opaque, reference-counted video-frame representation:

pub const CapturedVideoFrame = struct {
    timing: CaptureTiming,

    /// Index assigned to frames produced by this capture source.
    frame_index: u64,

    /// Source frames discarded before this delivered frame.
    dropped_before: u32,

    width: u32,
    height: u32,
    pixel_format: VideoPixelFormat,
    color_space: VideoColorSpace,
    rotation: VideoRotation,

    /// CPU planes or platform-native GPU-backed storage.
    storage: VideoFrameStorage,
};

This issue defines the type and ownership contract. A follow-up can expose a native consumer along these lines:

const consumer = try runtime.acquireVideoCaptureConsumer(key, .{
    .queue_depth = 3,
    .overflow = .fail,
});

while (try consumer.nextFrame()) |frame| {
    defer frame.release();
    try encoder.submitVideo(frame);
}

The TypeScript core receives lifecycle and diagnostic events only. It never receives raw pixel bytes or platform frame handles.

Preview delivery

A capture session may feed an existing media-surface.

Preview delivery uses the media surface's latest-wins behavior:

• Frames do not queue behind the UI.
• Intermediate frames may be replaced before presentation.
• Preview latency remains bounded.
• Pixels remain outside deterministic model state and session replay.
• Preview delivery is independent of an encoder-facing native consumer.

The first implementation may convert frames to the existing RGBA8 surface format. Zero-copy platform texture adoption can be added separately.

Capability flags

Add independent capabilities so applications can describe platform support honestly:

.display_capture
.window_capture
.camera_capture
.screen_source_enumeration
.camera_device_enumeration

A platform may support only a subset. For example, a backend could initially support display capture but not individual window capture.

The existence of the shared API must not imply that a production host supports a particular source kind.

Null-platform behavior

The null platform should:

• Report all new capability flags as unsupported.
• Compile the complete shared contract on every target.
• Reject videoCaptureStart exactly once with reason = .unsupported.
• Never emit started.
• Treat stop, discard and surface-detach for unknown keys as no-ops.
• Allocate no capture threads, surfaces or frame storage.
• Never emit pixel data.
• Remain deterministic under session recording and replay.

Lifecycle tests

Add tests covering:

• Unsupported start produces one rejected event.
• A rejected start does not leave a live session.
• Valid synthetic lifecycle: started → format_changed → stopped.
• Failure lifecycle: started → failed.
• Duplicate live keys are rejected without disturbing the original session.
• Multiple different video keys can be active concurrently.
• Stop seals accepted native frames for draining.
• Discard releases queued frames immediately.
• Unknown stop/discard operations are no-ops.
• A reused key receives a new generation.
• Late callbacks from an earlier generation are ignored.
• Negotiated format fields are propagated unchanged.
• framesProduced and framesDropped remain diagnostic counters.
• Lifecycle metadata survives journal/replay.
• Pixel contents and native frame handles are never journaled.
• Null-platform capability snapshots remain honest.

Suggested implementation split

1. Add shared timing, video source, format, lifecycle and reason types.
2. Add TypeScript command/event declarations and compiler surface fixtures.
3. Add Zig effect and runtime session state.
4. Add capability flags and platform API hooks.
5. Implement null-platform rejection behavior.
6. Add fake-executor lifecycle injection and generation handling.
7. Add journal/replay coverage for lifecycle metadata.
8. Document the contract and intended media-surface/native-consumer split.

Out of scope

• Screen/window source enumeration implementation.
• Camera device enumeration implementation.
• ScreenCaptureKit or AVFoundation capture.
• Windows Graphics Capture or Media Foundation.
• Linux portal, PipeWire or V4L2 capture.
• Audio capture changes beyond sharing timing terminology.
• Video encoding, muxing or file output.
• RTMP or other network streaming.
• Image composition or effects.
• Camera controls such as zoom, focus or exposure.
• Session recovery after device reconnection.
• Journaling raw video frames.

Follow-up work

Once this contract is accepted, platform work can proceed independently:

1. Screen/window enumeration and permissions under Screen capture support (equivalent to Electron's desktopCapturer) #240.
2. macOS display/window capture through ScreenCaptureKit.
3. macOS camera enumeration and AVFoundation capture.
4. Native frame-consumer delivery for encoders and compositors.
5. Windows display/window and camera backends.
6. Linux portal/PipeWire/V4L2 backends.