diff --git a/docs/README.md b/docs/README.md index eddabb53a4..0eb0ef16b8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -147,6 +147,7 @@ Integrate your 3D display hardware into DisplayXR. - [ADR-039](adr/ADR-039-one-fill-engine-for-every-tier.md) — One fill engine for every tier (same-adapter split) - [ADR-040](adr/ADR-040-rear-depth-budget.md) — Rear depth budget — the runtime owns the policy, the plug-in owns pixels, the app owns geometry - [ADR-041](adr/ADR-041-fixed-view-count-with-per-frame-activity.md) — Fixed view count with per-frame activity — inactive views alias, they do not disappear +- [ADR-043](adr/ADR-043-stereo-camera-source.md) — A display's stereo camera is a plug-in-provided source, owned by the service and privacy-gated by the runtime --- @@ -167,6 +168,7 @@ Design docs, status trackers, and plans — some shipped, some in progress. Afte - **[Roadmap Overview](roadmap/overview.md)** — milestone status and project trajectory - [Spatial Desktop PRD](roadmap/spatial-desktop-prd.md) — product vision - [PR FAQ](roadmap/prfaq.md) — press-release-style framing +- [Stereo Camera Source](roadmap/stereo-camera-source.md) — a display's (eye-tracking) stereo camera as a plug-in-provided, privacy-gated source for web calls: browser `getUserMedia` integration, Android, phased plan (design, ADR-043) - [Spatial Workspace Extensions Plan](roadmap/spatial-workspace-extensions-plan.md) — three-phase plan to decouple the shell from the runtime: boundary rename (Phase 1, done), policy migration behind extensions (Phase 2), repo severance (Phase 3) - [Workspace Extensions Header Sketch](roadmap/spatial-workspace-extensions-headers-draft.md) — `XR_DXR_spatial_workspace.h` C-level API draft (historical: the separate app-launcher extension sketched there was dropped; launcher tiles ship as `*.displayxr.json` manifests) - [Workspace Controller Detection](roadmap/spatial-workspace-controller-detection.md) — Phase 2.0 prep: orchestrator detects installed controller via sidecar `.controller.json` manifest diff --git a/docs/adr/ADR-043-stereo-camera-source.md b/docs/adr/ADR-043-stereo-camera-source.md new file mode 100644 index 0000000000..90df2a3fa4 --- /dev/null +++ b/docs/adr/ADR-043-stereo-camera-source.md @@ -0,0 +1,91 @@ +# ADR-043: A display's stereo camera is a plug-in-provided source, owned by the service and privacy-gated by the runtime + +**Status:** Proposed (2026-09-25) — design only, nothing implemented · introduces +[`XR_DXR_stereo_camera`](../specs/extensions/XR_DXR_stereo_camera.md) · appends optional +camera slots to `xrt_plugin_iface` under the [ADR-020](ADR-020-plugin-abi-compatibility-policy.md) +append-at-end rule · related: [ADR-019](ADR-019-vendor-plugin-aux-boundary.md), +[ADR-034](ADR-034-input-provider-plugins.md), [ADR-035](ADR-035-service-owned-arbitration-single-pipeline-isolated-satellites.md), +ADR-042 (vendor 2D→3D conversion; in flight on branch `feat/lift-ext`, not yet on `main`) (the same shape in reverse) · +integration + plan: [roadmap/stereo-camera-source.md](../roadmap/stereo-camera-source.md) + +## Context + +The web SDK's 3D call module (`@displayxr/inline3d/call`, RFC 0002 in `displayxr-web`) sends +full-width L|R side-by-side video when the sender has a stereo camera. Most 3D laptops and +monitors have one — the camera their eye tracker looks through. Field facts (2026-09-25, two +machines): + +- **While the vendor eye tracker runs, it holds the camera exclusively.** `getUserMedia` fails + with `NotReadableError: Device in use`. Since tracking runs whenever the display is in 3D, a + page effectively never gets the camera on the machines that most need it. +- **When the tracker was idle**, `getUserMedia` opened it as ONE device: 1280×480 @ 30 Hz = + 640×480 per eye, **greyscale, raw, unrectified** (visible vertical misalignment), with no + calibration attached. +- **The vendor's own call application gets frames anyway** — not from the device, from the + tracker: the vendor SDK publishes each tracker frame into named shared memory (a small local + header `{format, compressedSize, height, width}` followed by one SBS JPEG), and frames only + flow once a weave context has brought the tracker up. It rectifies with OpenCV from per-device + `intrinsics.yml`/`extrinsics.yml` files. Two defects in that application are instructive: + it picks the **first** device calibration folder on the machine (one test box had eleven), so a + correct design must key calibration by the **active** device; and the SDK's new-frame event is + **auto-reset**, so two readers waiting on it steal each other's wake-ups. +- 3D **tablets** carry a front stereo pair (15 mm and 25 mm baselines on two models, parallel) + owned by the vendor's camera/tracking service on Android. + +Everything a correct stereo camera needs — the private frame channel, the device's calibration, +keeping the tracker alive — is vendor knowledge. The runtime's plug-in model (ADR-019) exists +exactly so that knowledge lives in one vendor DLL and nowhere else. + +## Decision + +1. **The vendor plug-in is the camera source.** Optional slots appended to `xrt_plugin_iface` + (not a DP vtable — a camera is a sensor, graphics-API-neutral, and must outlive DP + re-creation) let a plug-in enumerate stereo cameras, return the **active device's** + calibration, and deliver decoded frames (GRAY8/NV12/BGRA8) from its own stack **without + taking the device from the tracker**. Opening a camera doubles as a tracker keep-alive. +2. **The service owns the camera.** One runtime-owned thread per open camera, one capture fanned + out to every authorised stream, latest-wins pinned rings, per-stream wake handles. Rectification + (Bouguet, parallel, zero disparity at infinity, **no convergence shear**) and format conversion + are vendor-neutral runtime code. In-process runtimes enumerate nothing: one owner per camera. +3. **Clients consume through `XR_DXR_stereo_camera`**, an instance-level extension (a capture + component has no session): enumerate with properties and state, calibration (raw/rectified), + start/stop, acquire newest frame with timestamps over shared memory (optional GPU transports). +4. **The runtime enforces camera privacy itself**, because these frames bypass the OS camera + stack: OS camera consent of the OS-derived peer, a DisplayXR per-executable consent (a + registered browser delegates per-origin consent to its own prompt), visible/foreground-only + delivery, a runtime-owned in-use indicator, a user kill switch, and hashed ids against + fingerprinting. +5. **The DisplayXR Browser exposes each camera as an ordinary `getUserMedia` device** with one + capability hint, so pages (the call module) need no DisplayXR-specific capture API. + +## Alternatives considered + +| Alternative | Verdict | +|---|---| +| **A. `getUserMedia` only** (page opens the physical camera; pause the tracker if needed) | **Rejected.** Fails exactly where it matters (exclusive tracker). Pausing tracking to free the device kills the weave's viewer tracking for the whole display during a call — the 3D experience the call is for. Still yields raw, uncalibrated pairs. Remains the path for plain USB stereo cameras that no plug-in claims (RFC 0002 §4 row 1). | +| **B. Bridge to the vendor's own app** (the vendor call app or a vendor helper re-publishes frames, e.g. as a virtual webcam) | **Rejected.** Couples DisplayXR to a vendor application's lifetime and install, adds a second privileged reader of the tracker (the auto-reset-event contention above), and a virtual webcam driver is a kernel/DirectShow-filter install per vendor with no consent model of ours. | +| **C. Plug-in source via the service** (this ADR) | **Accepted.** Vendor code stays in the plug-in (ADR-019); one owner, fan-out, runtime-enforced privacy; the same shape as `XR_DXR_lift` reversed (plug-in produces, service owns, client consumes). | +| **D. Vendor code in the browser** (the Chromium fork reads the vendor shared memory, decodes, rectifies) | **Rejected — violates vendor isolation.** Vendor names/paths/SDK headers in a vendor-agnostic browser; every vendor would need a browser patch; the browser's GPU/utility processes are sandboxed (Low IL, restricted token) and cannot open the vendor's named objects anyway; and no other client could reuse it. | +| **E. A separate "camera provider" plug-in type** (ADR-034 style) | **Deferred.** Right for a camera vendor that is not a display vendor. Today every source is a display's own tracker camera, whose calibration and keep-alive are the display plug-in's; a second plug-in type would split one device across two DLLs. The OpenXR surface does not change if a provider type is added later. | +| **F. Plug-in rectifies** | **Rejected as the default.** Rectification is vendor-neutral math; doing it once in the runtime gives every vendor correct, identical, testable (sim_display) output. A plug-in whose source is already rectified says so (`NATIVELY_RECTIFIED`). | +| **G. Bake a convergence shear into the source** (as the vendor call app does) | **Rejected.** Convergence depends on subject distance and is the receiver's choice (RFC 0002: per-eye crop offset `f·B/(2Z)`); the source stays a physically honest parallel pair. | + +## Consequences + +- **+** Pages get the tracker camera during tracking, rectified, with honest calibration, through + standard web APIs; the eye tracker is never disturbed. +- **+** Calibration is the active device's by construction; the "first folder" class of bug + cannot happen in a consumer. +- **+** Hardware-free end to end: sim_display's fake camera (synthetic distorted, misaligned SBS + with known depths) makes rectification and the privacy flow CI-testable. +- **+** One more place where DisplayXR adds, not subtracts, platform privacy: indicator and + consent exist even though the OS cannot see this consumer. +- **−** The runtime takes on camera privacy UX (tray prompt, indicator, consent store) it did not + have. It must be right the first time; a leak here is a headline. +- **−** The first real source rides an **undocumented vendor shared-memory contract** (header + defined by readers, no sequence number or timestamp, auto-reset wake). The plug-in must poll + defensively until the vendor SDK formalises it (roadmap open questions). +- **−** Frame quality is the tracker's: greyscale, 640×480, tracker-chosen rate/exposure. +- **−** Another service thread, and a CPU copy per frame (≈ 27 MB/s at 1280×480 NV12 30 Hz). +- **−** Android depends on the vendor camera SDK sharing frames at all (open question); until then + Android documents a platform difference rather than breaking isolation. diff --git a/docs/adr/README.md b/docs/adr/README.md index b18b2c81cb..0f67bfe705 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -45,3 +45,4 @@ - [ADR-039](ADR-039-one-fill-engine-for-every-tier.md) — One fill engine for every tier (same-adapter split) - [ADR-040](ADR-040-rear-depth-budget.md) — Rear depth budget — the runtime owns the policy, the plug-in owns pixels, the app owns geometry - [ADR-041](ADR-041-fixed-view-count-with-per-frame-activity.md) — Fixed view count with per-frame activity — inactive views alias, they do not disappear +- [ADR-043](ADR-043-stereo-camera-source.md) — A display's stereo camera is a plug-in-provided source, owned by the service and privacy-gated by the runtime diff --git a/docs/roadmap/stereo-camera-source.md b/docs/roadmap/stereo-camera-source.md new file mode 100644 index 0000000000..5026f11648 --- /dev/null +++ b/docs/roadmap/stereo-camera-source.md @@ -0,0 +1,200 @@ +# Stereo camera source — browser integration, vendor plug-in notes, phased plan + +**Status:** design (2026-09-25). Decision: [ADR-043](../adr/ADR-043-stereo-camera-source.md). +Extension: [`XR_DXR_stereo_camera`](../specs/extensions/XR_DXR_stereo_camera.md). Consumer +driving it: the web SDK's 3D call module (`@displayxr/inline3d/call`, RFC 0002 on +`displayxr-web` branch `feat/call-p1`, §4 *Capture*). + +This page holds what the vendor-neutral spec deliberately does not: how the DisplayXR Browser +turns a runtime camera into a web camera, how the first vendor plug-in (Leia SR) sources frames, +the Android story, and the plan. + +--- + +## A. The page-facing contract (goal) + +A page uses **standard web APIs only, plus one capability hint**: + +```js +const devices = await navigator.mediaDevices.enumerateDevices(); +// the runtime camera appears as an ordinary videoinput: label "3D Camera (DisplayXR)" +const stream = await navigator.mediaDevices.getUserMedia({ video: { deviceId: id } }); +const track = stream.getVideoTracks()[0]; +track.getSettings(); +// { width: 1280, height: 480, frameRate: 30, facingMode: 'user', ..., +// displayxrStereo: { layout: 'side-by-side', rectified: true, baselineMm: 50, horizontalFovDeg: 68 } } +``` + +- **The hint is one non-standard dictionary member, `displayxrStereo`, on `MediaTrackSettings`** + (mirrored in `getCapabilities()`), present only on DisplayXR stereo tracks. Absent ⇒ treat the + track as whatever the page already assumes. Values are **coarsened** — integer mm, integer + degrees — because exact calibration is a device fingerprint (spec §7.5); a page that needs + `f_px` derives it from `width/2` and `horizontalFovDeg`. +- **No selection API.** The call module's existing `camera: 'auto'` rule (aspect > 2.5 ⇒ stereo + SBS) already picks a 1280×480 track. With the hint it can stop guessing: prefer a device whose + track reports `displayxrStereo`, and fill its `hello` (`rectified`, `baselineMm`, `hfovDeg`) + from the hint instead of assuming `rectified: false`. That is the entire SDK change. +- **The frame is never mirrored** and is left-eye-left. Self-view mirroring (mirror each half AND + swap halves) stays the page's, as RFC 0002 §3 already says. +- **Permission, indicator, device picker are Chromium's own**, unchanged: the runtime camera is a + camera like any other to the permission model. +- Other browsers (stock Chrome/Edge) never see this camera; the SDK treats such senders as mono + (lifted on the receiver). A system-wide virtual webcam is out of scope (ADR-043 alternative B). + +## B. DisplayXR Browser — Windows + +### B.1 Where it plugs in + +Chromium enumerates and opens cameras in its **video capture service** +(`media/capture/video/…`, `VideoCaptureDeviceFactory` → `VideoCaptureDevice`). The integration +is a **second device factory composed in front of the platform one**: + +| Piece | Where | Notes | +|---|---|---| +| `DisplayXRStereoCameraDeviceFactory` | new files, `components/displayxr/capture/` | `GetDevicesInfo()` = `xrEnumerateStereoCamerasDXR`; one descriptor per camera: display name `"3D Camera (DisplayXR)"` (plus the plug-in's `displayName` when more than one), `device_id = "dxr-stereo:" + persistentId`, `facing = USER`, one format `2·eyeW × eyeH @ maxFrameRate, NV12` | +| Composition hook | one small hunk in `media/capture/video/create_video_capture_device_factory.cc` | wraps the platform factory; the only upstream-file edit (goes into `docs/integration-points.md`) | +| **Hide the busy duplicate** | same wrapper | drops the platform (MediaFoundation) descriptor whose symbolic link matches a camera's `platformDeviceHint` — otherwise the page also sees the physical device that fails with "Device in use" | +| `DisplayXRStereoCameraDevice : VideoCaptureDevice` | `components/displayxr/capture/` | `AllocateAndStart` → create + start a RECTIFIED (else RAW) NV12 shared-memory stream; a capture thread waits on the stream's wake handle, acquires, and calls `Client::OnIncomingCapturedData` with the pinned slot (a copy into Chromium's pool happens there), `reference_time`/`timestamp` from `captureTime` (QPC domain = `base::TimeTicks` on Windows). `StopAndDeAllocate` → stop + destroy. SUSPENDED ⇒ stop delivering (the track stays live and shows its last frame, like a paused webcam); INSTANCE_LOST ⇒ `Client::OnError` | +| Hint | Blink `MediaTrackSettings` / `MediaTrackCapabilities` IDL + the `VideoCaptureFormat`/device-info plumbing that already carries per-device metadata to the renderer | the one Blink-visible change; lives beside the existing inline-3D Blink surface | + +A greyscale source is delivered as NV12 with neutral chroma — every downstream stage (WebRTC +encoders, `