Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 72 additions & 1 deletion docs/adr/ADR-043-stereo-camera-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

**Status:** Proposed (2026-09-25) · **R1 implemented** 2026-09-26 (runtime, hardware-free; maintainer
defaults recorded in the roadmap §G: stereo-only, iface slots, service clients only, no raw frames
to pages) · **R2 implemented** 2026-09-26 (the service-side rectifier, decision 2 below) · introduces
to pages) · **R2 implemented** 2026-09-26 (the service-side rectifier, decision 2 below) · **Amendment 4**
2026-10-07 (a vendor-neutral UVC side-by-side source owned by the service) · introduces
[`XR_DXR_stereo_camera`](../specs/extensions/XR_DXR_stereo_camera.md) · appends optional
camera slots to `xrt_plugin_iface` under the [ADR-020](ADR-020-plugin-abi-compatibility-policy.md)
append-at-end rule · related: [ADR-019](ADR-019-vendor-plugin-aux-boundary.md),
Expand Down Expand Up @@ -170,3 +171,73 @@ Three changes close that (spec §2a, §7.1, §7.1.1 are normative):
A skipped delegation (declined or untrusted) also drops the foreground-rule exemption — the
client is an ordinary window-bearing app for that stream — but never the RAW refusal, which keys
on registration alone, so an untrusted entry can only ever narrow what a client gets.

## Amendment 4 (2026-10-07) — a vendor-neutral source for plain UVC side-by-side stereo webcams

Decision 1 made the display plug-in the only camera source, and Alternative A left "plain USB
stereo cameras that no plug-in claims" to `getUserMedia`. That leaves a gap the plug-in model
cannot fill: an external stereo webcam that outputs one side-by-side image over ordinary UVC
works with **any** display, belongs to **no** display vendor, and a page reading it through
`getUserMedia` gets raw, unaligned pairs with no calibration, no consent of ours and no
`XR_DXR_stereo_camera` path for native apps. Nothing about reading it is vendor knowledge: the
device is a standard UVC camera; the only device-specific facts are its SBS layout and eye order.

**Decision.** The service gains its **own** camera source, next to the plug-in's:

1. **Runtime-owned, vendor-neutral capture code** (`auxiliary/util/u_stereo_uvc.{h,c}` for
config, matching, the SBS split and the nominal model — pure, tested on every OS;
`auxiliary/os/os_uvc_capture*.{h,c,cpp}` for the OS call: Media Foundation on Windows, a
V4L2 stub on Linux to follow). It has exactly the six-operation shape of the plug-in camera
slots, and the camera manager dispatches on a per-camera source tag in one place; consent,
`CAMERA_CONSUMER` rules, the foreground rule, the in-use indicator, linger, rings,
rectification and refinement are one code path for both sources. No vendor names, no SDK.
2. **Opt-in per device, never automatic.** Nothing is claimed without the user's per-user
`stereo-cameras.json` (`%LOCALAPPDATA%\DisplayXR\`, `$XDG_CONFIG_HOME/displayxr/`;
`DXR_STEREO_CAMERA_UVC_CONFIG` overrides the path, `DXR_STEREO_CAMERA_UVC=0` turns the source
off). An entry matches by USB VID:PID and/or a friendly-name substring and states layout
(`sbs-full` | `sbs-half`), eye order, capture mode, per-eye output size, nominal baseline and
HFOV, optionally a calibration file. A **name-only** entry that matches more than one present
device claims nothing. The built-in allowlist is **empty**: no device is listed without a
public spec of its SBS layout.
3. **A plug-in's camera is never claimed.** Plug-in cameras are enumerated first; a device whose
OS id or USB VID:PID equals a plug-in camera's `platform_device_hint` is skipped — the
display's tracker camera stays the plug-in's and its eye tracker's.
4. **Uncalibrated is the normal case, and it is reported honestly.** Without a calibration file
the camera has no `CALIBRATED` flag; the service models it as an ideal parallel pinhole pair
(fx = fy from the configured HFOV, centred principal point, no distortion, R = I,
T = (−baseline, 0, 0)). RAW calibration is refused (`FEATURE_NOT_SUPPORTED`): a nominal model
is not a measurement of the device. RECTIFIED is offered anyway — it is a **pass-through**
(frames as delivered, described by the nominal pinhole, no remap, no zoom) until the online
vertical refinement (R2) has measured the rows and folded a correction in, from which point
the ordinary corrected LUT path runs. `baselineMm` / `horizontalFovDeg` are reported only
when the user configured them (0 = unknown, as before). A calibration file (OpenCV `K1 D1 K2
D2 R T`, YAML or JSON) makes the camera `CALIBRATED` and the R2 rectifier does the rest.
5. **The device is open only while a stream is started** (plus the 2 s linger), exactly like a
plug-in source; there is no tracker to keep alive. Enumeration never activates a device.

**What changes for consumers.** Nothing in the API (spec version unchanged): one rule is relaxed
— RECTIFIED no longer requires `CALIBRATED | NATIVELY_RECTIFIED`, it requires a camera the
service can rectify, which now includes a nominal-model UVC pair. A consumer that gates
RECTIFIED on `CALIBRATED` simply does not ask for it. The browser's capture component sees an
ordinary camera with a rectified output and a `platformDeviceHint` naming the physical webcam, so
it can hide the raw duplicate the OS also exposes. `displayxr-cli camera list` tags each camera
`plugin` or `uvc`; `camera uvc-devices [--modes]` shows the config and which present device it
claims (modes only for claimed devices, never started).

**Alternatives.** *A V4L2/MF virtual camera that re-publishes rectified frames* — rejected for the
reason Alternative B was: a driver install with no consent model of ours. *Letting each display
plug-in carry webcam support* — rejected: a webcam is not a display's, and every vendor would
duplicate the same capture code (ADR-019 isolates vendor code; this is the opposite case). *A
camera-provider plug-in type* (Alternative E) remains the right home for a **camera vendor's**
proprietary device (custom transport, factory calibration channel); a standards-compliant UVC
SBS webcam needs no vendor code, so it lives in the runtime.

**Consequences.** **+** Any display + any SBS UVC stereo webcam, no plug-in, same privacy model.
**+** The fake backend (`"fake"` in the config) runs the whole manager hardware-free
(`tests_stereo_camera_manager`). **−** MJPEG decoding the plug-in path does not have. A 4K60
webcam is decoded on the GPU by default (a hardware MJPEG decoder MFT on the adapter that has one,
via a DXGI device manager, scaled to the output size before it crosses to the CPU), with MF's
software decoder as the logged fallback; the first hardware run, before that path, delivered
11.3 Hz. **−** Linux has no backend yet (V4L2 is a TODO). **−** A
loose `match` could claim the wrong webcam; the ambiguity rule and `uvc-devices` mitigate, the
config is still the user's statement of which device is a stereo pair.
4 changes: 4 additions & 0 deletions docs/roadmap/control-panel-performance-settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -548,6 +548,10 @@ as `docs/specs/vendor/oem-android-platform-requirements.md` §R6.
| `DXR_HEALTH_MS` | `ipc/server/ipc_server_process.c:82` | `DEBUG_GET_ONCE_NUM` | 10000 | Svc | 2 | #951 `[HEALTH]` telemetry cadence; 0 disables |
| `DXR_MAX_CLIENTS` | `ipc/server/ipc_server_process.c:87` | `DEBUG_GET_ONCE_NUM`, consumed `:541-551` | 0 = auto | Svc | 1 | #959 admitted-client cap. Read once at service start |
| `DXR_STEREO_CAMERA` | `ipc/server/ipc_server_stereo_camera.c` (`DEBUG_GET_ONCE_BOOL_OPTION`, read at camera-manager create) | on | Svc | 1 | ADR-043 admin/dev kill switch: `0` = the service enumerates zero stereo cameras (indistinguishable from "none", on purpose) and every start returns `XR_ERROR_STEREO_CAMERA_DISABLED_DXR`. The user-facing twin shipped in R3: the tray / menu-bar "Share the 3D camera with apps" toggle, persisted in the consent store — a privacy control, not a performance lever |
| `DXR_STEREO_CAMERA_UVC` | `auxiliary/util/u_stereo_uvc.c` (`u_stereo_uvc_config_load`, `getenv` at camera-manager create) | unset (= on when configured) | Svc | 1 | ADR-043 Amendment 4: `0` = the service's vendor-neutral UVC side-by-side stereo webcam source claims nothing, whatever `stereo-cameras.json` says. The source is opt-in per device by that file; this is the admin/dev override |
| `DXR_STEREO_CAMERA_UVC_CONFIG` | `auxiliary/util/u_stereo_uvc.c` (`u_stereo_uvc_config_path`, `getenv`) | `%LOCALAPPDATA%\DisplayXR\stereo-cameras.json` / `$XDG_CONFIG_HOME/displayxr/stereo-cameras.json` | Svc | 1 | ADR-043 Amendment 4: path of the UVC stereo webcam config the service reads at start (the per-user file is the normal channel; the service reads ITS OWN environment, so this only helps a service launched with it) |
| `DXR_STEREO_CAMERA_UVC_DECODER` | `auxiliary/os/os_uvc_capture_mf.cpp` (`read_decoder_pref`, `getenv` per open) | `auto` | Svc | 1 | ADR-043 Amendment 4: MJPEG decode path of the UVC stereo webcam source — `auto` (hardware MJPEG decoder MFT on a GPU, else software, one WARN), `hardware` (never fall back: the open fails), `software`. A diagnosis / A-B lever, not a setting |
| `DXR_STEREO_CAMERA_UVC_ADAPTER` | `auxiliary/os/os_uvc_capture_mf.cpp` (`choose_adapter`, `getenv` per open) | unset (= the capable adapter with the most dedicated memory) | Svc | 1 | ADR-043 Amendment 4: description substring of the adapter the hardware MJPEG decode runs on (e.g. `Intel` for the iGPU) |
| `DXR_ULTRALEAP_SETTLE_MS` | `drivers/ultraleap/ultraleap_provider.cpp:1023` | `getenv` at provider init | `UL_SETTLE_DEFAULT_MS` | Both | 1 | How long to wait for a definitive Leap "attached" answer before role arbitration |

### Test / dev — never exposed
Expand Down
1 change: 1 addition & 0 deletions docs/roadmap/stereo-camera-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,7 @@ The implementation belongs in `displayxr-leia-plugin` (`src/drv_leia/`), per ADR
| **R1** ✅ | runtime | **Implemented 2026-09-26 (see "R1 as built" below).** Header `XR_DXR_stereo_camera.h` + `index.json` note (together — the catalog lint requires both); `xrt_plugin_iface` slots + `XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA` (appended after `create_dp_d3d11_lift`, so **lands after ADR-042's PR**); platform-neutral camera manager (thread, refcount + linger, 3-slot pinned ring, per-stream wake handles, decimation, NV12/BGRA conversion); IPC + OpenXR entry points; sim_display fake; `displayxr-cli camera list/calib/probe`; selftest check | `camera probe` on the sim fake writes frames on Windows + Linux CI; ring/pinning unit tests (mirror `tests_lift_mailbox`) |
| **R2** ✅ | runtime | **Implemented 2026-09-26 (see "R2 as built" below).** `u_stereo_rectify` (Bouguet, valid-region crop, LUT remap) with golden tests against OpenCV-generated fixtures; RECTIFIED output | sim fake: row error < 0.5 px, disparity = f·B/Z within 0.5 px — **measured 0.061 px worst block, 0.063 px worst disparity error** |
| **R3** ✅ | runtime | **Implemented 2026-10-03 (see "R3 as built" below).** Privacy: OS consent check, consent store + tray prompt, delegating-client registration, visibility/lock suspension, indicator, kill switches, camera-only client class, distinct refusal results, events delivered, keyed `persistentId`, read-only consumer section | Policy matrix unit-tested with fakes; sim-fake end to end on macOS: refused without consent → allowed after `camera allow --self` / `camera trust` → suspended on a (fake) lock → ended by the kill switch. **Windows tray + WTS path and the manual panel matrix (allow/deny/revoke, OS switch off, real lock screen, background app) still need the Windows box** |
| **U1** ✅ | runtime | **Implemented 2026-10-07 (ADR-043 Amendment 4, spec §9a).** A vendor-neutral, service-owned source for plain UVC side-by-side stereo webcams: opt-in `stereo-cameras.json` (VID:PID / name match, layout, eye order, mode, nominal baseline + HFOV, optional OpenCV calibration), never a plug-in's camera; Windows Media Foundation backend, Linux V4L2 stub; an uncalibrated pair = nominal model, RECTIFIED pass-through aligned by the online refinement; `camera list` source tag, `camera uvc-devices` | Hardware-free: `tests_stereo_uvc` + `tests_stereo_camera_manager` (the real manager on the fake backend: delivered dy +2.40 -> +0.004 px). **Panel box with a real SBS webcam: `camera probe --rectified` passes, the tracker camera untouched — owner run** |
| **L1** | leia-plugin (Windows) | Slots over the SR raw-camera channel per §D; calibration by active serial; keep-alive; repin runtime (feature-macro repin, `downstream-pins.json` `features` track) | On a panel box, incl. the multi-folder box: `camera probe --rectified` rows aligned; the vendor call app running at the same time still gets every frame; tracking/weave unaffected (frame-time + tracking-state logs) |
| **B1** | browser (Windows) | §B device factory + duplicate hiding + hint + delegating registration in the installer; web SDK: prefer the hint in `camera:'auto'`, fill `hello` from it | RFC 0002 P1 call between two panel laptops sends rectified SBS from the tracker camera **while both are tracking** |
| **A1** | leia-plugin (Android) + runtime | §C.1: the runtime owns the front pair and runs CNSDK in-app tracking from the same capture (worker thread, latest-frame slot); per-eye AHB transport + per-eye frame layout; Android consent + visibility | `camera probe` on the tablet via the runtime app **while the weave tracks from the same capture**; pair at 30.00 fps (not 24) |
Expand Down
Loading
Loading