Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
1b18047
feat(stereo-camera): XR_DXR_stereo_camera header, type-value block 10…
dfattal Oct 4, 2026
8a70bee
feat(stereo-camera): u_stereo_camera — latest-wins pinned ring, decim…
dfattal Sep 26, 2026
494a768
feat(stereo-camera): plug-in iface camera slots (XRT_PLUGIN_IFACE_HAS…
dfattal Sep 26, 2026
90dde1a
feat(stereo-camera): service camera manager + IPC protocol (stereo_ca…
dfattal Sep 26, 2026
5c04378
fix(sim_display): fake stereo camera — keep the suspend square wave o…
dfattal Sep 26, 2026
ec63d8f
feat(stereo-camera): displayxr-cli camera list | calib | probe
dfattal Sep 26, 2026
fe9dd3e
feat(stereo-camera): XR_DXR_stereo_camera OpenXR entry points (instan…
dfattal Sep 26, 2026
6275215
feat(stereo-camera): selftest stereo_camera_caps check
dfattal Sep 26, 2026
2668dfc
style(stereo-camera): clang-format 14 the new stereo camera files
dfattal Sep 26, 2026
a769549
fix(stereo-camera): no _Static_assert on XrFlags64 constants (not con…
dfattal Sep 26, 2026
8e589a5
docs(stereo-camera): R1 as built, maintainer decisions, measured Andr…
dfattal Sep 26, 2026
250a2f0
fix(stereo-camera): no cross-enum comparisons in the encoding pins (M…
dfattal Sep 27, 2026
56bd0c4
feat(stereo-camera): report the MEASURED source rate, 0 = unknown, ne…
dfattal Sep 27, 2026
0163e69
fix(stereo-camera): probe disparity search from the camera's geometry…
dfattal Sep 27, 2026
51712a2
chore(stereo-camera): rebase onto post-lift main — placeholder gone, …
dfattal Oct 4, 2026
00325bb
feat(stereo-camera): u_stereo_rectify — Bouguet rectification + LUT r…
dfattal Sep 26, 2026
fd7d75c
feat(sim_display): distorted, misaligned fake stereo camera with grou…
dfattal Sep 26, 2026
ee2a034
test(stereo-camera): golden rectifier tests vs OpenCV + end to end on…
dfattal Sep 26, 2026
c918101
feat(stereo-camera): the service rectifies — RECTIFIED frames + recti…
dfattal Sep 26, 2026
d1e06ee
feat(cli): camera calib --rectified prints P1/P2; probe reports row a…
dfattal Sep 26, 2026
2d57844
docs(stereo-camera): R2 as built — rectifier, distorted fake, probe, …
dfattal Sep 26, 2026
6ba44ef
feat(stereo-camera): WARN when a plug-in's calibration puts the right…
dfattal Sep 26, 2026
e4221f3
fix(stereo-camera): tests_stereo_rectify builds on MSVC (no M_PI)
dfattal Sep 27, 2026
378d711
fix(stereo-camera): probe row alignment = NCC, baseline-aware windows…
dfattal Sep 27, 2026
c86b8a6
feat(sim_display): uncalibrated vertical misalignment knob on the dis…
dfattal Sep 27, 2026
1641d72
feat(stereo-camera): online vertical-alignment refinement core (u_ste…
dfattal Sep 27, 2026
19552f1
feat(stereo-camera): the service refines row alignment online; probe …
dfattal Sep 27, 2026
f3322ba
docs(stereo-camera): R2 online vertical-alignment refinement as built
dfattal Sep 27, 2026
54e8402
fix(cli): probe row gate = |signed median dy| + MAD, not the unsigned…
dfattal Sep 27, 2026
6b9df2d
fix(ipc): PRESENT_OWNER quota counts owners (executables), not connec…
dfattal Sep 27, 2026
298b6a5
feat(util): stereo camera consent policy, platform store, keyed persi…
dfattal Oct 4, 2026
d658a97
feat(ipc): CAMERA_CONSUMER client class — camera-only consumers outsi…
dfattal Oct 4, 2026
93bff14
feat(stereo-camera): the service enforces consent, foreground and loc…
dfattal Oct 4, 2026
7f4678b
feat(service): Windows tray camera indicator, consent dialog, kill sw…
dfattal Oct 4, 2026
c3b6d41
feat(cli): camera consent/allow/deny/revoke/trust, status, stop-all, …
dfattal Oct 4, 2026
19d2d6c
docs(stereo-camera): R3 consent and privacy as built
dfattal Oct 4, 2026
fa524ce
fix(stereo-camera): probe exits 6 when camera sharing is off; tray pr…
Oct 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ Contribute to the DisplayXR runtime — compositors, state tracker, auxiliary co
- **[comp_multi One Pipeline (macOS/Linux/Android)](architecture/comp-multi-one-pipeline.md)** — #967 design note: bounded waits, shared-surface port to Android, issue split
- **[In-Process vs Service](architecture/in-process-vs-service.md)** — compositor deployment modes
- **[Implementing an Extension](guides/implementing-extension.md)** — how to add OpenXR extensions
- **[Stereo camera consent](guides/stereo-camera-consent.md)** — what gates `XR_DXR_stereo_camera` frames (consent prompt, delegating browsers, foreground rule, kill switches) and how an app or browser declares itself

### Internal Specs (`specs/runtime/`)

Expand Down
48 changes: 43 additions & 5 deletions docs/adr/ADR-043-stereo-camera-source.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
# ADR-043: A display's stereo camera is a plug-in-provided source, owned by the service and privacy-gated by the runtime

**Status:** Proposed (2026-09-25) — design only, nothing implemented · introduces
**Status:** Proposed (2026-09-25) · **R1 implemented** 2026-09-26 (runtime, hardware-free; maintainer
defaults recorded in the roadmap §G: stereo-only, iface slots, service clients only, no raw frames
to pages) · **R2 implemented** 2026-09-26 (the service-side rectifier, decision 2 below) · introduces
[`XR_DXR_stereo_camera`](../specs/extensions/XR_DXR_stereo_camera.md) · appends optional
camera slots to `xrt_plugin_iface` under the [ADR-020](ADR-020-plugin-abi-compatibility-policy.md)
append-at-end rule · related: [ADR-019](ADR-019-vendor-plugin-aux-boundary.md),
[ADR-034](ADR-034-input-provider-plugins.md), [ADR-035](ADR-035-service-owned-arbitration-single-pipeline-isolated-satellites.md),
ADR-042 (vendor 2D→3D conversion; in flight on branch `feat/lift-ext`, not yet on `main`) (the same shape in reverse) ·
[ADR-042](ADR-042-vendor-2d3d-conversion-supersedes-default.md) (vendor 2D→3D conversion — the same shape in reverse) ·
integration + plan: [roadmap/stereo-camera-source.md](../roadmap/stereo-camera-source.md)

## Context
Expand All @@ -30,7 +32,11 @@ machines):
correct design must key calibration by the **active** device; and the SDK's new-frame event is
**auto-reset**, so two readers waiting on it steal each other's wake-ups.
- 3D **tablets** carry a front stereo pair (15 mm and 25 mm baselines on two models, parallel)
owned by the vendor's camera/tracking service on Android.
owned by the vendor's camera/tracking service on Android. Measured 2026-09-26 (roadmap §C.0):
the pair is a hidden Camera2 **logical** camera (front 5, back 4) that delivers two raw 1280×720
NV12 streams at 30 fps. Opening the front one **evicts the vendor tracker, which never
reacquires**. The vendor camera SDK (CNSDK) supplies only calibration and in-app face tracking,
not frames. The DisplayXR Browser cannot see the pair at all (browser-pvt#175).

Everything a correct stereo camera needs — the private frame channel, the device's calibration,
keeping the tracker alive — is vendor knowledge. The runtime's plug-in model (ADR-019) exists
Expand Down Expand Up @@ -87,5 +93,37 @@ exactly so that knowledge lives in one vendor DLL and nowhere else.
defensively until the vendor SDK formalises it (roadmap open questions).
- **−** Frame quality is the tracker's: greyscale, 640×480, tracker-chosen rate/exposure.
- **−** Another service thread, and a CPU copy per frame (≈ 27 MB/s at 1280×480 NV12 30 Hz).
- **−** Android depends on the vendor camera SDK sharing frames at all (open question); until then
Android documents a platform difference rather than breaking isolation.
- **−** On Android the source cannot be a passenger on the tracker: the pair and the tracker cannot
both hold Camera2. So the runtime (the vendor plug-in in the runtime APK, ADR-038) must own the
front pair and run the vendor's in-app tracking from the same capture, on a worker thread
(roadmap §C.1). That is more vendor code inside the runtime APK, and more CPU, than on Windows.

## Amendment 1 (2026-10-03, R3 as built) — what changed from the decision text

- **A camera-only client class.** The decision assumed the browser's capture component would
be classed as the browser (PRESENT_OWNER, by enabling `XR_DXR_weave`). In the field the
video-capture *utility* process is a separate OS process with no window and no weave, and
classing it PRESENT_OWNER spent a panel-owner slot. R3 adds `CAMERA_CONSUMER`, declared
explicitly (`XrStereoCameraClientInfoDXR`), session-less by contract and outside the
present-owner quota. Consent is still keyed by the verified executable, so the browser's
installer registration (delegating) covers the utility too.
- **Distinct refusal results instead of one `PERMISSION_INSUFFICIENT`.** A browser must map
"the user said no" to `NotAllowedError` and "the device cannot be read" to
`NotReadableError`; one code could not carry both. `PERMISSION_INSUFFICIENT` now means the OS
camera switch (or a class that may not use cameras); consent, sharing-off, busy and
service-ended each have their own `XrResult`.
- **The foreground rule is window visibility of the peer pid, not session state.** The spec
said "a session in VISIBLE / FOCUSED"; a camera consumer may have no session at all, and the
service already has the peer pid. Window-bearing classes need a visible top-level window;
delegating, camera-consumer and diagnostic clients are exempt (they were granted explicit
consent). The OS-lock suspension is unconditional.
- **The prompt blocks the caller (≤ 60 s) rather than returning "pending".** A capture
device's `AllocateAndStart` wants a final answer, and the stream handle stays valid either
way; an unanswered prompt is a refusal the app may retry. Unanswered is never persisted.
- **Two user kill switches, not one.** "Stop camera sharing" ends every started stream now
(apps see `STREAM_ENDED`); "Share the 3D camera with apps" is the persistent toggle the
decision described (zero cameras enumerated).
- **The dev override stayed** (`DXR_STEREO_CAMERA_DEV_ALLOW=1`) as a documented, loud
development switch below the sharing-off checks — the R1 text had it as a stand-in to be
deleted.

145 changes: 145 additions & 0 deletions docs/guides/stereo-camera-consent.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# Stereo camera consent — for app and browser developers

How the DisplayXR runtime decides whether your process may read the display's stereo camera
(`XR_DXR_stereo_camera`, [ADR-043](../adr/ADR-043-stereo-camera-source.md)), what the user sees,
and what you have to do. The normative text is the extension spec,
[§7 *Privacy*](../specs/extensions/XR_DXR_stereo_camera.md#7-privacy); this page is the
walkthrough.

**Why the runtime does this at all.** The camera is the eye tracker's. The tracker opened the
device, so the OS camera permission model and its in-use light never see *you* as a consumer. The
runtime therefore enforces the equivalent itself: per-executable consent, a foreground rule, an
in-use indicator, kill switches, and an unlinkable device id. The vendor plug-in sees none of it.

## 1. The three kinds of client

| You are… | Declare | What gates frames |
|---|---|---|
| **A native app with a window** (a 3D call app) | nothing special — enable the extension | consent (§2) + your window must be visible (§3) |
| **A camera-only process** — a browser's video-capture utility, a capture helper, `displayxr-cli camera` | `XrStereoCameraClientInfoDXR` with `XR_STEREO_CAMERA_CLIENT_CONSUMER_ONLY_BIT_DXR` chained on `XrInstanceCreateInfo` | consent (§2); no window rule (you have none) |
| **A consent-delegating client** — a browser that shows its *own* per-origin camera prompt and in-use indicator | the camera-only declaration **and** a registration of your executable (§4) | your own prompt is the consent; your own visibility rule; never RAW frames |

The camera-only declaration makes the service admit you as the `CAMERA_CONSUMER` class: you may
call only the camera entry points (`xrCreateSession` is refused), and you are **not a panel
owner** — a browser's capture utility no longer spends one of the two present-owner slots, even
though its executable is the browser's. Chain it like this:

```c
XrStereoCameraClientInfoDXR consumer = {
.type = XR_TYPE_STEREO_CAMERA_CLIENT_INFO_DXR,
.flags = XR_STEREO_CAMERA_CLIENT_CONSUMER_ONLY_BIT_DXR,
};
XrInstanceCreateInfo ci = {XR_TYPE_INSTANCE_CREATE_INFO, &consumer, /* … XR_DXR_stereo_camera in enabledExtensionNames */};
```

## 2. Consent: what happens at `xrStartStereoCameraStreamDXR`

The service looks at the **OS-derived executable path** of your process (never anything you
send) and walks this list; the first hit decides. The same check runs for
`xrGetStereoCameraCalibrationDXR` (calibration identifies the device).

1. **Sharing off** — the user's tray / menu-bar toggle *"Share the 3D camera with apps"*, or
`DXR_STEREO_CAMERA=0` in the service's environment → `XR_ERROR_STEREO_CAMERA_DISABLED_DXR`.
(You also enumerate zero cameras.)
2. `DXR_STEREO_CAMERA_DEV_ALLOW=1` in the **service's** environment → allowed. A development
override; the service logs one WARN per run. Never on a user's machine.
3. Your executable path could not be verified → `XR_ERROR_STEREO_CAMERA_CONSENT_REFUSED_DXR`.
4. You are a **registered delegating client** → allowed, no prompt, nothing stored.
5. **Windows:** the OS camera privacy switch denies you (Settings → Privacy → Camera: globally,
for desktop apps, or for your app) → `XR_ERROR_PERMISSION_INSUFFICIENT`.
6. A **stored decision** for your executable (the user answered before, or an admin pre-seeded
it) → allowed, or `CONSENT_REFUSED`.
7. The user chose **Allow once** earlier for this very process → allowed.
8. The **prompt**: the service's tray (Windows) / menu-bar item (macOS) shows

> **"<your application name>" wants to use the 3D camera**
> *your-app.exe (pid 1234) will receive frames from the display's stereo camera. You can stop
> it at any time from the DisplayXR menu.*
> [ Deny ] [ Allow once ] [ Allow ]

**Allow** and **Deny** are remembered for your executable; **Allow once** lasts for this
process. Your `xrStartStereoCameraStreamDXR` call **blocks** while the dialog is up (up to
60 s). Unanswered, dismissed, or no prompt available (`DXR_STEREO_CAMERA_PROMPT=0`, a headless
Linux service) → `CONSENT_REFUSED`, nothing stored — you may try again later.

A refused stream is not destroyed: call start again after the user has allowed you. Map the
results like this in a browser:

| Result | Meaning | `getUserMedia` |
|---|---|---|
| `XR_ERROR_STEREO_CAMERA_CONSENT_REFUSED_DXR` | the user / policy said no | `NotAllowedError` |
| `XR_ERROR_PERMISSION_INSUFFICIENT` | the OS camera switch is off (or wrong client class / RAW asked) | `NotAllowedError` |
| `XR_ERROR_STEREO_CAMERA_DISABLED_DXR` | sharing is off | hide the device, or `NotReadableError` |
| `XR_ERROR_STEREO_CAMERA_BUSY_DXR` | the source cannot be opened right now | `NotReadableError` |
| `XR_ERROR_STEREO_CAMERA_STREAM_ENDED_DXR` | the service ended your stream (see §5) | end the track |

## 3. The foreground rule and the lock screen

If you are a window-bearing client (an ordinary app, or a present-owner), frames are published
only while your process **owns a visible, non-minimised top-level window** (checked every
250 ms). Otherwise your stream is `SUSPENDED`: no new frames, and the frame you had pinned is
cleared so nothing lingers. Camera-only and delegating clients are exempt — the browser's own
tab rule applies to its pages.

Every stream, of every client, is suspended while the **OS session is locked or switched away**
(Windows lock / disconnect, macOS screen lock / fast user switch) and resumes afterwards. You
see it as `XrEventDataStereoCameraStateChangedDXR` with `SUSPENDED` then `AVAILABLE` from
`xrPollEvent` (no session needed), and `xrEnumerateStereoCamerasDXR` reports `SUSPENDED`
meanwhile. Keep showing your last frame or a placeholder; do not tear the stream down.

## 4. Being a consent-delegating client (browsers)

A browser already asks the user per origin and shows its own in-use indicator, so the runtime
should not ask a second time per page. Register the browser's **executable path**:

- **Installer (machine-wide):** Windows `HKLM\Software\DisplayXR\CameraConsent\Delegating`, a
`REG_SZ` value whose *data* is the full path (value name: anything, e.g. the file name);
Linux `/etc/displayxr/camera-delegating.json`; macOS
`/Library/Application Support/DisplayXR/camera-delegating.json` — both
`{"delegating": ["/full/path/to/browser", …]}`.
- **Per user / development:** `displayxr-cli camera trust <exe>` (or `--self`), which writes the
user-level list (`HKCU\…\CameraConsent\Delegating`, `~/.config/…/camera_consent.json`).

Obligations that come with it: prompt the user yourself before opening a stream, show your own
in-use indicator, stop the stream when your page stops capturing, and never expose raw
calibration or the raw frame — the runtime enforces the last one (RAW is refused to delegating
clients; `persistentId` is already a per-executable keyed hash, coarsen the rest yourself).

An unsigned / development build that is not registered is simply a camera-only client: the
runtime prompts for it once (Allow is remembered for that path).

## 5. What the user can do to you at any time

- **"Stop camera sharing"** (tray / menu bar): every started stream ends. You receive
`XrEventDataStereoCameraStreamEndedDXR { stream, USER_STOPPED }`, your wake handle fires
once, and acquire / start on that handle return `XR_ERROR_STEREO_CAMERA_STREAM_ENDED_DXR`.
Destroy it. You may create and start a new one — that goes through §2 again (and "Allow once"
grants were forgotten).
- **"Share the 3D camera with apps" off**: same, with reason `DISABLED`; `xrEnumerate…` then
returns zero cameras until it is switched back on.
- Change their answer: `displayxr-cli camera allow|deny|revoke <exe>` edits the store; the next
start sees it.

While your stream runs the user sees the runtime's icon badge and *"3D camera in use by
your-app.exe"* in the tray / menu bar — in addition to whatever you show.

## 6. Developing and testing

```sh
# the service, with the sim fake camera; prompts off = a headless CI service
SIM_DISPLAY_FAKE_STEREO_CAMERA=1 DXR_STEREO_CAMERA_PROMPT=0 displayxr-service

displayxr-cli camera probe --seconds 2 # exit 4: CONSENT_REFUSED (no consent, no prompt)
displayxr-cli camera allow --self # or: camera trust --self (delegating)
displayxr-cli camera probe --seconds 2 # frames
displayxr-cli camera fake-lock on # every stream SUSPENDED (event printed by probe)
displayxr-cli camera fake-lock off
displayxr-cli camera stop-all # a running probe prints "ENDED … USER_STOPPED", exits 8
displayxr-cli camera sharing off # zero cameras; start -> DISABLED (exit 6)
displayxr-cli camera consent # what is stored, and this CLI's own path
```

`DXR_STEREO_CAMERA_DEV_ALLOW=1` on the service skips consent for everything (loud WARN) — use
it on a dev box when you are iterating on frames, not on the consent path. The full env-var
census is in
[`docs/roadmap/control-panel-performance-settings.md`](../roadmap/control-panel-performance-settings.md).
Loading
Loading