diff --git a/docs/adr/ADR-043-stereo-camera-source.md b/docs/adr/ADR-043-stereo-camera-source.md index 90df2a3fa4..9495750153 100644 --- a/docs/adr/ADR-043-stereo-camera-source.md +++ b/docs/adr/ADR-043-stereo-camera-source.md @@ -1,11 +1,13 @@ # ADR-043: A display's stereo camera is a plug-in-provided source, owned by the service and privacy-gated by the runtime -**Status:** Proposed (2026-09-25) — design only, nothing implemented · introduces +**Status:** Proposed (2026-09-25) · **R1 implemented** 2026-09-26 (runtime, hardware-free; maintainer +defaults recorded in the roadmap §G: stereo-only, iface slots, service clients only, no raw frames +to pages) · introduces [`XR_DXR_stereo_camera`](../specs/extensions/XR_DXR_stereo_camera.md) · appends optional camera slots to `xrt_plugin_iface` under the [ADR-020](ADR-020-plugin-abi-compatibility-policy.md) append-at-end rule · related: [ADR-019](ADR-019-vendor-plugin-aux-boundary.md), [ADR-034](ADR-034-input-provider-plugins.md), [ADR-035](ADR-035-service-owned-arbitration-single-pipeline-isolated-satellites.md), -ADR-042 (vendor 2D→3D conversion; in flight on branch `feat/lift-ext`, not yet on `main`) (the same shape in reverse) · +[ADR-042](ADR-042-vendor-2d3d-conversion-supersedes-default.md) (vendor 2D→3D conversion — the same shape in reverse) · integration + plan: [roadmap/stereo-camera-source.md](../roadmap/stereo-camera-source.md) ## Context @@ -30,7 +32,11 @@ machines): correct design must key calibration by the **active** device; and the SDK's new-frame event is **auto-reset**, so two readers waiting on it steal each other's wake-ups. - 3D **tablets** carry a front stereo pair (15 mm and 25 mm baselines on two models, parallel) - owned by the vendor's camera/tracking service on Android. + owned by the vendor's camera/tracking service on Android. Measured 2026-09-26 (roadmap §C.0): + the pair is a hidden Camera2 **logical** camera (front 5, back 4) that delivers two raw 1280×720 + NV12 streams at 30 fps. Opening the front one **evicts the vendor tracker, which never + reacquires**. The vendor camera SDK (CNSDK) supplies only calibration and in-app face tracking, + not frames. The DisplayXR Browser cannot see the pair at all (browser-pvt#175). Everything a correct stereo camera needs — the private frame channel, the device's calibration, keeping the tracker alive — is vendor knowledge. The runtime's plug-in model (ADR-019) exists @@ -87,5 +93,7 @@ exactly so that knowledge lives in one vendor DLL and nowhere else. defensively until the vendor SDK formalises it (roadmap open questions). - **−** Frame quality is the tracker's: greyscale, 640×480, tracker-chosen rate/exposure. - **−** Another service thread, and a CPU copy per frame (≈ 27 MB/s at 1280×480 NV12 30 Hz). -- **−** Android depends on the vendor camera SDK sharing frames at all (open question); until then - Android documents a platform difference rather than breaking isolation. +- **−** On Android the source cannot be a passenger on the tracker: the pair and the tracker cannot + both hold Camera2. So the runtime (the vendor plug-in in the runtime APK, ADR-038) must own the + front pair and run the vendor's in-app tracking from the same capture, on a worker thread + (roadmap §C.1). That is more vendor code inside the runtime APK, and more CPU, than on Windows. diff --git a/docs/reference/xrt_plugin_iface.md b/docs/reference/xrt_plugin_iface.md index 8a87f30823..40f5e18bb7 100644 --- a/docs/reference/xrt_plugin_iface.md +++ b/docs/reference/xrt_plugin_iface.md @@ -606,6 +606,47 @@ the runtime logs one warning and runs unpaced. Either way nothing fails — the difference is accuracy, not correctness. Application-side guidance: [INV-5.9](../guides/displayxr-app-rules.md). +## A stereo camera: the `stereo_camera_*` slots (ADR-043) + +Optional, appended per ADR-020, announced by `XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA`; the runtime +uses them only when `struct_size` covers `stereo_camera_close` **and** all six are non-NULL +(`xrt_plugin_iface_has_stereo_camera()`). They let a plug-in expose the display's own stereo +camera — usually the one its eye tracker looks through — as a runtime source +([`XR_DXR_stereo_camera`](../specs/extensions/XR_DXR_stereo_camera.md)). The service opens each +camera once and fans frames out to every authorised client; the plug-in only produces frames. + +```c +uint32_t (*stereo_camera_enumerate)(inst, capacity, struct xrt_plugin_stereo_camera_info *out); +xrt_result_t (*stereo_camera_get_calibration)(inst, index, struct xrt_plugin_stereo_camera_calibration *out); +xrt_result_t (*stereo_camera_open)(inst, index, struct xrt_plugin_stereo_camera **out_cam); +uint32_t (*stereo_camera_wait_frame)(cam, int64_t timeout_ns, struct xrt_plugin_stereo_camera_frame *out); +void (*stereo_camera_release_frame)(cam); +void (*stereo_camera_close)(cam); +``` + +| Rule | Why | +|---|---| +| **Never take the device from the tracker.** Read the frames the tracker already publishes (shared memory, a vendor camera service); never open, reconfigure or re-time the device. | The whole point: while tracking runs the device is exclusive (Windows `NotReadableError`), and on Android opening the front pair evicts the tracker (roadmap §C) | +| `stereo_camera_open` is a **tracker keep-alive**: while any camera is open, keep the tracker running. | Frames only exist while tracking runs | +| `wait_frame` is called from **one runtime-owned thread per open camera**, may block up to `timeout_ns`, and returns an `enum xrt_plugin_stereo_camera_wait` (`OK` / `TIMEOUT` / `SUSPENDED` / `ERROR`). Planes stay valid until `release_frame`. | The service maps `TIMEOUT` → `WAITING` (then `SUSPENDED` after 1 s without frames), `SUSPENDED` → `SUSPENDED`, `ERROR` → close + retry in 1 s | +| **Never wait on a wake object shared with other readers** of the vendor channel — poll it. | An auto-reset event wakes exactly one waiter: two readers steal each other's frames (ADR-043) | +| Decode your transport (JPEG, YUY2, …) to `GRAY8`, `NV12` or `BGRA8` yourself; always deliver **one side-by-side image**, `2·eye_width × eye_height`, left eye left, unmirrored, even extents. | The runtime owns format conversion, decimation, rectification (R2) and transport; one SBS layout for every vendor | +| Calibration is the **ACTIVE** device's — the unit you weave for — keyed by its own identity (serial). RAW, OpenCV convention: `k[eye] = fx fy cx cy`, `distortion_model` + coefficients, and `R`, `T` with **`x_R = R·x_L + T`** (T in mm; a right camera at +x of the left has `T = (−B, 0, 0)`). The runtime converts it to the client's pose-of-the-right-camera form. Return `XRT_ERROR_FEATURE_NOT_SUPPORTED` and do **not** set `CALIBRATED` if you cannot resolve it. | One test box carried eleven calibration folders; "the first one" is the wrong camera | +| `device_identity` is private: the service hashes it per consumer (`persistentId`) and never exports it. | Fingerprinting (spec §7.5) | +| Nothing throws across the boundary. | C ABI | + +Flags (`XRT_PLUGIN_STEREO_CAMERA_*`) mirror the XR bits: `SHARED_WITH_EYE_TRACKING`, +`USER_FACING`, `CALIBRATED`, `NATIVELY_RECTIFIED`, `MONOCHROME`. The reference implementation is +`sim_display`'s fake (`drivers/sim_display/sim_display_stereo_camera.c`, enabled with +`SIM_DISPLAY_FAKE_STEREO_CAMERA=1` in the **service** environment). + +**Slot order.** The six slots are the LAST members of the iface, appended (ADR-020) after +everything that was on `main` before them: `… vk_bundle_fn_table_offset` → `create_dp_d3d11_lift` +(ADR-042) → `get_platform_state` (ADR-045) → `stereo_camera_enumerate … stereo_camera_close`. +`tests_stereo_camera` pins that order (lift right after the vk fingerprint, platform-state right +after lift, the camera block right after platform-state and ending the struct), so a reorder or a +slot squeezed in between fails on the host, not on a vendor box. + ## Aux surface — separate from the iface Logging, debug-variable tracking, frame metrics, Perfetto tracing, and unique-ID generation are **not** plumbed through this iface. Plug-ins reach them by linking the runtime DLL's import library (`DisplayXRClient.lib`) and getting `__declspec(dllimport)`'d symbols. See [ADR-019](../adr/ADR-019-vendor-plugin-aux-boundary.md) for the rationale. diff --git a/docs/roadmap/control-panel-performance-settings.md b/docs/roadmap/control-panel-performance-settings.md index 03288919e7..6dbc0724cf 100644 --- a/docs/roadmap/control-panel-performance-settings.md +++ b/docs/roadmap/control-panel-performance-settings.md @@ -324,7 +324,7 @@ never claim a mode the runtime is not in, and "Custom" falls out for free. ## Appendix A — census Every `DXR_*` name read at runtime under `src/xrt`, with its read site, mechanism, default -and tier. **100 distinct names**; the two `DXR_BG2D_*` knobs reach the environment through +and tier. **102 distinct names**; the two `DXR_BG2D_*` knobs reach the environment through `bg2d_int_knob()` rather than a literal `getenv` at the listed line. Process column: **App** = the runtime DLL, loaded into the OpenXR app's process · @@ -546,6 +546,7 @@ as `docs/specs/vendor/oem-android-platform-requirements.md` §R6. | `DXR_DEVICE_REMOVED_EXIT_MS` | `compositor/d3d11_service/comp_d3d11_service.cpp:10279` | `getenv`, `static` cached | 2000 | Svc | 2 | Delay before the service exits on an unrecoverable `DEVICE_REMOVED` | | `DXR_HEALTH_MS` | `ipc/server/ipc_server_process.c:82` | `DEBUG_GET_ONCE_NUM` | 10000 | Svc | 2 | #951 `[HEALTH]` telemetry cadence; 0 disables | | `DXR_MAX_CLIENTS` | `ipc/server/ipc_server_process.c:87` | `DEBUG_GET_ONCE_NUM`, consumed `:541-551` | 0 = auto | Svc | 1 | #959 admitted-client cap. Read once at service start | +| `DXR_STEREO_CAMERA` | `ipc/server/ipc_server_stereo_camera.c` (`DEBUG_GET_ONCE_BOOL_OPTION`, read at camera-manager create) | on | Svc | 1 | ADR-043 admin/dev kill switch: `0` = the service enumerates zero stereo cameras (indistinguishable from "none", on purpose). The user-facing twin is the planned R3 "Share the 3D camera with apps" toggle — a privacy control, not a performance lever | | `DXR_ULTRALEAP_SETTLE_MS` | `drivers/ultraleap/ultraleap_provider.cpp:1023` | `getenv` at provider init | `UL_SETTLE_DEFAULT_MS` | Both | 1 | How long to wait for a definitive Leap "attached" answer before role arbitration | ### Test / dev — never exposed @@ -570,6 +571,7 @@ as `docs/specs/vendor/oem-android-platform-requirements.md` §R6. | `DXR_VIEW_CONFIG_LEGACY` | `state_trackers/oxr/oxr_system.c` (`oxr_system_fill_in`, `DEBUG_GET_ONCE_BOOL_OPTION`) | off | App | #1486 kill switch: restores the pre-#1486 single view configuration — `PRIMARY_STEREO` for any `view_count >= 2`, reporting the max across modes (4 on sim-display), and no `PRIMARY_MULTIVIEW_DXR`. Reintroduces the spec deviation and the CTS red on purpose, for one round of field bisection. Logs one `U_LOG_W` when active. 22 characters, so `debug.xrt.DXR_VIEW_CONFIG_LEGACY` is **stranded on Android** by the `PROP_NAME_MAX` trap above — env-only in practice. See [View-Configuration Model](../reference/view-configuration-model.md) | | `DXR_MODE_FLOOR` | `state_trackers/oxr/oxr_session.c` (`DEBUG_GET_ONCE_BOOL_OPTION`, one TU; `oxr_api_session.c` reads it through `oxr_session_mode_floor_enabled()`) | on | App | #1499 kill switch. `0` restores the pre-#1499 behaviour for EXTENSION sessions: `xrBeginSession` leaves the display in whatever rendering mode it found, and `xrRequestDisplayRenderingModeDXR` honours a request whose `viewCount` exceeds the session's view configuration — so the compositor's under-submit clamp paints the first tiles and the rest stay at the clear colour, which is exactly the state the issue objected to. It also silences the two #1499 log lines (the begin-time `UNFILLABLE` warning and the observation warning on a display-global mode change), because `0` promises pre-#1499 behaviour and a line the runtime never used to print is part of that promise. **One switch for BOTH halves on purpose**: a floor without the denial lets an app walk straight back into the mode the floor removed it from, and a denial without the floor refuses a mode the session is already in — half of this feature is not a state anyone should be able to bisect into. #1510's LEGACY floor is deliberately NOT affected (a legacy app was sized for its floored mode at `xrGetSystem`, so leaving it in a wider one would make the compromise view scale and the mode disagree). See [View-Configuration Model](../reference/view-configuration-model.md) | | `DXR_UNDER_SUBMIT` | `state_trackers/oxr/oxr_session_frame_end.c` (`DEBUG_GET_ONCE_NUM_OPTION`, one TU; the value→enum map is the pure `oxr_under_submit_from_setting()` in `oxr_view_config_rule.h`) | `1` | App | ADR-041 staging switch for the fixed-view-count submission contract. `0` = strict: every view configuration must submit exactly the count `xrLocateViews` returned. `1` (default) = strict **plus** the deprecated `PRIMARY_STEREO` 1-view arm (an `XR_DXR_display_info` instance submitting 1 while the active rendering mode is 1-view), which is accepted and logged **once per session** with a line naming the fix — released demos submit one view in 2D mode and need the compat window. `2` = kill switch: the pre-ADR-041 rules, `PRIMARY_MULTIVIEW_DXR` under-submit included. Out-of-range values **clamp to an end** rather than falling back to the default, so a typo cannot silently land on the default. The default flips to `0` in the first runtime release after `displayxr-common` and the five `displayxr-demo-*` demos ship the alias submission — the trigger is that shipment, not a date. Not a user-facing setting: it is a compat bisect knob with no performance dimension. Unreachable from a CTS run at any value (a conformance session never enables `XR_DXR_display_info`, which the compat arm requires). Distinct from `DXR_VIEW_CONFIG_LEGACY`, which restores the pre-#1486 *reporting*. See [ADR-041](../adr/ADR-041-fixed-view-count-with-per-frame-activity.md) and [View-Configuration Model](../reference/view-configuration-model.md) | +| `DXR_STEREO_CAMERA_DEV_ALLOW` | `ipc/server/ipc_server_stereo_camera.c` (`authorise_locked`, `DEBUG_GET_ONCE_BOOL_OPTION`) | off | Svc | ADR-043 **R1-only** stand-in for camera consent: stream start and calibration reads are REFUSED (`XR_ERROR_PERMISSION_INSUFFICIENT`) unless it is set in the SERVICE environment. Deleted when R3 lands the OS + DisplayXR consent path. Never a setting | | `DXR_IPC_FD` | `auxiliary/util/u_sandbox.c:181`; `ipc/client/ipc_client_connection.c:211, 234` | unset | Both / App | #1056 adopts an embedder-supplied service socket. Not a setting | | `DXR_IPC_HANDLE` | `ipc/client/ipc_client_connection.c:437, 456` | unset | App | Windows analogue of the above | | `DXR_WL_ORIGIN_STAMP` | `compositor/vk_native/comp_vk_native_compositor.c` (`vk_debug_origin_stamp`, Linux + Wayland) | off | App | #1748 measurement hook: writes a barcode of each frame's present origin into its top-left 352x8 device px — it **overwrites content**. Read back by the extension's `DISPLAYXR_STAMP_AUDIT=1` (shell environment) in `scripts/linux/move_sync/`. Never for a real session | diff --git a/docs/roadmap/stereo-camera-source.md b/docs/roadmap/stereo-camera-source.md index 5026f11648..575344dfc6 100644 --- a/docs/roadmap/stereo-camera-source.md +++ b/docs/roadmap/stereo-camera-source.md @@ -1,6 +1,7 @@ # Stereo camera source — browser integration, vendor plug-in notes, phased plan -**Status:** design (2026-09-25). Decision: [ADR-043](../adr/ADR-043-stereo-camera-source.md). +**Status:** design (2026-09-25); **R1 implemented** on `feat/stereo-camera-r1` (2026-09-26) — +see §E; maintainer decisions for R1 in §G. Decision: [ADR-043](../adr/ADR-043-stereo-camera-source.md). Extension: [`XR_DXR_stereo_camera`](../specs/extensions/XR_DXR_stereo_camera.md). Consumer driving it: the web SDK's 3D call module (`@displayxr/inline3d/call`, RFC 0002 on `displayxr-web` branch `feat/call-p1`, §4 *Capture*). @@ -94,32 +95,82 @@ process across milestones): ## C. Android -### C.1 Runtime + plug-in - -On the tablets the front stereo pair (15 mm and 25 mm baselines on the two current models, -parallel) belongs to the vendor's head-tracking service; the plug-in (`drv_leia_android`) talks -to it through the vendor camera SDK (CNSDK), which today gives it **face/eye data, not images**. - -- **If CNSDK can share frames** (an AHardwareBuffer or buffer stream of the pair the tracker - already captures, with the pair's calibration): the Android plug-in implements the same slots; - the runtime service's camera thread, rings (memfd), consent (CAMERA permission of the peer uid + - runtime-app consent) and visibility rule (`android_package_is_visible`) are the platform-neutral - code of spec §5–7. This is the target, and the **first open question to the vendor**. -- **If it cannot yet — documented platform difference:** Android enumerates no runtime stereo - camera. We do **not** have the plug-in or the browser open the front cameras through Camera2 - while tracking runs: Camera2 evicts the lower-priority client, which is the tracker, and the - weave would lose viewer tracking for the whole call. Pages see the tablet's ordinary cameras via - `getUserMedia`; the call module sends mono (lifted on the receiver). -- Frames on this path are likely **colour and larger** (per-eye 1280×720 class), which is where - the optional `AHARDWAREBUFFER` transport earns its keep. +### C.0 Field facts (NP02J / K68 tablet, measured 2026-09-26) + +**Camera topology.** Camera2 reports six ids on the tablet: 0 and 2 are back lenses, 1 and 3 front +lenses, and **4 = back and 5 = front are LOGICAL multi-cameras — the stereo pairs**. The logical +ids are hidden from `getCameraIdList()` and are opened by hard-coded id. Physical lens per eye: +front L = 3, R = 1; back L = 2, R = 0. + +**The browser cannot reach them** (DisplayXR/displayxr-browser-pvt#175): `enumerateDevices()` +lists no video inputs, `getUserMedia` only opens the front single camera, and +`facingMode: 'environment'` fails with `NotFoundError`. + +**How the vendor's own app gets stereo — and what CNSDK is, and is not.** The vendor app uses +`LeiaCameraBuilder(ctx).setStereoMode(true).setFront(b).setConvergenceMode(AUTO) +.setPreviewSurfaceView(surfaceTexture)` and receives a ready SBS image (2560×720) in a +`SurfaceTexture`. **`LeiaCameraBuilder` is not part of CNSDK:** it is a separate, private camera +wrapper over plain Camera2. It opens the one logical camera (5 front / 4 back) with an output per +physical lens, and its own shader does the SBS pack and the rectification. **CNSDK supplies only +the calibration** (per-camera rotations Rx/Ry/Rz from its display-config service, config v4.0; +the `/sdcard/rect.js` file the vendor app also reads is absent on the unit) **and in-app face +tracking.** + +**What the pair delivers.** Two separate 1280×720 NV12/YUV streams, one per lens, at exactly +30.00 fps, hardware-synchronised (identical timestamps per pair), about 97–100 ms from exposure to +the app. The frames arrive **raw**: not side-by-side, not rectified, not mirrored. For viewer +left/right the front eyes need mirroring. The residual vertical misalignment is about −2 px +(median), and the roll about 0.09°. + +**Exclusivity — the constraint that decides the design.** Opening the front pair (camera 5) +**evicts the head-tracking service's camera** (it held camera 3 at 10 fps). The tracker then stays +wedged ("Camera has been invalidated") and never reacquires it. The rear pair does not conflict. +The vendor's answer is **in-app tracking**: the capturing process feeds a third stream (640×480, +left lens) to CNSDK in-app mode. CNSDK's frame hand-off runs face detection **on the caller's +thread** (~119 ms). Called on the camera thread, that dragged the pair to 24 fps, so it must run +on a worker behind a latest-frame slot. + +**Windows, same week:** on two SR machines the eye tracker holds the stereo camera exclusively +(`getUserMedia` → `NotReadableError`). The design (plug-in source, service owner) follows from +that on both platforms. + +### C.1 Runtime + plug-in — design consequence + +On Android the provider lives **inside the runtime APK** (the vendor plug-in ships there, ADR-038). +It is not Chromium, and it is not a separate app. Because the front pair and the tracker cannot +coexist as two Camera2 clients: + +- **The runtime service is the ONE owner of the front pair, and it runs tracking in-process from + that same capture.** Three outputs of the logical camera: the two 1280×720 NV12 eye streams, and a + 640×480 left-lens stream that goes to CNSDK in-app tracking on a **worker thread behind a + latest-frame slot**, never on the camera thread. The runtime's eye tracking on the tablet then + comes from this capture, and camera streams to clients are passengers on it (the Windows model, + reversed: there the tracker owns the device and the plug-in reads its frames). +- **Nothing else may open the pair behind the tracker's back**: not a page, not another plug-in, not + an app through Camera2. Doing so evicts the capture that tracking depends on, and the display + loses viewer tracking for the whole call. +- **Transport:** AHardwareBuffer per eye (NV12). A 2×1280×720 pair at 30 Hz is ≈ 83 MB/s through a + CPU ring, so this is where the optional `AHARDWAREBUFFER` transport (spec §5.3) becomes the + default. Rectification runs in the runtime (R2's `u_stereo_rectify`, fed with the CNSDK + rotations) or in the consumer from the calibration. +- **Contract gaps this exposes (A1):** the R1 plug-in frame is ONE side-by-side image in CPU + memory. The tablet produces two per-lens hardware buffers that need mirroring. A1 either packs + them in the plug-in (a GPU blit into one SBS AHB, un-mirroring on the way), or appends a + per-eye layout to `xrt_plugin_stereo_camera_frame` (ADR-020 append-only: a layout field + per-eye + AHB handles). The second is preferred. The calibration slot stays as is: CNSDK's per-camera + rotations become `R`, and the baseline comes from the display config. +- Consent: CAMERA permission of the peer uid + the runtime app's consent, and + `android_package_is_visible` for the foreground rule (spec §7), are the platform-neutral code R3 + adds. ### C.2 Browser -The browser's Android arm is in the patch series (runtime fd brokered by the browser-process Java -via `DXR_IPC_FD`, patches 0083+). The camera lands on top of it the same way as Windows: a -DisplayXR device factory composed in front of `VideoCaptureDeviceFactoryAndroid`, fed over the -existing runtime connection, NV12 from the shared-memory ring. Chromium's own Android camera -permission request covers the uid check the runtime makes. +The browser's Android arm is in the patch series (the runtime fd is brokered by the +browser-process Java via `DXR_IPC_FD`, patches 0083+). The camera lands on top of it the same way +as Windows: a DisplayXR device factory composed in front of `VideoCaptureDeviceFactoryAndroid`, fed +over the existing runtime connection (AHB transport). Chromium's own Android camera permission +request covers the uid check the runtime makes. It fixes browser-pvt#175 for the stereo pair +without the browser ever opening camera 5. ## D. Leia SR plug-in (Windows) — sourcing notes @@ -142,17 +193,43 @@ The implementation belongs in `displayxr-leia-plugin` (`src/drv_leia/`), per ADR | Phase | Repo | Scope | Exit criterion | |---|---|---|---| | **R0** | runtime | This design: ADR-043, spec, this page | Maintainer sign-off on the open questions below | -| **R1** | runtime | Header `XR_DXR_stereo_camera.h` + `index.json` note (together — the catalog lint requires both); `xrt_plugin_iface` slots + `XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA` (appended after `create_dp_d3d11_lift`, so **lands after ADR-042's PR**); platform-neutral camera manager (thread, refcount + linger, 3-slot pinned ring, per-stream wake handles, decimation, NV12/BGRA conversion); IPC + OpenXR entry points; sim_display fake; `displayxr-cli camera list/calib/probe`; selftest check | `camera probe` on the sim fake writes frames on Windows + Linux CI; ring/pinning unit tests (mirror `tests_lift_mailbox`) | +| **R1** ✅ | runtime | **Implemented 2026-09-26 (see "R1 as built" below).** Header `XR_DXR_stereo_camera.h` + `index.json` note (together — the catalog lint requires both); `xrt_plugin_iface` slots + `XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA` (appended after `create_dp_d3d11_lift`, so **lands after ADR-042's PR**); platform-neutral camera manager (thread, refcount + linger, 3-slot pinned ring, per-stream wake handles, decimation, NV12/BGRA conversion); IPC + OpenXR entry points; sim_display fake; `displayxr-cli camera list/calib/probe`; selftest check | `camera probe` on the sim fake writes frames on Windows + Linux CI; ring/pinning unit tests (mirror `tests_lift_mailbox`) | | **R2** | runtime | `u_stereo_rectify` (Bouguet, valid-region crop, LUT remap) with golden tests against OpenCV-generated fixtures; RECTIFIED output | sim fake: row error < 0.5 px, disparity = f·B/Z within 0.5 px | | **R3** | runtime | Privacy: OS consent check, consent store + tray prompt, delegating-client registration, visibility/lock suspension, indicator, kill switches | Manual matrix: allow/deny/revoke, OS switch off, lock screen, background app — each blocks frames | | **L1** | leia-plugin (Windows) | Slots over the SR raw-camera channel per §D; calibration by active serial; keep-alive; repin runtime (feature-macro repin, `downstream-pins.json` `features` track) | On a panel box, incl. the multi-folder box: `camera probe --rectified` rows aligned; the vendor call app running at the same time still gets every frame; tracking/weave unaffected (frame-time + tracking-state logs) | | **B1** | browser (Windows) | §B device factory + duplicate hiding + hint + delegating registration in the installer; web SDK: prefer the hint in `camera:'auto'`, fill `hello` from it | RFC 0002 P1 call between two panel laptops sends rectified SBS from the tracker camera **while both are tracking** | -| **A1** | leia-plugin (Android) + runtime | Only if CNSDK shares frames (§C.1): Android slots, memfd/AHB transport, Android consent + visibility | `camera probe` on the tablet via the runtime app | +| **A1** | leia-plugin (Android) + runtime | §C.1: the runtime owns the front pair and runs CNSDK in-app tracking from the same capture (worker thread, latest-frame slot); per-eye AHB transport + per-eye frame layout; Android consent + visibility | `camera probe` on the tablet via the runtime app **while the weave tracks from the same capture**; pair at 30.00 fps (not 24) | | **B2** | browser (Android) | §C.2 | Tablet ↔ laptop 3D call, both directions stereo | R1–R3 need no hardware; L1 and B1 can proceed in parallel once R1's header exists (B1 develops against the sim fake). +### R1 as built + +- **Header + catalog:** `openxr/XR_DXR_stereo_camera.h`, type values `1004999300–310` (relocated + from `290–300` on the rebase onto post-lift `main`: `290` had been taken by `XR_DXR_weave` v14; + lift holds `270–289`); `index.json` note in the `capture` group. +- **Plug-in slots:** six `stereo_camera_*` slots + `XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA`, appended + after lift's `create_dp_d3d11_lift` (ADR-042, merged in v2.22.0) and ADR-045's + `get_platform_state` — the last members of the iface. `tests_stereo_camera` pins the order. +- **Service:** `ipc/server/ipc_server_stereo_camera.c`. It is platform-neutral (Windows D3D11 + service, macOS/Linux service, Android service). One thread per open camera, open on the first + start, 2 s linger, per-stream 3-slot rings in shared memory, conversion + decimation, and a + per-stream wake (Windows: an auto-reset event, handed out SYNCHRONIZE-only; POSIX: a pipe). + Windows hands the section out read-only (`FILE_MAP_READ` duplicate). +- **IPC:** 12 `stereo_camera_*` calls in `proto.json`. Streams are owned by the connection. +- **OpenXR:** nine entry points. An in-process instance enumerates zero. +- **sim_display fake:** `SIM_DISPLAY_FAKE_STEREO_CAMERA=1` (spec §10). +- **CLI:** `displayxr-cli camera list | calib | probe`; `selftest` gains `stereo_camera_caps`. +- **Privacy hooks (deny by default):** `DXR_STEREO_CAMERA_DEV_ALLOW=1` in the service environment + is the only way to start a stream or read calibration in R1. `DXR_STEREO_CAMERA=0` is the kill + switch. RAW is refused to `PRESENT_OWNER` (the browser). +- **Deferred beyond R1:** state-change events (structs defined, not delivered — they need a + server→client instance-event channel), the GPU transports (designed in the manager's file + comment: D3D11 texture ring + fence as the `XR_DXR_weave` output export; AHB / dma-buf ring + + sync_file as the #1699 weave path), a read-only POSIX section, and a distorted/misaligned fake + (all with R2/R3/A1). + ## F. Risks - **Undocumented vendor channel.** The raw-camera shared memory is a reader-defined contract with @@ -170,10 +247,38 @@ against the sim fake). hunk, and the device + factory are new files, to keep rebase conflicts small. - **Latency/sync.** Arrival timestamps hide tracker-internal latency; A/V sync relies on WebRTC's own jitter handling. Fine for calls, not for measurement. -- **ABI merge order.** The iface slots append after ADR-042's `create_dp_d3d11_lift`; merging out - of order would put two branches on the same offset. - -## G. Open questions for the maintainer +- **ABI merge order.** Lift (ADR-042) merged first (v2.22.0), then ADR-045's `get_platform_state`; + the camera slots were rebased to append after both. R1 originally held lift's offset with a + same-size placeholder so either order would have been ABI-safe; the placeholder is gone. +- **Android: the pair and the tracker cannot share Camera2.** Opening the front logical camera + evicts the vendor tracker, which then never reacquires (§C.0). The mitigation is architectural: + the runtime owns both, and tracking runs in-process from the same capture (§C.1). It adds + CNSDK's in-app face detection (~119 ms per hand-off) to the runtime's CPU budget on a worker. + **Vendor asks (CNSDK / head-tracking service):** (1) the head-tracking service must reacquire its + camera when another client releases it; today it stays wedged ("Camera has been invalidated") + until reboot. (2) Can the tracker run from the pair's own frames (the 1280×720 eye streams, or a + downscaled copy), so that no third stream is needed? (3) Expose the pair's calibration through a + documented CNSDK call, rather than the display-config service plus an absent `/sdcard/rect.js`. + +## G. Open questions & maintainer decisions + +**Decided for R1 (maintainer defaults, 2026-09-26):** + +- **Q1 → stereo-only for now.** Names and struct shapes stay open for other camera kinds + (`XrStereoCameraPropertiesDXR::viewCount`, always 2 in v1, and next chains). +- **Q2 → plug-in iface slots** behind `XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA` (ADR-020 append-only). + A separate camera-provider plug-in type waits for a camera vendor that is not a display vendor. +- **Q3 → service clients only.** In-process instances enumerate zero; there is no side IPC + connection. +- **Q9 → raw frames are not exposed to pages in R1.** Pages get RECTIFIED, or a RAW frame that + says so (`frame.output`) until R2's rectifier lands. The service refuses a RAW stream to + `PRESENT_OWNER` clients (the browser). +- **Q6 is answered by the field facts (§C.0):** CNSDK has no frames to share. The pair is plain + Camera2 and the vendor's camera wrapper is a separate library, so on Android the runtime owns + the pair and runs tracking from it (§C.1). The remaining CNSDK asks are in §F. +- Q4, Q5, Q7 and Q8 remain open (consent UX ownership, SR SDK asks, browser topology, hint shape). + +**Original questions:** 1. **Name and scope.** `XR_DXR_stereo_camera` (stereo only, SBS output) — or a broader `XR_DXR_camera_source` that also carries mono tracker cameras? The design is stereo-only; a diff --git a/docs/specs/extensions/XR_DXR_stereo_camera.md b/docs/specs/extensions/XR_DXR_stereo_camera.md index a9192ddd77..8ee1034f5a 100644 --- a/docs/specs/extensions/XR_DXR_stereo_camera.md +++ b/docs/specs/extensions/XR_DXR_stereo_camera.md @@ -3,10 +3,11 @@ | Field | Value | |---|---| | **Extension Name** | `XR_DXR_stereo_camera` | -| **Spec Version** | 1 (design) | +| **Spec Version** | 1 | | **Extension Type** | Instance extension, service path only (an in-process instance enumerates zero cameras) | -| **Header** | `src/external/openxr_includes/openxr/XR_DXR_stereo_camera.h` — **not written yet.** It lands with the first implementation PR, together with its `docs/specs/extensions/index.json` note (the catalog lint joins notes to headers, so neither can land alone) | -| **Status** | **Design** — not implemented. Provisional type-value block proposed: `1004999290–299` (after `XR_DXR_lift`'s `1004999270–280`), pending Khronos registry | +| **Header** | [`src/external/openxr_includes/openxr/XR_DXR_stereo_camera.h`](../../../src/external/openxr_includes/openxr/XR_DXR_stereo_camera.h) (+ its `index.json` catalog note) | +| **Status** | **R1 implemented** (runtime, hardware-free): header, plug-in slots, service camera manager, IPC, OpenXR entry points, sim_display fake, `displayxr-cli camera`, selftest check. **Not yet:** rectification (R2), consent / indicator / foreground rule (R3 — deny-by-default hooks in place), GPU transports, state-change events, the Leia provider (L1), the browser (B1). Provisional type values `1004999300–310` (relocated from `290–300` when `XR_DXR_weave` v14 took `290`; `XR_DXR_lift` holds `270–289`), pending Khronos registry | +| **R1 decisions** | stereo-only (names/structs kept open: `viewCount`, next chains) · camera provider = `xrt_plugin_iface` slots · service clients only (in-process enumerates zero) · raw frames never reach web pages (`RAW` refused to `PRESENT_OWNER` clients) — see [roadmap §G](../../roadmap/stereo-camera-source.md#g-open-questions--maintainer-decisions) | | **Decision record** | [ADR-043](../../adr/ADR-043-stereo-camera-source.md) | | **Plug-in contract** | appended `xrt_plugin_iface` camera slots, `XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA` (§9) | | **Browser / Android integration + plan** | [roadmap/stereo-camera-source.md](../../roadmap/stereo-camera-source.md) | @@ -61,6 +62,12 @@ if (xrAcquireStereoCameraFrameDXR(stream, &f) == XR_SUCCESS) { /* f.slot is pinn | Service with a camera source | one entry per camera, with its `state` (§3) | | `DXR_STEREO_CAMERA=0` in the **service's** environment, or the user kill switch (§7.4) | count 0 — indistinguishable from "no camera", on purpose | +**R1 dev gate.** Until the consent path (§7.1, R3) exists, the service refuses every +`xrStartStereoCameraStreamDXR` and `xrGetStereoCameraCalibrationDXR` with +`XR_ERROR_PERMISSION_INSUFFICIENT` **unless `DXR_STEREO_CAMERA_DEV_ALLOW=1` is set in the +service's environment** — deny by default. Enumeration works without it (it reveals no +calibration and no serial). + Instance-level, not session-level: a capture component (a browser's video-capture service) has no compositor session and needs none. Calls travel on the instance's IPC connection; streams belong to that connection and die with it. @@ -94,7 +101,8 @@ typedef struct XrStereoCameraPropertiesDXR { XrStereoCameraFlagsDXR flags; XrStereoCameraStateDXR state; XrExtent2Di eyeExtent; // native per-eye size (e.g. 640x480) - float maxFrameRate; // what the source can deliver, Hz + float maxFrameRate; // Hz the source delivers: MEASURED by the service after its + // first open, else the vendor's value; 0 = not known yet float baselineMm; // |T| of the pair; 0 if uncalibrated float horizontalFovDeg; // per eye, rectified; 0 if uncalibrated XrStereoCameraFormatFlagsDXR supportedFormats; // GRAY8 / NV12 / BGRA8 bits the service will produce @@ -289,6 +297,15 @@ When `SHARED_WITH_EYE_TRACKING` is set: ## 7. Privacy +> **R1 status.** Only the hook points exist, all deny-by-default, in +> `src/xrt/ipc/server/ipc_server_stereo_camera.c`: `authorise_locked()` (§7.1 — R1 allows only +> with `DXR_STEREO_CAMERA_DEV_ALLOW=1` in the service environment), `client_visible_locked()` +> (§7.2 — always true in R1), `output_allowed()` (RAW refused to `PRESENT_OWNER`, i.e. the +> browser, per the R1 decision), the `DXR_STEREO_CAMERA=0` kill switch (§7.4), and one WARN per +> stream start/stop naming the peer executable in place of the indicator (§7.3). `persistentId` +> (§7.5) is an unkeyed 128-bit hash of (device identity, peer executable) in R1; R3 keys it with a +> persisted per-user secret. + The frames bypass the OS camera stack (the tracker, not the consumer, opened the device), so OS camera permissions and in-use indicators do not see this consumer. The runtime therefore enforces the equivalent itself. All checks are made on the **OS-derived peer** of the connection @@ -350,7 +367,17 @@ passed §7.1 for that camera. A browser must coarsen what it gives pages (roadma ## 9. Plug-in contract (ADR-020 append-only) Appended to `struct xrt_plugin_iface` (after `create_dp_d3d11_lift`, ADR-042), gated by -`struct_size`, announced by `#define XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA 1`, no ABI bump. On the +`struct_size`, announced by `#define XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA 1`, no ABI bump. +**As implemented:** the six slots are the last members of the iface, after `create_dp_d3d11_lift` +(ADR-042) and `get_platform_state` (ADR-045), both of which were on `main` first; +`tests_stereo_camera` pins the order. The authoritative +header is `src/xrt/include/xrt/xrt_plugin.h`; the plug-in-facing rules are in +[`docs/reference/xrt_plugin_iface.md`](../../reference/xrt_plugin_iface.md#a-stereo-camera-the-stereo_camera_-slots-adr-043). +Two refinements over the sketch below: `wait_frame` returns an `enum xrt_plugin_stereo_camera_wait` +(`OK` / `TIMEOUT` / `SUSPENDED` / `ERROR`) rather than an `xrt_result_t`, so no new +`xrt_result_t` values were needed; and the extrinsics are **OpenCV's `x_R = R·x_L + T`** (T in +mm, `T = (−B, 0, 0)` for a right camera at +x) — the service converts them to the client's +`rightFromLeft` pose (the right camera in the left camera's frame, metres). On the **plug-in iface, not a display-processor vtable**: a camera is a sensor, not a weaver; it must be graphics-API-neutral (Windows D3D11 service, Android Vulkan runtime) and must not share a lifetime with a DP that is recreated on presenter changes. @@ -363,7 +390,7 @@ struct xrt_plugin_stereo_camera_info { char platform_device_hint[256]; // OS id of the physical device read, or "" uint32_t flags; // XRT_PLUGIN_STEREO_CAMERA_* (shared-with-tracking, user-facing, calibrated, rectified, mono) uint32_t eye_width, eye_height; - float max_frame_rate; + float max_frame_rate; // only if KNOWN (vendor value / measured earlier); 0 = unknown, never a guess uint32_t native_format; // GRAY8 / NV12 / BGRA8 — the plug-in decodes its transport (e.g. JPEG) itself }; @@ -416,12 +443,26 @@ owns threads, fan-out, rectification, format conversion, transport, consent and ## 10. sim_display fake +> **R1 as built** (`drivers/sim_display/sim_display_stereo_camera{,_pattern}.c`): the pair is +> **undistorted and aligned** and the camera reports `NATIVELY_RECTIFIED` (+ `CALIBRATED`, +> `SHARED_WITH_EYE_TRACKING`, `USER_FACING`, and `MONOCHROME` for gray8), so the RECTIFIED output +> path runs end to end today. The scene is a random-dot stereogram: background at 2.0 m and a bar +> at 0.6 m, which at the defaults (640 px per eye, 68° HFOV → fx 474.4 px, 50 mm) are **12 px and +> 40 px** of disparity, plus an 8-digit frame counter in both halves. Knobs, read by the SERVICE: +> `SIM_DISPLAY_FAKE_STEREO_CAMERA=1`, `_SIZE=WxH` (per eye), `_FPS=N`, `_FORMAT=gray8|nv12|bgra`, +> `_SUSPEND_PERIOD_MS=N`, `_ADVERTISED_FPS=N` (what enumerate claims, default = `_FPS`; `0` = unknown — +> exercises the service's measured-rate path), `_BASELINE_MM=N` (default 50; `120` reproduces the Leia SR +> tracking camera: the bar at 0.6 m is ~95 px). The distorted + misaligned variant below is R2's, with the rectifier. + `SIM_DISPLAY_FAKE_STEREO_CAMERA=1` makes sim_display advertise one camera (`SHARED_WITH_EYE_TRACKING | USER_FACING | CALIBRATED | MONOCHROME`, 640×480 per eye, 30 Hz, baseline 50 mm) producing a **synthetic SBS pattern**: a textured plane and a bar at known depths, rendered with a known synthetic calibration that includes radial distortion and a small vertical misalignment + roll between the eyes, plus a frame counter burned into each half. So: +- the descriptor's rate is testable: with `_ADVERTISED_FPS=0` the camera lists "rate unknown" until a + stream has run ~60 frames, then reports the measured rate (field finding: the Leia SR tracking + camera was hard-coded to advertise 30 Hz and actually ran 58–64 Hz); - rectification is testable exactly — after `RECTIFIED`, feature rows must align (|Δy| < 0.5 px) and the bar's disparity must equal `fx·B/Z`; - `SIM_DISPLAY_FAKE_STEREO_CAMERA_SUSPEND_PERIOD_MS=N` square-waves SUSPENDED/AVAILABLE, like the @@ -435,8 +476,20 @@ The whole path (enumerate → consent → start → acquire → rectify) then ru ``` displayxr-cli camera list [--json] # properties + state of every camera displayxr-cli camera calib [--raw|--rectified] [--json] -displayxr-cli camera probe [--raw] [--format gray8|nv12|bgra8] [--seconds S] [--out DIR] - # writes cam_.png, prints rate, drops, latency +displayxr-cli camera probe [] [--raw] [--format gray8|nv12|bgra8] [--fps F] + [--frames N] [--seconds S] [--out DIR] + # rate, index gaps, SBS layout, block + # disparities, service stats; --out writes cam_.png +``` + +As built in R1, e.g. against the sim fake on macOS: + +``` +stream 1 on camera 1: 1280x480 NV12 SBS (RECTIFIED), cap 30.0 Hz, ring 3 x 921600 B (section 2764800 B) +received 150 frames (index 1..150, 0 source frames not delivered to this stream), 150 wakes, 0 not-ready +measured delivery rate: 29.95 Hz over 4.98 s +block disparity (left x - right x): 12 px (38/54 blocks), 40 px (16/54 blocks), range 12..40 px +service stats: source 30.49 Hz, delivered 30.49 Hz, published 155, skipped 4, acquired 150, mean latency 12.417 ms ``` Runs as a DIAG IPC client, non-elevated. `probe` goes through the full consent path (so its @@ -457,4 +510,4 @@ camera state change and per stream start/stop with the peer executable; INFO sta | Version | Change | |---|---| -| 1 (design) | Enumerate + state events, calibration (raw / rectified), streams with start/stop, latest-wins acquire over a pinned 3-slot shared-memory ring with per-stream wake handles, optional GPU transports, runtime-enforced consent / foreground / indicator, plug-in iface slots, sim_display fake, CLI. | +| 1 | R1 implements: enumerate, calibration (RAW; RECTIFIED for natively rectified sources), streams with start/stop, latest-wins acquire over a pinned 3-slot shared-memory ring with per-stream wake handles, stream stats, plug-in slots, sim_display fake, CLI, selftest. Deferred: state-change events (structs defined, not delivered), GPU transports, rectifier (R2), consent/indicator (R3). Design scope: Enumerate + state events, calibration (raw / rectified), streams with start/stop, latest-wins acquire over a pinned 3-slot shared-memory ring with per-stream wake handles, optional GPU transports, runtime-enforced consent / foreground / indicator, plug-in iface slots, sim_display fake, CLI. | diff --git a/docs/specs/extensions/index.json b/docs/specs/extensions/index.json index c10cb4d566..dc343efa01 100644 --- a/docs/specs/extensions/index.json +++ b/docs/specs/extensions/index.json @@ -131,6 +131,11 @@ "group": "capture", "title": "Atlas Capture", "summary": "Vendor-neutral PNG snapshot of the multi-view atlas at a caller-selected compositor stage, for any app class" + }, + "XR_DXR_stereo_camera": { + "group": "capture", + "title": "Stereo Camera Source", + "summary": "The display's own stereo camera (usually the eye tracker's) as a runtime-owned, privacy-gated source: the vendor plug-in supplies frames without taking the device from the tracker, and the service fans newest-frame side-by-side pairs out to clients over shared memory" } } } diff --git a/src/external/openxr_includes/openxr/README.md b/src/external/openxr_includes/openxr/README.md index 67450000e2..0d869e65b5 100644 --- a/src/external/openxr_includes/openxr/README.md +++ b/src/external/openxr_includes/openxr/README.md @@ -49,7 +49,9 @@ Rules: | 1004999270–279 | `XR_DXR_lift` | 270 = `XR_TYPE_LIFT_PROPERTIES_DXR`, 271 = `XR_TYPE_LIFT_STREAM_CREATE_INFO_DXR`, 272 = `XR_TYPE_LIFT_FRAME_SUBMIT_INFO_DXR`, 273 = `XR_TYPE_LIFT_OPTIONS_DXR`, 274 = `XR_TYPE_LIFT_RESULT_DXR`, 275 = `XR_TYPE_WEAVE_SUBMIT_LIFT_RECTS_DXR`, 276 = `XR_TYPE_WEAVE_RECT_LIFT_DXR`, 277 = `XR_TYPE_LIFT_BLOB_DXR` (spec v1, ADR-042). 278 = `XR_OBJECT_TYPE_LIFT_STREAM_DXR` — an `XrObjectType`, not an `XrStructureType`. 279 = `XR_LIFT_NOT_READY_DXR` — a **success**-class `XrResult` (positive) | | 1004999280–289 | `XR_DXR_lift` (continued) | 280 = `XR_TYPE_LIFT_STREAM_STATS_DXR` (spec v1). A second decade because 270–279 is full | | 1004999290–299 | `XR_DXR_weave` (v14+ additions) | 290 = `XR_TYPE_WEAVE_SUBMIT_OVERLAY_UNCHANGED_DXR` (the v4 overlay atlas is unchanged since the previous submit, so a DP may reuse its lens prefilter of the 2D layer, spec v14, ADR-027 Amendment). The third weave decade (240–249 is full) | -| 1004999300+ | **next free** | | +| 1004999300–309 | `XR_DXR_stereo_camera` | 300 = `XR_TYPE_STEREO_CAMERA_PROPERTIES_DXR`, 301 = `XR_TYPE_STEREO_CAMERA_CALIBRATION_DXR`, 302 = `XR_TYPE_STEREO_CAMERA_STREAM_CREATE_INFO_DXR`, 303 = `XR_TYPE_STEREO_CAMERA_FRAME_DXR`, 304 = `XR_TYPE_STEREO_CAMERA_STREAM_INFO_DXR`, 305 = `XR_TYPE_STEREO_CAMERA_STREAM_TRANSPORT_DXR`, 306 = `XR_TYPE_STEREO_CAMERA_STREAM_STATS_DXR`, 307 = `XR_TYPE_EVENT_DATA_STEREO_CAMERA_STATE_CHANGED_DXR`, 308 = `XR_TYPE_EVENT_DATA_STEREO_CAMERAS_CHANGED_DXR` (spec v1, ADR-043). 309 = `XR_OBJECT_TYPE_STEREO_CAMERA_STREAM_DXR` — an `XrObjectType`, not an `XrStructureType`. **Relocated from 290–299** on the rebase onto post-lift `main`: 290 had meanwhile been taken by `XR_TYPE_WEAVE_SUBMIT_OVERLAY_UNCHANGED_DXR` (weave spec v14). Safe to move: spec v1, draft PR, no shipped app chains it | +| 1004999310–319 | `XR_DXR_stereo_camera` (continued) | 310 = `XR_STEREO_CAMERA_FRAME_NOT_READY_DXR` — a **success**-class `XrResult` (positive). A second decade because 300–309 is full | +| 1004999320+ | **next free** | | `XR_DXR_android_surface_binding` was implemented in #1037 and now has its own header and decade — see the row above for why its create-info type value sits diff --git a/src/external/openxr_includes/openxr/XR_DXR_stereo_camera.h b/src/external/openxr_includes/openxr/XR_DXR_stereo_camera.h new file mode 100644 index 0000000000..740871487c --- /dev/null +++ b/src/external/openxr_includes/openxr/XR_DXR_stereo_camera.h @@ -0,0 +1,372 @@ +// Copyright 2026, DisplayXR +// SPDX-License-Identifier: Apache-2.0 +// +// PROVISIONAL — DXR is DisplayXR's Khronos-registered OpenXR author ID, but +// the XR_DXR_* extensions in this header are NOT yet registered in the +// Khronos OpenXR registry: extension numbers and XrStructureType values sit +// in a provisional experimental block (1004999xxx) pending official +// assignment. Extension names are expected to be stable; numeric values are +// not. +// See GOVERNANCE.md. +// +/*! + * @file + * @brief Header for XR_DXR_stereo_camera extension + * @author David Fattal + * @ingroup external_openxr + * + * A display's STEREO CAMERA (usually the one its eye tracker looks through) as + * a runtime-provided source (ADR-043). The vendor display plug-in produces + * frames from its own stack without taking the device from the tracker; the + * runtime service owns the camera, opens it once and fans every frame out to + * each authorised stream; clients consume the newest frame over shared memory: + * + * xrEnumerateStereoCamerasDXR(instance, systemId, 0, &n, NULL); // + properties + * xrCreateStereoCameraStreamDXR(instance, &createInfo, &stream); + * xrStartStereoCameraStreamDXR(stream); // authorisation point + * xrGetStereoCameraStreamInfoDXR(stream, &info); // chain XrStereoCameraStreamTransportDXR + * // map info's section once; per frame (wake on the per-stream wake handle): + * if (xrAcquireStereoCameraFrameDXR(stream, &frame) == XR_SUCCESS) { read slot frame.slot } + * + * The frame is always ONE side-by-side image, 2*eyeWidth x eyeHeight, left eye + * in the left half, never mirrored, never woven (showing it in 3D is the + * weave path's job, ADR-007). + * + * Instance-level (a capture component has no session) and SERVICE-ONLY: an + * in-process instance enumerates zero cameras. The service is the single owner + * of each camera; an in-process runtime opening it too would recreate the + * exclusive-device fight this extension exists to end. + * + * Scope: stereo pairs. Names and struct shapes are kept open for other camera + * kinds (XrStereoCameraPropertiesDXR::viewCount, next chains), but v1 sources + * are two-view side-by-side only. + * + * Privacy: frames bypass the OS camera stack, so the runtime gates them itself + * (consent, foreground rule, in-use indicator, kill switch — spec §7). + * xrStartStereoCameraStreamDXR and xrGetStereoCameraCalibrationDXR return + * XR_ERROR_PERMISSION_INSUFFICIENT when refused. + * + * Spec: docs/specs/extensions/XR_DXR_stereo_camera.md. + * + * Version history: 1 = initial (enumerate + state, calibration raw/rectified, + * streams with start/stop, latest-wins acquire over a pinned 3-slot shared- + * memory ring with per-stream wake handles, stream stats). + */ +#ifndef XR_DXR_STEREO_CAMERA_H +#define XR_DXR_STEREO_CAMERA_H 1 + +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +#define XR_DXR_stereo_camera 1 +#define XR_DXR_stereo_camera_SPEC_VERSION 1 +#define XR_DXR_STEREO_CAMERA_EXTENSION_NAME "XR_DXR_stereo_camera" + +// Reserved 1004999300..319 (relocated from 290..309 on the post-lift rebase: 290 was taken by XR_DXR_weave v14). Allocation registry: README.md in this directory. +#define XR_TYPE_STEREO_CAMERA_PROPERTIES_DXR ((XrStructureType)1004999300) +#define XR_TYPE_STEREO_CAMERA_CALIBRATION_DXR ((XrStructureType)1004999301) +#define XR_TYPE_STEREO_CAMERA_STREAM_CREATE_INFO_DXR ((XrStructureType)1004999302) +#define XR_TYPE_STEREO_CAMERA_FRAME_DXR ((XrStructureType)1004999303) +#define XR_TYPE_STEREO_CAMERA_STREAM_INFO_DXR ((XrStructureType)1004999304) +#define XR_TYPE_STEREO_CAMERA_STREAM_TRANSPORT_DXR ((XrStructureType)1004999305) +#define XR_TYPE_STEREO_CAMERA_STREAM_STATS_DXR ((XrStructureType)1004999306) +#define XR_TYPE_EVENT_DATA_STEREO_CAMERA_STATE_CHANGED_DXR ((XrStructureType)1004999307) +#define XR_TYPE_EVENT_DATA_STEREO_CAMERAS_CHANGED_DXR ((XrStructureType)1004999308) +//! XrObjectType of an XrStereoCameraStreamDXR (debug-utils naming). +#define XR_OBJECT_TYPE_STEREO_CAMERA_STREAM_DXR ((XrObjectType)1004999309) +//! SUCCESS-class result: no frame newer than the last one this stream acquired. +//! Not an error — keep showing the previous frame. +#define XR_STEREO_CAMERA_FRAME_NOT_READY_DXR ((XrResult)1004999310) + +//! Size of XrStereoCameraPropertiesDXR::persistentId, NUL included. +#define XR_STEREO_CAMERA_PERSISTENT_ID_MAX_SIZE_DXR 64 +//! Size of XrStereoCameraPropertiesDXR::displayName, NUL included. +#define XR_STEREO_CAMERA_DISPLAY_NAME_MAX_SIZE_DXR 128 +//! Size of XrStereoCameraPropertiesDXR::platformDeviceHint, NUL included. +#define XR_STEREO_CAMERA_PLATFORM_HINT_MAX_SIZE_DXR 256 +//! Slots in a stream's shared-memory ring (latest + pinned + one to write). +#define XR_STEREO_CAMERA_RING_SLOTS_DXR 3 + +XR_DEFINE_HANDLE(XrStereoCameraStreamDXR) + +typedef XrFlags64 XrStereoCameraFlagsDXR; +//! The camera is the eye tracker's; frames exist only while tracking runs. +static const XrStereoCameraFlagsDXR XR_STEREO_CAMERA_SHARED_WITH_EYE_TRACKING_BIT_DXR = 0x00000001; +//! The camera faces the user (a 3D display's front camera). +static const XrStereoCameraFlagsDXR XR_STEREO_CAMERA_USER_FACING_BIT_DXR = 0x00000002; +//! Calibration is available for the ACTIVE device (RECTIFIED supported). +static const XrStereoCameraFlagsDXR XR_STEREO_CAMERA_CALIBRATED_BIT_DXR = 0x00000004; +//! The source already delivers a rectified pair. +static const XrStereoCameraFlagsDXR XR_STEREO_CAMERA_NATIVELY_RECTIFIED_BIT_DXR = 0x00000008; +//! Luma only (an IR / grey tracking sensor). +static const XrStereoCameraFlagsDXR XR_STEREO_CAMERA_MONOCHROME_BIT_DXR = 0x00000010; + +typedef enum XrStereoCameraStateDXR { + //! Frames flowing, or will flow on start. + XR_STEREO_CAMERA_STATE_AVAILABLE_DXR = 1, + //! Source up, no frame yet (tracker warming up). + XR_STEREO_CAMERA_STATE_WAITING_DXR = 2, + //! Temporarily no frames: tracker stopped, privacy lock, session locked. + XR_STEREO_CAMERA_STATE_SUSPENDED_DXR = 3, + //! Failed / removed. + XR_STEREO_CAMERA_STATE_UNAVAILABLE_DXR = 4, + XR_STEREO_CAMERA_STATE_MAX_ENUM_DXR = 0x7FFFFFFF +} XrStereoCameraStateDXR; + +typedef enum XrStereoCameraOutputDXR { + //! Parallel pair, zero disparity at infinity, no convergence shear. Needs + //! CALIBRATED or NATIVELY_RECTIFIED. + XR_STEREO_CAMERA_OUTPUT_RECTIFIED_DXR = 1, + //! The source's pixels as delivered (native clients only; never web pages). + XR_STEREO_CAMERA_OUTPUT_RAW_DXR = 2, + XR_STEREO_CAMERA_OUTPUT_MAX_ENUM_DXR = 0x7FFFFFFF +} XrStereoCameraOutputDXR; + +typedef enum XrStereoCameraFormatDXR { + XR_STEREO_CAMERA_FORMAT_GRAY8_DXR = 1, + //! A MONOCHROME source in NV12 is luma + neutral (128) chroma. + XR_STEREO_CAMERA_FORMAT_NV12_DXR = 2, + XR_STEREO_CAMERA_FORMAT_BGRA8_DXR = 3, + XR_STEREO_CAMERA_FORMAT_MAX_ENUM_DXR = 0x7FFFFFFF +} XrStereoCameraFormatDXR; + +typedef enum XrStereoCameraTransportDXR { + //! Required; every platform. + XR_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY_DXR = 1, + //! Optional (Windows); not implemented by spec v1 runtimes. + XR_STEREO_CAMERA_TRANSPORT_D3D11_TEXTURE_DXR = 2, + //! Optional (Android); not implemented by spec v1 runtimes. + XR_STEREO_CAMERA_TRANSPORT_AHARDWAREBUFFER_DXR = 3, + XR_STEREO_CAMERA_TRANSPORT_MAX_ENUM_DXR = 0x7FFFFFFF +} XrStereoCameraTransportDXR; + +typedef enum XrStereoCameraDistortionModelDXR { + XR_STEREO_CAMERA_DISTORTION_NONE_DXR = 0, + //! OpenCV k1 k2 p1 p2 k3. + XR_STEREO_CAMERA_DISTORTION_RADTAN5_DXR = 1, + //! + k4 k5 k6 (rational). + XR_STEREO_CAMERA_DISTORTION_RADTAN8_DXR = 2, + //! Fisheye (Kannala-Brandt k1..k4). + XR_STEREO_CAMERA_DISTORTION_KB4_DXR = 3, + XR_STEREO_CAMERA_DISTORTION_MAX_ENUM_DXR = 0x7FFFFFFF +} XrStereoCameraDistortionModelDXR; + +//! One bit per XrStereoCameraFormatDXR value: bit (1 << value). +typedef XrFlags64 XrStereoCameraFormatFlagsDXR; +static const XrStereoCameraFormatFlagsDXR XR_STEREO_CAMERA_FORMAT_GRAY8_BIT_DXR = 0x00000002; +static const XrStereoCameraFormatFlagsDXR XR_STEREO_CAMERA_FORMAT_NV12_BIT_DXR = 0x00000004; +static const XrStereoCameraFormatFlagsDXR XR_STEREO_CAMERA_FORMAT_BGRA8_BIT_DXR = 0x00000008; + +//! One bit per XrStereoCameraTransportDXR value: bit (1 << value). +typedef XrFlags64 XrStereoCameraTransportFlagsDXR; +static const XrStereoCameraTransportFlagsDXR XR_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY_BIT_DXR = 0x00000002; +static const XrStereoCameraTransportFlagsDXR XR_STEREO_CAMERA_TRANSPORT_D3D11_TEXTURE_BIT_DXR = 0x00000004; +static const XrStereoCameraTransportFlagsDXR XR_STEREO_CAMERA_TRANSPORT_AHARDWAREBUFFER_BIT_DXR = 0x00000008; + +/*! + * One camera, as enumerated. @c cameraId is valid for the service's lifetime + * and never reused for another camera within it. + */ +typedef struct XrStereoCameraPropertiesDXR { + XrStructureType type; + void* XR_MAY_ALIAS next; + uint64_t cameraId; + //! Stable per (device, consumer executable); never the raw serial (spec §7.5). + char persistentId[XR_STEREO_CAMERA_PERSISTENT_ID_MAX_SIZE_DXR]; + //! Plug-in supplied, human-readable ("Built-in 3D camera"). + char displayName[XR_STEREO_CAMERA_DISPLAY_NAME_MAX_SIZE_DXR]; + //! The OS's id of the physical device this source reads, "" if none — so a + //! capture stack can hide the raw (busy) duplicate. + char platformDeviceHint[XR_STEREO_CAMERA_PLATFORM_HINT_MAX_SIZE_DXR]; + XrStereoCameraFlagsDXR flags; + XrStereoCameraStateDXR state; + //! Views in the source. Always 2 in spec v1 (reserved for other camera kinds). + uint32_t viewCount; + //! Native per-eye size (e.g. 640x480). + XrExtent2Di eyeExtent; + //! What the source delivers, Hz: measured by the runtime once the source + //! has run, else the vendor's value; 0 = not known yet. + float maxFrameRate; + //! |T| of the pair, millimetres; 0 if uncalibrated. + float baselineMm; + //! Per eye, rectified, degrees; 0 if uncalibrated. + float horizontalFovDeg; + XrStereoCameraFormatFlagsDXR supportedFormats; + XrStereoCameraTransportFlagsDXR supportedTransports; +} XrStereoCameraPropertiesDXR; + +typedef struct XrStereoCameraIntrinsicsDXR { + //! The image these numbers describe (per eye). + XrExtent2Di imageExtent; + float fx, fy, cx, cy; + XrStereoCameraDistortionModelDXR model; + float coefficients[8]; +} XrStereoCameraIntrinsicsDXR; + +typedef struct XrStereoCameraCalibrationDXR { + XrStructureType type; + void* XR_MAY_ALIAS next; + //! IN: which output the numbers should describe. + XrStereoCameraOutputDXR output; + //! RAW: native + distortion. RECTIFIED: post-rectification, model NONE, equal fy/cy. + XrStereoCameraIntrinsicsDXR eye[2]; + //! RAW: the pair's extrinsics. RECTIFIED: pure +x translation of baselineMm/1000. + XrPosef rightFromLeft; + float baselineMm; +} XrStereoCameraCalibrationDXR; + +typedef struct XrStereoCameraStreamCreateInfoDXR { + XrStructureType type; + const void* XR_MAY_ALIAS next; + uint64_t cameraId; + XrStereoCameraOutputDXR output; + XrStereoCameraFormatDXR format; + XrStereoCameraTransportDXR transport; + //! <= 0 = the source's rate; the service decimates, never interpolates. + float maxFrameRate; +} XrStereoCameraStreamCreateInfoDXR; + +/*! + * The shared-memory ring of a started stream (chained on + * XrStereoCameraStreamInfoDXR). Handles are NEW handles duplicated into the + * caller (Windows HANDLE / POSIX fd): the caller closes them. The section is + * read-only for the caller. + */ +typedef struct XrStereoCameraStreamTransportDXR { + XrStructureType type; + void* XR_MAY_ALIAS next; + uint64_t sectionHandle; + uint64_t sectionSize; + uint32_t slotCount; + uint64_t slotStride; + //! Per-stream wake handle, signalled once per published frame. Windows: an + //! auto-reset event (SYNCHRONIZE access). POSIX: the read end of a + //! non-blocking pipe — one byte per frame; drain it before acquiring. + uint64_t wakeHandle; +} XrStereoCameraStreamTransportDXR; + +typedef struct XrStereoCameraStreamInfoDXR { + XrStructureType type; + void* XR_MAY_ALIAS next; + //! Full SBS extent (2*eyeWidth x eyeHeight). + XrExtent2Di extent; + XrStereoCameraFormatDXR format; + XrStereoCameraOutputDXR output; + XrStereoCameraTransportDXR transport; + //! The effective delivery cap, Hz. + float maxFrameRate; +} XrStereoCameraStreamInfoDXR; + +typedef struct XrStereoCameraFrameDXR { + XrStructureType type; + void* XR_MAY_ALIAS next; + //! Source frame counter, monotonic from 1; gaps = frames this stream missed. + uint64_t frameIndex; + //! Ring slot holding it; PINNED (byte-stable) until this stream's next acquire. + uint32_t slot; + XrExtent2Di extent; + XrStereoCameraFormatDXR format; + //! Plane pitches in bytes (NV12: Y, UV; single-plane formats: [0]). + uint32_t rowPitch[2]; + //! Byte offsets of the planes inside the slot. + uint64_t planeOffset[2]; + XrTime captureTime; + //! TRUE = sensor/exposure time; FALSE = arrival time at the plug-in. + XrBool32 captureTimeIsExposure; + //! What this frame is (RAW when rectification is unavailable). + XrStereoCameraOutputDXR output; + //! Bumps when calibration/rectification changed; re-query calibration. + uint32_t calibrationGeneration; +} XrStereoCameraFrameDXR; + +typedef struct XrStereoCameraStreamStatsDXR { + XrStructureType type; + void* XR_MAY_ALIAS next; + //! Rate the source delivered to the service, Hz (exponential average). + float sourceFrameRate; + //! Rate published to THIS stream after decimation, Hz. + float deliveredFrameRate; + uint64_t framesPublished; + //! Frames published but superseded before this stream acquired them. + uint64_t framesSkipped; + uint64_t framesAcquired; + //! Mean plug-in-timestamp -> publish latency, ns. + uint64_t meanLatencyNs; +} XrStereoCameraStreamStatsDXR; + +typedef struct XrEventDataStereoCameraStateChangedDXR { + XrStructureType type; + const void* XR_MAY_ALIAS next; + uint64_t cameraId; + XrStereoCameraStateDXR state; +} XrEventDataStereoCameraStateChangedDXR; + +typedef struct XrEventDataStereoCamerasChangedDXR { + XrStructureType type; + const void* XR_MAY_ALIAS next; +} XrEventDataStereoCamerasChangedDXR; + +typedef XrResult (XRAPI_PTR *PFN_xrEnumerateStereoCamerasDXR)( + XrInstance instance, XrSystemId systemId, uint32_t capacityInput, uint32_t* countOutput, + XrStereoCameraPropertiesDXR* cameras); +typedef XrResult (XRAPI_PTR *PFN_xrGetStereoCameraCalibrationDXR)( + XrInstance instance, uint64_t cameraId, XrStereoCameraCalibrationDXR* calibration); +typedef XrResult (XRAPI_PTR *PFN_xrCreateStereoCameraStreamDXR)( + XrInstance instance, const XrStereoCameraStreamCreateInfoDXR* createInfo, XrStereoCameraStreamDXR* stream); +typedef XrResult (XRAPI_PTR *PFN_xrDestroyStereoCameraStreamDXR)(XrStereoCameraStreamDXR stream); +typedef XrResult (XRAPI_PTR *PFN_xrStartStereoCameraStreamDXR)(XrStereoCameraStreamDXR stream); +typedef XrResult (XRAPI_PTR *PFN_xrStopStereoCameraStreamDXR)(XrStereoCameraStreamDXR stream); +typedef XrResult (XRAPI_PTR *PFN_xrGetStereoCameraStreamInfoDXR)( + XrStereoCameraStreamDXR stream, XrStereoCameraStreamInfoDXR* info); +typedef XrResult (XRAPI_PTR *PFN_xrAcquireStereoCameraFrameDXR)( + XrStereoCameraStreamDXR stream, XrStereoCameraFrameDXR* frame); +typedef XrResult (XRAPI_PTR *PFN_xrGetStereoCameraStreamStatsDXR)( + XrStereoCameraStreamDXR stream, XrStereoCameraStreamStatsDXR* stats); + +#ifndef XR_NO_PROTOTYPES +#ifdef XR_EXTENSION_PROTOTYPES + +//! Two-call idiom. An in-process instance, a plug-in with no camera, and the +//! kill switch all report count 0 — indistinguishable on purpose. +XRAPI_ATTR XrResult XRAPI_CALL xrEnumerateStereoCamerasDXR( + XrInstance instance, XrSystemId systemId, uint32_t capacityInput, uint32_t* countOutput, + XrStereoCameraPropertiesDXR* cameras); + +//! Calibration of the ACTIVE device. Consent-gated (spec §7.1, §7.5). +XRAPI_ATTR XrResult XRAPI_CALL xrGetStereoCameraCalibrationDXR( + XrInstance instance, uint64_t cameraId, XrStereoCameraCalibrationDXR* calibration); + +XRAPI_ATTR XrResult XRAPI_CALL xrCreateStereoCameraStreamDXR( + XrInstance instance, const XrStereoCameraStreamCreateInfoDXR* createInfo, XrStereoCameraStreamDXR* stream); + +XRAPI_ATTR XrResult XRAPI_CALL xrDestroyStereoCameraStreamDXR(XrStereoCameraStreamDXR stream); + +//! The authorisation point, and where the service opens the source. +XRAPI_ATTR XrResult XRAPI_CALL xrStartStereoCameraStreamDXR(XrStereoCameraStreamDXR stream); + +XRAPI_ATTR XrResult XRAPI_CALL xrStopStereoCameraStreamDXR(XrStereoCameraStreamDXR stream); + +//! Valid after start. Chain XrStereoCameraStreamTransportDXR to receive the +//! ring section + wake handle (new handles each call; caller closes). +XRAPI_ATTR XrResult XRAPI_CALL xrGetStereoCameraStreamInfoDXR( + XrStereoCameraStreamDXR stream, XrStereoCameraStreamInfoDXR* info); + +//! Newest frame newer than the last one acquired, or XR_STEREO_CAMERA_FRAME_NOT_READY_DXR. +XRAPI_ATTR XrResult XRAPI_CALL xrAcquireStereoCameraFrameDXR( + XrStereoCameraStreamDXR stream, XrStereoCameraFrameDXR* frame); + +XRAPI_ATTR XrResult XRAPI_CALL xrGetStereoCameraStreamStatsDXR( + XrStereoCameraStreamDXR stream, XrStereoCameraStreamStatsDXR* stats); + +#endif /* XR_EXTENSION_PROTOTYPES */ +#endif /* !XR_NO_PROTOTYPES */ + +#ifdef __cplusplus +} +#endif + +#endif // XR_DXR_STEREO_CAMERA_H diff --git a/src/xrt/auxiliary/util/CMakeLists.txt b/src/xrt/auxiliary/util/CMakeLists.txt index 6fa0f2db18..4c6484c67d 100644 --- a/src/xrt/auxiliary/util/CMakeLists.txt +++ b/src/xrt/auxiliary/util/CMakeLists.txt @@ -33,6 +33,8 @@ add_library( u_builders.h u_capture_dims.c u_capture_dims.h + u_stereo_camera.c + u_stereo_camera.h u_color_encoding.h u_debug.c u_debug.h diff --git a/src/xrt/auxiliary/util/u_stereo_camera.c b/src/xrt/auxiliary/util/u_stereo_camera.c new file mode 100644 index 0000000000..d11af289f3 --- /dev/null +++ b/src/xrt/auxiliary/util/u_stereo_camera.c @@ -0,0 +1,429 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera (ADR-043) building blocks — see u_stereo_camera.h. + * @ingroup aux_util + */ + +#include "util/u_stereo_camera.h" + +#include +#include + + +/* + * + * Ring. + * + */ + +void +u_stereo_camera_ring_init(struct u_stereo_camera_ring *r) +{ + memset(r, 0, sizeof(*r)); + r->latest = -1; + r->pinned = -1; + r->writing = -1; + r->latest_acquired = true; +} + +int32_t +u_stereo_camera_ring_begin_write(struct u_stereo_camera_ring *r) +{ + for (int32_t s = 0; s < U_STEREO_CAMERA_RING_SLOTS; s++) { + if (s != r->latest && s != r->pinned) { + r->writing = s; + return s; + } + } + // Unreachable with 3 slots: at most two are latest / pinned. + r->writing = -1; + return -1; +} + +void +u_stereo_camera_ring_publish(struct u_stereo_camera_ring *r, int32_t slot, uint64_t seq) +{ + if (slot < 0 || slot >= U_STEREO_CAMERA_RING_SLOTS || slot == r->pinned) { + r->writing = -1; + return; + } + if (r->latest >= 0 && !r->latest_acquired) { + r->skipped++; + } + r->latest = slot; + r->latest_seq = seq; + r->latest_acquired = false; + r->writing = -1; + r->published++; +} + +void +u_stereo_camera_ring_abort_write(struct u_stereo_camera_ring *r) +{ + r->writing = -1; +} + +bool +u_stereo_camera_ring_acquire(struct u_stereo_camera_ring *r, int32_t *out_slot, uint64_t *out_seq) +{ + if (r->latest < 0 || r->latest_acquired || r->latest_seq <= r->last_acquired_seq) { + return false; + } + r->pinned = r->latest; + r->last_acquired_seq = r->latest_seq; + r->latest_acquired = true; + r->acquired++; + *out_slot = r->pinned; + *out_seq = r->latest_seq; + return true; +} + +void +u_stereo_camera_ring_clear(struct u_stereo_camera_ring *r) +{ + r->pinned = -1; + r->latest = -1; + r->latest_acquired = true; +} + + +/* + * + * Source-rate meter. + * + */ + +void +u_stereo_camera_rate_meter_restart(struct u_stereo_camera_rate_meter *m) +{ + m->first_ns = 0; + m->last_ns = 0; + m->intervals = 0; +} + +bool +u_stereo_camera_rate_meter_push(struct u_stereo_camera_rate_meter *m, int64_t t_ns) +{ + if (m->first_ns == 0 || t_ns <= m->last_ns || t_ns - m->last_ns > U_STEREO_CAMERA_RATE_GAP_NS) { + m->first_ns = t_ns; + m->last_ns = t_ns; + m->intervals = 0; + return false; + } + m->last_ns = t_ns; + if (++m->intervals < U_STEREO_CAMERA_RATE_WINDOW) { + return false; + } + m->rate = (float)((double)m->intervals * 1e9 / (double)(m->last_ns - m->first_ns)); + m->first_ns = t_ns; // the next window starts here + m->intervals = 0; + return true; +} + + +/* + * + * Decimator. + * + */ + +void +u_stereo_camera_decimator_init(struct u_stereo_camera_decimator *d, float max_rate, float source_rate) +{ + memset(d, 0, sizeof(*d)); + if (max_rate <= 0.0f || (source_rate > 0.0f && max_rate >= source_rate * 0.999f)) { + d->period_ns = 0; + return; + } + d->period_ns = (int64_t)(1e9 / (double)max_rate); +} + +bool +u_stereo_camera_decimator_accept(struct u_stereo_camera_decimator *d, int64_t t_ns) +{ + if (d->period_ns <= 0) { + return true; + } + if (!d->started) { + d->started = true; + d->next_due_ns = t_ns + d->period_ns; + return true; + } + // A tenth of a period of slack absorbs source jitter without letting two + // frames through for one due time. + if (t_ns < d->next_due_ns - d->period_ns / 10) { + return false; + } + d->next_due_ns += d->period_ns; + if (d->next_due_ns <= t_ns) { + // Fell behind (source paused / suspended): re-anchor, never burst. + d->next_due_ns = t_ns + d->period_ns; + } + return true; +} + + +/* + * + * Layout + conversion. + * + */ + +static uint32_t +align64(uint32_t v) +{ + return (v + 63u) & ~63u; +} + +bool +u_stereo_camera_layout(uint32_t format, uint32_t width, uint32_t height, struct u_stereo_camera_planes *out) +{ + memset(out, 0, sizeof(*out)); + if (width == 0 || height == 0) { + return false; + } + switch (format) { + case 1: // GRAY8 + out->plane_count = 1; + out->pitch[0] = align64(width); + out->size = (uint64_t)out->pitch[0] * height; + return true; + case 2: // NV12 + if ((width & 1u) != 0 || (height & 1u) != 0) { + return false; + } + out->plane_count = 2; + out->pitch[0] = align64(width); + out->pitch[1] = align64(width); + out->offset[1] = (uint64_t)out->pitch[0] * height; + out->size = out->offset[1] + (uint64_t)out->pitch[1] * (height / 2); + return true; + case 3: // BGRA8 + out->plane_count = 1; + out->pitch[0] = align64(width * 4u); + out->size = (uint64_t)out->pitch[0] * height; + return true; + default: return false; + } +} + +static inline uint8_t +clamp_u8(int v) +{ + return (uint8_t)(v < 0 ? 0 : (v > 255 ? 255 : v)); +} + +static inline uint8_t +luma_601(int r, int g, int b) +{ + return clamp_u8(16 + ((66 * r + 129 * g + 25 * b + 128) >> 8)); +} + +bool +u_stereo_camera_convert(uint32_t src_format, + const uint8_t *const src_planes[2], + const uint32_t src_pitches[2], + uint32_t dst_format, + uint8_t *dst, + const struct u_stereo_camera_planes *dl, + uint32_t width, + uint32_t height) +{ + if (src_planes == NULL || src_planes[0] == NULL || dst == NULL || dl == NULL || width == 0 || height == 0) { + return false; + } + uint8_t *d0 = dst + dl->offset[0]; + uint8_t *d1 = dst + dl->offset[1]; + const uint8_t *s0 = src_planes[0]; + const uint8_t *s1 = src_planes[1]; + + // Same format: row copy (pitches may differ). + if (src_format == dst_format) { + uint32_t row = src_format == 3 ? width * 4u : width; + for (uint32_t y = 0; y < height; y++) { + memcpy(d0 + (size_t)y * dl->pitch[0], s0 + (size_t)y * src_pitches[0], row); + } + if (src_format == 2) { + if (s1 == NULL) { + return false; + } + for (uint32_t y = 0; y < height / 2; y++) { + memcpy(d1 + (size_t)y * dl->pitch[1], s1 + (size_t)y * src_pitches[1], width); + } + } + return src_format >= 1 && src_format <= 3; + } + + // GRAY8 / NV12 luma are copied verbatim between each other (a grey sensor's + // value IS its luma); only the RGB legs apply BT.601. + if ((src_format == 1 || src_format == 2) && dst_format == 1) { + for (uint32_t y = 0; y < height; y++) { + memcpy(d0 + (size_t)y * dl->pitch[0], s0 + (size_t)y * src_pitches[0], width); + } + return true; + } + if (src_format == 1 && dst_format == 2) { + for (uint32_t y = 0; y < height; y++) { + memcpy(d0 + (size_t)y * dl->pitch[0], s0 + (size_t)y * src_pitches[0], width); + } + for (uint32_t y = 0; y < height / 2; y++) { + memset(d1 + (size_t)y * dl->pitch[1], 128, width); + } + return true; + } + if (src_format == 1 && dst_format == 3) { + for (uint32_t y = 0; y < height; y++) { + const uint8_t *s = s0 + (size_t)y * src_pitches[0]; + uint8_t *o = d0 + (size_t)y * dl->pitch[0]; + for (uint32_t x = 0; x < width; x++) { + o[4 * x + 0] = o[4 * x + 1] = o[4 * x + 2] = s[x]; + o[4 * x + 3] = 255; + } + } + return true; + } + if (src_format == 2 && dst_format == 3) { + if (s1 == NULL) { + return false; + } + for (uint32_t y = 0; y < height; y++) { + const uint8_t *ys = s0 + (size_t)y * src_pitches[0]; + const uint8_t *uv = s1 + (size_t)(y / 2) * src_pitches[1]; + uint8_t *o = d0 + (size_t)y * dl->pitch[0]; + for (uint32_t x = 0; x < width; x++) { + int c = (int)ys[x] - 16; + int dd = (int)uv[x & ~1u] - 128; + int e = (int)uv[(x & ~1u) + 1] - 128; + o[4 * x + 2] = clamp_u8((298 * c + 409 * e + 128) >> 8); + o[4 * x + 1] = clamp_u8((298 * c - 100 * dd - 208 * e + 128) >> 8); + o[4 * x + 0] = clamp_u8((298 * c + 516 * dd + 128) >> 8); + o[4 * x + 3] = 255; + } + } + return true; + } + if (src_format == 3 && dst_format == 1) { + for (uint32_t y = 0; y < height; y++) { + const uint8_t *s = s0 + (size_t)y * src_pitches[0]; + uint8_t *o = d0 + (size_t)y * dl->pitch[0]; + for (uint32_t x = 0; x < width; x++) { + // Full-range luma: a GRAY8 consumer wants intensity, not video levels. + o[x] = + (uint8_t)((29 * s[4 * x + 0] + 150 * s[4 * x + 1] + 77 * s[4 * x + 2] + 128) >> 8); + } + } + return true; + } + if (src_format == 3 && dst_format == 2) { + if ((width & 1u) || (height & 1u)) { + return false; + } + for (uint32_t y = 0; y < height; y++) { + const uint8_t *s = s0 + (size_t)y * src_pitches[0]; + uint8_t *o = d0 + (size_t)y * dl->pitch[0]; + for (uint32_t x = 0; x < width; x++) { + o[x] = luma_601(s[4 * x + 2], s[4 * x + 1], s[4 * x + 0]); + } + } + for (uint32_t y = 0; y < height / 2; y++) { + const uint8_t *a = s0 + (size_t)(2 * y) * src_pitches[0]; + const uint8_t *b = a + src_pitches[0]; + uint8_t *o = d1 + (size_t)y * dl->pitch[1]; + for (uint32_t x = 0; x < width; x += 2) { + int bb = a[4 * x] + a[4 * x + 4] + b[4 * x] + b[4 * x + 4]; + int gg = a[4 * x + 1] + a[4 * x + 5] + b[4 * x + 1] + b[4 * x + 5]; + int rr = a[4 * x + 2] + a[4 * x + 6] + b[4 * x + 2] + b[4 * x + 6]; + bb = (bb + 2) / 4; + gg = (gg + 2) / 4; + rr = (rr + 2) / 4; + o[x] = clamp_u8(128 + ((-38 * rr - 74 * gg + 112 * bb + 128) >> 8)); + o[x + 1] = clamp_u8(128 + ((112 * rr - 94 * gg - 18 * bb + 128) >> 8)); + } + } + return true; + } + return false; +} + + +/* + * + * Identity + probe. + * + */ + +static uint64_t +fnv1a64(uint64_t h, const char *s) +{ + for (; s != NULL && *s != '\0'; s++) { + h ^= (uint8_t)*s; + h *= 0x100000001b3ull; + } + return h; +} + +void +u_stereo_camera_persistent_id(const char *device_identity, const char *consumer, char out[64]) +{ + uint64_t a = fnv1a64(0xcbf29ce484222325ull, "dxr-stereo-camera|"); + a = fnv1a64(a, device_identity); + a = fnv1a64(a, "|"); + a = fnv1a64(a, consumer); + // Second lane with a different basis so the id is 128 bits wide. + uint64_t b = fnv1a64(0x84222325cbf29ce4ull, consumer); + b = fnv1a64(b, "|"); + b = fnv1a64(b, device_identity); + snprintf(out, 64, "dxrcam-%016llx%016llx", (unsigned long long)a, (unsigned long long)b); +} + +bool +u_stereo_camera_estimate_disparity(const uint8_t *gray, + uint32_t pitch, + uint32_t eye_width, + uint32_t height, + uint32_t x0, + uint32_t y0, + uint32_t w, + uint32_t h, + uint32_t max_disparity, + float *out_disparity) +{ + if (gray == NULL || out_disparity == NULL || w == 0 || h == 0 || x0 + w > eye_width || y0 + h > height) { + return false; + } + uint32_t dmax = max_disparity < x0 ? max_disparity : x0; // right x = left x - d >= 0 + uint64_t best = UINT64_MAX; + uint32_t best_d = 0; + uint64_t sad[1024]; + if (dmax >= 1024) { + dmax = 1023; + } + for (uint32_t d = 0; d <= dmax; d++) { + uint64_t s = 0; + for (uint32_t y = y0; y < y0 + h; y++) { + const uint8_t *l = gray + (size_t)y * pitch + x0; + const uint8_t *r = gray + (size_t)y * pitch + eye_width + x0 - d; + for (uint32_t x = 0; x < w; x++) { + int diff = (int)l[x] - (int)r[x]; + s += (uint64_t)(diff < 0 ? -diff : diff); + } + } + sad[d] = s; + if (s < best) { + best = s; + best_d = d; + } + } + float sub = 0.0f; + if (best_d > 0 && best_d < dmax) { + double a = (double)sad[best_d - 1], b = (double)sad[best_d], c = (double)sad[best_d + 1]; + double den = a - 2.0 * b + c; + if (den > 0.0) { + sub = (float)(0.5 * (a - c) / den); + } + } + *out_disparity = (float)best_d + sub; + return true; +} diff --git a/src/xrt/auxiliary/util/u_stereo_camera.h b/src/xrt/auxiliary/util/u_stereo_camera.h new file mode 100644 index 0000000000..ab17c84ad2 --- /dev/null +++ b/src/xrt/auxiliary/util/u_stereo_camera.h @@ -0,0 +1,225 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera (ADR-043) platform-neutral building blocks: + * the per-stream latest-wins pinned ring, the frame-rate decimator, + * plane layout + format conversion, the per-consumer persistent id, + * and a block-matching disparity probe. + * + * Pure logic — no threads, no OS handles — so the service's camera manager + * (ipc_server_stereo_camera.c), the sim_display fake and `displayxr-cli + * camera` share one tested implementation (tests_stereo_camera). + * + * @ingroup aux_util + */ + +#pragma once + +#include "xrt/xrt_compiler.h" + +#include +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + + +/* + * + * Latest-wins ring with one consumer pin. + * + * One ring per stream. The camera thread writes into a slot that is neither + * the latest published slot nor the one the consumer has pinned, then + * publishes it. An acquire hands out the latest slot IF it is newer than the + * last one this stream acquired and pins it: the slot stays byte-stable until + * the next acquire. Three slots are therefore always enough. Frames the + * consumer did not acquire in time are skipped, never queued. + * + */ + +#define U_STEREO_CAMERA_RING_SLOTS 3 + +struct u_stereo_camera_ring +{ + int32_t latest; //!< slot of the newest published frame, -1 = none + int32_t pinned; //!< slot the consumer holds, -1 = none + int32_t writing; //!< slot handed out by begin_write, -1 = none + uint64_t latest_seq; //!< sequence of @ref latest (0 = none) + uint64_t last_acquired_seq; //!< sequence of the last acquire (0 = none) + bool latest_acquired; //!< the consumer has taken @ref latest + + uint64_t published; //!< frames published to this ring + uint64_t skipped; //!< published frames superseded before any acquire + uint64_t acquired; //!< successful acquires +}; + +void +u_stereo_camera_ring_init(struct u_stereo_camera_ring *r); + +//! A slot that is neither latest nor pinned. Always succeeds (3 slots). +int32_t +u_stereo_camera_ring_begin_write(struct u_stereo_camera_ring *r); + +//! Publish the slot from begin_write as the newest frame @p seq (> 0). +void +u_stereo_camera_ring_publish(struct u_stereo_camera_ring *r, int32_t slot, uint64_t seq); + +//! Abandon a begin_write (conversion failed): the slot stays free. +void +u_stereo_camera_ring_abort_write(struct u_stereo_camera_ring *r); + +/*! + * Pin and return the newest frame if newer than the last acquire. The + * previous pin is released first (it is only valid until the next acquire). + * @return false = NOT READY (the previous pin is kept). + */ +bool +u_stereo_camera_ring_acquire(struct u_stereo_camera_ring *r, int32_t *out_slot, uint64_t *out_seq); + +/*! + * Drop the pin and forget the latest frame — used on suspension so the last + * image does not linger for a consumer that may no longer see frames. + */ +void +u_stereo_camera_ring_clear(struct u_stereo_camera_ring *r); + + +/* + * + * Decimation: never interpolate, never burst. + * + */ + +struct u_stereo_camera_decimator +{ + int64_t period_ns; //!< 0 = pass everything + int64_t next_due_ns; + bool started; +}; + +//! @p max_rate <= 0 or >= @p source_rate = pass everything. +void +u_stereo_camera_decimator_init(struct u_stereo_camera_decimator *d, float max_rate, float source_rate); + +//! Should the frame stamped @p t_ns be delivered? +bool +u_stereo_camera_decimator_accept(struct u_stereo_camera_decimator *d, int64_t t_ns); + + +/* + * + * Source-rate meter: what the source ACTUALLY delivers, not what it advertises. + * + * A plug-in often cannot know its rate at enumerate time (the Leia SR tracking + * camera advertised 30 Hz and ran 58-64 Hz). The service therefore measures + * the first @ref U_STEREO_CAMERA_RATE_WINDOW frames after every open and + * reports that as the descriptor's rate; until a window completes the rate is + * the plug-in's own value, where 0 means "unknown". + * + */ + +#define U_STEREO_CAMERA_RATE_WINDOW 60 //!< intervals per measurement +#define U_STEREO_CAMERA_RATE_GAP_NS (500ll * 1000 * 1000) //!< a longer gap restarts the window + +struct u_stereo_camera_rate_meter +{ + int64_t first_ns; + int64_t last_ns; + uint32_t intervals; + float rate; //!< last completed measurement, Hz; 0 = none yet +}; + +//! Forget the window in progress (a source re-open); keeps the last @ref rate. +void +u_stereo_camera_rate_meter_restart(struct u_stereo_camera_rate_meter *m); + +/*! + * Feed one frame's timestamp. A gap over @ref U_STEREO_CAMERA_RATE_GAP_NS (a + * suspension) restarts the window rather than diluting it. + * @return true exactly when this frame completed a window (@ref rate updated). + */ +bool +u_stereo_camera_rate_meter_push(struct u_stereo_camera_rate_meter *m, int64_t t_ns); + + +/* + * + * Plane layout + conversion (GRAY8 = 1, NV12 = 2, BGRA8 = 3; the + * xrt_stereo_camera_format values). + * + */ + +struct u_stereo_camera_planes +{ + uint32_t plane_count; + uint32_t pitch[2]; + uint64_t offset[2]; + uint64_t size; //!< bytes the whole image needs +}; + +/*! + * Tight layout of a @p width x @p height image, pitches rounded up to 64 bytes + * (GPU-upload and SIMD friendly). NV12 needs even width and height. + * @return false on an unknown format or odd NV12 extent. + */ +bool +u_stereo_camera_layout(uint32_t format, uint32_t width, uint32_t height, struct u_stereo_camera_planes *out); + +/*! + * Convert one SBS image between formats. BT.601 limited range for the + * luma<->RGB legs; a GRAY8 source becomes NV12 with neutral (128) chroma. + * @return false on an unsupported pair or bad arguments. + */ +bool +u_stereo_camera_convert(uint32_t src_format, + const uint8_t *const src_planes[2], + const uint32_t src_pitches[2], + uint32_t dst_format, + uint8_t *dst, + const struct u_stereo_camera_planes *dst_layout, + uint32_t width, + uint32_t height); + + +/* + * + * Identity + probe helpers. + * + */ + +/*! + * A per-(device, consumer) id: "dxrcam-" + 32 hex chars. Never the raw device + * identity (fingerprinting, spec §7.5). + * + * R1: an unkeyed 128-bit FNV-1a pair over (device_identity, consumer). R3 + * keys it with a persisted per-user secret so ids cannot be recomputed from a + * known serial. + */ +void +u_stereo_camera_persistent_id(const char *device_identity, const char *consumer, char out[64]); + +/*! + * Horizontal disparity (left x - right x, pixels, sub-pixel) of the region + * [x0, x0+w) x [y0, y0+h) of the LEFT eye of a GRAY8 SBS image, by SAD block + * matching over 0..@p max_disparity. Used by `displayxr-cli camera probe` and + * the tests to check a fake source's known disparity. + * @return false if the region does not fit. + */ +bool +u_stereo_camera_estimate_disparity(const uint8_t *gray, + uint32_t pitch, + uint32_t eye_width, + uint32_t height, + uint32_t x0, + uint32_t y0, + uint32_t w, + uint32_t h, + uint32_t max_disparity, + float *out_disparity); + +#ifdef __cplusplus +} +#endif diff --git a/src/xrt/drivers/CMakeLists.txt b/src/xrt/drivers/CMakeLists.txt index 1f412e8776..d57543e2f9 100644 --- a/src/xrt/drivers/CMakeLists.txt +++ b/src/xrt/drivers/CMakeLists.txt @@ -75,6 +75,10 @@ set(SIM_DISPLAY_SOURCES sim_display/sim_display_device.c sim_display/sim_display_interface.h sim_display/sim_display_zone_common.h + sim_display/sim_display_stereo_camera.c + sim_display/sim_display_stereo_camera.h + sim_display/sim_display_stereo_camera_pattern.c + sim_display/sim_display_stereo_camera_pattern.h ) # Vulkan display processor (anaglyph/blend shaders) diff --git a/src/xrt/drivers/sim_display/sim_display_plugin.c b/src/xrt/drivers/sim_display/sim_display_plugin.c index b2c8e80c5b..906ff323dc 100644 --- a/src/xrt/drivers/sim_display/sim_display_plugin.c +++ b/src/xrt/drivers/sim_display/sim_display_plugin.c @@ -34,6 +34,7 @@ #include "xrt/xrt_results.h" #include "sim_display_interface.h" +#include "sim_display_stereo_camera.h" #if defined(XRT_HAVE_VULKAN) || !defined(_WIN32) #include "vk/vk_helpers.h" // #1243: sizeof(struct vk_bundle) fingerprint #endif @@ -283,6 +284,18 @@ static struct xrt_plugin_iface g_sim_display_iface = { #endif .get_platform_state = sim_display_plugin_get_platform_state, + + /* + * ADR-043 stereo camera source. The slots are always filled; the FAKE + * camera itself only exists under SIM_DISPLAY_FAKE_STEREO_CAMERA=1 (the + * enumerate slot reports zero cameras otherwise). + */ + .stereo_camera_enumerate = sim_display_stereo_camera_enumerate, + .stereo_camera_get_calibration = sim_display_stereo_camera_get_calibration, + .stereo_camera_open = sim_display_stereo_camera_open, + .stereo_camera_wait_frame = sim_display_stereo_camera_wait_frame, + .stereo_camera_release_frame = sim_display_stereo_camera_release_frame, + .stereo_camera_close = sim_display_stereo_camera_close, }; diff --git a/src/xrt/drivers/sim_display/sim_display_stereo_camera.c b/src/xrt/drivers/sim_display/sim_display_stereo_camera.c new file mode 100644 index 0000000000..9908a35015 --- /dev/null +++ b/src/xrt/drivers/sim_display/sim_display_stereo_camera.c @@ -0,0 +1,304 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief sim_display FAKE stereo camera provider (ADR-043). See the header. + * + * Paces itself on os_monotonic_get_ns (no thread of its own): wait_frame + * sleeps until the next frame's due time, renders the scene with the frame + * number burned in, and hands the runtime a pointer into its own buffer — + * exactly the shape a real plug-in has (a tracker publishes, the plug-in + * copies/decodes, the runtime's camera thread consumes). + * + * @ingroup drv_sim_display + */ + +#include "sim_display_stereo_camera.h" +#include "sim_display_stereo_camera_pattern.h" + +#include "os/os_time.h" +#include "util/u_logging.h" +#include "util/u_stereo_camera.h" + +#include +#include +#include +#include + +#define FAKE_HFOV_DEG 68.0 +#define FAKE_BASELINE_MM 50.0 +#define FAKE_BG_DEPTH_M 2.0 +#define FAKE_BAR_DEPTH_M 0.6 + +struct fake_config +{ + bool enabled; + uint32_t eye_w, eye_h; + float fps; + float advertised_fps; //!< what enumerate claims; 0 = "unknown" (the service measures) + double baseline_mm; //!< 50 mm default; 120 mm reproduces the Leia SR tracking camera's disparities + uint32_t format; + int64_t suspend_period_ns; +}; + +//! Completes the plug-in-owned opaque type. +struct xrt_plugin_stereo_camera +{ + struct fake_config cfg; + struct sim_stereo_camera_scene scene; + uint8_t *gray; + uint32_t gray_pitch; + uint8_t *converted; + struct u_stereo_camera_planes conv_layout; + int64_t t0_ns; //!< pacing anchor (re-anchored after a pause) + int64_t open_ns; //!< suspend square-wave origin (never moves) + int64_t period_ns; + uint64_t seq; +}; + +static const struct fake_config * +fake_config(void) +{ + static struct fake_config cfg; + static bool read = false; + if (read) { + return &cfg; + } + read = true; + const char *e = getenv("SIM_DISPLAY_FAKE_STEREO_CAMERA"); + cfg.enabled = e != NULL && e[0] != '\0' && e[0] != '0'; + cfg.eye_w = 640; + cfg.eye_h = 480; + cfg.fps = 30.0f; + cfg.format = XRT_PLUGIN_STEREO_CAMERA_FORMAT_GRAY8; + const char *sz = getenv("SIM_DISPLAY_FAKE_STEREO_CAMERA_SIZE"); + unsigned w = 0, h = 0; + if (sz != NULL && sscanf(sz, "%ux%u", &w, &h) == 2 && w >= 64 && h >= 64 && w <= 4096 && h <= 4096) { + cfg.eye_w = w & ~1u; // NV12 needs even extents + cfg.eye_h = h & ~1u; + } + const char *fps = getenv("SIM_DISPLAY_FAKE_STEREO_CAMERA_FPS"); + if (fps != NULL && atof(fps) >= 1.0 && atof(fps) <= 240.0) { + cfg.fps = (float)atof(fps); + } + cfg.advertised_fps = cfg.fps; + cfg.baseline_mm = FAKE_BASELINE_MM; + const char *bl = getenv("SIM_DISPLAY_FAKE_STEREO_CAMERA_BASELINE_MM"); + if (bl != NULL && atof(bl) >= 10.0 && atof(bl) <= 300.0) { + cfg.baseline_mm = atof(bl); + } + const char *afps = getenv("SIM_DISPLAY_FAKE_STEREO_CAMERA_ADVERTISED_FPS"); + if (afps != NULL && afps[0] != '\0' && atof(afps) >= 0.0 && atof(afps) <= 240.0) { + cfg.advertised_fps = (float)atof(afps); + } + const char *fmt = getenv("SIM_DISPLAY_FAKE_STEREO_CAMERA_FORMAT"); + if (fmt != NULL) { + if (strcmp(fmt, "nv12") == 0) { + cfg.format = XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12; + } else if (strcmp(fmt, "bgra") == 0 || strcmp(fmt, "bgra8") == 0) { + cfg.format = XRT_PLUGIN_STEREO_CAMERA_FORMAT_BGRA8; + } + } + const char *sp = getenv("SIM_DISPLAY_FAKE_STEREO_CAMERA_SUSPEND_PERIOD_MS"); + if (sp != NULL && atol(sp) > 0) { + cfg.suspend_period_ns = (int64_t)atol(sp) * 1000000; + } + if (cfg.enabled) { + U_LOG_W("sim_display: FAKE stereo camera ON — %ux%u per eye @ %.1f Hz, baseline %.1f mm, format %u%s", + cfg.eye_w, cfg.eye_h, cfg.fps, cfg.baseline_mm, cfg.format, + cfg.suspend_period_ns > 0 ? ", suspend square-wave" : ""); + } + return &cfg; +} + +static double +fake_fx(uint32_t eye_w) +{ + return (eye_w / 2.0) / tan(FAKE_HFOV_DEG * 0.5 * 3.14159265358979323846 / 180.0); +} + +uint32_t +sim_display_stereo_camera_enumerate(struct xrt_plugin_instance *inst, + uint32_t capacity, + struct xrt_plugin_stereo_camera_info *out) +{ + (void)inst; + const struct fake_config *cfg = fake_config(); + if (!cfg->enabled) { + return 0; + } + if (capacity >= 1 && out != NULL) { + uint32_t sz = out->struct_size; + struct xrt_plugin_stereo_camera_info info; + memset(&info, 0, sizeof(info)); + info.struct_size = (uint32_t)sizeof(info); + snprintf(info.display_name, sizeof(info.display_name), "Sim 3D camera (fake)"); + snprintf(info.device_identity, sizeof(info.device_identity), "sim-display-fake-stereo-camera-0001"); + info.flags = XRT_PLUGIN_STEREO_CAMERA_SHARED_WITH_EYE_TRACKING | XRT_PLUGIN_STEREO_CAMERA_USER_FACING | + XRT_PLUGIN_STEREO_CAMERA_CALIBRATED | XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED; + if (cfg->format == XRT_PLUGIN_STEREO_CAMERA_FORMAT_GRAY8) { + info.flags |= XRT_PLUGIN_STEREO_CAMERA_MONOCHROME; + } + info.eye_width = cfg->eye_w; + info.eye_height = cfg->eye_h; + info.max_frame_rate = cfg->advertised_fps; + info.native_format = cfg->format; + // Never write past the runtime's struct_size (ADR-020). + memcpy(out, &info, sz < sizeof(info) ? sz : sizeof(info)); + out->struct_size = sz; + } + return 1; +} + +xrt_result_t +sim_display_stereo_camera_get_calibration(struct xrt_plugin_instance *inst, + uint32_t index, + struct xrt_plugin_stereo_camera_calibration *out) +{ + (void)inst; + const struct fake_config *cfg = fake_config(); + if (!cfg->enabled || index != 0 || out == NULL) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + uint32_t sz = out->struct_size; + struct xrt_plugin_stereo_camera_calibration c; + memset(&c, 0, sizeof(c)); + c.struct_size = (uint32_t)sizeof(c); + c.image_width = cfg->eye_w; + c.image_height = cfg->eye_h; + double fx = fake_fx(cfg->eye_w); + for (int e = 0; e < 2; e++) { + c.k[e][0] = fx; + c.k[e][1] = fx; + c.k[e][2] = (cfg->eye_w - 1) * 0.5; + c.k[e][3] = (cfg->eye_h - 1) * 0.5; + } + c.distortion_model = XRT_PLUGIN_STEREO_CAMERA_DISTORTION_NONE; + c.rotation_right_from_left[0][0] = 1.0; + c.rotation_right_from_left[1][1] = 1.0; + c.rotation_right_from_left[2][2] = 1.0; + // OpenCV convention (x_R = R x_L + T): a right camera at +x of the left + // one has T = (-B, 0, 0). + c.translation_right_from_left_mm[0] = -cfg->baseline_mm; + memcpy(out, &c, sz < sizeof(c) ? sz : sizeof(c)); + out->struct_size = sz; + return XRT_SUCCESS; +} + +xrt_result_t +sim_display_stereo_camera_open(struct xrt_plugin_instance *inst, + uint32_t index, + struct xrt_plugin_stereo_camera **out_cam) +{ + (void)inst; + const struct fake_config *cfg = fake_config(); + if (!cfg->enabled || index != 0 || out_cam == NULL) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + struct xrt_plugin_stereo_camera *cam = calloc(1, sizeof(*cam)); + if (cam == NULL) { + return XRT_ERROR_ALLOCATION; + } + cam->cfg = *cfg; + sim_stereo_camera_scene_init(&cam->scene, cfg->eye_w, cfg->eye_h, fake_fx(cfg->eye_w), cfg->baseline_mm, + FAKE_BG_DEPTH_M, FAKE_BAR_DEPTH_M); + cam->gray_pitch = 2 * cfg->eye_w; + cam->gray = calloc((size_t)cam->gray_pitch * cfg->eye_h, 1); + if (cfg->format != XRT_PLUGIN_STEREO_CAMERA_FORMAT_GRAY8) { + u_stereo_camera_layout(cfg->format, 2 * cfg->eye_w, cfg->eye_h, &cam->conv_layout); + cam->converted = calloc((size_t)cam->conv_layout.size, 1); + } + if (cam->gray == NULL || (cfg->format != XRT_PLUGIN_STEREO_CAMERA_FORMAT_GRAY8 && cam->converted == NULL)) { + free(cam->gray); + free(cam->converted); + free(cam); + return XRT_ERROR_ALLOCATION; + } + cam->period_ns = (int64_t)(1e9 / cfg->fps); + cam->t0_ns = os_monotonic_get_ns(); + cam->open_ns = cam->t0_ns; + U_LOG_W("sim_display: FAKE stereo camera opened (bg disparity %u px, bar disparity %u px)", + cam->scene.bg_disparity, cam->scene.bar_disparity); + *out_cam = cam; + return XRT_SUCCESS; +} + +static void +sleep_ns(int64_t ns) +{ + if (ns > 0) { + os_nanosleep(ns); + } +} + +uint32_t +sim_display_stereo_camera_wait_frame(struct xrt_plugin_stereo_camera *cam, + int64_t timeout_ns, + struct xrt_plugin_stereo_camera_frame *out) +{ + if (cam == NULL || out == NULL) { + return XRT_PLUGIN_STEREO_CAMERA_WAIT_ERROR; + } + int64_t now = os_monotonic_get_ns(); + + // SUSPEND square wave: odd half-periods are "tracker stopped". + if (cam->cfg.suspend_period_ns > 0 && ((now - cam->open_ns) / cam->cfg.suspend_period_ns) % 2 == 1) { + sleep_ns(timeout_ns < 20000000 ? timeout_ns : 20000000); + return XRT_PLUGIN_STEREO_CAMERA_WAIT_SUSPENDED; + } + + int64_t due = cam->t0_ns + (int64_t)cam->seq * cam->period_ns; + if (now - due > cam->period_ns) { + // Back from a pause: re-anchor instead of bursting the missed frames. + cam->t0_ns = now - (int64_t)cam->seq * cam->period_ns; + due = now; + } + if (due - now > timeout_ns) { + sleep_ns(timeout_ns); + return XRT_PLUGIN_STEREO_CAMERA_WAIT_TIMEOUT; + } + sleep_ns(due - now); + + cam->seq++; + sim_stereo_camera_render_gray(&cam->scene, cam->seq, cam->gray, cam->gray_pitch); + + memset(out, 0, sizeof(*out)); + out->sequence = cam->seq; + out->time_ns = due; // the synthetic "exposure" + out->time_is_exposure = true; + out->width = 2 * cam->cfg.eye_w; + out->height = cam->cfg.eye_h; + out->format = cam->cfg.format; + if (cam->cfg.format == XRT_PLUGIN_STEREO_CAMERA_FORMAT_GRAY8) { + out->planes[0] = cam->gray; + out->pitches[0] = cam->gray_pitch; + } else { + const uint8_t *src[2] = {cam->gray, NULL}; + const uint32_t pitches[2] = {cam->gray_pitch, 0}; + u_stereo_camera_convert(XRT_PLUGIN_STEREO_CAMERA_FORMAT_GRAY8, src, pitches, cam->cfg.format, + cam->converted, &cam->conv_layout, out->width, out->height); + for (uint32_t p = 0; p < cam->conv_layout.plane_count; p++) { + out->planes[p] = cam->converted + cam->conv_layout.offset[p]; + out->pitches[p] = cam->conv_layout.pitch[p]; + } + } + return XRT_PLUGIN_STEREO_CAMERA_WAIT_OK; +} + +void +sim_display_stereo_camera_release_frame(struct xrt_plugin_stereo_camera *cam) +{ + (void)cam; // the buffer is ours and is only rewritten by the next wait_frame +} + +void +sim_display_stereo_camera_close(struct xrt_plugin_stereo_camera *cam) +{ + if (cam == NULL) { + return; + } + U_LOG_W("sim_display: FAKE stereo camera closed after %llu frames", (unsigned long long)cam->seq); + free(cam->gray); + free(cam->converted); + free(cam); +} diff --git a/src/xrt/drivers/sim_display/sim_display_stereo_camera.h b/src/xrt/drivers/sim_display/sim_display_stereo_camera.h new file mode 100644 index 0000000000..94f1f8809e --- /dev/null +++ b/src/xrt/drivers/sim_display/sim_display_stereo_camera.h @@ -0,0 +1,59 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief sim_display FAKE stereo camera provider (ADR-043) — the plug-in + * iface camera slots, filled under SIM_DISPLAY_FAKE_STEREO_CAMERA=1. + * + * Knobs (read by the process that loads the plug-in — the SERVICE): + * SIM_DISPLAY_FAKE_STEREO_CAMERA=1 advertise one camera + * SIM_DISPLAY_FAKE_STEREO_CAMERA_SIZE=WxH per-eye size (default 640x480) + * SIM_DISPLAY_FAKE_STEREO_CAMERA_FPS=N source rate (default 30) + * SIM_DISPLAY_FAKE_STEREO_CAMERA_FORMAT=gray8|nv12|bgra native format (default gray8) + * SIM_DISPLAY_FAKE_STEREO_CAMERA_SUSPEND_PERIOD_MS=N square-wave SUSPENDED/AVAILABLE + * + * The camera: SHARED_WITH_EYE_TRACKING | USER_FACING | CALIBRATED | + * NATIVELY_RECTIFIED (+ MONOCHROME for gray8), 68 deg horizontal FOV per eye, + * 50 mm baseline, background at 2.0 m and a bar at 0.6 m — at 640 px per eye + * that is fx = 474.4 px and disparities of 12 px and 40 px. + * + * @ingroup drv_sim_display + */ + +#pragma once + +#include "xrt/xrt_plugin.h" + +#ifdef __cplusplus +extern "C" { +#endif + +uint32_t +sim_display_stereo_camera_enumerate(struct xrt_plugin_instance *inst, + uint32_t capacity, + struct xrt_plugin_stereo_camera_info *out); + +xrt_result_t +sim_display_stereo_camera_get_calibration(struct xrt_plugin_instance *inst, + uint32_t index, + struct xrt_plugin_stereo_camera_calibration *out); + +xrt_result_t +sim_display_stereo_camera_open(struct xrt_plugin_instance *inst, + uint32_t index, + struct xrt_plugin_stereo_camera **out_cam); + +uint32_t +sim_display_stereo_camera_wait_frame(struct xrt_plugin_stereo_camera *cam, + int64_t timeout_ns, + struct xrt_plugin_stereo_camera_frame *out); + +void +sim_display_stereo_camera_release_frame(struct xrt_plugin_stereo_camera *cam); + +void +sim_display_stereo_camera_close(struct xrt_plugin_stereo_camera *cam); + +#ifdef __cplusplus +} +#endif diff --git a/src/xrt/drivers/sim_display/sim_display_stereo_camera_pattern.c b/src/xrt/drivers/sim_display/sim_display_stereo_camera_pattern.c new file mode 100644 index 0000000000..b0d52b7102 --- /dev/null +++ b/src/xrt/drivers/sim_display/sim_display_stereo_camera_pattern.c @@ -0,0 +1,137 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief sim_display FAKE stereo camera: the synthetic SBS scene. + * @ingroup drv_sim_display + */ + +#include "sim_display_stereo_camera_pattern.h" + +#include +#include +#include + +//! 5x7 digits, one byte per row, bit 4 = leftmost column. +static const uint8_t k_digits[10][7] = { + {0x0E, 0x11, 0x13, 0x15, 0x19, 0x11, 0x0E}, // 0 + {0x04, 0x0C, 0x04, 0x04, 0x04, 0x04, 0x0E}, // 1 + {0x0E, 0x11, 0x01, 0x02, 0x04, 0x08, 0x1F}, // 2 + {0x1F, 0x02, 0x04, 0x02, 0x01, 0x11, 0x0E}, // 3 + {0x02, 0x06, 0x0A, 0x12, 0x1F, 0x02, 0x02}, // 4 + {0x1F, 0x10, 0x1E, 0x01, 0x01, 0x11, 0x0E}, // 5 + {0x06, 0x08, 0x10, 0x1E, 0x11, 0x11, 0x0E}, // 6 + {0x1F, 0x01, 0x02, 0x04, 0x08, 0x08, 0x08}, // 7 + {0x0E, 0x11, 0x11, 0x0E, 0x11, 0x11, 0x0E}, // 8 + {0x0E, 0x11, 0x11, 0x0F, 0x01, 0x02, 0x0C}, // 9 +}; + +#define COUNTER_DIGITS 8 +#define COUNTER_SCALE 3 +#define COUNTER_X 8 +#define COUNTER_Y 8 + +static inline uint32_t +hash3(uint32_t x, uint32_t y, uint32_t seed) +{ + uint32_t h = x * 0x8da6b343u ^ y * 0xd8163841u ^ seed * 0xcb1ab31fu; + h ^= h >> 13; + h *= 0x5bd1e995u; + h ^= h >> 15; + return h; +} + +//! Value noise on 2-pixel cells: detail a block matcher locks onto, without +//! single-pixel aliasing. +static inline uint8_t +texel(uint32_t u, uint32_t v, uint32_t seed, uint8_t lo, uint8_t span) +{ + return (uint8_t)(lo + hash3(u >> 1, v >> 1, seed) % span); +} + +void +sim_stereo_camera_scene_init(struct sim_stereo_camera_scene *s, + uint32_t eye_width, + uint32_t eye_height, + double fx_px, + double baseline_mm, + double bg_depth_m, + double bar_depth_m) +{ + memset(s, 0, sizeof(*s)); + s->eye_width = eye_width; + s->eye_height = eye_height; + s->bg_disparity = (uint32_t)lround(fx_px * (baseline_mm / 1000.0) / bg_depth_m); + s->bar_disparity = (uint32_t)lround(fx_px * (baseline_mm / 1000.0) / bar_depth_m); + s->bar_x0 = eye_width * 3 / 8; + s->bar_x1 = eye_width * 5 / 8; + s->bar_y0 = eye_height / 4; + s->bar_y1 = eye_height * 3 / 4; +} + +static void +render_eye(const struct sim_stereo_camera_scene *s, uint8_t *dst, uint32_t pitch, uint32_t x_off, int right) +{ + for (uint32_t y = 0; y < s->eye_height; y++) { + uint8_t *row = dst + (size_t)y * pitch + x_off; + bool in_bar_rows = y >= s->bar_y0 && y < s->bar_y1; + for (uint32_t x = 0; x < s->eye_width; x++) { + // A feature at scene coordinate u shows at x = u in the left eye and at + // x = u - d in the right eye, so the right eye samples u = x + d. + uint32_t ub = right ? x + s->bg_disparity : x; + uint32_t ubar = right ? x + s->bar_disparity : x; + if (in_bar_rows && ubar >= s->bar_x0 && ubar < s->bar_x1) { + row[x] = texel(ubar, y, 0xBA5u, 128, 128); + } else { + row[x] = texel(ub, y, 0x0B6u, 0, 112); + } + } + } +} + +static void +render_counter(uint8_t *dst, uint32_t pitch, uint32_t x_off, uint32_t eye_w, uint32_t eye_h, uint64_t n) +{ + const uint32_t cw = 6 * COUNTER_SCALE, ch = 9 * COUNTER_SCALE; + const uint32_t box_w = COUNTER_DIGITS * cw + COUNTER_SCALE, box_h = ch; + if (COUNTER_X + box_w > eye_w || COUNTER_Y + box_h > eye_h) { + return; + } + for (uint32_t y = 0; y < box_h; y++) { + memset(dst + (size_t)(COUNTER_Y + y) * pitch + x_off + COUNTER_X, 0, box_w); + } + char digits[COUNTER_DIGITS]; + for (int i = COUNTER_DIGITS - 1; i >= 0; i--) { + digits[i] = (char)(n % 10); + n /= 10; + } + for (uint32_t i = 0; i < COUNTER_DIGITS; i++) { + const uint8_t *g = k_digits[(int)digits[i]]; + uint32_t gx = COUNTER_X + COUNTER_SCALE + i * cw; + for (uint32_t r = 0; r < 7; r++) { + for (uint32_t c = 0; c < 5; c++) { + if (((g[r] >> (4 - c)) & 1u) == 0) { + continue; + } + for (uint32_t sy = 0; sy < COUNTER_SCALE; sy++) { + uint8_t *p = + dst + (size_t)(COUNTER_Y + COUNTER_SCALE + r * COUNTER_SCALE + sy) * pitch + + x_off + gx + c * COUNTER_SCALE; + memset(p, 255, COUNTER_SCALE); + } + } + } + } +} + +void +sim_stereo_camera_render_gray(const struct sim_stereo_camera_scene *s, + uint64_t frame_number, + uint8_t *dst, + uint32_t pitch) +{ + render_eye(s, dst, pitch, 0, 0); + render_eye(s, dst, pitch, s->eye_width, 1); + render_counter(dst, pitch, 0, s->eye_width, s->eye_height, frame_number); + render_counter(dst, pitch, s->eye_width, s->eye_width, s->eye_height, frame_number); +} diff --git a/src/xrt/drivers/sim_display/sim_display_stereo_camera_pattern.h b/src/xrt/drivers/sim_display/sim_display_stereo_camera_pattern.h new file mode 100644 index 0000000000..81b4953522 --- /dev/null +++ b/src/xrt/drivers/sim_display/sim_display_stereo_camera_pattern.h @@ -0,0 +1,64 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief sim_display FAKE stereo camera (ADR-043): the synthetic SBS scene. + * + * Pure CPU rendering, no OS dependencies, so the unit tests pin it host-side + * (tests_stereo_camera) next to the disparity probe that measures it. + * + * The scene is a random-dot stereogram with two planes at KNOWN integer + * disparities (left x - right x, pixels): a textured background and a nearer + * textured bar in the middle of the frame. A frame counter is burned into the + * top-left of BOTH halves at zero disparity. The pair is parallel and + * undistorted (the R1 fake reports NATIVELY_RECTIFIED); a distorted, + * misaligned variant for the rectifier's tests is R2's. + * + * @ingroup drv_sim_display + */ + +#pragma once + +#include + +#ifdef __cplusplus +extern "C" { +#endif + +struct sim_stereo_camera_scene +{ + uint32_t eye_width; + uint32_t eye_height; + uint32_t bg_disparity; //!< background plane, pixels + uint32_t bar_disparity; //!< near bar, pixels + //! The bar's rectangle in LEFT-eye coordinates. + uint32_t bar_x0, bar_y0, bar_x1, bar_y1; +}; + +/*! + * Scene for a pinhole pair of focal @p fx_px and baseline @p baseline_mm, with + * the background at @p bg_depth_m and the bar at @p bar_depth_m: disparities + * are round(fx * B / Z). + */ +void +sim_stereo_camera_scene_init(struct sim_stereo_camera_scene *s, + uint32_t eye_width, + uint32_t eye_height, + double fx_px, + double baseline_mm, + double bg_depth_m, + double bar_depth_m); + +/*! + * Render frame @p frame_number into a GRAY8 SBS image (2*eye_width x + * eye_height, @p pitch bytes per row, left eye left). + */ +void +sim_stereo_camera_render_gray(const struct sim_stereo_camera_scene *s, + uint64_t frame_number, + uint8_t *dst, + uint32_t pitch); + +#ifdef __cplusplus +} +#endif diff --git a/src/xrt/include/xrt/xrt_instance.h b/src/xrt/include/xrt/xrt_instance.h index 61767f5bd8..9a93cd75ae 100644 --- a/src/xrt/include/xrt/xrt_instance.h +++ b/src/xrt/include/xrt/xrt_instance.h @@ -23,6 +23,9 @@ extern "C" { struct xrt_prober; struct xrt_device; struct xrt_instance_android; +struct xrt_stereo_camera_client; +struct xrt_plugin_iface; +struct xrt_plugin_instance; struct xrt_space_overseer; struct xrt_system; struct xrt_system_devices; @@ -245,6 +248,23 @@ struct xrt_instance * @see xrt_instance_android */ struct xrt_instance_android *android_instance; + + /*! + * XR_DXR_stereo_camera (ADR-043) client aspect. Set by the IPC client + * instance only; NULL for an in-process instance (which enumerates zero + * cameras — the service is each camera's single owner). + */ + struct xrt_stereo_camera_client *stereo_camera; + + /*! + * Optional: the plug-in that claimed the system for THIS instance, for + * service-side owners of plug-in-provided sources (the IPC server's + * stereo camera manager, ADR-043). Set by the native instance; NULL on + * the IPC client instance. Returns false when no plug-in is active. + */ + bool (*get_active_plugin)(struct xrt_instance *xinst, + const struct xrt_plugin_iface **out_iface, + struct xrt_plugin_instance **out_inst); }; /*! diff --git a/src/xrt/include/xrt/xrt_openxr_includes.h b/src/xrt/include/xrt/xrt_openxr_includes.h index 4016e59f60..74dd4afa25 100644 --- a/src/xrt/include/xrt/xrt_openxr_includes.h +++ b/src/xrt/include/xrt/xrt_openxr_includes.h @@ -87,3 +87,4 @@ typedef __eglMustCastToProperFunctionPointerType (*PFNEGLGETPROCADDRESSPROC)(con #include "openxr/XR_DXR_display_zones.h" #include "openxr/XR_DXR_weave.h" #include "openxr/XR_DXR_lift.h" +#include "openxr/XR_DXR_stereo_camera.h" diff --git a/src/xrt/include/xrt/xrt_plugin.h b/src/xrt/include/xrt/xrt_plugin.h index 8e11724f37..7804744823 100644 --- a/src/xrt/include/xrt/xrt_plugin.h +++ b/src/xrt/include/xrt/xrt_plugin.h @@ -43,6 +43,7 @@ #include "xrt/xrt_display_processor_metal.h" #include +#include #include #ifdef __cplusplus @@ -628,6 +629,133 @@ struct xrt_plugin_host_iface */ #define XRT_PLUGIN_HOST_HAS_ANDROID_PACKAGE_VISIBILITY 1 +/* + * + * Stereo camera source (ADR-043, XR_DXR_stereo_camera). + * + * A display's stereo camera — usually the one its eye tracker looks through — + * produced by the plug-in from its own stack WITHOUT taking the device from + * the tracker. The runtime service owns threads, fan-out, rectification, + * format conversion, transport, consent and the in-use indicator; the plug-in + * owns reading + decoding frames and the calibration of ITS ACTIVE device. + * Graphics-API-neutral on purpose: a camera is a sensor, not a weaver, and it + * must not share a lifetime with a display processor that is recreated on + * presenter changes. Slots on @ref xrt_plugin_iface (appended, ADR-020). + * + */ + +//! @ref xrt_plugin_stereo_camera_info::flags — values equal the +//! XrStereoCameraFlagsDXR bits (static-asserted in the state tracker). +#define XRT_PLUGIN_STEREO_CAMERA_SHARED_WITH_EYE_TRACKING (1u << 0) +#define XRT_PLUGIN_STEREO_CAMERA_USER_FACING (1u << 1) +#define XRT_PLUGIN_STEREO_CAMERA_CALIBRATED (1u << 2) +#define XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED (1u << 3) +#define XRT_PLUGIN_STEREO_CAMERA_MONOCHROME (1u << 4) + +//! Pixel formats a plug-in delivers (values equal XrStereoCameraFormatDXR). +enum xrt_plugin_stereo_camera_format +{ + XRT_PLUGIN_STEREO_CAMERA_FORMAT_GRAY8 = 1, + XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12 = 2, + XRT_PLUGIN_STEREO_CAMERA_FORMAT_BGRA8 = 3, +}; + +//! Lens models (values equal XrStereoCameraDistortionModelDXR). +enum xrt_plugin_stereo_camera_distortion +{ + XRT_PLUGIN_STEREO_CAMERA_DISTORTION_NONE = 0, + XRT_PLUGIN_STEREO_CAMERA_DISTORTION_RADTAN5 = 1, + XRT_PLUGIN_STEREO_CAMERA_DISTORTION_RADTAN8 = 2, + XRT_PLUGIN_STEREO_CAMERA_DISTORTION_KB4 = 3, +}; + +//! Result of @ref xrt_plugin_iface::stereo_camera_wait_frame. +enum xrt_plugin_stereo_camera_wait +{ + //! @p out holds a new frame; call stereo_camera_release_frame when done. + XRT_PLUGIN_STEREO_CAMERA_WAIT_OK = 0, + //! No frame within the timeout (source up; e.g. tracker warming up). + XRT_PLUGIN_STEREO_CAMERA_WAIT_TIMEOUT = 1, + //! The source is temporarily not delivering (tracker stopped). + XRT_PLUGIN_STEREO_CAMERA_WAIT_SUSPENDED = 2, + //! The source failed or went away; the runtime closes it. + XRT_PLUGIN_STEREO_CAMERA_WAIT_ERROR = 3, +}; + +/*! + * One camera, as a plug-in describes it. The runtime sets @ref struct_size to + * its own size before the call; the plug-in must not write past it. + */ +struct xrt_plugin_stereo_camera_info +{ + uint32_t struct_size; + //! Human-readable, UTF-8 ("Built-in 3D camera"). + char display_name[128]; + //! Stable device key (e.g. the panel serial). The SERVICE hashes it per + //! consumer; it is never exported to clients. + char device_identity[128]; + //! The OS's id of the physical device the frames come from, or "". + char platform_device_hint[256]; + //! XRT_PLUGIN_STEREO_CAMERA_* bits. + uint32_t flags; + //! Native per-eye size. + uint32_t eye_width; + uint32_t eye_height; + /*! + * What the source delivers, Hz — ONLY if the plug-in actually knows it + * (a vendor-provided value, or one it measured on an earlier open). 0 = + * unknown; never a placeholder guess. The service measures the real rate + * over the first frames after each open and reports that to apps. + */ + float max_frame_rate; + //! @ref xrt_plugin_stereo_camera_format the plug-in decodes to. + uint32_t native_format; +}; + +/*! + * RAW calibration of the ACTIVE device (the one this plug-in weaves for and + * whose tracker it reads) — never "the first calibration on the machine". + */ +struct xrt_plugin_stereo_camera_calibration +{ + uint32_t struct_size; + //! Per eye, pixels. + uint32_t image_width; + uint32_t image_height; + //! Per eye: fx fy cx cy, pixels. + double k[2][4]; + //! @ref xrt_plugin_stereo_camera_distortion. + uint32_t distortion_model; + double distortion[2][8]; + double rotation_right_from_left[3][3]; + double translation_right_from_left_mm[3]; +}; + +/*! + * One frame handed to the runtime. Planes stay valid until + * stereo_camera_release_frame. Always ONE side-by-side image, left eye left. + */ +struct xrt_plugin_stereo_camera_frame +{ + //! Monotonic per open, from 1. + uint64_t sequence; + //! os_monotonic_get_ns() domain. + int64_t time_ns; + //! true = sensor/exposure time; false = arrival time in the plug-in. + bool time_is_exposure; + //! Full SBS extent (2 * eye_width x eye_height). + uint32_t width; + uint32_t height; + //! @ref xrt_plugin_stereo_camera_format. + uint32_t format; + const uint8_t *planes[2]; + uint32_t pitches[2]; +}; + +//! Opaque plug-in-owned open camera. +struct xrt_plugin_stereo_camera; + + /*! * The plug-in's vtable. Filled in by the plug-in inside its * `xrtPluginNegotiate` implementation and handed back to the runtime via @@ -925,6 +1053,67 @@ struct xrt_plugin_iface * @ref struct_size; no XRT_PLUGIN_API_VERSION_CURRENT bump). */ bool (*get_platform_state)(struct xrt_plugin_platform_status *out_status); + + /*! + * @name Stereo camera source (ADR-043, XR_DXR_stereo_camera) + * + * Optional, all-or-nothing: the runtime uses the camera slots only when + * @ref struct_size covers @ref stereo_camera_close and every one of them + * is non-NULL. Appended per ADR-020 (append-only within a major; gated by + * @ref struct_size; no XRT_PLUGIN_API_VERSION_CURRENT bump) AFTER + * @ref get_platform_state (ADR-045), the last member before this block. + * Announced by @ref XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA. + * + * Threading: enumerate / get_calibration are cheap and callable from any + * thread. wait_frame / release_frame are called from ONE runtime-owned + * camera thread per open camera. Nothing throws across this boundary. + * @{ + */ + + /*! + * Fill up to @p capacity entries of @p out (each with struct_size preset + * by the runtime) and return the number of cameras. @p capacity 0 / NULL + * @p out = count query. Stable indices for the plug-in instance's life. + */ + uint32_t (*stereo_camera_enumerate)(struct xrt_plugin_instance *inst, + uint32_t capacity, + struct xrt_plugin_stereo_camera_info *out); + + /*! + * RAW calibration of camera @p index, for the ACTIVE device. Return + * XRT_ERROR_FEATURE_NOT_SUPPORTED when it cannot be resolved — the camera + * must then not report XRT_PLUGIN_STEREO_CAMERA_CALIBRATED. + */ + xrt_result_t (*stereo_camera_get_calibration)(struct xrt_plugin_instance *inst, + uint32_t index, + struct xrt_plugin_stereo_camera_calibration *out); + + /*! + * Open camera @p index. Doubles as a tracker keep-alive: while any camera + * is open, a SHARED_WITH_EYE_TRACKING source keeps its tracker running. + * Must never open / reconfigure / re-time the device away from the tracker. + */ + xrt_result_t (*stereo_camera_open)(struct xrt_plugin_instance *inst, + uint32_t index, + struct xrt_plugin_stereo_camera **out_cam); + + /*! + * Block up to @p timeout_ns for the next frame. Returns an @ref + * xrt_plugin_stereo_camera_wait. On OK the frame's planes stay valid until + * @ref stereo_camera_release_frame. Must not wait on a wake object shared + * with other readers of the vendor channel (poll it instead; ADR-043). + */ + uint32_t (*stereo_camera_wait_frame)(struct xrt_plugin_stereo_camera *cam, + int64_t timeout_ns, + struct xrt_plugin_stereo_camera_frame *out); + + //! Release the frame of the last OK wait_frame. + void (*stereo_camera_release_frame)(struct xrt_plugin_stereo_camera *cam); + + //! Close; drops the keep-alive when it was the last open camera. + void (*stereo_camera_close)(struct xrt_plugin_stereo_camera *cam); + + /*! @} */ }; /*! @@ -943,6 +1132,29 @@ struct xrt_plugin_iface */ #define XRT_PLUGIN_HAS_PLATFORM_STATE 1 +/*! + * Defined when @ref xrt_plugin_iface carries the stereo camera slots + * (ADR-043), so a plug-in built against an older runtime header can + * #ifdef-guard filling them. + */ +#define XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA 1 + +/*! + * True when @p iface fills the whole stereo camera slot set (and its + * struct_size covers it). + */ +static inline bool +xrt_plugin_iface_has_stereo_camera(const struct xrt_plugin_iface *iface) +{ + if (iface == NULL || iface->struct_size < offsetof(struct xrt_plugin_iface, stereo_camera_close) + + sizeof(iface->stereo_camera_close)) { + return false; + } + return iface->stereo_camera_enumerate != NULL && iface->stereo_camera_get_calibration != NULL && + iface->stereo_camera_open != NULL && iface->stereo_camera_wait_frame != NULL && + iface->stereo_camera_release_frame != NULL && iface->stereo_camera_close != NULL; +} + /* * diff --git a/src/xrt/include/xrt/xrt_stereo_camera.h b/src/xrt/include/xrt/xrt_stereo_camera.h new file mode 100644 index 0000000000..73fdcd7b57 --- /dev/null +++ b/src/xrt/include/xrt/xrt_stereo_camera.h @@ -0,0 +1,210 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera (ADR-043) runtime-side value types and the + * client aspect of @ref xrt_instance. + * + * The value structs are plain fixed-size POD: they ride the IPC wire as-is + * (proto.json) and are translated 1:1 to the XR structs by the state tracker. + * Enum values equal the XR ones (static-asserted in oxr_stereo_camera.c). + * + * @ingroup xrt_iface + */ + +#pragma once + +#include "xrt/xrt_compiler.h" +#include "xrt/xrt_handles.h" +#include "xrt/xrt_results.h" + +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +//! Most cameras the service exposes (per plug-in). +#define XRT_STEREO_CAMERA_MAX_CAMERAS 4 +//! Streams per camera, service-wide (spec §8: XR_ERROR_LIMIT_REACHED past it). +#define XRT_STEREO_CAMERA_MAX_STREAMS_PER_CAMERA 8 +//! Slots in a stream's ring: latest + pinned + one to write. +#define XRT_STEREO_CAMERA_RING_SLOTS 3 + +enum xrt_stereo_camera_state +{ + XRT_STEREO_CAMERA_STATE_AVAILABLE = 1, + XRT_STEREO_CAMERA_STATE_WAITING = 2, + XRT_STEREO_CAMERA_STATE_SUSPENDED = 3, + XRT_STEREO_CAMERA_STATE_UNAVAILABLE = 4, +}; + +enum xrt_stereo_camera_output +{ + XRT_STEREO_CAMERA_OUTPUT_RECTIFIED = 1, + XRT_STEREO_CAMERA_OUTPUT_RAW = 2, +}; + +//! Same values as xrt_plugin_stereo_camera_format and XrStereoCameraFormatDXR. +enum xrt_stereo_camera_format +{ + XRT_STEREO_CAMERA_FORMAT_GRAY8 = 1, + XRT_STEREO_CAMERA_FORMAT_NV12 = 2, + XRT_STEREO_CAMERA_FORMAT_BGRA8 = 3, +}; + +enum xrt_stereo_camera_transport +{ + XRT_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY = 1, + XRT_STEREO_CAMERA_TRANSPORT_D3D11_TEXTURE = 2, + XRT_STEREO_CAMERA_TRANSPORT_AHARDWAREBUFFER = 3, +}; + +//! Format / transport sets are one bit per enum value: (1 << value). +#define XRT_STEREO_CAMERA_BIT(v) (1ull << (uint32_t)(v)) + +/*! + * One enumerated camera (XrStereoCameraPropertiesDXR). + */ +struct xrt_stereo_camera_properties +{ + uint64_t camera_id; + char persistent_id[64]; + char display_name[128]; + char platform_device_hint[256]; + uint64_t flags; //!< XRT_PLUGIN_STEREO_CAMERA_* bits (= XrStereoCameraFlagsDXR) + uint32_t state; //!< enum xrt_stereo_camera_state + uint32_t view_count; + uint32_t eye_width; + uint32_t eye_height; + float max_frame_rate; + float baseline_mm; + float horizontal_fov_deg; + uint32_t reserved; + uint64_t supported_formats; //!< XRT_STEREO_CAMERA_BIT(format) + uint64_t supported_transports; //!< XRT_STEREO_CAMERA_BIT(transport) +}; + +struct xrt_stereo_camera_intrinsics +{ + uint32_t width; + uint32_t height; + float fx, fy, cx, cy; + uint32_t model; //!< xrt_plugin_stereo_camera_distortion + float coefficients[8]; +}; + +/*! + * Calibration as a client sees it (XrStereoCameraCalibrationDXR). + */ +struct xrt_stereo_camera_calibration +{ + uint32_t output; //!< enum xrt_stereo_camera_output these numbers describe + uint32_t reserved; + struct xrt_stereo_camera_intrinsics eye[2]; + float orientation[4]; //!< right-from-left rotation, quaternion x y z w + float position[3]; //!< right-from-left translation, metres + float baseline_mm; +}; + +/*! + * What a client asks for at stream create. + */ +struct xrt_stereo_camera_stream_request +{ + uint64_t camera_id; + uint32_t output; //!< enum xrt_stereo_camera_output + uint32_t format; //!< enum xrt_stereo_camera_format + uint32_t transport; //!< enum xrt_stereo_camera_transport + float max_frame_rate; +}; + +/*! + * A started stream's layout (XrStereoCameraStreamInfoDXR + transport). + */ +struct xrt_stereo_camera_stream_layout +{ + uint32_t width; //!< full SBS + uint32_t height; + uint32_t format; + uint32_t output; + uint32_t transport; + float max_frame_rate; //!< effective delivery cap + uint64_t section_size; + uint64_t slot_stride; + uint32_t slot_count; + uint32_t reserved; +}; + +/*! + * One acquired frame (XrStereoCameraFrameDXR). @ref capture_time_ns is on the + * os_monotonic clock; the state tracker converts it to XrTime. + */ +struct xrt_stereo_camera_frame_info +{ + uint64_t frame_index; + uint32_t slot; + uint32_t width; + uint32_t height; + uint32_t format; + uint32_t row_pitch[2]; + uint64_t plane_offset[2]; + int64_t capture_time_ns; + uint32_t time_is_exposure; + uint32_t output; + uint32_t calibration_generation; + uint32_t reserved; +}; + +struct xrt_stereo_camera_stream_stats +{ + float source_frame_rate; + float delivered_frame_rate; + uint64_t frames_published; + uint64_t frames_skipped; + uint64_t frames_acquired; + uint64_t mean_latency_ns; +}; + +/*! + * Client aspect of an @ref xrt_instance: set by the IPC client instance only + * (the camera's single owner is the service); NULL for an in-process instance, + * which therefore enumerates zero cameras. Streams are service-side ids owned + * by the instance's connection. + */ +struct xrt_stereo_camera_client +{ + xrt_result_t (*count)(struct xrt_stereo_camera_client *c, uint32_t *out_count); + xrt_result_t (*get_properties)(struct xrt_stereo_camera_client *c, + uint32_t index, + struct xrt_stereo_camera_properties *out_props); + xrt_result_t (*get_calibration)(struct xrt_stereo_camera_client *c, + uint64_t camera_id, + uint32_t output, + struct xrt_stereo_camera_calibration *out_calib); + xrt_result_t (*stream_create)(struct xrt_stereo_camera_client *c, + const struct xrt_stereo_camera_stream_request *req, + uint64_t *out_stream_id); + xrt_result_t (*stream_destroy)(struct xrt_stereo_camera_client *c, uint64_t stream_id); + xrt_result_t (*stream_start)(struct xrt_stereo_camera_client *c, uint64_t stream_id); + xrt_result_t (*stream_stop)(struct xrt_stereo_camera_client *c, uint64_t stream_id); + //! @p out_section / @p out_wake are NEW handles owned by the caller. + xrt_result_t (*stream_get_transport)(struct xrt_stereo_camera_client *c, + uint64_t stream_id, + struct xrt_stereo_camera_stream_layout *out_layout, + xrt_shmem_handle_t *out_section, + xrt_graphics_sync_handle_t *out_wake); + //! @p out_ready false = nothing newer than the last acquire. + xrt_result_t (*acquire)(struct xrt_stereo_camera_client *c, + uint64_t stream_id, + bool *out_ready, + struct xrt_stereo_camera_frame_info *out_frame); + xrt_result_t (*stats)(struct xrt_stereo_camera_client *c, + uint64_t stream_id, + struct xrt_stereo_camera_stream_stats *out_stats); +}; + +#ifdef __cplusplus +} +#endif diff --git a/src/xrt/ipc/CMakeLists.txt b/src/xrt/ipc/CMakeLists.txt index 3ae3b601f9..845865f079 100644 --- a/src/xrt/ipc/CMakeLists.txt +++ b/src/xrt/ipc/CMakeLists.txt @@ -70,6 +70,8 @@ add_library( client/ipc_client_connection.c client/ipc_client_lift.c client/ipc_client_lift.h + client/ipc_client_stereo_camera.c + client/ipc_client_stereo_camera.h client/ipc_client_device.c client/ipc_client_hmd.c client/ipc_client_instance.c @@ -121,6 +123,8 @@ add_library( server/ipc_server_peer_creds.c server/ipc_server_peer_creds.h server/ipc_server_process.c + server/ipc_server_stereo_camera.c + server/ipc_server_stereo_camera.h ) target_include_directories( ipc_server diff --git a/src/xrt/ipc/client/ipc_client_instance.c b/src/xrt/ipc/client/ipc_client_instance.c index 223174c19c..9cce353915 100644 --- a/src/xrt/ipc/client/ipc_client_instance.c +++ b/src/xrt/ipc/client/ipc_client_instance.c @@ -30,6 +30,7 @@ #include "client/ipc_client.h" #include "client/ipc_client_interface.h" #include "client/ipc_client_connection.h" +#include "client/ipc_client_stereo_camera.h" #include "ipc_client_generated.h" @@ -229,6 +230,9 @@ ipc_client_instance_destroy(struct xrt_instance *xinst) { struct ipc_client_instance *ii = ipc_client_instance(xinst); + free(ii->base.stereo_camera); + ii->base.stereo_camera = NULL; + // service considers us to be connected until fd is closed ipc_client_connection_fini(&ii->ipc_c); @@ -334,6 +338,9 @@ ipc_instance_create(const struct xrt_instance_info *i_info, struct xrt_instance ii->base.startup_timestamp = ii->ipc_c.ism->startup_timestamp; + // XR_DXR_stereo_camera (ADR-043): cameras are reached over this connection. + ii->base.stereo_camera = ipc_client_stereo_camera_client_create(&ii->ipc_c); + *out_xinst = &ii->base; return XRT_SUCCESS; diff --git a/src/xrt/ipc/client/ipc_client_stereo_camera.c b/src/xrt/ipc/client/ipc_client_stereo_camera.c new file mode 100644 index 0000000000..a87c9d6db8 --- /dev/null +++ b/src/xrt/ipc/client/ipc_client_stereo_camera.c @@ -0,0 +1,266 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera (ADR-043) client-side IPC calls (see + * ipc_client_stereo_camera.h). + * @ingroup ipc_client + */ + +#include "xrt/xrt_config_os.h" + +#include "util/u_misc.h" + +#include "shared/ipc_protocol.h" +#include "shared/ipc_shmem.h" +#include "client/ipc_client.h" +#include "client/ipc_client_connection.h" +#include "client/ipc_client_stereo_camera.h" +#include "ipc_client_generated.h" + +#include +#include + +xrt_result_t +ipc_client_stereo_camera_count(struct ipc_connection *ipc_c, uint32_t *out_count) +{ + *out_count = 0; + if (ipc_c == NULL) { + return XRT_ERROR_IPC_FAILURE; + } + return ipc_call_stereo_camera_count(ipc_c, out_count); +} + +xrt_result_t +ipc_client_stereo_camera_get_properties(struct ipc_connection *ipc_c, + uint32_t index, + struct xrt_stereo_camera_properties *out_props) +{ + U_ZERO(out_props); + if (ipc_c == NULL) { + return XRT_ERROR_IPC_FAILURE; + } + xrt_result_t xret = ipc_call_stereo_camera_get_properties(ipc_c, index, out_props); + out_props->persistent_id[sizeof(out_props->persistent_id) - 1] = '\0'; + out_props->display_name[sizeof(out_props->display_name) - 1] = '\0'; + out_props->platform_device_hint[sizeof(out_props->platform_device_hint) - 1] = '\0'; + return xret; +} + +xrt_result_t +ipc_client_stereo_camera_get_calibration(struct ipc_connection *ipc_c, + uint64_t camera_id, + uint32_t output, + struct xrt_stereo_camera_calibration *out_calib) +{ + U_ZERO(out_calib); + if (ipc_c == NULL) { + return XRT_ERROR_IPC_FAILURE; + } + return ipc_call_stereo_camera_get_calibration(ipc_c, camera_id, output, out_calib); +} + +xrt_result_t +ipc_client_stereo_camera_stream_create(struct ipc_connection *ipc_c, + const struct xrt_stereo_camera_stream_request *req, + uint64_t *out_stream_id) +{ + *out_stream_id = 0; + if (ipc_c == NULL) { + return XRT_ERROR_IPC_FAILURE; + } + return ipc_call_stereo_camera_stream_create(ipc_c, req, out_stream_id); +} + +xrt_result_t +ipc_client_stereo_camera_stream_destroy(struct ipc_connection *ipc_c, uint64_t stream_id) +{ + if (ipc_c == NULL) { + return XRT_ERROR_IPC_FAILURE; + } + return ipc_call_stereo_camera_stream_destroy(ipc_c, stream_id); +} + +xrt_result_t +ipc_client_stereo_camera_stream_start(struct ipc_connection *ipc_c, uint64_t stream_id) +{ + if (ipc_c == NULL) { + return XRT_ERROR_IPC_FAILURE; + } + return ipc_call_stereo_camera_stream_start(ipc_c, stream_id); +} + +xrt_result_t +ipc_client_stereo_camera_stream_stop(struct ipc_connection *ipc_c, uint64_t stream_id) +{ + if (ipc_c == NULL) { + return XRT_ERROR_IPC_FAILURE; + } + return ipc_call_stereo_camera_stream_stop(ipc_c, stream_id); +} + +xrt_result_t +ipc_client_stereo_camera_stream_get_transport(struct ipc_connection *ipc_c, + uint64_t stream_id, + struct xrt_stereo_camera_stream_layout *out_layout, + xrt_shmem_handle_t *out_section, + xrt_graphics_sync_handle_t *out_wake) +{ + U_ZERO(out_layout); + *out_section = XRT_SHMEM_HANDLE_INVALID; + *out_wake = XRT_GRAPHICS_SYNC_HANDLE_INVALID; + if (ipc_c == NULL) { + return XRT_ERROR_IPC_FAILURE; + } + xrt_shmem_handle_t section = XRT_SHMEM_HANDLE_INVALID; + xrt_result_t xret = ipc_call_stereo_camera_stream_get_section(ipc_c, stream_id, out_layout, §ion, 1); + if (xret != XRT_SUCCESS) { + return xret; + } + xrt_graphics_sync_handle_t wake = XRT_GRAPHICS_SYNC_HANDLE_INVALID; + xret = ipc_call_stereo_camera_stream_get_wake(ipc_c, stream_id, &wake, 1); + if (xret != XRT_SUCCESS) { + // The section crossed; do not leak it. + void *none = NULL; + ipc_shmem_destroy(§ion, &none, 0); + return xret; + } + *out_section = section; + *out_wake = wake; + return XRT_SUCCESS; +} + +xrt_result_t +ipc_client_stereo_camera_acquire(struct ipc_connection *ipc_c, + uint64_t stream_id, + bool *out_ready, + struct xrt_stereo_camera_frame_info *out_frame) +{ + *out_ready = false; + U_ZERO(out_frame); + if (ipc_c == NULL) { + return XRT_ERROR_IPC_FAILURE; + } + return ipc_call_stereo_camera_acquire(ipc_c, stream_id, out_ready, out_frame); +} + +xrt_result_t +ipc_client_stereo_camera_stats(struct ipc_connection *ipc_c, + uint64_t stream_id, + struct xrt_stereo_camera_stream_stats *out_stats) +{ + U_ZERO(out_stats); + if (ipc_c == NULL) { + return XRT_ERROR_IPC_FAILURE; + } + return ipc_call_stereo_camera_stream_stats(ipc_c, stream_id, out_stats); +} + + +/* + * + * xrt_stereo_camera_client aspect. + * + */ + +struct ipc_stereo_camera_client +{ + struct xrt_stereo_camera_client base; + struct ipc_connection *ipc_c; +}; + +static inline struct ipc_connection * +conn(struct xrt_stereo_camera_client *c) +{ + return ((struct ipc_stereo_camera_client *)c)->ipc_c; +} + +static xrt_result_t +c_count(struct xrt_stereo_camera_client *c, uint32_t *out_count) +{ + return ipc_client_stereo_camera_count(conn(c), out_count); +} + +static xrt_result_t +c_get_properties(struct xrt_stereo_camera_client *c, uint32_t index, struct xrt_stereo_camera_properties *out) +{ + return ipc_client_stereo_camera_get_properties(conn(c), index, out); +} + +static xrt_result_t +c_get_calibration(struct xrt_stereo_camera_client *c, + uint64_t camera_id, + uint32_t output, + struct xrt_stereo_camera_calibration *out) +{ + return ipc_client_stereo_camera_get_calibration(conn(c), camera_id, output, out); +} + +static xrt_result_t +c_stream_create(struct xrt_stereo_camera_client *c, + const struct xrt_stereo_camera_stream_request *req, + uint64_t *out_stream_id) +{ + return ipc_client_stereo_camera_stream_create(conn(c), req, out_stream_id); +} + +static xrt_result_t +c_stream_destroy(struct xrt_stereo_camera_client *c, uint64_t stream_id) +{ + return ipc_client_stereo_camera_stream_destroy(conn(c), stream_id); +} + +static xrt_result_t +c_stream_start(struct xrt_stereo_camera_client *c, uint64_t stream_id) +{ + return ipc_client_stereo_camera_stream_start(conn(c), stream_id); +} + +static xrt_result_t +c_stream_stop(struct xrt_stereo_camera_client *c, uint64_t stream_id) +{ + return ipc_client_stereo_camera_stream_stop(conn(c), stream_id); +} + +static xrt_result_t +c_stream_get_transport(struct xrt_stereo_camera_client *c, + uint64_t stream_id, + struct xrt_stereo_camera_stream_layout *out_layout, + xrt_shmem_handle_t *out_section, + xrt_graphics_sync_handle_t *out_wake) +{ + return ipc_client_stereo_camera_stream_get_transport(conn(c), stream_id, out_layout, out_section, out_wake); +} + +static xrt_result_t +c_acquire(struct xrt_stereo_camera_client *c, + uint64_t stream_id, + bool *out_ready, + struct xrt_stereo_camera_frame_info *out_frame) +{ + return ipc_client_stereo_camera_acquire(conn(c), stream_id, out_ready, out_frame); +} + +static xrt_result_t +c_stats(struct xrt_stereo_camera_client *c, uint64_t stream_id, struct xrt_stereo_camera_stream_stats *out) +{ + return ipc_client_stereo_camera_stats(conn(c), stream_id, out); +} + +struct xrt_stereo_camera_client * +ipc_client_stereo_camera_client_create(struct ipc_connection *ipc_c) +{ + struct ipc_stereo_camera_client *c = U_TYPED_CALLOC(struct ipc_stereo_camera_client); + c->ipc_c = ipc_c; + c->base.count = c_count; + c->base.get_properties = c_get_properties; + c->base.get_calibration = c_get_calibration; + c->base.stream_create = c_stream_create; + c->base.stream_destroy = c_stream_destroy; + c->base.stream_start = c_stream_start; + c->base.stream_stop = c_stream_stop; + c->base.stream_get_transport = c_stream_get_transport; + c->base.acquire = c_acquire; + c->base.stats = c_stats; + return &c->base; +} diff --git a/src/xrt/ipc/client/ipc_client_stereo_camera.h b/src/xrt/ipc/client/ipc_client_stereo_camera.h new file mode 100644 index 0000000000..ab8bf10789 --- /dev/null +++ b/src/xrt/ipc/client/ipc_client_stereo_camera.h @@ -0,0 +1,86 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera (ADR-043) client side: connection-level calls, + * shared by the IPC instance's @ref xrt_stereo_camera_client aspect + * and `displayxr-cli camera`. + * + * Streams belong to the IPC connection (not a session), so every call takes a + * bare @ref ipc_connection. + * + * @ingroup ipc_client + */ + +#pragma once + +#include "xrt/xrt_handles.h" +#include "xrt/xrt_results.h" +#include "xrt/xrt_stereo_camera.h" + +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +struct ipc_connection; + +xrt_result_t +ipc_client_stereo_camera_count(struct ipc_connection *ipc_c, uint32_t *out_count); + +xrt_result_t +ipc_client_stereo_camera_get_properties(struct ipc_connection *ipc_c, + uint32_t index, + struct xrt_stereo_camera_properties *out_props); + +xrt_result_t +ipc_client_stereo_camera_get_calibration(struct ipc_connection *ipc_c, + uint64_t camera_id, + uint32_t output, + struct xrt_stereo_camera_calibration *out_calib); + +xrt_result_t +ipc_client_stereo_camera_stream_create(struct ipc_connection *ipc_c, + const struct xrt_stereo_camera_stream_request *req, + uint64_t *out_stream_id); + +xrt_result_t +ipc_client_stereo_camera_stream_destroy(struct ipc_connection *ipc_c, uint64_t stream_id); + +xrt_result_t +ipc_client_stereo_camera_stream_start(struct ipc_connection *ipc_c, uint64_t stream_id); + +xrt_result_t +ipc_client_stereo_camera_stream_stop(struct ipc_connection *ipc_c, uint64_t stream_id); + +//! The started stream's ring section + wake handle (NEW handles, caller-owned). +xrt_result_t +ipc_client_stereo_camera_stream_get_transport(struct ipc_connection *ipc_c, + uint64_t stream_id, + struct xrt_stereo_camera_stream_layout *out_layout, + xrt_shmem_handle_t *out_section, + xrt_graphics_sync_handle_t *out_wake); + +xrt_result_t +ipc_client_stereo_camera_acquire(struct ipc_connection *ipc_c, + uint64_t stream_id, + bool *out_ready, + struct xrt_stereo_camera_frame_info *out_frame); + +xrt_result_t +ipc_client_stereo_camera_stats(struct ipc_connection *ipc_c, + uint64_t stream_id, + struct xrt_stereo_camera_stream_stats *out_stats); + +/*! + * The @ref xrt_stereo_camera_client aspect over @p ipc_c (which must outlive + * it). Caller frees with free(). + */ +struct xrt_stereo_camera_client * +ipc_client_stereo_camera_client_create(struct ipc_connection *ipc_c); + +#ifdef __cplusplus +} +#endif diff --git a/src/xrt/ipc/server/ipc_server.h b/src/xrt/ipc/server/ipc_server.h index fa29e255b7..43d3aedaf0 100644 --- a/src/xrt/ipc/server/ipc_server.h +++ b/src/xrt/ipc/server/ipc_server.h @@ -60,6 +60,7 @@ extern "C" { #define IPC_MAX_CLIENT_SPACES 128 struct xrt_instance; +struct ipc_server_stereo_camera; struct xrt_compositor; struct xrt_compositor_native; @@ -218,6 +219,14 @@ struct ipc_client_state * every stream it owns destroyed — at client teardown. */ uint64_t lift_owner; + + /*! + * XR_DXR_stereo_camera (ADR-043): this connection's stream-owner token, + * taken from the camera manager on first stream create (0 = never used a + * camera). A token, not the ics pointer: the thread slot is reused by later + * connections. Every stream it owns is destroyed at client teardown. + */ + uint64_t stereo_camera_owner; }; enum ipc_thread_state @@ -509,6 +518,10 @@ struct ipc_server struct os_mutex lock; } global_state; + + //! XR_DXR_stereo_camera (ADR-043): the camera manager (never NULL once + //! the server is initialised; it may expose zero cameras). + struct ipc_server_stereo_camera *stereo_camera; }; diff --git a/src/xrt/ipc/server/ipc_server_per_client_thread.c b/src/xrt/ipc/server/ipc_server_per_client_thread.c index 9ad3f07849..d0e9459ad0 100644 --- a/src/xrt/ipc/server/ipc_server_per_client_thread.c +++ b/src/xrt/ipc/server/ipc_server_per_client_thread.c @@ -22,6 +22,7 @@ DEBUG_GET_ONCE_NUM_OPTION(test_exit_on_disconnect, "DXR_TEST_EXIT_ON_DISCONNECT" #include "shared/ipc_shmem.h" #include "shared/ipc_utils.h" #include "server/ipc_server.h" +#include "server/ipc_server_stereo_camera.h" #include "ipc_server_generated.h" #include "xrt/xrt_config_have.h" @@ -127,6 +128,9 @@ common_shutdown(volatile struct ipc_client_state *ics) // XR_DXR_lift (ADR-042): streams are owned by the connection, not a session. ipc_server_client_lift_release(ics); + // XR_DXR_stereo_camera (ADR-043): streams are owned by the connection. + ipc_server_client_stereo_camera_release(ics); + // Make sure undestroyed spaces are unreferenced for (uint32_t i = 0; i < IPC_MAX_CLIENT_SPACES; i++) { // Cast away volatile. diff --git a/src/xrt/ipc/server/ipc_server_process.c b/src/xrt/ipc/server/ipc_server_process.c index 726e6fb34d..4e44b32e30 100644 --- a/src/xrt/ipc/server/ipc_server_process.c +++ b/src/xrt/ipc/server/ipc_server_process.c @@ -34,6 +34,7 @@ #include "shared/ipc_protocol.h" #include "shared/ipc_shmem.h" #include "server/ipc_server.h" +#include "server/ipc_server_stereo_camera.h" #include "server/ipc_server_interface.h" #include "server/ipc_server_peer_creds.h" @@ -298,6 +299,9 @@ teardown_all(struct ipc_server *s) { u_var_remove_root(s); + // Before the instance: the camera threads call into the plug-in. + ipc_server_stereo_camera_destroy(&s->stereo_camera); + xrt_syscomp_destroy(&s->xsysc); teardown_idevs(s); @@ -619,6 +623,9 @@ init_all(struct ipc_server *s, enum u_logging_level log_level) xret = xrt_instance_create_system(s->xinst, &s->xsys, &s->xsysd, &s->xso, &s->xsysc); IPC_CHK_WITH_GOTO(s, xret, "xrt_instance_create_system", error); + // XR_DXR_stereo_camera (ADR-043): the service owns every plug-in camera. + s->stereo_camera = ipc_server_stereo_camera_create(s->xinst); + // Always succeeds. init_idevs(s); init_tracking_origins(s); diff --git a/src/xrt/ipc/server/ipc_server_stereo_camera.c b/src/xrt/ipc/server/ipc_server_stereo_camera.c new file mode 100644 index 0000000000..7c343449c1 --- /dev/null +++ b/src/xrt/ipc/server/ipc_server_stereo_camera.c @@ -0,0 +1,1342 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera (ADR-043): the service's camera manager and its + * IPC handlers. See ipc_server_stereo_camera.h. + * + * Locking: ONE manager mutex guards cameras, streams and rings. Plug-in calls + * that may block (open, wait_frame, close) are made with it RELEASED; the + * per-frame fan-out (conversion into each stream's slot) runs under it — a + * 1280x480 NV12 copy is well under a millisecond, and it keeps an acquire from + * ever observing a half-written slot. Only the camera thread closes a source. + * + * Privacy (spec §7) — R1 ships the HOOK POINTS, deny-by-default: + * - authorise_locked(): every start and every calibration read. Allowed in R1 + * only with DXR_STEREO_CAMERA_DEV_ALLOW=1 in the SERVICE environment; + * otherwise XRT_ERROR_NOT_AUTHORIZED (XR_ERROR_PERMISSION_INSUFFICIENT). + * R3 replaces the env check with OS consent + the DisplayXR consent store + + * delegating-client registration. + * - client_visible_locked(): the foreground rule, evaluated per published + * frame. R1 always true; R3 wires session visibility / OS lock. + * - output_allowed(): RAW is refused to PRESENT_OWNER clients (the browser, + * which exposes cameras to web pages) — maintainer decision for R1. + * - DXR_STEREO_CAMERA=0: kill switch, zero cameras enumerated. + * - the in-use indicator (tray badge naming consumers) is R3; R1 logs one + * WARN per stream start/stop naming the peer executable. + * + * Transports — R1 implements SHARED_MEMORY only. The GPU transports are + * designed to reuse this ring's state machine unchanged, only the slot storage + * differs: + * - D3D11_TEXTURE (Windows): TODO(L1/B1). Three D3D11 textures on the + * service's lift/weave device created SHARED_NTHANDLE|SHARED_KEYEDMUTEX-free + * with one ID3D11Fence; the publish step becomes an UpdateSubresource into + * the free slot + Signal(fence, seq); acquire returns (slot, fence value); + * the NT handles + fence are handed out on the first acquire and on + * reallocation, the XR_DXR_weave output-export pattern (legacy DXGI handles + * low-bit tagged for Low-IL callers). The browser's capture device imports + * them and waits on the fence instead of reading the section. + * - Vulkan / AHARDWAREBUFFER (Android, and desktop Linux via dma-buf): TODO(A1). + * Three AHardwareBuffers (or VkImages exported as dma-buf / opaque fd) per + * stream, sent once over the IPC socket like weave's dma-buf outputs + * (#1699), with a sync_file per publish instead of a fence value. NV12 per + * eye maps to AHARDWAREBUFFER_FORMAT_Y8Cb8Cr8_420; the tablet's per-lens + * 1280x720 NV12 pair makes this transport the default there. + * + * @ingroup ipc_server + */ + +#include "server/ipc_server.h" +#include "server/ipc_server_peer_creds.h" +#include "server/ipc_server_stereo_camera.h" +#include "ipc_server_generated.h" + +#include "xrt/xrt_plugin.h" +#include "xrt/xrt_stereo_camera.h" + +#include "os/os_threading.h" +#include "os/os_time.h" +#include "shared/ipc_shmem.h" +#include "util/u_debug.h" +#include "util/u_logging.h" +#include "util/u_misc.h" +#include "util/u_stereo_camera.h" + +#include +#include +#include + +#ifdef XRT_OS_WINDOWS +#include +#else +#include +#include +#include +#endif + +DEBUG_GET_ONCE_BOOL_OPTION(stereo_camera_enabled, "DXR_STEREO_CAMERA", true) +DEBUG_GET_ONCE_BOOL_OPTION(stereo_camera_dev_allow, "DXR_STEREO_CAMERA_DEV_ALLOW", false) + +#define MAX_STREAMS (XRT_STEREO_CAMERA_MAX_CAMERAS * XRT_STEREO_CAMERA_MAX_STREAMS_PER_CAMERA) +#define LINGER_NS (2000ll * 1000 * 1000) +#define WAIT_TIMEOUT_NS (100ll * 1000 * 1000) +#define SUSPEND_AFTER_NS (1000ll * 1000 * 1000) +#define REOPEN_BACKOFF_NS (1000ll * 1000 * 1000) +#define STATS_LOG_NS (5000ll * 1000 * 1000) + + +/* + * + * Wake handle: an auto-reset event (Windows) / a non-blocking pipe (POSIX). + * + */ + +struct scam_wake +{ +#ifdef XRT_OS_WINDOWS + HANDLE event; +#else + int fds[2]; //!< [0] = read end (sent to the client), [1] = write end +#endif +}; + +static bool +wake_create(struct scam_wake *w) +{ +#ifdef XRT_OS_WINDOWS + w->event = CreateEventW(NULL, FALSE, FALSE, NULL); + return w->event != NULL; +#else + w->fds[0] = w->fds[1] = -1; + if (pipe(w->fds) != 0) { + return false; + } + for (int i = 0; i < 2; i++) { + int fl = fcntl(w->fds[i], F_GETFL, 0); + fcntl(w->fds[i], F_SETFL, fl | O_NONBLOCK); + fcntl(w->fds[i], F_SETFD, FD_CLOEXEC); + } + return true; +#endif +} + +static void +wake_signal(struct scam_wake *w) +{ +#ifdef XRT_OS_WINDOWS + if (w->event != NULL) { + SetEvent(w->event); + } +#else + if (w->fds[1] >= 0) { + const char b = 1; + // EAGAIN = the consumer is not draining: it already has a pending wake. + ssize_t r = write(w->fds[1], &b, 1); + (void)r; + } +#endif +} + +static void +wake_destroy(struct scam_wake *w) +{ +#ifdef XRT_OS_WINDOWS + if (w->event != NULL) { + CloseHandle(w->event); + w->event = NULL; + } +#else + for (int i = 0; i < 2; i++) { + if (w->fds[i] >= 0) { + close(w->fds[i]); + w->fds[i] = -1; + } + } +#endif +} + + +/* + * + * Structs. + * + */ + +struct scam_stream +{ + bool used; + uint64_t id; + uint64_t owner; //!< ics->stereo_camera_owner of the creating connection + volatile struct ipc_client_state *ics; + uint32_t camera; //!< index into mgr->cams + struct xrt_stereo_camera_stream_request req; + uint32_t output; //!< what frames actually are (RAW when rectification is unavailable) + char consumer[260]; + + bool started; + bool allocated; + uint32_t width, height; + struct u_stereo_camera_planes layout; + uint64_t slot_stride; + uint64_t section_size; + xrt_shmem_handle_t section; + uint8_t *map; + struct scam_wake wake; + struct u_stereo_camera_ring ring; + struct u_stereo_camera_decimator dec; + struct xrt_stereo_camera_frame_info slot_info[U_STEREO_CAMERA_RING_SLOTS]; + +#ifdef XRT_OS_WINDOWS + //! Restricted duplicates handed to the last get_section / get_wake reply. + //! The generated dispatch duplicates them into the peer AFTER the handler + //! returns, so they are closed on the next call / stream destroy. + HANDLE sent_section_dup; + HANDLE sent_wake_dup; +#endif + + // stats + float delivered_rate; + int64_t last_publish_ns; + uint64_t latency_sum_ns; + uint64_t latency_count; +}; + +struct scam_camera +{ + uint32_t index; //!< plug-in index + uint64_t camera_id; + struct xrt_plugin_stereo_camera_info info; + bool have_calib; + struct xrt_plugin_stereo_camera_calibration calib; + uint32_t state; + uint32_t calibration_generation; + + struct xrt_plugin_stereo_camera *src; //!< owned by the camera thread + uint32_t started_count; + int64_t linger_deadline_ns; + int64_t last_frame_ns; + int64_t retry_after_ns; + float source_rate; + //! Measured over the first frames after each open; what the descriptor + //! reports once known (the plug-in's enumerate-time rate is a guess, and + //! was 2x wrong on the Leia SR tracking camera). + struct u_stereo_camera_rate_meter rate_meter; + bool rate_logged; + int64_t last_stats_log_ns; + + bool thread_started; + struct os_thread thread; + struct os_cond cond; + struct ipc_server_stereo_camera *mgr; +}; + +struct ipc_server_stereo_camera +{ + struct os_mutex lock; + bool shutting_down; + const struct xrt_plugin_iface *iface; + struct xrt_plugin_instance *inst; + uint32_t camera_count; + struct scam_camera cams[XRT_STEREO_CAMERA_MAX_CAMERAS]; + struct scam_stream streams[MAX_STREAMS]; + uint64_t next_stream_id; + uint64_t next_owner; +}; + + +/* + * + * Helpers. + * + */ + +static const char * +state_str(uint32_t s) +{ + switch (s) { + case XRT_STEREO_CAMERA_STATE_AVAILABLE: return "AVAILABLE"; + case XRT_STEREO_CAMERA_STATE_WAITING: return "WAITING"; + case XRT_STEREO_CAMERA_STATE_SUSPENDED: return "SUSPENDED"; + case XRT_STEREO_CAMERA_STATE_UNAVAILABLE: return "UNAVAILABLE"; + default: return "?"; + } +} + +static struct ipc_server_stereo_camera * +mgr_of(volatile struct ipc_client_state *ics) +{ + return (ics != NULL && ics->server != NULL) ? ics->server->stereo_camera : NULL; +} + +static uint64_t +owner_of(struct ipc_server_stereo_camera *m, volatile struct ipc_client_state *ics) +{ + if (ics->stereo_camera_owner == 0) { + ics->stereo_camera_owner = ++m->next_owner; + } + return ics->stereo_camera_owner; +} + +static struct scam_camera * +camera_by_id_locked(struct ipc_server_stereo_camera *m, uint64_t id) +{ + for (uint32_t i = 0; i < m->camera_count; i++) { + if (m->cams[i].camera_id == id) { + return &m->cams[i]; + } + } + return NULL; +} + +static struct scam_stream * +stream_by_id_locked(struct ipc_server_stereo_camera *m, volatile struct ipc_client_state *ics, uint64_t id) +{ + if (ics->stereo_camera_owner == 0) { + return NULL; + } + for (uint32_t i = 0; i < MAX_STREAMS; i++) { + struct scam_stream *s = &m->streams[i]; + if (s->used && s->id == id && s->owner == ics->stereo_camera_owner) { + return s; + } + } + return NULL; +} + +//! Who may use cameras at all: apps, the browser, and the diagnostic CLI. +static xrt_result_t +require_camera_client(volatile struct ipc_client_state *ics, const char *what) +{ + uint32_t cls = ics->client_state.client_class; + if (cls != XRT_CLIENT_CLASS_APP && cls != XRT_CLIENT_CLASS_PRESENT_OWNER && cls != XRT_CLIENT_CLASS_DIAG) { + U_LOG_W("%s: denied — pid %ld is class %s (stereo camera: APP / PRESENT_OWNER / DIAG).", what, + ics->peer_pid, ipc_server_client_class_str(cls)); + return XRT_ERROR_NOT_AUTHORIZED; + } + return XRT_SUCCESS; +} + +/*! + * Spec §7.1 authorisation HOOK. R1: deny by default; the dev override is the + * only way in. R3 replaces the body: OS camera consent of the OS-derived peer + * (Windows CapabilityAccessManager ConsentStore\webcam, Android CAMERA of the + * peer uid), then the DisplayXR per-executable consent (HKCU allow list / tray + * prompt; a registered consent-delegating browser passes per executable and + * prompts per origin itself). + */ +static xrt_result_t +authorise_locked(volatile struct ipc_client_state *ics, const struct scam_camera *cam, const char *what) +{ + (void)cam; + if (debug_get_bool_option_stereo_camera_dev_allow()) { + return XRT_SUCCESS; + } + U_LOG_W( + "%s: refused for pid %ld — stereo camera consent is not implemented yet (R3); set " + "DXR_STEREO_CAMERA_DEV_ALLOW=1 in the SERVICE environment for development.", + what, ics->peer_pid); + return XRT_ERROR_NOT_AUTHORIZED; +} + +/*! + * Spec §7.2 foreground-rule HOOK, evaluated per published frame. R1: always + * visible. R3: an app's session must be VISIBLE/FOCUSED; a delegating client + * follows its own rule; every stream is suspended while the OS session is + * locked / switched away (Android: android_package_is_visible of the peer). + */ +static bool +client_visible_locked(volatile struct ipc_client_state *ics) +{ + (void)ics; + return true; +} + +//! Maintainer decision (R1): raw frames never reach web pages — the browser +//! (PRESENT_OWNER) gets RECTIFIED only; native clients may ask for RAW. +static bool +output_allowed(volatile struct ipc_client_state *ics, uint32_t output) +{ + return output != XRT_STEREO_CAMERA_OUTPUT_RAW || + ics->client_state.client_class != XRT_CLIENT_CLASS_PRESENT_OWNER; +} + +static bool +camera_can_rectify(const struct scam_camera *cam) +{ + // R1 has no rectifier (u_stereo_rectify is R2): only a natively rectified + // source yields RECTIFIED frames; a CALIBRATED one is delivered RAW-flagged. + return (cam->info.flags & XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED) != 0; +} + +static void +set_state_locked(struct scam_camera *cam, uint32_t state) +{ + if (cam->state == state) { + return; + } + U_LOG_W("stereo camera %llu \"%s\": %s -> %s", (unsigned long long)cam->camera_id, cam->info.display_name, + state_str(cam->state), state_str(state)); + cam->state = state; + if (state == XRT_STEREO_CAMERA_STATE_SUSPENDED || state == XRT_STEREO_CAMERA_STATE_UNAVAILABLE) { + // Spec §7.2: the last image must not linger for a suspended consumer. + struct ipc_server_stereo_camera *m = cam->mgr; + for (uint32_t i = 0; i < MAX_STREAMS; i++) { + struct scam_stream *s = &m->streams[i]; + if (s->used && s->camera == cam->index) { + u_stereo_camera_ring_clear(&s->ring); + } + } + } + // TODO(R1 follow-up): queue XrEventDataStereoCameraStateChangedDXR to the + // owning connections (needs a server->client instance-event channel). +} + +static void +mat3_to_quat(const double r[3][3], float out[4]) +{ + double tr = r[0][0] + r[1][1] + r[2][2]; + double x, y, z, w; + if (tr > 0.0) { + double s = sqrt(tr + 1.0) * 2.0; + w = 0.25 * s; + x = (r[2][1] - r[1][2]) / s; + y = (r[0][2] - r[2][0]) / s; + z = (r[1][0] - r[0][1]) / s; + } else if (r[0][0] > r[1][1] && r[0][0] > r[2][2]) { + double s = sqrt(1.0 + r[0][0] - r[1][1] - r[2][2]) * 2.0; + w = (r[2][1] - r[1][2]) / s; + x = 0.25 * s; + y = (r[0][1] + r[1][0]) / s; + z = (r[0][2] + r[2][0]) / s; + } else if (r[1][1] > r[2][2]) { + double s = sqrt(1.0 + r[1][1] - r[0][0] - r[2][2]) * 2.0; + w = (r[0][2] - r[2][0]) / s; + x = (r[0][1] + r[1][0]) / s; + y = 0.25 * s; + z = (r[1][2] + r[2][1]) / s; + } else { + double s = sqrt(1.0 + r[2][2] - r[0][0] - r[1][1]) * 2.0; + w = (r[1][0] - r[0][1]) / s; + x = (r[0][2] + r[2][0]) / s; + y = (r[1][2] + r[2][1]) / s; + z = 0.25 * s; + } + out[0] = (float)x; + out[1] = (float)y; + out[2] = (float)z; + out[3] = (float)w; +} + +static double +baseline_mm(const struct xrt_plugin_stereo_camera_calibration *c) +{ + const double *t = c->translation_right_from_left_mm; + return sqrt(t[0] * t[0] + t[1] * t[1] + t[2] * t[2]); +} + + +/* + * + * Stream storage. + * + */ + +static void +stream_free_storage(struct scam_stream *s) +{ + if (s->map != NULL || s->section != XRT_SHMEM_HANDLE_INVALID) { + void *map = s->map; + ipc_shmem_destroy(&s->section, &map, (size_t)s->section_size); + s->map = NULL; + } + wake_destroy(&s->wake); +#ifdef XRT_OS_WINDOWS + if (s->sent_section_dup != NULL) { + CloseHandle(s->sent_section_dup); + s->sent_section_dup = NULL; + } + if (s->sent_wake_dup != NULL) { + CloseHandle(s->sent_wake_dup); + s->sent_wake_dup = NULL; + } +#endif + s->allocated = false; +} + +static xrt_result_t +stream_allocate_locked(struct scam_stream *s, const struct scam_camera *cam) +{ + if (s->allocated) { + return XRT_SUCCESS; + } + s->width = 2 * cam->info.eye_width; + s->height = cam->info.eye_height; + if (!u_stereo_camera_layout(s->req.format, s->width, s->height, &s->layout)) { + return XRT_ERROR_ALLOCATION; + } + s->slot_stride = (s->layout.size + 4095u) & ~(uint64_t)4095u; + s->section_size = s->slot_stride * U_STEREO_CAMERA_RING_SLOTS; + void *map = NULL; + s->section = XRT_SHMEM_HANDLE_INVALID; + xrt_result_t xret = ipc_shmem_create((size_t)s->section_size, &s->section, &map); + if (xret != XRT_SUCCESS) { + return xret; + } + s->map = map; + memset(s->map, 0, (size_t)s->section_size); + // TODO(R3): hand POSIX consumers a read-only fd (Linux: reopen + // /proc/self/fd/N O_RDONLY; macOS: shm_open the name O_RDONLY before unlink; + // Android: an ASharedMemory with PROT_READ set via ASharedMemory_setProt). + if (!wake_create(&s->wake)) { + stream_free_storage(s); + return XRT_ERROR_ALLOCATION; + } + u_stereo_camera_ring_init(&s->ring); + s->allocated = true; + return XRT_SUCCESS; +} + + +//! The descriptor's rate: measured once known, else the plug-in's (0 = unknown). +static float +source_rate_hz(const struct scam_camera *cam) +{ + return cam->rate_meter.rate > 0.0f ? cam->rate_meter.rate : cam->info.max_frame_rate; +} + + +/* + * + * Camera thread. + * + */ + +static void +publish_locked(struct ipc_server_stereo_camera *m, + struct scam_camera *cam, + const struct xrt_plugin_stereo_camera_frame *f, + int64_t now) +{ + if (cam->last_frame_ns > 0) { + double dt = (double)(now - cam->last_frame_ns) * 1e-9; + if (dt > 0.0) { + float inst = (float)(1.0 / dt); + cam->source_rate = cam->source_rate <= 0.0f ? inst : cam->source_rate * 0.9f + inst * 0.1f; + } + } + cam->last_frame_ns = now; + set_state_locked(cam, XRT_STEREO_CAMERA_STATE_AVAILABLE); + + if (u_stereo_camera_rate_meter_push(&cam->rate_meter, f->time_ns > 0 ? f->time_ns : now)) { + float adv = cam->info.max_frame_rate; + bool off = adv <= 0.0f || fabsf(cam->rate_meter.rate - adv) > 0.1f * adv; + if (!cam->rate_logged) { + cam->rate_logged = true; + U_LOG_W( + "stereo camera %llu: measured source rate %.1f Hz over %d frames (plug-in advertised %.1f " + "Hz%s)%s", + (unsigned long long)cam->camera_id, cam->rate_meter.rate, U_STEREO_CAMERA_RATE_WINDOW, adv, + adv > 0.0f ? "" : " = unknown", + off ? " — the descriptor now reports the measured rate" : ""); + } else { + U_LOG_I("stereo camera %llu: measured source rate %.1f Hz", (unsigned long long)cam->camera_id, + cam->rate_meter.rate); + } + } + + if (f->width != 2 * cam->info.eye_width || f->height != cam->info.eye_height) { + U_LOG_W("stereo camera %llu: frame %ux%u does not match the advertised %ux%u SBS — dropped", + (unsigned long long)cam->camera_id, f->width, f->height, 2 * cam->info.eye_width, + cam->info.eye_height); + return; + } + + for (uint32_t i = 0; i < MAX_STREAMS; i++) { + struct scam_stream *s = &m->streams[i]; + if (!s->used || !s->started || !s->allocated || s->camera != cam->index) { + continue; + } + if (!client_visible_locked(s->ics)) { + u_stereo_camera_ring_clear(&s->ring); + continue; + } + if (!u_stereo_camera_decimator_accept(&s->dec, f->time_ns)) { + continue; + } + int32_t slot = u_stereo_camera_ring_begin_write(&s->ring); + if (slot < 0) { + continue; + } + uint8_t *dst = s->map + (uint64_t)slot * s->slot_stride; + if (!u_stereo_camera_convert(f->format, f->planes, f->pitches, s->req.format, dst, &s->layout, f->width, + f->height)) { + u_stereo_camera_ring_abort_write(&s->ring); + continue; + } + struct xrt_stereo_camera_frame_info *fi = &s->slot_info[slot]; + U_ZERO(fi); + fi->frame_index = f->sequence; + fi->slot = (uint32_t)slot; + fi->width = f->width; + fi->height = f->height; + fi->format = s->req.format; + fi->row_pitch[0] = s->layout.pitch[0]; + fi->row_pitch[1] = s->layout.pitch[1]; + fi->plane_offset[0] = s->layout.offset[0]; + fi->plane_offset[1] = s->layout.offset[1]; + fi->capture_time_ns = f->time_ns; + fi->time_is_exposure = f->time_is_exposure ? 1u : 0u; + fi->output = s->output; + fi->calibration_generation = cam->calibration_generation; + u_stereo_camera_ring_publish(&s->ring, slot, f->sequence); + + int64_t pub = os_monotonic_get_ns(); + if (pub > f->time_ns) { + s->latency_sum_ns += (uint64_t)(pub - f->time_ns); + s->latency_count++; + } + if (s->last_publish_ns > 0 && pub > s->last_publish_ns) { + float inst = (float)(1e9 / (double)(pub - s->last_publish_ns)); + s->delivered_rate = s->delivered_rate <= 0.0f ? inst : s->delivered_rate * 0.9f + inst * 0.1f; + } + s->last_publish_ns = pub; + wake_signal(&s->wake); + } + + if (now - cam->last_stats_log_ns >= STATS_LOG_NS) { + cam->last_stats_log_ns = now; + U_LOG_I("stereo camera %llu: source %.1f Hz, %u started stream(s)", (unsigned long long)cam->camera_id, + cam->source_rate, cam->started_count); + } +} + +static void * +camera_thread(void *ptr) +{ + struct scam_camera *cam = (struct scam_camera *)ptr; + struct ipc_server_stereo_camera *m = cam->mgr; + const struct xrt_plugin_iface *iface = m->iface; + + os_mutex_lock(&m->lock); + while (!m->shutting_down) { + int64_t now = os_monotonic_get_ns(); + + // Idle: close the source once the linger expires, then sleep until a + // stream starts (or the manager shuts down). + if (cam->started_count == 0 && (cam->src == NULL || now >= cam->linger_deadline_ns)) { + if (cam->src != NULL) { + struct xrt_plugin_stereo_camera *src = cam->src; + cam->src = NULL; + os_mutex_unlock(&m->lock); + iface->stereo_camera_close(src); + os_mutex_lock(&m->lock); + U_LOG_W("stereo camera %llu: source closed (no started stream for %lld ms)", + (unsigned long long)cam->camera_id, (long long)(LINGER_NS / 1000000)); + cam->last_frame_ns = 0; + cam->source_rate = 0.0f; + if (cam->state != XRT_STEREO_CAMERA_STATE_UNAVAILABLE) { + cam->state = XRT_STEREO_CAMERA_STATE_AVAILABLE; // idle: "will flow on start" + } + continue; + } + os_cond_wait(&cam->cond, &m->lock); + continue; + } + + if (cam->src == NULL) { + if (now < cam->retry_after_ns) { + os_cond_wait_timeout_ns(&cam->cond, &m->lock, (uint64_t)(cam->retry_after_ns - now)); + continue; + } + struct xrt_plugin_stereo_camera *src = NULL; + os_mutex_unlock(&m->lock); + xrt_result_t xret = iface->stereo_camera_open(m->inst, cam->index, &src); + os_mutex_lock(&m->lock); + if (xret != XRT_SUCCESS || src == NULL) { + U_LOG_W("stereo camera %llu: plug-in open failed (%d); retrying in 1 s", + (unsigned long long)cam->camera_id, (int)xret); + set_state_locked(cam, XRT_STEREO_CAMERA_STATE_UNAVAILABLE); + cam->retry_after_ns = os_monotonic_get_ns() + REOPEN_BACKOFF_NS; + continue; + } + if (m->shutting_down) { + os_mutex_unlock(&m->lock); + iface->stereo_camera_close(src); + os_mutex_lock(&m->lock); + break; + } + cam->src = src; + cam->last_frame_ns = 0; + u_stereo_camera_rate_meter_restart(&cam->rate_meter); + U_LOG_W("stereo camera %llu: source opened", (unsigned long long)cam->camera_id); + set_state_locked(cam, XRT_STEREO_CAMERA_STATE_WAITING); + } + + struct xrt_plugin_stereo_camera *src = cam->src; + struct xrt_plugin_stereo_camera_frame f; + memset(&f, 0, sizeof(f)); + os_mutex_unlock(&m->lock); + uint32_t w = iface->stereo_camera_wait_frame(src, WAIT_TIMEOUT_NS, &f); + os_mutex_lock(&m->lock); + now = os_monotonic_get_ns(); + + switch (w) { + case XRT_PLUGIN_STEREO_CAMERA_WAIT_OK: + publish_locked(m, cam, &f, now); + iface->stereo_camera_release_frame(src); + break; + case XRT_PLUGIN_STEREO_CAMERA_WAIT_TIMEOUT: + if (cam->last_frame_ns == 0) { + set_state_locked(cam, XRT_STEREO_CAMERA_STATE_WAITING); + } else if (now - cam->last_frame_ns > SUSPEND_AFTER_NS) { + set_state_locked(cam, XRT_STEREO_CAMERA_STATE_SUSPENDED); + } + break; + case XRT_PLUGIN_STEREO_CAMERA_WAIT_SUSPENDED: + set_state_locked(cam, XRT_STEREO_CAMERA_STATE_SUSPENDED); + cam->last_frame_ns = 0; + break; + default: + U_LOG_W("stereo camera %llu: plug-in reported a source error; closing, retry in 1 s", + (unsigned long long)cam->camera_id); + set_state_locked(cam, XRT_STEREO_CAMERA_STATE_UNAVAILABLE); + cam->src = NULL; + os_mutex_unlock(&m->lock); + iface->stereo_camera_close(src); + os_mutex_lock(&m->lock); + cam->retry_after_ns = os_monotonic_get_ns() + REOPEN_BACKOFF_NS; + break; + } + } + if (cam->src != NULL) { + struct xrt_plugin_stereo_camera *src = cam->src; + cam->src = NULL; + os_mutex_unlock(&m->lock); + iface->stereo_camera_close(src); + os_mutex_lock(&m->lock); + } + os_mutex_unlock(&m->lock); + return NULL; +} + + +/* + * + * Manager lifecycle. + * + */ + +struct ipc_server_stereo_camera * +ipc_server_stereo_camera_create(struct xrt_instance *xinst) +{ + struct ipc_server_stereo_camera *m = U_TYPED_CALLOC(struct ipc_server_stereo_camera); + os_mutex_init(&m->lock); + for (uint32_t i = 0; i < MAX_STREAMS; i++) { + m->streams[i].section = XRT_SHMEM_HANDLE_INVALID; + } + + if (!debug_get_bool_option_stereo_camera_enabled()) { + U_LOG_W("stereo camera: disabled by DXR_STEREO_CAMERA=0 (kill switch) — zero cameras"); + return m; + } + const struct xrt_plugin_iface *iface = NULL; + struct xrt_plugin_instance *inst = NULL; + if (xinst == NULL || xinst->get_active_plugin == NULL || !xinst->get_active_plugin(xinst, &iface, &inst) || + !xrt_plugin_iface_has_stereo_camera(iface)) { + U_LOG_I("stereo camera: the active plug-in provides no camera source"); + return m; + } + m->iface = iface; + m->inst = inst; + + struct xrt_plugin_stereo_camera_info infos[XRT_STEREO_CAMERA_MAX_CAMERAS]; + memset(infos, 0, sizeof(infos)); + for (uint32_t i = 0; i < XRT_STEREO_CAMERA_MAX_CAMERAS; i++) { + infos[i].struct_size = (uint32_t)sizeof(infos[i]); + } + uint32_t n = iface->stereo_camera_enumerate(inst, XRT_STEREO_CAMERA_MAX_CAMERAS, infos); + if (n > XRT_STEREO_CAMERA_MAX_CAMERAS) { + n = XRT_STEREO_CAMERA_MAX_CAMERAS; + } + for (uint32_t i = 0; i < n; i++) { + struct scam_camera *cam = &m->cams[m->camera_count]; + struct xrt_plugin_stereo_camera_info *info = &infos[i]; + info->display_name[sizeof(info->display_name) - 1] = '\0'; + info->device_identity[sizeof(info->device_identity) - 1] = '\0'; + info->platform_device_hint[sizeof(info->platform_device_hint) - 1] = '\0'; + if (info->eye_width == 0 || info->eye_height == 0 || (info->eye_width & 1u) || + (info->eye_height & 1u) || info->native_format < XRT_PLUGIN_STEREO_CAMERA_FORMAT_GRAY8 || + info->native_format > XRT_PLUGIN_STEREO_CAMERA_FORMAT_BGRA8) { + U_LOG_W( + "stereo camera: plug-in camera %u has a malformed description (%ux%u, format %u) — " + "skipped", + i, info->eye_width, info->eye_height, info->native_format); + continue; + } + cam->index = i; + cam->camera_id = (uint64_t)m->camera_count + 1; // never reused in this service lifetime + cam->info = *info; + cam->mgr = m; + cam->state = XRT_STEREO_CAMERA_STATE_AVAILABLE; + os_cond_init(&cam->cond); + if (info->flags & XRT_PLUGIN_STEREO_CAMERA_CALIBRATED) { + cam->calib.struct_size = (uint32_t)sizeof(cam->calib); + if (iface->stereo_camera_get_calibration(inst, i, &cam->calib) == XRT_SUCCESS) { + cam->have_calib = true; + } else { + U_LOG_W( + "stereo camera: \"%s\" claims CALIBRATED but returned no calibration — RAW only", + info->display_name); + cam->info.flags &= ~XRT_PLUGIN_STEREO_CAMERA_CALIBRATED; + } + } + if (!(info->max_frame_rate >= 0.0f && info->max_frame_rate <= 1000.0f)) { + cam->info.max_frame_rate = 0.0f; // NaN / negative / absurd = unknown + } + char rate[48]; + if (cam->info.max_frame_rate > 0.0f) { + snprintf(rate, sizeof(rate), "advertised %.1f Hz", cam->info.max_frame_rate); + } else { + snprintf(rate, sizeof(rate), "rate unknown"); + } + U_LOG_W( + "stereo camera %llu: \"%s\" %ux%u per eye, %s (measured on first open), flags 0x%x, native " + "format %u", + (unsigned long long)cam->camera_id, info->display_name, info->eye_width, info->eye_height, rate, + info->flags, info->native_format); + m->camera_count++; + } + return m; +} + +void +ipc_server_stereo_camera_destroy(struct ipc_server_stereo_camera **mgr_ptr) +{ + struct ipc_server_stereo_camera *m = *mgr_ptr; + if (m == NULL) { + return; + } + os_mutex_lock(&m->lock); + m->shutting_down = true; + for (uint32_t i = 0; i < m->camera_count; i++) { + os_cond_broadcast(&m->cams[i].cond); + } + os_mutex_unlock(&m->lock); + for (uint32_t i = 0; i < m->camera_count; i++) { + if (m->cams[i].thread_started) { + os_thread_join(&m->cams[i].thread); + os_thread_destroy(&m->cams[i].thread); + } + os_cond_destroy(&m->cams[i].cond); + } + for (uint32_t i = 0; i < MAX_STREAMS; i++) { + if (m->streams[i].used) { + stream_free_storage(&m->streams[i]); + } + } + os_mutex_destroy(&m->lock); + free(m); + *mgr_ptr = NULL; +} + +static void +stream_stop_locked(struct scam_stream *s, struct scam_camera *cam) +{ + if (!s->started) { + return; + } + s->started = false; + u_stereo_camera_ring_clear(&s->ring); + if (cam->started_count > 0) { + cam->started_count--; + } + if (cam->started_count == 0) { + cam->linger_deadline_ns = os_monotonic_get_ns() + LINGER_NS; + } + U_LOG_W("stereo camera %llu: stream %llu stopped (%s); %u stream(s) still started", + (unsigned long long)cam->camera_id, (unsigned long long)s->id, s->consumer, cam->started_count); +} + +static void +stream_destroy_locked(struct ipc_server_stereo_camera *m, struct scam_stream *s) +{ + stream_stop_locked(s, &m->cams[s->camera]); + stream_free_storage(s); + memset(s, 0, sizeof(*s)); + s->section = XRT_SHMEM_HANDLE_INVALID; +#ifndef XRT_OS_WINDOWS + s->wake.fds[0] = s->wake.fds[1] = -1; +#endif +} + +void +ipc_server_client_stereo_camera_release(volatile struct ipc_client_state *ics) +{ + struct ipc_server_stereo_camera *m = mgr_of(ics); + if (m == NULL || ics->stereo_camera_owner == 0) { + return; + } + os_mutex_lock(&m->lock); + for (uint32_t i = 0; i < MAX_STREAMS; i++) { + struct scam_stream *s = &m->streams[i]; + if (s->used && s->owner == ics->stereo_camera_owner) { + stream_destroy_locked(m, s); + } + } + os_mutex_unlock(&m->lock); + ics->stereo_camera_owner = 0; +} + + +/* + * + * IPC handlers. + * + */ + +xrt_result_t +ipc_handle_stereo_camera_count(volatile struct ipc_client_state *ics, uint32_t *out_count) +{ + *out_count = 0; + struct ipc_server_stereo_camera *m = mgr_of(ics); + if (require_camera_client(ics, "stereo_camera_count") != XRT_SUCCESS || m == NULL) { + return XRT_SUCCESS; // zero cameras, indistinguishable from "none" on purpose + } + os_mutex_lock(&m->lock); + *out_count = m->camera_count; + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; +} + +xrt_result_t +ipc_handle_stereo_camera_get_properties(volatile struct ipc_client_state *ics, + uint32_t index, + struct xrt_stereo_camera_properties *out_props) +{ + U_ZERO(out_props); + struct ipc_server_stereo_camera *m = mgr_of(ics); + xrt_result_t auth = require_camera_client(ics, "stereo_camera_get_properties"); + if (auth != XRT_SUCCESS) { + return auth; + } + if (m == NULL) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + char exe[260] = {0}; + ipc_server_peer_exe_path(ics->peer_pid, exe, sizeof(exe)); + + os_mutex_lock(&m->lock); + if (index >= m->camera_count) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + const struct scam_camera *cam = &m->cams[index]; + out_props->camera_id = cam->camera_id; + u_stereo_camera_persistent_id(cam->info.device_identity, exe, out_props->persistent_id); + snprintf(out_props->display_name, sizeof(out_props->display_name), "%s", cam->info.display_name); + snprintf(out_props->platform_device_hint, sizeof(out_props->platform_device_hint), "%s", + cam->info.platform_device_hint); + out_props->flags = cam->info.flags; + out_props->state = cam->state; + out_props->view_count = 2; + out_props->eye_width = cam->info.eye_width; + out_props->eye_height = cam->info.eye_height; + out_props->max_frame_rate = source_rate_hz(cam); + if (cam->have_calib) { + out_props->baseline_mm = (float)baseline_mm(&cam->calib); + double fx = cam->calib.k[0][0]; + if (fx > 0.0) { + out_props->horizontal_fov_deg = + (float)(2.0 * atan(cam->info.eye_width / (2.0 * fx)) * 180.0 / 3.14159265358979323846); + } + } + out_props->supported_formats = XRT_STEREO_CAMERA_BIT(XRT_STEREO_CAMERA_FORMAT_GRAY8) | + XRT_STEREO_CAMERA_BIT(XRT_STEREO_CAMERA_FORMAT_NV12) | + XRT_STEREO_CAMERA_BIT(XRT_STEREO_CAMERA_FORMAT_BGRA8); + out_props->supported_transports = XRT_STEREO_CAMERA_BIT(XRT_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY); + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; +} + +xrt_result_t +ipc_handle_stereo_camera_get_calibration(volatile struct ipc_client_state *ics, + uint64_t camera_id, + uint32_t output, + struct xrt_stereo_camera_calibration *out_calib) +{ + U_ZERO(out_calib); + struct ipc_server_stereo_camera *m = mgr_of(ics); + xrt_result_t auth = require_camera_client(ics, "stereo_camera_get_calibration"); + if (auth != XRT_SUCCESS) { + return auth; + } + if (m == NULL) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + os_mutex_lock(&m->lock); + struct scam_camera *cam = camera_by_id_locked(m, camera_id); + if (cam == NULL || (output != XRT_STEREO_CAMERA_OUTPUT_RAW && output != XRT_STEREO_CAMERA_OUTPUT_RECTIFIED)) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + // Spec §7.5: calibration identifies the device — only after §7.1 passes. + xrt_result_t xret = authorise_locked(ics, cam, "stereo_camera_get_calibration"); + if (xret == XRT_SUCCESS && !output_allowed(ics, output)) { + xret = XRT_ERROR_NOT_AUTHORIZED; + } + if (xret == XRT_SUCCESS && !cam->have_calib) { + xret = XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + if (xret == XRT_SUCCESS && output == XRT_STEREO_CAMERA_OUTPUT_RECTIFIED && !camera_can_rectify(cam)) { + xret = XRT_ERROR_FEATURE_NOT_SUPPORTED; // R2 computes post-rectification numbers + } + if (xret != XRT_SUCCESS) { + os_mutex_unlock(&m->lock); + return xret; + } + const struct xrt_plugin_stereo_camera_calibration *c = &cam->calib; + out_calib->output = output; + for (int e = 0; e < 2; e++) { + struct xrt_stereo_camera_intrinsics *in = &out_calib->eye[e]; + in->width = c->image_width; + in->height = c->image_height; + in->fx = (float)c->k[e][0]; + in->fy = (float)c->k[e][1]; + in->cx = (float)c->k[e][2]; + in->cy = (float)c->k[e][3]; + in->model = output == XRT_STEREO_CAMERA_OUTPUT_RECTIFIED ? XRT_PLUGIN_STEREO_CAMERA_DISTORTION_NONE + : c->distortion_model; + if (in->model != XRT_PLUGIN_STEREO_CAMERA_DISTORTION_NONE) { + for (int k = 0; k < 8; k++) { + in->coefficients[k] = (float)c->distortion[e][k]; + } + } + } + // Plug-in: OpenCV x_R = R x_L + T. Client: the pose of the right camera in + // the left camera's frame = (R^T, -R^T T), metres. + double rt[3][3]; + for (int i = 0; i < 3; i++) { + for (int j = 0; j < 3; j++) { + rt[i][j] = c->rotation_right_from_left[j][i]; + } + } + mat3_to_quat(rt, out_calib->orientation); + for (int i = 0; i < 3; i++) { + double p = 0.0; + for (int j = 0; j < 3; j++) { + p -= rt[i][j] * c->translation_right_from_left_mm[j]; + } + out_calib->position[i] = (float)(p / 1000.0); + } + out_calib->baseline_mm = (float)baseline_mm(c); + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; +} + +xrt_result_t +ipc_handle_stereo_camera_stream_create(volatile struct ipc_client_state *ics, + const struct xrt_stereo_camera_stream_request *req, + uint64_t *out_stream_id) +{ + *out_stream_id = 0; + struct ipc_server_stereo_camera *m = mgr_of(ics); + xrt_result_t auth = require_camera_client(ics, "stereo_camera_stream_create"); + if (auth != XRT_SUCCESS) { + return auth; + } + if (m == NULL) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + if (req->transport != XRT_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; // GPU transports: see the file comment + } + if (req->format < XRT_STEREO_CAMERA_FORMAT_GRAY8 || req->format > XRT_STEREO_CAMERA_FORMAT_BGRA8 || + (req->output != XRT_STEREO_CAMERA_OUTPUT_RAW && req->output != XRT_STEREO_CAMERA_OUTPUT_RECTIFIED)) { + return XRT_ERROR_INPUT_UNSUPPORTED; + } + if (!output_allowed(ics, req->output)) { + U_LOG_W("stereo_camera_stream_create: RAW refused for pid %ld (PRESENT_OWNER: rectified only)", + ics->peer_pid); + return XRT_ERROR_NOT_AUTHORIZED; + } + char exe[260] = {0}; + ipc_server_peer_exe_path(ics->peer_pid, exe, sizeof(exe)); + + os_mutex_lock(&m->lock); + struct scam_camera *cam = camera_by_id_locked(m, req->camera_id); + if (cam == NULL) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_INPUT_UNSUPPORTED; + } + bool calibrated = (cam->info.flags & + (XRT_PLUGIN_STEREO_CAMERA_CALIBRATED | XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED)) != 0; + if (req->output == XRT_STEREO_CAMERA_OUTPUT_RECTIFIED && !calibrated) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_INPUT_UNSUPPORTED; + } + uint32_t per_camera = 0; + struct scam_stream *slot = NULL; + for (uint32_t i = 0; i < MAX_STREAMS; i++) { + struct scam_stream *s = &m->streams[i]; + if (s->used && s->camera == cam->index) { + per_camera++; + } else if (!s->used && slot == NULL) { + slot = s; + } + } + if (per_camera >= XRT_STEREO_CAMERA_MAX_STREAMS_PER_CAMERA || slot == NULL) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_CLIENT_LIMIT_REACHED; + } + memset(slot, 0, sizeof(*slot)); + slot->section = XRT_SHMEM_HANDLE_INVALID; +#ifndef XRT_OS_WINDOWS + slot->wake.fds[0] = slot->wake.fds[1] = -1; +#endif + slot->used = true; + slot->id = ++m->next_stream_id; + slot->owner = owner_of(m, ics); + slot->ics = ics; + slot->camera = cam->index; + slot->req = *req; + // R1: RECTIFIED only from a natively rectified source; a calibrated one is + // delivered RAW and FLAGGED (frame.output) until u_stereo_rectify lands (R2). + slot->output = (req->output == XRT_STEREO_CAMERA_OUTPUT_RECTIFIED && camera_can_rectify(cam)) + ? XRT_STEREO_CAMERA_OUTPUT_RECTIFIED + : XRT_STEREO_CAMERA_OUTPUT_RAW; + snprintf(slot->consumer, sizeof(slot->consumer), "%s (pid %ld)", exe[0] ? exe : "?", ics->peer_pid); + *out_stream_id = slot->id; + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; +} + +xrt_result_t +ipc_handle_stereo_camera_stream_destroy(volatile struct ipc_client_state *ics, uint64_t stream_id) +{ + struct ipc_server_stereo_camera *m = mgr_of(ics); + if (m == NULL) { + return XRT_SUCCESS; + } + os_mutex_lock(&m->lock); + struct scam_stream *s = stream_by_id_locked(m, ics, stream_id); + if (s != NULL) { + stream_destroy_locked(m, s); + } + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; +} + +xrt_result_t +ipc_handle_stereo_camera_stream_start(volatile struct ipc_client_state *ics, uint64_t stream_id) +{ + struct ipc_server_stereo_camera *m = mgr_of(ics); + xrt_result_t auth = require_camera_client(ics, "stereo_camera_stream_start"); + if (auth != XRT_SUCCESS) { + return auth; + } + if (m == NULL) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + os_mutex_lock(&m->lock); + struct scam_stream *s = stream_by_id_locked(m, ics, stream_id); + if (s == NULL) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_INPUT_UNSUPPORTED; + } + if (s->started) { + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; + } + struct scam_camera *cam = &m->cams[s->camera]; + // Spec §7.1: start is THE authorisation point. + xrt_result_t xret = authorise_locked(ics, cam, "stereo_camera_stream_start"); + if (xret == XRT_SUCCESS) { + xret = stream_allocate_locked(s, cam); + } + if (xret != XRT_SUCCESS) { + os_mutex_unlock(&m->lock); + return xret; + } + u_stereo_camera_decimator_init(&s->dec, s->req.max_frame_rate, source_rate_hz(cam)); + u_stereo_camera_ring_clear(&s->ring); + s->started = true; + cam->started_count++; + if (!cam->thread_started) { + os_thread_init(&cam->thread); + if (os_thread_start(&cam->thread, camera_thread, cam) != 0) { + s->started = false; + cam->started_count--; + os_mutex_unlock(&m->lock); + return XRT_ERROR_THREADING_INIT_FAILURE; + } + cam->thread_started = true; + } + os_cond_signal(&cam->cond); + // TODO(R3): runtime-owned in-use indicator naming s->consumer. + U_LOG_W("stereo camera %llu: stream %llu started by %s — output %s, format %u, max %.1f Hz; %u started", + (unsigned long long)cam->camera_id, (unsigned long long)s->id, s->consumer, + s->output == XRT_STEREO_CAMERA_OUTPUT_RECTIFIED ? "RECTIFIED" : "RAW", s->req.format, + s->req.max_frame_rate, cam->started_count); + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; +} + +xrt_result_t +ipc_handle_stereo_camera_stream_stop(volatile struct ipc_client_state *ics, uint64_t stream_id) +{ + struct ipc_server_stereo_camera *m = mgr_of(ics); + if (m == NULL) { + return XRT_SUCCESS; + } + os_mutex_lock(&m->lock); + struct scam_stream *s = stream_by_id_locked(m, ics, stream_id); + if (s != NULL) { + stream_stop_locked(s, &m->cams[s->camera]); + } + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; +} + +xrt_result_t +ipc_handle_stereo_camera_stream_get_section(volatile struct ipc_client_state *ics, + uint64_t stream_id, + struct xrt_stereo_camera_stream_layout *out_layout, + uint32_t max_handle_count, + xrt_shmem_handle_t *out_handles, + uint32_t *out_handle_count) +{ + U_ZERO(out_layout); + *out_handle_count = 0; + struct ipc_server_stereo_camera *m = mgr_of(ics); + if (m == NULL || max_handle_count < 1) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + os_mutex_lock(&m->lock); + struct scam_stream *s = stream_by_id_locked(m, ics, stream_id); + if (s == NULL || !s->allocated) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_INPUT_UNSUPPORTED; // not started yet + } + out_layout->width = s->width; + out_layout->height = s->height; + out_layout->format = s->req.format; + out_layout->output = s->output; + out_layout->transport = s->req.transport; + out_layout->max_frame_rate = + s->dec.period_ns > 0 ? (float)(1e9 / (double)s->dec.period_ns) : source_rate_hz(&m->cams[s->camera]); + out_layout->section_size = s->section_size; + out_layout->slot_stride = s->slot_stride; + out_layout->slot_count = U_STEREO_CAMERA_RING_SLOTS; +#ifdef XRT_OS_WINDOWS + // A READ-ONLY duplicate: the consumer can map it but never write the ring. + if (s->sent_section_dup != NULL) { + CloseHandle(s->sent_section_dup); + s->sent_section_dup = NULL; + } + if (!DuplicateHandle(GetCurrentProcess(), s->section, GetCurrentProcess(), &s->sent_section_dup, FILE_MAP_READ, + FALSE, 0)) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_IPC_FAILURE; + } + out_handles[0] = s->sent_section_dup; +#else + out_handles[0] = s->section; // SCM_RIGHTS installs a copy; ours stays open +#endif + *out_handle_count = 1; + ics->handles_sent = true; // browser#103 RC-1 + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; +} + +xrt_result_t +ipc_handle_stereo_camera_stream_get_wake(volatile struct ipc_client_state *ics, + uint64_t stream_id, + uint32_t max_handle_count, + xrt_graphics_sync_handle_t *out_handles, + uint32_t *out_handle_count) +{ + *out_handle_count = 0; + struct ipc_server_stereo_camera *m = mgr_of(ics); + if (m == NULL || max_handle_count < 1) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + os_mutex_lock(&m->lock); + struct scam_stream *s = stream_by_id_locked(m, ics, stream_id); + if (s == NULL || !s->allocated) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_INPUT_UNSUPPORTED; + } +#ifdef XRT_OS_WINDOWS + // SYNCHRONIZE only: the consumer can wait, never signal / reset it. + if (s->sent_wake_dup != NULL) { + CloseHandle(s->sent_wake_dup); + s->sent_wake_dup = NULL; + } + if (!DuplicateHandle(GetCurrentProcess(), s->wake.event, GetCurrentProcess(), &s->sent_wake_dup, SYNCHRONIZE, + FALSE, 0)) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_IPC_FAILURE; + } + out_handles[0] = s->sent_wake_dup; +#else + out_handles[0] = s->wake.fds[0]; +#endif + *out_handle_count = 1; + ics->handles_sent = true; + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; +} + +xrt_result_t +ipc_handle_stereo_camera_acquire(volatile struct ipc_client_state *ics, + uint64_t stream_id, + bool *out_ready, + struct xrt_stereo_camera_frame_info *out_frame) +{ + *out_ready = false; + U_ZERO(out_frame); + struct ipc_server_stereo_camera *m = mgr_of(ics); + if (m == NULL) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + os_mutex_lock(&m->lock); + struct scam_stream *s = stream_by_id_locked(m, ics, stream_id); + if (s == NULL) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_INPUT_UNSUPPORTED; + } + int32_t slot = -1; + uint64_t seq = 0; + if (s->started && s->allocated && u_stereo_camera_ring_acquire(&s->ring, &slot, &seq)) { + *out_frame = s->slot_info[slot]; + *out_ready = true; + } + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; +} + +xrt_result_t +ipc_handle_stereo_camera_stream_stats(volatile struct ipc_client_state *ics, + uint64_t stream_id, + struct xrt_stereo_camera_stream_stats *out_stats) +{ + U_ZERO(out_stats); + struct ipc_server_stereo_camera *m = mgr_of(ics); + if (m == NULL) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + os_mutex_lock(&m->lock); + struct scam_stream *s = stream_by_id_locked(m, ics, stream_id); + if (s == NULL) { + os_mutex_unlock(&m->lock); + return XRT_ERROR_INPUT_UNSUPPORTED; + } + out_stats->source_frame_rate = m->cams[s->camera].source_rate; + out_stats->delivered_frame_rate = s->delivered_rate; + out_stats->frames_published = s->ring.published; + out_stats->frames_skipped = s->ring.skipped; + out_stats->frames_acquired = s->ring.acquired; + out_stats->mean_latency_ns = s->latency_count > 0 ? s->latency_sum_ns / s->latency_count : 0; + os_mutex_unlock(&m->lock); + return XRT_SUCCESS; +} diff --git a/src/xrt/ipc/server/ipc_server_stereo_camera.h b/src/xrt/ipc/server/ipc_server_stereo_camera.h new file mode 100644 index 0000000000..6d979be1c1 --- /dev/null +++ b/src/xrt/ipc/server/ipc_server_stereo_camera.h @@ -0,0 +1,50 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera (ADR-043): the service's camera manager. + * + * The service is the single owner of each plug-in-provided stereo camera: one + * runtime-owned thread per open camera pulls frames from the plug-in's + * wait_frame slot, and fans each frame out to every started, authorised stream + * — per stream a 3-slot latest-wins shared-memory ring (u_stereo_camera_ring), + * format conversion, decimation, and a per-stream wake handle. The source is + * opened on the first start and closed a linger (2 s) after the last stop. + * + * Platform-neutral (Windows D3D11 service, macOS / Linux service, Android + * runtime service): the only OS-specific pieces are the wake handle (event / + * pipe) and the read-only section duplicate, both in ipc_server_stereo_camera.c. + * + * @ingroup ipc_server + */ + +#pragma once + +#include "xrt/xrt_instance.h" + +#ifdef __cplusplus +extern "C" { +#endif + +struct ipc_server_stereo_camera; +struct ipc_client_state; + +/*! + * Create the manager over the active plug-in of @p xinst (the service's native + * instance). Always returns a manager; with no plug-in, no camera slots, or the + * kill switch (DXR_STEREO_CAMERA=0) it simply exposes zero cameras. + */ +struct ipc_server_stereo_camera * +ipc_server_stereo_camera_create(struct xrt_instance *xinst); + +//! Stop every camera thread, close every source, free every stream. +void +ipc_server_stereo_camera_destroy(struct ipc_server_stereo_camera **mgr_ptr); + +//! Destroy every stream @p ics created. Called once at client teardown. +void +ipc_server_client_stereo_camera_release(volatile struct ipc_client_state *ics); + +#ifdef __cplusplus +} +#endif diff --git a/src/xrt/ipc/shared/ipc_protocol.h b/src/xrt/ipc/shared/ipc_protocol.h index 562eb1d43d..bea1dab4ff 100644 --- a/src/xrt/ipc/shared/ipc_protocol.h +++ b/src/xrt/ipc/shared/ipc_protocol.h @@ -25,6 +25,7 @@ #include "xrt/xrt_space.h" #include "xrt/xrt_tracking.h" #include "xrt/xrt_display_metrics.h" +#include "xrt/xrt_stereo_camera.h" #include "xrt/xrt_config_build.h" #include diff --git a/src/xrt/ipc/shared/proto.json b/src/xrt/ipc/shared/proto.json index 67c40f1784..bcc283c338 100644 --- a/src/xrt/ipc/shared/proto.json +++ b/src/xrt/ipc/shared/proto.json @@ -1194,5 +1194,93 @@ "out": [ {"name": "accepted", "type": "uint32_t"} ] + }, + + "stereo_camera_count": { + "out": [ + {"name": "count", "type": "uint32_t"} + ] + }, + + "stereo_camera_get_properties": { + "in": [ + {"name": "index", "type": "uint32_t"} + ], + "out": [ + {"name": "props", "type": "struct xrt_stereo_camera_properties"} + ] + }, + + "stereo_camera_get_calibration": { + "in": [ + {"name": "camera_id", "type": "uint64_t"}, + {"name": "output", "type": "uint32_t"} + ], + "out": [ + {"name": "calib", "type": "struct xrt_stereo_camera_calibration"} + ] + }, + + "stereo_camera_stream_create": { + "in": [ + {"name": "req", "type": "struct xrt_stereo_camera_stream_request"} + ], + "out": [ + {"name": "stream_id", "type": "uint64_t"} + ] + }, + + "stereo_camera_stream_destroy": { + "in": [ + {"name": "stream_id", "type": "uint64_t"} + ] + }, + + "stereo_camera_stream_start": { + "in": [ + {"name": "stream_id", "type": "uint64_t"} + ] + }, + + "stereo_camera_stream_stop": { + "in": [ + {"name": "stream_id", "type": "uint64_t"} + ] + }, + + "stereo_camera_stream_get_section": { + "in": [ + {"name": "stream_id", "type": "uint64_t"} + ], + "out": [ + {"name": "layout", "type": "struct xrt_stereo_camera_stream_layout"} + ], + "out_handles": {"type": "xrt_shmem_handle_t"} + }, + + "stereo_camera_stream_get_wake": { + "in": [ + {"name": "stream_id", "type": "uint64_t"} + ], + "out_handles": {"type": "xrt_graphics_sync_handle_t"} + }, + + "stereo_camera_acquire": { + "in": [ + {"name": "stream_id", "type": "uint64_t"} + ], + "out": [ + {"name": "ready", "type": "bool"}, + {"name": "frame", "type": "struct xrt_stereo_camera_frame_info"} + ] + }, + + "stereo_camera_stream_stats": { + "in": [ + {"name": "stream_id", "type": "uint64_t"} + ], + "out": [ + {"name": "stats", "type": "struct xrt_stereo_camera_stream_stats"} + ] } } diff --git a/src/xrt/state_trackers/oxr/CMakeLists.txt b/src/xrt/state_trackers/oxr/CMakeLists.txt index a141f7efa5..6c21fc979e 100644 --- a/src/xrt/state_trackers/oxr/CMakeLists.txt +++ b/src/xrt/state_trackers/oxr/CMakeLists.txt @@ -61,6 +61,7 @@ add_library( oxr_views_change.h oxr_weave.c oxr_lift.c + oxr_stereo_camera.c oxr_workspace.c oxr_workspace_modal_win32.c oxr_workspace_file_dialog.c diff --git a/src/xrt/state_trackers/oxr/oxr_api_funcs.h b/src/xrt/state_trackers/oxr/oxr_api_funcs.h index 7774d11efa..051679369f 100644 --- a/src/xrt/state_trackers/oxr/oxr_api_funcs.h +++ b/src/xrt/state_trackers/oxr/oxr_api_funcs.h @@ -1112,6 +1112,50 @@ XRAPI_ATTR XrResult XRAPI_CALL oxr_xrGetLiftStreamStatsDXR(XrLiftStreamDXR stream, XrLiftStreamStatsDXR *stats); #endif +#ifdef OXR_HAVE_DXR_stereo_camera +//! OpenXR API function @ep{xrEnumerateStereoCamerasDXR} +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrEnumerateStereoCamerasDXR(XrInstance instance, + XrSystemId systemId, + uint32_t capacityInput, + uint32_t *countOutput, + XrStereoCameraPropertiesDXR *cameras); + +//! OpenXR API function @ep{xrGetStereoCameraCalibrationDXR} +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrGetStereoCameraCalibrationDXR(XrInstance instance, uint64_t cameraId, XrStereoCameraCalibrationDXR *calibration); + +//! OpenXR API function @ep{xrCreateStereoCameraStreamDXR} +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrCreateStereoCameraStreamDXR(XrInstance instance, + const XrStereoCameraStreamCreateInfoDXR *createInfo, + XrStereoCameraStreamDXR *stream); + +//! OpenXR API function @ep{xrDestroyStereoCameraStreamDXR} +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrDestroyStereoCameraStreamDXR(XrStereoCameraStreamDXR stream); + +//! OpenXR API function @ep{xrStartStereoCameraStreamDXR} +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrStartStereoCameraStreamDXR(XrStereoCameraStreamDXR stream); + +//! OpenXR API function @ep{xrStopStereoCameraStreamDXR} +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrStopStereoCameraStreamDXR(XrStereoCameraStreamDXR stream); + +//! OpenXR API function @ep{xrGetStereoCameraStreamInfoDXR} +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrGetStereoCameraStreamInfoDXR(XrStereoCameraStreamDXR stream, XrStereoCameraStreamInfoDXR *info); + +//! OpenXR API function @ep{xrAcquireStereoCameraFrameDXR} +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrAcquireStereoCameraFrameDXR(XrStereoCameraStreamDXR stream, XrStereoCameraFrameDXR *frame); + +//! OpenXR API function @ep{xrGetStereoCameraStreamStatsDXR} +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrGetStereoCameraStreamStatsDXR(XrStereoCameraStreamDXR stream, XrStereoCameraStreamStatsDXR *stats); +#endif + #ifdef OXR_HAVE_EXT_conformance_automation //! OpenXR API function @ep{xrSetInputDeviceActiveEXT} XRAPI_ATTR XrResult XRAPI_CALL diff --git a/src/xrt/state_trackers/oxr/oxr_api_negotiate.c b/src/xrt/state_trackers/oxr/oxr_api_negotiate.c index 0a8fe96344..9481242809 100644 --- a/src/xrt/state_trackers/oxr/oxr_api_negotiate.c +++ b/src/xrt/state_trackers/oxr/oxr_api_negotiate.c @@ -527,6 +527,18 @@ handle_non_null(struct oxr_instance *inst, struct oxr_logger *log, const char *n ENTRY_IF_EXT(xrGetLiftStreamStatsDXR, DXR_lift); #endif +#ifdef OXR_HAVE_DXR_stereo_camera + ENTRY_IF_EXT(xrEnumerateStereoCamerasDXR, DXR_stereo_camera); + ENTRY_IF_EXT(xrGetStereoCameraCalibrationDXR, DXR_stereo_camera); + ENTRY_IF_EXT(xrCreateStereoCameraStreamDXR, DXR_stereo_camera); + ENTRY_IF_EXT(xrDestroyStereoCameraStreamDXR, DXR_stereo_camera); + ENTRY_IF_EXT(xrStartStereoCameraStreamDXR, DXR_stereo_camera); + ENTRY_IF_EXT(xrStopStereoCameraStreamDXR, DXR_stereo_camera); + ENTRY_IF_EXT(xrGetStereoCameraStreamInfoDXR, DXR_stereo_camera); + ENTRY_IF_EXT(xrAcquireStereoCameraFrameDXR, DXR_stereo_camera); + ENTRY_IF_EXT(xrGetStereoCameraStreamStatsDXR, DXR_stereo_camera); +#endif + #ifdef OXR_HAVE_DXR_workspace_file_dialog ENTRY_IF_EXT(xrRequestFilePickerDXR, DXR_workspace_file_dialog); ENTRY_IF_EXT(xrGetFilePickerRequestDXR, DXR_workspace_file_dialog); diff --git a/src/xrt/state_trackers/oxr/oxr_api_verify.h b/src/xrt/state_trackers/oxr/oxr_api_verify.h index 8ea5037cbb..f27222b22e 100644 --- a/src/xrt/state_trackers/oxr/oxr_api_verify.h +++ b/src/xrt/state_trackers/oxr/oxr_api_verify.h @@ -96,6 +96,9 @@ struct oxr_subaction_paths; OXR_VERIFY_AND_SET_AND_INIT(log, thing, new_thing, oxr_local_3d_zone_ext, LOCAL3DZONE, name, new_thing->sess->sys->inst) #define OXR_VERIFY_LIFT_STREAM_AND_INIT_LOG(log, thing, new_thing, name) \ OXR_VERIFY_AND_SET_AND_INIT(log, thing, new_thing, oxr_lift_stream_dxr, LIFTSTREAM, name, new_thing->sess->sys->inst) + +#define OXR_VERIFY_STEREO_CAMERA_STREAM_AND_INIT_LOG(log, thing, new_thing, name) \ + OXR_VERIFY_AND_SET_AND_INIT(log, thing, new_thing, oxr_stereo_camera_stream_dxr, STEREOCAMSTREAM, name, new_thing->inst) // clang-format on #define OXR_VERIFY_INSTANCE_NOT_NULL(log, arg, new_arg) OXR_VERIFY_SET(log, arg, new_arg, oxr_instance, INSTANCE); diff --git a/src/xrt/state_trackers/oxr/oxr_defines.h b/src/xrt/state_trackers/oxr/oxr_defines.h index dbb77e62df..25b8fe2af9 100644 --- a/src/xrt/state_trackers/oxr/oxr_defines.h +++ b/src/xrt/state_trackers/oxr/oxr_defines.h @@ -38,6 +38,8 @@ #define OXR_XR_DEBUG_LOCAL3DZONE (*(uint64_t *)"oxrl3dz\0") // XR_DXR_lift stream (ADR-042) #define OXR_XR_DEBUG_LIFTSTREAM (*(uint64_t *)"oxrlift\0") +// XR_DXR_stereo_camera stream (ADR-043) +#define OXR_XR_DEBUG_STEREOCAMSTREAM (*(uint64_t *)"oxrscam\0") // clang-format on /*! diff --git a/src/xrt/state_trackers/oxr/oxr_extension_support.h b/src/xrt/state_trackers/oxr/oxr_extension_support.h index 3574034b8f..ad72ecc781 100644 --- a/src/xrt/state_trackers/oxr/oxr_extension_support.h +++ b/src/xrt/state_trackers/oxr/oxr_extension_support.h @@ -768,6 +768,24 @@ #define OXR_EXTENSION_SUPPORT_DXR_lift(_) #endif +/* + * XR_DXR_stereo_camera + * + * Hand-added DisplayXR extension (generate_oxr_ext_support.py knows nothing of + * the DisplayXR blocks — keep them when regenerating). A display's stereo + * camera as a plug-in-provided, service-owned source (ADR-043). Advertised on + * every platform: the camera manager is platform-neutral service code, and an + * instance with no camera (in-process, no plug-in source, kill switch) simply + * enumerates zero. + */ +#if defined(XR_DXR_stereo_camera) +#define OXR_HAVE_DXR_stereo_camera +#define OXR_EXTENSION_SUPPORT_DXR_stereo_camera(_) \ + _(DXR_stereo_camera, DXR_STEREO_CAMERA) +#else +#define OXR_EXTENSION_SUPPORT_DXR_stereo_camera(_) +#endif + /* * XR_DXR_workspace_file_dialog @@ -1282,6 +1300,7 @@ OXR_EXTENSION_SUPPORT_DXR_display_zones(_) \ OXR_EXTENSION_SUPPORT_DXR_weave(_) \ OXR_EXTENSION_SUPPORT_DXR_lift(_) \ + OXR_EXTENSION_SUPPORT_DXR_stereo_camera(_) \ OXR_EXTENSION_SUPPORT_DXR_workspace_file_dialog(_) \ OXR_EXTENSION_SUPPORT_DXR_mcp_tools(_) \ OXR_EXTENSION_SUPPORT_BD_controller_interaction(_) \ diff --git a/src/xrt/state_trackers/oxr/oxr_objects.h b/src/xrt/state_trackers/oxr/oxr_objects.h index 745f1b71b1..7753ab179b 100644 --- a/src/xrt/state_trackers/oxr/oxr_objects.h +++ b/src/xrt/state_trackers/oxr/oxr_objects.h @@ -135,6 +135,7 @@ struct oxr_xdev_list; struct oxr_plane_detector_ext; struct oxr_local_3d_zone_ext; struct oxr_lift_stream_dxr; +struct oxr_stereo_camera_stream_dxr; #define XRT_MAX_HANDLE_CHILDREN 256 #define OXR_MAX_BINDINGS_PER_ACTION 32 @@ -4355,6 +4356,27 @@ struct oxr_lift_stream_dxr }; #endif // OXR_HAVE_DXR_lift +#ifdef OXR_HAVE_DXR_stereo_camera +/*! + * One XR_DXR_stereo_camera stream (ADR-043). The camera and its ring live in + * the service (ipc_server_stereo_camera.c); this is the handle and the + * service-side stream id, owned by the instance's IPC connection. + * + * Parent type/handle is @ref oxr_instance + * + * @obj{XrStereoCameraStreamDXR} + * @extends oxr_handle_base + */ +struct oxr_stereo_camera_stream_dxr +{ + struct oxr_handle_base handle; + struct oxr_instance *inst; + uint64_t id; //!< service-side stream id + uint64_t camera_id; //!< service-side camera id + bool started; +}; +#endif // OXR_HAVE_DXR_stereo_camera + #ifdef OXR_HAVE_EXT_user_presence XrResult diff --git a/src/xrt/state_trackers/oxr/oxr_stereo_camera.c b/src/xrt/state_trackers/oxr/oxr_stereo_camera.c new file mode 100644 index 0000000000..5664d170c6 --- /dev/null +++ b/src/xrt/state_trackers/oxr/oxr_stereo_camera.c @@ -0,0 +1,474 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera API entry points (ADR-043). + * @author David Fattal + * @ingroup oxr_api + * + * Instance-level: a capture component has no session. Every call goes through + * the instance's @ref xrt_stereo_camera_client aspect, which only the IPC + * client instance carries — an in-process instance enumerates zero cameras + * (the service is each camera's single owner; maintainer decision for R1, + * roadmap §G.3). + * + * Error contract: + * - dead pipe (XRT_ERROR_IPC_FAILURE) -> XR_ERROR_INSTANCE_LOST + * - refused by consent (XRT_ERROR_NOT_AUTHORIZED) -> XR_ERROR_PERMISSION_INSUFFICIENT + * (retryable: the stream stays created) + * - no such camera / bad format / output -> XR_ERROR_VALIDATION_FAILURE + * - transport or output the runtime lacks -> XR_ERROR_FEATURE_UNSUPPORTED + * - 8 streams on the camera already -> XR_ERROR_LIMIT_REACHED + * - anything else on a healthy pipe -> XR_ERROR_RUNTIME_FAILURE + */ + +#include "oxr_objects.h" +#include "oxr_logger.h" +#include "oxr_handle.h" +#include "oxr_api_funcs.h" +#include "oxr_api_verify.h" +#include "oxr_chain.h" + +#include "util/u_misc.h" +#include "util/u_time.h" +#include "util/u_trace_marker.h" + +#include "xrt/xrt_instance.h" +#include "xrt/xrt_plugin.h" +#include "xrt/xrt_stereo_camera.h" + +#include "xrt/xrt_config_os.h" + +#include +#if defined(XRT_OS_WINDOWS) +#include +#else +#include +#endif + +#ifdef OXR_HAVE_DXR_stereo_camera + +// The wire carries the xrt encodings; they equal the XR ones. The XR flag bits +// are `static const XrFlags64` — not constant expressions in C (MSVC and GCC +// reject them in _Static_assert; clang only accepts them as an extension) — so +// the flag / format-bit mirrors are pinned against the header's literal values +// here and cross-checked against the real constants in verify_encodings(). +_Static_assert(XRT_PLUGIN_STEREO_CAMERA_SHARED_WITH_EYE_TRACKING == 0x1, "flag mirror"); +_Static_assert(XRT_PLUGIN_STEREO_CAMERA_USER_FACING == 0x2, "flag mirror"); +_Static_assert(XRT_PLUGIN_STEREO_CAMERA_CALIBRATED == 0x4, "flag mirror"); +_Static_assert(XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED == 0x8, "flag mirror"); +_Static_assert(XRT_PLUGIN_STEREO_CAMERA_MONOCHROME == 0x10, "flag mirror"); +// The enum mirrors are pinned the same way, each side against the literal on +// its own: a direct XR-vs-xrt comparison is "operands are different enum types" +// (MSVC C5287 at /W4), and MSVC's C front end raises it even through (int) +// casts. Two enum-vs-literal checks carry the same proof with no cross-enum op. +#define SCAM_PIN(xr, xrt, v) _Static_assert((xr) == (v) && (xrt) == (v), #xr " != " #xrt) +SCAM_PIN(XR_STEREO_CAMERA_STATE_AVAILABLE_DXR, XRT_STEREO_CAMERA_STATE_AVAILABLE, 1); +SCAM_PIN(XR_STEREO_CAMERA_STATE_WAITING_DXR, XRT_STEREO_CAMERA_STATE_WAITING, 2); +SCAM_PIN(XR_STEREO_CAMERA_STATE_SUSPENDED_DXR, XRT_STEREO_CAMERA_STATE_SUSPENDED, 3); +SCAM_PIN(XR_STEREO_CAMERA_STATE_UNAVAILABLE_DXR, XRT_STEREO_CAMERA_STATE_UNAVAILABLE, 4); +SCAM_PIN(XR_STEREO_CAMERA_OUTPUT_RECTIFIED_DXR, XRT_STEREO_CAMERA_OUTPUT_RECTIFIED, 1); +SCAM_PIN(XR_STEREO_CAMERA_OUTPUT_RAW_DXR, XRT_STEREO_CAMERA_OUTPUT_RAW, 2); +SCAM_PIN(XR_STEREO_CAMERA_FORMAT_GRAY8_DXR, XRT_STEREO_CAMERA_FORMAT_GRAY8, 1); +SCAM_PIN(XR_STEREO_CAMERA_FORMAT_NV12_DXR, XRT_STEREO_CAMERA_FORMAT_NV12, 2); +SCAM_PIN(XR_STEREO_CAMERA_FORMAT_BGRA8_DXR, XRT_STEREO_CAMERA_FORMAT_BGRA8, 3); +SCAM_PIN(XR_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY_DXR, XRT_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY, 1); +#undef SCAM_PIN +_Static_assert(XR_STEREO_CAMERA_RING_SLOTS_DXR == XRT_STEREO_CAMERA_RING_SLOTS, "ring slots"); +_Static_assert(XR_STEREO_CAMERA_PERSISTENT_ID_MAX_SIZE_DXR == + sizeof(((struct xrt_stereo_camera_properties *)0)->persistent_id), + "id size"); +_Static_assert(XR_STEREO_CAMERA_DISPLAY_NAME_MAX_SIZE_DXR == + sizeof(((struct xrt_stereo_camera_properties *)0)->display_name), + "name size"); +_Static_assert(XR_STEREO_CAMERA_PLATFORM_HINT_MAX_SIZE_DXR == + sizeof(((struct xrt_stereo_camera_properties *)0)->platform_device_hint), + "hint size"); + +//! Runtime half of the encoding check (see the _Static_assert block above). +static bool +verify_encodings(void) +{ + return XR_STEREO_CAMERA_SHARED_WITH_EYE_TRACKING_BIT_DXR == XRT_PLUGIN_STEREO_CAMERA_SHARED_WITH_EYE_TRACKING && + XR_STEREO_CAMERA_USER_FACING_BIT_DXR == XRT_PLUGIN_STEREO_CAMERA_USER_FACING && + XR_STEREO_CAMERA_CALIBRATED_BIT_DXR == XRT_PLUGIN_STEREO_CAMERA_CALIBRATED && + XR_STEREO_CAMERA_NATIVELY_RECTIFIED_BIT_DXR == XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED && + XR_STEREO_CAMERA_MONOCHROME_BIT_DXR == XRT_PLUGIN_STEREO_CAMERA_MONOCHROME && + XR_STEREO_CAMERA_FORMAT_GRAY8_BIT_DXR == XRT_STEREO_CAMERA_BIT(XRT_STEREO_CAMERA_FORMAT_GRAY8) && + XR_STEREO_CAMERA_FORMAT_NV12_BIT_DXR == XRT_STEREO_CAMERA_BIT(XRT_STEREO_CAMERA_FORMAT_NV12) && + XR_STEREO_CAMERA_FORMAT_BGRA8_BIT_DXR == XRT_STEREO_CAMERA_BIT(XRT_STEREO_CAMERA_FORMAT_BGRA8) && + XR_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY_BIT_DXR == + XRT_STEREO_CAMERA_BIT(XRT_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY); +} + +static struct xrt_stereo_camera_client * +client_of(struct oxr_instance *inst) +{ + return inst->xinst != NULL ? inst->xinst->stereo_camera : NULL; +} + +static XrResult +scam_xret(struct oxr_logger *log, xrt_result_t xret, const char *what) +{ + switch (xret) { + case XRT_SUCCESS: return XR_SUCCESS; + case XRT_ERROR_IPC_FAILURE: + return oxr_error(log, XR_ERROR_INSTANCE_LOST, "%s: the runtime service connection is gone", what); + case XRT_ERROR_NOT_AUTHORIZED: + return oxr_error(log, XR_ERROR_PERMISSION_INSUFFICIENT, + "%s: the runtime refused camera access (consent; see XR_DXR_stereo_camera §7)", what); + case XRT_ERROR_INPUT_UNSUPPORTED: + return oxr_error(log, XR_ERROR_VALIDATION_FAILURE, "%s: unknown camera/stream or unsupported request", + what); + case XRT_ERROR_FEATURE_NOT_SUPPORTED: + return oxr_error(log, XR_ERROR_FEATURE_UNSUPPORTED, "%s: not supported by this runtime / camera", what); + case XRT_ERROR_CLIENT_LIMIT_REACHED: + return oxr_error(log, XR_ERROR_LIMIT_REACHED, "%s: too many streams on this camera", what); + default: + return oxr_error(log, XR_ERROR_RUNTIME_FAILURE, "%s: service refused (xrt_result=%d)", what, (int)xret); + } +} + +static XrResult +stream_destroy_cb(struct oxr_logger *log, struct oxr_handle_base *hb) +{ + struct oxr_stereo_camera_stream_dxr *s = (struct oxr_stereo_camera_stream_dxr *)hb; + struct xrt_stereo_camera_client *c = client_of(s->inst); + if (c != NULL && s->id != 0) { + // Best effort: a dead connection already released every stream. + (void)c->stream_destroy(c, s->id); + } + free(s); + return XR_SUCCESS; +} + +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrEnumerateStereoCamerasDXR(XrInstance instance, + XrSystemId systemId, + uint32_t capacityInput, + uint32_t *countOutput, + XrStereoCameraPropertiesDXR *cameras) +{ + OXR_TRACE_MARKER(); + struct oxr_instance *inst = NULL; + struct oxr_logger log; + OXR_VERIFY_INSTANCE_AND_INIT_LOG(&log, instance, inst, "xrEnumerateStereoCamerasDXR"); + OXR_VERIFY_EXTENSION(&log, inst, DXR_stereo_camera); + OXR_VERIFY_ARG_NOT_NULL(&log, countOutput); + XrResult sr = oxr_system_verify_id(&log, inst, systemId); + if (sr != XR_SUCCESS) { + return sr; + } + if (capacityInput > 0 && cameras == NULL) { + return oxr_error(&log, XR_ERROR_VALIDATION_FAILURE, "(cameras == NULL) with capacityInput > 0"); + } + + *countOutput = 0; + struct xrt_stereo_camera_client *c = client_of(inst); + if (c == NULL) { + return XR_SUCCESS; // in-process instance: zero cameras, on purpose + } + if (!verify_encodings()) { + return oxr_error(&log, XR_ERROR_RUNTIME_FAILURE, "XR_DXR_stereo_camera encoding mismatch (build bug)"); + } + uint32_t n = 0; + xrt_result_t xret = c->count(c, &n); + if (xret != XRT_SUCCESS) { + return scam_xret(&log, xret, "xrEnumerateStereoCamerasDXR"); + } + *countOutput = n; + if (capacityInput == 0) { + return XR_SUCCESS; + } + if (capacityInput < n) { + return oxr_error(&log, XR_ERROR_SIZE_INSUFFICIENT, "(capacityInput == %u) < %u cameras", capacityInput, + n); + } + for (uint32_t i = 0; i < n; i++) { + if (cameras[i].type != XR_TYPE_STEREO_CAMERA_PROPERTIES_DXR) { + return oxr_error(&log, XR_ERROR_VALIDATION_FAILURE, "cameras[%u].type", i); + } + } + for (uint32_t i = 0; i < n; i++) { + struct xrt_stereo_camera_properties p; + xret = c->get_properties(c, i, &p); + if (xret != XRT_SUCCESS) { + return scam_xret(&log, xret, "xrEnumerateStereoCamerasDXR"); + } + XrStereoCameraPropertiesDXR *o = &cameras[i]; + o->cameraId = p.camera_id; + memcpy(o->persistentId, p.persistent_id, sizeof(o->persistentId)); + memcpy(o->displayName, p.display_name, sizeof(o->displayName)); + memcpy(o->platformDeviceHint, p.platform_device_hint, sizeof(o->platformDeviceHint)); + o->flags = p.flags; + o->state = (XrStereoCameraStateDXR)p.state; + o->viewCount = p.view_count; + o->eyeExtent.width = (int32_t)p.eye_width; + o->eyeExtent.height = (int32_t)p.eye_height; + o->maxFrameRate = p.max_frame_rate; + o->baselineMm = p.baseline_mm; + o->horizontalFovDeg = p.horizontal_fov_deg; + o->supportedFormats = p.supported_formats; + o->supportedTransports = p.supported_transports; + } + return XR_SUCCESS; +} + +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrGetStereoCameraCalibrationDXR(XrInstance instance, uint64_t cameraId, XrStereoCameraCalibrationDXR *calibration) +{ + OXR_TRACE_MARKER(); + struct oxr_instance *inst = NULL; + struct oxr_logger log; + OXR_VERIFY_INSTANCE_AND_INIT_LOG(&log, instance, inst, "xrGetStereoCameraCalibrationDXR"); + OXR_VERIFY_EXTENSION(&log, inst, DXR_stereo_camera); + OXR_VERIFY_ARG_TYPE_AND_NOT_NULL(&log, calibration, XR_TYPE_STEREO_CAMERA_CALIBRATION_DXR); + if (calibration->output != XR_STEREO_CAMERA_OUTPUT_RECTIFIED_DXR && + calibration->output != XR_STEREO_CAMERA_OUTPUT_RAW_DXR) { + return oxr_error(&log, XR_ERROR_VALIDATION_FAILURE, "calibration->output"); + } + struct xrt_stereo_camera_client *c = client_of(inst); + if (c == NULL) { + return oxr_error(&log, XR_ERROR_VALIDATION_FAILURE, "no stereo camera %llu (in-process instance)", + (unsigned long long)cameraId); + } + struct xrt_stereo_camera_calibration k; + xrt_result_t xret = c->get_calibration(c, cameraId, (uint32_t)calibration->output, &k); + if (xret != XRT_SUCCESS) { + return scam_xret(&log, xret, "xrGetStereoCameraCalibrationDXR"); + } + for (int e = 0; e < 2; e++) { + XrStereoCameraIntrinsicsDXR *o = &calibration->eye[e]; + o->imageExtent.width = (int32_t)k.eye[e].width; + o->imageExtent.height = (int32_t)k.eye[e].height; + o->fx = k.eye[e].fx; + o->fy = k.eye[e].fy; + o->cx = k.eye[e].cx; + o->cy = k.eye[e].cy; + o->model = (XrStereoCameraDistortionModelDXR)k.eye[e].model; + memcpy(o->coefficients, k.eye[e].coefficients, sizeof(o->coefficients)); + } + calibration->rightFromLeft.orientation.x = k.orientation[0]; + calibration->rightFromLeft.orientation.y = k.orientation[1]; + calibration->rightFromLeft.orientation.z = k.orientation[2]; + calibration->rightFromLeft.orientation.w = k.orientation[3]; + calibration->rightFromLeft.position.x = k.position[0]; + calibration->rightFromLeft.position.y = k.position[1]; + calibration->rightFromLeft.position.z = k.position[2]; + calibration->baselineMm = k.baseline_mm; + return XR_SUCCESS; +} + +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrCreateStereoCameraStreamDXR(XrInstance instance, + const XrStereoCameraStreamCreateInfoDXR *createInfo, + XrStereoCameraStreamDXR *stream) +{ + OXR_TRACE_MARKER(); + struct oxr_instance *inst = NULL; + struct oxr_logger log; + OXR_VERIFY_INSTANCE_AND_INIT_LOG(&log, instance, inst, "xrCreateStereoCameraStreamDXR"); + OXR_VERIFY_EXTENSION(&log, inst, DXR_stereo_camera); + OXR_VERIFY_ARG_TYPE_AND_NOT_NULL(&log, createInfo, XR_TYPE_STEREO_CAMERA_STREAM_CREATE_INFO_DXR); + OXR_VERIFY_ARG_NOT_NULL(&log, stream); + + if (createInfo->format < XR_STEREO_CAMERA_FORMAT_GRAY8_DXR || + createInfo->format > XR_STEREO_CAMERA_FORMAT_BGRA8_DXR) { + return oxr_error(&log, XR_ERROR_VALIDATION_FAILURE, "createInfo->format"); + } + if (createInfo->output != XR_STEREO_CAMERA_OUTPUT_RECTIFIED_DXR && + createInfo->output != XR_STEREO_CAMERA_OUTPUT_RAW_DXR) { + return oxr_error(&log, XR_ERROR_VALIDATION_FAILURE, "createInfo->output"); + } + if (createInfo->transport != XR_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY_DXR) { + return oxr_error(&log, XR_ERROR_FEATURE_UNSUPPORTED, + "createInfo->transport: only SHARED_MEMORY is implemented (spec v1)"); + } + struct xrt_stereo_camera_client *c = client_of(inst); + if (c == NULL) { + return oxr_error(&log, XR_ERROR_VALIDATION_FAILURE, "no stereo camera %llu (in-process instance)", + (unsigned long long)createInfo->cameraId); + } + + struct xrt_stereo_camera_stream_request req = { + .camera_id = createInfo->cameraId, + .output = (uint32_t)createInfo->output, + .format = (uint32_t)createInfo->format, + .transport = (uint32_t)createInfo->transport, + .max_frame_rate = createInfo->maxFrameRate, + }; + uint64_t id = 0; + xrt_result_t xret = c->stream_create(c, &req, &id); + if (xret != XRT_SUCCESS) { + return scam_xret(&log, xret, "xrCreateStereoCameraStreamDXR"); + } + + struct oxr_stereo_camera_stream_dxr *s = NULL; + XrResult ret = OXR_ALLOCATE_HANDLE(&log, s, OXR_XR_DEBUG_STEREOCAMSTREAM, stream_destroy_cb, &inst->handle); + if (ret != XR_SUCCESS) { + (void)c->stream_destroy(c, id); + return ret; + } + s->inst = inst; + s->id = id; + s->camera_id = createInfo->cameraId; + *stream = XRT_CAST_PTR_TO_OXR_HANDLE(XrStereoCameraStreamDXR, s); + return XR_SUCCESS; +} + +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrDestroyStereoCameraStreamDXR(XrStereoCameraStreamDXR stream) +{ + OXR_TRACE_MARKER(); + struct oxr_stereo_camera_stream_dxr *s = NULL; + struct oxr_logger log; + OXR_VERIFY_STEREO_CAMERA_STREAM_AND_INIT_LOG(&log, stream, s, "xrDestroyStereoCameraStreamDXR"); + return oxr_handle_destroy(&log, &s->handle); +} + +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrStartStereoCameraStreamDXR(XrStereoCameraStreamDXR stream) +{ + OXR_TRACE_MARKER(); + struct oxr_stereo_camera_stream_dxr *s = NULL; + struct oxr_logger log; + OXR_VERIFY_STEREO_CAMERA_STREAM_AND_INIT_LOG(&log, stream, s, "xrStartStereoCameraStreamDXR"); + struct xrt_stereo_camera_client *c = client_of(s->inst); + xrt_result_t xret = c->stream_start(c, s->id); + if (xret != XRT_SUCCESS) { + return scam_xret(&log, xret, "xrStartStereoCameraStreamDXR"); + } + s->started = true; + return XR_SUCCESS; +} + +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrStopStereoCameraStreamDXR(XrStereoCameraStreamDXR stream) +{ + OXR_TRACE_MARKER(); + struct oxr_stereo_camera_stream_dxr *s = NULL; + struct oxr_logger log; + OXR_VERIFY_STEREO_CAMERA_STREAM_AND_INIT_LOG(&log, stream, s, "xrStopStereoCameraStreamDXR"); + struct xrt_stereo_camera_client *c = client_of(s->inst); + xrt_result_t xret = c->stream_stop(c, s->id); + if (xret != XRT_SUCCESS) { + return scam_xret(&log, xret, "xrStopStereoCameraStreamDXR"); + } + s->started = false; + return XR_SUCCESS; +} + +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrGetStereoCameraStreamInfoDXR(XrStereoCameraStreamDXR stream, XrStereoCameraStreamInfoDXR *info) +{ + OXR_TRACE_MARKER(); + struct oxr_stereo_camera_stream_dxr *s = NULL; + struct oxr_logger log; + OXR_VERIFY_STEREO_CAMERA_STREAM_AND_INIT_LOG(&log, stream, s, "xrGetStereoCameraStreamInfoDXR"); + OXR_VERIFY_ARG_TYPE_AND_NOT_NULL(&log, info, XR_TYPE_STEREO_CAMERA_STREAM_INFO_DXR); + if (!s->started) { + return oxr_error(&log, XR_ERROR_CALL_ORDER_INVALID, "the stream has not been started"); + } + struct xrt_stereo_camera_client *c = client_of(s->inst); + struct xrt_stereo_camera_stream_layout lay; + xrt_shmem_handle_t section = XRT_SHMEM_HANDLE_INVALID; + xrt_graphics_sync_handle_t wake = XRT_GRAPHICS_SYNC_HANDLE_INVALID; + xrt_result_t xret = c->stream_get_transport(c, s->id, &lay, §ion, &wake); + if (xret != XRT_SUCCESS) { + return scam_xret(&log, xret, "xrGetStereoCameraStreamInfoDXR"); + } + info->extent.width = (int32_t)lay.width; + info->extent.height = (int32_t)lay.height; + info->format = (XrStereoCameraFormatDXR)lay.format; + info->output = (XrStereoCameraOutputDXR)lay.output; + info->transport = (XrStereoCameraTransportDXR)lay.transport; + info->maxFrameRate = lay.max_frame_rate; + + XrStereoCameraStreamTransportDXR *t = OXR_GET_OUTPUT_FROM_CHAIN( + info, XR_TYPE_STEREO_CAMERA_STREAM_TRANSPORT_DXR, XrStereoCameraStreamTransportDXR); + if (t != NULL) { + // NEW handles, now the caller's to close. + t->sectionHandle = (uint64_t)(uintptr_t)section; + t->sectionSize = lay.section_size; + t->slotCount = lay.slot_count; + t->slotStride = lay.slot_stride; + t->wakeHandle = (uint64_t)(uintptr_t)wake; + } else { + // Nobody to hand them to. +#if defined(XRT_OS_WINDOWS) + if (section != NULL) { + CloseHandle(section); + } + if (wake != NULL) { + CloseHandle(wake); + } +#else + if (section >= 0) { + close(section); + } + if (wake >= 0) { + close(wake); + } +#endif + } + return XR_SUCCESS; +} + +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrAcquireStereoCameraFrameDXR(XrStereoCameraStreamDXR stream, XrStereoCameraFrameDXR *frame) +{ + OXR_TRACE_MARKER(); + struct oxr_stereo_camera_stream_dxr *s = NULL; + struct oxr_logger log; + OXR_VERIFY_STEREO_CAMERA_STREAM_AND_INIT_LOG(&log, stream, s, "xrAcquireStereoCameraFrameDXR"); + OXR_VERIFY_ARG_TYPE_AND_NOT_NULL(&log, frame, XR_TYPE_STEREO_CAMERA_FRAME_DXR); + struct xrt_stereo_camera_client *c = client_of(s->inst); + bool ready = false; + struct xrt_stereo_camera_frame_info fi; + xrt_result_t xret = c->acquire(c, s->id, &ready, &fi); + if (xret != XRT_SUCCESS) { + return scam_xret(&log, xret, "xrAcquireStereoCameraFrameDXR"); + } + if (!ready) { + return XR_STEREO_CAMERA_FRAME_NOT_READY_DXR; + } + frame->frameIndex = fi.frame_index; + frame->slot = fi.slot; + frame->extent.width = (int32_t)fi.width; + frame->extent.height = (int32_t)fi.height; + frame->format = (XrStereoCameraFormatDXR)fi.format; + frame->rowPitch[0] = fi.row_pitch[0]; + frame->rowPitch[1] = fi.row_pitch[1]; + frame->planeOffset[0] = fi.plane_offset[0]; + frame->planeOffset[1] = fi.plane_offset[1]; + frame->captureTime = time_state_monotonic_to_ts_ns(s->inst->timekeeping, fi.capture_time_ns); + frame->captureTimeIsExposure = fi.time_is_exposure ? XR_TRUE : XR_FALSE; + frame->output = (XrStereoCameraOutputDXR)fi.output; + frame->calibrationGeneration = fi.calibration_generation; + return XR_SUCCESS; +} + +XRAPI_ATTR XrResult XRAPI_CALL +oxr_xrGetStereoCameraStreamStatsDXR(XrStereoCameraStreamDXR stream, XrStereoCameraStreamStatsDXR *stats) +{ + OXR_TRACE_MARKER(); + struct oxr_stereo_camera_stream_dxr *s = NULL; + struct oxr_logger log; + OXR_VERIFY_STEREO_CAMERA_STREAM_AND_INIT_LOG(&log, stream, s, "xrGetStereoCameraStreamStatsDXR"); + OXR_VERIFY_ARG_TYPE_AND_NOT_NULL(&log, stats, XR_TYPE_STEREO_CAMERA_STREAM_STATS_DXR); + struct xrt_stereo_camera_client *c = client_of(s->inst); + struct xrt_stereo_camera_stream_stats st; + xrt_result_t xret = c->stats(c, s->id, &st); + if (xret != XRT_SUCCESS) { + return scam_xret(&log, xret, "xrGetStereoCameraStreamStatsDXR"); + } + stats->sourceFrameRate = st.source_frame_rate; + stats->deliveredFrameRate = st.delivered_frame_rate; + stats->framesPublished = st.frames_published; + stats->framesSkipped = st.frames_skipped; + stats->framesAcquired = st.frames_acquired; + stats->meanLatencyNs = st.mean_latency_ns; + return XR_SUCCESS; +} + +#endif // OXR_HAVE_DXR_stereo_camera diff --git a/src/xrt/targets/cli/CMakeLists.txt b/src/xrt/targets/cli/CMakeLists.txt index 6778feaeed..2e5d64859a 100644 --- a/src/xrt/targets/cli/CMakeLists.txt +++ b/src/xrt/targets/cli/CMakeLists.txt @@ -43,6 +43,7 @@ add_executable( cli_cmd_test.c cli_cmd_clients.c cli_cmd_lift.cpp + cli_cmd_camera.c cli_common.h cli_main.c ) @@ -91,6 +92,8 @@ endif() # ADR-042: `displayxr-cli lift probe` loads / writes images (stb, header-only, # compiled static into cli_cmd_lift.cpp) and drives D3D11 shared textures. +# ADR-043: `displayxr-cli camera probe --out` writes PNGs with the same stb +# (compiled static into cli_cmd_camera.c). target_include_directories(cli PRIVATE ${PROJECT_SOURCE_DIR}/src/external/stb) if(WIN32) target_link_libraries(cli PRIVATE d3d11 dxgi) diff --git a/src/xrt/targets/cli/cli_cmd_camera.c b/src/xrt/targets/cli/cli_cmd_camera.c new file mode 100644 index 0000000000..1b2c20a492 --- /dev/null +++ b/src/xrt/targets/cli/cli_cmd_camera.c @@ -0,0 +1,663 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief `displayxr-cli camera` — XR_DXR_stereo_camera (ADR-043) over IPC. + * + * camera list [--json] properties + state of every camera + * camera calib [--raw|--rectified] [--json] + * camera probe [] [--raw] [--format gray8|nv12|bgra8] [--fps F] + * [--frames N] [--seconds S] [--out DIR] + * + * `probe` is a real consumer: it creates and starts a stream (so it goes + * through the service's authorisation — in R1, DXR_STEREO_CAMERA_DEV_ALLOW=1 + * on the SERVICE), maps the ring READ-ONLY, waits on the per-stream wake + * handle, acquires, and reports the measured delivery rate, frame-index gaps, + * the SBS layout, block-matched disparities of the last frame (the sim fake's + * scene is 12 px background / 40 px bar), and the service's stream stats. + * With --out it writes the last frame as cam_.png. + * + * Connects as a DIAG client, like `clients`: on Windows run it NON-elevated. + */ + +#include "cli_common.h" + +#ifdef CLI_HAVE_IPC + +#include "xrt/xrt_instance.h" +#include "xrt/xrt_plugin.h" +#include "xrt/xrt_results.h" +#include "xrt/xrt_stereo_camera.h" +#include "os/os_time.h" +#include "util/u_logging.h" +#include "util/u_stereo_camera.h" + +#include "client/ipc_client_connection.h" +#include "client/ipc_client.h" +#include "client/ipc_client_stereo_camera.h" + +#define STB_IMAGE_WRITE_STATIC +#define STB_IMAGE_WRITE_IMPLEMENTATION +#include "stb_image_write.h" + +#include +#include +#include +#include + +#ifdef XRT_OS_WINDOWS +#include +#else +#include +#include +#include +#endif + +static const char * +state_str(uint32_t s) +{ + switch (s) { + case XRT_STEREO_CAMERA_STATE_AVAILABLE: return "AVAILABLE"; + case XRT_STEREO_CAMERA_STATE_WAITING: return "WAITING"; + case XRT_STEREO_CAMERA_STATE_SUSPENDED: return "SUSPENDED"; + case XRT_STEREO_CAMERA_STATE_UNAVAILABLE: return "UNAVAILABLE"; + default: return "?"; + } +} + +static const char * +format_str(uint32_t f) +{ + switch (f) { + case XRT_STEREO_CAMERA_FORMAT_GRAY8: return "GRAY8"; + case XRT_STEREO_CAMERA_FORMAT_NV12: return "NV12"; + case XRT_STEREO_CAMERA_FORMAT_BGRA8: return "BGRA8"; + default: return "?"; + } +} + +static void +flags_str(uint64_t f, char *out, size_t cap) +{ + out[0] = '\0'; + static const struct + { + uint64_t bit; + const char *name; + } k[] = { + {XRT_PLUGIN_STEREO_CAMERA_SHARED_WITH_EYE_TRACKING, "SHARED_WITH_EYE_TRACKING"}, + {XRT_PLUGIN_STEREO_CAMERA_USER_FACING, "USER_FACING"}, + {XRT_PLUGIN_STEREO_CAMERA_CALIBRATED, "CALIBRATED"}, + {XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED, "NATIVELY_RECTIFIED"}, + {XRT_PLUGIN_STEREO_CAMERA_MONOCHROME, "MONOCHROME"}, + }; + for (size_t i = 0; i < sizeof(k) / sizeof(k[0]); i++) { + if (f & k[i].bit) { + size_t n = strlen(out); + snprintf(out + n, cap - n, "%s%s", n ? "|" : "", k[i].name); + } + } +} + +static const char * +result_hint(xrt_result_t x) +{ + switch (x) { + case XRT_ERROR_NOT_AUTHORIZED: + return " = NOT_AUTHORIZED (R1: set DXR_STEREO_CAMERA_DEV_ALLOW=1 in the SERVICE environment)"; + case XRT_ERROR_FEATURE_NOT_SUPPORTED: return " = FEATURE_NOT_SUPPORTED"; + case XRT_ERROR_INPUT_UNSUPPORTED: return " = INPUT_UNSUPPORTED (bad id / format / output)"; + case XRT_ERROR_CLIENT_LIMIT_REACHED: return " = LIMIT_REACHED (8 streams per camera)"; + case XRT_ERROR_IPC_FAILURE: return " = IPC_FAILURE"; + default: return ""; + } +} + +static bool +cam_connect(struct ipc_connection *ipc_c) +{ + struct xrt_instance_info ii = {0}; + snprintf(ii.app_info.application_name, sizeof(ii.app_info.application_name), "%s", "displayxr-cli"); + ii.app_info.declared_client_class = XRT_CLIENT_CLASS_DIAG; + xrt_result_t xret = ipc_client_connection_init(ipc_c, U_LOGGING_ERROR, &ii); + if (xret != XRT_SUCCESS) { + printf("displayxr-cli camera: not connected to the service (xrt_result=%d).\n", (int)xret); + printf(" Is displayxr-service running? On Windows, run from a NON-elevated prompt.\n"); + return false; + } + return true; +} + + +/* + * + * list / calib. + * + */ + +static int +cmd_list(struct ipc_connection *ipc_c, bool json) +{ + uint32_t n = 0; + xrt_result_t xret = ipc_client_stereo_camera_count(ipc_c, &n); + if (xret != XRT_SUCCESS) { + printf("stereo_camera_count failed: %d%s\n", (int)xret, result_hint(xret)); + return 2; + } + if (json) { + printf("{\"cameras\": ["); + } else { + printf("%u stereo camera(s)\n", n); + } + for (uint32_t i = 0; i < n; i++) { + struct xrt_stereo_camera_properties p; + xret = ipc_client_stereo_camera_get_properties(ipc_c, i, &p); + if (xret != XRT_SUCCESS) { + continue; + } + char flags[160]; + flags_str(p.flags, flags, sizeof(flags)); + if (json) { + printf( + "%s{\"id\": %llu, \"name\": \"%s\", \"persistentId\": \"%s\", \"state\": \"%s\", " + "\"flags\": \"%s\", \"viewCount\": %u, \"eyeWidth\": %u, \"eyeHeight\": %u, " + "\"maxFrameRate\": %.2f, \"baselineMm\": %.2f, \"horizontalFovDeg\": %.2f, " + "\"formats\": \"0x%llx\", \"transports\": \"0x%llx\", \"platformDeviceHint\": \"%s\"}", + i ? ", " : "", (unsigned long long)p.camera_id, p.display_name, p.persistent_id, + state_str(p.state), flags, p.view_count, p.eye_width, p.eye_height, p.max_frame_rate, + p.baseline_mm, p.horizontal_fov_deg, (unsigned long long)p.supported_formats, + (unsigned long long)p.supported_transports, p.platform_device_hint); + } else { + printf(" [%llu] \"%s\" %s\n", (unsigned long long)p.camera_id, p.display_name, + state_str(p.state)); + printf(" persistentId %s\n", p.persistent_id); + printf(" flags %s\n", flags); + char rate[64]; + if (p.max_frame_rate > 0.0f) { + snprintf(rate, sizeof(rate), "%.1f Hz", p.max_frame_rate); + } else { + snprintf(rate, sizeof(rate), "rate unknown (measured once a stream runs)"); + } + printf(" eye %ux%u (SBS %ux%u), %u views, %s\n", p.eye_width, p.eye_height, + 2 * p.eye_width, p.eye_height, p.view_count, rate); + printf(" baseline %.2f mm, HFOV %.2f deg per eye\n", p.baseline_mm, + p.horizontal_fov_deg); + printf(" formats 0x%llx transports 0x%llx platform hint \"%s\"\n", + (unsigned long long)p.supported_formats, (unsigned long long)p.supported_transports, + p.platform_device_hint); + } + } + if (json) { + printf("]}\n"); + } + return 0; +} + +static int +cmd_calib(struct ipc_connection *ipc_c, uint64_t id, uint32_t output, bool json) +{ + struct xrt_stereo_camera_calibration c; + xrt_result_t xret = ipc_client_stereo_camera_get_calibration(ipc_c, id, output, &c); + if (xret != XRT_SUCCESS) { + printf("stereo_camera_get_calibration(%llu) failed: %d%s\n", (unsigned long long)id, (int)xret, + result_hint(xret)); + return 2; + } + if (json) { + printf("{\"output\": \"%s\", \"baselineMm\": %.3f, \"eyes\": [", + output == XRT_STEREO_CAMERA_OUTPUT_RAW ? "RAW" : "RECTIFIED", c.baseline_mm); + for (int e = 0; e < 2; e++) { + printf( + "%s{\"w\": %u, \"h\": %u, \"fx\": %.3f, \"fy\": %.3f, \"cx\": %.3f, \"cy\": %.3f, " + "\"model\": %u}", + e ? ", " : "", c.eye[e].width, c.eye[e].height, c.eye[e].fx, c.eye[e].fy, c.eye[e].cx, + c.eye[e].cy, c.eye[e].model); + } + printf( + "], \"rightFromLeft\": {\"orientation\": [%.6f, %.6f, %.6f, %.6f], \"position\": [%.6f, " + "%.6f, %.6f]}}\n", + c.orientation[0], c.orientation[1], c.orientation[2], c.orientation[3], c.position[0], + c.position[1], c.position[2]); + return 0; + } + printf("calibration (%s), baseline %.3f mm\n", output == XRT_STEREO_CAMERA_OUTPUT_RAW ? "RAW" : "RECTIFIED", + c.baseline_mm); + for (int e = 0; e < 2; e++) { + printf(" %s eye: %ux%u fx %.3f fy %.3f cx %.3f cy %.3f model %u\n", e ? "right" : "left ", + c.eye[e].width, c.eye[e].height, c.eye[e].fx, c.eye[e].fy, c.eye[e].cx, c.eye[e].cy, + c.eye[e].model); + } + printf(" right camera in left frame: q (%.5f %.5f %.5f %.5f) p (%.5f %.5f %.5f) m\n", c.orientation[0], + c.orientation[1], c.orientation[2], c.orientation[3], c.position[0], c.position[1], c.position[2]); + return 0; +} + + +/* + * + * probe. + * + */ + +struct ro_map +{ + const uint8_t *ptr; + uint64_t size; +#ifdef XRT_OS_WINDOWS + HANDLE section; +#else + int section; +#endif +}; + +static bool +map_readonly(struct ro_map *m, xrt_shmem_handle_t h, uint64_t size) +{ + m->size = size; + m->section = h; +#ifdef XRT_OS_WINDOWS + m->ptr = (const uint8_t *)MapViewOfFile(h, FILE_MAP_READ, 0, 0, (SIZE_T)size); + return m->ptr != NULL; +#else + void *p = mmap(NULL, (size_t)size, PROT_READ, MAP_SHARED, h, 0); + m->ptr = p == MAP_FAILED ? NULL : (const uint8_t *)p; + return m->ptr != NULL; +#endif +} + +static void +unmap_readonly(struct ro_map *m) +{ +#ifdef XRT_OS_WINDOWS + if (m->ptr != NULL) { + UnmapViewOfFile(m->ptr); + } + if (m->section != NULL) { + CloseHandle(m->section); + } +#else + if (m->ptr != NULL) { + munmap((void *)m->ptr, (size_t)m->size); + } + if (m->section >= 0) { + close(m->section); + } +#endif + memset(m, 0, sizeof(*m)); +} + +//! Wait for the stream's wake handle, then drain it. true = signalled. +static bool +wait_wake(xrt_graphics_sync_handle_t w, int timeout_ms) +{ +#ifdef XRT_OS_WINDOWS + return WaitForSingleObject(w, (DWORD)timeout_ms) == WAIT_OBJECT_0; +#else + struct pollfd pfd = {.fd = w, .events = POLLIN, .revents = 0}; + if (poll(&pfd, 1, timeout_ms) <= 0) { + return false; + } + char buf[64]; + while (read(w, buf, sizeof(buf)) > 0) { + } + return true; +#endif +} + +static void +close_wake(xrt_graphics_sync_handle_t w) +{ +#ifdef XRT_OS_WINDOWS + if (w != NULL) { + CloseHandle(w); + } +#else + if (w >= 0) { + close(w); + } +#endif +} + +static int +cmp_int(const void *a, const void *b) +{ + return *(const int *)a - *(const int *)b; +} + +//! Nearest subject the probe expects: its disparity bounds the search. +#define PROBE_NEAREST_M 0.4 + +/*! + * Disparity search bound from the camera's own geometry: the disparity of a + * subject at PROBE_NEAREST_M, f * B / Z, with f from the per-eye HFOV. A fixed + * 64 px was fine for the 50 mm sim fake and useless on the 120 mm Leia SR + * tracking camera, where a face at 0.6 m is ~95 px and every block clamped at + * 64. Uncalibrated: a quarter of the eye width. Always within [64, 0.6 * eye]. + */ +static uint32_t +probe_max_disparity(const struct xrt_stereo_camera_properties *p, uint32_t eye_w) +{ + double d = eye_w / 4.0; + if (p != NULL && p->baseline_mm > 0.0f && p->horizontal_fov_deg > 1.0f && p->horizontal_fov_deg < 179.0f) { + double f = (eye_w / 2.0) / tan(p->horizontal_fov_deg * 0.5 * 3.14159265358979323846 / 180.0); + d = f * (p->baseline_mm / 1000.0) / PROBE_NEAREST_M; + } + double cap = eye_w * 0.6; + if (d > cap) { + d = cap; + } + return d < 64.0 ? 64u : (uint32_t)ceil(d); +} + +//! Block-match the last frame on a 64x64 grid and print the two dominant +//! disparities (the sim fake: background 12 px, bar 40 px). Blocks whose best +//! match sits ON the search bound are a clamp, not a measurement: counted and +//! reported, never folded into the modes. +static void +report_disparity(const uint8_t *gray, uint32_t pitch, uint32_t eye_w, uint32_t h, uint32_t max_d, char *out, size_t cap) +{ + int vals[512]; + int n = 0, at_edge = 0; + for (uint32_t y = 48; y + 64 <= h && n < 512; y += 64) { + for (uint32_t x = 64; x + 64 <= eye_w && n < 512; x += 64) { + float d = 0.0f; + if (u_stereo_camera_estimate_disparity(gray, pitch, eye_w, h, x, y, 64, 64, max_d, &d)) { + uint32_t lim = max_d < x ? max_d : x; + if (d >= (float)lim - 0.5f) { + at_edge++; + continue; + } + vals[n++] = (int)(d + 0.5f); + } + } + } + if (n == 0) { + snprintf(out, cap, "n/a (%s)", at_edge > 0 ? "every block hit the search bound" : "frame too small"); + return; + } + qsort(vals, (size_t)n, sizeof(int), cmp_int); + // Histogram modes. + int best_v[2] = {-1, -1}, best_c[2] = {0, 0}; + for (int i = 0; i < n;) { + int j = i; + while (j < n && vals[j] == vals[i]) { + j++; + } + int c = j - i; + if (c > best_c[0]) { + best_v[1] = best_v[0]; + best_c[1] = best_c[0]; + best_v[0] = vals[i]; + best_c[0] = c; + } else if (c > best_c[1]) { + best_v[1] = vals[i]; + best_c[1] = c; + } + i = j; + } + int k = snprintf(out, cap, "%d px (%d/%d blocks), %d px (%d/%d blocks), range %d..%d px (search 0..%u)", + best_v[0], best_c[0], n, best_v[1], best_c[1], n, vals[0], vals[n - 1], max_d); + if (at_edge > 0 && k > 0 && (size_t)k < cap) { + snprintf(out + k, cap - (size_t)k, "; %d block(s) AT THE SEARCH BOUND, excluded", at_edge); + } +} + +static int +cmd_probe(struct ipc_connection *ipc_c, int argc, const char **argv) +{ + uint64_t id = 0; + uint32_t output = XRT_STEREO_CAMERA_OUTPUT_RECTIFIED; + uint32_t format = XRT_STEREO_CAMERA_FORMAT_NV12; + float fps = 0.0f; + int frames = 90; + float seconds = 0.0f; + const char *out_dir = NULL; + for (int i = 3; i < argc; i++) { + if (strcmp(argv[i], "--raw") == 0) { + output = XRT_STEREO_CAMERA_OUTPUT_RAW; + } else if (strcmp(argv[i], "--format") == 0 && i + 1 < argc) { + const char *f = argv[++i]; + format = strcmp(f, "gray8") == 0 ? XRT_STEREO_CAMERA_FORMAT_GRAY8 + : strcmp(f, "bgra8") == 0 ? XRT_STEREO_CAMERA_FORMAT_BGRA8 + : XRT_STEREO_CAMERA_FORMAT_NV12; + } else if (strcmp(argv[i], "--fps") == 0 && i + 1 < argc) { + fps = (float)atof(argv[++i]); + } else if (strcmp(argv[i], "--frames") == 0 && i + 1 < argc) { + frames = atoi(argv[++i]); + } else if (strcmp(argv[i], "--seconds") == 0 && i + 1 < argc) { + seconds = (float)atof(argv[++i]); + } else if (strcmp(argv[i], "--out") == 0 && i + 1 < argc) { + out_dir = argv[++i]; + } else if (argv[i][0] != '-') { + id = strtoull(argv[i], NULL, 10); + } + } + struct xrt_stereo_camera_properties props; + memset(&props, 0, sizeof(props)); + { + uint32_t n = 0; + bool found = false; + if (ipc_client_stereo_camera_count(ipc_c, &n) == XRT_SUCCESS) { + for (uint32_t i = 0; i < n && !found; i++) { + if (ipc_client_stereo_camera_get_properties(ipc_c, i, &props) == XRT_SUCCESS && + (id == 0 || props.camera_id == id)) { + found = true; + } + } + } + if (!found && id == 0) { + printf("camera probe: the service exposes no stereo camera.\n"); + return 3; + } + if (!found) { + memset(&props, 0, sizeof(props)); // stream_create reports the bad id + } + id = found ? props.camera_id : id; + } + + struct xrt_stereo_camera_stream_request req = { + .camera_id = id, + .output = output, + .format = format, + .transport = XRT_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY, + .max_frame_rate = fps, + }; + uint64_t sid = 0; + xrt_result_t xret = ipc_client_stereo_camera_stream_create(ipc_c, &req, &sid); + if (xret != XRT_SUCCESS) { + printf("stream create failed: %d%s\n", (int)xret, result_hint(xret)); + return 2; + } + xret = ipc_client_stereo_camera_stream_start(ipc_c, sid); + if (xret != XRT_SUCCESS) { + printf("stream start failed: %d%s\n", (int)xret, result_hint(xret)); + ipc_client_stereo_camera_stream_destroy(ipc_c, sid); + return 2; + } + struct xrt_stereo_camera_stream_layout lay; + xrt_shmem_handle_t section; + xrt_graphics_sync_handle_t wake; + xret = ipc_client_stereo_camera_stream_get_transport(ipc_c, sid, &lay, §ion, &wake); + if (xret != XRT_SUCCESS) { + printf("stream transport failed: %d%s\n", (int)xret, result_hint(xret)); + ipc_client_stereo_camera_stream_destroy(ipc_c, sid); + return 2; + } + struct ro_map map; + memset(&map, 0, sizeof(map)); + if (!map_readonly(&map, section, lay.section_size)) { + printf("could not map the ring read-only\n"); + close_wake(wake); + ipc_client_stereo_camera_stream_destroy(ipc_c, sid); + return 2; + } + char cap[48]; + if (lay.max_frame_rate > 0.0f) { + snprintf(cap, sizeof(cap), "cap %.1f Hz", lay.max_frame_rate); + } else { + snprintf(cap, sizeof(cap), "cap = source rate (not measured yet)"); + } + printf("stream %llu on camera %llu: %ux%u %s SBS (%s), %s, ring %u x %llu B (section %llu B)\n", + (unsigned long long)sid, (unsigned long long)id, lay.width, lay.height, format_str(lay.format), + lay.output == XRT_STEREO_CAMERA_OUTPUT_RECTIFIED ? "RECTIFIED" : "RAW (flagged)", cap, lay.slot_count, + (unsigned long long)lay.slot_stride, (unsigned long long)lay.section_size); + + int64_t t_start = os_monotonic_get_ns(); + int64_t deadline = seconds > 0.0f ? t_start + (int64_t)(seconds * 1e9) : t_start + 60ll * 1000000000; + int got = 0, wakes = 0, not_ready = 0; + uint64_t first_index = 0, last_index = 0, gaps = 0; + int64_t t_first = 0, t_last = 0; + struct xrt_stereo_camera_frame_info last; + memset(&last, 0, sizeof(last)); + while ((seconds > 0.0f || got < frames) && os_monotonic_get_ns() < deadline) { + if (!wait_wake(wake, 1000)) { + continue; + } + wakes++; + bool ready = false; + struct xrt_stereo_camera_frame_info fi; + xret = ipc_client_stereo_camera_acquire(ipc_c, sid, &ready, &fi); + if (xret != XRT_SUCCESS) { + printf("acquire failed: %d%s\n", (int)xret, result_hint(xret)); + break; + } + if (!ready) { + not_ready++; + continue; + } + int64_t now = os_monotonic_get_ns(); + if (got == 0) { + first_index = fi.frame_index; + t_first = now; + } else if (fi.frame_index > last_index + 1) { + gaps += fi.frame_index - last_index - 1; + } + last_index = fi.frame_index; + t_last = now; + last = fi; + got++; + } + double span = (double)(t_last - t_first) * 1e-9; + double rate = (got > 1 && span > 0.0) ? (got - 1) / span : 0.0; + printf( + "received %d frames (index %llu..%llu, %llu source frames not delivered to this stream), %d wakes, " + "%d not-ready\n", + got, (unsigned long long)first_index, (unsigned long long)last_index, (unsigned long long)gaps, wakes, + not_ready); + printf("measured delivery rate: %.2f Hz over %.2f s\n", rate, span); + + if (got > 0) { + const uint8_t *slot = map.ptr + (uint64_t)last.slot * lay.slot_stride; + printf( + "last frame: index %llu, slot %u, %ux%u %s, pitch %u/%u, planes @%llu/%llu, %s time, output %s, " + "calibration gen %u\n", + (unsigned long long)last.frame_index, last.slot, last.width, last.height, format_str(last.format), + last.row_pitch[0], last.row_pitch[1], (unsigned long long)last.plane_offset[0], + (unsigned long long)last.plane_offset[1], last.time_is_exposure ? "exposure" : "arrival", + last.output == XRT_STEREO_CAMERA_OUTPUT_RECTIFIED ? "RECTIFIED" : "RAW", + last.calibration_generation); + + // Luma of the last frame, for the disparity probe and a GRAY8 PNG. + struct u_stereo_camera_planes gl; + u_stereo_camera_layout(XRT_STEREO_CAMERA_FORMAT_GRAY8, last.width, last.height, &gl); + uint8_t *gray = calloc((size_t)gl.size, 1); + const uint8_t *src[2] = {slot + last.plane_offset[0], slot + last.plane_offset[1]}; + if (gray != NULL && + u_stereo_camera_convert(last.format, src, last.row_pitch, XRT_STEREO_CAMERA_FORMAT_GRAY8, gray, &gl, + last.width, last.height)) { + char disp[256]; + report_disparity(gray, gl.pitch[0], last.width / 2, last.height, + probe_max_disparity(&props, last.width / 2), disp, sizeof(disp)); + printf("block disparity (left x - right x): %s\n", disp); + } + if (out_dir != NULL) { + char path[1024]; + snprintf(path, sizeof(path), "%s/cam_%llu.png", out_dir, (unsigned long long)last.frame_index); + int ok = 0; + if (last.format == XRT_STEREO_CAMERA_FORMAT_GRAY8 && gray != NULL) { + ok = stbi_write_png(path, (int)last.width, (int)last.height, 1, gray, (int)gl.pitch[0]); + } else { + struct u_stereo_camera_planes bl; + u_stereo_camera_layout(XRT_STEREO_CAMERA_FORMAT_BGRA8, last.width, last.height, &bl); + uint8_t *rgba = calloc((size_t)bl.size, 1); + if (rgba != NULL && u_stereo_camera_convert(last.format, src, last.row_pitch, + XRT_STEREO_CAMERA_FORMAT_BGRA8, rgba, &bl, + last.width, last.height)) { + for (uint32_t y = 0; y < last.height; y++) { + uint8_t *r = rgba + (size_t)y * bl.pitch[0]; + for (uint32_t x = 0; x < last.width; x++) { + uint8_t t = r[4 * x]; + r[4 * x] = r[4 * x + 2]; + r[4 * x + 2] = t; + } + } + ok = stbi_write_png(path, (int)last.width, (int)last.height, 4, rgba, + (int)bl.pitch[0]); + } + free(rgba); + } + printf("%s %s\n", ok ? "wrote" : "FAILED to write", path); + } + free(gray); + } + + struct xrt_stereo_camera_stream_stats st; + if (ipc_client_stereo_camera_stats(ipc_c, sid, &st) == XRT_SUCCESS) { + printf( + "service stats: source %.2f Hz, delivered %.2f Hz, published %llu, skipped %llu, acquired %llu, " + "mean latency %.3f ms\n", + st.source_frame_rate, st.delivered_frame_rate, (unsigned long long)st.frames_published, + (unsigned long long)st.frames_skipped, (unsigned long long)st.frames_acquired, + (double)st.mean_latency_ns * 1e-6); + } + + unmap_readonly(&map); + close_wake(wake); + ipc_client_stereo_camera_stream_stop(ipc_c, sid); + ipc_client_stereo_camera_stream_destroy(ipc_c, sid); + return got > 0 ? 0 : 4; +} + +int +cli_cmd_camera(int argc, const char **argv) +{ + const char *sub = argc >= 3 ? argv[2] : "list"; + bool json = cli_has_flag(argc, argv, "--json"); + struct ipc_connection ipc_c = {0}; + if (!cam_connect(&ipc_c)) { + return 2; + } + int ret; + if (strcmp(sub, "list") == 0) { + ret = cmd_list(&ipc_c, json); + } else if (strcmp(sub, "calib") == 0 && argc >= 4) { + uint32_t output = cli_has_flag(argc, argv, "--rectified") ? XRT_STEREO_CAMERA_OUTPUT_RECTIFIED + : XRT_STEREO_CAMERA_OUTPUT_RAW; + ret = cmd_calib(&ipc_c, strtoull(argv[3], NULL, 10), output, json); + } else if (strcmp(sub, "probe") == 0) { + ret = cmd_probe(&ipc_c, argc, argv); + } else { + printf( + "usage: displayxr-cli camera list [--json] | calib [--raw|--rectified] [--json] |\n" + " probe [] [--raw] [--format gray8|nv12|bgra8] [--fps F] [--frames N] [--seconds S] " + "[--out DIR]\n"); + ret = 1; + } + ipc_client_connection_fini(&ipc_c); + return ret; +} + +#else // !CLI_HAVE_IPC + +#include + +int +cli_cmd_camera(int argc, const char **argv) +{ + (void)argc; + (void)argv; + printf("displayxr-cli camera: this build has no IPC client (XRT_FEATURE_SERVICE off).\n"); + return 1; +} + +#endif diff --git a/src/xrt/targets/cli/cli_common.h b/src/xrt/targets/cli/cli_common.h index 9d32e5ab8f..9c6ec7d4b9 100644 --- a/src/xrt/targets/cli/cli_common.h +++ b/src/xrt/targets/cli/cli_common.h @@ -52,6 +52,10 @@ cli_cmd_test(int argc, const char **argv); int cli_cmd_lift(int argc, const char **argv); +//! `camera` — XR_DXR_stereo_camera (ADR-043): `camera list|calib|probe`, over IPC (DIAG). +int +cli_cmd_camera(int argc, const char **argv); + /*! * True if @p flag appears anywhere in argv[2..]. Commands take raw * argc/argv; this is the one shared option-scan (e.g. `--json`). diff --git a/src/xrt/targets/cli/cli_main.c b/src/xrt/targets/cli/cli_main.c index eb017a351a..34412613a7 100644 --- a/src/xrt/targets/cli/cli_main.c +++ b/src/xrt/targets/cli/cli_main.c @@ -51,6 +51,10 @@ cli_print_help(int argc, const char **argv) P(" lift <...> - 2D->3D conversion module (XR_DXR_lift, ADR-042), over IPC (DIAG).\n"); P(" 'lift caps [--json]', 'lift probe [--mode depth|sbs|\n"); P(" nview|gaussians] [--n N]' (Windows) — writes lift_out_.png/.ply.\n"); + P(" camera <...> - Stereo camera sources (XR_DXR_stereo_camera, ADR-043), over IPC (DIAG).\n"); + P(" 'camera list [--json]', 'camera calib [--raw|--rectified]',\n"); + P(" 'camera probe [] [--raw] [--format gray8|nv12|bgra8] [--fps F]\n"); + P(" [--frames N] [--seconds S] [--out DIR]' — rate, layout, disparity, PNG.\n"); P(" test - List found devices and role assignments, for prober testing.\n"); P(" probe - Just probe and then exit.\n"); @@ -106,5 +110,8 @@ main(int argc, const char **argv) if (strcmp(argv[1], "lift") == 0) { return cli_cmd_lift(argc, argv); } + if (strcmp(argv[1], "camera") == 0) { + return cli_cmd_camera(argc, argv); + } return cli_print_help(argc, argv); } diff --git a/src/xrt/targets/cli/cli_query.c b/src/xrt/targets/cli/cli_query.c index 29d325bdf0..e3fc04875c 100644 --- a/src/xrt/targets/cli/cli_query.c +++ b/src/xrt/targets/cli/cli_query.c @@ -71,7 +71,70 @@ or_q(const char *s) return (s != NULL && s[0] != '\0') ? s : "?"; } +/*! + * ADR-043 `stereo_camera_caps`: enumerate the plug-in's stereo cameras straight + * through the iface (this is the in-process no-comp instance — the service's + * camera manager is not involved) and validate each description. + */ +static void +probe_stereo_camera(struct cli_query_result *r, const struct xrt_plugin_iface *iface) +{ + r->stereo_camera_count = 0; + r->stereo_camera_malformed = false; + if (!xrt_plugin_iface_has_stereo_camera(iface)) { + snprintf(r->stereo_camera_note, sizeof(r->stereo_camera_note), + "plug-in has no stereo camera slots (OK)"); + return; + } + struct xrt_plugin_instance *inst = target_plugin_get_active_instance(); + struct xrt_plugin_stereo_camera_info infos[4]; + memset(infos, 0, sizeof(infos)); + for (int i = 0; i < 4; i++) { + infos[i].struct_size = (uint32_t)sizeof(infos[i]); + } + uint32_t n = iface->stereo_camera_enumerate(inst, 4, infos); + r->stereo_camera_count = n; + if (n == 0) { + snprintf(r->stereo_camera_note, sizeof(r->stereo_camera_note), "0 cameras (OK)"); + return; + } + for (uint32_t i = 0; i < n && i < 4; i++) { + const struct xrt_plugin_stereo_camera_info *c = &infos[i]; + const char *why = NULL; + if (c->eye_width == 0 || c->eye_height == 0 || (c->eye_width & 1u) || (c->eye_height & 1u)) { + why = "zero or odd eye extent"; + } else if (c->native_format < 1 || c->native_format > 3) { + why = "unknown native format"; + } else if (!(c->max_frame_rate >= 0.0f && c->max_frame_rate <= 1000.0f)) { + // 0 is legal: "unknown until measured" (the service measures it). + why = "max_frame_rate negative, NaN or > 1000"; + } else if (c->flags & XRT_PLUGIN_STEREO_CAMERA_CALIBRATED) { + struct xrt_plugin_stereo_camera_calibration k; + memset(&k, 0, sizeof(k)); + k.struct_size = (uint32_t)sizeof(k); + const double *t = k.translation_right_from_left_mm; + if (iface->stereo_camera_get_calibration(inst, i, &k) != XRT_SUCCESS) { + why = "CALIBRATED but get_calibration failed"; + } else if (!(k.k[0][0] > 0.0) || !(k.k[1][0] > 0.0) || + !(t[0] * t[0] + t[1] * t[1] + t[2] * t[2] > 0.0)) { + why = "CALIBRATED with a non-positive focal length or zero baseline"; + } + } + if (why != NULL) { + r->stereo_camera_malformed = true; + snprintf(r->stereo_camera_note, sizeof(r->stereo_camera_note), "camera %u \"%.60s\": %s", i, + c->display_name, why); + return; + } + } + snprintf(r->stereo_camera_note, sizeof(r->stereo_camera_note), + "%u camera(s); [0] \"%.60s\" %ux%u/eye, advertised %.1f Hz (0 = unknown), flags 0x%x", n, + infos[0].display_name, infos[0].eye_width, infos[0].eye_height, (double)infos[0].max_frame_rate, + infos[0].flags); +} + #ifdef XRT_OS_WINDOWS + /*! * #224 / ADR-027 P4 — headless zone-caps probe. Creates a D3D11 WARP device * (no GPU / display required), asks the active plug-in's D3D11 DP factory @@ -1697,6 +1760,13 @@ cli_query_fill(struct cli_query_result *r, struct cli_query_handles *h, const st "not probed: the lift module is a D3D11-service feature (Windows-only); modes 0 (OK)"); #endif + // ADR-043 — stereo camera slots. Absence never fails; only a camera the + // plug-in describes out of contract flips the verdict. + probe_stereo_camera(r, iface); + if (r->stereo_camera_malformed && r->result_code == CLI_SELFTEST_PASS) { + r->result_code = CLI_SELFTEST_BAD_STEREO_CAMERA; + } + // #1234 / #902 — can the Vulkan loader still reach the queue-lock layer? // Pure reporting: never touches result_code. See the field comment in // cli_query.h for why this must not be fatal. @@ -2653,6 +2723,13 @@ build_checks(const struct cli_query_result *r, struct check *out) // #224 / ADR-027 P4 — zone-caps probe. ABSENCE NEVER FAILS: ok stays // true for legacy plug-ins / no factory / non-Windows; only a // present-but-malformed caps struct fails (BAD_ZONE_CAPS). + // ADR-043 — stereo camera slots. ABSENCE NEVER FAILS. + c = &out[n++]; + c->name = "stereo_camera_caps"; + c->ok = !r->stereo_camera_malformed; + snprintf(c->detail, sizeof(c->detail), "%s", + r->stereo_camera_note[0] != '\0' ? r->stereo_camera_note : "not evaluated"); + c = &out[n++]; c->name = "zone_caps"; c->ok = !r->zone_caps_malformed; diff --git a/src/xrt/targets/cli/cli_query.h b/src/xrt/targets/cli/cli_query.h index 44565dd558..2a5c1f7524 100644 --- a/src/xrt/targets/cli/cli_query.h +++ b/src/xrt/targets/cli/cli_query.h @@ -119,6 +119,15 @@ enum cli_selftest_result * one, and of sim_display — passes. */ CLI_SELFTEST_BAD_LIFT_CAPS = 11, + + /*! + * The active plug-in fills the stereo camera slots (ADR-043) and one of + * its cameras is described out of contract: zero / odd eye extent, an + * unknown native format, a non-positive frame rate, or CALIBRATED with no + * (or a degenerate) calibration. ABSENCE NEVER FAILS: no slots or zero + * cameras is the state of almost every box. + */ + CLI_SELFTEST_BAD_STEREO_CAMERA = 12, }; //! Hardware adapters reported by the GPU-topology probe (#918). @@ -283,6 +292,12 @@ struct cli_query_result bool zone_caps_probed; bool zone_caps_malformed; char zone_probe_note[128]; + + /* Stereo camera slots (ADR-043, `stereo_camera_caps`). Absence never + * fails; only an out-of-contract camera description does. */ + uint32_t stereo_camera_count; + bool stereo_camera_malformed; + char stereo_camera_note[200]; struct xrt_dp_local_zone_caps zone_caps; /* ADR-042 lift-caps probe (Windows-only: WARP D3D11 device + the plug-in's diff --git a/src/xrt/targets/common/target_instance.c b/src/xrt/targets/common/target_instance.c index c8779b399f..e0dfbddd91 100644 --- a/src/xrt/targets/common/target_instance.c +++ b/src/xrt/targets/common/target_instance.c @@ -935,6 +935,22 @@ t_instance_create_system(struct xrt_instance *xinst, * */ +/*! + * xrt_instance::get_active_plugin — the plug-in that claimed the system. The + * service's stereo camera manager (ADR-043) reads its camera slots through it. + */ +static bool +t_instance_get_active_plugin(struct xrt_instance *xinst, + const struct xrt_plugin_iface **out_iface, + struct xrt_plugin_instance **out_inst) +{ + (void)xinst; + const struct xrt_plugin_iface *iface = target_plugin_get_active(); + *out_iface = iface; + *out_inst = iface != NULL ? target_plugin_get_active_instance() : NULL; + return iface != NULL; +} + #ifdef XRT_FEATURE_HYBRID_MODE // In hybrid mode, export as native_instance_create to avoid symbol conflict // with ipc_instance_create from the IPC client library @@ -960,6 +976,7 @@ xrt_instance_create(struct xrt_instance_info *ii, struct xrt_instance **out_xins tinst->base.create_system = t_instance_create_system; tinst->base.get_prober = t_instance_get_prober; tinst->base.destroy = t_instance_destroy; + tinst->base.get_active_plugin = t_instance_get_active_plugin; tinst->xp = xp; tinst->base.startup_timestamp = os_monotonic_get_ns(); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 9ab14e6071..f3c18c6555 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -66,6 +66,7 @@ set(tests tests_comp_layer_view_camera tests_comp_color_policy tests_m_matrix_projection + tests_stereo_camera ) if(XRT_BUILD_DRIVER_QWERTY) list(APPEND tests tests_qwerty_camera_profile tests_qwerty_velocity) @@ -702,3 +703,8 @@ add_test(NAME tests_ipc_proto COMMAND ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURC # lift mailbox tests (the module that emits it is Windows/D3D11-only). target_sources(tests_lift_mailbox PRIVATE ${PROJECT_SOURCE_DIR}/src/xrt/drivers/sim_display/sim_display_fake_ply.c) target_include_directories(tests_lift_mailbox PRIVATE ${PROJECT_SOURCE_DIR}/src/xrt/drivers/sim_display) + +# ADR-043: the sim_display FAKE stereo camera's scene is pinned host-side next to +# the ring / conversion / disparity-probe tests (it compiles into the plug-in). +target_sources(tests_stereo_camera PRIVATE ${PROJECT_SOURCE_DIR}/src/xrt/drivers/sim_display/sim_display_stereo_camera_pattern.c) +target_include_directories(tests_stereo_camera PRIVATE ${PROJECT_SOURCE_DIR}/src/xrt/drivers/sim_display) diff --git a/tests/tests_stereo_camera.cpp b/tests/tests_stereo_camera.cpp new file mode 100644 index 0000000000..1829067169 --- /dev/null +++ b/tests/tests_stereo_camera.cpp @@ -0,0 +1,329 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera (ADR-043) R1: the platform-neutral pieces the + * service's camera manager is built from, pinned host-side. + * + * - the per-stream latest-wins ring: a pinned slot is never written, the + * writer always finds a free slot, NOT READY until something newer lands, + * skipped frames are counted, suspension clears the pin; + * - the decimator: exact ratios (30 -> 15, 30 -> 20), no burst after a pause; + * - plane layout + format conversion round trips; + * - the persistent id: stable, per consumer, never the raw identity; + * - the sim_display FAKE scene: the disparity probe measures exactly the + * disparities the scene was built with, and the frame counter differs + * between frames; + * - the plug-in iface: the camera slots are appended after the ADR-042 lift + * slot and the ADR-045 platform-state slot (ADR-020 append-only), and end + * the struct. + */ + +#include "catch_amalgamated.hpp" + +#include "util/u_stereo_camera.h" +#include "xrt/xrt_plugin.h" +#include "xrt/xrt_stereo_camera.h" + +#include "sim_display_stereo_camera_pattern.h" + +#include +#include +#include +#include +#include + +TEST_CASE("stereo camera ring: latest-wins with one pin", "[stereo_camera]") +{ + u_stereo_camera_ring r; + u_stereo_camera_ring_init(&r); + + int32_t slot = -1; + uint64_t seq = 0; + CHECK_FALSE(u_stereo_camera_ring_acquire(&r, &slot, &seq)); // nothing yet + + int32_t w1 = u_stereo_camera_ring_begin_write(&r); + REQUIRE(w1 >= 0); + u_stereo_camera_ring_publish(&r, w1, 1); + REQUIRE(u_stereo_camera_ring_acquire(&r, &slot, &seq)); + CHECK(slot == w1); + CHECK(seq == 1); + CHECK_FALSE(u_stereo_camera_ring_acquire(&r, &slot, &seq)); // not newer + + // Many publishes while the consumer holds its pin: the writer must never + // be handed the pinned slot, and every unacquired frame counts as skipped. + for (uint64_t s = 2; s < 50; s++) { + int32_t w = u_stereo_camera_ring_begin_write(&r); + REQUIRE(w >= 0); + REQUIRE(w != r.pinned); + REQUIRE(w != r.latest); + u_stereo_camera_ring_publish(&r, w, s); + } + CHECK(r.skipped == 47); // 2..48 superseded, 49 is the latest + REQUIRE(u_stereo_camera_ring_acquire(&r, &slot, &seq)); + CHECK(seq == 49); + CHECK(r.acquired == 2); + + // Suspension: pin dropped, latest forgotten, next publish is not a skip. + u_stereo_camera_ring_clear(&r); + CHECK(r.pinned == -1); + CHECK_FALSE(u_stereo_camera_ring_acquire(&r, &slot, &seq)); + int32_t w = u_stereo_camera_ring_begin_write(&r); + u_stereo_camera_ring_publish(&r, w, 50); + CHECK(r.skipped == 47); + REQUIRE(u_stereo_camera_ring_acquire(&r, &slot, &seq)); + CHECK(seq == 50); +} + +TEST_CASE("stereo camera ring: a pinned slot stays byte-stable", "[stereo_camera]") +{ + // Simulate the payload: each slot holds the sequence last written into it. + u_stereo_camera_ring r; + u_stereo_camera_ring_init(&r); + uint64_t payload[U_STEREO_CAMERA_RING_SLOTS] = {0, 0, 0}; + int32_t slot = -1; + uint64_t seq = 0, held = 0; + int32_t held_slot = -1; + for (uint64_t s = 1; s <= 1000; s++) { + int32_t w = u_stereo_camera_ring_begin_write(&r); + payload[w] = s; + u_stereo_camera_ring_publish(&r, w, s); + if (held_slot >= 0) { + REQUIRE(payload[held_slot] == held); // never overwritten under the pin + } + if (s % 7 == 0 && u_stereo_camera_ring_acquire(&r, &slot, &seq)) { + REQUIRE(payload[slot] == seq); + held = seq; + held_slot = slot; + } + } +} + +TEST_CASE("stereo camera decimator", "[stereo_camera]") +{ + const int64_t p30 = 33333333; + auto count = [&](float max_rate, int frames, int64_t pause_at = -1) { + u_stereo_camera_decimator d; + u_stereo_camera_decimator_init(&d, max_rate, 30.0f); + int n = 0; + int64_t t = 1000000000; + for (int i = 0; i < frames; i++) { + if (i == pause_at) { + t += 2000000000; // a 2 s gap (suspended source) + } + n += u_stereo_camera_decimator_accept(&d, t) ? 1 : 0; + t += p30; + } + return n; + }; + CHECK(count(0.0f, 300) == 300); + CHECK(count(30.0f, 300) == 300); + CHECK(count(60.0f, 300) == 300); // never interpolates up + CHECK(count(15.0f, 300) == 150); + CHECK(count(20.0f, 300) == 200); + CHECK(count(10.0f, 300) == 100); + // After a pause the gate re-anchors: no burst of "missed" frames. + int with_pause = count(15.0f, 300, 150); + CHECK(with_pause >= 150); + CHECK(with_pause <= 151); +} + +TEST_CASE("stereo camera rate meter measures, never trusts the advert", "[stereo_camera]") +{ + u_stereo_camera_rate_meter m = {}; + CHECK(m.rate == 0.0f); // unknown until a window completes + // A 60 Hz source with +-2 ms arrival jitter (the SR tracker's shape). + int64_t t = 1000000000; + int completed = 0; + for (int i = 0; i <= U_STEREO_CAMERA_RATE_WINDOW; i++) { + int64_t jitter = (i % 2 == 0 ? 1 : -1) * 2000000; + completed += + u_stereo_camera_rate_meter_push(&m, t + (i == 0 || i == U_STEREO_CAMERA_RATE_WINDOW ? 0 : jitter)) + ? 1 + : 0; + t += 16666667; + } + CHECK(completed == 1); + CHECK(m.rate == Catch::Approx(60.0f).margin(0.05f)); + + // A suspension gap restarts the window instead of dragging the rate down. + u_stereo_camera_rate_meter g = {}; + t = 1000000000; + for (int i = 0; i < 30; i++, t += 33333333) { + u_stereo_camera_rate_meter_push(&g, t); + } + t += 3000000000ll; + bool done = false; + for (int i = 0; i <= U_STEREO_CAMERA_RATE_WINDOW; i++, t += 33333333) { + done = u_stereo_camera_rate_meter_push(&g, t) || done; + } + CHECK(done); + CHECK(g.rate == Catch::Approx(30.0f).margin(0.05f)); + + // Restart (source re-opened) keeps the last measurement. + u_stereo_camera_rate_meter_restart(&g); + CHECK(g.rate == Catch::Approx(30.0f).margin(0.05f)); +} + +TEST_CASE("stereo camera layout", "[stereo_camera]") +{ + u_stereo_camera_planes l; + REQUIRE(u_stereo_camera_layout(1, 1280, 480, &l)); + CHECK(l.plane_count == 1); + CHECK(l.pitch[0] == 1280); + CHECK(l.size == 1280ull * 480); + REQUIRE(u_stereo_camera_layout(2, 1280, 480, &l)); + CHECK(l.plane_count == 2); + CHECK(l.offset[1] == 1280ull * 480); + CHECK(l.size == 1280ull * 480 * 3 / 2); + REQUIRE(u_stereo_camera_layout(3, 1000, 10, &l)); + CHECK(l.pitch[0] == 4032); // 4000 rounded up to 64 + CHECK_FALSE(u_stereo_camera_layout(2, 1281, 480, &l)); // odd NV12 + CHECK_FALSE(u_stereo_camera_layout(9, 64, 64, &l)); +} + +TEST_CASE("stereo camera format conversion", "[stereo_camera]") +{ + const uint32_t W = 64, H = 8; + std::vector gray(W * H); + for (uint32_t i = 0; i < W * H; i++) { + gray[i] = (uint8_t)(i * 7); + } + const uint8_t *src[2] = {gray.data(), nullptr}; + const uint32_t sp[2] = {W, 0}; + + SECTION("gray -> nv12 keeps luma, neutral chroma") + { + u_stereo_camera_planes l; + REQUIRE(u_stereo_camera_layout(2, W, H, &l)); + std::vector out(l.size); + REQUIRE(u_stereo_camera_convert(1, src, sp, 2, out.data(), &l, W, H)); + CHECK(std::memcmp(out.data(), gray.data(), W) == 0); + for (uint64_t i = l.offset[1]; i < l.size; i++) { + REQUIRE(out[i] == 128); + } + // and back to gray + u_stereo_camera_planes g; + REQUIRE(u_stereo_camera_layout(1, W, H, &g)); + std::vector back(g.size); + const uint8_t *nsrc[2] = {out.data(), out.data() + l.offset[1]}; + REQUIRE(u_stereo_camera_convert(2, nsrc, l.pitch, 1, back.data(), &g, W, H)); + CHECK(std::memcmp(back.data(), gray.data(), W * H) == 0); + } + SECTION("gray -> bgra -> gray is lossless") + { + u_stereo_camera_planes b; + REQUIRE(u_stereo_camera_layout(3, W, H, &b)); + std::vector bgra(b.size); + REQUIRE(u_stereo_camera_convert(1, src, sp, 3, bgra.data(), &b, W, H)); + CHECK(bgra[3] == 255); + u_stereo_camera_planes g; + REQUIRE(u_stereo_camera_layout(1, W, H, &g)); + std::vector back(g.size); + const uint8_t *bsrc[2] = {bgra.data(), nullptr}; + REQUIRE(u_stereo_camera_convert(3, bsrc, b.pitch, 1, back.data(), &g, W, H)); + CHECK(std::memcmp(back.data(), gray.data(), W * H) == 0); + } + SECTION("bgra -> nv12 -> bgra stays within video-range rounding") + { + u_stereo_camera_planes b; + REQUIRE(u_stereo_camera_layout(3, W, H, &b)); + std::vector bgra(b.size, 0); + for (uint32_t y = 0; y < H; y++) { + for (uint32_t x = 0; x < W; x++) { + uint8_t *p = &bgra[y * b.pitch[0] + 4 * x]; + p[0] = 60; // uniform colour: chroma subsampling is exact + p[1] = 120; + p[2] = 200; + p[3] = 255; + } + } + u_stereo_camera_planes n; + REQUIRE(u_stereo_camera_layout(2, W, H, &n)); + std::vector nv(n.size); + const uint8_t *bs[2] = {bgra.data(), nullptr}; + REQUIRE(u_stereo_camera_convert(3, bs, b.pitch, 2, nv.data(), &n, W, H)); + std::vector rt(b.size); + const uint8_t *ns[2] = {nv.data(), nv.data() + n.offset[1]}; + REQUIRE(u_stereo_camera_convert(2, ns, n.pitch, 3, rt.data(), &b, W, H)); + for (int c = 0; c < 3; c++) { + int d = (int)rt[c] - (int)bgra[c]; + CHECK(d >= -3); + CHECK(d <= 3); + } + } + CHECK_FALSE(u_stereo_camera_convert(1, src, sp, 7, gray.data(), nullptr, W, H)); +} + +TEST_CASE("stereo camera persistent id", "[stereo_camera]") +{ + char a[64], b[64], c[64]; + u_stereo_camera_persistent_id("SERIAL-123", "C:/apps/chrome.exe", a); + u_stereo_camera_persistent_id("SERIAL-123", "C:/apps/chrome.exe", b); + u_stereo_camera_persistent_id("SERIAL-123", "C:/apps/other.exe", c); + CHECK(std::strcmp(a, b) == 0); + CHECK(std::strcmp(a, c) != 0); + CHECK(std::strncmp(a, "dxrcam-", 7) == 0); + CHECK(std::strlen(a) == 7 + 32); + CHECK(std::strstr(a, "SERIAL") == nullptr); +} + +TEST_CASE("sim_display fake scene has the disparities it was built with", "[stereo_camera]") +{ + sim_stereo_camera_scene s; + // 640 px per eye, 68 deg HFOV, 50 mm, 2.0 m / 0.6 m — the fake's defaults. + const double fx = 320.0 / std::tan(34.0 * 3.14159265358979323846 / 180.0); + sim_stereo_camera_scene_init(&s, 640, 480, fx, 50.0, 2.0, 0.6); + CHECK(s.bg_disparity == 12); + CHECK(s.bar_disparity == 40); + + const uint32_t pitch = 1280; + std::vector f1(pitch * 480), f2(pitch * 480); + sim_stereo_camera_render_gray(&s, 1, f1.data(), pitch); + sim_stereo_camera_render_gray(&s, 2, f2.data(), pitch); + CHECK(std::memcmp(f1.data(), f2.data(), f1.size()) != 0); // the counter moved + + float d = 0.0f; + // Inside the bar (left-eye coordinates). + REQUIRE(u_stereo_camera_estimate_disparity(f1.data(), pitch, 640, 480, 272, 200, 64, 64, 64, &d)); + CHECK(d == Catch::Approx(40.0f).margin(0.5f)); + // Background, right of the bar, below the counter. + REQUIRE(u_stereo_camera_estimate_disparity(f1.data(), pitch, 640, 480, 480, 300, 64, 64, 64, &d)); + CHECK(d == Catch::Approx(12.0f).margin(0.5f)); + // Out-of-range region is refused. + CHECK_FALSE(u_stereo_camera_estimate_disparity(f1.data(), pitch, 640, 480, 600, 0, 64, 64, 64, &d)); +} + +TEST_CASE("plug-in iface: camera slots are appended after lift and platform-state (ADR-020)", "[stereo_camera]") +{ + // ADR-020 append-only: the camera block comes AFTER every slot that was on + // main before it — lift's factory (ADR-042) and then the platform-state + // probe (ADR-045), which is the last pre-camera member. Pin both so a + // reorder (or a slot squeezed in between) fails here, not on a vendor box. + static_assert(sizeof(((xrt_plugin_iface *)0)->get_platform_state) == sizeof(void *), "one slot"); + static_assert(sizeof(((xrt_plugin_iface *)0)->create_dp_d3d11_lift) == sizeof(void *), "one slot"); + const size_t lift_end = + offsetof(xrt_plugin_iface, create_dp_d3d11_lift) + sizeof(((xrt_plugin_iface *)0)->create_dp_d3d11_lift); + const size_t platform_state_end = + offsetof(xrt_plugin_iface, get_platform_state) + sizeof(((xrt_plugin_iface *)0)->get_platform_state); + // lift sits right after the #1243 vk fingerprint, platform-state right + // after lift, and the camera block right after platform-state. + CHECK(offsetof(xrt_plugin_iface, create_dp_d3d11_lift) == + offsetof(xrt_plugin_iface, vk_bundle_fn_table_offset) + sizeof(uint32_t)); + CHECK(offsetof(xrt_plugin_iface, get_platform_state) == lift_end); + CHECK(offsetof(xrt_plugin_iface, stereo_camera_enumerate) == platform_state_end); + CHECK(offsetof(xrt_plugin_iface, stereo_camera_close) + sizeof(void *) == sizeof(xrt_plugin_iface)); + + xrt_plugin_iface iface{}; + iface.struct_size = (uint32_t)offsetof(xrt_plugin_iface, stereo_camera_enumerate); + CHECK_FALSE(xrt_plugin_iface_has_stereo_camera(&iface)); // an older plug-in + CHECK_FALSE(xrt_plugin_iface_has_stereo_camera(nullptr)); +} + +TEST_CASE("stereo camera value types agree with the plug-in encodings", "[stereo_camera]") +{ + STATIC_REQUIRE((int)XRT_STEREO_CAMERA_FORMAT_GRAY8 == (int)XRT_PLUGIN_STEREO_CAMERA_FORMAT_GRAY8); + STATIC_REQUIRE((int)XRT_STEREO_CAMERA_FORMAT_NV12 == (int)XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12); + STATIC_REQUIRE((int)XRT_STEREO_CAMERA_FORMAT_BGRA8 == (int)XRT_PLUGIN_STEREO_CAMERA_FORMAT_BGRA8); + STATIC_REQUIRE(XRT_STEREO_CAMERA_RING_SLOTS == U_STEREO_CAMERA_RING_SLOTS); +}