diff --git a/docs/adr/ADR-043-stereo-camera-source.md b/docs/adr/ADR-043-stereo-camera-source.md index ff049749ae..6aa537fe6a 100644 --- a/docs/adr/ADR-043-stereo-camera-source.md +++ b/docs/adr/ADR-043-stereo-camera-source.md @@ -2,7 +2,8 @@ **Status:** Proposed (2026-09-25) · **R1 implemented** 2026-09-26 (runtime, hardware-free; maintainer defaults recorded in the roadmap §G: stereo-only, iface slots, service clients only, no raw frames -to pages) · **R2 implemented** 2026-09-26 (the service-side rectifier, decision 2 below) · introduces +to pages) · **R2 implemented** 2026-09-26 (the service-side rectifier, decision 2 below) · **Amendment 4** +2026-10-07 (a vendor-neutral UVC side-by-side source owned by the service) · introduces [`XR_DXR_stereo_camera`](../specs/extensions/XR_DXR_stereo_camera.md) · appends optional camera slots to `xrt_plugin_iface` under the [ADR-020](ADR-020-plugin-abi-compatibility-policy.md) append-at-end rule · related: [ADR-019](ADR-019-vendor-plugin-aux-boundary.md), @@ -170,3 +171,73 @@ Three changes close that (spec §2a, §7.1, §7.1.1 are normative): A skipped delegation (declined or untrusted) also drops the foreground-rule exemption — the client is an ordinary window-bearing app for that stream — but never the RAW refusal, which keys on registration alone, so an untrusted entry can only ever narrow what a client gets. + +## Amendment 4 (2026-10-07) — a vendor-neutral source for plain UVC side-by-side stereo webcams + +Decision 1 made the display plug-in the only camera source, and Alternative A left "plain USB +stereo cameras that no plug-in claims" to `getUserMedia`. That leaves a gap the plug-in model +cannot fill: an external stereo webcam that outputs one side-by-side image over ordinary UVC +works with **any** display, belongs to **no** display vendor, and a page reading it through +`getUserMedia` gets raw, unaligned pairs with no calibration, no consent of ours and no +`XR_DXR_stereo_camera` path for native apps. Nothing about reading it is vendor knowledge: the +device is a standard UVC camera; the only device-specific facts are its SBS layout and eye order. + +**Decision.** The service gains its **own** camera source, next to the plug-in's: + +1. **Runtime-owned, vendor-neutral capture code** (`auxiliary/util/u_stereo_uvc.{h,c}` for + config, matching, the SBS split and the nominal model — pure, tested on every OS; + `auxiliary/os/os_uvc_capture*.{h,c,cpp}` for the OS call: Media Foundation on Windows, a + V4L2 stub on Linux to follow). It has exactly the six-operation shape of the plug-in camera + slots, and the camera manager dispatches on a per-camera source tag in one place; consent, + `CAMERA_CONSUMER` rules, the foreground rule, the in-use indicator, linger, rings, + rectification and refinement are one code path for both sources. No vendor names, no SDK. +2. **Opt-in per device, never automatic.** Nothing is claimed without the user's per-user + `stereo-cameras.json` (`%LOCALAPPDATA%\DisplayXR\`, `$XDG_CONFIG_HOME/displayxr/`; + `DXR_STEREO_CAMERA_UVC_CONFIG` overrides the path, `DXR_STEREO_CAMERA_UVC=0` turns the source + off). An entry matches by USB VID:PID and/or a friendly-name substring and states layout + (`sbs-full` | `sbs-half`), eye order, capture mode, per-eye output size, nominal baseline and + HFOV, optionally a calibration file. A **name-only** entry that matches more than one present + device claims nothing. The built-in allowlist is **empty**: no device is listed without a + public spec of its SBS layout. +3. **A plug-in's camera is never claimed.** Plug-in cameras are enumerated first; a device whose + OS id or USB VID:PID equals a plug-in camera's `platform_device_hint` is skipped — the + display's tracker camera stays the plug-in's and its eye tracker's. +4. **Uncalibrated is the normal case, and it is reported honestly.** Without a calibration file + the camera has no `CALIBRATED` flag; the service models it as an ideal parallel pinhole pair + (fx = fy from the configured HFOV, centred principal point, no distortion, R = I, + T = (−baseline, 0, 0)). RAW calibration is refused (`FEATURE_NOT_SUPPORTED`): a nominal model + is not a measurement of the device. RECTIFIED is offered anyway — it is a **pass-through** + (frames as delivered, described by the nominal pinhole, no remap, no zoom) until the online + vertical refinement (R2) has measured the rows and folded a correction in, from which point + the ordinary corrected LUT path runs. `baselineMm` / `horizontalFovDeg` are reported only + when the user configured them (0 = unknown, as before). A calibration file (OpenCV `K1 D1 K2 + D2 R T`, YAML or JSON) makes the camera `CALIBRATED` and the R2 rectifier does the rest. +5. **The device is open only while a stream is started** (plus the 2 s linger), exactly like a + plug-in source; there is no tracker to keep alive. Enumeration never activates a device. + +**What changes for consumers.** Nothing in the API (spec version unchanged): one rule is relaxed +— RECTIFIED no longer requires `CALIBRATED | NATIVELY_RECTIFIED`, it requires a camera the +service can rectify, which now includes a nominal-model UVC pair. A consumer that gates +RECTIFIED on `CALIBRATED` simply does not ask for it. The browser's capture component sees an +ordinary camera with a rectified output and a `platformDeviceHint` naming the physical webcam, so +it can hide the raw duplicate the OS also exposes. `displayxr-cli camera list` tags each camera +`plugin` or `uvc`; `camera uvc-devices [--modes]` shows the config and which present device it +claims (modes only for claimed devices, never started). + +**Alternatives.** *A V4L2/MF virtual camera that re-publishes rectified frames* — rejected for the +reason Alternative B was: a driver install with no consent model of ours. *Letting each display +plug-in carry webcam support* — rejected: a webcam is not a display's, and every vendor would +duplicate the same capture code (ADR-019 isolates vendor code; this is the opposite case). *A +camera-provider plug-in type* (Alternative E) remains the right home for a **camera vendor's** +proprietary device (custom transport, factory calibration channel); a standards-compliant UVC +SBS webcam needs no vendor code, so it lives in the runtime. + +**Consequences.** **+** Any display + any SBS UVC stereo webcam, no plug-in, same privacy model. +**+** The fake backend (`"fake"` in the config) runs the whole manager hardware-free +(`tests_stereo_camera_manager`). **−** MJPEG decoding the plug-in path does not have. A 4K60 +webcam is decoded on the GPU by default (a hardware MJPEG decoder MFT on the adapter that has one, +via a DXGI device manager, scaled to the output size before it crosses to the CPU), with MF's +software decoder as the logged fallback; the first hardware run, before that path, delivered +11.3 Hz. **−** Linux has no backend yet (V4L2 is a TODO). **−** A +loose `match` could claim the wrong webcam; the ambiguity rule and `uvc-devices` mitigate, the +config is still the user's statement of which device is a stereo pair. diff --git a/docs/roadmap/control-panel-performance-settings.md b/docs/roadmap/control-panel-performance-settings.md index 32355f9963..9a7ce849fc 100644 --- a/docs/roadmap/control-panel-performance-settings.md +++ b/docs/roadmap/control-panel-performance-settings.md @@ -548,6 +548,10 @@ as `docs/specs/vendor/oem-android-platform-requirements.md` §R6. | `DXR_HEALTH_MS` | `ipc/server/ipc_server_process.c:82` | `DEBUG_GET_ONCE_NUM` | 10000 | Svc | 2 | #951 `[HEALTH]` telemetry cadence; 0 disables | | `DXR_MAX_CLIENTS` | `ipc/server/ipc_server_process.c:87` | `DEBUG_GET_ONCE_NUM`, consumed `:541-551` | 0 = auto | Svc | 1 | #959 admitted-client cap. Read once at service start | | `DXR_STEREO_CAMERA` | `ipc/server/ipc_server_stereo_camera.c` (`DEBUG_GET_ONCE_BOOL_OPTION`, read at camera-manager create) | on | Svc | 1 | ADR-043 admin/dev kill switch: `0` = the service enumerates zero stereo cameras (indistinguishable from "none", on purpose) and every start returns `XR_ERROR_STEREO_CAMERA_DISABLED_DXR`. The user-facing twin shipped in R3: the tray / menu-bar "Share the 3D camera with apps" toggle, persisted in the consent store — a privacy control, not a performance lever | +| `DXR_STEREO_CAMERA_UVC` | `auxiliary/util/u_stereo_uvc.c` (`u_stereo_uvc_config_load`, `getenv` at camera-manager create) | unset (= on when configured) | Svc | 1 | ADR-043 Amendment 4: `0` = the service's vendor-neutral UVC side-by-side stereo webcam source claims nothing, whatever `stereo-cameras.json` says. The source is opt-in per device by that file; this is the admin/dev override | +| `DXR_STEREO_CAMERA_UVC_CONFIG` | `auxiliary/util/u_stereo_uvc.c` (`u_stereo_uvc_config_path`, `getenv`) | `%LOCALAPPDATA%\DisplayXR\stereo-cameras.json` / `$XDG_CONFIG_HOME/displayxr/stereo-cameras.json` | Svc | 1 | ADR-043 Amendment 4: path of the UVC stereo webcam config the service reads at start (the per-user file is the normal channel; the service reads ITS OWN environment, so this only helps a service launched with it) | +| `DXR_STEREO_CAMERA_UVC_DECODER` | `auxiliary/os/os_uvc_capture_mf.cpp` (`read_decoder_pref`, `getenv` per open) | `auto` | Svc | 1 | ADR-043 Amendment 4: MJPEG decode path of the UVC stereo webcam source — `auto` (hardware MJPEG decoder MFT on a GPU, else software, one WARN), `hardware` (never fall back: the open fails), `software`. A diagnosis / A-B lever, not a setting | +| `DXR_STEREO_CAMERA_UVC_ADAPTER` | `auxiliary/os/os_uvc_capture_mf.cpp` (`choose_adapter`, `getenv` per open) | unset (= the capable adapter with the most dedicated memory) | Svc | 1 | ADR-043 Amendment 4: description substring of the adapter the hardware MJPEG decode runs on (e.g. `Intel` for the iGPU) | | `DXR_ULTRALEAP_SETTLE_MS` | `drivers/ultraleap/ultraleap_provider.cpp:1023` | `getenv` at provider init | `UL_SETTLE_DEFAULT_MS` | Both | 1 | How long to wait for a definitive Leap "attached" answer before role arbitration | ### Test / dev — never exposed diff --git a/docs/roadmap/stereo-camera-source.md b/docs/roadmap/stereo-camera-source.md index 4089a8a1e4..ea26446f30 100644 --- a/docs/roadmap/stereo-camera-source.md +++ b/docs/roadmap/stereo-camera-source.md @@ -200,6 +200,7 @@ The implementation belongs in `displayxr-leia-plugin` (`src/drv_leia/`), per ADR | **R1** ✅ | runtime | **Implemented 2026-09-26 (see "R1 as built" below).** Header `XR_DXR_stereo_camera.h` + `index.json` note (together — the catalog lint requires both); `xrt_plugin_iface` slots + `XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA` (appended after `create_dp_d3d11_lift`, so **lands after ADR-042's PR**); platform-neutral camera manager (thread, refcount + linger, 3-slot pinned ring, per-stream wake handles, decimation, NV12/BGRA conversion); IPC + OpenXR entry points; sim_display fake; `displayxr-cli camera list/calib/probe`; selftest check | `camera probe` on the sim fake writes frames on Windows + Linux CI; ring/pinning unit tests (mirror `tests_lift_mailbox`) | | **R2** ✅ | runtime | **Implemented 2026-09-26 (see "R2 as built" below).** `u_stereo_rectify` (Bouguet, valid-region crop, LUT remap) with golden tests against OpenCV-generated fixtures; RECTIFIED output | sim fake: row error < 0.5 px, disparity = f·B/Z within 0.5 px — **measured 0.061 px worst block, 0.063 px worst disparity error** | | **R3** ✅ | runtime | **Implemented 2026-10-03 (see "R3 as built" below).** Privacy: OS consent check, consent store + tray prompt, delegating-client registration, visibility/lock suspension, indicator, kill switches, camera-only client class, distinct refusal results, events delivered, keyed `persistentId`, read-only consumer section | Policy matrix unit-tested with fakes; sim-fake end to end on macOS: refused without consent → allowed after `camera allow --self` / `camera trust` → suspended on a (fake) lock → ended by the kill switch. **Windows tray + WTS path and the manual panel matrix (allow/deny/revoke, OS switch off, real lock screen, background app) still need the Windows box** | +| **U1** ✅ | runtime | **Implemented 2026-10-07 (ADR-043 Amendment 4, spec §9a).** A vendor-neutral, service-owned source for plain UVC side-by-side stereo webcams: opt-in `stereo-cameras.json` (VID:PID / name match, layout, eye order, mode, nominal baseline + HFOV, optional OpenCV calibration), never a plug-in's camera; Windows Media Foundation backend, Linux V4L2 stub; an uncalibrated pair = nominal model, RECTIFIED pass-through aligned by the online refinement; `camera list` source tag, `camera uvc-devices` | Hardware-free: `tests_stereo_uvc` + `tests_stereo_camera_manager` (the real manager on the fake backend: delivered dy +2.40 -> +0.004 px). **Panel box with a real SBS webcam: `camera probe --rectified` passes, the tracker camera untouched — owner run** | | **L1** | leia-plugin (Windows) | Slots over the SR raw-camera channel per §D; calibration by active serial; keep-alive; repin runtime (feature-macro repin, `downstream-pins.json` `features` track) | On a panel box, incl. the multi-folder box: `camera probe --rectified` rows aligned; the vendor call app running at the same time still gets every frame; tracking/weave unaffected (frame-time + tracking-state logs) | | **B1** | browser (Windows) | §B device factory + duplicate hiding + hint + delegating registration in the installer; web SDK: prefer the hint in `camera:'auto'`, fill `hello` from it | RFC 0002 P1 call between two panel laptops sends rectified SBS from the tracker camera **while both are tracking** | | **A1** | leia-plugin (Android) + runtime | §C.1: the runtime owns the front pair and runs CNSDK in-app tracking from the same capture (worker thread, latest-frame slot); per-eye AHB transport + per-eye frame layout; Android consent + visibility | `camera probe` on the tablet via the runtime app **while the weave tracks from the same capture**; pair at 30.00 fps (not 24) | diff --git a/docs/specs/extensions/XR_DXR_stereo_camera.md b/docs/specs/extensions/XR_DXR_stereo_camera.md index eef9c8d3df..c8bfaa53e0 100644 --- a/docs/specs/extensions/XR_DXR_stereo_camera.md +++ b/docs/specs/extensions/XR_DXR_stereo_camera.md @@ -6,10 +6,11 @@ | **Spec Version** | 3 | | **Extension Type** | Instance extension, service path only (an in-process instance enumerates zero cameras) | | **Header** | [`src/external/openxr_includes/openxr/XR_DXR_stereo_camera.h`](../../../src/external/openxr_includes/openxr/XR_DXR_stereo_camera.h) (+ its `index.json` catalog note) | -| **Status** | **R1 implemented** (runtime, hardware-free): header, plug-in slots, service camera manager, IPC, OpenXR entry points, sim_display fake, `displayxr-cli camera`, selftest check. **R2 implemented**: the service-side rectifier (`u_stereo_rectify`, CPU) — RECTIFIED frames + rectified calibration from any CALIBRATED source, golden-tested against OpenCV, and a distorted sim fake with ground truth. **Not yet:** consent / indicator / foreground rule (R3 — deny-by-default hooks in place), GPU transports (the rectifier has the seam), state-change events, the Leia provider (L1), the browser (B1). Provisional type values `1004999300–310` (relocated from `290–300` when `XR_DXR_weave` v14 took `290`; `XR_DXR_lift` holds `270–289`), pending Khronos registry | +| **Status** | **R1 implemented** (runtime, hardware-free): header, plug-in slots, service camera manager, IPC, OpenXR entry points, sim_display fake, `displayxr-cli camera`, selftest check. **R2 implemented**: the service-side rectifier (`u_stereo_rectify`, CPU) — RECTIFIED frames + rectified calibration from any CALIBRATED source, golden-tested against OpenCV, and a distorted sim fake with ground truth. **Not yet:** consent / indicator / foreground rule (R3 — deny-by-default hooks in place), GPU transports (the rectifier has the seam), state-change events, the Leia provider (L1), the browser (B1). **Amendment 4 implemented** (2026-10-07): a vendor-neutral, service-owned source for plain UVC side-by-side stereo webcams (§9a) — Windows Media Foundation, Linux V4L2 to follow. Provisional type values `1004999300–310` (relocated from `290–300` when `XR_DXR_weave` v14 took `290`; `XR_DXR_lift` holds `270–289`), pending Khronos registry | | **R1 decisions** | stereo-only (names/structs kept open: `viewCount`, next chains) · camera provider = `xrt_plugin_iface` slots · service clients only (in-process enumerates zero) · raw frames never reach web pages (`RAW` refused to `PRESENT_OWNER` clients) — see [roadmap §G](../../roadmap/stereo-camera-source.md#g-open-questions--maintainer-decisions) | | **Decision record** | [ADR-043](../../adr/ADR-043-stereo-camera-source.md) | | **Plug-in contract** | appended `xrt_plugin_iface` camera slots, `XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA` (§9) | +| **Second source** | the service's own UVC side-by-side webcam source, opt-in per device (§9a, ADR-043 Amendment 4) | | **Browser / Android integration + plan** | [roadmap/stereo-camera-source.md](../../roadmap/stereo-camera-source.md) | ## 1. What it is @@ -60,6 +61,7 @@ if (xrAcquireStereoCameraFrameDXR(stream, &f) == XR_SUCCESS) { /* f.slot is pinn | In-process instance (any platform) | `xrEnumerateStereoCamerasDXR` → `XR_SUCCESS`, count 0. The single owner of a camera is the service; an in-process runtime opening it too would recreate the exclusive-device fight this extension exists to end | | Service whose plug-in has no camera slots (or `sim_display` without its fake) | count 0 | | Service with a camera source | one entry per camera, with its `state` (§3) | +| Service with an opted-in UVC side-by-side webcam (`stereo-cameras.json`, §9a) | one more entry per claimed webcam, after the plug-in's; no plug-in needed | | `DXR_STEREO_CAMERA=0` in the **service's** environment, or the user kill switch (§7.4) | count 0 — indistinguishable from "no camera", on purpose | Instance-level, not session-level: a capture component (a browser's video-capture service) has @@ -127,8 +129,9 @@ typedef struct XrStereoCameraPropertiesDXR { XrExtent2Di eyeExtent; // native per-eye size (e.g. 640x480) float maxFrameRate; // Hz the source delivers: MEASURED by the service after its // first open, else the vendor's value; 0 = not known yet - float baselineMm; // |T| of the pair; 0 if uncalibrated - float horizontalFovDeg; // per eye, rectified; 0 if uncalibrated + float baselineMm; // |T| of the pair; 0 if uncalibrated (a §9a UVC camera: the + // CONFIGURED nominal value, else 0) + float horizontalFovDeg; // per eye, rectified; 0 if uncalibrated (§9a: as baselineMm) XrStereoCameraFormatFlagsDXR supportedFormats; // GRAY8 / NV12 / BGRA8 bits the service will produce XrStereoCameraTransportFlagsDXR supportedTransports; // SHARED_MEMORY always; GPU transports optional (§5.3) } XrStereoCameraPropertiesDXR; @@ -178,6 +181,14 @@ device's own identity (serial). It never picks "the first calibration folder" on one field box carried eleven. A plug-in that cannot resolve the active device's calibration reports the camera without `CALIBRATED`, and the service offers RAW only. +**An uncalibrated UVC webcam (§9a)** is the one exception to "RAW only": the service models it +as an ideal parallel pinhole pair from its configured HFOV and baseline. It is reported without +`CALIBRATED`; `xrGetStereoCameraCalibrationDXR(RAW)` fails (`XR_ERROR_FEATURE_UNSUPPORTED`: a +nominal model is not a measurement of the device); RECTIFIED is offered and returns the numbers +of the frames actually delivered — the nominal pinhole while they are passed through, the +corrected geometry once the online refinement has aligned the rows (the frame's +`calibrationGeneration` moves then). + **Rectification** (when not `NATIVELY_RECTIFIED`) is the service's, vendor-neutral (`auxiliary/util/u_stereo_rectify.{h,c}`, R2): Bouguet rectification to a **parallel** pair with zero disparity at infinity, cropped to the valid region (no black corners), per-eye size @@ -225,7 +236,8 @@ typedef enum XrStereoCameraTransportDXR { typedef struct XrStereoCameraStreamCreateInfoDXR { XrStructureType type; const void* XR_MAY_ALIAS next; uint64_t cameraId; - XrStereoCameraOutputDXR output; // RECTIFIED needs CALIBRATED or NATIVELY_RECTIFIED + XrStereoCameraOutputDXR output; // RECTIFIED needs CALIBRATED or NATIVELY_RECTIFIED, or a + // nominal-model UVC camera the service aligns (§9a) XrStereoCameraFormatDXR format; // a MONOCHROME source in NV12 = luma + neutral chroma XrStereoCameraTransportDXR transport; float maxFrameRate; // ≤ 0 = the source's rate; the service decimates, never interpolates @@ -499,7 +511,7 @@ The service queues, per connection (16 deep, oldest dropped), and the client dra | Result | When | |---|---| | `XR_STEREO_CAMERA_FRAME_NOT_READY_DXR` (success) | no frame newer than the last acquired | -| `XR_ERROR_VALIDATION_FAILURE` | struct out of contract (unknown camera, format/transport not in the supported bits, RECTIFIED on an uncalibrated camera) | +| `XR_ERROR_VALIDATION_FAILURE` | struct out of contract (unknown camera, format/transport not in the supported bits, RECTIFIED on a camera the service cannot rectify) | | `XR_ERROR_PERMISSION_INSUFFICIENT` | the OS camera privacy switch denies the app (§7.1 row 4, delegating clients included); RAW asked by a browser / delegating client; a client class that may not use cameras (or `xrCreateSession` from a `CAMERA_CONSUMER`) | | `XR_ERROR_STEREO_CAMERA_CONSENT_REFUSED_DXR` | §7.1: the user / the stored decision refused, the prompt was unanswered or unavailable; retryable after consent. A browser maps it to `NotAllowedError`. A registered delegating client gets it too when its delegation was skipped — it declined (`DECLINE_DELEGATION_BIT`, §2a) or its entry is untrusted (§7.1.1) — and no stored Allow / prompt answer allowed it | | `XR_ERROR_STEREO_CAMERA_DISABLED_DXR` | §7.4: sharing is off (user toggle or `DXR_STEREO_CAMERA=0`) | @@ -587,6 +599,120 @@ owns threads, fan-out, rectification, format conversion, transport, consent and | authorisation, foreground rule, indicator, kill switch | calibration of **its active device**, by device identity | | events, stats, CLI | tracker keep-alive while open; shedding camera before tracking | +## 9a. Service-owned UVC side-by-side source (ADR-043 Amendment 4) + +A plain stereo webcam that sends **one side-by-side image** over standard UVC belongs to no +display vendor, so it gets no plug-in: the **service** reads it with its own vendor-neutral +capture code and exposes it like any other camera — same entry points, same consent path and +`CAMERA_CONSUMER` rules, same foreground rule, indicator, kill switches, linger, rings and +rectifier. Code: `auxiliary/util/u_stereo_uvc.{h,c}` (config, matching, SBS split, nominal +model, calibration files, the fake backend — pure, tested on every OS by `tests_stereo_uvc`), +`auxiliary/os/os_uvc_capture*.{h,c,cpp}` (the OS call), and the `src_*` dispatch in +`ipc_server_stereo_camera.c` (the only place the manager tells the two sources apart). + +**Opt-in only.** With no config the source enumerates nothing and touches no device. The SERVICE +reads, once at start: + +| Where | | +|---|---| +| Windows | `%LOCALAPPDATA%\DisplayXR\stereo-cameras.json` | +| Linux / macOS | `$XDG_CONFIG_HOME/displayxr/stereo-cameras.json`, else `~/.config/displayxr/stereo-cameras.json` | +| override | `DXR_STEREO_CAMERA_UVC_CONFIG=` in the service's environment; `DXR_STEREO_CAMERA_UVC=0` turns the source off | + +```jsonc +{"uvc": [{ + "match": "Eyes", // friendly-name substring, case-insensitive } at least one; + "vid_pid": "2bdf:0281", // USB VID:PID, hex } both = both must hold + "name": "Stereo webcam", // display name ("" = the device's own) + "layout": "sbs-half", // "sbs-full": each half one full-res eye; "sbs-half": each half an eye squeezed 2:1 + "eyes": "lr", // "lr" = the device's left lens in the left half; "rl" = swapped + "mode": "3840x2160@60", // capture mode; omitted = the largest, then fastest, the device lists + "eye_size": "1280x720", // per-eye output; omitted = see "Frames" + "baseline_mm": 60, // nominal lens separation (omitted: 60 internally, reported 0) + "hfov_deg": 90, // nominal per-eye horizontal FOV (omitted: 70 internally, reported 0) + "calibration": "C:/cal/eyes.yaml" // optional OpenCV-style calibration (below) +}]} +``` + +(The file itself is strict JSON — no comments.) Up to 4 entries; any malformed value fails the +whole file (logged, nothing claimed): a typo must never claim a different camera. **No built-in +allowlist** ships: a device is listed only with a public spec of its SBS layout, and none was +available. + +**Which device.** Entries are resolved in order, one device each: the first present device that +matches and is not already claimed. A **name-only** entry matching several devices claims none +(a vendor prefix shared by a display's tracking camera and the webcam must not pick one by +enumeration order). **A device a display plug-in reports as its camera is never claimed**: plug-in +cameras are enumerated first, and a device whose OS id, or whose USB VID:PID, equals a plug-in +camera's `platformDeviceHint` (e.g. `"04f2:b70c"`) is skipped. + +**Frames.** The backend delivers decoded NV12 or YUY2 (Windows: an async `IMFSourceReader` with +advanced video processing; the reader keeps only the newest sample) — preferably already at the +output size, 2 × eye width × eye height (below). The source splits each frame into an NV12 SBS image of two +square-pixel eyes, LEFT lens left: SBS_FULL eyes are `(W/2) × H`, SBS_HALF eyes `(W/2) × (H/2)` +(a 2:1-squeezed half carries that much information; the vertical 2:1 is an exact two-row +average), both capped at 1280 px wide with the aspect kept, unless `eye_size` says otherwise +(integer box pre-reduction + bilinear). Timestamps are arrival time (`captureTimeIsExposure` +FALSE). Flags: `USER_FACING`, plus `CALIBRATED` with a calibration file; never +`SHARED_WITH_EYE_TRACKING`. `platformDeviceHint` = the OS device id (Windows: the MF symbolic +link), so a capture stack can hide the raw webcam it also sees. The device is opened on the first +started stream and closed after the 2 s linger; an open that fails (another app holds it) makes +the camera `UNAVAILABLE`, and a start meanwhile gets `XR_ERROR_STEREO_CAMERA_BUSY_DXR`. + +**Calibration.** None (the usual case): the nominal model of §4 — fx = fy = +`(eyeWidth/2) / tan(hfov/2)`, principal point `((w−1)/2, (h−1)/2)`, no distortion, R = I, +T = (−baseline, 0, 0) mm. RECTIFIED frames are the delivered frames unchanged (no remap, no +zoom) until the online vertical refinement (§4, R2) — which runs on them exactly as on a +calibrated camera — folds a correction in; from then on they go through the corrected LUT and +the rectified calibration describes that geometry. A `calibration` file makes the camera +`CALIBRATED`: OpenCV FileStorage YAML (`K1: !!opencv-matrix ... data: [...]`) or JSON, keys +`K1`/`M1`, `D1`, `K2`/`M2`, `D2`, `R` (3×3 or a rotation vector), `T` (mm; |T| < 1 is taken as +metres), optional `image_width` / `image_height` (default: the raw per-eye size `mode/2 × mode`, +i.e. the squeezed half for SBS_HALF — an anisotropic K is fine), optional +`distortion_model: fisheye` for 4-coefficient KB4; 4–5 coefficients are RADTAN5, 8 RADTAN8. A +file that cannot be read or parsed is logged and the nominal model is used. + +**Backends.** Windows: Media Foundation (`mf`, `mfplat`, `mfreadwrite`, `mfuuid`, `d3d11`, `dxgi`, +linked only into the service and `displayxr-cli`, via `aux_os_uvc`). **Decode path** — a 4K60 MJPEG +webcam is the expensive case (first hardware run: 11.3 Hz, mostly a CPU split of the full 4K frame): + +| Path | What runs | When | +|---|---|---| +| **Hardware** (default) | a D3D11 device (`VIDEO_SUPPORT`, multithread-protected) on the adapter that has a hardware MJPEG decoder MFT (matched by PCI vendor; the most dedicated memory wins, `DXR_STEREO_CAMERA_UVC_ADAPTER=` picks), handed to the reader as `MF_SOURCE_READER_D3D_MANAGER` + `MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS`; NV12 is requested **at the output size**, so the GPU decodes **and** downscales (half-SBS 3840×2160 → 2560×720) — with an anamorphic output pixel aspect (`PAR_out = PAR_src · src_w·out_h / (src_h·out_w)`, 1:2 here), because the video processor otherwise preserves the display aspect and **letterboxes** (the first real-webcam run got a centred ~1280 px picture between black bars); readback is **pipelined** over a ring of 3 staging NV12 textures: each read queues a `CopySubresourceRegion` + event query and delivers the newest copy that has already finished (waiting, by polling — never a blocking `Map` that would hold the device lock MF's decoder needs — on the older one only when two are in flight), one `Map`, copy out, `Unmap`, the sample returned at once. A synchronous copy measured 22–25 ms per frame on the real webcam (the copy waits behind the decode) and capped it at 36 Hz | an adapter with a hardware MJPEG decoder exists, and MF actually inserted a hardware decoder (checked after the topology is built; a software decoder behind the device manager is reopened on the software path) | +| **Software** (fallback, one WARN with the reason) | the plain reader; MF's multithreaded MJPEG decoder + video processor, also asked for the output size | no capable adapter, the hardware open failed, or `DXR_STEREO_CAMERA_UVC_DECODER=software` | + +`DXR_STEREO_CAMERA_UVC_DECODER=hardware` refuses the fallback (the open fails instead). The service +logs which ran, per open: `uvc capture (Media Foundation): started 3840x2160@60.00 — decoder NVIDIA +MJPEG Video Decoder MFT (hardware), HARDWARE (GPU decode + scale) path, reader output NV12 +2560x720`, and at close the frame count and mean readback. `displayxr-cli camera uvc-devices +--decoder` prints the policy, the decoders registered and the adapter it would use, and +`displayxr-cli camera uvc-decode-bench ` runs a media file (a 4K MJPEG AVI) through the +identical open / read / split path with requests paced to the file's frame rate — no camera. +The bench also checks the picture: per eye, the mean luma of the outer 5 % columns against the +centre, and exits 4 on dark side bars (`tests_uvc_capture_mf` asserts the same through the real +backend on a generated YUY2 AVI, both policies). On the development laptop (RTX 3080 + Intel iGPU, +a bright 3840×2160@60 half-SBS MJPEG file, out 2560×720, no bars on any path): hardware Intel +60.0 Hz (readback 0.7 ms), hardware NVIDIA 51–53 Hz (readback 6–7 ms — a file lets the +asynchronous NVIDIA MFT decode ~5× ahead of the paced requests and the copies queue behind that +backlog; a 60 Hz camera has none), software with MF scaling 59.7 Hz (0 superseded, CPU-heavy); the source's own 4K → 2 × 1280×720 resample, +used only when no scaler is available, costs ~25 ms (was ~110 ms), a pre-scaled frame 0.3 ms. Linux: V4L2 is a TODO +(`os_uvc_capture_stubs.c` records the plan); other platforms: none. A config with `"fake": true` +(or `{"disparity": 0.04, "dy": 0.005, "pixel": "nv12"|"yuy2", "fps": 30}`, fractions of the eye +width / height) REPLACES real capture with a synthetic device ("DisplayXR synthetic SBS camera", +id `dxr-fake-uvc`) that renders a known disparity and a known right-eye vertical offset in any +layout — the hardware-free path through the whole manager (`tests_stereo_camera_manager`, and +`camera probe --rectified` against a service on any OS). + +**Diagnostics.** (See also the decode-path tools above.) `displayxr-cli camera list` tags each camera `(source: plugin|uvc)` (`"source"` in +`--json`; the wire carries it in the former `reserved` word of the properties — the OpenXR struct +is unchanged). `displayxr-cli camera uvc-devices [--modes]` runs locally (no service): the config +path and parse result, every capture device present with its VID:PID and OS id, which entry +claims it, ambiguous entries, and with `--modes` the capture modes of the CLAIMED devices only and +the per-eye output each would give (the device object is created to read its media types, never +started). Service log: one WARN per claimed device (`stereo camera (uvc): CLAIMED ...`), the +nominal-model line (`NOMINAL model (uncalibrated) — RECTIFIED = the frames as delivered ...`), +then the usual per-camera, per-stream and refinement lines. + ## 10. sim_display fake > **R1 as built** (`drivers/sim_display/sim_display_stereo_camera{,_pattern}.c`): the pair is @@ -630,7 +756,9 @@ The whole path (enumerate → consent → start → acquire → rectify) then ru ## 11. Probe and diagnostics ``` -displayxr-cli camera list [--json] # properties + state of every camera +displayxr-cli camera list [--json] # properties + state + source (plugin|uvc) of every camera +displayxr-cli camera uvc-devices [--modes] [--decoder] # §9a: UVC webcam config + devices + decode path (local) +displayxr-cli camera uvc-decode-bench # §9a: a media file through the webcam decode path displayxr-cli camera calib [--raw|--rectified] [--json] displayxr-cli camera probe [] [--raw] [--format gray8|nv12|bgra8] [--fps F] [--frames N] [--seconds S] [--out DIR] @@ -718,6 +846,7 @@ camera state change and per stream start/stop with the peer executable; INFO sta | Version | Change | |---|---| +| 3 (Amendment 4) | No API change. The service's own vendor-neutral UVC side-by-side source (§9a): opt-in per device by `stereo-cameras.json`, never a plug-in's camera; an uncalibrated pair is reported without `CALIBRATED`, its RAW calibration is refused, its RECTIFIED output is a pass-through aligned by the online refinement — so RECTIFIED now requires a camera the service can rectify rather than `CALIBRATED \| NATIVELY_RECTIFIED`. `displayxr-cli camera list` shows the source; new `camera uvc-devices` | | 3 | Delegation hardening: `XR_STEREO_CAMERA_CLIENT_DECLINE_DELEGATION_BIT_DXR` (`0x2`) — a delegating client opts out of row 6 for one instance (§2a); a delegating entry for a user-writable executable path is applied only with a recorded signer its valid Authenticode signature matches (`.signer`, Windows; POSIX path-only) and a user-level entry's first use per executable per service run is announced (§7.1.1). Appended; nothing renumbered | | 2 (R3) | Consent and privacy as built (§7): `XrStereoCameraClientInfoDXR` + `XR_STEREO_CAMERA_CLIENT_CONSUMER_ONLY_BIT_DXR` (the `CAMERA_CONSUMER` class, §2a); distinct results `XR_ERROR_STEREO_CAMERA_CONSENT_REFUSED_DXR` / `_DISABLED_DXR` / `_BUSY_DXR` / `_STREAM_ENDED_DXR`; `XrEventDataStereoCameraStreamEndedDXR`; the state-changed event is now delivered; `persistentId` keyed with a per-user secret; read-only consumer section on every platform; `DXR_STEREO_CAMERA_DEV_ALLOW` becomes a documented dev override, `DXR_STEREO_CAMERA_PROMPT=0` disables the prompt. Everything appended at `1004999311–316`; nothing renumbered | | 1 (R2) | No API change. RECTIFIED output and rectified calibration now come from the service's rectifier for any CALIBRATED, non-natively-rectified source; the browser is refused a camera it cannot rectify; sim_display `_DISTORT=1` fake; `camera probe` row alignment + `--rectified` | diff --git a/docs/specs/extensions/index.json b/docs/specs/extensions/index.json index b2dadcc567..905e66afcc 100644 --- a/docs/specs/extensions/index.json +++ b/docs/specs/extensions/index.json @@ -140,7 +140,7 @@ "XR_DXR_stereo_camera": { "group": "capture", "title": "Stereo Camera Source", - "summary": "The display's own stereo camera (usually the eye tracker's) as a runtime-owned, privacy-gated source: the vendor plug-in supplies frames without taking the device from the tracker, and the service fans newest-frame side-by-side pairs out to clients over shared memory" + "summary": "A runtime-owned, privacy-gated stereo camera source: the display's own camera (usually the eye tracker's, supplied by the vendor plug-in without taking it from the tracker) or an opted-in plain UVC side-by-side webcam, fanned out by the service to clients as rectified newest-frame side-by-side pairs over shared memory" } } } diff --git a/src/xrt/auxiliary/os/CMakeLists.txt b/src/xrt/auxiliary/os/CMakeLists.txt index 3d67512879..91ff1da5b9 100644 --- a/src/xrt/auxiliary/os/CMakeLists.txt +++ b/src/xrt/auxiliary/os/CMakeLists.txt @@ -93,6 +93,27 @@ if(APPLE) target_sources(aux_os PRIVATE os_macos.c) endif() +#### +# UVC capture backend of the vendor-neutral stereo camera source (ADR-043 +# Amendment 4, util/u_stereo_uvc.h). Its own library so the capture DLL +# imports (Media Foundation) land only in the processes that capture — the +# service and displayxr-cli — never in the runtime library apps load. +# + +add_library(aux_os_uvc STATIC os_uvc_capture.h) +target_link_libraries(aux_os_uvc PUBLIC aux-includes aux_os aux_util) +if(WIN32) + target_sources(aux_os_uvc PRIVATE os_uvc_capture_mf.cpp) + # mf: MFEnumDeviceSources / MFCreateDeviceSource; mfplat: MFStartup, media + # types; mfreadwrite: the source reader; mfuuid: the MF GUIDs. + # d3d11 + dxgi: the hardware MJPEG decode path (IMFDXGIDeviceManager on the + # adapter that has a hardware MJPEG decoder MFT; staging readback). + target_link_libraries(aux_os_uvc PRIVATE mf mfplat mfreadwrite mfuuid ole32 d3d11 dxgi) +else() + # Desktop Linux: V4L2 is a TODO (see the file); everything else has none. + target_sources(aux_os_uvc PRIVATE os_uvc_capture_stubs.c) +endif() + #### # BLE library # diff --git a/src/xrt/auxiliary/os/os_uvc_capture.h b/src/xrt/auxiliary/os/os_uvc_capture.h new file mode 100644 index 0000000000..942691cc9a --- /dev/null +++ b/src/xrt/auxiliary/os/os_uvc_capture.h @@ -0,0 +1,52 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief The OS capture backend of the vendor-neutral UVC stereo camera source + * (u_stereo_uvc.h, ADR-043 Amendment 4). + * + * One function per platform fills a @ref u_stereo_uvc_backend: + * - Windows: Media Foundation (os_uvc_capture_mf.cpp) — devices from + * MFEnumDeviceSources (no device is activated to list them), modes from the + * device's media types (the device object is created, never started), and + * an ASYNC IMFSourceReader per open device whose callback keeps only the + * newest sample; MJPEG / YUY2 / NV12 are decoded / converted to NV12 by the + * reader's own decoder MFT (advanced video processing), never by us. + * - Linux desktop: V4L2 — TODO (os_uvc_capture_stubs.c reports no backend). + * - Everything else: no backend; the source then only ever sees the fake. + * + * A separate static library (aux_os_uvc) so the capture DLL imports land only + * in the processes that capture: the service and displayxr-cli. + * + * @ingroup aux_os + */ + +#pragma once + +#include "util/u_stereo_uvc.h" + +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/*! + * Fill @p out with this platform's capture backend. + * @return false when the platform has none (@p out is zeroed). + */ +bool +os_uvc_capture_backend(struct u_stereo_uvc_backend *out); + +/*! + * Human-readable report of the decode path open() would take (`displayxr-cli + * camera uvc-devices --decoder`): the decoder policy, the hardware / software + * MJPEG decoders registered, the adapters and which one would decode. Touches + * no camera. A platform without a backend says so. + */ +void +os_uvc_capture_decoder_report(char *out, size_t cap); + +#ifdef __cplusplus +} +#endif diff --git a/src/xrt/auxiliary/os/os_uvc_capture_mf.cpp b/src/xrt/auxiliary/os/os_uvc_capture_mf.cpp new file mode 100644 index 0000000000..a514a08d92 --- /dev/null +++ b/src/xrt/auxiliary/os/os_uvc_capture_mf.cpp @@ -0,0 +1,1569 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief Media Foundation capture backend for the UVC stereo camera source — + * see os_uvc_capture.h. + * + * Privacy-relevant facts, in order of the calls: + * - enumerate: MFEnumDeviceSources only. IMFActivate objects are read for + * their friendly name + symbolic link and released; nothing is activated. + * - list_modes: MFCreateDeviceSource + the presentation descriptor's media + * types, then IMFMediaSource::Shutdown. The device object exists for the + * duration of the call; no stream is selected or started (no frames, no + * camera light). Only ever called for a device the user's config matched. + * - open: the device source + an ASYNC source reader; streaming starts with + * the first ReadSample. close flushes, releases the reader and shuts the + * source down. The service calls these only while a stream is started. + * - the decoder report (`displayxr-cli camera uvc-devices --decoder`) touches + * no camera at all: DXGI adapters + the registered MJPEG decoder MFTs. + * + * Decode paths (the reason this file is more than a reader): + * - HARDWARE (default when possible): a 4K60 MJPEG webcam decoded in software + * delivers ~11 Hz with ~90 ms latency on a laptop CPU. So open() first looks + * for an adapter that has a hardware MJPEG decoder MFT (MFTEnumEx HARDWARE, + * matched to DXGI adapters by vendor id), creates a D3D11 device on it + * (VIDEO_SUPPORT, multithread-protected), hands the reader an + * IMFDXGIDeviceManager (MF_SOURCE_READER_D3D_MANAGER + + * MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS + advanced video processing), and + * asks for NV12 AT THE OUTPUT SIZE the source wants (2 x eye width x eye + * height) — so the GPU decodes AND downscales (half-SBS 3840x2160 -> + * 2560x720) and only that small frame crosses to the CPU: one + * CopySubresourceRegion into a staging NV12 texture + one Map per frame. + * After the topology is built the reader's decoder is inspected; if MF put + * a SOFTWARE decoder behind the device manager anyway (nothing gained, a + * GPU round trip added) the device is reopened on the software path. + * - SOFTWARE (fallback, one WARN saying why): the plain reader with advanced + * video processing at the native size; the source splits / downscales on + * the CPU. + * Knobs (service environment): DXR_STEREO_CAMERA_UVC_DECODER=auto|hardware| + * software (hardware = never fall back: the open fails instead), + * DXR_STEREO_CAMERA_UVC_ADAPTER= to + * pick among several capable adapters (default: the one with the most + * dedicated video memory, i.e. the discrete GPU). + * + * Threading: open / read / close run on the service's camera thread (COM MTA, + * initialised here and balanced in close); the reader's callback runs on a + * Media Foundation work-queue thread and only swaps the newest sample in under + * a lock + signals an auto-reset event. read() waits on that event, takes the + * newest sample (older ones are simply dropped: latest-wins, like every other + * camera source) and keeps its buffer LOCKED / its staging texture MAPPED until + * the next read / close so the planes it hands out stay valid. The D3D11 device + * is shared with MF's own threads, hence ID3D10Multithread protection. + * + * @ingroup aux_os + */ + +#include "os/os_uvc_capture.h" +#include "os/os_time.h" +#include "util/u_logging.h" + +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#include +#include +#include // ID3D10Multithread +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +namespace { + +template +void +safe_release(T *&p) +{ + if (p != nullptr) { + p->Release(); + p = nullptr; + } +} + +//! COM (MTA) + Media Foundation for the duration of one call or one open device. +struct mf_scope +{ + bool com = false; + bool mf = false; + + bool + begin() + { + HRESULT hr = CoInitializeEx(nullptr, COINIT_MULTITHREADED); + com = SUCCEEDED(hr); // RPC_E_CHANGED_MODE: an STA thread — MF works there too, nothing to balance + mf = SUCCEEDED(MFStartup(MF_VERSION, MFSTARTUP_NOSOCKET)); + return mf; + } + + void + end() + { + if (mf) { + MFShutdown(); + mf = false; + } + if (com) { + CoUninitialize(); + com = false; + } + } +}; + +void +wide_to_utf8(const wchar_t *w, char *out, size_t cap) +{ + out[0] = '\0'; + if (w == nullptr) { + return; + } + int n = WideCharToMultiByte(CP_UTF8, 0, w, -1, out, (int)cap, nullptr, nullptr); + if (n <= 0) { + out[0] = '\0'; + } + out[cap - 1] = '\0'; +} + +bool +utf8_to_wide(const char *s, wchar_t *out, size_t cap) +{ + int n = MultiByteToWideChar(CP_UTF8, 0, s, -1, out, (int)cap); + return n > 0; +} + +//! Append printf-style to a bounded buffer. +void +appendf(char *buf, size_t cap, const char *fmt, ...) +{ + size_t n = std::strlen(buf); + if (n + 1 >= cap) { + return; + } + va_list ap; + va_start(ap, fmt); + vsnprintf(buf + n, cap - n, fmt, ap); + va_end(ap); +} + +//! The device source for symbolic link @p id (not started). +HRESULT +create_device_source(const char *id, IMFMediaSource **out) +{ + *out = nullptr; + // "file:": a media FILE through the identical reader / decoder path — + // `displayxr-cli camera uvc-decode-bench` measures the decoders with it, no + // camera involved. Never an enumerated id, so the service cannot reach it. + if (std::strncmp(id, "file:", 5) == 0) { + wchar_t path[1024]; + if (!utf8_to_wide(id + 5, path, sizeof(path) / sizeof(path[0]))) { + return E_INVALIDARG; + } + IMFSourceResolver *res = nullptr; + IUnknown *obj = nullptr; + MF_OBJECT_TYPE type = MF_OBJECT_INVALID; + HRESULT hr = MFCreateSourceResolver(&res); + if (SUCCEEDED(hr)) { + hr = res->CreateObjectFromURL(path, MF_RESOLUTION_MEDIASOURCE, nullptr, &type, &obj); + } + if (SUCCEEDED(hr)) { + hr = obj->QueryInterface(IID_PPV_ARGS(out)); + } + safe_release(obj); + safe_release(res); + return hr; + } + wchar_t link[512]; + if (!utf8_to_wide(id, link, sizeof(link) / sizeof(link[0]))) { + return E_INVALIDARG; + } + IMFAttributes *attr = nullptr; + HRESULT hr = MFCreateAttributes(&attr, 2); + if (SUCCEEDED(hr)) { + hr = attr->SetGUID(MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE, MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE_VIDCAP_GUID); + } + if (SUCCEEDED(hr)) { + hr = attr->SetString(MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE_VIDCAP_SYMBOLIC_LINK, link); + } + if (SUCCEEDED(hr)) { + hr = MFCreateDeviceSource(attr, out); + } + safe_release(attr); + return hr; +} + +//! The subtypes the reader can turn into NV12 for us. +int +subtype_rank(const GUID &st) +{ + if (st == MFVideoFormat_NV12) { + return 3; // no conversion + } + if (st == MFVideoFormat_YUY2) { + return 2; + } + if (st == MFVideoFormat_MJPG) { + return 1; // decoded by an MJPEG decoder MFT + } + return 0; +} + +bool +type_mode(IMFMediaType *t, u_stereo_uvc_mode *m, GUID *subtype) +{ + UINT32 w = 0, h = 0, num = 0, den = 0; + if (FAILED(MFGetAttributeSize(t, MF_MT_FRAME_SIZE, &w, &h)) || w == 0 || h == 0) { + return false; + } + if (FAILED(MFGetAttributeRatio(t, MF_MT_FRAME_RATE, &num, &den)) || den == 0) { + num = 0; + den = 1; + } + GUID st = GUID_NULL; + t->GetGUID(MF_MT_SUBTYPE, &st); + if (subtype != nullptr) { + *subtype = st; + } + m->width = w; + m->height = h; + m->fps = (float)((double)num / (double)den); + return true; +} + + +/* + * + * Decoder policy: which adapter has a hardware MJPEG decoder. + * + */ + +enum class decoder_pref +{ + automatic, + hardware, + software, +}; + +decoder_pref +read_decoder_pref() +{ + const char *e = getenv("DXR_STEREO_CAMERA_UVC_DECODER"); + if (e == nullptr || e[0] == '\0' || std::strcmp(e, "auto") == 0) { + return decoder_pref::automatic; + } + if (std::strcmp(e, "hardware") == 0 || std::strcmp(e, "hw") == 0) { + return decoder_pref::hardware; + } + if (std::strcmp(e, "software") == 0 || std::strcmp(e, "sw") == 0) { + return decoder_pref::software; + } + U_LOG_W("uvc capture: DXR_STEREO_CAMERA_UVC_DECODER=\"%s\" is not auto|hardware|software — using auto", e); + return decoder_pref::automatic; +} + +struct hw_decoder +{ + char name[128]; + uint32_t vendor; //!< PCI vendor id from MFT_ENUM_HARDWARE_VENDOR_ID_Attribute ("VEN_xxxx"), 0 = unknown +}; + +//! Registered decoder MFTs taking MJPEG (@p hardware: the HARDWARE ones). +uint32_t +list_mjpeg_decoders(bool hardware, hw_decoder *out, uint32_t cap) +{ + MFT_REGISTER_TYPE_INFO in = {MFMediaType_Video, MFVideoFormat_MJPG}; + IMFActivate **acts = nullptr; + UINT32 count = 0; + UINT32 flags = (hardware ? MFT_ENUM_FLAG_HARDWARE : (MFT_ENUM_FLAG_SYNCMFT | MFT_ENUM_FLAG_ASYNCMFT)) | + MFT_ENUM_FLAG_SORTANDFILTER; + if (FAILED(MFTEnumEx(MFT_CATEGORY_VIDEO_DECODER, flags, &in, nullptr, &acts, &count))) { + return 0; + } + uint32_t n = 0; + for (UINT32 i = 0; i < count; i++) { + if (n < cap) { + hw_decoder *d = &out[n++]; + std::memset(d, 0, sizeof(*d)); + WCHAR *s = nullptr; + UINT32 len = 0; + if (SUCCEEDED(acts[i]->GetAllocatedString(MFT_FRIENDLY_NAME_Attribute, &s, &len))) { + wide_to_utf8(s, d->name, sizeof(d->name)); + CoTaskMemFree(s); + } else { + std::snprintf(d->name, sizeof(d->name), "(unnamed decoder)"); + } + s = nullptr; + if (SUCCEEDED(acts[i]->GetAllocatedString(MFT_ENUM_HARDWARE_VENDOR_ID_Attribute, &s, &len))) { + char v[64]; + wide_to_utf8(s, v, sizeof(v)); + CoTaskMemFree(s); + const char *p = std::strstr(v, "VEN_"); + d->vendor = p != nullptr ? (uint32_t)std::strtoul(p + 4, nullptr, 16) : 0; + } + } + acts[i]->Release(); + } + CoTaskMemFree(acts); + return n; +} + +struct adapter_choice +{ + bool found = false; + IDXGIAdapter1 *adapter = nullptr; //!< owned + char desc[128] = {0}; + char decoder[128] = {0}; + char why[160] = {0}; +}; + +/*! + * Pick the adapter to decode on: one whose vendor has a hardware MJPEG decoder + * MFT; DXR_STEREO_CAMERA_UVC_ADAPTER (description substring) wins, else the + * most dedicated video memory. @p report (may be NULL) gets the whole picture. + */ +adapter_choice +choose_adapter(char *report, size_t report_cap) +{ + adapter_choice c; + hw_decoder hw[16]; + uint32_t nhw = list_mjpeg_decoders(true, hw, 16); + if (report != nullptr) { + appendf(report, report_cap, "hardware MJPEG decoder MFTs: %u\n", nhw); + for (uint32_t i = 0; i < nhw; i++) { + appendf(report, report_cap, " - \"%s\" (vendor 0x%04x)\n", hw[i].name, hw[i].vendor); + } + hw_decoder sw[16]; + uint32_t nsw = list_mjpeg_decoders(false, sw, 16); + appendf(report, report_cap, "software MJPEG decoder MFTs: %u\n", nsw); + for (uint32_t i = 0; i < nsw; i++) { + appendf(report, report_cap, " - \"%s\"\n", sw[i].name); + } + } + const char *want = getenv("DXR_STEREO_CAMERA_UVC_ADAPTER"); + IDXGIFactory1 *f = nullptr; + if (FAILED(CreateDXGIFactory1(IID_PPV_ARGS(&f)))) { + std::snprintf(c.why, sizeof(c.why), "no DXGI factory"); + return c; + } + SIZE_T best_mem = 0; + bool best_named = false; + if (report != nullptr) { + appendf(report, report_cap, "adapters:\n"); + } + for (UINT i = 0;; i++) { + IDXGIAdapter1 *a = nullptr; + if (f->EnumAdapters1(i, &a) == DXGI_ERROR_NOT_FOUND) { + break; + } + DXGI_ADAPTER_DESC1 d; + if (FAILED(a->GetDesc1(&d)) || (d.Flags & DXGI_ADAPTER_FLAG_SOFTWARE) != 0) { + safe_release(a); + continue; + } + char name[128]; + wide_to_utf8(d.Description, name, sizeof(name)); + int dec = -1; + for (uint32_t k = 0; k < nhw && dec < 0; k++) { + if (hw[k].vendor == d.VendorId) { + dec = (int)k; + } + } + if (report != nullptr) { + appendf(report, report_cap, + " [%u] \"%s\" vendor 0x%04x, %llu MB dedicated, hardware MJPEG decoder: %s\n", i, name, + d.VendorId, (unsigned long long)(d.DedicatedVideoMemory >> 20), + dec >= 0 ? hw[dec].name : "none"); + } + bool named = want != nullptr && want[0] != '\0' && std::strstr(name, want) != nullptr; + bool better = dec >= 0 && (!c.found || (named && !best_named) || + (named == best_named && d.DedicatedVideoMemory > best_mem)); + if (better) { + safe_release(c.adapter); + c.adapter = a; + c.adapter->AddRef(); + c.found = true; + best_mem = d.DedicatedVideoMemory; + best_named = named; + std::snprintf(c.desc, sizeof(c.desc), "%s", name); + std::snprintf(c.decoder, sizeof(c.decoder), "%s", hw[dec].name); + } + safe_release(a); + } + safe_release(f); + if (!c.found) { + std::snprintf(c.why, sizeof(c.why), "%s", + nhw == 0 ? "no hardware MJPEG decoder MFT is registered on this machine" + : "no adapter matches a hardware MJPEG decoder MFT's vendor"); + } else if (want != nullptr && want[0] != '\0' && !best_named) { + std::snprintf(c.why, sizeof(c.why), "DXR_STEREO_CAMERA_UVC_ADAPTER=\"%s\" matches no capable adapter", + want); + } + return c; +} + + +/* + * + * The async reader: newest sample wins. + * + */ + +class reader_cb final : public IMFSourceReaderCallback +{ +public: + reader_cb() + { + InitializeCriticalSection(&lock_); + event_ = CreateEventW(nullptr, FALSE, FALSE, nullptr); + flushed_ = CreateEventW(nullptr, TRUE, FALSE, nullptr); + } + + // IUnknown + STDMETHODIMP + QueryInterface(REFIID iid, void **ppv) override + { + if (ppv == nullptr) { + return E_POINTER; + } + if (iid == __uuidof(IUnknown) || iid == __uuidof(IMFSourceReaderCallback)) { + *ppv = static_cast(this); + AddRef(); + return S_OK; + } + *ppv = nullptr; + return E_NOINTERFACE; + } + + STDMETHODIMP_(ULONG) + AddRef() override + { + return (ULONG)InterlockedIncrement(&refs_); + } + + STDMETHODIMP_(ULONG) + Release() override + { + LONG r = InterlockedDecrement(&refs_); + if (r == 0) { + delete this; + } + return (ULONG)r; + } + + // IMFSourceReaderCallback + STDMETHODIMP + OnReadSample(HRESULT hr, DWORD, DWORD flags, LONGLONG, IMFSample *sample) override + { + const int64_t now = os_monotonic_get_ns(); + EnterCriticalSection(&lock_); + if (FAILED(hr) || (flags & (MF_SOURCE_READERF_ERROR | MF_SOURCE_READERF_ENDOFSTREAM)) != 0) { + if (!failed_) { + failed_ = true; + fail_hr_ = FAILED(hr) ? hr : E_FAIL; + fail_flags_ = flags; + } + } else if (sample != nullptr) { + if (pending_ != nullptr) { + dropped_++; + pending_->Release(); + } + pending_ = sample; + pending_->AddRef(); + pending_ns_ = now; + delivered_++; + } + // Keep the pipe full until close — but issue the next ReadSample OUTSIDE + // the lock: the reader may decode the next sample inline and call us + // back re-entrantly from inside ReadSample, and a held (recursive) lock + // would then starve read() for as long as the chain runs (a whole file; + // a decode's worth of latency per frame on a camera). The reader is + // AddRef'd under the lock, so teardown releasing it meanwhile is safe. + IMFSourceReader *next = nullptr; + if (!failed_ && !stopping_ && reader_ != nullptr) { + next = reader_; + next->AddRef(); + } + LeaveCriticalSection(&lock_); + SetEvent(event_); + if (next != nullptr && pace_ns_ > 0) { + // file: ids only (the decode bench) — a file is not paced by a + // sensor: the pacer thread requests the next frame at the file's + // own rate, like a camera delivers it. Never sleep here: blocking + // this MF work-queue thread stalls the reader. + next->Release(); + SetEvent(pace_event_); + return S_OK; + } + if (next != nullptr) { + HRESULT rr = next->ReadSample((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, 0, nullptr, nullptr, + nullptr, nullptr); + if (FAILED(rr)) { + EnterCriticalSection(&lock_); + if (!failed_ && !stopping_) { + failed_ = true; + fail_hr_ = rr; + fail_flags_ = 0xffffffffu; // the re-issued ReadSample itself failed + } + LeaveCriticalSection(&lock_); + SetEvent(event_); + } + next->Release(); + } + return S_OK; + } + + STDMETHODIMP + OnFlush(DWORD) override + { + SetEvent(flushed_); + return S_OK; + } + + STDMETHODIMP + OnEvent(DWORD, IMFMediaEvent *) override + { + return S_OK; + } + + // Ours. + CRITICAL_SECTION lock_; + HANDLE event_ = nullptr; + HANDLE flushed_ = nullptr; + IMFSourceReader *reader_ = nullptr; //!< raw: the reader owns us, not the reverse + IMFSample *pending_ = nullptr; + int64_t pending_ns_ = 0; + bool failed_ = false; + bool stopping_ = false; + HRESULT fail_hr_ = S_OK; + DWORD fail_flags_ = 0; //!< MF_SOURCE_READERF_* of the failing callback (all ones: ReadSample failed) + uint64_t dropped_ = 0; + uint64_t delivered_ = 0; //!< samples the reader handed us + int64_t pace_ns_ = 0; //!< file: ids only — request frames at this period + HANDLE pace_event_ = nullptr; + HANDLE pace_thread_ = nullptr; + volatile LONG pace_quit_ = 0; + + //! file: ids only: a thread that issues ReadSample every pace_ns_. + bool + start_pacer(int64_t period_ns) + { + pace_ns_ = period_ns; + pace_event_ = CreateEventW(nullptr, FALSE, FALSE, nullptr); + AddRef(); // the thread's reference + pace_thread_ = CreateThread(nullptr, 0, pacer_main, this, 0, nullptr); + if (pace_thread_ == nullptr) { + Release(); + pace_ns_ = 0; + return false; + } + return true; + } + + void + stop_pacer() + { + if (pace_thread_ != nullptr) { + InterlockedExchange(&pace_quit_, 1); + SetEvent(pace_event_); + WaitForSingleObject(pace_thread_, 2000); + CloseHandle(pace_thread_); + pace_thread_ = nullptr; + } + } + + static DWORD WINAPI + pacer_main(LPVOID p) + { + reader_cb *cb = (reader_cb *)p; + const bool com = SUCCEEDED(CoInitializeEx(nullptr, COINIT_MULTITHREADED)); + int64_t t0 = 0; + uint64_t n = 0; + bool warned = false; + while (cb->pace_quit_ == 0) { + if (WaitForSingleObject(cb->pace_event_, 500) != WAIT_OBJECT_0 || cb->pace_quit_ != 0) { + continue; + } + int64_t now = os_monotonic_get_ns(); + if (t0 == 0) { + t0 = now; + } + int64_t wait = t0 + (int64_t)(++n) * cb->pace_ns_ - now; + if (wait > 0) { + os_nanosleep(wait); + } + EnterCriticalSection(&cb->lock_); + IMFSourceReader *r = (!cb->failed_ && !cb->stopping_) ? cb->reader_ : nullptr; + if (r != nullptr) { + r->AddRef(); + } + LeaveCriticalSection(&cb->lock_); + if (r != nullptr) { + HRESULT hr = r->ReadSample((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, 0, nullptr, + nullptr, nullptr, nullptr); + if (FAILED(hr) && !warned) { + warned = true; + U_LOG_W("uvc capture (decode bench pacer): ReadSample failed, hr 0x%08lx", + (unsigned long)hr); + } + r->Release(); + } + } + if (com) { + CoUninitialize(); + } + cb->Release(); + return 0; + } + +private: + ~reader_cb() + { + if (pending_ != nullptr) { + pending_->Release(); + } + if (event_ != nullptr) { + CloseHandle(event_); + } + if (flushed_ != nullptr) { + CloseHandle(flushed_); + } + if (pace_event_ != nullptr) { + CloseHandle(pace_event_); + } + DeleteCriticalSection(&lock_); + } + + volatile LONG refs_ = 1; +}; + +//! One open device. +struct mf_handle +{ + mf_scope scope; + DWORD thread_id = 0; + IMFMediaSource *source = nullptr; + IMFSourceReader *reader = nullptr; + reader_cb *cb = nullptr; + uint32_t pixel = 0; //!< enum u_stereo_uvc_pixel of the reader's OUTPUT + uint32_t width = 0, height = 0; + LONG default_stride = 0; + + // Hardware path. + bool hw = false; + ID3D11Device *dev = nullptr; + ID3D11DeviceContext *ctx = nullptr; + IMFDXGIDeviceManager *dxgi_mgr = nullptr; + //! The pipelined readback ring (read_back_texture). + static const int RING = 3; + ID3D11Texture2D *ring[RING] = {}; + ID3D11Query *ring_q[RING] = {}; + int64_t ring_t[RING] = {}; + int pend[RING] = {}; //!< slots whose copy is queued, not yet returned (oldest first) + int npend = 0; + uint8_t *cpu = nullptr; //!< the read-back frame (tight NV12), valid until the next read + size_t cpu_size = 0; + + // The sample whose buffer read() handed out, locked until the next read / close. + IMFSample *held = nullptr; + IMFMediaBuffer *held_buf = nullptr; + IMF2DBuffer2 *held_2d = nullptr; + bool held_locked = false; //!< held_buf->Lock() (not 2D) is outstanding + + // Diagnostics. + char decoder[160]; + uint64_t frames = 0; + int64_t readback_ns = 0; +}; + +void +unlock_held(mf_handle *h) +{ + if (h->held_2d != nullptr) { + h->held_2d->Unlock2D(); + safe_release(h->held_2d); + } + if (h->held_locked) { + h->held_buf->Unlock(); + h->held_locked = false; + } + safe_release(h->held_buf); + safe_release(h->held); +} + + +/* + * + * Backend ops. + * + */ + +uint32_t +mf_enumerate(void *ctx, u_stereo_uvc_device *out, uint32_t cap) +{ + (void)ctx; + mf_scope scope; + if (!scope.begin()) { + scope.end(); + return 0; + } + IMFAttributes *attr = nullptr; + IMFActivate **devs = nullptr; + UINT32 count = 0; + HRESULT hr = MFCreateAttributes(&attr, 1); + if (SUCCEEDED(hr)) { + hr = attr->SetGUID(MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE, MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE_VIDCAP_GUID); + } + if (SUCCEEDED(hr)) { + hr = MFEnumDeviceSources(attr, &devs, &count); + } + uint32_t n = 0; + for (UINT32 i = 0; SUCCEEDED(hr) && i < count; i++) { + if (n < cap && out != nullptr) { + u_stereo_uvc_device *d = &out[n]; + std::memset(d, 0, sizeof(*d)); + WCHAR *s = nullptr; + UINT32 len = 0; + if (SUCCEEDED(devs[i]->GetAllocatedString(MF_DEVSOURCE_ATTRIBUTE_FRIENDLY_NAME, &s, &len))) { + wide_to_utf8(s, d->name, sizeof(d->name)); + CoTaskMemFree(s); + } + s = nullptr; + if (SUCCEEDED(devs[i]->GetAllocatedString( + MF_DEVSOURCE_ATTRIBUTE_SOURCE_TYPE_VIDCAP_SYMBOLIC_LINK, &s, &len))) { + wide_to_utf8(s, d->id, sizeof(d->id)); + CoTaskMemFree(s); + } + d->has_vid_pid = u_stereo_uvc_parse_vid_pid(d->id, &d->vid, &d->pid); + n++; + } + devs[i]->Release(); + } + CoTaskMemFree(devs); + safe_release(attr); + scope.end(); + return n; +} + +uint32_t +mf_list_modes(void *ctx, const char *id, u_stereo_uvc_mode *out, uint32_t cap) +{ + (void)ctx; + mf_scope scope; + if (!scope.begin()) { + scope.end(); + return 0; + } + IMFMediaSource *src = nullptr; + IMFPresentationDescriptor *pd = nullptr; + IMFStreamDescriptor *sd = nullptr; + IMFMediaTypeHandler *th = nullptr; + uint32_t n = 0; + BOOL selected = FALSE; + DWORD types = 0; + HRESULT hr = create_device_source(id, &src); + if (SUCCEEDED(hr)) { + hr = src->CreatePresentationDescriptor(&pd); + } + if (SUCCEEDED(hr)) { + hr = pd->GetStreamDescriptorByIndex(0, &selected, &sd); + } + if (SUCCEEDED(hr)) { + hr = sd->GetMediaTypeHandler(&th); + } + if (SUCCEEDED(hr)) { + hr = th->GetMediaTypeCount(&types); + } + for (DWORD i = 0; SUCCEEDED(hr) && i < types && n < cap; i++) { + IMFMediaType *t = nullptr; + if (FAILED(th->GetMediaTypeByIndex(i, &t))) { + continue; + } + u_stereo_uvc_mode m; + GUID st; + if (type_mode(t, &m, &st) && subtype_rank(st) > 0) { + bool dup = false; + for (uint32_t k = 0; k < n; k++) { + dup |= out[k].width == m.width && out[k].height == m.height && out[k].fps == m.fps; + } + if (!dup && out != nullptr) { + out[n++] = m; + } + } + safe_release(t); + } + safe_release(th); + safe_release(sd); + safe_release(pd); + if (src != nullptr) { + src->Shutdown(); + safe_release(src); + } + scope.end(); + return n; +} + +//! Pick the native type: the exact size, the rate closest to @p want (0 = fastest), NV12 > YUY2 > MJPG. +IMFMediaType * +pick_native(IMFSourceReader *reader, const u_stereo_uvc_mode *want) +{ + IMFMediaType *best = nullptr; + float best_fps = 0.0f; + int best_rank = 0; + for (DWORD i = 0;; i++) { + IMFMediaType *t = nullptr; + HRESULT hr = reader->GetNativeMediaType((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, i, &t); + if (hr == MF_E_NO_MORE_TYPES || FAILED(hr)) { + break; + } + u_stereo_uvc_mode m; + GUID st = GUID_NULL; + const bool sized = type_mode(t, &m, &st); + const int rank = sized ? subtype_rank(st) : 0; + if (rank > 0 && m.width == want->width && m.height == want->height) { + bool better; + if (best == nullptr) { + better = true; + } else if (want->fps > 0.0f) { + float d = m.fps > want->fps ? m.fps - want->fps : want->fps - m.fps; + float bd = best_fps > want->fps ? best_fps - want->fps : want->fps - best_fps; + better = d < bd - 0.01f || (d <= bd + 0.01f && rank > best_rank); + } else { + better = m.fps > best_fps + 0.01f || (m.fps >= best_fps - 0.01f && rank > best_rank); + } + if (better) { + safe_release(best); + best = t; + best->AddRef(); + best_fps = m.fps; + best_rank = rank; + } + } + safe_release(t); + } + return best; +} + +//! Ask the reader for @p subtype output at @p w x @p h (0 = the native size). +HRESULT +set_output(IMFSourceReader *reader, IMFMediaType *native, const GUID &subtype, uint32_t w, uint32_t h) +{ + IMFMediaType *out = nullptr; + HRESULT hr = MFCreateMediaType(&out); + if (SUCCEEDED(hr)) { + hr = out->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video); + } + if (SUCCEEDED(hr)) { + hr = out->SetGUID(MF_MT_SUBTYPE, subtype); + } + UINT32 nw = 0, nh = 0, num = 0, den = 0; + if (SUCCEEDED(hr) && SUCCEEDED(MFGetAttributeSize(native, MF_MT_FRAME_SIZE, &nw, &nh))) { + hr = MFSetAttributeSize(out, MF_MT_FRAME_SIZE, w > 0 ? w : nw, h > 0 ? h : nh); + } + if (SUCCEEDED(hr) && SUCCEEDED(MFGetAttributeRatio(native, MF_MT_FRAME_RATE, &num, &den))) { + hr = MFSetAttributeRatio(out, MF_MT_FRAME_RATE, num, den); + } + if (SUCCEEDED(hr)) { + // The video processor preserves the DISPLAY aspect ratio: asked for + // 2560x720 from a 16:9 3840x2160 frame with square pixels it LETTERBOXES + // (measured on the real webcam: a centred ~1280 px picture between + // black bars). Declare the output's pixels anamorphic so its display + // aspect equals the source's — the scale is then a plain per-axis + // resize of the whole SBS frame, each half staying its own eye: + // PAR_out = PAR_src * (src_w * out_h) / (src_h * out_w) (1:2 here) + UINT32 pn = 1, pd = 1; + if (FAILED(MFGetAttributeRatio(native, MF_MT_PIXEL_ASPECT_RATIO, &pn, &pd)) || pn == 0 || pd == 0) { + pn = pd = 1; + } + const uint32_t ow = w > 0 ? w : nw, oh = h > 0 ? h : nh; + uint64_t num = (uint64_t)pn * nw * oh, den = (uint64_t)pd * nh * ow; + for (uint64_t a = num, b = den; b != 0;) { // reduce by the gcd + uint64_t r = a % b; + a = b; + b = r; + if (b == 0) { + num /= a; + den /= a; + } + } + while (num > 0xffffffffull || den > 0xffffffffull) { + num >>= 1; + den >>= 1; + } + hr = MFSetAttributeRatio(out, MF_MT_PIXEL_ASPECT_RATIO, (UINT32)(num > 0 ? num : 1), + (UINT32)(den > 0 ? den : 1)); + } + if (SUCCEEDED(hr)) { + hr = out->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive); + } + if (SUCCEEDED(hr)) { + hr = reader->SetCurrentMediaType((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, nullptr, out); + } + safe_release(out); + return hr; +} + +/*! + * The decoder the reader put in front of our output, after the topology is + * built: " (hardware)" / "(software)" into @p out. Returns whether it is + * a hardware MFT. No decoder (an NV12 / YUY2 native type) counts as hardware- + * neutral: true, "none (uncompressed native type)". + */ +bool +describe_decoder(IMFSourceReader *reader, char *out, size_t cap) +{ + std::snprintf(out, cap, "none (uncompressed native type)"); + IMFSourceReaderEx *ex = nullptr; + if (FAILED(reader->QueryInterface(IID_PPV_ARGS(&ex)))) { + std::snprintf(out, cap, "unknown (no IMFSourceReaderEx)"); + return false; + } + bool hw = true; + for (DWORD i = 0;; i++) { + GUID cat = GUID_NULL; + IMFTransform *t = nullptr; + if (FAILED(ex->GetTransformForStream((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, i, &cat, &t))) { + break; + } + if (cat == MFT_CATEGORY_VIDEO_DECODER) { + IMFAttributes *a = nullptr; + char name[128] = "(unnamed decoder)"; + bool is_hw = false; + if (SUCCEEDED(t->GetAttributes(&a)) && a != nullptr) { + WCHAR *s = nullptr; + UINT32 len = 0; + if (SUCCEEDED(a->GetAllocatedString(MFT_FRIENDLY_NAME_Attribute, &s, &len))) { + wide_to_utf8(s, name, sizeof(name)); + CoTaskMemFree(s); + } + // A hardware MFT carries its device URL and is asynchronous. + UINT32 url_len = 0; + is_hw = SUCCEEDED(a->GetStringLength(MFT_ENUM_HARDWARE_URL_Attribute, &url_len)) || + MFGetAttributeUINT32(a, MF_TRANSFORM_ASYNC, FALSE) != FALSE; + safe_release(a); + } + std::snprintf(out, cap, "%s (%s)", name, is_hw ? "hardware" : "software"); + hw = is_hw; + } + safe_release(t); + } + safe_release(ex); + return hw; +} + +void +mf_close(void *handle); + +/*! + * Create the D3D11 device + DXGI device manager on @p adapter for the + * hardware path. Returns false (handle untouched beyond its d3d members) on + * any failure. + */ +bool +create_hw_device(mf_handle *h, IDXGIAdapter1 *adapter) +{ + static const D3D_FEATURE_LEVEL levels[] = {D3D_FEATURE_LEVEL_11_1, D3D_FEATURE_LEVEL_11_0}; + UINT flags = D3D11_CREATE_DEVICE_VIDEO_SUPPORT | D3D11_CREATE_DEVICE_BGRA_SUPPORT; + HRESULT hr = + D3D11CreateDevice(adapter, D3D_DRIVER_TYPE_UNKNOWN, nullptr, flags, levels, + (UINT)(sizeof(levels) / sizeof(levels[0])), D3D11_SDK_VERSION, &h->dev, nullptr, &h->ctx); + if (FAILED(hr)) { + return false; + } + ID3D10Multithread *mt = nullptr; + if (SUCCEEDED(h->dev->QueryInterface(IID_PPV_ARGS(&mt)))) { + mt->SetMultithreadProtected(TRUE); // MF's decoder threads share this device + safe_release(mt); + } + UINT token = 0; + hr = MFCreateDXGIDeviceManager(&token, &h->dxgi_mgr); + if (SUCCEEDED(hr)) { + hr = h->dxgi_mgr->ResetDevice(h->dev, token); + } + return SUCCEEDED(hr); +} + +/*! + * One attempt at opening the reader. @p adapter non-NULL = the hardware path + * (D3D manager, GPU scaling to @p out_w x @p out_h). + */ +HRESULT +open_reader( + mf_handle *h, const char *id, const u_stereo_uvc_mode *mode, IDXGIAdapter1 *adapter, uint32_t out_w, uint32_t out_h) +{ + h->hw = adapter != nullptr; + h->cb = new (std::nothrow) reader_cb(); + IMFAttributes *attr = nullptr; + IMFMediaType *native = nullptr; + IMFMediaType *cur = nullptr; + HRESULT hr = h->cb != nullptr && h->cb->event_ != nullptr ? S_OK : E_OUTOFMEMORY; + if (SUCCEEDED(hr) && h->hw && !create_hw_device(h, adapter)) { + hr = E_FAIL; + } + if (SUCCEEDED(hr)) { + hr = create_device_source(id, &h->source); + } + if (SUCCEEDED(hr)) { + hr = MFCreateAttributes(&attr, 4); + } + if (SUCCEEDED(hr)) { + hr = attr->SetUnknown(MF_SOURCE_READER_ASYNC_CALLBACK, static_cast(h->cb)); + } + if (SUCCEEDED(hr)) { + // The reader inserts the MJPEG decoder MFT + colour conversion (+ scaling) to NV12. + hr = attr->SetUINT32(MF_SOURCE_READER_ENABLE_ADVANCED_VIDEO_PROCESSING, TRUE); + } + if (SUCCEEDED(hr) && h->hw) { + hr = attr->SetUINT32(MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, TRUE); + if (SUCCEEDED(hr)) { + hr = attr->SetUnknown(MF_SOURCE_READER_D3D_MANAGER, h->dxgi_mgr); + } + } + if (SUCCEEDED(hr)) { + hr = MFCreateSourceReaderFromMediaSource(h->source, attr, &h->reader); + } + if (SUCCEEDED(hr)) { + native = pick_native(h->reader, mode); + hr = native != nullptr ? S_OK : MF_E_INVALIDMEDIATYPE; + } + if (SUCCEEDED(hr)) { + hr = h->reader->SetCurrentMediaType((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, nullptr, native); + } + if (SUCCEEDED(hr)) { + // Ask for the OUTPUT size on both paths: hardware decodes AND scales on + // the GPU; software lets MF's (multithreaded) video processor scale, + // which beats decoding 4K and resampling it on the camera thread. If + // the scaler refuses, native size and the source downscales. + if (SUCCEEDED(set_output(h->reader, native, MFVideoFormat_NV12, out_w, out_h))) { + h->pixel = U_STEREO_UVC_PIXEL_NV12; + } else if (out_w > 0 && SUCCEEDED(set_output(h->reader, native, MFVideoFormat_NV12, 0, 0))) { + h->pixel = U_STEREO_UVC_PIXEL_NV12; // no scaler: full size, the source downscales + } else if (!h->hw && SUCCEEDED(set_output(h->reader, native, MFVideoFormat_YUY2, 0, 0))) { + h->pixel = U_STEREO_UVC_PIXEL_YUY2; + } else { + hr = MF_E_INVALIDMEDIATYPE; + } + } + if (SUCCEEDED(hr)) { + hr = h->reader->GetCurrentMediaType((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, &cur); + } + if (SUCCEEDED(hr)) { + UINT32 w = 0, hh = 0; + MFGetAttributeSize(cur, MF_MT_FRAME_SIZE, &w, &hh); + h->width = w; + h->height = hh; + UINT32 stride = 0; + if (SUCCEEDED(cur->GetUINT32(MF_MT_DEFAULT_STRIDE, &stride))) { + h->default_stride = (LONG)stride; + } else { + h->default_stride = (LONG)(h->pixel == U_STEREO_UVC_PIXEL_YUY2 ? 2 * w : w); + } + } + if (SUCCEEDED(hr)) { + bool dec_hw = describe_decoder(h->reader, h->decoder, sizeof(h->decoder)); + if (h->hw && !dec_hw) { + hr = MF_E_TOPO_CODEC_NOT_FOUND; // a software decoder behind a D3D manager: nothing gained + } + } + u_stereo_uvc_mode nm = {0, 0, 0.0f}; + if (native != nullptr) { + type_mode(native, &nm, nullptr); + } + if (SUCCEEDED(hr) && std::strncmp(id, "file:", 5) == 0 && nm.fps > 0.0f) { + h->cb->start_pacer((int64_t)(1e9 / nm.fps)); // the decode bench: camera-like pacing + } + if (SUCCEEDED(hr)) { + EnterCriticalSection(&h->cb->lock_); + h->cb->reader_ = h->reader; + LeaveCriticalSection(&h->cb->lock_); + hr = h->reader->ReadSample((DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, 0, nullptr, nullptr, nullptr, + nullptr); + } + if (SUCCEEDED(hr)) { + U_LOG_W( + "uvc capture (Media Foundation): started %ux%u@%.2f — decoder %s, %s path, " + "reader output NV12 %ux%u", + nm.width, nm.height, nm.fps, h->decoder, h->hw ? "HARDWARE (GPU decode + scale)" : "SOFTWARE", + h->width, h->height); + } + safe_release(cur); + safe_release(native); + safe_release(attr); + return hr; +} + +//! Tear down everything open_reader built (keeps the COM/MF scope). +void +teardown_reader(mf_handle *h) +{ + unlock_held(h); + if (h->cb != nullptr) { + EnterCriticalSection(&h->cb->lock_); + h->cb->stopping_ = true; + LeaveCriticalSection(&h->cb->lock_); + h->cb->stop_pacer(); + } + if (h->reader != nullptr) { + // Drain the outstanding ReadSample before the reader goes away. + if (h->cb != nullptr && h->cb->reader_ != nullptr && + SUCCEEDED(h->reader->Flush((DWORD)MF_SOURCE_READER_ALL_STREAMS))) { + WaitForSingleObject(h->cb->flushed_, 1000); + } + if (h->cb != nullptr) { + EnterCriticalSection(&h->cb->lock_); + h->cb->reader_ = nullptr; + LeaveCriticalSection(&h->cb->lock_); + } + safe_release(h->reader); + } + if (h->source != nullptr) { + h->source->Shutdown(); + safe_release(h->source); + } + if (h->cb != nullptr) { + if (h->cb->dropped_ > 0) { + U_LOG_I("uvc capture (Media Foundation): closed; %llu sample(s) superseded before a read", + (unsigned long long)h->cb->dropped_); + } + h->cb->Release(); + h->cb = nullptr; + } + for (int i = 0; i < mf_handle::RING; i++) { + safe_release(h->ring_q[i]); + safe_release(h->ring[i]); + } + h->npend = 0; + safe_release(h->dxgi_mgr); + safe_release(h->ctx); + safe_release(h->dev); + h->hw = false; +} + +bool +mf_open(void *ctx, const char *id, const u_stereo_uvc_mode *mode, uint32_t out_w, uint32_t out_h, void **out_handle) +{ + (void)ctx; + *out_handle = nullptr; + mf_handle *h = new (std::nothrow) mf_handle(); + if (h == nullptr) { + return false; + } + std::snprintf(h->decoder, sizeof(h->decoder), "?"); + h->thread_id = GetCurrentThreadId(); + if (!h->scope.begin()) { + mf_close(h); + return false; + } + const decoder_pref pref = read_decoder_pref(); + HRESULT hr = E_FAIL; + char why[256] = {0}; + if (pref != decoder_pref::software) { + adapter_choice c = choose_adapter(nullptr, 0); + if (c.found) { + hr = open_reader(h, id, mode, c.adapter, out_w, out_h); + if (FAILED(hr)) { + std::snprintf(why, sizeof(why), + "hardware path on \"%s\" (decoder MFT \"%s\") failed, hr 0x%08lx%s", + c.desc, c.decoder, (unsigned long)hr, + hr == MF_E_TOPO_CODEC_NOT_FOUND ? " — MF chose a software decoder anyway" + : ""); + teardown_reader(h); + } else if (c.why[0] != '\0') { + U_LOG_W("uvc capture: %s — decoding on \"%s\"", c.why, c.desc); + } + } else { + std::snprintf(why, sizeof(why), "%s", c.why); + } + safe_release(c.adapter); + if (FAILED(hr) && pref == decoder_pref::hardware) { + U_LOG_W( + "uvc capture (Media Foundation): DXR_STEREO_CAMERA_UVC_DECODER=hardware and no " + "hardware path (%s) — not opening", + why); + mf_close(h); + return false; + } + } else { + std::snprintf(why, sizeof(why), "DXR_STEREO_CAMERA_UVC_DECODER=software"); + } + if (FAILED(hr)) { + U_LOG_W( + "uvc capture (Media Foundation): SOFTWARE decode (%s) — slower and more CPU than the hardware path", + why); + hr = open_reader(h, id, mode, nullptr, out_w, out_h); + } + if (FAILED(hr)) { + U_LOG_W("uvc capture (Media Foundation): open %ux%u@%.1f failed, hr 0x%08lx", mode->width, mode->height, + mode->fps, (unsigned long)hr); + mf_close(h); + return false; + } + *out_handle = h; + return true; +} + +/*! + * Hardware path readback, PIPELINED over a ring of @ref RING staging textures: + * this read queues the GPU copy of the frame just decoded into one slot and + * returns the frame queued by the PREVIOUS read, whose copy has long finished + * — so the readback overlaps the decode of the next frame instead of waiting + * for it (a synchronous copy + Map measured 22-25 ms per 4K MJPEG frame on + * the real webcam: the copy waits behind the decode). Cost: one frame of + * latency. The very first read only primes the ring. + * @return 1 = @p out holds a frame (stamped @p out_t), 0 = primed (no frame + * yet), -1 = not a DXGI texture / failure (caller locks the buffer). + */ +int +map_slot(mf_handle *h, int take, int64_t t0, u_stereo_uvc_raw_frame *out, int64_t *out_t); + +int +read_back_texture(mf_handle *h, IMFMediaBuffer *buf, int64_t t_ns, u_stereo_uvc_raw_frame *out, int64_t *out_t) +{ + IMFDXGIBuffer *db = nullptr; + if (FAILED(buf->QueryInterface(IID_PPV_ARGS(&db)))) { + return -1; + } + ID3D11Texture2D *tex = nullptr; + UINT sub = 0; + bool ok = SUCCEEDED(db->GetResource(IID_PPV_ARGS(&tex))) && SUCCEEDED(db->GetSubresourceIndex(&sub)); + safe_release(db); + if (!ok) { + safe_release(tex); + return -1; + } + D3D11_TEXTURE2D_DESC td; + tex->GetDesc(&td); + if (td.Format != DXGI_FORMAT_NV12 || td.Width < h->width || td.Height < h->height) { + safe_release(tex); + return -1; + } + for (int i = 0; i < mf_handle::RING; i++) { + if (h->ring[i] == nullptr) { + D3D11_TEXTURE2D_DESC sd = {}; + sd.Width = h->width; + sd.Height = h->height; + sd.MipLevels = 1; + sd.ArraySize = 1; + sd.Format = DXGI_FORMAT_NV12; + sd.SampleDesc.Count = 1; + sd.Usage = D3D11_USAGE_STAGING; + sd.CPUAccessFlags = D3D11_CPU_ACCESS_READ; + D3D11_QUERY_DESC qd = {D3D11_QUERY_EVENT, 0}; + if (FAILED(h->dev->CreateTexture2D(&sd, nullptr, &h->ring[i])) || + FAILED(h->dev->CreateQuery(&qd, &h->ring_q[i]))) { + safe_release(tex); + return -1; + } + } + } + const int64_t t0 = os_monotonic_get_ns(); + // Queue this frame's copy into a free slot (the GPU orders it after the + // decode that wrote the texture; the decoder may reuse the texture after). + int s = -1; + for (int c = 0; c < mf_handle::RING && s < 0; c++) { + bool used = false; + for (int k = 0; k < h->npend; k++) { + used |= h->pend[k] == c; + } + s = used ? -1 : c; + } + if (s < 0) { // cannot happen (at most RING - 1 pending): reset the ring + h->npend = 0; + s = 0; + } + D3D11_BOX box = {0, 0, 0, h->width, h->height, 1}; + h->ctx->CopySubresourceRegion(h->ring[s], 0, 0, 0, 0, tex, sub, &box); + safe_release(tex); + h->ctx->End(h->ring_q[s]); + h->ctx->Flush(); + h->ring_t[s] = t_ns; + h->pend[h->npend++] = s; + // Deliver the NEWEST copy that has already finished — possibly this very + // frame when the GPU is quick — and drop the older ones (latest wins). + for (int k = h->npend - 1; k >= 0; k--) { + if (h->ctx->GetData(h->ring_q[h->pend[k]], nullptr, 0, D3D11_ASYNC_GETDATA_DONOTFLUSH) == S_OK) { + const int take = h->pend[k]; + const int remain = h->npend - 1 - k; + for (int r = 0; r < remain; r++) { + h->pend[r] = h->pend[k + 1 + r]; + } + h->npend = remain; + return map_slot(h, take, t0, out, out_t); + } + } + // None finished yet. With two copies in flight, wait for the older one (it + // was queued a frame ago); with one, the frame comes out on a later read. + if (h->npend >= 2) { + const int take = h->pend[0]; + for (int r = 0; r + 1 < h->npend; r++) { + h->pend[r] = h->pend[r + 1]; + } + h->npend--; + return map_slot(h, take, t0, out, out_t); + } + return 0; +} + +/*! + * Map ring slot @p take (its copy was queued earlier), copy the frame out to + * h->cpu, Unmap. 1 = frame in @p out, -1 = failure. + */ +int +map_slot(mf_handle *h, int take, int64_t t0, u_stereo_uvc_raw_frame *out, int64_t *out_t) +{ + // The previous copy: normally done long ago. Poll (never a blocking Map: + // the device is multithread-protected and shared with MF's decoder). + const int64_t give_up = os_monotonic_get_ns() + 100ll * 1000 * 1000; + while (h->ctx->GetData(h->ring_q[take], nullptr, 0, D3D11_ASYNC_GETDATA_DONOTFLUSH) == S_FALSE && + os_monotonic_get_ns() < give_up) { + Sleep(0); + } + D3D11_MAPPED_SUBRESOURCE m = {}; + if (FAILED(h->ctx->Map(h->ring[take], 0, D3D11_MAP_READ, 0, &m))) { + return -1; + } + // Copy out and Unmap at once: no staging texture or decoder sample is held + // while the frame is consumed. + const size_t pitch = h->width; + const size_t need = pitch * h->height * 3 / 2; + if (h->cpu_size < need) { + uint8_t *nb = (uint8_t *)std::realloc(h->cpu, need); + if (nb == nullptr) { + h->ctx->Unmap(h->ring[take], 0); + return -1; + } + h->cpu = nb; + h->cpu_size = need; + } + const uint8_t *src_uv = (const uint8_t *)m.pData + (size_t)m.RowPitch * h->height; // UV follows Y + for (uint32_t y = 0; y < h->height; y++) { + std::memcpy(h->cpu + y * pitch, (const uint8_t *)m.pData + (size_t)y * m.RowPitch, pitch); + } + for (uint32_t y = 0; y < h->height / 2; y++) { + std::memcpy(h->cpu + (h->height + y) * pitch, src_uv + (size_t)y * m.RowPitch, pitch); + } + h->ctx->Unmap(h->ring[take], 0); + h->readback_ns += os_monotonic_get_ns() - t0; + out->pixel = U_STEREO_UVC_PIXEL_NV12; + out->width = h->width; + out->height = h->height; + out->planes[0] = h->cpu; + out->pitches[0] = (uint32_t)pitch; + out->planes[1] = h->cpu + pitch * h->height; + out->pitches[1] = (uint32_t)pitch; + *out_t = h->ring_t[take]; + return 1; +} + +uint32_t +mf_read(void *handle, int64_t timeout_ns, u_stereo_uvc_raw_frame *out) +{ + mf_handle *h = (mf_handle *)handle; + unlock_held(h); // the previous frame's planes are released by contract + + IMFSample *sample = nullptr; + int64_t t_ns = 0; + bool failed = false; + // Wait against a DEADLINE: the auto-reset event can be left signalled by an + // earlier callback whose sample was already taken, so one wait may return + // with nothing pending — that is not a timeout yet. + const int64_t deadline = os_monotonic_get_ns() + (timeout_ns > 0 ? timeout_ns : 0); + for (;;) { + EnterCriticalSection(&h->cb->lock_); + bool have = h->cb->pending_ != nullptr || h->cb->failed_; + LeaveCriticalSection(&h->cb->lock_); + const int64_t left = deadline - os_monotonic_get_ns(); + if (have || left <= 0) { + break; + } + WaitForSingleObject(h->cb->event_, (DWORD)((left + 999999) / 1000000)); + } + EnterCriticalSection(&h->cb->lock_); + sample = h->cb->pending_; + h->cb->pending_ = nullptr; + t_ns = h->cb->pending_ns_; + failed = h->cb->failed_; + HRESULT fail_hr = h->cb->fail_hr_; + DWORD fail_flags = h->cb->fail_flags_; + const unsigned long long delivered = h->cb->delivered_, superseded = h->cb->dropped_; + LeaveCriticalSection(&h->cb->lock_); + if (sample == nullptr && h->hw && h->npend > 0) { + // No newer frame within the timeout (end of a file, a paused camera): + // deliver the newest one still in the readback ring rather than lose it. + const int take = h->pend[h->npend - 1]; + h->npend = 0; + std::memset(out, 0, sizeof(*out)); + int64_t t_take = 0; + if (map_slot(h, take, os_monotonic_get_ns(), out, &t_take) > 0) { + out->time_ns = t_take; + out->time_is_exposure = false; + h->frames++; + return U_STEREO_UVC_READ_OK; + } + } + if (sample == nullptr) { + if (failed) { + U_LOG_W( + "uvc capture (Media Foundation): the device stopped delivering (hr 0x%08lx, reader flags " + "0x%lx%s) after %llu frame(s) read; %llu delivered by the reader, %llu superseded", + (unsigned long)fail_hr, (unsigned long)fail_flags, + (fail_flags != 0xffffffffu && (fail_flags & MF_SOURCE_READERF_ENDOFSTREAM)) + ? " = end of stream" + : "", + (unsigned long long)h->frames, delivered, superseded); + return U_STEREO_UVC_READ_ERROR; + } + return U_STEREO_UVC_READ_TIMEOUT; + } + + h->held = sample; + if (FAILED(sample->GetBufferByIndex(0, &h->held_buf))) { + unlock_held(h); + return U_STEREO_UVC_READ_TIMEOUT; + } + std::memset(out, 0, sizeof(*out)); + const int rb = h->hw ? read_back_texture(h, h->held_buf, t_ns, out, &t_ns) : -1; + if (rb == 0) { + unlock_held(h); // primed: the copy is queued, the sample can go back to the decoder + return U_STEREO_UVC_READ_TIMEOUT; + } + if (rb > 0) { + unlock_held(h); // the frame is in h->cpu now: give the sample back to the decoder + out->time_ns = t_ns; + out->time_is_exposure = false; + if (++h->frames == 300) { + U_LOG_W( + "uvc capture (Media Foundation): hardware path GPU->CPU readback %.3f ms/frame over the " + "first 300 frames (%ux%u NV12)", + (double)h->readback_ns / 300.0 * 1e-6, h->width, h->height); + } + return U_STEREO_UVC_READ_OK; + } + BYTE *base = nullptr; + LONG pitch = 0; + DWORD len = 0; + IMF2DBuffer2 *b2 = nullptr; + if (SUCCEEDED(h->held_buf->QueryInterface(IID_PPV_ARGS(&b2)))) { + BYTE *start = nullptr; + if (SUCCEEDED(b2->Lock2DSize(MF2DBuffer_LockFlags_Read, &base, &pitch, &start, &len))) { + h->held_2d = b2; // keeps the ref; Unlock2D on release + b2 = nullptr; + } else { + base = nullptr; + } + safe_release(b2); + } + if (base == nullptr) { + DWORD max = 0; + if (FAILED(h->held_buf->Lock(&base, &max, &len))) { + unlock_held(h); + return U_STEREO_UVC_READ_TIMEOUT; + } + h->held_locked = true; + pitch = h->default_stride; + } + if (pitch <= 0) { // bottom-up never happens for YUV; refuse rather than mis-read + unlock_held(h); + return U_STEREO_UVC_READ_TIMEOUT; + } + out->pixel = h->pixel; + out->width = h->width; + out->height = h->height; + out->planes[0] = base; + out->pitches[0] = (uint32_t)pitch; + if (h->pixel == U_STEREO_UVC_PIXEL_NV12) { + // The UV plane follows the (possibly row-padded, e.g. 1088 for 1080) Y plane. + uint32_t rows = (uint32_t)(len / (DWORD)pitch) * 2 / 3; + if (rows < h->height) { + unlock_held(h); + return U_STEREO_UVC_READ_TIMEOUT; + } + out->planes[1] = base + (size_t)rows * (size_t)pitch; + out->pitches[1] = (uint32_t)pitch; + } else if (len < (DWORD)pitch * h->height) { + unlock_held(h); + return U_STEREO_UVC_READ_TIMEOUT; + } + out->time_ns = t_ns; + out->time_is_exposure = false; + h->frames++; + return U_STEREO_UVC_READ_OK; +} + +void +mf_close(void *handle) +{ + mf_handle *h = (mf_handle *)handle; + if (h == nullptr) { + return; + } + if (h->frames > 0) { + U_LOG_W("uvc capture (Media Foundation): closed after %llu frame(s) — decoder %s%s", + (unsigned long long)h->frames, h->decoder, h->hw ? "" : " (software path)"); + if (h->hw) { + U_LOG_W("uvc capture (Media Foundation): mean GPU->CPU readback %.3f ms/frame (%ux%u NV12)", + (double)h->readback_ns / (double)h->frames * 1e-6, h->width, h->height); + } + } + teardown_reader(h); + std::free(h->cpu); + if (GetCurrentThreadId() == h->thread_id) { + h->scope.end(); + } + delete h; +} + +} // namespace + +extern "C" bool +os_uvc_capture_backend(struct u_stereo_uvc_backend *out) +{ + std::memset(out, 0, sizeof(*out)); + out->name = "media-foundation"; + out->enumerate = mf_enumerate; + out->list_modes = mf_list_modes; + out->open = mf_open; + out->read = mf_read; + out->close = mf_close; + return true; +} + +extern "C" void +os_uvc_capture_decoder_report(char *out, size_t cap) +{ + if (out == nullptr || cap == 0) { + return; + } + out[0] = '\0'; + mf_scope scope; + if (!scope.begin()) { + scope.end(); + appendf(out, cap, "Media Foundation unavailable\n"); + return; + } + const decoder_pref pref = read_decoder_pref(); + appendf(out, cap, "decoder policy: %s (DXR_STEREO_CAMERA_UVC_DECODER)\n", + pref == decoder_pref::software ? "software" + : pref == decoder_pref::hardware ? "hardware only" + : "auto (hardware, else software)"); + adapter_choice c = choose_adapter(out, cap); + if (pref == decoder_pref::software) { + appendf(out, cap, "=> SOFTWARE decode (forced)\n"); + } else if (c.found) { + appendf(out, cap, + "=> HARDWARE decode on \"%s\" with \"%s\"%s%s (confirmed per open: the service logs the " + "decoder MF actually inserted)\n", + c.desc, c.decoder, c.why[0] ? "; " : "", c.why); + } else { + appendf(out, cap, "=> %s decode: %s\n", + pref == decoder_pref::hardware ? "NO (hardware only)" : "SOFTWARE", c.why); + } + safe_release(c.adapter); + scope.end(); +} diff --git a/src/xrt/auxiliary/os/os_uvc_capture_stubs.c b/src/xrt/auxiliary/os/os_uvc_capture_stubs.c new file mode 100644 index 0000000000..d65aa0961f --- /dev/null +++ b/src/xrt/auxiliary/os/os_uvc_capture_stubs.c @@ -0,0 +1,38 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief No UVC capture backend on this platform (yet) — see os_uvc_capture.h. + * + * TODO(linux): a V4L2 backend for desktop Linux (XRT_OS_LINUX_DESKTOP): + * enumerate /dev/video* with VIDIOC_QUERYCAP (V4L2_CAP_VIDEO_CAPTURE, skipping + * metadata nodes) and the USB VID:PID from sysfs + * (/sys/class/video4linux/videoN/device/../idVendor, idProduct); modes from + * VIDIOC_ENUM_FMT / ENUM_FRAMESIZES / ENUM_FRAMEINTERVALS; open = S_FMT + S_PARM + * + mmap'd buffers + STREAMON on a reader thread keeping the newest buffer; + * YUYV is handed over as U_STEREO_UVC_PIXEL_YUY2, MJPEG needs a decoder (libjpeg- + * turbo, dlopen'd so the runtime keeps no hard dependency). Until then a Linux + * service sees only the config's fake device. + * + * @ingroup aux_os + */ + +#include "os/os_uvc_capture.h" + +#include +#include + +bool +os_uvc_capture_backend(struct u_stereo_uvc_backend *out) +{ + memset(out, 0, sizeof(*out)); + return false; +} + +void +os_uvc_capture_decoder_report(char *out, size_t cap) +{ + if (out != NULL && cap > 0) { + snprintf(out, cap, "no UVC capture backend on this platform (Linux V4L2 is a TODO)\n"); + } +} diff --git a/src/xrt/auxiliary/util/CMakeLists.txt b/src/xrt/auxiliary/util/CMakeLists.txt index aaaa09608d..d2e748298f 100644 --- a/src/xrt/auxiliary/util/CMakeLists.txt +++ b/src/xrt/auxiliary/util/CMakeLists.txt @@ -54,6 +54,8 @@ add_library( u_stereo_rectify.h u_stereo_vrefine.c u_stereo_vrefine.h + u_stereo_uvc.c + u_stereo_uvc.h u_color_encoding.h u_debug.c u_debug.h diff --git a/src/xrt/auxiliary/util/u_stereo_uvc.c b/src/xrt/auxiliary/util/u_stereo_uvc.c new file mode 100644 index 0000000000..98980d69a3 --- /dev/null +++ b/src/xrt/auxiliary/util/u_stereo_uvc.c @@ -0,0 +1,1695 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief Vendor-neutral UVC side-by-side stereo camera source — see u_stereo_uvc.h. + * @ingroup aux_util + */ + +#include "util/u_stereo_uvc.h" +#include "util/u_stereo_rectify.h" +#include "util/u_logging.h" + +#include "os/os_time.h" +#include "xrt/xrt_config_os.h" + +#include + +#include +#include +#include +#include +#include + +#ifdef XRT_OS_WINDOWS +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#include +#include // SHGetFolderPathA +#endif + +#define PI_D 3.14159265358979323846 + +static void +seterr(char *err, size_t cap, const char *msg) +{ + if (err != NULL && cap > 0) { + snprintf(err, cap, "%s", msg); + } +} + +static int +lower_ascii(int c) +{ + return (c >= 'A' && c <= 'Z') ? c + ('a' - 'A') : c; +} + +//! Case-insensitive (ASCII) substring search; UTF-8 bytes above 0x7f compare exactly. +static const char * +strcasestr_ascii(const char *hay, const char *needle) +{ + size_t n = strlen(needle); + if (n == 0) { + return hay; + } + for (const char *p = hay; *p != '\0'; p++) { + size_t i = 0; + while (i < n && p[i] != '\0' && + lower_ascii((unsigned char)p[i]) == lower_ascii((unsigned char)needle[i])) { + i++; + } + if (i == n) { + return p; + } + } + return NULL; +} + +static bool +streq_ascii_nocase(const char *a, const char *b) +{ + while (*a != '\0' && *b != '\0') { + if (lower_ascii((unsigned char)*a) != lower_ascii((unsigned char)*b)) { + return false; + } + a++; + b++; + } + return *a == *b; +} + +static int +hexval(int c) +{ + if (c >= '0' && c <= '9') { + return c - '0'; + } + c = lower_ascii(c); + if (c >= 'a' && c <= 'f') { + return c - 'a' + 10; + } + return -1; +} + +//! Exactly 4 hex digits at @p s. +static bool +hex4(const char *s, uint16_t *out) +{ + uint32_t v = 0; + for (int i = 0; i < 4; i++) { + int h = hexval((unsigned char)s[i]); + if (h < 0) { + return false; + } + v = (v << 4) | (uint32_t)h; + } + *out = (uint16_t)v; + return true; +} + + +/* + * + * Config. + * + */ + +void +u_stereo_uvc_entry_defaults(struct u_stereo_uvc_entry *e) +{ + memset(e, 0, sizeof(*e)); + e->layout = U_STEREO_UVC_LAYOUT_SBS_FULL; +} + +//! "WxH" or "WxH@fps". +static bool +parse_size(const char *s, uint32_t *w, uint32_t *h, float *fps) +{ + unsigned a = 0, b = 0; + float f = 0.0f; + int n = sscanf(s, "%ux%u@%f", &a, &b, &f); + if (n < 2 || a == 0 || b == 0 || a > 16384 || b > 16384) { + return false; + } + if (fps != NULL) { + *fps = n == 3 ? f : 0.0f; + if (n == 3 && !(f > 0.0f && f <= 1000.0f)) { + return false; + } + } else if (n == 3) { + return false; + } + *w = a; + *h = b; + return true; +} + +static bool +parse_entry(const cJSON *j, uint32_t idx, struct u_stereo_uvc_entry *e, char *err, size_t cap) +{ + char msg[256]; + u_stereo_uvc_entry_defaults(e); + if (!cJSON_IsObject(j)) { + snprintf(msg, sizeof(msg), "uvc[%u] is not an object", idx); + seterr(err, cap, msg); + return false; + } + const cJSON *v = cJSON_GetObjectItemCaseSensitive(j, "match"); + if (v != NULL) { + if (!cJSON_IsString(v) || v->valuestring[0] == '\0') { + snprintf(msg, sizeof(msg), "uvc[%u].match must be a non-empty string", idx); + seterr(err, cap, msg); + return false; + } + snprintf(e->name_contains, sizeof(e->name_contains), "%s", v->valuestring); + } + v = cJSON_GetObjectItemCaseSensitive(j, "vid_pid"); + if (v != NULL) { + if (!cJSON_IsString(v) || !u_stereo_uvc_parse_vid_pid(v->valuestring, &e->vid, &e->pid)) { + snprintf(msg, sizeof(msg), "uvc[%u].vid_pid must look like \"1234:abcd\"", idx); + seterr(err, cap, msg); + return false; + } + e->match_vid_pid = true; + } + if (e->name_contains[0] == '\0' && !e->match_vid_pid) { + snprintf(msg, sizeof(msg), "uvc[%u] needs \"match\" (name substring) and/or \"vid_pid\"", idx); + seterr(err, cap, msg); + return false; + } + v = cJSON_GetObjectItemCaseSensitive(j, "name"); + if (cJSON_IsString(v)) { + snprintf(e->display_name, sizeof(e->display_name), "%s", v->valuestring); + } + v = cJSON_GetObjectItemCaseSensitive(j, "layout"); + if (v != NULL) { + const char *s = cJSON_IsString(v) ? v->valuestring : ""; + if (strcmp(s, "sbs-full") == 0 || strcmp(s, "full") == 0 || strcmp(s, "sbs") == 0) { + e->layout = U_STEREO_UVC_LAYOUT_SBS_FULL; + } else if (strcmp(s, "sbs-half") == 0 || strcmp(s, "half") == 0) { + e->layout = U_STEREO_UVC_LAYOUT_SBS_HALF; + } else { + snprintf(msg, sizeof(msg), "uvc[%u].layout must be \"sbs-full\" or \"sbs-half\"", idx); + seterr(err, cap, msg); + return false; + } + } + v = cJSON_GetObjectItemCaseSensitive(j, "eyes"); + if (v != NULL) { + const char *s = cJSON_IsString(v) ? v->valuestring : ""; + if (strcmp(s, "lr") == 0) { + e->swap_eyes = false; + } else if (strcmp(s, "rl") == 0) { + e->swap_eyes = true; + } else { + snprintf(msg, sizeof(msg), "uvc[%u].eyes must be \"lr\" or \"rl\"", idx); + seterr(err, cap, msg); + return false; + } + } + v = cJSON_GetObjectItemCaseSensitive(j, "mode"); + if (v != NULL) { + if (!cJSON_IsString(v) || !parse_size(v->valuestring, &e->mode_width, &e->mode_height, &e->mode_fps) || + (e->mode_width % 4) != 0 || (e->mode_height % 2) != 0) { + snprintf(msg, sizeof(msg), + "uvc[%u].mode must look like \"3840x2160@60\" (width a multiple of 4, even height)", + idx); + seterr(err, cap, msg); + return false; + } + } + v = cJSON_GetObjectItemCaseSensitive(j, "eye_size"); + if (v != NULL) { + if (!cJSON_IsString(v) || !parse_size(v->valuestring, &e->eye_width, &e->eye_height, NULL) || + e->eye_width < 16 || e->eye_height < 16) { + snprintf(msg, sizeof(msg), "uvc[%u].eye_size must look like \"1280x720\"", idx); + seterr(err, cap, msg); + return false; + } + e->eye_width &= ~1u; // NV12: even extents + e->eye_height &= ~1u; + } + v = cJSON_GetObjectItemCaseSensitive(j, "baseline_mm"); + if (v != NULL) { + if (!cJSON_IsNumber(v) || !(v->valuedouble >= 1.0 && v->valuedouble <= 1000.0)) { + snprintf(msg, sizeof(msg), "uvc[%u].baseline_mm must be a number in [1, 1000]", idx); + seterr(err, cap, msg); + return false; + } + e->baseline_mm = (float)v->valuedouble; + } + v = cJSON_GetObjectItemCaseSensitive(j, "hfov_deg"); + if (v != NULL) { + if (!cJSON_IsNumber(v) || !(v->valuedouble >= 10.0 && v->valuedouble <= 170.0)) { + snprintf(msg, sizeof(msg), "uvc[%u].hfov_deg must be a number in [10, 170]", idx); + seterr(err, cap, msg); + return false; + } + e->hfov_deg = (float)v->valuedouble; + } + v = cJSON_GetObjectItemCaseSensitive(j, "calibration"); + if (v != NULL) { + if (!cJSON_IsString(v)) { + snprintf(msg, sizeof(msg), "uvc[%u].calibration must be a file path", idx); + seterr(err, cap, msg); + return false; + } + snprintf(e->calibration, sizeof(e->calibration), "%s", v->valuestring); + } + return true; +} + +bool +u_stereo_uvc_config_parse(const char *json, struct u_stereo_uvc_config *out, char *err, size_t err_cap) +{ + memset(out, 0, sizeof(*out)); + out->fake_params.disparity = 0.03; + out->fake_params.dy = 0.0; + out->fake_params.pixel = U_STEREO_UVC_PIXEL_NV12; + out->fake_params.fps = 30.0f; + if (json == NULL) { + seterr(err, err_cap, "no document"); + return false; + } + cJSON *root = cJSON_Parse(json); + if (root == NULL || !cJSON_IsObject(root)) { + cJSON_Delete(root); + seterr(err, err_cap, "not a JSON object"); + return false; + } + bool ok = true; + const cJSON *list = cJSON_GetObjectItemCaseSensitive(root, "uvc"); + if (list != NULL && !cJSON_IsArray(list)) { + seterr(err, err_cap, "\"uvc\" must be an array"); + ok = false; + } + if (ok && list != NULL) { + int n = cJSON_GetArraySize(list); + if (n > U_STEREO_UVC_MAX_ENTRIES) { + seterr(err, err_cap, "too many \"uvc\" entries (at most 4)"); + ok = false; + } + for (int i = 0; ok && i < n; i++) { + ok = parse_entry(cJSON_GetArrayItem(list, i), (uint32_t)i, &out->entries[out->count], err, + err_cap); + if (ok) { + out->count++; + } + } + } + const cJSON *fake = cJSON_GetObjectItemCaseSensitive(root, "fake"); + if (ok && fake != NULL) { + if (cJSON_IsBool(fake)) { + out->fake = cJSON_IsTrue(fake); + } else if (cJSON_IsObject(fake)) { + out->fake = true; + const cJSON *v = cJSON_GetObjectItemCaseSensitive(fake, "disparity"); + if (cJSON_IsNumber(v) && fabs(v->valuedouble) < 0.5) { + out->fake_params.disparity = v->valuedouble; + } + v = cJSON_GetObjectItemCaseSensitive(fake, "dy"); + if (cJSON_IsNumber(v) && fabs(v->valuedouble) < 0.1) { + out->fake_params.dy = v->valuedouble; + } + v = cJSON_GetObjectItemCaseSensitive(fake, "pixel"); + if (cJSON_IsString(v) && strcmp(v->valuestring, "yuy2") == 0) { + out->fake_params.pixel = U_STEREO_UVC_PIXEL_YUY2; + } + v = cJSON_GetObjectItemCaseSensitive(fake, "fps"); + if (cJSON_IsNumber(v) && v->valuedouble >= 1.0 && v->valuedouble <= 240.0) { + out->fake_params.fps = (float)v->valuedouble; + } + } else { + seterr(err, err_cap, "\"fake\" must be true/false or an object"); + ok = false; + } + } + cJSON_Delete(root); + if (!ok) { + out->count = 0; + out->fake = false; + } + return ok; +} + +bool +u_stereo_uvc_config_path(char *out, size_t cap) +{ + const char *env = getenv("DXR_STEREO_CAMERA_UVC_CONFIG"); + if (env != NULL && env[0] != '\0') { + snprintf(out, cap, "%s", env); + return true; + } +#ifdef XRT_OS_WINDOWS + char appdata[MAX_PATH]; + if (FAILED(SHGetFolderPathA(NULL, CSIDL_LOCAL_APPDATA, NULL, 0, appdata))) { + return false; + } + snprintf(out, cap, "%s\\DisplayXR\\" U_STEREO_UVC_CONFIG_FILENAME, appdata); +#else + const char *config_home = getenv("XDG_CONFIG_HOME"); + if (config_home != NULL && config_home[0] != '\0') { + snprintf(out, cap, "%s/displayxr/" U_STEREO_UVC_CONFIG_FILENAME, config_home); + } else { + const char *home = getenv("HOME"); + if (home == NULL || home[0] == '\0') { + return false; + } + snprintf(out, cap, "%s/.config/displayxr/" U_STEREO_UVC_CONFIG_FILENAME, home); + } +#endif + return true; +} + +//! Whole small text file, NUL-terminated; NULL if absent / too big. +static char * +read_text_file(const char *path, size_t max) +{ + FILE *f = fopen(path, "rb"); + if (f == NULL) { + return NULL; + } + fseek(f, 0, SEEK_END); + long len = ftell(f); + fseek(f, 0, SEEK_SET); + if (len < 0 || (size_t)len > max) { + fclose(f); + return NULL; + } + char *buf = (char *)malloc((size_t)len + 1); + if (buf == NULL) { + fclose(f); + return NULL; + } + size_t got = fread(buf, 1, (size_t)len, f); + fclose(f); + buf[got] = '\0'; + return buf; +} + +enum u_stereo_uvc_config_load_result +u_stereo_uvc_config_load(struct u_stereo_uvc_config *out, char *path, size_t path_cap, char *err, size_t err_cap) +{ + memset(out, 0, sizeof(*out)); + if (path != NULL && path_cap > 0) { + path[0] = '\0'; + } + const char *off = getenv("DXR_STEREO_CAMERA_UVC"); + if (off != NULL && strcmp(off, "0") == 0) { + seterr(err, err_cap, "DXR_STEREO_CAMERA_UVC=0"); + return U_STEREO_UVC_CONFIG_NONE; + } + char p[1024]; + if (!u_stereo_uvc_config_path(p, sizeof(p))) { + seterr(err, err_cap, "no per-user config directory"); + return U_STEREO_UVC_CONFIG_NONE; + } + if (path != NULL && path_cap > 0) { + snprintf(path, path_cap, "%s", p); + } + char *text = read_text_file(p, 256 * 1024); + if (text == NULL) { + seterr(err, err_cap, "no config file"); + return U_STEREO_UVC_CONFIG_NONE; + } + bool ok = u_stereo_uvc_config_parse(text, out, err, err_cap); + free(text); + return ok ? U_STEREO_UVC_CONFIG_OK : U_STEREO_UVC_CONFIG_INVALID; +} + + +/* + * + * Devices + matching. + * + */ + +bool +u_stereo_uvc_parse_vid_pid(const char *s, uint16_t *out_vid, uint16_t *out_pid) +{ + if (s == NULL) { + return false; + } + const char *v = strcasestr_ascii(s, "vid_"); + if (v != NULL) { + const char *p = strcasestr_ascii(v, "pid_"); + return p != NULL && hex4(v + 4, out_vid) && hex4(p + 4, out_pid); + } + // "XXXX:YYYY" (optionally 0x-prefixed halves). + const char *a = s; + if (a[0] == '0' && (a[1] == 'x' || a[1] == 'X')) { + a += 2; + } + if (strlen(a) < 9 || !hex4(a, out_vid) || a[4] != ':') { + return false; + } + const char *b = a + 5; + if (b[0] == '0' && (b[1] == 'x' || b[1] == 'X')) { + b += 2; + } + return strlen(b) == 4 && hex4(b, out_pid); +} + +bool +u_stereo_uvc_entry_matches(const struct u_stereo_uvc_entry *e, const struct u_stereo_uvc_device *d) +{ + if (e->name_contains[0] == '\0' && !e->match_vid_pid) { + return false; // never "match everything" + } + if (e->name_contains[0] != '\0' && strcasestr_ascii(d->name, e->name_contains) == NULL) { + return false; + } + if (e->match_vid_pid && (!d->has_vid_pid || d->vid != e->vid || d->pid != e->pid)) { + return false; + } + return true; +} + +void +u_stereo_uvc_assign(const struct u_stereo_uvc_config *cfg, + const struct u_stereo_uvc_device *devs, + uint32_t device_count, + const bool *unavailable, + int32_t out_device_for_entry[U_STEREO_UVC_MAX_ENTRIES]) +{ + bool taken[U_STEREO_UVC_MAX_DEVICES] = {0}; + uint32_t n = device_count > U_STEREO_UVC_MAX_DEVICES ? U_STEREO_UVC_MAX_DEVICES : device_count; + for (uint32_t i = 0; i < n; i++) { + taken[i] = unavailable != NULL && unavailable[i]; + } + for (uint32_t e = 0; e < U_STEREO_UVC_MAX_ENTRIES; e++) { + out_device_for_entry[e] = U_STEREO_UVC_ASSIGN_NONE; + if (e >= cfg->count) { + continue; + } + const struct u_stereo_uvc_entry *ent = &cfg->entries[e]; + int32_t hit = U_STEREO_UVC_ASSIGN_NONE; + uint32_t matches = 0; + for (uint32_t i = 0; i < n; i++) { + if (!taken[i] && u_stereo_uvc_entry_matches(ent, &devs[i])) { + if (matches++ == 0) { + hit = (int32_t)i; + } + } + } + if (matches > 1 && !ent->match_vid_pid) { + out_device_for_entry[e] = U_STEREO_UVC_ASSIGN_AMBIGUOUS; + continue; + } + if (hit >= 0) { + taken[hit] = true; + } + out_device_for_entry[e] = hit; + } +} + +bool +u_stereo_uvc_device_claimed_by_hint(const struct u_stereo_uvc_device *d, const char *hint) +{ + if (hint == NULL || hint[0] == '\0') { + return false; + } + if (d->id[0] != '\0' && streq_ascii_nocase(d->id, hint)) { + return true; + } + uint16_t vid = 0, pid = 0; + return d->has_vid_pid && u_stereo_uvc_parse_vid_pid(hint, &vid, &pid) && vid == d->vid && pid == d->pid; +} + + +/* + * + * Geometry. + * + */ + +bool +u_stereo_uvc_eye_size(uint32_t layout, + uint32_t frame_w, + uint32_t frame_h, + uint32_t want_w, + uint32_t want_h, + uint32_t *out_w, + uint32_t *out_h) +{ + if (frame_w < 64 || frame_h < 32 || (frame_w % 4) != 0 || (frame_h % 2) != 0) { + return false; + } + if (want_w > 0 && want_h > 0) { + *out_w = want_w & ~1u; + *out_h = want_h & ~1u; + return *out_w >= 16 && *out_h >= 16; + } + double w = frame_w / 2.0; + double h = layout == U_STEREO_UVC_LAYOUT_SBS_HALF ? frame_h / 2.0 : (double)frame_h; + if (w > U_STEREO_UVC_DEFAULT_MAX_EYE_WIDTH) { + h = h * U_STEREO_UVC_DEFAULT_MAX_EYE_WIDTH / w; + w = U_STEREO_UVC_DEFAULT_MAX_EYE_WIDTH; + } + *out_w = (uint32_t)(w + 0.5) & ~1u; + *out_h = (uint32_t)(h + 0.5) & ~1u; + return *out_w >= 16 && *out_h >= 16; +} + +void +u_stereo_uvc_nominal_calibration(uint32_t eye_w, + uint32_t eye_h, + double hfov_deg, + double baseline_mm, + struct xrt_plugin_stereo_camera_calibration *out) +{ + uint32_t sz = out->struct_size; + struct xrt_plugin_stereo_camera_calibration c; + memset(&c, 0, sizeof(c)); + c.struct_size = (uint32_t)sizeof(c); + c.image_width = eye_w; + c.image_height = eye_h; + double f = (eye_w / 2.0) / tan(hfov_deg * 0.5 * PI_D / 180.0); + for (int e = 0; e < 2; e++) { + c.k[e][0] = f; + c.k[e][1] = f; + c.k[e][2] = (eye_w - 1) / 2.0; + c.k[e][3] = (eye_h - 1) / 2.0; + } + c.distortion_model = XRT_PLUGIN_STEREO_CAMERA_DISTORTION_NONE; + c.rotation_right_from_left[0][0] = 1.0; + c.rotation_right_from_left[1][1] = 1.0; + c.rotation_right_from_left[2][2] = 1.0; + c.translation_right_from_left_mm[0] = -baseline_mm; + size_t n = (sz == 0 || sz > sizeof(c)) ? sizeof(c) : sz; + memcpy(out, &c, n); + out->struct_size = sz == 0 ? (uint32_t)sizeof(c) : sz; +} + + +/* + * + * Calibration file (OpenCV FileStorage YAML or JSON). + * + */ + +static bool +is_ident_char(int c) +{ + return isalnum(c) || c == '_'; +} + +//! Position just after "key" + optional quote + optional spaces + ':' — or NULL. +static const char * +find_key(const char *text, const char *key) +{ + size_t n = strlen(key); + for (const char *p = text; (p = strstr(p, key)) != NULL; p += n) { + if (p > text && is_ident_char((unsigned char)p[-1])) { + continue; + } + const char *q = p + n; + if (is_ident_char((unsigned char)*q)) { + continue; + } + if (*q == '"' || *q == '\'') { + q++; + } + while (*q == ' ' || *q == '\t') { + q++; + } + if (*q == ':') { + return q + 1; + } + } + return NULL; +} + +/*! + * Numbers of the first [...] after @p p (nested brackets flattened). Stops at + * the bracket that closes the first one. Returns the count, -1 on no array. + */ +static int +read_array(const char *p, double *out, int cap) +{ + // The array must come before the next line that starts a new key: scan to + // the first '['; a YAML matrix header (!!opencv-matrix, rows, cols, dt) is skipped. + const char *b = strchr(p, '['); + if (b == NULL) { + return -1; + } + int depth = 0, n = 0; + const char *q = b; + while (*q != '\0') { + if (*q == '[') { + depth++; + q++; + } else if (*q == ']') { + depth--; + q++; + if (depth == 0) { + return n; + } + } else if (*q == '-' || *q == '+' || *q == '.' || isdigit((unsigned char)*q)) { + char *end = NULL; + double v = strtod(q, &end); + if (end == q) { + q++; + continue; + } + if (n < cap) { + out[n] = v; + } + n++; + q = end; + } else { + q++; + } + } + return -1; +} + +static bool +read_scalar(const char *text, const char *key, double *out) +{ + const char *p = find_key(text, key); + if (p == NULL) { + return false; + } + char *end = NULL; + double v = strtod(p, &end); + if (end == p) { + return false; + } + *out = v; + return true; +} + +static bool +read_matrix(const char *text, const char *k1, const char *k2, double *out, int cap, int *out_n) +{ + const char *p = find_key(text, k1); + if (p == NULL && k2 != NULL) { + p = find_key(text, k2); + } + if (p == NULL) { + return false; + } + int n = read_array(p, out, cap); + if (n < 0 || n > cap) { + return false; + } + *out_n = n; + return true; +} + +bool +u_stereo_uvc_calibration_parse(const char *text, + uint32_t default_w, + uint32_t default_h, + struct xrt_plugin_stereo_camera_calibration *out, + char *err, + size_t err_cap) +{ + if (text == NULL) { + seterr(err, err_cap, "no text"); + return false; + } + uint32_t sz = out->struct_size; + struct xrt_plugin_stereo_camera_calibration c; + memset(&c, 0, sizeof(c)); + c.struct_size = (uint32_t)sizeof(c); + + double k[2][9], d[2][14], r[9], t[3]; + int nk[2], nd[2], nr = 0, nt = 0; + if (!read_matrix(text, "K1", "M1", k[0], 9, &nk[0]) || nk[0] != 9 || + !read_matrix(text, "K2", "M2", k[1], 9, &nk[1]) || nk[1] != 9) { + seterr(err, err_cap, "K1/M1 and K2/M2 must be 3x3 matrices"); + return false; + } + if (!read_matrix(text, "D1", NULL, d[0], 14, &nd[0]) || !read_matrix(text, "D2", NULL, d[1], 14, &nd[1]) || + nd[0] != nd[1]) { + seterr(err, err_cap, "D1 and D2 must be distortion vectors of the same length"); + return false; + } + if (!read_matrix(text, "R", NULL, r, 9, &nr) || (nr != 9 && nr != 3)) { + seterr(err, err_cap, "R must be a 3x3 matrix or a rotation vector"); + return false; + } + if (!read_matrix(text, "T", NULL, t, 3, &nt) || nt != 3) { + seterr(err, err_cap, "T must be a 3-vector"); + return false; + } + bool fisheye = false; + const char *dm = find_key(text, "distortion_model"); + if (dm != NULL) { + dm += strspn(dm, " \t\"'"); + char tok[16] = {0}; + for (size_t i = 0; i + 1 < sizeof(tok) && isalnum((unsigned char)dm[i]); i++) { + tok[i] = (char)lower_ascii((unsigned char)dm[i]); + } + fisheye = strcmp(tok, "fisheye") == 0 || strcmp(tok, "kb4") == 0 || strcmp(tok, "equidistant") == 0; + } + if (fisheye && nd[0] == 4) { + c.distortion_model = XRT_PLUGIN_STEREO_CAMERA_DISTORTION_KB4; + } else if (nd[0] == 4 || nd[0] == 5) { + c.distortion_model = XRT_PLUGIN_STEREO_CAMERA_DISTORTION_RADTAN5; + } else if (nd[0] == 8) { + c.distortion_model = XRT_PLUGIN_STEREO_CAMERA_DISTORTION_RADTAN8; + } else if (nd[0] == 0) { + c.distortion_model = XRT_PLUGIN_STEREO_CAMERA_DISTORTION_NONE; + } else { + seterr(err, err_cap, "distortion vectors must have 4, 5 or 8 coefficients"); + return false; + } + for (int e = 0; e < 2; e++) { + // OpenCV K = [fx 0 cx; 0 fy cy; 0 0 1]. + if (!(k[e][0] > 0.0) || !(k[e][4] > 0.0) || k[e][8] != 1.0) { + seterr(err, err_cap, "K must be [fx 0 cx; 0 fy cy; 0 0 1] with positive focals"); + return false; + } + c.k[e][0] = k[e][0]; + c.k[e][1] = k[e][4]; + c.k[e][2] = k[e][2]; + c.k[e][3] = k[e][5]; + for (int i = 0; i < nd[e] && i < 8; i++) { + c.distortion[e][i] = d[e][i]; + } + } + if (nr == 3) { + u_stereo_rectify_rodrigues(r, c.rotation_right_from_left); + } else { + for (int i = 0; i < 9; i++) { + c.rotation_right_from_left[i / 3][i % 3] = r[i]; + } + } + double tl = sqrt(t[0] * t[0] + t[1] * t[1] + t[2] * t[2]); + if (!(tl > 0.0)) { + seterr(err, err_cap, "T is zero"); + return false; + } + double scale = tl < 1.0 ? 1000.0 : 1.0; // metres -> mm + for (int i = 0; i < 3; i++) { + c.translation_right_from_left_mm[i] = t[i] * scale; + } + double w = 0.0, h = 0.0, size[2]; + int nsize = 0; + if (read_scalar(text, "image_width", &w) && read_scalar(text, "image_height", &h)) { + c.image_width = (uint32_t)w; + c.image_height = (uint32_t)h; + } else if (read_matrix(text, "image_size", "imageSize", size, 2, &nsize) && nsize == 2) { + c.image_width = (uint32_t)size[0]; + c.image_height = (uint32_t)size[1]; + } else { + c.image_width = default_w; + c.image_height = default_h; + } + if (c.image_width == 0 || c.image_height == 0) { + seterr(err, err_cap, "image size is zero"); + return false; + } + size_t n = (sz == 0 || sz > sizeof(c)) ? sizeof(c) : sz; + memcpy(out, &c, n); + out->struct_size = sz == 0 ? (uint32_t)sizeof(c) : sz; + return true; +} + + +/* + * + * Split + resample. + * + */ + +//! One 8-bit channel of a plane region: samples at base + y * pitch + x * step. +struct chan +{ + const uint8_t *base; + uint32_t pitch; + uint32_t step; + uint32_t w, h; +}; + +struct dchan +{ + uint8_t *base; + uint32_t pitch; + uint32_t step; + uint32_t w, h; +}; + +/*! + * One row of the box-reduced intermediate grid: row @p m averages source rows + * [m * iy, m * iy + iy) and columns in groups of @p ix. Rows are walked in + * memory order and summed into @p acc (16-bit: ix * iy <= 256), then scaled by + * a fixed-point reciprocal — no per-sample division. A 1 x 1 reduction of a + * packed (step 1) channel needs no work at all: the source row itself is + * returned. + */ +static const uint8_t * +box_row(const struct chan *s, uint32_t ix, uint32_t iy, uint32_t mw, uint32_t m, uint16_t *acc, uint8_t *row) +{ + const size_t step = s->step; + if (ix == 1 && iy == 1) { + const uint8_t *p = s->base + (size_t)m * s->pitch; + if (step == 1) { + return p; + } + for (uint32_t x = 0; x < mw; x++) { + row[x] = p[x * step]; + } + return row; + } + memset(acc, 0, sizeof(uint16_t) * mw); + for (uint32_t yy = 0; yy < iy; yy++) { + const uint8_t *p = s->base + (size_t)(m * iy + yy) * s->pitch; + if (ix == 1 && step == 1) { + for (uint32_t x = 0; x < mw; x++) { + acc[x] = (uint16_t)(acc[x] + p[x]); + } + } else if (ix == 1) { + for (uint32_t x = 0; x < mw; x++) { + acc[x] = (uint16_t)(acc[x] + p[x * step]); + } + } else { + for (uint32_t x = 0; x < mw; x++) { + const uint8_t *q = p + (size_t)x * ix * step; + uint32_t a = 0; + for (uint32_t xx = 0; xx < ix; xx++) { + a += q[xx * step]; + } + acc[x] = (uint16_t)(acc[x] + a); + } + } + } + const uint32_t div = ix * iy; + const uint32_t recip = (65536u + div / 2) / div; // sum * recip >> 16 ~= sum / div + for (uint32_t x = 0; x < mw; x++) { + uint32_t v = ((uint32_t)acc[x] * recip + 32768u) >> 16; + row[x] = (uint8_t)(v > 255 ? 255 : v); + } + return row; +} + +//! Pixel-centre bilinear tap from a @p dst-sized grid into a @p src-sized one. +static void +tap(uint32_t i, uint32_t src, uint32_t dst, uint32_t *i0, uint32_t *i1, uint32_t *w) +{ + double f = ((double)i + 0.5) * (double)src / (double)dst - 0.5; + if (f < 0.0) { + f = 0.0; + } + if (f > (double)(src - 1)) { + f = (double)(src - 1); + } + uint32_t a = (uint32_t)f; + uint32_t wt = (uint32_t)((f - (double)a) * 256.0 + 0.5); + if (wt >= 256) { + a++; + wt = 0; + } + if (a > src - 1) { + a = src - 1; + } + *i0 = a; + *i1 = a + 1 < src ? a + 1 : src - 1; + *w = wt; +} + +static bool +resample(const struct chan *s, const struct dchan *d) +{ + if (s->w == 0 || s->h == 0 || d->w == 0 || d->h == 0) { + return false; + } + if (s->w == d->w && s->h == d->h) { + for (uint32_t y = 0; y < d->h; y++) { + const uint8_t *sp = s->base + (size_t)y * s->pitch; + uint8_t *dp = d->base + (size_t)y * d->pitch; + if (s->step == 1 && d->step == 1) { + memcpy(dp, sp, d->w); + } else { + for (uint32_t x = 0; x < d->w; x++) { + dp[(size_t)x * d->step] = sp[(size_t)x * s->step]; + } + } + } + return true; + } + // Integer box pre-reduction (anti-aliasing), then bilinear for the rest. + uint32_t ix = s->w / d->w, iy = s->h / d->h; + ix = ix < 1 ? 1 : ix > 16 ? 16 : ix; + iy = iy < 1 ? 1 : iy > 16 ? 16 : iy; + const uint32_t mw = s->w / ix, mh = s->h / iy; + uint32_t *x0 = (uint32_t *)malloc(sizeof(uint32_t) * d->w * 3); + uint8_t *rows = (uint8_t *)malloc((size_t)mw * 2); + uint16_t *acc = (uint16_t *)malloc(sizeof(uint16_t) * mw); + if (x0 == NULL || rows == NULL || acc == NULL) { + free(x0); + free(rows); + free(acc); + return false; + } + uint32_t *x1 = x0 + d->w, *wx = x0 + 2 * d->w; + bool h_identity = mw == d->w; + for (uint32_t x = 0; x < d->w; x++) { + tap(x, mw, d->w, &x0[x], &x1[x], &wx[x]); + } + // Two cached intermediate rows, slot = row parity: y1 is y0 or y0 + 1, so + // the two rows one output row needs never share a slot. + uint8_t *rowbuf[2] = {rows, rows + mw}; + const uint8_t *row[2] = {NULL, NULL}; + int64_t have[2] = {-1, -1}; + const size_t dstep = d->step; + for (uint32_t y = 0; y < d->h; y++) { + uint32_t y0, y1, wy; + tap(y, mh, d->h, &y0, &y1, &wy); + const uint32_t need[2] = {y0, y1}; + const uint8_t *r[2]; + const int rows_needed = wy == 0 ? 1 : 2; // an exact row needs no second one + for (int k = 0; k < rows_needed; k++) { + const uint32_t slot = need[k] & 1u; + if (have[slot] != (int64_t)need[k]) { + row[slot] = box_row(s, ix, iy, mw, need[k], acc, rowbuf[slot]); + have[slot] = need[k]; + } + r[k] = row[slot]; + } + uint8_t *dp = d->base + (size_t)y * d->pitch; + if (rows_needed == 1) { + const uint8_t *r0 = r[0]; + if (h_identity) { + for (uint32_t x = 0; x < d->w; x++) { + dp[x * dstep] = r0[x]; + } + } else { + for (uint32_t x = 0; x < d->w; x++) { + uint32_t w = wx[x]; + dp[x * dstep] = (uint8_t)((r0[x0[x]] * (256 - w) + r0[x1[x]] * w + 128) >> 8); + } + } + continue; + } + const uint8_t *r0 = r[0], *r1 = r[1]; + for (uint32_t x = 0; x < d->w; x++) { + uint32_t a = h_identity ? x : x0[x], b = h_identity ? x : x1[x], w = h_identity ? 0 : wx[x]; + uint32_t top = r0[a] * (256 - w) + r0[b] * w; + uint32_t bot = r1[a] * (256 - w) + r1[b] * w; + dp[x * dstep] = (uint8_t)((top * (256 - wy) + bot * wy + 32768) >> 16); + } + } + free(x0); + free(rows); + free(acc); + return true; +} + +/*! + * Fast path: the frame already has the output size (a GPU-scaled frame), NV12 + * in and out — each eye is a plain row copy of both planes. + */ +static bool +copy_nv12_halves(const struct u_stereo_uvc_raw_frame *in, + bool swap_eyes, + uint32_t eye_w, + uint32_t eye_h, + uint8_t *dst, + const struct u_stereo_camera_planes *dst_layout) +{ + for (uint32_t e = 0; e < 2; e++) { + const uint32_t h = swap_eyes ? 1u - e : e; + for (uint32_t y = 0; y < eye_h; y++) { + memcpy(dst + dst_layout->offset[0] + (size_t)y * dst_layout->pitch[0] + (size_t)e * eye_w, + in->planes[0] + (size_t)y * in->pitches[0] + (size_t)h * eye_w, eye_w); + } + for (uint32_t y = 0; y < eye_h / 2; y++) { + memcpy(dst + dst_layout->offset[1] + (size_t)y * dst_layout->pitch[1] + (size_t)e * eye_w, + in->planes[1] + (size_t)y * in->pitches[1] + (size_t)h * eye_w, eye_w); + } + } + return true; +} + +bool +u_stereo_uvc_split(const struct u_stereo_uvc_raw_frame *in, + bool swap_eyes, + uint32_t eye_w, + uint32_t eye_h, + uint8_t *dst, + const struct u_stereo_camera_planes *dst_layout) +{ + if (in == NULL || dst == NULL || dst_layout == NULL || in->planes[0] == NULL || (in->width % 4) != 0 || + (in->height % 2) != 0 || in->width < 8 || in->height < 2 || (eye_w % 2) != 0 || (eye_h % 2) != 0 || + eye_w == 0 || eye_h == 0 || dst_layout->plane_count != 2) { + return false; + } + const uint32_t W = in->width, H = in->height, hw = W / 2; + if (in->pixel == U_STEREO_UVC_PIXEL_NV12 && in->planes[1] != NULL && hw == eye_w && H == eye_h) { + return copy_nv12_halves(in, swap_eyes, eye_w, eye_h, dst, dst_layout); + } + for (uint32_t e = 0; e < 2; e++) { + const uint32_t h = swap_eyes ? 1u - e : e; // which half of the source holds output eye e + struct chan sy, su, sv; + if (in->pixel == U_STEREO_UVC_PIXEL_NV12) { + if (in->planes[1] == NULL) { + return false; + } + sy = (struct chan){in->planes[0] + (size_t)h * hw, in->pitches[0], 1, hw, H}; + su = (struct chan){in->planes[1] + (size_t)h * hw, in->pitches[1], 2, hw / 2, H / 2}; + sv = su; + sv.base += 1; + } else if (in->pixel == U_STEREO_UVC_PIXEL_YUY2) { + const uint8_t *p = in->planes[0] + (size_t)h * hw * 2; + sy = (struct chan){p, in->pitches[0], 2, hw, H}; + su = (struct chan){p + 1, in->pitches[0], 4, hw / 2, H}; + sv = (struct chan){p + 3, in->pitches[0], 4, hw / 2, H}; + } else { + return false; + } + uint8_t *yb = dst + dst_layout->offset[0] + (size_t)e * eye_w; + uint8_t *uvb = dst + dst_layout->offset[1] + (size_t)e * eye_w; // eye_w bytes = eye_w/2 UV pairs + struct dchan dy = {yb, dst_layout->pitch[0], 1, eye_w, eye_h}; + struct dchan du = {uvb, dst_layout->pitch[1], 2, eye_w / 2, eye_h / 2}; + struct dchan dv = {uvb + 1, dst_layout->pitch[1], 2, eye_w / 2, eye_h / 2}; + if (!resample(&sy, &dy) || !resample(&su, &du) || !resample(&sv, &dv)) { + return false; + } + } + return true; +} + + +/* + * + * The fake backend. + * + */ + +#define FAKE_ID "dxr-fake-uvc" +#define FAKE_NAME "DisplayXR synthetic SBS camera" +#define FAKE_CELLS_X 96.0 +#define FAKE_CELLS_Y 72.0 + +struct fake_handle +{ + struct u_stereo_uvc_fake_params p; + struct u_stereo_uvc_mode mode; + uint8_t *buf; + uint32_t pitch[2]; + uint64_t offset[2]; + int64_t t0_ns; + int64_t period_ns; + uint64_t seq; +}; + +static uint32_t +fake_hash(int32_t x, int32_t y) +{ + uint32_t h = (uint32_t)x * 374761393u + (uint32_t)y * 668265263u; + h = (h ^ (h >> 13)) * 1274126177u; + return h ^ (h >> 16); +} + +//! Smooth value noise in [40, 215] at lattice coordinates (u, v). +static double +fake_tex(double u, double v) +{ + double fu = floor(u), fv = floor(v); + int32_t iu = (int32_t)fu, iv = (int32_t)fv; + double a = u - fu, b = v - fv; + // Smoothstep keeps the gradient continuous at lattice lines. + a = a * a * (3.0 - 2.0 * a); + b = b * b * (3.0 - 2.0 * b); + double v00 = (double)(fake_hash(iu, iv) & 0xff), v10 = (double)(fake_hash(iu + 1, iv) & 0xff); + double v01 = (double)(fake_hash(iu, iv + 1) & 0xff), v11 = (double)(fake_hash(iu + 1, iv + 1) & 0xff); + double t = v00 + (v10 - v00) * a, bt = v01 + (v11 - v01) * a; + return 40.0 + (t + (bt - t) * b) * (175.0 / 255.0); +} + +/*! + * Luma of raw pixel (@p px, @p py) of half @p half: the scene is defined in + * EYE-NORMALISED coordinates, so every layout (full / half SBS) reads back the + * same scene after the split. The right half (the right lens) sees the + * texture shifted by the disparity and lowered by dy. + */ +static uint8_t +fake_luma(const struct u_stereo_uvc_fake_params *p, uint32_t half, uint32_t hw, uint32_t h, uint32_t px, uint32_t py) +{ + double x = ((double)px + 0.5) / hw; + double y = ((double)py + 0.5) / h; + if (half == 1) { + x += p->disparity; + y -= p->dy; + } + double v = fake_tex(x * FAKE_CELLS_X, y * FAKE_CELLS_Y); + return (uint8_t)(v + 0.5); +} + +static void +fake_render(struct fake_handle *f) +{ + const uint32_t W = f->mode.width, H = f->mode.height, hw = W / 2; + for (uint32_t y = 0; y < H; y++) { + uint8_t *row = f->buf + f->offset[0] + (size_t)y * f->pitch[0]; + for (uint32_t x = 0; x < W; x++) { + uint32_t half = x >= hw ? 1u : 0u; + uint8_t l = fake_luma(&f->p, half, hw, H, x - half * hw, y); + if (f->p.pixel == U_STEREO_UVC_PIXEL_YUY2) { + row[2 * (size_t)x] = l; + row[2 * (size_t)x + 1] = 128; + } else { + row[x] = l; + } + } + } + if (f->p.pixel == U_STEREO_UVC_PIXEL_NV12) { + memset(f->buf + f->offset[1], 128, (size_t)f->pitch[1] * (H / 2)); + } +} + +static uint32_t +fake_enumerate(void *ctx, struct u_stereo_uvc_device *out, uint32_t cap) +{ + (void)ctx; + if (out != NULL && cap >= 1) { + memset(out, 0, sizeof(*out)); + snprintf(out->name, sizeof(out->name), "%s", FAKE_NAME); + snprintf(out->id, sizeof(out->id), "%s", FAKE_ID); + } + return 1; +} + +static uint32_t +fake_list_modes(void *ctx, const char *id, struct u_stereo_uvc_mode *out, uint32_t cap) +{ + const struct u_stereo_uvc_fake_params *p = (const struct u_stereo_uvc_fake_params *)ctx; + if (id == NULL || strcmp(id, FAKE_ID) != 0) { + return 0; + } + static const struct u_stereo_uvc_mode modes[] = { + {1280, 480, 30.0f}, + {2560, 720, 30.0f}, + {1280, 960, 30.0f}, + }; + uint32_t n = (uint32_t)(sizeof(modes) / sizeof(modes[0])); + for (uint32_t i = 0; i < n && i < cap && out != NULL; i++) { + out[i] = modes[i]; + out[i].fps = p != NULL ? p->fps : modes[i].fps; + } + return n < cap ? n : cap; +} + +static bool +fake_open( + void *ctx, const char *id, const struct u_stereo_uvc_mode *mode, uint32_t out_w, uint32_t out_h, void **out_handle) +{ + const struct u_stereo_uvc_fake_params *p = (const struct u_stereo_uvc_fake_params *)ctx; + // The fake ignores the output-size hint: it always delivers the full + // capture, so the source's own split/downscale stays under test. + (void)out_w; + (void)out_h; + if (id == NULL || strcmp(id, FAKE_ID) != 0 || mode == NULL || mode->width < 64 || (mode->width % 4) != 0 || + mode->height < 32 || (mode->height % 2) != 0 || mode->width > 8192 || mode->height > 8192) { + return false; + } + struct fake_handle *f = (struct fake_handle *)calloc(1, sizeof(*f)); + if (f == NULL) { + return false; + } + f->p = *p; + f->mode = *mode; + float fps = mode->fps > 0.0f ? mode->fps : p->fps; + f->period_ns = (int64_t)(1e9 / (fps > 0.0f ? fps : 30.0f)); + if (p->pixel == U_STEREO_UVC_PIXEL_YUY2) { + f->pitch[0] = mode->width * 2; + f->offset[0] = 0; + f->buf = (uint8_t *)malloc((size_t)f->pitch[0] * mode->height); + } else { + f->pitch[0] = mode->width; + f->pitch[1] = mode->width; + f->offset[1] = (uint64_t)mode->width * mode->height; + f->buf = (uint8_t *)malloc((size_t)mode->width * mode->height * 3 / 2); + } + if (f->buf == NULL) { + free(f); + return false; + } + fake_render(f); // static scene: render once + f->t0_ns = os_monotonic_get_ns(); + *out_handle = f; + return true; +} + +static uint32_t +fake_read(void *handle, int64_t timeout_ns, struct u_stereo_uvc_raw_frame *out) +{ + struct fake_handle *f = (struct fake_handle *)handle; + int64_t due = f->t0_ns + (int64_t)(f->seq + 1) * f->period_ns; + int64_t now = os_monotonic_get_ns(); + if (due - now > timeout_ns) { + if (timeout_ns > 0) { + os_nanosleep(timeout_ns); + } + return U_STEREO_UVC_READ_TIMEOUT; + } + if (due > now) { + os_nanosleep(due - now); + } + f->seq++; + memset(out, 0, sizeof(*out)); + out->pixel = f->p.pixel; + out->width = f->mode.width; + out->height = f->mode.height; + out->planes[0] = f->buf + f->offset[0]; + out->pitches[0] = f->pitch[0]; + if (f->p.pixel == U_STEREO_UVC_PIXEL_NV12) { + out->planes[1] = f->buf + f->offset[1]; + out->pitches[1] = f->pitch[1]; + } + out->time_ns = os_monotonic_get_ns(); + out->time_is_exposure = false; + return U_STEREO_UVC_READ_OK; +} + +static void +fake_close(void *handle) +{ + struct fake_handle *f = (struct fake_handle *)handle; + if (f != NULL) { + free(f->buf); + free(f); + } +} + +void +u_stereo_uvc_fake_backend_init(struct u_stereo_uvc_backend *out, const struct u_stereo_uvc_fake_params *params) +{ + memset(out, 0, sizeof(*out)); + out->name = "fake"; + out->ctx = (void *)params; // the caller keeps it alive (u_stereo_uvc keeps its own copy) + out->enumerate = fake_enumerate; + out->list_modes = fake_list_modes; + out->open = fake_open; + out->read = fake_read; + out->close = fake_close; +} + + +/* + * + * The source. + * + */ + +struct uvc_cam +{ + struct u_stereo_uvc_entry entry; + struct u_stereo_uvc_device dev; + struct u_stereo_uvc_mode mode; + uint32_t eye_w, eye_h; + struct xrt_plugin_stereo_camera_calibration calib; + bool calibrated; //!< from a calibration file + bool baseline_known, hfov_known; +}; + +struct u_stereo_uvc +{ + struct u_stereo_uvc_backend backend; //!< .read == NULL: none + struct u_stereo_uvc_fake_params fake; //!< ctx of the fake backend + uint32_t count; + struct uvc_cam cams[U_STEREO_UVC_MAX_ENTRIES]; +}; + +struct u_stereo_uvc_stream +{ + struct u_stereo_uvc *uvc; + uint32_t index; + void *handle; + uint8_t *buf; + struct u_stereo_camera_planes layout; + uint64_t seq; + bool size_warned; +}; + +static bool +pick_mode(const struct u_stereo_uvc *u, + const struct u_stereo_uvc_entry *e, + const char *id, + struct u_stereo_uvc_mode *out) +{ + struct u_stereo_uvc_mode modes[U_STEREO_UVC_MAX_MODES]; + uint32_t n = u->backend.list_modes != NULL + ? u->backend.list_modes(u->backend.ctx, id, modes, U_STEREO_UVC_MAX_MODES) + : 0; + if (e->mode_width > 0) { + out->width = e->mode_width; + out->height = e->mode_height; + out->fps = e->mode_fps; + bool listed = n == 0; + float best_fps = 0.0f; + for (uint32_t i = 0; i < n; i++) { + if (modes[i].width == out->width && modes[i].height == out->height) { + listed = true; + best_fps = modes[i].fps > best_fps ? modes[i].fps : best_fps; + } + } + if (!listed) { + U_LOG_W( + "stereo camera (uvc): configured mode %ux%u is not among the %u modes the device lists — " + "trying it anyway", + out->width, out->height, n); + } + if (out->fps <= 0.0f) { + out->fps = best_fps; + } + return true; + } + // Auto: the largest frame that splits cleanly, then the fastest. + bool found = false; + for (uint32_t i = 0; i < n; i++) { + const struct u_stereo_uvc_mode *m = &modes[i]; + if ((m->width % 4) != 0 || (m->height % 2) != 0 || m->width < 64) { + continue; + } + uint64_t area = (uint64_t)m->width * m->height, best = (uint64_t)out->width * out->height; + if (!found || area > best || (area == best && m->fps > out->fps)) { + *out = *m; + found = true; + } + } + return found; +} + +static void +resolve_calibration(struct uvc_cam *c) +{ + const struct u_stereo_uvc_entry *e = &c->entry; + c->baseline_known = e->baseline_mm > 0.0f; + c->hfov_known = e->hfov_deg > 0.0f; + if (e->calibration[0] != '\0') { + char *text = read_text_file(e->calibration, 1024 * 1024); + char err[160] = {0}; + c->calib.struct_size = (uint32_t)sizeof(c->calib); + if (text == NULL) { + U_LOG_W( + "stereo camera (uvc) \"%s\": calibration file \"%s\" not readable — using the NOMINAL " + "model (uncalibrated)", + c->dev.name, e->calibration); + } else if (!u_stereo_uvc_calibration_parse(text, c->mode.width / 2, c->mode.height, &c->calib, err, + sizeof(err))) { + U_LOG_W( + "stereo camera (uvc) \"%s\": calibration file \"%s\" rejected (%s) — using the NOMINAL " + "model (uncalibrated)", + c->dev.name, e->calibration, err); + } else { + c->calibrated = true; + c->baseline_known = true; + c->hfov_known = true; + U_LOG_W( + "stereo camera (uvc) \"%s\": calibration from \"%s\" (%ux%u per eye, fx %.1f, |T| %.1f mm)", + c->dev.name, e->calibration, c->calib.image_width, c->calib.image_height, c->calib.k[0][0], + sqrt(c->calib.translation_right_from_left_mm[0] * + c->calib.translation_right_from_left_mm[0] + + c->calib.translation_right_from_left_mm[1] * + c->calib.translation_right_from_left_mm[1] + + c->calib.translation_right_from_left_mm[2] * + c->calib.translation_right_from_left_mm[2])); + } + free(text); + if (c->calibrated) { + return; + } + } + c->calib.struct_size = (uint32_t)sizeof(c->calib); + u_stereo_uvc_nominal_calibration( + c->eye_w, c->eye_h, c->hfov_known ? e->hfov_deg : U_STEREO_UVC_DEFAULT_HFOV_DEG, + c->baseline_known ? e->baseline_mm : U_STEREO_UVC_DEFAULT_BASELINE_MM, &c->calib); +} + +struct u_stereo_uvc * +u_stereo_uvc_create(const struct u_stereo_uvc_config *cfg, + const struct u_stereo_uvc_backend *backend, + const char *const *exclude_hints, + uint32_t exclude_count) +{ + struct u_stereo_uvc *u = (struct u_stereo_uvc *)calloc(1, sizeof(*u)); + if (u == NULL || cfg == NULL) { + return u; + } + if (cfg->fake) { + // The synthetic device REPLACES OS capture: no real camera is considered. + u->fake = cfg->fake_params; + u_stereo_uvc_fake_backend_init(&u->backend, &u->fake); + } else if (backend != NULL) { + u->backend = *backend; + } + if (u->backend.enumerate == NULL || u->backend.open == NULL || u->backend.read == NULL || + u->backend.close == NULL) { + if (cfg->count > 0) { + U_LOG_W("stereo camera (uvc): %u configured device(s) but no capture backend on this platform", + cfg->count); + } + memset(&u->backend, 0, sizeof(u->backend)); + return u; + } + struct u_stereo_uvc_device devs[U_STEREO_UVC_MAX_DEVICES]; + memset(devs, 0, sizeof(devs)); + uint32_t nd = u->backend.enumerate(u->backend.ctx, devs, U_STEREO_UVC_MAX_DEVICES); + nd = nd > U_STEREO_UVC_MAX_DEVICES ? U_STEREO_UVC_MAX_DEVICES : nd; + bool used[U_STEREO_UVC_MAX_DEVICES] = {0}; + for (uint32_t i = 0; i < nd; i++) { + devs[i].name[sizeof(devs[i].name) - 1] = '\0'; + devs[i].id[sizeof(devs[i].id) - 1] = '\0'; + if (!devs[i].has_vid_pid) { + devs[i].has_vid_pid = u_stereo_uvc_parse_vid_pid(devs[i].id, &devs[i].vid, &devs[i].pid); + } + for (uint32_t h = 0; h < exclude_count; h++) { + if (u_stereo_uvc_device_claimed_by_hint(&devs[i], exclude_hints[h])) { + used[i] = true; // a plug-in's camera (the tracker's): never ours + U_LOG_I("stereo camera (uvc): \"%s\" is a display plug-in's camera — never claimed", + devs[i].name); + } + } + } + int32_t assigned[U_STEREO_UVC_MAX_ENTRIES]; + u_stereo_uvc_assign(cfg, devs, nd, used, assigned); + for (uint32_t e = 0; e < cfg->count && u->count < U_STEREO_UVC_MAX_ENTRIES; e++) { + const struct u_stereo_uvc_entry *ent = &cfg->entries[e]; + const int32_t hit = assigned[e]; + if (hit == U_STEREO_UVC_ASSIGN_AMBIGUOUS) { + U_LOG_W( + "stereo camera (uvc): config entry %u (match \"%s\") matches SEVERAL capture devices — " + "none claimed; make \"match\" more specific or add \"vid_pid\" " + "(`displayxr-cli camera uvc-devices`)", + e, ent->name_contains); + continue; + } + if (hit < 0) { + U_LOG_W("stereo camera (uvc): config entry %u (match \"%s\"%s) matches no free capture device", + e, ent->name_contains, ent->match_vid_pid ? " + vid_pid" : ""); + continue; + } + struct uvc_cam *c = &u->cams[u->count]; + memset(c, 0, sizeof(*c)); + c->entry = *ent; + c->dev = devs[hit]; + if (!pick_mode(u, ent, c->dev.id, &c->mode)) { + U_LOG_W( + "stereo camera (uvc) \"%s\": the device lists no usable mode — set \"mode\" in the config", + c->dev.name); + continue; + } + if (!u_stereo_uvc_eye_size(ent->layout, c->mode.width, c->mode.height, ent->eye_width, ent->eye_height, + &c->eye_w, &c->eye_h)) { + U_LOG_W("stereo camera (uvc) \"%s\": mode %ux%u cannot be split into two eyes", c->dev.name, + c->mode.width, c->mode.height); + continue; + } + used[hit] = true; + resolve_calibration(c); + char vp[24] = "no VID:PID"; + if (c->dev.has_vid_pid) { + snprintf(vp, sizeof(vp), "VID:PID %04x:%04x", c->dev.vid, c->dev.pid); + } + U_LOG_W("stereo camera (uvc): CLAIMED \"%s\" (%s) — %s %ux%u@%.1f, eyes %s, %ux%u per eye out, %s", + c->dev.name, vp, ent->layout == U_STEREO_UVC_LAYOUT_SBS_HALF ? "half-SBS" : "full-SBS", + c->mode.width, c->mode.height, c->mode.fps, ent->swap_eyes ? "rl" : "lr", c->eye_w, c->eye_h, + c->calibrated ? "CALIBRATED (file)" : "uncalibrated (nominal pinhole + online row alignment)"); + u->count++; + } + return u; +} + +void +u_stereo_uvc_destroy(struct u_stereo_uvc **uvc_ptr) +{ + if (uvc_ptr == NULL || *uvc_ptr == NULL) { + return; + } + free(*uvc_ptr); + *uvc_ptr = NULL; +} + +uint32_t +u_stereo_uvc_enumerate(struct u_stereo_uvc *uvc, uint32_t capacity, struct xrt_plugin_stereo_camera_info *out) +{ + if (uvc == NULL) { + return 0; + } + for (uint32_t i = 0; i < uvc->count && i < capacity && out != NULL; i++) { + const struct uvc_cam *c = &uvc->cams[i]; + uint32_t sz = out[i].struct_size; + struct xrt_plugin_stereo_camera_info info; + memset(&info, 0, sizeof(info)); + info.struct_size = (uint32_t)sizeof(info); + snprintf(info.display_name, sizeof(info.display_name), "%s", + c->entry.display_name[0] != '\0' ? c->entry.display_name : c->dev.name); + snprintf(info.device_identity, sizeof(info.device_identity), "uvc:%.120s", c->dev.id); + snprintf(info.platform_device_hint, sizeof(info.platform_device_hint), "%s", c->dev.id); + info.flags = XRT_PLUGIN_STEREO_CAMERA_USER_FACING; + if (c->calibrated) { + info.flags |= XRT_PLUGIN_STEREO_CAMERA_CALIBRATED; + } + info.eye_width = c->eye_w; + info.eye_height = c->eye_h; + info.max_frame_rate = c->mode.fps; // the negotiated mode's rate; the service measures the real one + info.native_format = XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12; + size_t n = (sz == 0 || sz > sizeof(info)) ? sizeof(info) : sz; + memcpy(&out[i], &info, n); + out[i].struct_size = sz; + } + return uvc->count; +} + +xrt_result_t +u_stereo_uvc_get_calibration(struct u_stereo_uvc *uvc, + uint32_t index, + struct xrt_plugin_stereo_camera_calibration *out, + bool *out_nominal, + bool *out_baseline_known, + bool *out_hfov_known) +{ + if (uvc == NULL || index >= uvc->count || out == NULL) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + const struct uvc_cam *c = &uvc->cams[index]; + uint32_t sz = out->struct_size; + size_t n = (sz == 0 || sz > sizeof(c->calib)) ? sizeof(c->calib) : sz; + memcpy(out, &c->calib, n); + out->struct_size = sz == 0 ? (uint32_t)sizeof(c->calib) : sz; + if (out_nominal != NULL) { + *out_nominal = !c->calibrated; + } + if (out_baseline_known != NULL) { + *out_baseline_known = c->baseline_known; + } + if (out_hfov_known != NULL) { + *out_hfov_known = c->hfov_known; + } + return XRT_SUCCESS; +} + +bool +u_stereo_uvc_get_mode(struct u_stereo_uvc *uvc, uint32_t index, struct u_stereo_uvc_mode *out, uint32_t *out_layout) +{ + if (uvc == NULL || index >= uvc->count) { + return false; + } + if (out != NULL) { + *out = uvc->cams[index].mode; + } + if (out_layout != NULL) { + *out_layout = uvc->cams[index].entry.layout; + } + return true; +} + +xrt_result_t +u_stereo_uvc_open(struct u_stereo_uvc *uvc, uint32_t index, struct u_stereo_uvc_stream **out_stream) +{ + *out_stream = NULL; + if (uvc == NULL || index >= uvc->count || uvc->backend.open == NULL) { + return XRT_ERROR_FEATURE_NOT_SUPPORTED; + } + const struct uvc_cam *c = &uvc->cams[index]; + struct u_stereo_uvc_stream *s = (struct u_stereo_uvc_stream *)calloc(1, sizeof(*s)); + if (s == NULL) { + return XRT_ERROR_ALLOCATION; + } + s->uvc = uvc; + s->index = index; + if (!u_stereo_camera_layout(XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12, 2 * c->eye_w, c->eye_h, &s->layout)) { + free(s); + return XRT_ERROR_ALLOCATION; + } + s->buf = (uint8_t *)malloc((size_t)s->layout.size); + if (s->buf == NULL) { + free(s); + return XRT_ERROR_ALLOCATION; + } + // Hint: the SBS frame we resample to, so a GPU backend can decode + scale there. + if (!uvc->backend.open(uvc->backend.ctx, c->dev.id, &c->mode, 2 * c->eye_w, c->eye_h, &s->handle)) { + U_LOG_W( + "stereo camera (uvc) \"%s\": could not open %ux%u@%.1f (in use by another app, unplugged, or " + "the mode is not offered)", + c->dev.name, c->mode.width, c->mode.height, c->mode.fps); + free(s->buf); + free(s); + return XRT_ERROR_DEVICE_CREATION_FAILED; + } + *out_stream = s; + return XRT_SUCCESS; +} + +uint32_t +u_stereo_uvc_wait_frame(struct u_stereo_uvc_stream *s, int64_t timeout_ns, struct xrt_plugin_stereo_camera_frame *out) +{ + struct u_stereo_uvc *uvc = s->uvc; + const struct uvc_cam *c = &uvc->cams[s->index]; + struct u_stereo_uvc_raw_frame raw; + memset(&raw, 0, sizeof(raw)); + uint32_t r = uvc->backend.read(s->handle, timeout_ns, &raw); + if (r == U_STEREO_UVC_READ_TIMEOUT) { + return XRT_PLUGIN_STEREO_CAMERA_WAIT_TIMEOUT; + } + if (r != U_STEREO_UVC_READ_OK) { + return XRT_PLUGIN_STEREO_CAMERA_WAIT_ERROR; + } + if ((raw.width != c->mode.width || raw.height != c->mode.height) && !s->size_warned) { + s->size_warned = true; + U_LOG_W( + "stereo camera (uvc) \"%s\": device delivers %ux%u, not the requested %ux%u — resampled to the " + "advertised %ux%u per eye", + c->dev.name, raw.width, raw.height, c->mode.width, c->mode.height, c->eye_w, c->eye_h); + } + if (!u_stereo_uvc_split(&raw, c->entry.swap_eyes, c->eye_w, c->eye_h, s->buf, &s->layout)) { + return XRT_PLUGIN_STEREO_CAMERA_WAIT_TIMEOUT; // an unsplittable frame is dropped, not fatal + } + memset(out, 0, sizeof(*out)); + out->sequence = ++s->seq; + out->time_ns = raw.time_ns > 0 ? raw.time_ns : os_monotonic_get_ns(); + out->time_is_exposure = raw.time_is_exposure; + out->width = 2 * c->eye_w; + out->height = c->eye_h; + out->format = XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12; + out->planes[0] = s->buf + s->layout.offset[0]; + out->planes[1] = s->buf + s->layout.offset[1]; + out->pitches[0] = s->layout.pitch[0]; + out->pitches[1] = s->layout.pitch[1]; + return XRT_PLUGIN_STEREO_CAMERA_WAIT_OK; +} + +void +u_stereo_uvc_release_frame(struct u_stereo_uvc_stream *s) +{ + (void)s; // the split buffer is reused by the next wait_frame +} + +void +u_stereo_uvc_close(struct u_stereo_uvc_stream *s) +{ + if (s == NULL) { + return; + } + if (s->handle != NULL) { + s->uvc->backend.close(s->handle); + } + free(s->buf); + free(s); +} diff --git a/src/xrt/auxiliary/util/u_stereo_uvc.h b/src/xrt/auxiliary/util/u_stereo_uvc.h new file mode 100644 index 0000000000..f5c9999b96 --- /dev/null +++ b/src/xrt/auxiliary/util/u_stereo_uvc.h @@ -0,0 +1,460 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera (ADR-043 Amendment 4): a vendor-neutral stereo + * camera source for plain UVC side-by-side stereo webcams. + * + * Every other stereo camera comes from a display plug-in (the tracker camera + * the plug-in reads without taking it from the eye tracker). This one is the + * SERVICE's own: an external stereo webcam that delivers ONE side-by-side + * image per frame over plain UVC, opted in by the user, works with any display + * and needs no vendor plug-in. Nothing here is vendor-specific. + * + * Layers, so everything but the OS capture call is testable on every OS: + * + * 1. **Config** (@ref u_stereo_uvc_config_parse / _load): which devices are + * claimed and how to read them. OPT-IN only — with no config file the + * source enumerates nothing and never touches a camera. A JSON file in + * the per-user DisplayXR config dir (the u_setting.h directory): + * Windows %LOCALAPPDATA%\DisplayXR\stereo-cameras.json + * POSIX $XDG_CONFIG_HOME/displayxr/stereo-cameras.json + * (else ~/.config/displayxr/stereo-cameras.json) + * or the file named by DXR_STEREO_CAMERA_UVC_CONFIG in the SERVICE's + * environment; DXR_STEREO_CAMERA_UVC=0 turns the source off. + * @code + * {"uvc": [{ + * "match": "eyes", // friendly-name substring, case-insensitive, and/or + * "vid_pid": "1234:abcd", // USB VID:PID (hex); both given = both must match + * "name": "Stereo webcam", // display name ("" = the device's own) + * "layout": "sbs-half", // "sbs-full": each half is a full-res eye; + * // "sbs-half": each half is an eye squeezed 2:1 horizontally + * "eyes": "lr", // "lr" (left lens in the left half) | "rl" + * "mode": "3840x2160@60", // capture mode (optional: else the largest, fastest) + * "eye_size": "1280x720", // per-eye output (optional: else the information-honest size, + * // capped at 1280 px wide) + * "baseline_mm": 60, // nominal lens separation (optional) + * "hfov_deg": 90, // nominal per-eye horizontal field of view (optional) + * "calibration": "C:/cal/eyes.yaml" // OpenCV-style K1 D1 K2 D2 R T (optional) + * }]} + * @endcode + * A built-in device allowlist is deliberately empty: none is shipped + * without a public spec for the device's SBS layout. + * 2. **Matching** (@ref u_stereo_uvc_entry_matches): name substring and/or + * USB VID:PID. A device a display plug-in already reports as its camera + * (its platform_device_hint) is never claimed — the tracker camera belongs + * to the plug-in and its eye tracker (@ref u_stereo_uvc_device_claimed_by_hint). + * 3. **Geometry**: the SBS split (@ref u_stereo_uvc_split: either eye order, + * full or half SBS, resampled to square-pixel eyes, NV12 out) and, for an + * uncalibrated device, a NOMINAL pinhole (@ref + * u_stereo_uvc_nominal_calibration: from HFOV + eye size + baseline, no + * distortion, a parallel pair). The service reports such a camera WITHOUT + * CALIBRATED; its RECTIFIED output is the frames as delivered with the + * service's online vertical refinement doing the row alignment. + * A calibration file (@ref u_stereo_uvc_calibration_parse) makes it + * CALIBRATED and the service's ordinary rectifier runs. + * 4. **The source** (@ref u_stereo_uvc_create ... _close): the same six + * operations as the plug-in camera slots (enumerate, calibration, open, + * wait_frame, release_frame, close), over a capture BACKEND + * (@ref u_stereo_uvc_backend). The backend is the only OS-specific part: + * Media Foundation on Windows (os_uvc_capture_mf.cpp), V4L2 on Linux (a + * stub for now), and @ref u_stereo_uvc_fake_backend_init — a synthetic SBS + * camera with a known disparity and vertical misalignment, for tests and + * hardware-free development (`"fake": {...}` in the config). + * + * The device is opened only while the service holds the source open, i.e. + * while a stream is started (plus the manager's 2 s linger). + * + * @ingroup aux_util + */ + +#pragma once + +#include "xrt/xrt_compiler.h" +#include "xrt/xrt_plugin.h" +#include "xrt/xrt_results.h" + +#include "util/u_stereo_camera.h" + +#include +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +//! Config entries (= cameras) at most. +#define U_STEREO_UVC_MAX_ENTRIES 4 +//! OS capture devices considered at most. +#define U_STEREO_UVC_MAX_DEVICES 16 +//! Modes a backend reports per device at most. +#define U_STEREO_UVC_MAX_MODES 64 +//! The default per-eye output width cap (a 3D call does not need more). +#define U_STEREO_UVC_DEFAULT_MAX_EYE_WIDTH 1280u +//! Used for the nominal model when the config gives none (reported as 0 = unknown). +#define U_STEREO_UVC_DEFAULT_HFOV_DEG 70.0 +#define U_STEREO_UVC_DEFAULT_BASELINE_MM 60.0 +//! File name in the per-user DisplayXR config dir. +#define U_STEREO_UVC_CONFIG_FILENAME "stereo-cameras.json" + + +/* + * + * Config. + * + */ + +enum u_stereo_uvc_layout +{ + //! Each half of the frame is one eye at full horizontal resolution (2 sensors side by side). + U_STEREO_UVC_LAYOUT_SBS_FULL = 1, + //! Each half is one eye squeezed 2:1 horizontally (a W x H frame carries two W x H views). + U_STEREO_UVC_LAYOUT_SBS_HALF = 2, +}; + +struct u_stereo_uvc_entry +{ + char name_contains[128]; //!< "" = not matched by name + bool match_vid_pid; + uint16_t vid, pid; + char display_name[128]; //!< "" = the device's friendly name + uint32_t layout; //!< enum u_stereo_uvc_layout + //! "eyes": "rl" — the device puts its RIGHT lens in the LEFT half. + bool swap_eyes; + uint32_t mode_width, mode_height; //!< 0 = auto + float mode_fps; //!< 0 = auto + uint32_t eye_width, eye_height; //!< 0 = auto + float baseline_mm; //!< 0 = not configured + float hfov_deg; //!< 0 = not configured + char calibration[512]; //!< "" = none (nominal model) +}; + +//! The synthetic camera's scene (fake backend). +struct u_stereo_uvc_fake_params +{ + //! Horizontal disparity of the scene, left x - right x, fraction of the eye width. + double disparity; + //! Vertical misalignment: features appear this much LOWER in the right eye, fraction of the eye height. + double dy; + //! Raw pixel format the fake delivers: 1 = NV12, 2 = YUY2 (enum u_stereo_uvc_pixel). + uint32_t pixel; + float fps; +}; + +struct u_stereo_uvc_config +{ + uint32_t count; + struct u_stereo_uvc_entry entries[U_STEREO_UVC_MAX_ENTRIES]; + //! "fake": a synthetic device the entries can match ("DisplayXR synthetic SBS camera"). + bool fake; + struct u_stereo_uvc_fake_params fake_params; +}; + +//! One entry with every default filled (sbs-full, lr, auto everything). +void +u_stereo_uvc_entry_defaults(struct u_stereo_uvc_entry *e); + +/*! + * Parse a config document (see the file comment). Unknown keys are ignored; + * an entry with neither "match" nor "vid_pid" or a bad value fails the whole + * document (a typo must not silently claim a different camera). + * @return false + a message in @p err on an invalid document. + */ +bool +u_stereo_uvc_config_parse(const char *json, struct u_stereo_uvc_config *out, char *err, size_t err_cap); + +//! The config file path: DXR_STEREO_CAMERA_UVC_CONFIG, else the per-user default. +bool +u_stereo_uvc_config_path(char *out, size_t cap); + +enum u_stereo_uvc_config_load_result +{ + U_STEREO_UVC_CONFIG_NONE = 0, //!< no file (or DXR_STEREO_CAMERA_UVC=0): the source is off + U_STEREO_UVC_CONFIG_OK = 1, //!< parsed + U_STEREO_UVC_CONFIG_INVALID = 2, //!< present but unusable (@p err says why): the source is off +}; + +//! Read and parse the config file. @p path gets the path consulted. +enum u_stereo_uvc_config_load_result +u_stereo_uvc_config_load(struct u_stereo_uvc_config *out, char *path, size_t path_cap, char *err, size_t err_cap); + + +/* + * + * Devices + matching. + * + */ + +struct u_stereo_uvc_device +{ + char name[128]; //!< friendly name, UTF-8 + //! The OS device id: Windows the Media Foundation symbolic link + //! (\\?\usb#vid_xxxx&pid_yyyy&mi_00#...), Linux /dev/videoN. + char id[256]; + bool has_vid_pid; + uint16_t vid, pid; +}; + +/*! + * Find a USB VID:PID in @p s: "vid_XXXX&pid_YYYY" anywhere (any case, as in + * Windows device ids and symbolic links) or a plain "XXXX:YYYY". + */ +bool +u_stereo_uvc_parse_vid_pid(const char *s, uint16_t *out_vid, uint16_t *out_pid); + +//! Does config entry @p e claim device @p d? +bool +u_stereo_uvc_entry_matches(const struct u_stereo_uvc_entry *e, const struct u_stereo_uvc_device *d); + +//! @ref u_stereo_uvc_assign: the entry matched no available device. +#define U_STEREO_UVC_ASSIGN_NONE (-1) +//! @ref u_stereo_uvc_assign: a NAME-ONLY entry matched several devices — claims none. +#define U_STEREO_UVC_ASSIGN_AMBIGUOUS (-2) + +/*! + * Which device each config entry claims, in entry order: the first matching + * device not @p unavailable (a plug-in's camera) and not claimed by an earlier + * entry. A name-substring entry WITHOUT vid_pid that matches more than one + * such device claims nothing (@ref U_STEREO_UVC_ASSIGN_AMBIGUOUS): a loose + * name ("3D", a vendor prefix) must never pick, say, a display's own tracking + * camera over the intended webcam by enumeration order. An entry with a + * VID:PID takes the first of several identical devices. + * @param unavailable may be NULL. + */ +void +u_stereo_uvc_assign(const struct u_stereo_uvc_config *cfg, + const struct u_stereo_uvc_device *devs, + uint32_t device_count, + const bool *unavailable, + int32_t out_device_for_entry[U_STEREO_UVC_MAX_ENTRIES]); + +/*! + * Is @p d the physical device behind a plug-in camera whose + * platform_device_hint is @p hint? Same OS id (case-insensitive), or the same + * USB VID:PID when the hint carries one. Such a device is never claimed. + */ +bool +u_stereo_uvc_device_claimed_by_hint(const struct u_stereo_uvc_device *d, const char *hint); + + +/* + * + * Geometry. + * + */ + +/*! + * The per-eye output size for a @p frame_w x @p frame_h capture. Requested + * @p want_w x @p want_h when non-zero (rounded down to even), else the + * information-honest square-pixel size — SBS_FULL: (W/2, H); SBS_HALF: + * (W/2, H/2) — scaled down to at most @ref U_STEREO_UVC_DEFAULT_MAX_EYE_WIDTH + * wide, aspect kept, even extents. + * @return false on a frame that cannot be split (odd/zero width, too small). + */ +bool +u_stereo_uvc_eye_size(uint32_t layout, + uint32_t frame_w, + uint32_t frame_h, + uint32_t want_w, + uint32_t want_h, + uint32_t *out_w, + uint32_t *out_h); + +/*! + * The nominal RAW calibration of an uncalibrated pair: per eye a pinhole with + * fx = fy = (eye_w / 2) / tan(hfov / 2), principal point at the image centre + * ((w - 1) / 2, (h - 1) / 2, OpenCV pixel centres), no distortion; the right + * camera a pure +x translation (R = I, T = (-baseline, 0, 0) mm). + */ +void +u_stereo_uvc_nominal_calibration(uint32_t eye_w, + uint32_t eye_h, + double hfov_deg, + double baseline_mm, + struct xrt_plugin_stereo_camera_calibration *out); + +/*! + * Parse an OpenCV-style stereo calibration — FileStorage YAML (`K1: !!opencv-matrix + * ... data: [...]`) or JSON (`"K1": [[...]]` or flat) — keys K1/M1, D1, K2/M2, + * D2, R, T, optional image_width / image_height (default @p default_w x @p + * default_h: the per-eye size the camera delivers before any resampling). + * D with 4 or 5 values is RADTAN5, 8 is RADTAN8; "distortion_model": "fisheye" + * makes 4 values KB4. T in mm; a |T| below 1 is taken as metres and scaled. + */ +bool +u_stereo_uvc_calibration_parse(const char *text, + uint32_t default_w, + uint32_t default_h, + struct xrt_plugin_stereo_camera_calibration *out, + char *err, + size_t err_cap); + + +/* + * + * Frames. + * + */ + +//! Raw pixel formats a backend hands over (decoded: MJPEG is the backend's job). +enum u_stereo_uvc_pixel +{ + U_STEREO_UVC_PIXEL_NV12 = 1, + U_STEREO_UVC_PIXEL_YUY2 = 2, +}; + +struct u_stereo_uvc_raw_frame +{ + uint32_t pixel; //!< enum u_stereo_uvc_pixel + uint32_t width, height; + const uint8_t *planes[2]; //!< NV12: Y, UV. YUY2: one packed plane. + uint32_t pitches[2]; + int64_t time_ns; //!< os_monotonic_get_ns() domain + bool time_is_exposure; +}; + +/*! + * Split + resample one raw SBS frame into an NV12 SBS image of two + * @p eye_w x @p eye_h eyes, LEFT lens left (swapping the halves when + * @p swap_eyes). Each eye samples only its own half. Downscale is an integer + * box pre-reduction then bilinear (pixel-centre mapping), so a 2:1 vertical + * reduction of a half-SBS frame is an exact 2-row average. + * @p dst_layout must be the NV12 layout of (2 * eye_w) x eye_h. + */ +bool +u_stereo_uvc_split(const struct u_stereo_uvc_raw_frame *in, + bool swap_eyes, + uint32_t eye_w, + uint32_t eye_h, + uint8_t *dst, + const struct u_stereo_camera_planes *dst_layout); + + +/* + * + * Capture backend (the OS part). + * + */ + +struct u_stereo_uvc_mode +{ + uint32_t width, height; + float fps; +}; + +//! Result of @ref u_stereo_uvc_backend::read. +enum u_stereo_uvc_read +{ + U_STEREO_UVC_READ_OK = 0, + U_STEREO_UVC_READ_TIMEOUT = 1, + U_STEREO_UVC_READ_ERROR = 2, //!< the device failed / went away +}; + +struct u_stereo_uvc_backend +{ + const char *name; //!< "media-foundation", "v4l2", "fake" + void *ctx; + //! Capture devices present now. Must NOT open or start any of them. + uint32_t (*enumerate)(void *ctx, struct u_stereo_uvc_device *out, uint32_t cap); + /*! + * The capture modes of device @p id. May instantiate the OS device object + * to read its media types, but never starts streaming. NULL / 0 = unknown. + */ + uint32_t (*list_modes)(void *ctx, const char *id, struct u_stereo_uvc_mode *out, uint32_t cap); + /*! + * Open + start @p id at @p mode (exact size; fps best effort). @p out_w x + * @p out_h is the frame size the source will resample to (2 x eye width x + * eye height): a backend that can decode + scale on the GPU delivers that + * instead of the full capture size (half-SBS 3840x2160 -> 2560x720). 0 = + * no preference. The source resamples whatever arrives, so honouring it is + * optional. + */ + bool (*open)(void *ctx, + const char *id, + const struct u_stereo_uvc_mode *mode, + uint32_t out_w, + uint32_t out_h, + void **out_handle); + //! Block up to @p timeout_ns for the next decoded frame; valid until the next read / close. + uint32_t (*read)(void *handle, int64_t timeout_ns, struct u_stereo_uvc_raw_frame *out); + void (*close)(void *handle); +}; + +/*! + * The synthetic backend: one device ("DisplayXR synthetic SBS camera", id + * "dxr-fake-uvc", no VID:PID), listing 1280x480 / 2560x720 / 1280x960 and + * opening any even mode, rendering a smooth random texture with the + * configured disparity + right-eye vertical offset. The scene is defined in + * EYE-NORMALISED coordinates, so a full-SBS and a half-SBS entry read back + * the same scene after the split. @p params must outlive the backend (it is + * the backend's ctx). A config with `"fake"` uses it INSTEAD of OS capture. + */ +void +u_stereo_uvc_fake_backend_init(struct u_stereo_uvc_backend *out, const struct u_stereo_uvc_fake_params *params); + + +/* + * + * The source: the plug-in camera slots' shape, over a backend. + * + */ + +struct u_stereo_uvc; +struct u_stereo_uvc_stream; + +/*! + * Resolve the config against the devices present. @p backend NULL = no OS + * backend (only a fake device, if configured, can match). Devices whose id / + * VID:PID equals one of @p exclude_hints (the plug-ins' platform_device_hint) + * are skipped. One camera per config entry: the first matching, unclaimed + * device. Always returns an object (zero cameras when nothing matched). + */ +struct u_stereo_uvc * +u_stereo_uvc_create(const struct u_stereo_uvc_config *cfg, + const struct u_stereo_uvc_backend *backend, + const char *const *exclude_hints, + uint32_t exclude_count); + +void +u_stereo_uvc_destroy(struct u_stereo_uvc **uvc_ptr); + +//! Like the plug-in slot: fill up to @p capacity infos (struct_size preset), return the count. +uint32_t +u_stereo_uvc_enumerate(struct u_stereo_uvc *uvc, uint32_t capacity, struct xrt_plugin_stereo_camera_info *out); + +/*! + * Calibration of camera @p index: the file's (@p out_nominal false; the + * camera then reports CALIBRATED) or the nominal pinhole (@p out_nominal + * true). @p out_baseline_known / @p out_hfov_known: whether the config gave + * them (an unknown one is a default the service must not report). + */ +xrt_result_t +u_stereo_uvc_get_calibration(struct u_stereo_uvc *uvc, + uint32_t index, + struct xrt_plugin_stereo_camera_calibration *out, + bool *out_nominal, + bool *out_baseline_known, + bool *out_hfov_known); + +//! The capture mode camera @p index opens at (diagnostics). +bool +u_stereo_uvc_get_mode(struct u_stereo_uvc *uvc, uint32_t index, struct u_stereo_uvc_mode *out, uint32_t *out_layout); + +//! Open (start capturing) camera @p index. +xrt_result_t +u_stereo_uvc_open(struct u_stereo_uvc *uvc, uint32_t index, struct u_stereo_uvc_stream **out_stream); + +//! An enum xrt_plugin_stereo_camera_wait. On OK @p out stays valid until release. +uint32_t +u_stereo_uvc_wait_frame(struct u_stereo_uvc_stream *s, int64_t timeout_ns, struct xrt_plugin_stereo_camera_frame *out); + +void +u_stereo_uvc_release_frame(struct u_stereo_uvc_stream *s); + +void +u_stereo_uvc_close(struct u_stereo_uvc_stream *s); + +#ifdef __cplusplus +} +#endif diff --git a/src/xrt/include/xrt/xrt_stereo_camera.h b/src/xrt/include/xrt/xrt_stereo_camera.h index 9310badc19..375e00cd83 100644 --- a/src/xrt/include/xrt/xrt_stereo_camera.h +++ b/src/xrt/include/xrt/xrt_stereo_camera.h @@ -73,6 +73,19 @@ enum xrt_stereo_camera_transport //! Format / transport sets are one bit per enum value: (1 << value). #define XRT_STEREO_CAMERA_BIT(v) (1ull << (uint32_t)(v)) +/*! + * Where a camera's frames come from (diagnostics only: `displayxr-cli camera + * list`; not exposed through the OpenXR API, which is source-neutral). + */ +enum xrt_stereo_camera_source +{ + //! The active display plug-in's stereo_camera_* slots (ADR-043). + XRT_STEREO_CAMERA_SOURCE_PLUGIN = 0, + //! The service's own vendor-neutral UVC side-by-side source (ADR-043 + //! Amendment 4): an opted-in plain stereo webcam. + XRT_STEREO_CAMERA_SOURCE_UVC = 1, +}; + /*! * One enumerated camera (XrStereoCameraPropertiesDXR). */ @@ -90,7 +103,7 @@ struct xrt_stereo_camera_properties float max_frame_rate; float baseline_mm; float horizontal_fov_deg; - uint32_t reserved; + uint32_t source; //!< enum xrt_stereo_camera_source (was reserved; same wire size) uint64_t supported_formats; //!< XRT_STEREO_CAMERA_BIT(format) uint64_t supported_transports; //!< XRT_STEREO_CAMERA_BIT(transport) }; diff --git a/src/xrt/ipc/CMakeLists.txt b/src/xrt/ipc/CMakeLists.txt index a1d9d0b76a..85a67b8a5e 100644 --- a/src/xrt/ipc/CMakeLists.txt +++ b/src/xrt/ipc/CMakeLists.txt @@ -135,6 +135,8 @@ target_include_directories( PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/../compositor # For d3d11_service headers ) target_link_libraries(ipc_server PRIVATE aux_util aux_util_process aux_util_debug_gui ipc_shared aux_math) +# ADR-043 Amendment 4: the UVC side-by-side stereo camera source's capture backend. +target_link_libraries(ipc_server PRIVATE aux_os_uvc) # Link the D3D11 service compositor — the IPC handlers call its # vendor-agnostic helpers for predicted eye positions, per-client window diff --git a/src/xrt/ipc/server/ipc_server_stereo_camera.c b/src/xrt/ipc/server/ipc_server_stereo_camera.c index 1c796231b3..562c1e1e9b 100644 --- a/src/xrt/ipc/server/ipc_server_stereo_camera.c +++ b/src/xrt/ipc/server/ipc_server_stereo_camera.c @@ -87,6 +87,21 @@ * px, |b| <= 2 px / 100 px), gated (>= 50 matches over >= 3 frames), * re-verified on every re-open, and DXR_STEREO_CAMERA_REFINE=0 turns it off. * + * Sources (Amendment 4) — a camera comes from the active plug-in's + * stereo_camera_* slots or from the service's own vendor-neutral UVC + * side-by-side source (util/u_stereo_uvc.h over os/os_uvc_capture.h: Media + * Foundation on Windows, a V4L2 TODO elsewhere), opted in per device by the + * user's stereo-cameras.json. Both have the same six-operation shape; the + * src_* dispatch is the only place that tells them apart, so consent, the + * foreground rule, the indicator, linger, rates, rings and rectification are + * one code path. Plug-in cameras are enumerated first and their + * platform_device_hints are never claimed by the UVC source. An uncalibrated + * UVC pair gets a NOMINAL model (pinhole from its configured HFOV + baseline, + * parallel, undistorted): the camera reports no CALIBRATED, RAW calibration is + * FEATURE_NOT_SUPPORTED, and RECTIFIED is a PASS-THROUGH (no remap, nominal + * numbers) until the online refinement below folds a vertical correction in — + * from then on the ordinary LUT path with the corrected geometry. + * * Transports — R1 implements SHARED_MEMORY only. The GPU transports are * designed to reuse this ring's state machine unchanged, only the slot storage * differs: @@ -128,7 +143,9 @@ #include "util/u_misc.h" #include "util/u_stereo_camera.h" #include "util/u_stereo_rectify.h" +#include "util/u_stereo_uvc.h" #include "util/u_stereo_vrefine.h" +#include "os/os_uvc_capture.h" #include #include @@ -317,6 +334,15 @@ struct scam_rectifier bool (*apply)(struct scam_rectifier *r, const struct xrt_plugin_stereo_camera_frame *in, struct xrt_plugin_stereo_camera_frame *out); + /*! + * Amendment 4: the camera's model is a NOMINAL parallel pinhole pair (an + * uncalibrated UVC stereo webcam) and no vertical correction has been + * folded in yet — the rectified frames ARE the source frames (no remap, + * no zoom), described by the nominal intrinsics below. The first applied + * refinement swaps in real maps and clears it. + */ + bool passthrough; + double pass_f, pass_cx, pass_cy; // stats uint64_t ns_sum; uint64_t count; @@ -352,10 +378,17 @@ struct scam_rectifier struct scam_camera { - uint32_t index; //!< plug-in index + uint32_t index; //!< position in mgr->cams (streams key on it) + //! enum xrt_stereo_camera_source: the plug-in's slots or the service's UVC source. + uint32_t source; + uint32_t source_index; //!< the camera's index inside its source uint64_t camera_id; struct xrt_plugin_stereo_camera_info info; bool have_calib; + //! Amendment 4: @ref calib is a NOMINAL model (an uncalibrated UVC pair): + //! it drives the rectifier, but is never handed out as RAW calibration. + bool calib_nominal; + bool baseline_known, hfov_known; //!< nominal model: did the user configure them struct xrt_plugin_stereo_camera_calibration calib; uint32_t state; uint32_t calibration_generation; @@ -363,7 +396,9 @@ struct scam_camera //! sane; NULL = no service-side rectification (RAW-flagged fallback). struct scam_rectifier *rect; - struct xrt_plugin_stereo_camera *src; //!< owned by the camera thread + //! Owned by the camera thread: a struct xrt_plugin_stereo_camera (PLUGIN) + //! or a struct u_stereo_uvc_stream (UVC); see the src_* dispatch below. + void *src; uint32_t started_count; int64_t linger_deadline_ns; int64_t last_frame_ns; @@ -397,6 +432,8 @@ struct ipc_server_stereo_camera bool shutting_down; const struct xrt_plugin_iface *iface; struct xrt_plugin_instance *inst; + //! Amendment 4: the service's own UVC side-by-side source (NULL = none configured). + struct u_stereo_uvc *uvc; uint32_t camera_count; struct scam_camera cams[XRT_STEREO_CAMERA_MAX_CAMERAS]; struct scam_stream streams[MAX_STREAMS]; @@ -861,6 +898,70 @@ baseline_mm(const struct xrt_plugin_stereo_camera_calibration *c) } +/* + * + * Source dispatch (Amendment 4): a camera is the plug-in's (its stereo_camera_* + * slots) or the service's own UVC source (u_stereo_uvc). Both have the same + * six-operation shape; everything above and below this block is shared. + * + */ + +static const char * +source_str(uint32_t source) +{ + return source == XRT_STEREO_CAMERA_SOURCE_UVC ? "uvc" : "plug-in"; +} + +static xrt_result_t +src_open(struct ipc_server_stereo_camera *m, const struct scam_camera *cam, void **out) +{ + *out = NULL; + if (cam->source == XRT_STEREO_CAMERA_SOURCE_UVC) { + struct u_stereo_uvc_stream *s = NULL; + xrt_result_t xret = u_stereo_uvc_open(m->uvc, cam->source_index, &s); + *out = s; + return xret; + } + struct xrt_plugin_stereo_camera *pc = NULL; + xrt_result_t xret = m->iface->stereo_camera_open(m->inst, cam->source_index, &pc); + *out = pc; + return xret; +} + +static uint32_t +src_wait_frame(struct ipc_server_stereo_camera *m, + const struct scam_camera *cam, + void *src, + int64_t timeout_ns, + struct xrt_plugin_stereo_camera_frame *out) +{ + if (cam->source == XRT_STEREO_CAMERA_SOURCE_UVC) { + return u_stereo_uvc_wait_frame((struct u_stereo_uvc_stream *)src, timeout_ns, out); + } + return m->iface->stereo_camera_wait_frame((struct xrt_plugin_stereo_camera *)src, timeout_ns, out); +} + +static void +src_release_frame(struct ipc_server_stereo_camera *m, const struct scam_camera *cam, void *src) +{ + if (cam->source == XRT_STEREO_CAMERA_SOURCE_UVC) { + u_stereo_uvc_release_frame((struct u_stereo_uvc_stream *)src); + return; + } + m->iface->stereo_camera_release_frame((struct xrt_plugin_stereo_camera *)src); +} + +static void +src_close(struct ipc_server_stereo_camera *m, const struct scam_camera *cam, void *src) +{ + if (cam->source == XRT_STEREO_CAMERA_SOURCE_UVC) { + u_stereo_uvc_close((struct u_stereo_uvc_stream *)src); + return; + } + m->iface->stereo_camera_close((struct xrt_plugin_stereo_camera *)src); +} + + /* * * Rectifier (R2). @@ -924,6 +1025,69 @@ rectify_apply_cpu(struct scam_rectifier *r, return true; } +//! Pass-through backend (Amendment 4): the nominal parallel pair needs no remap. +static bool +rectify_apply_passthrough(struct scam_rectifier *r, + const struct xrt_plugin_stereo_camera_frame *in, + struct xrt_plugin_stereo_camera_frame *out) +{ + if (in->width != 2 * r->geo.width || in->height != r->geo.height || in->planes[0] == NULL) { + return false; + } + *out = *in; + return true; +} + +/*! + * Is @p c an exactly parallel, undistorted pair with one pinhole for both + * eyes at the frame size — i.e. are the frames already rectified, so that + * the nominal intrinsics describe them with no remap? (A nominal model is + * built that way; a real calibration practically never is.) + */ +static bool +calibration_is_parallel_pinhole(const struct xrt_plugin_stereo_camera_calibration *c, uint32_t w, uint32_t h) +{ + if (c->distortion_model != XRT_PLUGIN_STEREO_CAMERA_DISTORTION_NONE || c->image_width != w || + c->image_height != h) { + return false; + } + for (int i = 0; i < 4; i++) { + if (c->k[0][i] != c->k[1][i]) { + return false; + } + } + if (c->k[0][0] != c->k[0][1]) { + return false; + } + for (int i = 0; i < 3; i++) { + for (int j = 0; j < 3; j++) { + if (c->rotation_right_from_left[i][j] != (i == j ? 1.0 : 0.0)) { + return false; + } + } + } + const double *t = c->translation_right_from_left_mm; + return t[0] < 0.0 && t[1] == 0.0 && t[2] == 0.0; +} + +/*! + * The rectified intrinsics clients are told: the pass-through's nominal ones, + * else the rectifier's geometry. + */ +static void +rectified_intrinsics(const struct scam_rectifier *r, double *f, double *cx, double *cy) +{ + if (r->passthrough) { + *f = r->pass_f; + *cx = r->pass_cx; + *cy = r->pass_cy; + } else { + *f = r->geo.f; + *cx = r->geo.cx; + *cy = r->geo.cy; + } +} + static void rectifier_destroy(struct scam_rectifier **rp) { @@ -1008,6 +1172,17 @@ rectifier_create(const struct scam_camera *cam) return NULL; } r->apply = rectify_apply_cpu; + if (cam->calib_nominal && calibration_is_parallel_pinhole(c, in.width, in.height)) { + // Amendment 4: an uncalibrated pair modelled as an ideal parallel rig. + // Bouguet of that is the identity up to the alpha = 0 border zoom, so + // deliver the frames untouched (no remap, no zoom) with the nominal + // numbers; the refinement below aligns the rows from the frames. + r->passthrough = true; + r->pass_f = c->k[0][0]; + r->pass_cx = c->k[0][2]; + r->pass_cy = c->k[0][3]; + r->apply = rectify_apply_passthrough; + } r->in = in; r->mgr = cam->mgr; r->camera_id = cam->camera_id; @@ -1043,6 +1218,15 @@ rectifier_create(const struct scam_camera *cam) "look swapped; rectified disparities will be NEGATIVE", (unsigned long long)cam->camera_id); } + if (r->passthrough) { + U_LOG_W( + "stereo camera %llu: NOMINAL model (uncalibrated) — RECTIFIED = the frames as delivered " + "(f %.2f px, principal point (%.2f, %.2f), nominal baseline %.2f mm) until the online " + "refinement aligns the rows%s", + (unsigned long long)cam->camera_id, r->pass_f, r->pass_cx, r->pass_cy, r->geo.baseline, + r->refine ? "" : " (refinement OFF: rows are as the device delivers them)"); + return r; + } U_LOG_W( "stereo camera %llu: rectifier ready (CPU) — f %.2f px, principal point (%.2f, %.2f), baseline %.2f mm, " "valid-region zoom x%.4f, %u/%u valid taps, built in %.1f ms", @@ -1168,6 +1352,9 @@ rectifier_swap_pending_locked(struct scam_camera *cam) r->lut[0] = r->pending_lut[0]; r->lut[1] = r->pending_lut[1]; r->pending_ready = false; + // A folded-in correction needs real maps: a nominal pass-through ends here. + r->passthrough = false; + r->apply = rectify_apply_cpu; // Consumers re-read the rectified calibration (f / principal point move // by the alpha = 0 re-crop) on the next frame's generation change. cam->calibration_generation++; @@ -1437,7 +1624,6 @@ camera_thread(void *ptr) { struct scam_camera *cam = (struct scam_camera *)ptr; struct ipc_server_stereo_camera *m = cam->mgr; - const struct xrt_plugin_iface *iface = m->iface; os_mutex_lock(&m->lock); while (!m->shutting_down) { @@ -1447,10 +1633,10 @@ camera_thread(void *ptr) // stream starts (or the manager shuts down). if (cam->started_count == 0 && (cam->src == NULL || now >= cam->linger_deadline_ns)) { if (cam->src != NULL) { - struct xrt_plugin_stereo_camera *src = cam->src; + void *src = cam->src; cam->src = NULL; os_mutex_unlock(&m->lock); - iface->stereo_camera_close(src); + src_close(m, cam, src); os_mutex_lock(&m->lock); U_LOG_W("stereo camera %llu: source closed (no started stream for %lld ms)", (unsigned long long)cam->camera_id, (long long)(LINGER_NS / 1000000)); @@ -1470,20 +1656,20 @@ camera_thread(void *ptr) os_cond_wait_timeout_ns(&cam->cond, &m->lock, (uint64_t)(cam->retry_after_ns - now)); continue; } - struct xrt_plugin_stereo_camera *src = NULL; + void *src = NULL; os_mutex_unlock(&m->lock); - xrt_result_t xret = iface->stereo_camera_open(m->inst, cam->index, &src); + xrt_result_t xret = src_open(m, cam, &src); os_mutex_lock(&m->lock); if (xret != XRT_SUCCESS || src == NULL) { - U_LOG_W("stereo camera %llu: plug-in open failed (%d); retrying in 1 s", - (unsigned long long)cam->camera_id, (int)xret); + U_LOG_W("stereo camera %llu: %s source open failed (%d); retrying in 1 s", + (unsigned long long)cam->camera_id, source_str(cam->source), (int)xret); set_state_locked(cam, XRT_STEREO_CAMERA_STATE_UNAVAILABLE); cam->retry_after_ns = os_monotonic_get_ns() + REOPEN_BACKOFF_NS; continue; } if (m->shutting_down) { os_mutex_unlock(&m->lock); - iface->stereo_camera_close(src); + src_close(m, cam, src); os_mutex_lock(&m->lock); break; } @@ -1498,11 +1684,11 @@ camera_thread(void *ptr) set_state_locked(cam, XRT_STEREO_CAMERA_STATE_WAITING); } - struct xrt_plugin_stereo_camera *src = cam->src; + void *src = cam->src; struct xrt_plugin_stereo_camera_frame f; memset(&f, 0, sizeof(f)); os_mutex_unlock(&m->lock); - uint32_t w = iface->stereo_camera_wait_frame(src, WAIT_TIMEOUT_NS, &f); + uint32_t w = src_wait_frame(m, cam, src, WAIT_TIMEOUT_NS, &f); os_mutex_lock(&m->lock); now = os_monotonic_get_ns(); @@ -1529,7 +1715,7 @@ camera_thread(void *ptr) rfp = ok ? &rf : NULL; } publish_locked(m, cam, &f, rfp, now); - iface->stereo_camera_release_frame(src); + src_release_frame(m, cam, src); break; } case XRT_PLUGIN_STEREO_CAMERA_WAIT_TIMEOUT: @@ -1544,22 +1730,22 @@ camera_thread(void *ptr) cam->last_frame_ns = 0; break; default: - U_LOG_W("stereo camera %llu: plug-in reported a source error; closing, retry in 1 s", - (unsigned long long)cam->camera_id); + U_LOG_W("stereo camera %llu: %s source reported an error; closing, retry in 1 s", + (unsigned long long)cam->camera_id, source_str(cam->source)); set_state_locked(cam, XRT_STEREO_CAMERA_STATE_UNAVAILABLE); cam->src = NULL; os_mutex_unlock(&m->lock); - iface->stereo_camera_close(src); + src_close(m, cam, src); os_mutex_lock(&m->lock); cam->retry_after_ns = os_monotonic_get_ns() + REOPEN_BACKOFF_NS; break; } } if (cam->src != NULL) { - struct xrt_plugin_stereo_camera *src = cam->src; + void *src = cam->src; cam->src = NULL; os_mutex_unlock(&m->lock); - iface->stereo_camera_close(src); + src_close(m, cam, src); os_mutex_lock(&m->lock); } os_mutex_unlock(&m->lock); @@ -1573,8 +1759,189 @@ camera_thread(void *ptr) * */ +/*! + * Validate + register one camera description @p info from @p source at its + * source index @p source_index. Manager lock not needed (create time). + */ +static struct scam_camera * +add_camera(struct ipc_server_stereo_camera *m, + uint32_t source, + uint32_t source_index, + struct xrt_plugin_stereo_camera_info *info) +{ + if (m->camera_count >= XRT_STEREO_CAMERA_MAX_CAMERAS) { + U_LOG_W("stereo camera: more than %d cameras — \"%s\" (%s) skipped", XRT_STEREO_CAMERA_MAX_CAMERAS, + info->display_name, source_str(source)); + return NULL; + } + info->display_name[sizeof(info->display_name) - 1] = '\0'; + info->device_identity[sizeof(info->device_identity) - 1] = '\0'; + info->platform_device_hint[sizeof(info->platform_device_hint) - 1] = '\0'; + if (info->eye_width == 0 || info->eye_height == 0 || (info->eye_width & 1u) || (info->eye_height & 1u) || + info->native_format < XRT_PLUGIN_STEREO_CAMERA_FORMAT_GRAY8 || + info->native_format > XRT_PLUGIN_STEREO_CAMERA_FORMAT_BGRA8) { + U_LOG_W("stereo camera: %s camera %u has a malformed description (%ux%u, format %u) — skipped", + source_str(source), source_index, info->eye_width, info->eye_height, info->native_format); + return NULL; + } + struct scam_camera *cam = &m->cams[m->camera_count]; + cam->index = m->camera_count; + cam->source = source; + cam->source_index = source_index; + cam->camera_id = (uint64_t)m->camera_count + 1; // never reused in this service lifetime + cam->info = *info; + cam->mgr = m; + cam->state = XRT_STEREO_CAMERA_STATE_AVAILABLE; + os_cond_init(&cam->cond); + m->camera_count++; + return cam; +} + +//! Shared tail of camera registration: rectifier, rate sanity, the WARN line. +static void +finish_camera(struct scam_camera *cam) +{ + if (cam->have_calib && (cam->info.flags & XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED) == 0) { + cam->rect = rectifier_create(cam); + } + if (!(cam->info.max_frame_rate >= 0.0f && cam->info.max_frame_rate <= 1000.0f)) { + cam->info.max_frame_rate = 0.0f; // NaN / negative / absurd = unknown + } + char rate[48]; + if (cam->info.max_frame_rate > 0.0f) { + snprintf(rate, sizeof(rate), "advertised %.1f Hz", cam->info.max_frame_rate); + } else { + snprintf(rate, sizeof(rate), "rate unknown"); + } + U_LOG_W( + "stereo camera %llu: \"%s\" (%s source) %ux%u per eye, %s (measured on first open), flags 0x%x, native " + "format %u%s", + (unsigned long long)cam->camera_id, cam->info.display_name, source_str(cam->source), cam->info.eye_width, + cam->info.eye_height, rate, cam->info.flags, cam->info.native_format, + cam->calib_nominal ? ", NOMINAL model (uncalibrated)" : ""); +} + +static void +add_plugin_cameras(struct ipc_server_stereo_camera *m, struct xrt_instance *xinst) +{ + const struct xrt_plugin_iface *iface = NULL; + struct xrt_plugin_instance *inst = NULL; + if (xinst == NULL || xinst->get_active_plugin == NULL || !xinst->get_active_plugin(xinst, &iface, &inst) || + !xrt_plugin_iface_has_stereo_camera(iface)) { + U_LOG_I("stereo camera: the active plug-in provides no camera source"); + return; + } + m->iface = iface; + m->inst = inst; + + struct xrt_plugin_stereo_camera_info infos[XRT_STEREO_CAMERA_MAX_CAMERAS]; + memset(infos, 0, sizeof(infos)); + for (uint32_t i = 0; i < XRT_STEREO_CAMERA_MAX_CAMERAS; i++) { + infos[i].struct_size = (uint32_t)sizeof(infos[i]); + } + uint32_t n = iface->stereo_camera_enumerate(inst, XRT_STEREO_CAMERA_MAX_CAMERAS, infos); + if (n > XRT_STEREO_CAMERA_MAX_CAMERAS) { + n = XRT_STEREO_CAMERA_MAX_CAMERAS; + } + for (uint32_t i = 0; i < n; i++) { + struct scam_camera *cam = add_camera(m, XRT_STEREO_CAMERA_SOURCE_PLUGIN, i, &infos[i]); + if (cam == NULL) { + continue; + } + if (cam->info.flags & XRT_PLUGIN_STEREO_CAMERA_CALIBRATED) { + cam->calib.struct_size = (uint32_t)sizeof(cam->calib); + if (iface->stereo_camera_get_calibration(inst, i, &cam->calib) == XRT_SUCCESS) { + cam->have_calib = true; + } else { + U_LOG_W( + "stereo camera: \"%s\" claims CALIBRATED but returned no calibration — RAW only", + cam->info.display_name); + cam->info.flags &= ~XRT_PLUGIN_STEREO_CAMERA_CALIBRATED; + } + } + finish_camera(cam); + } +} + +/*! + * Amendment 4: the service's own UVC side-by-side source. Opt-in only — with + * no config (stereo-cameras.json / DXR_STEREO_CAMERA_UVC_CONFIG) nothing is + * enumerated and no device is touched. Never claims a device a plug-in camera + * names as its platform_device_hint (the tracker camera is the plug-in's). + */ +static void +add_uvc_cameras(struct ipc_server_stereo_camera *m, const struct ipc_server_stereo_camera_options *opts) +{ + struct u_stereo_uvc_config cfg; + if (opts != NULL && opts->uvc_config != NULL) { + cfg = *opts->uvc_config; + } else { + char path[1024], err[256]; + enum u_stereo_uvc_config_load_result r = + u_stereo_uvc_config_load(&cfg, path, sizeof(path), err, sizeof(err)); + if (r == U_STEREO_UVC_CONFIG_INVALID) { + U_LOG_W("stereo camera (uvc): config \"%s\" is INVALID (%s) — no UVC camera", path, err); + return; + } + if (r != U_STEREO_UVC_CONFIG_OK) { + U_LOG_I("stereo camera (uvc): no UVC stereo camera configured (%s: %s)", path[0] ? path : "-", + err); + return; + } + U_LOG_W("stereo camera (uvc): config \"%s\": %u entr%s%s", path, cfg.count, + cfg.count == 1 ? "y" : "ies", + cfg.fake ? ", FAKE synthetic device (no real camera is considered)" : ""); + } + if (cfg.count == 0) { + return; + } + const char *hints[XRT_STEREO_CAMERA_MAX_CAMERAS]; + uint32_t nh = 0; + for (uint32_t i = 0; i < m->camera_count; i++) { + if (m->cams[i].info.platform_device_hint[0] != '\0') { + hints[nh++] = m->cams[i].info.platform_device_hint; + } + } + struct u_stereo_uvc_backend os_backend; + const struct u_stereo_uvc_backend *backend = NULL; + if (opts != NULL && opts->uvc_backend != NULL) { + backend = opts->uvc_backend; + } else if (os_uvc_capture_backend(&os_backend)) { + backend = &os_backend; + } + m->uvc = u_stereo_uvc_create(&cfg, backend, hints, nh); + + struct xrt_plugin_stereo_camera_info infos[XRT_STEREO_CAMERA_MAX_CAMERAS]; + memset(infos, 0, sizeof(infos)); + for (uint32_t i = 0; i < XRT_STEREO_CAMERA_MAX_CAMERAS; i++) { + infos[i].struct_size = (uint32_t)sizeof(infos[i]); + } + uint32_t n = u_stereo_uvc_enumerate(m->uvc, XRT_STEREO_CAMERA_MAX_CAMERAS, infos); + n = n > XRT_STEREO_CAMERA_MAX_CAMERAS ? XRT_STEREO_CAMERA_MAX_CAMERAS : n; + for (uint32_t i = 0; i < n; i++) { + struct scam_camera *cam = add_camera(m, XRT_STEREO_CAMERA_SOURCE_UVC, i, &infos[i]); + if (cam == NULL) { + continue; + } + cam->calib.struct_size = (uint32_t)sizeof(cam->calib); + bool nominal = true; + if (u_stereo_uvc_get_calibration(m->uvc, i, &cam->calib, &nominal, &cam->baseline_known, + &cam->hfov_known) == XRT_SUCCESS) { + cam->have_calib = true; + cam->calib_nominal = nominal; + } + finish_camera(cam); + } +} + struct ipc_server_stereo_camera * ipc_server_stereo_camera_create(struct xrt_instance *xinst) +{ + return ipc_server_stereo_camera_create_ex(xinst, NULL); +} + +struct ipc_server_stereo_camera * +ipc_server_stereo_camera_create_ex(struct xrt_instance *xinst, const struct ipc_server_stereo_camera_options *opts) { struct ipc_server_stereo_camera *m = U_TYPED_CALLOC(struct ipc_server_stereo_camera); os_mutex_init(&m->lock); @@ -1586,7 +1953,19 @@ ipc_server_stereo_camera_create(struct xrt_instance *xinst) // R3: the consent policy over the real store; the prompt arrives later // from the host (tray / menu bar) through set_prompt_provider. - u_camera_consent_init(&m->consent, u_camera_consent_store_default(), NULL, &env_ops, m); + const struct u_camera_consent_store_ops *store = u_camera_consent_store_default(); + void *store_ctx = NULL; + const struct u_camera_consent_env_ops *env = &env_ops; + void *env_ctx = m; + if (opts != NULL && opts->consent_store != NULL) { + store = opts->consent_store; + store_ctx = opts->consent_store_ctx; + } + if (opts != NULL && opts->consent_env != NULL) { + env = opts->consent_env; + env_ctx = opts->consent_env_ctx; + } + u_camera_consent_init(&m->consent, store, store_ctx, env, env_ctx); m->consent.kill_switch = !debug_get_bool_option_stereo_camera_enabled(); m->consent.dev_override = debug_get_bool_option_stereo_camera_dev_allow(); m->consent.prompt_enabled = debug_get_bool_option_stereo_camera_prompt(); @@ -1616,79 +1995,14 @@ ipc_server_stereo_camera_create(struct xrt_instance *xinst) U_LOG_W("stereo camera: camera sharing is OFF (user toggle) — zero cameras until it is switched on"); } if (!m->consent.prompt_enabled) { - U_LOG_W("stereo camera: consent prompt disabled (DXR_STEREO_CAMERA_PROMPT=0) — an app without a stored " - "or delegated consent is refused"); - } - const struct xrt_plugin_iface *iface = NULL; - struct xrt_plugin_instance *inst = NULL; - if (xinst == NULL || xinst->get_active_plugin == NULL || !xinst->get_active_plugin(xinst, &iface, &inst) || - !xrt_plugin_iface_has_stereo_camera(iface)) { - U_LOG_I("stereo camera: the active plug-in provides no camera source"); - return m; - } - m->iface = iface; - m->inst = inst; - - struct xrt_plugin_stereo_camera_info infos[XRT_STEREO_CAMERA_MAX_CAMERAS]; - memset(infos, 0, sizeof(infos)); - for (uint32_t i = 0; i < XRT_STEREO_CAMERA_MAX_CAMERAS; i++) { - infos[i].struct_size = (uint32_t)sizeof(infos[i]); - } - uint32_t n = iface->stereo_camera_enumerate(inst, XRT_STEREO_CAMERA_MAX_CAMERAS, infos); - if (n > XRT_STEREO_CAMERA_MAX_CAMERAS) { - n = XRT_STEREO_CAMERA_MAX_CAMERAS; - } - for (uint32_t i = 0; i < n; i++) { - struct scam_camera *cam = &m->cams[m->camera_count]; - struct xrt_plugin_stereo_camera_info *info = &infos[i]; - info->display_name[sizeof(info->display_name) - 1] = '\0'; - info->device_identity[sizeof(info->device_identity) - 1] = '\0'; - info->platform_device_hint[sizeof(info->platform_device_hint) - 1] = '\0'; - if (info->eye_width == 0 || info->eye_height == 0 || (info->eye_width & 1u) || - (info->eye_height & 1u) || info->native_format < XRT_PLUGIN_STEREO_CAMERA_FORMAT_GRAY8 || - info->native_format > XRT_PLUGIN_STEREO_CAMERA_FORMAT_BGRA8) { - U_LOG_W( - "stereo camera: plug-in camera %u has a malformed description (%ux%u, format %u) — " - "skipped", - i, info->eye_width, info->eye_height, info->native_format); - continue; - } - cam->index = i; - cam->camera_id = (uint64_t)m->camera_count + 1; // never reused in this service lifetime - cam->info = *info; - cam->mgr = m; - cam->state = XRT_STEREO_CAMERA_STATE_AVAILABLE; - os_cond_init(&cam->cond); - if (info->flags & XRT_PLUGIN_STEREO_CAMERA_CALIBRATED) { - cam->calib.struct_size = (uint32_t)sizeof(cam->calib); - if (iface->stereo_camera_get_calibration(inst, i, &cam->calib) == XRT_SUCCESS) { - cam->have_calib = true; - } else { - U_LOG_W( - "stereo camera: \"%s\" claims CALIBRATED but returned no calibration — RAW only", - info->display_name); - cam->info.flags &= ~XRT_PLUGIN_STEREO_CAMERA_CALIBRATED; - } - } - if (cam->have_calib && (cam->info.flags & XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED) == 0) { - cam->rect = rectifier_create(cam); - } - if (!(info->max_frame_rate >= 0.0f && info->max_frame_rate <= 1000.0f)) { - cam->info.max_frame_rate = 0.0f; // NaN / negative / absurd = unknown - } - char rate[48]; - if (cam->info.max_frame_rate > 0.0f) { - snprintf(rate, sizeof(rate), "advertised %.1f Hz", cam->info.max_frame_rate); - } else { - snprintf(rate, sizeof(rate), "rate unknown"); - } U_LOG_W( - "stereo camera %llu: \"%s\" %ux%u per eye, %s (measured on first open), flags 0x%x, native " - "format %u", - (unsigned long long)cam->camera_id, info->display_name, info->eye_width, info->eye_height, rate, - info->flags, info->native_format); - m->camera_count++; + "stereo camera: consent prompt disabled (DXR_STEREO_CAMERA_PROMPT=0) — an app without a stored " + "or delegated consent is refused"); } + // Plug-in cameras first: their platform_device_hints keep the UVC source + // off the devices they own. + add_plugin_cameras(m, xinst); + add_uvc_cameras(m, opts); return m; } @@ -1719,6 +2033,8 @@ ipc_server_stereo_camera_destroy(struct ipc_server_stereo_camera **mgr_ptr) stream_free_storage(&m->streams[i]); } } + // Every camera thread has closed its source above: the UVC source can go. + u_stereo_uvc_destroy(&m->uvc); if (g_mgr == m) { g_mgr = NULL; } @@ -1860,12 +2176,20 @@ ipc_handle_stereo_camera_get_properties(volatile struct ipc_client_state *ics, out_props->eye_width = cam->info.eye_width; out_props->eye_height = cam->info.eye_height; out_props->max_frame_rate = source_rate_hz(cam); + out_props->source = cam->source; if (cam->have_calib) { - out_props->baseline_mm = (float)baseline_mm(&cam->calib); // Spec §3: per eye, RECTIFIED (the service's rectified focal when it // rectifies; the plug-in's own for a natively rectified source). - double fx = cam->rect != NULL ? cam->rect->geo.f : cam->calib.k[0][0]; - if (fx > 0.0) { + double fx = cam->calib.k[0][0], cx = 0.0, cy = 0.0; + if (cam->rect != NULL) { + rectified_intrinsics(cam->rect, &fx, &cx, &cy); + } + // Amendment 4: a NOMINAL model reports only what the user configured; + // a default it fell back to is not a fact about the device (0 = unknown). + if (!cam->calib_nominal || cam->baseline_known) { + out_props->baseline_mm = (float)baseline_mm(&cam->calib); + } + if (fx > 0.0 && (!cam->calib_nominal || cam->hfov_known)) { out_props->horizontal_fov_deg = (float)(2.0 * atan(cam->info.eye_width / (2.0 * fx)) * 180.0 / 3.14159265358979323846); } @@ -1923,6 +2247,11 @@ ipc_handle_stereo_camera_get_calibration(volatile struct ipc_client_state *ics, if (xret == XRT_SUCCESS && !cam->have_calib) { xret = XRT_ERROR_FEATURE_NOT_SUPPORTED; } + // Amendment 4: a nominal model is not a RAW calibration of the device — + // only the rectified numbers (which describe the frames delivered) are given. + if (xret == XRT_SUCCESS && cam->calib_nominal && output == XRT_STEREO_CAMERA_OUTPUT_RAW) { + xret = XRT_ERROR_FEATURE_NOT_SUPPORTED; + } if (xret == XRT_SUCCESS && output == XRT_STEREO_CAMERA_OUTPUT_RECTIFIED && !camera_can_rectify(cam)) { xret = XRT_ERROR_FEATURE_NOT_SUPPORTED; } @@ -1936,15 +2265,17 @@ ipc_handle_stereo_camera_get_calibration(volatile struct ipc_client_state *ics, // infinity), no distortion, and the right camera a pure +x translation // of the baseline (OpenCV P1/P2 with P2[0][3] = f * Tx, Tx = -baseline). const struct u_stereo_rectify_result *g = &cam->rect->geo; + double f = 0.0, cx = 0.0, cy = 0.0; + rectified_intrinsics(cam->rect, &f, &cx, &cy); // a nominal pass-through: no zoom out_calib->output = output; for (int e = 0; e < 2; e++) { struct xrt_stereo_camera_intrinsics *in = &out_calib->eye[e]; in->width = g->width; in->height = g->height; - in->fx = (float)g->f; - in->fy = (float)g->f; - in->cx = (float)g->cx; - in->cy = (float)g->cy; + in->fx = (float)f; + in->fy = (float)f; + in->cx = (float)cx; + in->cy = (float)cy; in->model = XRT_PLUGIN_STEREO_CAMERA_DISTORTION_NONE; } out_calib->orientation[3] = 1.0f; // identity @@ -2031,8 +2362,12 @@ ipc_handle_stereo_camera_stream_create(volatile struct ipc_client_state *ics, os_mutex_unlock(&m->lock); return m->sharing_enabled ? XRT_ERROR_INPUT_UNSUPPORTED : XRT_ERROR_STEREO_CAMERA_DISABLED; } + // RECTIFIED needs a calibration (CALIBRATED / NATIVELY_RECTIFIED) — or, + // Amendment 4, a nominal model the service rectifies (an uncalibrated UVC + // pair: frames as delivered + the online row alignment). bool calibrated = (cam->info.flags & - (XRT_PLUGIN_STEREO_CAMERA_CALIBRATED | XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED)) != 0; + (XRT_PLUGIN_STEREO_CAMERA_CALIBRATED | XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED)) != 0 || + (cam->calib_nominal && cam->rect != NULL); if (req->output == XRT_STEREO_CAMERA_OUTPUT_RECTIFIED && !calibrated) { os_mutex_unlock(&m->lock); return XRT_ERROR_INPUT_UNSUPPORTED; diff --git a/src/xrt/ipc/server/ipc_server_stereo_camera.h b/src/xrt/ipc/server/ipc_server_stereo_camera.h index 8a72a18983..30d1d0b50b 100644 --- a/src/xrt/ipc/server/ipc_server_stereo_camera.h +++ b/src/xrt/ipc/server/ipc_server_stereo_camera.h @@ -4,9 +4,10 @@ * @file * @brief XR_DXR_stereo_camera (ADR-043): the service's camera manager. * - * The service is the single owner of each plug-in-provided stereo camera: one - * runtime-owned thread per open camera pulls frames from the plug-in's - * wait_frame slot, and fans each frame out to every started, authorised stream + * The service is the single owner of each stereo camera — plug-in-provided, or + * (Amendment 4) an opted-in plain UVC side-by-side webcam read by the service's + * own vendor-neutral source: one runtime-owned thread per open camera pulls + * frames from the source's wait_frame, and fans each frame out to every started, authorised stream * — per stream a 3-slot latest-wins shared-memory ring (u_stereo_camera_ring), * format conversion, decimation, and a per-stream wake handle. The source is * opened on the first start and closed a linger (2 s) after the last stop. @@ -22,6 +23,7 @@ #include "xrt/xrt_instance.h" #include "util/u_camera_consent.h" +#include "util/u_stereo_uvc.h" #include #include @@ -35,12 +37,36 @@ struct ipc_client_state; /*! * Create the manager over the active plug-in of @p xinst (the service's native - * instance). Always returns a manager; with no plug-in, no camera slots, or the - * kill switch (DXR_STEREO_CAMERA=0) it simply exposes zero cameras. + * instance) plus the service's own UVC side-by-side source (ADR-043 Amendment + * 4; opt-in by the per-user stereo-cameras.json, util/u_stereo_uvc.h). Always + * returns a manager; with no camera source or the kill switch + * (DXR_STEREO_CAMERA=0) it simply exposes zero cameras. */ struct ipc_server_stereo_camera * ipc_server_stereo_camera_create(struct xrt_instance *xinst); +/*! + * What @ref ipc_server_stereo_camera_create_ex may replace — the seams a test + * drives the whole manager through (tests_stereo_camera_manager). Every NULL + * field means "the service's real one". + */ +struct ipc_server_stereo_camera_options +{ + //! Consent store (default: the per-user store, u_camera_consent_store_default()). + const struct u_camera_consent_store_ops *consent_store; + void *consent_store_ctx; + //! Consent environment (default: OS camera switch, tray prompt, signer checks). + const struct u_camera_consent_env_ops *consent_env; + void *consent_env_ctx; + //! UVC source config (default: u_stereo_uvc_config_load()). + const struct u_stereo_uvc_config *uvc_config; + //! UVC capture backend (default: os_uvc_capture_backend()). + const struct u_stereo_uvc_backend *uvc_backend; +}; + +struct ipc_server_stereo_camera * +ipc_server_stereo_camera_create_ex(struct xrt_instance *xinst, const struct ipc_server_stereo_camera_options *opts); + //! Stop every camera thread, close every source, free every stream. void ipc_server_stereo_camera_destroy(struct ipc_server_stereo_camera **mgr_ptr); diff --git a/src/xrt/targets/cli/CMakeLists.txt b/src/xrt/targets/cli/CMakeLists.txt index 844ff2dc0e..913c86aded 100644 --- a/src/xrt/targets/cli/CMakeLists.txt +++ b/src/xrt/targets/cli/CMakeLists.txt @@ -75,6 +75,8 @@ target_link_libraries( xrt-interfaces aux_os aux_os_ble + # ADR-043 Amendment 4: `camera uvc-devices` lists capture devices + the UVC config. + aux_os_uvc aux_util aux_math cli_query diff --git a/src/xrt/targets/cli/cli_cmd_camera.c b/src/xrt/targets/cli/cli_cmd_camera.c index 7c914cc357..2f95dc12f7 100644 --- a/src/xrt/targets/cli/cli_cmd_camera.c +++ b/src/xrt/targets/cli/cli_cmd_camera.c @@ -28,6 +28,18 @@ * camera stop-all the user kill switch (IPC) * camera sharing on|off the persistent sharing toggle (IPC) * camera fake-lock on|off simulate the OS session lock (IPC, diagnostics) + * camera uvc-devices [--modes] ADR-043 Amendment 4, LOCAL: the UVC stereo webcam + * config (stereo-cameras.json) + the capture devices + * present, each marked with the entry claiming it; + * --modes lists the modes of CLAIMED devices only; + * --decoder: the MJPEG decode path (hardware MFT + * + adapter, or software) — touches no camera + * camera uvc-decode-bench [--layout sbs-half|sbs-full] [--eye WxH] [--frames N] [--verbose] + * a media file (4K MJPEG AVI) through the identical + * webcam decode / readback / split path, no camera + * + * `list` prints each camera's source: "plugin" (the display plug-in's slots) or + * "uvc" (the service's own side-by-side webcam source; Amendment 4). * * `list` / `calib` / `probe` connect as the CAMERA_CONSUMER class (ADR-043 R3) * — the same class a browser's video-capture utility declares — so the CLI @@ -78,6 +90,8 @@ #include "os/os_time.h" #include "util/u_logging.h" #include "util/u_stereo_camera.h" +#include "util/u_stereo_uvc.h" +#include "os/os_uvc_capture.h" #include "client/ipc_client_connection.h" #include "client/ipc_client.h" @@ -253,19 +267,32 @@ cmd_list(struct ipc_connection *ipc_c, bool json) } char flags[160]; flags_str(p.flags, flags, sizeof(flags)); + // Amendment 4: which source the service reads it from ("plugin" / "uvc"). + const char *source = p.source == XRT_STEREO_CAMERA_SOURCE_UVC ? "uvc" : "plugin"; + // A device id (Windows symbolic link) carries backslashes: JSON-escape them. + char hint[2 * sizeof(p.platform_device_hint)]; + size_t hn = 0; + for (const char *c = p.platform_device_hint; *c != '\0' && hn + 2 < sizeof(hint); c++) { + if (*c == '\\' || *c == '"') { + hint[hn++] = '\\'; + } + hint[hn++] = *c; + } + hint[hn] = '\0'; if (json) { printf( - "%s{\"id\": %llu, \"name\": \"%s\", \"persistentId\": \"%s\", \"state\": \"%s\", " + "%s{\"id\": %llu, \"name\": \"%s\", \"source\": \"%s\", \"persistentId\": \"%s\", " + "\"state\": \"%s\", " "\"flags\": \"%s\", \"viewCount\": %u, \"eyeWidth\": %u, \"eyeHeight\": %u, " "\"maxFrameRate\": %.2f, \"baselineMm\": %.2f, \"horizontalFovDeg\": %.2f, " "\"formats\": \"0x%llx\", \"transports\": \"0x%llx\", \"platformDeviceHint\": \"%s\"}", - i ? ", " : "", (unsigned long long)p.camera_id, p.display_name, p.persistent_id, + i ? ", " : "", (unsigned long long)p.camera_id, p.display_name, source, p.persistent_id, state_str(p.state), flags, p.view_count, p.eye_width, p.eye_height, p.max_frame_rate, p.baseline_mm, p.horizontal_fov_deg, (unsigned long long)p.supported_formats, - (unsigned long long)p.supported_transports, p.platform_device_hint); + (unsigned long long)p.supported_transports, hint); } else { - printf(" [%llu] \"%s\" %s\n", (unsigned long long)p.camera_id, p.display_name, - state_str(p.state)); + printf(" [%llu] \"%s\" %s (source: %s)\n", (unsigned long long)p.camera_id, p.display_name, + state_str(p.state), source); printf(" persistentId %s\n", p.persistent_id); printf(" flags %s\n", flags); char rate[64]; @@ -276,8 +303,16 @@ cmd_list(struct ipc_connection *ipc_c, bool json) } printf(" eye %ux%u (SBS %ux%u), %u views, %s\n", p.eye_width, p.eye_height, 2 * p.eye_width, p.eye_height, p.view_count, rate); - printf(" baseline %.2f mm, HFOV %.2f deg per eye\n", p.baseline_mm, - p.horizontal_fov_deg); + if (p.source == XRT_STEREO_CAMERA_SOURCE_UVC && + (p.flags & XRT_PLUGIN_STEREO_CAMERA_CALIBRATED) == 0) { + printf( + " baseline %.2f mm, HFOV %.2f deg per eye (NOMINAL, from the config; " + "0 = not configured)\n", + p.baseline_mm, p.horizontal_fov_deg); + } else { + printf(" baseline %.2f mm, HFOV %.2f deg per eye\n", p.baseline_mm, + p.horizontal_fov_deg); + } printf(" formats 0x%llx transports 0x%llx platform hint \"%s\"\n", (unsigned long long)p.supported_formats, (unsigned long long)p.supported_transports, p.platform_device_hint); @@ -1306,6 +1341,297 @@ on_off_arg(int argc, const char **argv, int idx, uint32_t *out) return false; } +/* + * + * uvc-devices (ADR-043 Amendment 4): local, no service. + * + */ + +static const char * +uvc_layout_str(uint32_t l) +{ + return l == U_STEREO_UVC_LAYOUT_SBS_HALF ? "sbs-half" : "sbs-full"; +} + +/*! + * `camera uvc-devices [--modes]`: the UVC stereo camera config as the service + * would read it (path, parse result, entries) and the capture devices present, + * each marked with the entry that would claim it. Enumeration activates no + * device. --modes lists the capture modes of the MATCHED devices only (that + * instantiates the device object, never starts it): an unmatched device — the + * display's own tracker camera, say — is never touched. + */ +static int +cmd_uvc_devices(int argc, const char **argv) +{ + bool modes = cli_has_flag(argc, argv, "--modes"); + if (cli_has_flag(argc, argv, "--decoder")) { + // The decode path the service's open() would take — touches no camera. + static char report[8192]; + os_uvc_capture_decoder_report(report, sizeof(report)); + printf("UVC MJPEG decode path:\n%s", report); + printf( + "(the service log line \"uvc capture (Media Foundation): started ... decoder ...\" confirms " + "the decoder MF actually inserted, per open)\n\n"); + } + struct u_stereo_uvc_config cfg; + char path[1024], err[256] = {0}; + enum u_stereo_uvc_config_load_result r = u_stereo_uvc_config_load(&cfg, path, sizeof(path), err, sizeof(err)); + printf("UVC stereo camera config: %s\n", path[0] ? path : "(no per-user config directory)"); + switch (r) { + case U_STEREO_UVC_CONFIG_OK: + printf(" OK — %u entr%s%s\n", cfg.count, cfg.count == 1 ? "y" : "ies", + cfg.fake ? ", FAKE synthetic device (replaces real capture)" : ""); + break; + case U_STEREO_UVC_CONFIG_INVALID: printf(" INVALID — %s (the service claims nothing)\n", err); break; + default: printf(" none — %s (the service claims nothing; this is the default)\n", err); break; + } + for (uint32_t e = 0; e < cfg.count; e++) { + const struct u_stereo_uvc_entry *en = &cfg.entries[e]; + char mode[48] = "auto", eye[32] = "auto"; + if (en->mode_width > 0) { + snprintf(mode, sizeof(mode), "%ux%u@%.1f", en->mode_width, en->mode_height, en->mode_fps); + } + if (en->eye_width > 0) { + snprintf(eye, sizeof(eye), "%ux%u", en->eye_width, en->eye_height); + } + printf(" entry %u: match \"%s\"", e, en->name_contains); + if (en->match_vid_pid) { + printf(" vid_pid %04x:%04x", en->vid, en->pid); + } + printf(", layout %s, eyes %s, mode %s, eye_size %s, baseline %.1f mm, hfov %.1f deg%s%s\n", + uvc_layout_str(en->layout), en->swap_eyes ? "rl" : "lr", mode, eye, en->baseline_mm, + en->hfov_deg, en->calibration[0] ? ", calibration " : " (nominal model)", en->calibration); + } + + struct u_stereo_uvc_backend backend; + if (cfg.fake) { + u_stereo_uvc_fake_backend_init(&backend, &cfg.fake_params); + } else if (!os_uvc_capture_backend(&backend)) { + printf("capture backend: none on this platform (Linux V4L2 is a TODO)\n"); + return 0; + } + struct u_stereo_uvc_device devs[U_STEREO_UVC_MAX_DEVICES]; + memset(devs, 0, sizeof(devs)); + uint32_t n = backend.enumerate(backend.ctx, devs, U_STEREO_UVC_MAX_DEVICES); + n = n > U_STEREO_UVC_MAX_DEVICES ? U_STEREO_UVC_MAX_DEVICES : n; + printf("capture backend: %s, %u device(s)\n", backend.name, n); + // The service's own rule (u_stereo_uvc_assign), minus the plug-in exclusion + // the CLI cannot see without the service. + int32_t assigned[U_STEREO_UVC_MAX_ENTRIES]; + u_stereo_uvc_assign(&cfg, devs, n, NULL, assigned); + for (uint32_t e = 0; e < cfg.count; e++) { + if (assigned[e] == U_STEREO_UVC_ASSIGN_AMBIGUOUS) { + printf( + " entry %u: AMBIGUOUS — \"%s\" matches several devices; the service claims none for it " + "(make \"match\" more specific or add \"vid_pid\")\n", + e, cfg.entries[e].name_contains); + } else if (assigned[e] < 0) { + printf(" entry %u: matches no device present\n", e); + } + } + for (uint32_t i = 0; i < n; i++) { + struct u_stereo_uvc_device *d = &devs[i]; + int32_t entry = -1; + for (uint32_t e = 0; e < cfg.count; e++) { + if (assigned[e] == (int32_t)i) { + entry = (int32_t)e; + } + } + char vp[16] = "none"; + if (d->has_vid_pid) { + snprintf(vp, sizeof(vp), "%04x:%04x", d->vid, d->pid); + } + printf(" [%u] \"%s\" VID:PID %s %s\n", i, d->name, vp, + entry >= 0 ? "<- CLAIMED by the config" : "(not claimed: never touched)"); + printf(" id %s\n", d->id); + if (entry >= 0) { + const struct u_stereo_uvc_entry *en = &cfg.entries[entry]; + printf(" entry %d; the service skips it if a display plug-in reports it as its camera\n", + entry); + if (modes && backend.list_modes != NULL) { + struct u_stereo_uvc_mode ms[U_STEREO_UVC_MAX_MODES]; + uint32_t nm = backend.list_modes(backend.ctx, d->id, ms, U_STEREO_UVC_MAX_MODES); + printf(" %u mode(s) (NV12 / YUY2 / MJPEG):\n", nm); + for (uint32_t k = 0; k < nm; k++) { + uint32_t ew = 0, eh = 0; + bool ok = u_stereo_uvc_eye_size(en->layout, ms[k].width, ms[k].height, + en->eye_width, en->eye_height, &ew, &eh); + printf(" %ux%u@%.2f", ms[k].width, ms[k].height, ms[k].fps); + if (ok) { + printf(" -> %s %ux%u per eye (SBS %ux%u)", uvc_layout_str(en->layout), + ew, eh, 2 * ew, eh); + } + printf("\n"); + } + } + } + } + if (!modes && cfg.count > 0) { + printf("(add --modes to list the capture modes of the claimed device(s); the device is not started)\n"); + } + return 0; +} + +/*! + * `camera uvc-decode-bench [--layout sbs-half|sbs-full] [--eye WxH] [--frames N]`: + * decode a media FILE (e.g. a 4K MJPEG AVI) through the exact open / read + * path the service uses for a UVC webcam (same decoder policy, GPU decode + + * scale, staging readback, then the source's split), unpaced, and report the + * throughput. No camera, no service. DXR_STEREO_CAMERA_UVC_DECODER applies. + */ +static int +cmd_uvc_decode_bench(int argc, const char **argv) +{ + if (argc < 4 || argv[3][0] == '-') { + printf( + "usage: displayxr-cli camera uvc-decode-bench [--layout sbs-half|sbs-full] [--eye WxH] " + "[--frames N]\n"); + return 1; + } + uint32_t layout = U_STEREO_UVC_LAYOUT_SBS_HALF, want_w = 0, want_h = 0, max_frames = 600; + bool verbose = cli_has_flag(argc, argv, "--verbose"); + for (int i = 4; i < argc; i++) { + if (strcmp(argv[i], "--layout") == 0 && i + 1 < argc) { + layout = strcmp(argv[++i], "sbs-full") == 0 ? U_STEREO_UVC_LAYOUT_SBS_FULL + : U_STEREO_UVC_LAYOUT_SBS_HALF; + } else if (strcmp(argv[i], "--eye") == 0 && i + 1 < argc) { + if (sscanf(argv[++i], "%ux%u", &want_w, &want_h) != 2) { + want_w = want_h = 0; + } + } else if (strcmp(argv[i], "--frames") == 0 && i + 1 < argc) { + max_frames = (uint32_t)strtoul(argv[++i], NULL, 10); + } + } + struct u_stereo_uvc_backend b; + if (!os_uvc_capture_backend(&b)) { + printf("no capture backend on this platform\n"); + return 2; + } + char id[1100]; + snprintf(id, sizeof(id), "file:%s", argv[3]); + struct u_stereo_uvc_mode modes[U_STEREO_UVC_MAX_MODES]; + uint32_t nm = b.list_modes(b.ctx, id, modes, U_STEREO_UVC_MAX_MODES); + if (nm == 0) { + printf("cannot read \"%s\" (no NV12 / YUY2 / MJPEG video stream)\n", argv[3]); + return 2; + } + uint32_t ew = 0, eh = 0; + if (!u_stereo_uvc_eye_size(layout, modes[0].width, modes[0].height, want_w, want_h, &ew, &eh)) { + printf("%ux%u cannot be split\n", modes[0].width, modes[0].height); + return 2; + } + struct u_stereo_camera_planes lay; + if (!u_stereo_camera_layout(XRT_STEREO_CAMERA_FORMAT_NV12, 2 * ew, eh, &lay)) { + return 2; + } + uint8_t *dst = malloc((size_t)lay.size); + void *h = NULL; + if (dst == NULL || !b.open(b.ctx, id, &modes[0], 2 * ew, eh, &h)) { + printf("open failed (see the WARN lines above)\n"); + free(dst); + return 2; + } + printf("file %ux%u@%.2f -> %s, %ux%u per eye (SBS %ux%u)\n", modes[0].width, modes[0].height, modes[0].fps, + layout == U_STEREO_UVC_LAYOUT_SBS_FULL ? "sbs-full" : "sbs-half", ew, eh, 2 * ew, eh); + uint32_t frames = 0, raw_w = 0, raw_h = 0; + int64_t read_ns = 0, split_ns = 0, lat_ns = 0, t_first = 0, t_last = 0; + const int64_t t_start = os_monotonic_get_ns(); + while (frames < max_frames) { + struct u_stereo_uvc_raw_frame raw; + int64_t t0 = os_monotonic_get_ns(); + uint32_t r = b.read(h, 100ll * 1000 * 1000, &raw); + int64_t t1 = os_monotonic_get_ns(); + // A TIMEOUT is not the end (the GPU path's first read only primes its + // readback ring): stop after 2 s without a frame, or on ERROR (end of file). + if (r == U_STEREO_UVC_READ_TIMEOUT && t1 - (frames > 0 ? t_last : t_start) < 2000ll * 1000 * 1000) { + continue; + } + if (r != U_STEREO_UVC_READ_OK) { + if (verbose) { + printf(" read #%u -> %s after %.2f ms\n", frames, + r == U_STEREO_UVC_READ_TIMEOUT ? "TIMEOUT" : "ERROR", (double)(t1 - t0) * 1e-6); + } + break; // end of file (or a failure) + } + bool ok = u_stereo_uvc_split(&raw, false, ew, eh, dst, &lay); + int64_t t2 = os_monotonic_get_ns(); + if (verbose && frames < 8) { + printf(" frame %u: read %.2f ms, split %.2f ms, %ux%u\n", frames, (double)(t1 - t0) * 1e-6, + (double)(t2 - t1) * 1e-6, raw.width, raw.height); + } + if (!ok) { + printf("split failed on a %ux%u frame\n", raw.width, raw.height); + break; + } + if (frames == 0) { + t_first = t2; + raw_w = raw.width; + raw_h = raw.height; + } else { + read_ns += t1 - t0; + split_ns += t2 - t1; + // Arrival of the decoded sample in the reader callback -> split done. + lat_ns += raw.time_ns > 0 && t2 > raw.time_ns ? t2 - raw.time_ns : 0; + } + t_last = t2; + frames++; + } + b.close(h); + // Image-content check of the last delivered SBS frame: per eye, the mean + // luma of its outer 5 % columns against its centre half. A letterboxed / + // pillarboxed scale (black side bars) shows as dark edges — meaningful on + // a test file whose picture is bright edge to edge. + bool bars = false; + if (frames > 0) { + const uint32_t edge = ew / 20 > 0 ? ew / 20 : 1; + for (uint32_t e = 0; e < 2; e++) { + double sum[3] = {0, 0, 0}; + uint64_t cnt[3] = {0, 0, 0}; + for (uint32_t y = 0; y < eh; y++) { + const uint8_t *row = dst + lay.offset[0] + (size_t)y * lay.pitch[0] + (size_t)e * ew; + for (uint32_t x = 0; x < ew; x++) { + int k = x < edge ? 0 + : x >= ew - edge ? 2 + : (x >= ew / 4 && x < ew * 3 / 4) ? 1 + : -1; + if (k >= 0) { + sum[k] += row[x]; + cnt[k]++; + } + } + } + double l = sum[0] / (double)cnt[0], c = sum[1] / (double)cnt[1], r = sum[2] / (double)cnt[2]; + bool dark = c > 40.0 && (l < 0.5 * c || r < 0.5 * c); + bars |= dark; + printf("content check, %s eye: mean luma outer-left %.1f / centre %.1f / outer-right %.1f%s\n", + e == 0 ? "left" : "right", l, c, r, + dark ? " <- DARK SIDE BARS (letterboxed scale?)" : ""); + } + } + free(dst); + if (frames < 2) { + printf("decoded %u frame(s) — nothing to measure\n", frames); + return 3; + } + double secs = (double)(t_last - t_first) * 1e-9; + printf( + "delivered %u frames in %.2f s = %.1f frames/s (read rate; requests paced to the file's frame rate); " + "raw frame %ux%u%s\n", + frames, secs, (frames - 1) / secs, raw_w, raw_h, + raw_w == 2 * ew && raw_h == eh ? " (scaled by the decoder path)" : " (full size: CPU downscale)"); + printf( + "per frame: read (wait + readback) %.2f ms, split / resample %.2f ms; decoded-sample -> split-done " + "latency %.2f ms\n", + (double)read_ns / (frames - 1) * 1e-6, (double)split_ns / (frames - 1) * 1e-6, + (double)lat_ns / (frames - 1) * 1e-6); + if (bars) { + printf("FAIL: dark side bars in the delivered frame — the scale is not a plain resize\n"); + return 4; + } + return 0; +} + static void usage(void) { @@ -1323,6 +1649,11 @@ usage(void) "Deny;\n" " a user-writable needs --signer, spec 7.1.1)\n" " signer |--self path class + Authenticode signer (exit 3 = not validly signed)\n" + " uvc-devices [--modes] [--decoder] UVC stereo webcam config + capture devices (local;\n" + " --modes: capture modes of the CLAIMED devices only, never\n" + " started; --decoder: hardware/software MJPEG decode path)\n" + " uvc-decode-bench [--layout sbs-half|sbs-full] [--eye WxH] [--frames N]\n" + " decode a media file through the webcam path (no camera)\n" " status | stop-all | sharing on|off | fake-lock on|off (over IPC, DIAG)\n" "exit codes: 4 consent refused, 6 sharing off, 7 busy, 8 ended by the service, 5 rows misaligned\n"); } @@ -1344,6 +1675,12 @@ cli_cmd_camera(int argc, const char **argv) if (strcmp(sub, "signer") == 0) { return cmd_signer(argc, argv); } + if (strcmp(sub, "uvc-devices") == 0) { + return cmd_uvc_devices(argc, argv); + } + if (strcmp(sub, "uvc-decode-bench") == 0) { + return cmd_uvc_decode_bench(argc, argv); + } // Control ops: DIAG class over IPC. uint32_t onoff = 0; if (strcmp(sub, "status") == 0) { diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index ad00bc4951..c11f41245a 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -81,6 +81,7 @@ set(tests tests_camera_consent tests_stereo_rectify tests_stereo_vrefine + tests_stereo_uvc ) if(XRT_BUILD_DRIVER_QWERTY) list(APPEND tests tests_qwerty_camera_profile tests_qwerty_velocity) @@ -808,3 +809,34 @@ target_include_directories(tests_stereo_rectify PRIVATE ${PROJECT_SOURCE_DIR}/sr target_sources(tests_stereo_vrefine PRIVATE ${PROJECT_SOURCE_DIR}/src/xrt/drivers/sim_display/sim_display_stereo_camera_pattern.c) target_include_directories(tests_stereo_vrefine PRIVATE ${PROJECT_SOURCE_DIR}/src/xrt/drivers/sim_display ${CMAKE_CURRENT_SOURCE_DIR}/fixtures ${PROJECT_SOURCE_DIR}/src/external/stb) target_compile_definitions(tests_stereo_vrefine PRIVATE STEREO_FIXTURE_DIR="${CMAKE_CURRENT_SOURCE_DIR}/fixtures") + +# ADR-043 Amendment 4: the service's REAL camera manager end to end, driven +# through its IPC handlers with the UVC source on its fake backend (no device, +# no Media Foundation, no service process). Compiles the manager straight in +# rather than linking ipc_server, whose handler file pulls in every compositor; +# the glue stubs the two symbols it would have taken from there. +if(TARGET ipc_server) + add_executable( + tests_stereo_camera_manager + tests_stereo_camera_manager.cpp tests_stereo_camera_manager_glue.c tests_stereo_camera_manager_glue.h + ${PROJECT_SOURCE_DIR}/src/xrt/ipc/server/ipc_server_stereo_camera.c + ${PROJECT_SOURCE_DIR}/src/xrt/ipc/server/ipc_server_peer_creds.c + ) + target_link_libraries( + tests_stereo_camera_manager PRIVATE xrt-external-catch2 xrt-interfaces aux_util aux_os aux_os_uvc ipc_shared + ) + # ipc_server_generated.h (handler prototypes) is generated with ipc_server. + add_dependencies(tests_stereo_camera_manager ipc_server) + add_test(NAME tests_stereo_camera_manager COMMAND tests_stereo_camera_manager --success + --allow-running-no-tests + ) +endif() + +# ADR-043 Amendment 4, Windows: the Media Foundation UVC backend's scaled output +# (GPU and CPU video processor) is a plain resize, never letterboxed — through +# the real backend on a generated YUY2 AVI. SKIPs without Media Foundation. +if(WIN32 AND TARGET aux_os_uvc) + add_executable(tests_uvc_capture_mf tests_uvc_capture_mf.cpp) + target_link_libraries(tests_uvc_capture_mf PRIVATE xrt-external-catch2 aux_util aux_os aux_os_uvc) + add_test(NAME tests_uvc_capture_mf COMMAND tests_uvc_capture_mf --success --allow-running-no-tests) +endif() diff --git a/tests/tests_stereo_camera_manager.cpp b/tests/tests_stereo_camera_manager.cpp new file mode 100644 index 0000000000..38c54feb86 --- /dev/null +++ b/tests/tests_stereo_camera_manager.cpp @@ -0,0 +1,228 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera (ADR-043 Amendment 4) END TO END through the + * service's REAL camera manager (ipc_server_stereo_camera.c, its IPC + * handlers called directly) with the UVC source on its fake backend — + * no Media Foundation, no device, no service process, so it runs on + * every CI host: + * + * - enumerate: one camera, source UVC, no CALIBRATED, nominal baseline/HFOV + * reported only because the config gives them; + * - calibration: RAW refused (a nominal model is no device calibration), + * RECTIFIED = the nominal pinhole (pass-through, no zoom); + * - a RECTIFIED NV12 stream starts through the consent path (stored Allow), + * frames arrive in the read-only ring, the online refinement APPLIES a + * correction for the fake's 2.4 px vertical misalignment, the calibration + * generation moves, and the delivered frames' rows align (<= 0.3 px); + * - RAW streams are delivered RAW-flagged with the misalignment intact. + */ + +#include "catch_amalgamated.hpp" + +#include "tests_stereo_camera_manager_glue.h" + +#include "os/os_time.h" +#include "util/u_stereo_camera.h" + +#include +#include +#include +#include +#include +#include + +namespace { + +double +median(std::vector v) +{ + if (v.empty()) { + return 0.0; + } + std::sort(v.begin(), v.end()); + return v[v.size() / 2]; +} + +//! Signed median dy (right - left) over a block grid of an NV12/GRAY8 SBS luma plane. +double +row_offset(const uint8_t *y, uint32_t pitch, uint32_t ew, uint32_t eh, size_t *out_blocks) +{ + std::vector dy; + for (uint32_t by = 40; by + 32 + 40 < eh; by += 48) { + for (uint32_t bx = 56; bx + 32 + 8 < ew; bx += 48) { + float fx = 0.0f, fy = 0.0f; + if (u_stereo_camera_estimate_offset(y, pitch, ew, eh, bx, by, 32, 32, 48, 8, &fx, &fy)) { + dy.push_back(fy); + } + } + } + *out_blocks = dy.size(); + return median(dy); +} + +u_stereo_uvc_config +fake_config() +{ + u_stereo_uvc_config cfg; + char err[256] = {0}; + const char *json = R"({"fake": {"disparity": 0.04, "dy": 0.005, "fps": 60}, + "uvc": [{"match": "synthetic", "name": "Test SBS webcam", "mode": "1280x480@60", + "baseline_mm": 60, "hfov_deg": 70}]})"; + bool ok = u_stereo_uvc_config_parse(json, &cfg, err, sizeof(err)); + INFO(err); + REQUIRE(ok); + return cfg; +} + +} // namespace + +TEST_CASE("manager + UVC fake: enumerate and calibration", "[stereo_camera_manager]") +{ + u_stereo_uvc_config cfg = fake_config(); + scm_fixture *f = scm_create(&cfg); + REQUIRE(f != nullptr); + REQUIRE(scm_count(f) == 1); + xrt_stereo_camera_properties p; + REQUIRE(scm_properties(f, 0, &p) == XRT_SUCCESS); + CHECK(p.source == XRT_STEREO_CAMERA_SOURCE_UVC); + CHECK(std::string(p.display_name) == "Test SBS webcam"); + CHECK((p.flags & XRT_PLUGIN_STEREO_CAMERA_CALIBRATED) == 0); + CHECK((p.flags & XRT_PLUGIN_STEREO_CAMERA_NATIVELY_RECTIFIED) == 0); + CHECK(p.eye_width == 640); + CHECK(p.eye_height == 480); + CHECK(p.view_count == 2); + CHECK(p.baseline_mm == Catch::Approx(60.0f)); + CHECK(p.horizontal_fov_deg == Catch::Approx(70.0f).margin(0.01)); + CHECK(std::string(p.platform_device_hint) == "dxr-fake-uvc"); + CHECK(std::string(p.persistent_id).rfind("dxrcam-", 0) == 0); + + xrt_stereo_camera_calibration c; + CHECK(scm_calibration(f, p.camera_id, XRT_STEREO_CAMERA_OUTPUT_RAW, &c) == XRT_ERROR_FEATURE_NOT_SUPPORTED); + REQUIRE(scm_calibration(f, p.camera_id, XRT_STEREO_CAMERA_OUTPUT_RECTIFIED, &c) == XRT_SUCCESS); + const double f_nom = 320.0 / std::tan(35.0 * 3.14159265358979323846 / 180.0); + CHECK(c.eye[0].fx == Catch::Approx(f_nom).epsilon(1e-6)); // pass-through: no alpha = 0 zoom + CHECK(c.eye[1].fy == Catch::Approx(f_nom).epsilon(1e-6)); + CHECK(c.eye[0].cx == Catch::Approx(319.5)); + CHECK(c.eye[0].cy == Catch::Approx(239.5)); + CHECK(c.eye[0].model == XRT_PLUGIN_STEREO_CAMERA_DISTORTION_NONE); + CHECK(c.baseline_mm == Catch::Approx(60.0f)); + CHECK(c.position[0] == Catch::Approx(0.060f).margin(1e-5)); // right camera at +x, metres + CHECK(c.orientation[3] == 1.0f); + scm_destroy(f); +} + +TEST_CASE("manager + UVC fake: RECTIFIED stream converges, RAW stays raw", "[stereo_camera_manager]") +{ + u_stereo_uvc_config cfg = fake_config(); + scm_fixture *f = scm_create(&cfg); + REQUIRE(f != nullptr); + REQUIRE(scm_count(f) == 1); + xrt_stereo_camera_properties p; + REQUIRE(scm_properties(f, 0, &p) == XRT_SUCCESS); + + SECTION("RECTIFIED") + { + xrt_stereo_camera_stream_request req = {p.camera_id, XRT_STEREO_CAMERA_OUTPUT_RECTIFIED, + XRT_STEREO_CAMERA_FORMAT_NV12, + XRT_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY, 0.0f}; + uint64_t id = 0; + REQUIRE(scm_stream_create(f, &req, &id) == XRT_SUCCESS); + REQUIRE(scm_stream_start(f, id) == XRT_SUCCESS); + xrt_stereo_camera_stream_layout lay; + REQUIRE(scm_stream_map(f, id, &lay) == XRT_SUCCESS); + CHECK(lay.width == 1280); + CHECK(lay.height == 480); + CHECK(lay.output == XRT_STEREO_CAMERA_OUTPUT_RECTIFIED); + + // Run until the refinement applied a correction AND a frame of the new + // generation arrived (fast phase: a measurement every 250 ms), max 10 s. + const int64_t t_end = os_monotonic_get_ns() + 10ll * 1000 * 1000 * 1000; + uint32_t frames = 0, first_gen = UINT32_MAX, last_gen = 0; + double first_dy = 0.0, last_dy = 99.0; + size_t blocks = 0; + xrt_stereo_camera_stream_stats st; + std::memset(&st, 0, sizeof(st)); + while (os_monotonic_get_ns() < t_end) { + bool ready = false; + xrt_stereo_camera_frame_info fi; + const uint8_t *slot = nullptr; + REQUIRE(scm_acquire(f, id, &ready, &fi, &slot) == XRT_SUCCESS); + if (!ready) { + os_nanosleep(5 * 1000 * 1000); + continue; + } + REQUIRE(slot != nullptr); + CHECK(fi.output == XRT_STEREO_CAMERA_OUTPUT_RECTIFIED); + CHECK(fi.format == XRT_STEREO_CAMERA_FORMAT_NV12); + frames++; + if (first_gen == UINT32_MAX) { + first_gen = fi.calibration_generation; + first_dy = row_offset(slot + fi.plane_offset[0], fi.row_pitch[0], 640, 480, &blocks); + } + last_gen = fi.calibration_generation; + REQUIRE(scm_stats(f, id, &st) == XRT_SUCCESS); + if (st.refine_state == XRT_STEREO_CAMERA_REFINE_APPLIED && last_gen != first_gen) { + last_dy = row_offset(slot + fi.plane_offset[0], fi.row_pitch[0], 640, 480, &blocks); + if (std::fabs(last_dy) <= 0.3) { + break; + } + } + } + std::printf( + "[manager uvc] %u frames; refine state %u, %u update(s), applied a %+.3f px b %+.3f/100px; " + "generation %u -> %u; delivered dy %+.3f -> %+.3f px (%zu blocks); source %.1f Hz\n", + frames, st.refine_state, st.refine_updates, st.refine_offset_px, st.refine_slope_per_100px, + first_gen, last_gen, first_dy, last_dy, blocks, st.source_frame_rate); + CHECK(frames > 10); + CHECK(first_dy == Catch::Approx(2.4).margin(0.35)); // pass-through: the device's own misalignment + CHECK(st.refine_state == XRT_STEREO_CAMERA_REFINE_APPLIED); + CHECK(last_gen != first_gen); + CHECK(std::fabs(last_dy) <= 0.3); + + // The rectified calibration now describes the corrected maps. + xrt_stereo_camera_calibration c; + REQUIRE(scm_calibration(f, p.camera_id, XRT_STEREO_CAMERA_OUTPUT_RECTIFIED, &c) == XRT_SUCCESS); + CHECK(c.eye[0].fx > 0.0f); + CHECK(c.eye[0].fx == c.eye[1].fx); + CHECK(c.baseline_mm == Catch::Approx(60.0f).margin(0.5)); + CHECK(scm_stream_destroy(f, id) == XRT_SUCCESS); + } + SECTION("RAW") + { + xrt_stereo_camera_stream_request req = {p.camera_id, XRT_STEREO_CAMERA_OUTPUT_RAW, + XRT_STEREO_CAMERA_FORMAT_GRAY8, + XRT_STEREO_CAMERA_TRANSPORT_SHARED_MEMORY, 0.0f}; + uint64_t id = 0; + REQUIRE(scm_stream_create(f, &req, &id) == XRT_SUCCESS); + REQUIRE(scm_stream_start(f, id) == XRT_SUCCESS); + xrt_stereo_camera_stream_layout lay; + REQUIRE(scm_stream_map(f, id, &lay) == XRT_SUCCESS); + const int64_t t_end = os_monotonic_get_ns() + 5ll * 1000 * 1000 * 1000; + bool got = false; + while (!got && os_monotonic_get_ns() < t_end) { + bool ready = false; + xrt_stereo_camera_frame_info fi; + const uint8_t *slot = nullptr; + REQUIRE(scm_acquire(f, id, &ready, &fi, &slot) == XRT_SUCCESS); + if (!ready) { + os_nanosleep(5 * 1000 * 1000); + continue; + } + got = true; + CHECK(fi.output == XRT_STEREO_CAMERA_OUTPUT_RAW); + CHECK(fi.format == XRT_STEREO_CAMERA_FORMAT_GRAY8); + size_t blocks = 0; + double dy = row_offset(slot + fi.plane_offset[0], fi.row_pitch[0], 640, 480, &blocks); + CHECK(blocks > 20); + CHECK(dy == Catch::Approx(2.4).margin(0.35)); + } + CHECK(got); + xrt_stereo_camera_stream_stats st; + REQUIRE(scm_stats(f, id, &st) == XRT_SUCCESS); + CHECK(st.refine_state == XRT_STEREO_CAMERA_REFINE_NONE); + CHECK(scm_stream_destroy(f, id) == XRT_SUCCESS); + } + scm_destroy(f); +} diff --git a/tests/tests_stereo_camera_manager_glue.c b/tests/tests_stereo_camera_manager_glue.c new file mode 100644 index 0000000000..c1dc4b88e7 --- /dev/null +++ b/tests/tests_stereo_camera_manager_glue.c @@ -0,0 +1,269 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief C side of tests_stereo_camera_manager: a minimal ipc_server + + * ipc_client_state around the REAL camera manager + * (ipc_server_stereo_camera.c, compiled into the test), its IPC + * handlers called directly, an in-memory consent store, and the UVC + * source on its fake backend. No pipe, no service process. + */ + +#include "tests_stereo_camera_manager_glue.h" + +#include "server/ipc_server.h" +#include "server/ipc_server_stereo_camera.h" +#include "ipc_server_generated.h" + +#include "xrt/xrt_config_os.h" +#include "util/u_camera_consent.h" + +#include +#include + +#ifdef XRT_OS_WINDOWS +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#include +#else +#include +#include +#endif + +/* + * What ipc_server_stereo_camera.c needs from the rest of ipc_server (which + * this test does not link: the handler file drags in every compositor). + */ + +const char * +ipc_server_client_class_str(uint32_t client_class) +{ + (void)client_class; + return "TEST"; +} + +#ifdef XRT_OS_MACOS +bool +ipc_server_macos_pid_app_visible(long pid) +{ + (void)pid; + return true; +} +#endif + + +/* + * In-memory consent: every executable has a stored Allow, the OS switch is on. + */ + +static bool +store_get(void *ctx, const char *exe, enum u_camera_consent_stored *out) +{ + (void)ctx; + (void)exe; + *out = U_CAMERA_CONSENT_STORED_ALLOW; + return true; +} + +static bool +store_sharing(void *ctx) +{ + (void)ctx; + return true; +} + +static const struct u_camera_consent_store_ops k_store = { + .get = store_get, + .sharing_enabled = store_sharing, +}; + +static bool +env_os_allowed(void *ctx, const char *exe) +{ + (void)ctx; + (void)exe; + return true; +} + +static bool +env_user_writable(void *ctx, const char *exe) +{ + (void)ctx; + (void)exe; + return false; +} + +static const struct u_camera_consent_env_ops k_env = { + .os_camera_allowed = env_os_allowed, + .path_user_writable = env_user_writable, +}; + + +struct scm_fixture +{ + struct ipc_server server; + struct ipc_client_state ics; + struct ipc_server_stereo_camera *mgr; + struct u_stereo_uvc_config cfg; + // Read-only view of the last stream's ring. + const uint8_t *map; + uint64_t map_size; + uint64_t slot_stride; +#ifdef XRT_OS_WINDOWS + HANDLE section; +#else + int section; +#endif +}; + +struct scm_fixture * +scm_create(const struct u_stereo_uvc_config *cfg) +{ + struct scm_fixture *f = (struct scm_fixture *)calloc(1, sizeof(*f)); + if (f == NULL) { + return NULL; + } + f->cfg = *cfg; + struct ipc_server_stereo_camera_options o; + memset(&o, 0, sizeof(o)); + o.consent_store = &k_store; + o.consent_env = &k_env; + o.uvc_config = &f->cfg; + f->mgr = ipc_server_stereo_camera_create_ex(NULL, &o); + f->server.stereo_camera = f->mgr; + f->ics.server = &f->server; + f->ics.client_state.client_class = XRT_CLIENT_CLASS_CAMERA_CONSUMER; +#ifdef XRT_OS_WINDOWS + f->ics.peer_pid = (long)GetCurrentProcessId(); + f->section = NULL; +#else + f->ics.peer_pid = (long)getpid(); + f->section = -1; +#endif + return f; +} + +static void +unmap(struct scm_fixture *f) +{ + if (f->map != NULL) { +#ifdef XRT_OS_WINDOWS + UnmapViewOfFile((LPCVOID)f->map); +#else + munmap((void *)f->map, (size_t)f->map_size); +#endif + f->map = NULL; + } +#ifdef XRT_OS_WINDOWS + if (f->section != NULL) { + CloseHandle(f->section); + f->section = NULL; + } +#else + if (f->section >= 0) { + close(f->section); + f->section = -1; + } +#endif +} + +void +scm_destroy(struct scm_fixture *f) +{ + if (f == NULL) { + return; + } + unmap(f); + ipc_server_client_stereo_camera_release(&f->ics); + ipc_server_stereo_camera_destroy(&f->mgr); + free(f); +} + +uint32_t +scm_count(struct scm_fixture *f) +{ + uint32_t n = 0; + ipc_handle_stereo_camera_count(&f->ics, &n); + return n; +} + +xrt_result_t +scm_properties(struct scm_fixture *f, uint32_t index, struct xrt_stereo_camera_properties *out) +{ + return ipc_handle_stereo_camera_get_properties(&f->ics, index, out); +} + +xrt_result_t +scm_calibration(struct scm_fixture *f, uint64_t camera_id, uint32_t output, struct xrt_stereo_camera_calibration *out) +{ + return ipc_handle_stereo_camera_get_calibration(&f->ics, camera_id, output, out); +} + +xrt_result_t +scm_stream_create(struct scm_fixture *f, const struct xrt_stereo_camera_stream_request *req, uint64_t *out_id) +{ + return ipc_handle_stereo_camera_stream_create(&f->ics, req, out_id); +} + +xrt_result_t +scm_stream_start(struct scm_fixture *f, uint64_t id) +{ + return ipc_handle_stereo_camera_stream_start(&f->ics, id); +} + +xrt_result_t +scm_stream_map(struct scm_fixture *f, uint64_t id, struct xrt_stereo_camera_stream_layout *out_layout) +{ + unmap(f); + xrt_shmem_handle_t h = XRT_SHMEM_HANDLE_INVALID; + uint32_t count = 0; + xrt_result_t xret = ipc_handle_stereo_camera_stream_get_section(&f->ics, id, out_layout, 1, &h, &count); + if (xret != XRT_SUCCESS || count != 1) { + return xret != XRT_SUCCESS ? xret : XRT_ERROR_IPC_FAILURE; + } + // The handler hands out the manager's own read-only handle (the dispatch + // would duplicate it into the peer): map a duplicate, read-only. +#ifdef XRT_OS_WINDOWS + if (!DuplicateHandle(GetCurrentProcess(), h, GetCurrentProcess(), &f->section, 0, FALSE, + DUPLICATE_SAME_ACCESS)) { + return XRT_ERROR_IPC_FAILURE; + } + f->map = (const uint8_t *)MapViewOfFile(f->section, FILE_MAP_READ, 0, 0, (SIZE_T)out_layout->section_size); +#else + f->section = dup(h); + void *p = mmap(NULL, (size_t)out_layout->section_size, PROT_READ, MAP_SHARED, f->section, 0); + f->map = p == MAP_FAILED ? NULL : (const uint8_t *)p; +#endif + f->map_size = out_layout->section_size; + f->slot_stride = out_layout->slot_stride; + return f->map != NULL ? XRT_SUCCESS : XRT_ERROR_IPC_FAILURE; +} + +xrt_result_t +scm_acquire(struct scm_fixture *f, + uint64_t id, + bool *out_ready, + struct xrt_stereo_camera_frame_info *out_frame, + const uint8_t **out_slot) +{ + xrt_result_t xret = ipc_handle_stereo_camera_acquire(&f->ics, id, out_ready, out_frame); + *out_slot = NULL; + if (xret == XRT_SUCCESS && *out_ready && f->map != NULL) { + *out_slot = f->map + (uint64_t)out_frame->slot * f->slot_stride; + } + return xret; +} + +xrt_result_t +scm_stats(struct scm_fixture *f, uint64_t id, struct xrt_stereo_camera_stream_stats *out) +{ + return ipc_handle_stereo_camera_stream_stats(&f->ics, id, out); +} + +xrt_result_t +scm_stream_destroy(struct scm_fixture *f, uint64_t id) +{ + unmap(f); + return ipc_handle_stereo_camera_stream_destroy(&f->ics, id); +} diff --git a/tests/tests_stereo_camera_manager_glue.h b/tests/tests_stereo_camera_manager_glue.h new file mode 100644 index 0000000000..92d9fdebff --- /dev/null +++ b/tests/tests_stereo_camera_manager_glue.h @@ -0,0 +1,62 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief The C fixture tests_stereo_camera_manager drives the real camera + * manager through (see tests_stereo_camera_manager_glue.c). + */ + +#pragma once + +#include "xrt/xrt_stereo_camera.h" +#include "util/u_stereo_uvc.h" + +#ifdef __cplusplus +extern "C" { +#endif + +struct scm_fixture; + +//! A manager whose only source is the UVC source on @p cfg (use a "fake" config). +struct scm_fixture * +scm_create(const struct u_stereo_uvc_config *cfg); + +void +scm_destroy(struct scm_fixture *f); + +uint32_t +scm_count(struct scm_fixture *f); + +xrt_result_t +scm_properties(struct scm_fixture *f, uint32_t index, struct xrt_stereo_camera_properties *out); + +xrt_result_t +scm_calibration(struct scm_fixture *f, uint64_t camera_id, uint32_t output, struct xrt_stereo_camera_calibration *out); + +xrt_result_t +scm_stream_create(struct scm_fixture *f, const struct xrt_stereo_camera_stream_request *req, uint64_t *out_id); + +xrt_result_t +scm_stream_start(struct scm_fixture *f, uint64_t id); + +//! Map the started stream's ring read-only (as a consumer does). +xrt_result_t +scm_stream_map(struct scm_fixture *f, uint64_t id, struct xrt_stereo_camera_stream_layout *out_layout); + +//! @p out_slot points at the acquired frame's slot inside the mapped ring. +xrt_result_t +scm_acquire(struct scm_fixture *f, + uint64_t id, + bool *out_ready, + struct xrt_stereo_camera_frame_info *out_frame, + const uint8_t **out_slot); + +xrt_result_t +scm_stats(struct scm_fixture *f, uint64_t id, struct xrt_stereo_camera_stream_stats *out); + +xrt_result_t +scm_stream_destroy(struct scm_fixture *f, uint64_t id); + +#ifdef __cplusplus +} +#endif diff --git a/tests/tests_stereo_uvc.cpp b/tests/tests_stereo_uvc.cpp new file mode 100644 index 0000000000..1fe0b37e82 --- /dev/null +++ b/tests/tests_stereo_uvc.cpp @@ -0,0 +1,880 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief XR_DXR_stereo_camera (ADR-043 Amendment 4): the vendor-neutral UVC + * side-by-side stereo camera source (u_stereo_uvc) — every part that + * is not the OS capture call, on every OS: + * + * - config: the documented document parses; every mistake fails the whole + * document (a typo must never claim a different camera); fake knob; + * - matching: VID:PID from Windows symbolic links / "XXXX:YYYY", name + * substring (case-insensitive), both, never "match everything"; a device a + * plug-in names in its platform_device_hint is never claimed; + * - geometry: per-eye output size (full vs half SBS, the 1280 cap, explicit), + * the nominal pinhole, and that it is an exact parallel pair the rectifier + * maps as (almost) identity; + * - calibration files: OpenCV FileStorage YAML and JSON, rotation vectors, + * metres, fisheye, rejection of malformed input; + * - the SBS split: eye order, full vs half (an exact 2:1 row average), NV12 + * and YUY2 input, chroma; + * - the source end to end on the FAKE backend (enumerate -> calibration -> + * open -> wait_frame): the known disparity and vertical offset come back, + * in both layouts and both pixel formats, and "rl" swaps the halves; + * - CLOSED LOOP as the service runs it for an uncalibrated pair: frames + * passed through, the online refinement measures the fake's 2.4 px + * vertical misalignment, the corrected maps bring it to <= 0.25 px. + */ + +#include "catch_amalgamated.hpp" + +#include "util/u_stereo_camera.h" +#include "util/u_stereo_rectify.h" +#include "util/u_stereo_uvc.h" +#include "util/u_stereo_vrefine.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace { + +constexpr double k_pi = 3.14159265358979323846; + +u_stereo_uvc_config +parse_ok(const char *json) +{ + u_stereo_uvc_config cfg; + char err[256] = {0}; + bool ok = u_stereo_uvc_config_parse(json, &cfg, err, sizeof(err)); + INFO(err); + REQUIRE(ok); + return cfg; +} + +std::string +parse_err(const char *json) +{ + u_stereo_uvc_config cfg; + char err[256] = {0}; + CHECK_FALSE(u_stereo_uvc_config_parse(json, &cfg, err, sizeof(err))); + CHECK(cfg.count == 0); + return err; +} + +u_stereo_uvc_device +device(const char *name, const char *id) +{ + u_stereo_uvc_device d; + std::memset(&d, 0, sizeof(d)); + std::snprintf(d.name, sizeof(d.name), "%s", name); + std::snprintf(d.id, sizeof(d.id), "%s", id); + d.has_vid_pid = u_stereo_uvc_parse_vid_pid(d.id, &d.vid, &d.pid); + return d; +} + +double +median(std::vector v) +{ + if (v.empty()) { + return 0.0; + } + std::sort(v.begin(), v.end()); + return v[v.size() / 2]; +} + +//! Block offsets (dx, dy) of a GRAY8 / NV12-Y SBS image over a grid of blocks. +void +block_offsets(const uint8_t *y, + uint32_t pitch, + uint32_t ew, + uint32_t eh, + uint32_t max_d, + std::vector &dx, + std::vector &dy) +{ + dx.clear(); + dy.clear(); + const uint32_t b = 32; + for (uint32_t by = 40; by + b + 40 < eh; by += 48) { + for (uint32_t bx = max_d + 8; bx + b + 8 < ew; bx += 48) { + float fx = 0.0f, fy = 0.0f; + if (u_stereo_camera_estimate_offset(y, pitch, ew, eh, bx, by, b, b, max_d, 8, &fx, &fy)) { + dx.push_back(fx); + dy.push_back(fy); + } + } + } +} + +const char *k_full_doc = R"({ + "uvc": [{ + "match": "Eyes", + "vid_pid": "04f2:b71a", + "name": "Stereo webcam", + "layout": "sbs-half", + "eyes": "rl", + "mode": "3840x2160@60", + "eye_size": "1281x721", + "baseline_mm": 60, + "hfov_deg": 90, + "calibration": "C:/cal/eyes.yaml" + }] +})"; + +} // namespace + + +TEST_CASE("uvc config: the documented document parses", "[stereo_uvc]") +{ + u_stereo_uvc_config cfg = parse_ok(k_full_doc); + REQUIRE(cfg.count == 1); + const u_stereo_uvc_entry &e = cfg.entries[0]; + CHECK(std::string(e.name_contains) == "Eyes"); + CHECK(e.match_vid_pid); + CHECK(e.vid == 0x04f2); + CHECK(e.pid == 0xb71a); + CHECK(std::string(e.display_name) == "Stereo webcam"); + CHECK(e.layout == U_STEREO_UVC_LAYOUT_SBS_HALF); + CHECK(e.swap_eyes); + CHECK(e.mode_width == 3840); + CHECK(e.mode_height == 2160); + CHECK(e.mode_fps == Catch::Approx(60.0f)); + CHECK(e.eye_width == 1280); // rounded down to even (NV12) + CHECK(e.eye_height == 720); + CHECK(e.baseline_mm == Catch::Approx(60.0f)); + CHECK(e.hfov_deg == Catch::Approx(90.0f)); + CHECK(std::string(e.calibration) == "C:/cal/eyes.yaml"); + CHECK_FALSE(cfg.fake); + + // Defaults: full SBS, left lens left, everything auto, nothing configured. + u_stereo_uvc_config d = parse_ok(R"({"uvc": [{"vid_pid": "VID_1234&PID_ABCD"}]})"); + REQUIRE(d.count == 1); + CHECK(d.entries[0].layout == U_STEREO_UVC_LAYOUT_SBS_FULL); + CHECK_FALSE(d.entries[0].swap_eyes); + CHECK(d.entries[0].mode_width == 0); + CHECK(d.entries[0].baseline_mm == 0.0f); + CHECK(d.entries[0].vid == 0x1234); + CHECK(d.entries[0].pid == 0xabcd); + + // No list at all = nothing claimed (not an error). + u_stereo_uvc_config none = parse_ok("{}"); + CHECK(none.count == 0); + + // The fake knob, with and without parameters. + u_stereo_uvc_config f = parse_ok(R"({"fake": {"disparity": 0.05, "dy": 0.004, "pixel": "yuy2", "fps": 90}, + "uvc": [{"match": "synthetic"}]})"); + CHECK(f.fake); + CHECK(f.fake_params.disparity == Catch::Approx(0.05)); + CHECK(f.fake_params.dy == Catch::Approx(0.004)); + CHECK(f.fake_params.pixel == U_STEREO_UVC_PIXEL_YUY2); + CHECK(f.fake_params.fps == Catch::Approx(90.0f)); + CHECK(parse_ok(R"({"fake": true})").fake); +} + +TEST_CASE("uvc config: every mistake fails the whole document", "[stereo_uvc]") +{ + CHECK(parse_err("not json").find("JSON") != std::string::npos); + CHECK(parse_err(R"({"uvc": {}})").find("array") != std::string::npos); + CHECK(parse_err(R"({"uvc": [{"name": "x"}]})").find("match") != std::string::npos); // no selector + CHECK(parse_err(R"({"uvc": [{"match": ""}]})").find("match") != std::string::npos); + CHECK(parse_err(R"({"uvc": [{"vid_pid": "12:34"}]})").find("vid_pid") != std::string::npos); + CHECK(parse_err(R"({"uvc": [{"match": "a", "layout": "tab"}]})").find("layout") != std::string::npos); + CHECK(parse_err(R"({"uvc": [{"match": "a", "eyes": "left"}]})").find("eyes") != std::string::npos); + CHECK(parse_err(R"({"uvc": [{"match": "a", "mode": "1920"}]})").find("mode") != std::string::npos); + CHECK(parse_err(R"({"uvc": [{"match": "a", "mode": "1922x1080"}]})").find("mode") != std::string::npos); + CHECK(parse_err(R"({"uvc": [{"match": "a", "baseline_mm": -3}]})").find("baseline") != std::string::npos); + CHECK(parse_err(R"({"uvc": [{"match": "a", "hfov_deg": 200}]})").find("hfov") != std::string::npos); + CHECK(parse_err(R"({"uvc": [{"match": "a", "calibration": 3}]})").find("calibration") != std::string::npos); + CHECK(parse_err(R"({"fake": 3})").find("fake") != std::string::npos); + // A bad SECOND entry voids the first too. + CHECK(parse_err(R"({"uvc": [{"match": "a"}, {"match": "b", "eyes": "x"}]})").find("uvc[1]") != + std::string::npos); + CHECK(parse_err(R"({"uvc": [{"match": "a"}, {"match": "b"}, {"match": "c"}, {"match": "d"}, {"match": "e"}]})") + .find("too many") != std::string::npos); +} + +TEST_CASE("uvc matching: VID:PID, name, both; never everything", "[stereo_uvc]") +{ + uint16_t v = 0, p = 0; + CHECK(u_stereo_uvc_parse_vid_pid( + "\\\\?\\usb#vid_04f2&pid_b70c&mi_00#6&18588fc8&0&0000#{e5323777-f976-4f5b-9b55-b94699c46e44}\\global", &v, + &p)); + CHECK(v == 0x04f2); + CHECK(p == 0xb70c); + CHECK(u_stereo_uvc_parse_vid_pid("USB\\VID_1D6B&PID_0102&MI_00\\x", &v, &p)); + CHECK(v == 0x1d6b); + CHECK(p == 0x0102); + CHECK(u_stereo_uvc_parse_vid_pid("0x0bda:0x5830", &v, &p)); + CHECK(v == 0x0bda); + CHECK(p == 0x5830); + CHECK_FALSE(u_stereo_uvc_parse_vid_pid("/dev/video0", &v, &p)); + CHECK_FALSE(u_stereo_uvc_parse_vid_pid("vid_04f2", &v, &p)); + CHECK_FALSE(u_stereo_uvc_parse_vid_pid("04f2:b70", &v, &p)); + + u_stereo_uvc_device tracker = + device("Display Tracking Camera", "\\\\?\\usb#vid_04f2&pid_b70c&mi_00#6&1#{e5323777}\\global"); + u_stereo_uvc_device eyes = + device("Stereo Eyes Cam", "\\\\?\\usb#vid_2bdf&pid_0281&mi_00#7&2#{e5323777}\\global"); + u_stereo_uvc_device builtin = device("HD WebCam", "\\\\?\\usb#vid_04f2&pid_b5c5&mi_00#6&3#{e5323777}\\global"); + + u_stereo_uvc_entry e; + u_stereo_uvc_entry_defaults(&e); + CHECK_FALSE(u_stereo_uvc_entry_matches(&e, &eyes)); // no selector = nothing + + std::snprintf(e.name_contains, sizeof(e.name_contains), "eyes"); + CHECK(u_stereo_uvc_entry_matches(&e, &eyes)); // case-insensitive + CHECK_FALSE(u_stereo_uvc_entry_matches(&e, &tracker)); + CHECK_FALSE(u_stereo_uvc_entry_matches(&e, &builtin)); + + u_stereo_uvc_entry_defaults(&e); + e.match_vid_pid = true; + e.vid = 0x04f2; + e.pid = 0xb5c5; + CHECK(u_stereo_uvc_entry_matches(&e, &builtin)); + CHECK_FALSE(u_stereo_uvc_entry_matches(&e, &tracker)); // same VID, other PID + + // Both given: both must hold. + std::snprintf(e.name_contains, sizeof(e.name_contains), "Eyes"); + CHECK_FALSE(u_stereo_uvc_entry_matches(&e, &builtin)); + e.pid = 0x0281; + e.vid = 0x2bdf; + CHECK(u_stereo_uvc_entry_matches(&e, &eyes)); +} + +TEST_CASE("uvc assignment: one device per entry; a loose name claims nothing", "[stereo_uvc]") +{ + // A display's tracking camera and a stereo webcam from the same vendor + // share a name prefix — the case the ambiguity rule exists for. + u_stereo_uvc_device devs[] = { + device("Acme Tracking Camera", "\\\\?\\usb#vid_04f2&pid_b70c&mi_00#1#{x}\\global"), + device("Acme Eyes", "\\\\?\\usb#vid_2bdf&pid_0281&mi_00#2#{x}\\global"), + device("Acme Eyes", "\\\\?\\usb#vid_2bdf&pid_0281&mi_00#3#{x}\\global"), + device("HD WebCam", "\\\\?\\usb#vid_04f2&pid_b5c5&mi_00#4#{x}\\global"), + }; + int32_t a[U_STEREO_UVC_MAX_ENTRIES]; + + u_stereo_uvc_config cfg = parse_ok(R"({"uvc": [{"match": "acme"}]})"); + u_stereo_uvc_assign(&cfg, devs, 4, nullptr, a); + CHECK(a[0] == U_STEREO_UVC_ASSIGN_AMBIGUOUS); + CHECK(a[1] == U_STEREO_UVC_ASSIGN_NONE); + + // Specific enough by name, but two identical webcams: still ambiguous by name... + cfg = parse_ok(R"({"uvc": [{"match": "eyes"}]})"); + u_stereo_uvc_assign(&cfg, devs, 4, nullptr, a); + CHECK(a[0] == U_STEREO_UVC_ASSIGN_AMBIGUOUS); + // ... with one of them unavailable (a plug-in's), it is unique. + bool unavailable[4] = {false, true, false, false}; + u_stereo_uvc_assign(&cfg, devs, 4, unavailable, a); + CHECK(a[0] == 2); + + // VID:PID: identical devices go to successive entries in order. + cfg = parse_ok(R"({"uvc": [{"vid_pid": "2bdf:0281"}, {"vid_pid": "2bdf:0281"}, {"vid_pid": "2bdf:0281"}, + {"match": "tracking"}]})"); + u_stereo_uvc_assign(&cfg, devs, 4, nullptr, a); + CHECK(a[0] == 1); + CHECK(a[1] == 2); + CHECK(a[2] == U_STEREO_UVC_ASSIGN_NONE); + CHECK(a[3] == 0); // unique by name + + // A plug-in's camera is never assigned, whatever the entry says. + bool tracker_out[4] = {true, false, false, false}; + cfg = parse_ok(R"({"uvc": [{"vid_pid": "04f2:b70c"}]})"); + u_stereo_uvc_assign(&cfg, devs, 4, tracker_out, a); + CHECK(a[0] == U_STEREO_UVC_ASSIGN_NONE); +} + +TEST_CASE("uvc matching: a plug-in's camera is never claimed", "[stereo_uvc]") +{ + u_stereo_uvc_device tracker = + device("Display Tracking Camera", "\\\\?\\usb#vid_04f2&pid_b70c&mi_00#6&1#{e5323777}\\global"); + // The same OS id, any case. + CHECK( + u_stereo_uvc_device_claimed_by_hint(&tracker, "\\\\?\\USB#VID_04F2&PID_B70C&MI_00#6&1#{E5323777}\\GLOBAL")); + // A hint that is only a PnP instance id still names the VID:PID. + CHECK(u_stereo_uvc_device_claimed_by_hint(&tracker, "USB\\VID_04F2&PID_B70C&MI_00\\6&18588FC8&0&0000")); + // The form a display plug-in reports for its tracker camera: USB "vvvv:pppp". + CHECK(u_stereo_uvc_device_claimed_by_hint(&tracker, "04f2:b70c")); + CHECK(u_stereo_uvc_device_claimed_by_hint(&tracker, "04F2:B70C")); + CHECK_FALSE(u_stereo_uvc_device_claimed_by_hint(&tracker, "USB\\VID_04F2&PID_B5C5&MI_00\\x")); + CHECK_FALSE(u_stereo_uvc_device_claimed_by_hint(&tracker, "")); + CHECK_FALSE(u_stereo_uvc_device_claimed_by_hint(&tracker, nullptr)); + + // Through the source: an entry that WOULD match the tracker claims nothing + // once a plug-in names it (fake backend; its device id is "dxr-fake-uvc"). + u_stereo_uvc_config cfg = parse_ok(R"({"fake": true, "uvc": [{"match": "synthetic"}]})"); + const char *hints[] = {"dxr-fake-uvc"}; + u_stereo_uvc *u = u_stereo_uvc_create(&cfg, nullptr, hints, 1); + CHECK(u_stereo_uvc_enumerate(u, 0, nullptr) == 0); + u_stereo_uvc_destroy(&u); + u = u_stereo_uvc_create(&cfg, nullptr, nullptr, 0); + CHECK(u_stereo_uvc_enumerate(u, 0, nullptr) == 1); + u_stereo_uvc_destroy(&u); + CHECK(u == nullptr); +} + +TEST_CASE("uvc geometry: per-eye output size", "[stereo_uvc]") +{ + uint32_t w = 0, h = 0; + // Full SBS: each half is an eye. + REQUIRE(u_stereo_uvc_eye_size(U_STEREO_UVC_LAYOUT_SBS_FULL, 1280, 480, 0, 0, &w, &h)); + CHECK(w == 640); + CHECK(h == 480); + REQUIRE(u_stereo_uvc_eye_size(U_STEREO_UVC_LAYOUT_SBS_FULL, 2560, 720, 0, 0, &w, &h)); + CHECK(w == 1280); + CHECK(h == 720); + // Half SBS: a 2:1-squeezed half has W/2 x H/2 square-pixel information. + REQUIRE(u_stereo_uvc_eye_size(U_STEREO_UVC_LAYOUT_SBS_HALF, 1920, 1080, 0, 0, &w, &h)); + CHECK(w == 960); + CHECK(h == 540); + // ... capped at 1280 wide, aspect kept (3840x2160 half -> 1920x1080 -> 1280x720). + REQUIRE(u_stereo_uvc_eye_size(U_STEREO_UVC_LAYOUT_SBS_HALF, 3840, 2160, 0, 0, &w, &h)); + CHECK(w == 1280); + CHECK(h == 720); + REQUIRE(u_stereo_uvc_eye_size(U_STEREO_UVC_LAYOUT_SBS_FULL, 3840, 1080, 0, 0, &w, &h)); + CHECK(w == 1280); + CHECK(h == 720); + // Explicit, rounded down to even. + REQUIRE(u_stereo_uvc_eye_size(U_STEREO_UVC_LAYOUT_SBS_HALF, 3840, 2160, 1921, 1081, &w, &h)); + CHECK(w == 1920); + CHECK(h == 1080); + // Cannot split. + CHECK_FALSE(u_stereo_uvc_eye_size(U_STEREO_UVC_LAYOUT_SBS_FULL, 1282, 480, 0, 0, &w, &h)); + CHECK_FALSE(u_stereo_uvc_eye_size(U_STEREO_UVC_LAYOUT_SBS_FULL, 0, 480, 0, 0, &w, &h)); +} + +TEST_CASE("uvc geometry: the nominal pinhole is an exact parallel pair", "[stereo_uvc]") +{ + xrt_plugin_stereo_camera_calibration c; + std::memset(&c, 0, sizeof(c)); + c.struct_size = sizeof(c); + u_stereo_uvc_nominal_calibration(1280, 720, 90.0, 60.0, &c); + CHECK(c.image_width == 1280); + CHECK(c.image_height == 720); + CHECK(c.k[0][0] == Catch::Approx(640.0)); // (w/2) / tan(45 deg) + CHECK(c.k[0][1] == c.k[0][0]); + CHECK(c.k[0][2] == Catch::Approx(639.5)); + CHECK(c.k[0][3] == Catch::Approx(359.5)); + for (int i = 0; i < 4; i++) { + CHECK(c.k[1][i] == c.k[0][i]); + } + CHECK(c.distortion_model == XRT_PLUGIN_STEREO_CAMERA_DISTORTION_NONE); + CHECK(c.translation_right_from_left_mm[0] == Catch::Approx(-60.0)); + CHECK(c.translation_right_from_left_mm[1] == 0.0); + CHECK(c.rotation_right_from_left[1][1] == 1.0); + CHECK(c.rotation_right_from_left[0][1] == 0.0); + + // What the service's rectifier makes of it: the same pinhole, baseline, + // principal row — the maps are the identity up to the alpha = 0 border + // check's hair of zoom, which the service's pass-through skips entirely. + u_stereo_rectify_input in; + std::memset(&in, 0, sizeof(in)); + in.width = 1280; + in.height = 720; + for (int e = 0; e < 2; e++) { + in.eye[e].fx = c.k[e][0]; + in.eye[e].fy = c.k[e][1]; + in.eye[e].cx = c.k[e][2]; + in.eye[e].cy = c.k[e][3]; + in.eye[e].model = c.distortion_model; + } + std::memcpy(in.R, c.rotation_right_from_left, sizeof(in.R)); + std::memcpy(in.T, c.translation_right_from_left_mm, sizeof(in.T)); + u_stereo_rectify_result g; + REQUIRE(u_stereo_rectify_compute(&in, &g)); + CHECK(g.baseline == Catch::Approx(60.0)); + CHECK(g.cy == Catch::Approx(359.5).margin(1e-6)); + CHECK(g.crop_scale < 1.0025); + CHECK(g.f / c.k[0][0] == Catch::Approx(g.crop_scale).epsilon(1e-9)); + CHECK(g.t_rect[0] == Catch::Approx(-60.0)); +} + +TEST_CASE("uvc calibration file: OpenCV YAML and JSON", "[stereo_uvc]") +{ + const char *yaml = R"(%YAML:1.0 +--- +image_width: 1920 +image_height: 2160 +M1: !!opencv-matrix + rows: 3 + cols: 3 + dt: d + data: [ 9.5e+02, 0., 9.6e+02, 0., 1.9e+03, + 1.08e+03, 0., 0., 1. ] +D1: !!opencv-matrix + rows: 1 + cols: 5 + dt: d + data: [ -1.2e-01, 3.0e-02, 1.0e-04, -2.0e-04, 0. ] +M2: !!opencv-matrix + rows: 3 + cols: 3 + dt: d + data: [ 9.4e+02, 0., 9.55e+02, 0., 1.89e+03, 1.07e+03, 0., 0., 1. ] +D2: !!opencv-matrix + rows: 1 + cols: 5 + dt: d + data: [ -1.1e-01, 2.0e-02, 0., 0., 0. ] +R: !!opencv-matrix + rows: 3 + cols: 3 + dt: d + data: [ 1., 0., 0., 0., 1., 0., 0., 0., 1. ] +T: !!opencv-matrix + rows: 3 + cols: 1 + dt: d + data: [ -6.1e+01, 2.0e-01, -1.0e-01 ] +)"; + xrt_plugin_stereo_camera_calibration c; + std::memset(&c, 0, sizeof(c)); + c.struct_size = sizeof(c); + char err[160] = {0}; + REQUIRE(u_stereo_uvc_calibration_parse(yaml, 1, 1, &c, err, sizeof(err))); + CHECK(c.image_width == 1920); + CHECK(c.image_height == 2160); + CHECK(c.k[0][0] == Catch::Approx(950.0)); + CHECK(c.k[0][1] == Catch::Approx(1900.0)); // a half-SBS calibration: anisotropic, fine + CHECK(c.k[0][3] == Catch::Approx(1080.0)); + CHECK(c.k[1][2] == Catch::Approx(955.0)); + CHECK(c.distortion_model == XRT_PLUGIN_STEREO_CAMERA_DISTORTION_RADTAN5); + CHECK(c.distortion[0][0] == Catch::Approx(-0.12)); + CHECK(c.distortion[1][1] == Catch::Approx(0.02)); + CHECK(c.translation_right_from_left_mm[0] == Catch::Approx(-61.0)); + CHECK(c.translation_right_from_left_mm[2] == Catch::Approx(-0.1)); + CHECK(c.rotation_right_from_left[2][2] == 1.0); + + // JSON, nested matrices, a rotation VECTOR, T in metres, no size (default). + const char *json = R"({"K1": [[700, 0, 320], [0, 700, 240], [0, 0, 1]], "D1": [0.1, 0.0, 0.0, 0.0], + "K2": [700, 0, 321, 0, 701, 239, 0, 0, 1], "D2": [0.1, 0.0, 0.0, 0.0], + "R": [0.0, 0.01, 0.0], "T": [-0.05, 0.0, 0.0], "distortion_model": "fisheye"})"; + REQUIRE(u_stereo_uvc_calibration_parse(json, 640, 480, &c, err, sizeof(err))); + CHECK(c.image_width == 640); + CHECK(c.image_height == 480); + CHECK(c.k[1][1] == Catch::Approx(701.0)); + CHECK(c.distortion_model == XRT_PLUGIN_STEREO_CAMERA_DISTORTION_KB4); + CHECK(c.translation_right_from_left_mm[0] == Catch::Approx(-50.0)); // metres -> mm + CHECK(c.rotation_right_from_left[0][0] == Catch::Approx(std::cos(0.01))); + CHECK(c.rotation_right_from_left[0][2] == Catch::Approx(std::sin(0.01))); + + // Rejections. + CHECK_FALSE(u_stereo_uvc_calibration_parse(R"({"K1": [1, 2, 3]})", 640, 480, &c, err, sizeof(err))); + CHECK(std::string(err).find("K1") != std::string::npos); + CHECK_FALSE(u_stereo_uvc_calibration_parse( + R"({"K1": [700,0,320,0,700,240,0,0,1], "K2": [700,0,320,0,700,240,0,0,1], "D1": [0,0,0,0,0], + "D2": [0,0,0,0], "R": [1,0,0,0,1,0,0,0,1], "T": [-50,0,0]})", + 640, 480, &c, err, sizeof(err))); + CHECK_FALSE(u_stereo_uvc_calibration_parse( + R"({"K1": [700,0,320,0,700,240,0,0,1], "K2": [700,0,320,0,700,240,0,0,1], "D1": [0,0,0,0,0], + "D2": [0,0,0,0,0], "R": [1,0,0,0,1,0,0,0,1], "T": [0,0,0]})", + 640, 480, &c, err, sizeof(err))); + CHECK_FALSE(u_stereo_uvc_calibration_parse(nullptr, 640, 480, &c, err, sizeof(err))); +} + +TEST_CASE("uvc split: eye order, full vs half, NV12 and YUY2", "[stereo_uvc]") +{ + // A 16x8 NV12 SBS frame: left half luma = 40 + row, right half = 200 - row; + // chroma U/V differ per half too. + const uint32_t W = 16, H = 8; + std::vector nv12(W * H * 3 / 2); + for (uint32_t y = 0; y < H; y++) { + for (uint32_t x = 0; x < W; x++) { + nv12[y * W + x] = (uint8_t)(x < W / 2 ? 40 + y : 200 - y); + } + } + for (uint32_t y = 0; y < H / 2; y++) { + for (uint32_t x = 0; x < W / 2; x++) { + uint8_t *uv = &nv12[W * H + y * W + 2 * x]; + uv[0] = (uint8_t)(x < W / 4 ? 90 : 100); // U + uv[1] = (uint8_t)(x < W / 4 ? 160 : 170); // V + } + } + u_stereo_uvc_raw_frame in; + std::memset(&in, 0, sizeof(in)); + in.pixel = U_STEREO_UVC_PIXEL_NV12; + in.width = W; + in.height = H; + in.planes[0] = nv12.data(); + in.planes[1] = nv12.data() + W * H; + in.pitches[0] = in.pitches[1] = W; + + SECTION("full SBS, lr: a straight copy") + { + u_stereo_camera_planes lay; + REQUIRE(u_stereo_camera_layout(XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12, 16, 8, &lay)); + std::vector out(lay.size); + REQUIRE(u_stereo_uvc_split(&in, false, 8, 8, out.data(), &lay)); + for (uint32_t y = 0; y < 8; y++) { + CHECK(out[lay.offset[0] + y * lay.pitch[0] + 0] == 40 + y); + CHECK(out[lay.offset[0] + y * lay.pitch[0] + 15] == 200 - y); + } + CHECK(out[lay.offset[1] + 0] == 90); + CHECK(out[lay.offset[1] + 1] == 160); + CHECK(out[lay.offset[1] + 8] == 100); + CHECK(out[lay.offset[1] + 9] == 170); + } + SECTION("full SBS, rl: the halves swap") + { + u_stereo_camera_planes lay; + REQUIRE(u_stereo_camera_layout(XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12, 16, 8, &lay)); + std::vector out(lay.size); + REQUIRE(u_stereo_uvc_split(&in, true, 8, 8, out.data(), &lay)); + CHECK(out[lay.offset[0] + 3 * lay.pitch[0] + 0] == 197); + CHECK(out[lay.offset[0] + 3 * lay.pitch[0] + 15] == 43); + CHECK(out[lay.offset[1] + 0] == 100); + CHECK(out[lay.offset[1] + 8] == 90); + } + SECTION("half SBS: 2:1 vertical is an exact two-row average") + { + // 8x8 halves -> 8x4 eyes. + u_stereo_camera_planes lay; + REQUIRE(u_stereo_camera_layout(XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12, 16, 4, &lay)); + std::vector out(lay.size); + REQUIRE(u_stereo_uvc_split(&in, false, 8, 4, out.data(), &lay)); + for (uint32_t y = 0; y < 4; y++) { + // rows 2y, 2y+1: (40+2y + 40+2y+1) / 2 = 40 + 2y + 0.5 -> rounds up + CHECK(out[lay.offset[0] + y * lay.pitch[0] + 2] == 41 + 2 * y); + CHECK(out[lay.offset[0] + y * lay.pitch[0] + 12] == 200 - 2 * y); + } + } + SECTION("YUY2 in, NV12 out") + { + std::vector yuy2(W * H * 2); + for (uint32_t y = 0; y < H; y++) { + for (uint32_t x = 0; x < W; x += 2) { + uint8_t *p = &yuy2[y * W * 2 + x * 2]; + bool left = x < W / 2; + p[0] = (uint8_t)(left ? 40 + y : 200 - y); + p[1] = (uint8_t)(left ? 90 : 100); + p[2] = p[0]; + p[3] = (uint8_t)(left ? 160 : 170); + } + } + u_stereo_uvc_raw_frame yin = in; + yin.pixel = U_STEREO_UVC_PIXEL_YUY2; + yin.planes[0] = yuy2.data(); + yin.planes[1] = nullptr; + yin.pitches[0] = W * 2; + u_stereo_camera_planes lay; + REQUIRE(u_stereo_camera_layout(XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12, 16, 8, &lay)); + std::vector out(lay.size); + REQUIRE(u_stereo_uvc_split(&yin, false, 8, 8, out.data(), &lay)); + CHECK(out[lay.offset[0] + 5 * lay.pitch[0] + 1] == 45); + CHECK(out[lay.offset[0] + 5 * lay.pitch[0] + 9] == 195); + CHECK(out[lay.offset[1] + 2 * lay.pitch[1] + 0] == 90); + CHECK(out[lay.offset[1] + 2 * lay.pitch[1] + 1] == 160); + CHECK(out[lay.offset[1] + 2 * lay.pitch[1] + 10] == 100); + CHECK(out[lay.offset[1] + 2 * lay.pitch[1] + 11] == 170); + } + SECTION("bad input") + { + u_stereo_camera_planes lay; + REQUIRE(u_stereo_camera_layout(XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12, 16, 8, &lay)); + std::vector out(lay.size); + u_stereo_uvc_raw_frame bad = in; + bad.width = 18; // not a multiple of 4 + CHECK_FALSE(u_stereo_uvc_split(&bad, false, 8, 8, out.data(), &lay)); + CHECK_FALSE(u_stereo_uvc_split(&in, false, 7, 8, out.data(), &lay)); + bad = in; + bad.pixel = 99; + CHECK_FALSE(u_stereo_uvc_split(&bad, false, 8, 8, out.data(), &lay)); + } +} + +TEST_CASE("uvc split: cost of a half-SBS 3840x2160 NV12 frame to 2 x 1280x720", "[stereo_uvc][.perf]") +{ + const uint32_t W = 3840, H = 2160; + std::vector nv12((size_t)W * H * 3 / 2); + for (size_t i = 0; i < nv12.size(); i++) { + nv12[i] = (uint8_t)(i * 2654435761u >> 24); + } + u_stereo_uvc_raw_frame in; + std::memset(&in, 0, sizeof(in)); + in.pixel = U_STEREO_UVC_PIXEL_NV12; + in.width = W; + in.height = H; + in.planes[0] = nv12.data(); + in.planes[1] = nv12.data() + (size_t)W * H; + in.pitches[0] = in.pitches[1] = W; + u_stereo_camera_planes lay; + REQUIRE(u_stereo_camera_layout(XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12, 2560, 720, &lay)); + std::vector out(lay.size); + for (uint32_t eye_w : {1280u, 2560u / 2}) { + auto t0 = std::chrono::steady_clock::now(); + const int n = 20; + for (int i = 0; i < n; i++) { + REQUIRE(u_stereo_uvc_split(&in, false, eye_w, 720, out.data(), &lay)); + } + double ms = + std::chrono::duration(std::chrono::steady_clock::now() - t0).count() / n; + std::printf("[uvc split perf] 3840x2160 NV12 half-SBS -> 2 x %ux720: %.2f ms/frame\n", eye_w, ms); + } + // The GPU path hands over 2560x720 already: a straight copy. + in.width = 2560; + in.height = 720; + in.pitches[0] = in.pitches[1] = 2560; + in.planes[1] = nv12.data() + (size_t)2560 * 720; + auto t0 = std::chrono::steady_clock::now(); + for (int i = 0; i < 50; i++) { + REQUIRE(u_stereo_uvc_split(&in, false, 1280, 720, out.data(), &lay)); + } + double ms = std::chrono::duration(std::chrono::steady_clock::now() - t0).count() / 50; + std::printf("[uvc split perf] 2560x720 (GPU-scaled) -> 2 x 1280x720: %.3f ms/frame\n", ms); +} + +namespace { + +struct source_run +{ + uint32_t eye_w = 0, eye_h = 0; + uint32_t flags = 0; + std::vector y; //!< the last frame's luma, tight 2*eye_w pitch + uint32_t frames = 0; +}; + +//! Enumerate -> open -> read @p n frames through the source on the fake backend. +source_run +run_source(const char *json, uint32_t n) +{ + source_run r; + u_stereo_uvc_config cfg = parse_ok(json); + u_stereo_uvc *u = u_stereo_uvc_create(&cfg, nullptr, nullptr, 0); + REQUIRE(u_stereo_uvc_enumerate(u, 0, nullptr) == 1); + xrt_plugin_stereo_camera_info info; + std::memset(&info, 0, sizeof(info)); + info.struct_size = sizeof(info); + REQUIRE(u_stereo_uvc_enumerate(u, 1, &info) == 1); + CHECK(info.struct_size == sizeof(info)); + CHECK(info.native_format == XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12); + CHECK(std::string(info.platform_device_hint) == "dxr-fake-uvc"); + CHECK(std::string(info.device_identity) == "uvc:dxr-fake-uvc"); + r.eye_w = info.eye_width; + r.eye_h = info.eye_height; + r.flags = info.flags; + + u_stereo_uvc_stream *s = nullptr; + REQUIRE(u_stereo_uvc_open(u, 0, &s) == XRT_SUCCESS); + uint64_t last_seq = 0; + for (uint32_t i = 0; i < n; i++) { + xrt_plugin_stereo_camera_frame f; + uint32_t w = u_stereo_uvc_wait_frame(s, 500ll * 1000 * 1000, &f); + REQUIRE(w == XRT_PLUGIN_STEREO_CAMERA_WAIT_OK); + CHECK(f.sequence == last_seq + 1); + last_seq = f.sequence; + CHECK(f.width == 2 * r.eye_w); + CHECK(f.height == r.eye_h); + CHECK(f.format == XRT_PLUGIN_STEREO_CAMERA_FORMAT_NV12); + CHECK(f.time_ns > 0); + r.y.resize((size_t)f.width * f.height); + for (uint32_t row = 0; row < f.height; row++) { + std::memcpy(&r.y[(size_t)row * f.width], f.planes[0] + (size_t)row * f.pitches[0], f.width); + } + CHECK(f.planes[1][0] == 128); // the fake is grey: neutral chroma survives the split + u_stereo_uvc_release_frame(s); + r.frames++; + } + u_stereo_uvc_close(s); + u_stereo_uvc_destroy(&u); + return r; +} + +} // namespace + +TEST_CASE("uvc source on the FAKE backend: the known disparity and dy come back", "[stereo_uvc]") +{ + struct variant + { + const char *json; + uint32_t eye_w, eye_h; + } v[] = { + // Full SBS, NV12: 1280x480 -> 640x480 per eye. + {R"({"fake": {"disparity": 0.04, "dy": 0.005, "fps": 240}, + "uvc": [{"match": "synthetic", "mode": "1280x480@240"}]})", + 640, 480}, + // Half SBS, YUY2: 1280x960 -> 640x480 per eye (2:1 vertical). + {R"({"fake": {"disparity": 0.04, "dy": 0.005, "pixel": "yuy2", "fps": 240}, + "uvc": [{"match": "SYNTHETIC", "layout": "sbs-half", "mode": "1280x960@240"}]})", + 640, 480}, + }; + for (const variant &t : v) { + source_run r = run_source(t.json, 3); + CHECK(r.eye_w == t.eye_w); + CHECK(r.eye_h == t.eye_h); + CHECK((r.flags & XRT_PLUGIN_STEREO_CAMERA_CALIBRATED) == 0); // nominal model + CHECK((r.flags & XRT_PLUGIN_STEREO_CAMERA_SHARED_WITH_EYE_TRACKING) == 0); + std::vector dx, dy; + block_offsets(r.y.data(), 2 * r.eye_w, r.eye_w, r.eye_h, 48, dx, dy); + REQUIRE(dx.size() > 20); + double mdx = median(dx), mdy = median(dy); + std::printf( + "[uvc fake] eye %ux%u: disparity %.2f px (want %.2f), dy %+.2f px (want %+.2f), %zu blocks\n", + r.eye_w, r.eye_h, mdx, 0.04 * r.eye_w, mdy, 0.005 * r.eye_h, dx.size()); + CHECK(mdx == Catch::Approx(0.04 * r.eye_w).margin(0.35)); + CHECK(mdy == Catch::Approx(0.005 * r.eye_h).margin(0.35)); + } + + // "rl": the device's RIGHT lens is in the left half -> the halves swap. + source_run lr = run_source(R"({"fake": {"fps": 240}, "uvc": [{"match": "synthetic", "mode": "1280x480"}]})", 1); + source_run rl = run_source( + R"({"fake": {"fps": 240}, "uvc": [{"match": "synthetic", "mode": "1280x480", "eyes": "rl"}]})", 1); + REQUIRE(lr.y.size() == rl.y.size()); + bool swapped = true; + for (uint32_t row = 0; row < lr.eye_h && swapped; row++) { + const uint8_t *a = &lr.y[(size_t)row * 2 * lr.eye_w]; + const uint8_t *b = &rl.y[(size_t)row * 2 * rl.eye_w]; + swapped = std::memcmp(a, b + lr.eye_w, lr.eye_w) == 0 && std::memcmp(a + lr.eye_w, b, lr.eye_w) == 0; + } + CHECK(swapped); +} + +TEST_CASE("uvc source: calibration — nominal by default, the file's when given", "[stereo_uvc]") +{ + u_stereo_uvc_config cfg = parse_ok( + R"({"fake": true, "uvc": [{"match": "synthetic", "mode": "1280x480", "baseline_mm": 65, "hfov_deg": 80}]})"); + u_stereo_uvc *u = u_stereo_uvc_create(&cfg, nullptr, nullptr, 0); + REQUIRE(u_stereo_uvc_enumerate(u, 0, nullptr) == 1); + xrt_plugin_stereo_camera_calibration c; + std::memset(&c, 0, sizeof(c)); + c.struct_size = sizeof(c); + bool nominal = false, bl = false, hf = false; + REQUIRE(u_stereo_uvc_get_calibration(u, 0, &c, &nominal, &bl, &hf) == XRT_SUCCESS); + CHECK(nominal); + CHECK(bl); + CHECK(hf); + CHECK(c.k[0][0] == Catch::Approx(320.0 / std::tan(40.0 * k_pi / 180.0))); + CHECK(c.translation_right_from_left_mm[0] == Catch::Approx(-65.0)); + CHECK(u_stereo_uvc_get_calibration(u, 1, &c, &nominal, &bl, &hf) == XRT_ERROR_FEATURE_NOT_SUPPORTED); + u_stereo_uvc_mode m; + uint32_t layout = 0; + REQUIRE(u_stereo_uvc_get_mode(u, 0, &m, &layout)); + CHECK(m.width == 1280); + CHECK(layout == U_STEREO_UVC_LAYOUT_SBS_FULL); + u_stereo_uvc_destroy(&u); + + // Unconfigured baseline / HFOV: defaults drive the model, flagged unknown. + cfg = parse_ok(R"({"fake": true, "uvc": [{"match": "synthetic"}]})"); + u = u_stereo_uvc_create(&cfg, nullptr, nullptr, 0); + REQUIRE(u_stereo_uvc_get_calibration(u, 0, &c, &nominal, &bl, &hf) == XRT_SUCCESS); + CHECK(nominal); + CHECK_FALSE(bl); + CHECK_FALSE(hf); + CHECK(c.translation_right_from_left_mm[0] == Catch::Approx(-U_STEREO_UVC_DEFAULT_BASELINE_MM)); + // Auto mode = the largest the device lists (the fake's 2560x720). + REQUIRE(u_stereo_uvc_get_mode(u, 0, &m, &layout)); + CHECK(m.width == 2560); + CHECK(m.height == 720); + u_stereo_uvc_destroy(&u); + + // A missing calibration file falls back to the nominal model (and says so). + cfg = parse_ok(R"({"fake": true, "uvc": [{"match": "synthetic", "calibration": "/nonexistent/cal.yaml"}]})"); + u = u_stereo_uvc_create(&cfg, nullptr, nullptr, 0); + REQUIRE(u_stereo_uvc_get_calibration(u, 0, &c, &nominal, &bl, &hf) == XRT_SUCCESS); + CHECK(nominal); + u_stereo_uvc_destroy(&u); + + // No backend at all (a platform without one): configured entries claim nothing. + cfg = parse_ok(R"({"uvc": [{"match": "x"}]})"); + u = u_stereo_uvc_create(&cfg, nullptr, nullptr, 0); + CHECK(u_stereo_uvc_enumerate(u, 0, nullptr) == 0); + u_stereo_uvc *none = nullptr; + u_stereo_uvc_stream *ss = nullptr; + CHECK(u_stereo_uvc_open(none, 0, &ss) != XRT_SUCCESS); + CHECK(ss == nullptr); + u_stereo_uvc_destroy(&u); +} + +TEST_CASE("uvc CLOSED LOOP: pass-through + online refinement aligns an uncalibrated pair", "[stereo_uvc]") +{ + // The service's path for an uncalibrated UVC pair, step by step: the + // nominal model, frames passed through untouched, the refinement measuring + // the device's own vertical misalignment (2.4 px here) and the corrected + // maps it rebuilds. Judged by an independent block matcher. + source_run r = run_source(R"({"fake": {"disparity": 0.04, "dy": 0.005, "fps": 240}, + "uvc": [{"match": "synthetic", "mode": "1280x480", "baseline_mm": 60, + "hfov_deg": 70}]})", + 2); + const uint32_t W = r.eye_w, H = r.eye_h; + xrt_plugin_stereo_camera_calibration c; + std::memset(&c, 0, sizeof(c)); + c.struct_size = sizeof(c); + u_stereo_uvc_nominal_calibration(W, H, 70.0, 60.0, &c); + u_stereo_rectify_input in; + std::memset(&in, 0, sizeof(in)); + in.width = W; + in.height = H; + in.calib_width = c.image_width; + in.calib_height = c.image_height; + for (int e = 0; e < 2; e++) { + in.eye[e].fx = c.k[e][0]; + in.eye[e].fy = c.k[e][1]; + in.eye[e].cx = c.k[e][2]; + in.eye[e].cy = c.k[e][3]; + in.eye[e].model = c.distortion_model; + } + std::memcpy(in.R, c.rotation_right_from_left, sizeof(in.R)); + std::memcpy(in.T, c.translation_right_from_left_mm, sizeof(in.T)); + u_stereo_rectify_result geo; + REQUIRE(u_stereo_rectify_compute(&in, &geo)); + + u_stereo_vrefine_config cfg; + u_stereo_vrefine_config_defaults(&cfg, H, geo.cy); + u_stereo_vrefine ref; + u_stereo_vrefine_init(&ref, &cfg); + u_stereo_vrefine_measure_params mp; + double dmax = geo.f * geo.baseline / 300.0; + dmax = dmax < 24.0 ? 24.0 : dmax > W / 2.0 ? W / 2.0 : dmax; + u_stereo_vrefine_measure_defaults(&mp, (uint32_t)dmax); + std::vector s(512); + + const std::vector &raw = r.y; // the static fake scene + std::vector rect = raw; // pass-through until a correction applies + u_stereo_rectify_lut lut; + std::memset(&lut, 0, sizeof(lut)); + bool have_lut = false; + + std::vector dx, dy; + block_offsets(rect.data(), 2 * W, W, H, 48, dx, dy); + const double before = median(dy); + + int64_t now = 1000000000; + uint32_t updates = 0; + for (uint32_t frame = 0; frame < 90; frame++, now += 33333333) { + if (!u_stereo_vrefine_due(&ref, now)) { + continue; + } + uint32_t n = u_stereo_vrefine_measure(rect.data(), 2 * W, W, H, &mp, s.data(), (uint32_t)s.size()); + if (u_stereo_vrefine_push(&ref, now, s.data(), n) == U_STEREO_VREFINE_UPDATED) { + updates++; + u_stereo_rectify_input ci = in; + ci.v_offset = ref.a / geo.f; + ci.v_slope = ref.b; + u_stereo_rectify_result g; + REQUIRE(u_stereo_rectify_compute(&ci, &g)); + if (have_lut) { + u_stereo_rectify_lut_fini(&lut); + } + REQUIRE(u_stereo_rectify_lut_init(&lut, &g, 1)); + have_lut = true; + u_stereo_rectify_lut_apply(&lut, 1, raw.data(), 2 * W, rect.data(), 2 * W); + } + } + block_offsets(rect.data(), 2 * W, W, H, 48, dx, dy); + const double after = median(dy); + std::printf( + "[uvc closed loop] %u map rebuild(s): correction a %+.3f px b %+.5f; signed dy median %+.3f -> " + "%+.3f px (%zu blocks), disparity %.2f px\n", + updates, ref.a, ref.b, before, after, dy.size(), median(dx)); + CHECK(before == Catch::Approx(2.4).margin(0.35)); + CHECK(updates >= 1); + CHECK(ref.applied); + CHECK(std::fabs(after) <= 0.25); + CHECK(median(dx) == Catch::Approx(0.04 * W).margin(0.6)); // the correction is vertical only + if (have_lut) { + u_stereo_rectify_lut_fini(&lut); + } +} diff --git a/tests/tests_uvc_capture_mf.cpp b/tests/tests_uvc_capture_mf.cpp new file mode 100644 index 0000000000..e03ef7f397 --- /dev/null +++ b/tests/tests_uvc_capture_mf.cpp @@ -0,0 +1,253 @@ +// Copyright 2026, The DisplayXR Project +// SPDX-License-Identifier: BSL-1.0 +/*! + * @file + * @brief ADR-043 Amendment 4, Windows: the Media Foundation UVC backend's + * SCALED output is a plain per-axis resize — never letterboxed. + * + * The first real-webcam run of the GPU path asked the video processor for + * 2560x720 from a 16:9 3840x2160 half-SBS frame and got the picture + * PILLARBOXED (the processor preserves display aspect): a centred ~1280 px + * image between black bars, so each "eye" was black + half the picture. + * + * This writes a small uncompressed YUY2 AVI (a uniformly bright texture, + * 16:9) and opens it through the real backend (`file:` id) asking for a 32:9 + * output — the same shape change — on BOTH decode paths (the hardware policy: + * a D3D11 device manager + GPU video processor when an adapter qualifies; the + * software policy: MF's CPU video processor). It asserts the frame arrives at + * the requested size and that its outer 5 % columns are as bright as the + * centre. No camera. SKIPs where Media Foundation cannot read the file. + */ + +#include "catch_amalgamated.hpp" + +#include "os/os_uvc_capture.h" +#include "util/u_stereo_uvc.h" + +#include +#include +#include +#include +#include +#include + +#include + +namespace { + +void +put32(std::vector &b, uint32_t v) +{ + for (int i = 0; i < 4; i++) { + b.push_back((uint8_t)(v >> (8 * i))); + } +} + +void +put16(std::vector &b, uint16_t v) +{ + b.push_back((uint8_t)v); + b.push_back((uint8_t)(v >> 8)); +} + +void +fourcc(std::vector &b, const char *c) +{ + b.insert(b.end(), c, c + 4); +} + +void +patch32(std::vector &b, size_t at, uint32_t v) +{ + for (int i = 0; i < 4; i++) { + b[at + i] = (uint8_t)(v >> (8 * i)); + } +} + +//! A minimal uncompressed YUY2 AVI: @p frames frames of a bright texture. +std::vector +make_yuy2_avi(uint32_t w, uint32_t h, uint32_t frames, uint32_t fps) +{ + const uint32_t frame_bytes = w * h * 2; + std::vector f; + fourcc(f, "RIFF"); + size_t riff_size = f.size(); + put32(f, 0); + fourcc(f, "AVI "); + + fourcc(f, "LIST"); + size_t hdrl_size = f.size(); + put32(f, 0); + fourcc(f, "hdrl"); + fourcc(f, "avih"); + put32(f, 56); + put32(f, 1000000 / fps); // dwMicroSecPerFrame + put32(f, frame_bytes * fps); // dwMaxBytesPerSec + put32(f, 0); // dwPaddingGranularity + put32(f, 0x10); // AVIF_HASINDEX + put32(f, frames); // dwTotalFrames + put32(f, 0); // dwInitialFrames + put32(f, 1); // dwStreams + put32(f, frame_bytes); // dwSuggestedBufferSize + put32(f, w); + put32(f, h); + for (int i = 0; i < 4; i++) { + put32(f, 0); + } + fourcc(f, "LIST"); + size_t strl_size = f.size(); + put32(f, 0); + fourcc(f, "strl"); + fourcc(f, "strh"); + put32(f, 56); + fourcc(f, "vids"); + fourcc(f, "YUY2"); + put32(f, 0); // dwFlags + put16(f, 0); // wPriority + put16(f, 0); // wLanguage + put32(f, 0); // dwInitialFrames + put32(f, 1); // dwScale + put32(f, fps); // dwRate + put32(f, 0); // dwStart + put32(f, frames); // dwLength + put32(f, frame_bytes); // dwSuggestedBufferSize + put32(f, 0xffffffffu); // dwQuality + put32(f, 0); // dwSampleSize + put16(f, 0); + put16(f, 0); + put16(f, (uint16_t)w); + put16(f, (uint16_t)h); + fourcc(f, "strf"); + put32(f, 40); + put32(f, 40); // biSize + put32(f, w); + put32(f, h); + put16(f, 1); // biPlanes + put16(f, 16); // biBitCount + fourcc(f, "YUY2"); + put32(f, frame_bytes); + put32(f, 0); + put32(f, 0); + put32(f, 0); + put32(f, 0); + patch32(f, strl_size, (uint32_t)(f.size() - strl_size - 4)); + patch32(f, hdrl_size, (uint32_t)(f.size() - hdrl_size - 4)); + + fourcc(f, "LIST"); + size_t movi_size = f.size(); + put32(f, 0); + size_t movi_fcc = f.size(); + fourcc(f, "movi"); + std::vector offsets; + for (uint32_t n = 0; n < frames; n++) { + offsets.push_back((uint32_t)(f.size() - movi_fcc)); + fourcc(f, "00db"); + put32(f, frame_bytes); + for (uint32_t y = 0; y < h; y++) { + for (uint32_t x = 0; x < w; x += 2) { + // Bright everywhere (luma 110..210): any black bar is unmistakable. + auto lum = [&](uint32_t px) { + return (uint8_t)(160 + 50 * std::sin(px * 0.21 + n) * std::cos(y * 0.17)); + }; + f.push_back(lum(x)); + f.push_back(128); + f.push_back(lum(x + 1)); + f.push_back(128); + } + } + } + patch32(f, movi_size, (uint32_t)(f.size() - movi_size - 4)); + fourcc(f, "idx1"); + put32(f, frames * 16); + for (uint32_t n = 0; n < frames; n++) { + fourcc(f, "00db"); + put32(f, 0x10); // AVIIF_KEYFRAME + put32(f, offsets[n]); + put32(f, frame_bytes); + } + patch32(f, riff_size, (uint32_t)(f.size() - 8)); + return f; +} + +double +mean_luma(const u_stereo_uvc_raw_frame &r, uint32_t x0, uint32_t x1) +{ + double s = 0.0; + uint64_t n = 0; + const uint32_t step = r.pixel == U_STEREO_UVC_PIXEL_YUY2 ? 2 : 1; + for (uint32_t y = 0; y < r.height; y++) { + const uint8_t *row = r.planes[0] + (size_t)y * r.pitches[0]; + for (uint32_t x = x0; x < x1; x++) { + s += row[(size_t)x * step]; + n++; + } + } + return n > 0 ? s / (double)n : 0.0; +} + +} // namespace + +TEST_CASE("uvc capture (MF): a scaled output is a plain resize, never letterboxed", "[uvc_capture_mf]") +{ + char tmp[MAX_PATH]; + GetTempPathA(sizeof(tmp), tmp); + std::string path = std::string(tmp) + "dxr_tests_uvc_aspect.avi"; + std::vector avi = make_yuy2_avi(1280, 720, 30, 30); // 16:9 + FILE *fp = std::fopen(path.c_str(), "wb"); + REQUIRE(fp != nullptr); + std::fwrite(avi.data(), 1, avi.size(), fp); + std::fclose(fp); + + u_stereo_uvc_backend b; + REQUIRE(os_uvc_capture_backend(&b)); + const std::string id = "file:" + path; + u_stereo_uvc_mode modes[8]; + uint32_t nm = b.list_modes(b.ctx, id.c_str(), modes, 8); + if (nm == 0) { + std::remove(path.c_str()); + SKIP("Media Foundation cannot read the test AVI here"); + } + REQUIRE(modes[0].width == 1280); + REQUIRE(modes[0].height == 720); + + for (const char *policy : {"hardware", "software"}) { + _putenv_s("DXR_STEREO_CAMERA_UVC_DECODER", policy); + void *h = nullptr; + // 1280x360 from 1280x720 = 32:9 from 16:9, the 3840x2160 -> 2560x720 shape. + if (!b.open(b.ctx, id.c_str(), &modes[0], 1280, 360, &h)) { + std::printf("[uvc aspect] %s policy: no such path on this machine — skipped\n", policy); + continue; + } + u_stereo_uvc_raw_frame r; + bool got = false; + uint32_t timeouts = 0, errors = 0; + for (int i = 0; i < 40 && !got; i++) { + uint32_t rr = b.read(h, 500ll * 1000 * 1000, &r); + got = rr == U_STEREO_UVC_READ_OK; + timeouts += rr == U_STEREO_UVC_READ_TIMEOUT ? 1 : 0; + if (rr == U_STEREO_UVC_READ_ERROR) { + errors++; + break; + } + } + INFO(policy << " policy: " << timeouts << " timeout(s), " << errors << " error(s) before a frame"); + REQUIRE(got); + const uint32_t edge = r.width / 20; // 5 % + const double left = mean_luma(r, 0, edge); + const double right = mean_luma(r, r.width - edge, r.width); + const double centre = mean_luma(r, r.width / 4, r.width * 3 / 4); + std::printf( + "[uvc aspect] %s policy: delivered %ux%u, mean luma outer-left %.1f / centre %.1f / " + "outer-right %.1f\n", + policy, r.width, r.height, left, centre, right); + CHECK(r.width == 1280); + CHECK(r.height == 360); + CHECK(centre > 120.0); + // A letterboxed frame has ~16 (black) here; the picture is ~160 everywhere. + CHECK(left > 0.8 * centre); + CHECK(right > 0.8 * centre); + b.close(h); + } + _putenv_s("DXR_STEREO_CAMERA_UVC_DECODER", ""); + std::remove(path.c_str()); +}