feat(stereo-camera): R2 — service-side rectifier (u_stereo_rectify), distorted fake, probe --rectified

docs/adr/ADR-043-stereo-camera-source.md

@@ -2,7 +2,7 @@
 
 **Status:** Proposed (2026-09-25) · **R1 implemented** 2026-09-26 (runtime, hardware-free; maintainer
 defaults recorded in the roadmap §G: stereo-only, iface slots, service clients only, no raw frames
-to pages) · introduces
+to pages) · **R2 implemented** 2026-09-26 (the service-side rectifier, decision 2 below) · introduces
 [`XR_DXR_stereo_camera`](../specs/extensions/XR_DXR_stereo_camera.md) · appends optional
 camera slots to `xrt_plugin_iface` under the [ADR-020](ADR-020-plugin-abi-compatibility-policy.md)
 append-at-end rule · related: [ADR-019](ADR-019-vendor-plugin-aux-boundary.md),

docs/reference/xrt_plugin_iface.md

@@ -631,7 +631,7 @@ void         (*stereo_camera_close)(cam);
 | `wait_frame` is called from **one runtime-owned thread per open camera**, may block up to `timeout_ns`, and returns an `enum xrt_plugin_stereo_camera_wait` (`OK` / `TIMEOUT` / `SUSPENDED` / `ERROR`). Planes stay valid until `release_frame`. | The service maps `TIMEOUT` → `WAITING` (then `SUSPENDED` after 1 s without frames), `SUSPENDED` → `SUSPENDED`, `ERROR` → close + retry in 1 s |
 | **Never wait on a wake object shared with other readers** of the vendor channel — poll it. | An auto-reset event wakes exactly one waiter: two readers steal each other's frames (ADR-043) |
 | Decode your transport (JPEG, YUY2, …) to `GRAY8`, `NV12` or `BGRA8` yourself; always deliver **one side-by-side image**, `2·eye_width × eye_height`, left eye left, unmirrored, even extents. | The runtime owns format conversion, decimation, rectification (R2) and transport; one SBS layout for every vendor |
-| Calibration is the **ACTIVE** device's — the unit you weave for — keyed by its own identity (serial). RAW, OpenCV convention: `k[eye] = fx fy cx cy`, `distortion_model` + coefficients, and `R`, `T` with **`x_R = R·x_L + T`** (T in mm; a right camera at +x of the left has `T = (−B, 0, 0)`). The runtime converts it to the client's pose-of-the-right-camera form. Return `XRT_ERROR_FEATURE_NOT_SUPPORTED` and do **not** set `CALIBRATED` if you cannot resolve it. | One test box carried eleven calibration folders; "the first one" is the wrong camera |
+| Calibration is the **ACTIVE** device's — the unit you weave for — keyed by its own identity (serial). RAW, OpenCV convention: `k[eye] = fx fy cx cy`, `distortion_model` + coefficients, and `R`, `T` with **`x_R = R·x_L + T`** (T in mm; a right camera at +x of the left has `T = (−B, 0, 0)`). The runtime converts it to the client's pose-of-the-right-camera form. Return `XRT_ERROR_FEATURE_NOT_SUPPORTED` and do **not** set `CALIBRATED` if you cannot resolve it. `image_width/height` = the size the intrinsics were calibrated at; the service rescales them to the frame's eye size when the two differ. Unless you set `NATIVELY_RECTIFIED`, the service rectifies from exactly these numbers (R2, `u_stereo_rectify`), so a wrong calibration shows up as misaligned rows in `displayxr-cli camera probe --rectified`. | One test box carried eleven calibration folders; "the first one" is the wrong camera |
 | `device_identity` is private: the service hashes it per consumer (`persistentId`) and never exports it. | Fingerprinting (spec §7.5) |
 | Nothing throws across the boundary. | C ABI |
 

docs/roadmap/stereo-camera-source.md

@@ -1,7 +1,8 @@
 # Stereo camera source — browser integration, vendor plug-in notes, phased plan
 
-**Status:** design (2026-09-25); **R1 implemented** on `feat/stereo-camera-r1` (2026-09-26) —
-see §E; maintainer decisions for R1 in §G. Decision: [ADR-043](../adr/ADR-043-stereo-camera-source.md).
+**Status:** design (2026-09-25); **R1 implemented** on `feat/stereo-camera-r1` (2026-09-26),
+**R2 implemented** on `feat/stereo-camera-r2` (2026-09-26, stacked) — see §E; maintainer decisions
+for R1 in §G. **L1** (Leia provider) is on `displayxr-leia-plugin` `feat/stereo-camera-l1`. Decision: [ADR-043](../adr/ADR-043-stereo-camera-source.md).
 Extension: [`XR_DXR_stereo_camera`](../specs/extensions/XR_DXR_stereo_camera.md). Consumer
 driving it: the web SDK's 3D call module (`@displayxr/inline3d/call`, RFC 0002 on
 `displayxr-web` branch `feat/call-p1`, §4 *Capture*).
@@ -194,7 +195,7 @@ The implementation belongs in `displayxr-leia-plugin` (`src/drv_leia/`), per ADR
 |---|---|---|---|
 | **R0** | runtime | This design: ADR-043, spec, this page | Maintainer sign-off on the open questions below |
 | **R1** ✅ | runtime | **Implemented 2026-09-26 (see "R1 as built" below).** Header `XR_DXR_stereo_camera.h` + `index.json` note (together — the catalog lint requires both); `xrt_plugin_iface` slots + `XRT_PLUGIN_IFACE_HAS_STEREO_CAMERA` (appended after `create_dp_d3d11_lift`, so **lands after ADR-042's PR**); platform-neutral camera manager (thread, refcount + linger, 3-slot pinned ring, per-stream wake handles, decimation, NV12/BGRA conversion); IPC + OpenXR entry points; sim_display fake; `displayxr-cli camera list/calib/probe`; selftest check | `camera probe` on the sim fake writes frames on Windows + Linux CI; ring/pinning unit tests (mirror `tests_lift_mailbox`) |
-| **R2** | runtime | `u_stereo_rectify` (Bouguet, valid-region crop, LUT remap) with golden tests against OpenCV-generated fixtures; RECTIFIED output | sim fake: row error < 0.5 px, disparity = f·B/Z within 0.5 px |
+| **R2** ✅ | runtime | **Implemented 2026-09-26 (see "R2 as built" below).** `u_stereo_rectify` (Bouguet, valid-region crop, LUT remap) with golden tests against OpenCV-generated fixtures; RECTIFIED output | sim fake: row error < 0.5 px, disparity = f·B/Z within 0.5 px — **measured 0.061 px worst block, 0.063 px worst disparity error** |
 | **R3** | runtime | Privacy: OS consent check, consent store + tray prompt, delegating-client registration, visibility/lock suspension, indicator, kill switches | Manual matrix: allow/deny/revoke, OS switch off, lock screen, background app — each blocks frames |
 | **L1** | leia-plugin (Windows) | Slots over the SR raw-camera channel per §D; calibration by active serial; keep-alive; repin runtime (feature-macro repin, `downstream-pins.json` `features` track) | On a panel box, incl. the multi-folder box: `camera probe --rectified` rows aligned; the vendor call app running at the same time still gets every frame; tracking/weave unaffected (frame-time + tracking-state logs) |
 | **B1** | browser (Windows) | §B device factory + duplicate hiding + hint + delegating registration in the installer; web SDK: prefer the hint in `camera:'auto'`, fill `hello` from it | RFC 0002 P1 call between two panel laptops sends rectified SBS from the tracker camera **while both are tracking** |
@@ -230,6 +231,64 @@ against the sim fake).
   sync_file as the #1699 weave path), a read-only POSIX section, and a distorted/misaligned fake
   (all with R2/R3/A1).
 
+### R2 as built
+
+- **`auxiliary/util/u_stereo_rectify.{h,c}`**, no OpenCV. Three layers so a GPU path replaces only
+  the last: geometry (`u_stereo_rectify_compute`: OpenCV's `stereoRectify(CALIB_ZERO_DISPARITY,
+  alpha = 0)` in C), float maps (`u_stereo_rectify_build_map`, what a GPU backend uploads as an
+  RG32F texture), and the CPU backend (a fixed-point bilinear LUT over the whole SBS image; NV12 =
+  full-res Y LUT + half-res UV LUT). RADTAN5 / RADTAN8 / KB4. Calibration at a different size than
+  the frames is rescaled. The alpha = 0 crop is verified on every border pixel of the real maps.
+- **Service:** a per-camera `struct scam_rectifier` built at manager create from the plug-in's
+  RAW calibration (WARN line with f, principal point, baseline, zoom, build time). It runs once
+  per source frame on the camera thread with the lock released, only while a started stream wants
+  RECTIFIED; RAW streams read the original. `get_calibration(RECTIFIED)` returns the same
+  geometry (one pinhole for both eyes, pure +x baseline); `horizontalFovDeg` is the rectified one.
+  The browser is refused (`INPUT_UNSUPPORTED`) a camera the rectifier rejected rather than being
+  given the RAW-flagged fallback. INFO stats add `rectify ms/frame`.
+- **Fake:** `SIM_DISPLAY_FAKE_STEREO_CAMERA_DISTORT=1` — the raw distorted, vertically misaligned,
+  rolled pair with exact ground truth (spec §10).
+- **Tests** (`tests_stereo_rectify`, fixtures in `tests/fixtures/` from
+  `gen_stereo_rectify_fixtures.py`, OpenCV 4.10, offline): R1/R2 to 1e-12, principal point to the
+  converged reference within 1e-4 px (and OpenCV's own P1 within 0.75 px — OpenCV stops
+  `undistortPoints` at 5 iterations), maps to 3e-5 px, no black pixel, and end to end on the fake:
+  raw |Δy| median 2.9 px → rectified worst block 0.061 px; disparity 11.33 / 37.77 px ground truth
+  vs 11.31 / 37.79 measured (worst block 0.063 px off).
+- **CLI:** `camera calib --rectified` prints P1/P2 and `Z = f·B/d`; every probe reports row
+  alignment and the dominant disparities as depths; `probe --rectified` exits 5 unless rows align.
+  Against the fake: |Δy| median 0.016 px, depths read back 2.004 m and 0.599 m.
+- **Perf:** 0.94 ms GRAY8 / 1.36 ms NV12 / 2.25 ms BGRA8 per 1280×480 frame, M1 Pro, -O2, one
+  thread (3 ms GRAY8 in a Debug service). At 30 Hz that is ~3 % of one core for the Leia GRAY8
+  source; **budget: ≤ 2 ms per frame on the camera thread**, above which the rows split across
+  threads (`u_stereo_rectify_lut_apply_rows`) or the GPU backend takes over. Setup (geometry + both
+  LUTs) 9–18 ms, once per calibration; LUT memory 6 bytes per output sample (3.7 MB + 0.9 MB).
+- **Online vertical-alignment refinement** (`auxiliary/util/u_stereo_vrefine.{h,c}` + the service
+  worker). Why: on a Leia SR laptop the rectified rows were still +1.71 px apart, and OpenCV's own
+  `stereoRectify` on the same frame and calibration gave the same +1.71 px — the stored calibration
+  is slightly off for the frames, not the rectifier. Model `dy = a + b·(y − cy)` (right − left,
+  output px); the x (roll) term measured 0.04 px / 100 px on that data and is left out. A worker
+  measures rectified frames the camera thread hands it (Shi-Tomasi patches on a 2× downscale, NCC
+  over d × dy, full-res NCC refinement ±2 px with parabolic sub-pixel; gated on NCC ≥ 0.9, d > 0,
+  unclamped peaks, real vertical curvature), fits robustly (median start, Tukey IRLS; the slope
+  only when rows spread ≥ 40 px), and when the correction moves by > 0.2 px rebuilds the maps with
+  it folded into the geometry: `v_offset = a/f`, `v_slope = b`, split symmetrically as a per-eye
+  vertical affine in normalized rectified coordinates **before** the principal point and the
+  alpha = 0 crop are computed, so P1/P2 describe the corrected frames (still one pinhole, pure +x
+  baseline) and `calibrationGeneration` bumps on the swap. Duty cycle: every 250 ms for 5 s after
+  an open or an update, then one ≥ 3-frame window every 30 s; 2.7 ms per measurement at -O2 on the
+  worker (< 0.4 ms per frame amortised at 30 Hz), 8 ms per rebuild. Safety: |a| ≤ 6 px,
+  |b| ≤ 2 px / 100 px, ≥ 50 matches over ≥ 3 frames, re-verified on every re-open, WARN on first
+  application and INFO after, `DXR_STEREO_CAMERA_REFINE=0` kill switch. Fake:
+  `SIM_DISPLAY_FAKE_STEREO_CAMERA_DY` / `_DY_SLOPE` (px, px per 100 px) inject an uncalibrated
+  misalignment. `probe` prints the state, the applied a/b and dy before → after.
+  Tests (`tests_stereo_vrefine`): the fake with the SR model (+2.6 px, −0.47 px / 100 px) converges
+  to +0.000 px (probe matcher, 338 blocks) in one rebuild; flat / uncorrelated / vertical-stripe
+  frames never apply; clamps hold. **Real SR frame** (`tests/fixtures/stereo_vrefine_sr_raw.jpg`,
+  62 KB, + the device calibration + 63 OpenCV SIFT matches as an independent reference): SIFT
+  signed dy median +1.715 px → **+0.075 px**.
+- **Not in R2:** the GPU backend (seam in place), and a rectifier for a camera whose calibration
+  changes at runtime (the refinement's reset hook is there for it).
+
 ## F. Risks
 
 - **Undocumented vendor channel.** The raw-camera shared memory is a reader-defined contract with