Skip to content

feat(lift): honour the runtime's viewpoint policy (rect-relative, baseline, y/z, per-stream knobs) - #317

Merged
dfattal merged 2 commits into
mainfrom
feat/lift-viewpoint-policy
Oct 8, 2026
Merged

dfattal merged 2 commits into
mainfrom
feat/lift-viewpoint-policy

Conversation

@dfattal

@dfattal dfattal commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Part of DisplayXR/displayxr-runtime#1858, addresses #316 (P1).

Plug-in half of Phase 1 of the lift camera design (window-aware lookaround). The runtime now owns the viewpoint policy (rebase to the lifted rect, ipd/parallax factors, axis mask, clamp, recentering ease — runtime ADR-048); the plug-in only translates units.

Depends on DisplayXR/displayxr-runtime#1864 for the new fields (XRT_DP_LIFT_HAS_VIEWPOINT_POLICY, branch feat/lift-viewpoint-policy), but works without it: against runtime headers that lack the macro the new code compiles out, and against a runtime binary that passes a short xrt_dp_lift_params::struct_size the behaviour is byte-for-byte today's (x only, 63 mm unit, ±3 clamp).

What changes

  • New pure mapping module src/drv_leia/leia_lift_viewpoint.{h,c} (platform-neutral C, no NeurD/SR/D3D):
    • Legacy: x_n = clamp(G·x/0.063, ±3), y_n = z_n = 0 (old runtime, or eyes from the DP's own tracker).
    • Policy: unit = the nominal 63 mm (NeurD's ±0.5 default pair), so the runtime's ipd factor shows through: x_n = G·x/0.063; y_n = Gy·G·y/0.063 when axis_mode ≥ XY; z_n = Gz·(z − z_ref)/z_ref (clamped ±1) when axis_mode == XYZ; x/y clamp ±G·(max_offset_m + baseline_m/2)/0.063 when the runtime clamped (baseline 0 → 63 mm), else ±3. baseline_m is used only for that clamp margin and the log.
  • DP (leia_display_processor_d3d11.cpp): reads the appended fields with LEIA_LIFT_COVERS (gated on the last field, viewpoint_frame), #ifdef XRT_DP_LIFT_HAS_VIEWPOINT_POLICY; z_ref = the panel's nominal viewing distance from the cached SR geometry (leiasr_geometry_get), i.e. the same value the runtime receives via get_display_info and pins z to; unknown → 0.5 m (the runtime's default). static_assert that the axis values match the runtime's.
  • Per-stream knobs: ViewGain, new YGain (default 1.0) and ZGain (default 0.5, signed), plus ConvGain/DepthGain/Dilate/Scale are re-read (env > HKLM > default) and snapshotted at each stream create, so a registry change applies to the next stream without a service restart. Process-level knobs (enable, backend, versions, video model) unchanged. Stream-create WARN lists the snapshot.
  • Logging: one WARN per stream per contract change (viewpoints = runtime policy (frame rect, axis X, baseline …) / legacy mapping (…)); one process-wide WARN the first time vertical look-around is honoured (vertical parallax used to cause jumps).
  • Docs (docs/lift-neurd.md): viewpoint mapping rewritten (fixes the stale "y_n = clamp(G·y/0.063)" claim), z assumption documented, knob table gains YGain/ZGain/DepthGain/Dilate/VideoModel, knob lifetimes, Verifying modes=0x3 (was 0x7; NVIEW is never offered).
  • Host test src/drv_leia/tests/test_lift_viewpoint.c, built + run by the Linux CI lane (same pattern as test_display_claims_win).

Decisions worth a reviewer's eye

  • Nominal unit, not baseline_m: the runtime already applied ipdFactor to the eyes; dividing by the post-factor baseline would cancel it (every pair → ±0.5). A 32 mm pair maps to ±0.254·G.
  • Clamp margin: the runtime clamps the eyes' midpoint; each eye is baseline_m/2 off it, so the per-eye limit adds that half-pair (otherwise the pair collapses at the edge). A runtime clamp wider than ±3 is honoured.
  • z mapping is fractional distance change (not baseline units) and uncalibrated; conservative default gain 0.5, sign flippable.

Verified

  • Plug-in DLL builds against the runtime branch headers (policy path compiled in) and against runtime origin/main (policy compiled out).
  • test_lift_viewpoint passes (MSVC /W4 /WX locally; Linux CI runs it).

Owed (hardware, panel + NeurD)

  • Old runtime + this plug-in: lift SBS on the panel looks and logs as before (viewpoints = legacy mapping (… runtime predates the viewpoint policy)).
  • Policy runtime, default axis X: lookaround is now relative to the window/rect centre; log viewpoints = runtime policy (frame rect, axis X, baseline ~6x mm …).
  • Axis XY: vertical lookaround with no jumps; honouring VERTICAL look-around logged once; tune YGain if needed.
  • Axis XYZ: lean in/out — sanity-check NeurD z sign/magnitude; tune ZGain.

🤖 Generated with Claude Code

dfattal and others added 2 commits October 8, 2026 15:51
…inal unit, y/z, per-stream knobs)

Plug-in half of lift Phase 1 (DisplayXR/displayxr-runtime#1858, #316 P1;
runtime side DisplayXR/displayxr-runtime#1864). The runtime now owns the
viewpoint policy (rebase to the lifted rect, ipd/parallax factors, axis mask,
clamp, recentering — runtime ADR-048); the plug-in only translates units.

- leia_lift_viewpoint.{h,c}: pure mapping, host-tested
  (tests/test_lift_viewpoint.c, Linux CI). Legacy contract unchanged (x only,
  63 mm, +-3). Policy: the unit stays the NOMINAL 63 mm (NeurD's +-0.5 pair)
  so the runtime's ipd factor shows through (a 32 mm pair -> +-0.254);
  honour y on axis XY/XYZ (YGain); map z on XYZ as the fractional distance
  change from the panel's nominal viewing distance (ZGain, default 0.5,
  clamped +-1); map the runtime's midpoint clamp to
  +-ViewGain*(max_offset + baseline/2)/0.063 so the full eye pair survives at
  the edge. baseline_m is used only for that margin and the log.
- D3D11 DP reads the appended xrt_dp_lift_params fields with
  LEIA_LIFT_COVERS behind XRT_DP_LIFT_HAS_VIEWPOINT_POLICY; a short
  struct_size or older runtime headers keep today's behaviour exactly.
- Per-convert knobs (ViewGain, YGain, ZGain, ConvGain, DepthGain, Dilate,
  Scale) are re-read and snapshotted per stream at stream create.
- docs/lift-neurd.md: mapping rewritten (stale y claim fixed), z assumption,
  knob table + lifetimes, Verifying modes=0x3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
glibc's <math.h> does not define NULL, so the host-run test_lift_viewpoint
target failed to compile on every Linux lane ('NULL' undeclared). MSVC's
headers happened to pull it in transitively.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dfattal
dfattal force-pushed the feat/lift-viewpoint-policy branch from 1e259c6 to 911871b Compare October 8, 2026 22:51
@dfattal
dfattal merged commit caf61b9 into main Oct 8, 2026
16 checks passed
@dfattal
dfattal deleted the feat/lift-viewpoint-policy branch October 8, 2026 23:00
dfattal added a commit that referenced this pull request Oct 8, 2026
glibc's <math.h> does not define NULL, so the host-run test_lift_depth
target failed to compile on every Linux lane (same fix as #317's
leia_lift_viewpoint.c).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dfattal added a commit that referenced this pull request Oct 9, 2026
glibc's <math.h> does not define NULL, so the host-run test_lift_depth
target failed to compile on every Linux lane (same fix as #317's
leia_lift_viewpoint.c).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dfattal added a commit that referenced this pull request Oct 9, 2026
glibc's <math.h> does not define NULL, so the host-run test_lift_depth
target failed to compile on every Linux lane (same fix as #317's
leia_lift_viewpoint.c).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dfattal added a commit that referenced this pull request Oct 12, 2026
glibc's <math.h> does not define NULL, so the host-run test_lift_depth
target failed to compile on every Linux lane (same fix as #317's
leia_lift_viewpoint.c).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dfattal added a commit that referenced this pull request Oct 12, 2026
…9) (#318)

* feat(lift): metric depth export + NeurD off-axis camera (NeurD >= 0.4.9)

Plug-in Phase 3 (P2) of the lift camera / metric depth design
(DisplayXR/displayxr-runtime#1858, #316), plus adoption of NeurD's metric
off-axis camera. Runtime contract: feat/lift-app-rig-and-depth (ADR-048
Addendum A). NeurD side: LeiaInc/media_sdk #559 (per-stream props + float
depth) and #560 (convert_stream_dx_offaxis), both INTRODUCED_IN 0.4.9.

- Per-stream props: with NeurD_set/get_stream_prop_*, inpaint, auto-
  convergence, gain, dilation and convergence are set per NeurD stream
  (CONVERGENCE always overridden so each stream keeps its own auto result);
  a refusal falls back to the global path once, with one WARN.
- Depth export: caps report aux_outputs = AUX_DEPTH (METRIC under
  VideoModel=metric, else RELATIVE) when NeurD has get_stream_depth_dx on
  DirectML. Streams created with AUX_DEPTH set RETAIN_FLOAT_DEPTH; after each
  convert the R32F depth of the SAME inference is bridged to the caller's
  device like the SBS output (legacy-shared texture, same event-query drain)
  and reported by the new slot 33 lift_get_depth (XRT_DP_D3D11_HAS_LIFT_DEPTH):
  metric = metres, LINEAR scale 1 (#559's shader already divides S / raw);
  relative = raw 0..255, INVERSE scale 1/255. Intrinsics passed through (depth
  texels); convergence_depth = the depth at zero disparity: h0 = 0.5 - 5c
  (legacy renderer) or 1 - C/D (off-axis), metric S / (255 h0), relative 1/h0.
- Off-axis camera: with convert_stream_dx_offaxis and the runtime's viewpoint
  policy (RECT frame, rect size), SBS views use NeurD's metric Kooima camera:
  screen = rect in metres, eyes = the runtime's rect-relative viewpoints
  (z = distance from the rect plane, as given; app-rig sources reproduced
  without gain), nominal eye = the runtime's nominal_z_m. Relief: relative
  model ReliefDepthM (0.08 m, matches the calibrated legacy look) at the
  requested / NeurD-auto convergence; metric model matches the metric depth
  gradient at the screen (D = N / h0), capped by MetricReliefMaxM (0.3 m).
  OffAxis=0 forces the legacy dimensionless path; a refused call falls back
  for that frame.
- Both 0.4.9 features compile only when the fetched NeurD.h declares them
  (CMake detects each separately) and run only when the loaded NeurD's table
  has them, so NeurD 0.4.6-0.4.8 and the v0.4.6 header pin are unchanged.
- leia_lift_depth.{h,c}: the pure math, host-tested on the Linux lane
  (tests/test_lift_depth.c). docs/lift-neurd.md: depth export, off-axis
  camera, knob table (OffAxis, ReliefDepthM, MetricReliefMaxM), gating.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(lift): include <stddef.h> for NULL in leia_lift_depth.c

glibc's <math.h> does not define NULL, so the host-run test_lift_depth
target failed to compile on every Linux lane (same fix as #317's
leia_lift_viewpoint.c).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lift): use NeurD_convert_stream_dx_ex (views + depth of one call)

NeurD's 0.4.9 API was reworked in media_sdk #561 (supersedes #559/#560): the float depth
now comes back from the same convert call as the views, and the metric off-axis camera
is a camera option of that call.

- Build gates: DXR_LEIA_NEURD_HAS_STREAM_PROPS (NeurD_set_stream_prop_1i) and
  DXR_LEIA_NEURD_HAS_CONVERT_EX (NeurD_convert_stream_dx_ex) replace
  _HAS_FLOAT_DEPTH / _HAS_OFFAXIS. Run time still gates on LEIA_NEURD_HAS.
- A stream that needs depth (aux depth, or the metric off-axis scale) converts with
  want_depth: the OFFAXIS camera when it applies, else the dimensionless pattern
  (NeurD's default +/-0.5 pair when untracked). No RETAIN prop, no separate getter.
- Off-axis failures still fall back to the legacy camera for that frame. A depth-only
  _ex failure falls back to the plain convert with no depth for that frame.
- Builds against NeurD v0.4.6 headers compile all of this out, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(lift): pin NeurD headers to v0.4.9-DEV (NeurD_convert_stream_dx_ex)

media_sdk#561 merged to dev as fd9590f and is tagged v0.4.9-DEV, the first
tag whose NeurD.h declares per-stream props and NeurD_convert_stream_dx_ex,
so this build compiles both features in. Move to v0.4.9 once that release
tag exists; a version's -DEV / -QA / release tags carry the same headers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(lift): pin NeurD headers to the v0.4.9 release; resolve rebase onto main

NeurD v0.4.9 shipped to PROD on 2026-10-11 (media_sdk dev@20bf39e, the
commit that passed QA as v0.4.9-QA). Headers are identical to v0.4.9-DEV.
Also keep both sides of the leia_lift_neurd_caps conflict with #345
(module info + aux depth fields).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant