Skip to content

feat(lift): NeurD-backed 2D→3D lift slots on the D3D11 DP - #283

Merged
dfattal merged 21 commits into
mainfrom
feat/lift-main
Oct 1, 2026
Merged

feat(lift): NeurD-backed 2D→3D lift slots on the D3D11 DP#283
dfattal merged 21 commits into
mainfrom
feat/lift-main

Conversation

@dfattal

@dfattal dfattal commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

NeurD-backed 2D→3D lift on the D3D11 display processor

Lands the plug-in half of XR_DXR_lift (runtime PR DisplayXR/displayxr-runtime#1781, ADR-042). 20 lift commits, rebased onto main, plus the Windows runtime re-pin.

What it adds

  • leia_lift_neurd.{cpp,h}: the five D3D11 DP lift slots and a lift-only DP factory, backed by a dynamically loaded NeurD.dll (located via HKLM\SOFTWARE\LeiaInc\NeurD). NeurD absent ⇒ caps UNAVAILABLE and nothing is loaded.
  • Knobs under HKLM\SOFTWARE\DisplayXR\Leia\Lift (env wins): Backend (default directml), VideoModel, DepthGain (default 2.0), Dilate (default 2), InteractiveMin, ViewGain, ConvGain.
  • NeurD headers are private and fetched at build time from LeiaInc/media_sdk at NEURD_SDK_REF (v0.4.6); nothing private is committed. Without them the plug-in builds without lift.
  • docs/lift-neurd.md.

Compatibility

  • Plug-in ABI stays v5; the slots are appended and struct_size-gated, so this DLL loads on older runtimes (they never read the slots).
  • Compiles the slots only against runtime headers that define XRT_DP_D3D11_HAS_LIFT, so the Windows pin moves from v2.16.9 to v2.22.0 (the release carrying runtime#1781). The installer's derived minimum-runtime floor becomes 2.22.0.
  • CI caveat: LEIALOFT_GITHUB_TOKEN cannot read LeiaInc/media_sdk (404), so the GitHub-hosted build compiles lift OUT (warning only). The signing runner can read it: a signed build of this branch (codesign-runner run 36930017941) carries lift slots WIRED. Releases ship the signed installer; fixing the token is a follow-up.

Verification

  • Built locally against the lift runtime: DLL carries lift slots WIRED; displayxr-cli selftest PASS incl. lift_caps (neurd-directml), lift caps READY with NeurD 0.4.6.
  • Panel-tested via the DisplayXR Browser lift builds (video + photo, depth gain, dilation, no vertical parallax).
  • No lift code path can throw into the runtime (lift_convert is wrapped).

🤖 Generated with Claude Code

dfattal and others added 21 commits October 1, 2026 14:24
@dfattal @claude
feat(lift): NeurD-backed 2D->3D lift module for the D3D11 DP
9689da8 
Dynamic NeurD.dll loader (PATH -> NEURD_PATH -> HKLM\SOFTWARE\LeiaInc\NeurD),
one process-wide NeurD instance activated on a background thread (licence
network failure reported as ACTIVATING and retried), per-DP handle with
ref-count + stream table, LeiaMeet-style device bridge (shared textures on
NeurD's device, pack/unpack compute shaders, event-query drains), mode ->
NeurD property mapping, tracked-eye -> NeurD viewpoint mapping.

Plug-in-owned types only; the DP maps the runtime's lift contract onto it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dfattal @claude
feat(lift): wire the four NeurD lift slots into the D3D11 DP
3f6f00e 
lift_get_caps / lift_stream_create / lift_stream_destroy / lift_convert,
compiled only when the runtime headers announce XRT_DP_D3D11_HAS_LIFT
(runtime feat/lift-ext); struct_size-bounded reads of every versioned
struct. Tracked eyes come from the existing SR eye path. The lift handle is
created at DP create (reads DXR_LEIA_LIFT* once, loads nothing) and destroyed
before the weaver.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dfattal @claude
docs(lift): NeurD lift architecture, threading, licensing, bridge, kn…
5bf1544 
…obs, verification

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dfattal @claude
fix(lift): caps never waits behind an in-flight convert
5a96158 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dfattal @claude
fix(lift): match the runtime's landed lift contract (xrt_dp_lift.h)
8ce86fe 
- NVIEW is one row, view 0 leftmost (was a square grid); max_views 8 so the
  row fits D3D11's 16384 limit at 1440p.
- DEPTH returns a single-channel R8_UNORM texture (R8 unpack shader).
- strength 0 = flat (only negative falls back to 1.0); inpaint non-zero =
  blur fill.
- input_scale 1 = native; DXR_LEIA_LIFT_SCALE, when set, overrides streams.
- lift_convert_blob (GAUSSIANS) left NULL.
- docs: contract deltas, convergence-units gap, lift-DP weaver caveat.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dfattal @claude
chore(lift): assert lift mode/state values against the runtime's
8d39657 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dfattal @claude
refactor(lift): build against the real NeurD headers, fetched from pr…
cb6537a 
…ivate media_sdk

Remove src/drv_leia/leia_neurd_abi.h. It was a hand-written mirror of NeurD's
public header — enum values, struct layouts and the function-table order only;
no NeurD code was ever included. Leia's NeurD header is proprietary, so the
public plug-in now fetches the REAL headers at build time instead, the same way
it fetches the SR SDK:

- NEURD_SDK_REF (v0.4.6, a media_sdk release tag; headers byte-identical to
  dev@25a713d93) + NEURD_SDK_REPO pinned in scripts/build-windows.bat and
  build-windows.yml, kept equal by check_sr_pins.py.
- scripts/fetch-neurd-headers.ps1 pulls NeurD.h, _NeurD_detail.h,
  _NeurD_table.h via gh (CI: LEIALOFT_GITHUB_TOKEN) and generates
  NeurD_version.h from NeurD_version.h.in + the ref's project VERSION, into
  the gitignored NeurD-SDK-<ref>/include.
- Fetch failure is SOFT everywhere: CMake compiles the lift module out
  (DXR_LEIA_HAS_NEURD undefined), the DP's lift slots stay NULL, the runtime
  reports lift unavailable, the rest of the plug-in is unchanged.
- leia_lift_neurd.cpp still loads NeurD.dll dynamically (GetProcAddress
  NeurD_load, PFN_NeurD_load) and calls through the returned table using the
  real header's version-gated inline wrappers — no import lib, and any NeurD
  signature drift is now a compile error.
- docs/lift-neurd.md + CLAUDE.md: private-repo access prerequisite, pins,
  soft-failure behaviour, win-box build steps.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dfattal @claude
feat(lift): lift-only D3D11 DP factory + resync to the final lift header
3f4dfc5 
- create_dp_d3d11_lift (xrt_plugin.h, ADR-042): a DP with NO SR weaver, window,
  tracker or lens control — only the NeurD lift slots; every other slot NULL.
  Filled when the iface header has XRT_PLUGIN_IFACE_HAS_D3D11_LIFT_FACTORY and
  the NeurD headers were fetched; otherwise NULL (runtime falls back).
- convergence is now the runtime's relative depth in [0,1] (<0 = auto), mapped
  to NeurD's [-0.2,0.2] as K*(c-0.5), K = DXR_LEIA_LIFT_CONV_GAIN (default
  0.4; uncalibrated, calibration recipe in docs/lift-neurd.md).
- focal_px (appended) ignored; lift_convert_blob stays NULL.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dfattal
fix(lift): log NeurD's activation/init callback lines at WARN until R…
d449868 
…EADY — the only place NeurD says why init failed was dropped at the default log level
@dfattal @claude
feat(lift): run on NeurD 0.3.11+ — get_dx_device optional, own bridge…
d911a98 
… device fallback

NeurD 0.3.12 is the officially supported 2D->3D line but lacks
NeurD_get_dx_device (introduced 0.4.3), which the lift module hard-gated on.

ABI facts (media_sdk tags v0.3.12 / v0.4.6):
- The function table (_NeurD_table.h) is append-only: 0.3.12's 27 entries are
  an exact prefix of 0.4.6's 36. NeurD_load in 0.3.12 does no version check
  and returns its table with version = 0.3.12, so a plug-in built against the
  0.4.6 header loads it, and LEIA_NEURD_HAS (version test first) correctly
  reports the 0.4.x entries absent.
- 0.3.12 fills create_stream, destroy_stream, convert_stream_dx, set_backend,
  get_backend, shrink_memory_pool, set_logger_callback, set_prop_1i/1f, init.

Changes:
- activation gate: drop get_dx_device; need 0.3.11+ (the stream API).
- setup_nd_device_locked: use get_dx_device when present; otherwise create the
  plug-in's own D3D11 device on the default adapter (as NeurD's example does),
  flagged nd_dev_owned so the setup-failure path Releases it
  (release_nd_dev_locked). One WARN names the case + LUID.
- caps: drop NVIEW once READY on a NeurD without interactive convert
  (< 0.4.5); interactive_unavailable is now atomic since caps reads it
  lock-free.

Caveat: in 0.3.12, media_sdk_convert_dx is compiled only under USE_TRT (CUDA):
it GetDevice()s the caller's input buffer, binds CUDA-D3D11 interop to that
device's adapter, and returns an R8G8B8A8_UNORM texture on the same device —
which the existing stage_output texture branch already copies into the
bridge. The DirectML build's DX path is unverified, so on 0.3.x the service
must run with DXR_LEIA_LIFT_BACKEND=cuda, and the default adapter must be the
NVIDIA GPU. 0.3.x is SBS/DEPTH only, default models (NeurD_init). Documented
in docs/lift-neurd.md. Not compiled or run on Windows yet.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@dfattal @claude
feat(lift): DXR_LEIA_LIFT_INTERACTIVE_MIN — opt-in interactive gate f…
9b6114f 
…or the 0.4.4 internal-interactive NeurD package

The NeurD-Runtime-0.4.4-internal-interactive package reports version 0.4.4 but
carries convert_stream_dx_interactive. In media_sdk the interactive API landed
in 5b4faa8 (NEURDW-80) as a 35-entry table that is an exact prefix of 0.4.6's,
with the entries tagged 0.4.5; neither the v0.4.4 nor the v0.4.5 tag has them,
so this is a dev-branch build and the header's 0.4.5 INTRODUCED_IN blocks it.

Only NeurD_load is exported, so GetProcAddress cannot discriminate, and the
slot cannot be probed (a stock 0.4.4 table has 31 entries; index 33 is past
it). Hence an explicit, demo-only opt-in:

  DXR_LEIA_LIFT_INTERACTIVE_MIN=0.4.4  (service env; clamped to >= 0.4.4)

When set, the interactive availability test uses it instead of
NEURD_convert_stream_dx_interactive_INTRODUCED_IN, with one WARN at
activation. On a stock 0.4.4 this reads past the table and crashes — the WARN
and docs say so. The NEURD_UNAVAILABLE_OUTDATED_RUNTIME fallback in the
convert path is unchanged; no other gate is touched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@dfattal @claude
fix(lift): DEPTH polarity — NeurD is near=high, the spec's RELATIVE i…
c729013 
…s larger=farther; flip in the R8 unpack

Panel-verified: DEPTH results came out inverted downstream (background in
front in the browser's explore mode). The plug-in reports depth_semantics 0
(RELATIVE), which XR_DXR_lift defines as "larger = FARTHER, no unit", but
NeurD's DEPTH output is disparity-like, near = HIGH (measured: foreground mean
236, background 1 in the R8 result). Consumers trusting the label (browser
SendDepth flips RELATIVE to nearness) therefore inverted it.

Flip at the bridge: kUnpackR8Cs now writes 1 - v into the R8_UNORM UAV, so
the result obeys the spec. SBS/N-view unpack untouched. The convergence map
uses the runtime's RELATIVE parameter, not NeurD's texture, so it is
unaffected; nothing else in the plug-in reads the DEPTH texture.

Known gap: the texture-result branch of stage_output (only NeurD 0.3.x's
CUDA DX path) copies DEPTH as-is and is NOT flipped; commented + documented.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@dfattal @claude
fix(lift): DXR_LEIA_LIFT_INTERACTIVE_MIN calls the table slot directl…
e8f65d7 
…y — the header's inline wrapper re-checks 0.4.5

Panel result on bdb0e0f: the override activated ("interactive viewpoints
available"), but the first convert logged "NeurD interactive convert
unavailable" and N-view failed 5/5. The v0.4.6 header's inline
NeurD_convert_stream_dx_interactive does its own
NEURD_FUNC_IS_UNAVAILABLE check (version >= 0.4.5) before touching the slot,
so a 0.4.4-reporting runtime got NEURD_UNAVAILABLE_OUTDATED_RUNTIME and the
plug-in latched interactive_unavailable.

Add nd_convert_stream_dx_interactive(): when the override admitted a runtime
older than NEURD_convert_stream_dx_interactive_INTRODUCED_IN
(g.interactive_direct_slot, set at activation) it calls
nd->convert_stream_dx_interactive(stream, input, xyz, size, fmt, output)
directly — the PFN signature from _NeurD_detail.h, i.e. the wrapper's
arguments minus nd; otherwise the wrapper, so the OUTDATED_RUNTIME latch on
the normal path is unchanged.

Docs: the runtime polls lift_get_caps only until READY, so a convert-time
latch does not remove NVIEW from the runtime's cached caps — known limit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@dfattal @claude
fix(lift): NeurD cannot tile beyond SBS through the public API — drop…
9b7af2e 
… NVIEW from caps, derive view_count from the output

Panel result on bd6b52b: an N-view probe (--views 4) came back as a plain
2-tile SBS (4096x1362) stamped view_count=4. In media_sdk (v0.4.6, same as the
dev build) Config::adjust() runs on every convert via adjust_cfg_for_image()
— including the interactive path — and re-derives tiled_cols/rows from
output_type (SBS 2x1, TB 1x2, DEPTH 1x1), and NeurD_set_prop_1i(OUTPUT_TYPE)
accepts only 0..2. So OUTPUT_TILES_W/H are dead through the public API and
N > 2 tiles per convert is impossible until a tiled output type exists.

- caps: never advertise NVIEW (pre-READY or READY).
- stream_create(NVIEW): refused with a WARN naming the reason.
- convert: check NeurD's actual layout (output aspect / input aspect = tile
  count) against the mode's expected tiles and drop a mismatching frame, so a
  result can never be delivered under the wrong label.
- Interactive SBS viewpoint path unchanged.
- docs: N-view section + asks for Leia media_sdk.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@dfattal @claude
feat(lift): lift knobs from HKLM\SOFTWARE\DisplayXR\Leia\Lift (env st…
a4468c0 
…ill wins) — a respawned service no longer loses them silently

The lift knobs lived only in displayxr-service.exe's environment. Any respawn
(tray relaunch, HKLM Run at logon, crash restart) starts the service with the
logon env, and the knobs vanished silently — the only tell was "interactive
viewpoints UNAVAILABLE". Registry gates, not env vars.

- Each knob now resolves env > HKLM\SOFTWARE\DisplayXR\Leia\Lift (REG_SZ,
  opened with KEY_WOW64_64KEY) > default. Values: Backend, InteractiveMin,
  Scale, ViewGain, ConvGain — same grammar as DXR_LEIA_LIFT_BACKEND /
  _INTERACTIVE_MIN / _SCALE / _VIEW_GAIN / _CONV_GAIN. DXR_LEIA_LIFT (the
  kill switch) stays env-only.
- One parser per knob shared by both sources (knob_lookup returns the raw
  UTF-8 string + its source; parse_backend/parse_scale/parse_float_in/
  parse_interactive_min own the grammar). InteractiveMin is now read with the
  other knobs at DP create instead of ad hoc at activation.
- kick() takes the first acquirer's knobs; activation logs ONE WARN with
  every knob's effective value and source.
- scripts/set-lift-knobs.bat (elevated): Backend=directml +
  InteractiveMin=0.4.4, --clear deletes the key.
- docs: Registry value column, why, exact reg add lines.

No behaviour change with no key and no env. The gain knobs now parse
strictly (strtof, whole string) where atof accepted trailing junk.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@dfattal @claude
feat(lift): VideoModel knob -- pick NeurD 0.4.6's metric video model
e036d3f 
DXR_LEIA_LIFT_VIDEO_MODEL / HKLM\SOFTWARE\DisplayXR\Leia\Lift VideoModel:
'fast' (default, NEURD_MODEL_VIDEO_RELATIVE_FAST, unchanged) or 'metric'
(NEURD_MODEL_VIDEO_METRIC_QUALITY) for NeurD_init_with_options' video slot,
so the metric real-time model can be A/B'd for fps and quality without a
rebuild. Photo streams keep the relative photo model (the only one that slot
accepts). The activation knobs WARN reports it with its source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dfattal @claude
feat(lift): log which viewpoints a convert used (tracked / explicit /…
8a73cd9 
… default)

SBS only sends viewpoints when the eyes are tracked (or given explicitly);
otherwise NeurD renders its default pattern and DXR_LEIA_LIFT_VIEW_GAIN is
never applied. A ViewGain A/B run with nobody in front of the tracker
therefore produced byte-identical output on NeurD 0.4.6 and read as 'the
interactive path ignores our viewpoints'. One WARN per change of source
makes the tell explicit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dfattal @claude
feat(lift): DepthGain knob, default 2.0 -- NeurD gain = strength x De…
cda7f99 
…pthGain

NeurD's GAIN_MULTIPLIER 1 read as too weak on the Leia panel; strength 2
(set live via dxr-lift-strength) was judged right as the default (David,
2026-09-26). The element's strength keeps meaning 1 = nominal, and the
calibration lives here with NeurD's units: DXR_LEIA_LIFT_DEPTH_GAIN /
HKLM\SOFTWARE\DisplayXR\Leia\Lift DepthGain (default 2.0), reported in the
activation knobs WARN. (ViewGain is not a depth control: NeurD bounds the
viewpoint excursion by its own disparity budget, so gain 2 = gain 1.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dfattal @claude
fix(lift): never pass the tracked eye HEIGHT to NeurD (vertical paral…
874a553 
…lax)

leia_lift_neurd_map_viewpoint mapped the eye y offset into NeurD's
viewpoint, so with the viewer above the display centre NeurD rendered the
lifted frame as seen from above: the image shifted vertically with head
height ('the converted asset jumps up/down') and a mirror-filled band
appeared at the top edge. Weave dumps on the panel: both eyes shifted
+7 px (top) / -7 px (bottom) vs the element, where the untracked probe
shows NeurD's own +/-4 px. A lenticular panel has horizontal parallax only,
so y is zeroed. (x keeps the eye midpoint = horizontal look-around.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dfattal @claude
feat(lift): Dilate knob, default 2 (NeurD's own default is 3)
ad147ee 
NEURD_PROP_DILATE_RADIO was never set, so NeurD dilated the disparity map by
its default 3 px and the foreground visibly grew past its silhouette on the
panel (David, 2026-09-26: 'the disparity map dilation is a bit too much,
reduce a bit'). Now set every convert (cached like the other props) from
DXR_LEIA_LIFT_DILATE / HKLM\SOFTWARE\DisplayXR\Leia\Lift Dilate, default 2;
reported in the activation knobs WARN.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dfattal @claude
build(windows): pin the runtime to v2.22.0 — first release with the D…
f86015d 
…3D11 lift slots

The NeurD lift module compiles only against runtime headers that define
XRT_DP_D3D11_HAS_LIFT (runtime#1781, released in v2.22.0). The slots are
appended and struct_size-gated, so the plug-in ABI stays v5; the installer's
derived minimum-runtime floor moves to 2.22.0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@dfattal
dfattal marked this pull request as ready for review October 1, 2026 21:49
@dfattal
dfattal merged commit 4460347 into main Oct 1, 2026
16 checks passed
@dfattal
dfattal deleted the feat/lift-main branch October 1, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dfattal