Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 110 additions & 13 deletions docs/adr/ADR-046-depth-aware-cursor.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ADR-046: Depth-aware cursor — opt-in only; the app knows the depth, the runtime places the cursor

**Status:** Proposed (2026-10-06) · Phase 1 implemented · Phase 3a implemented (Metal) · Amendment 1 (lifted content, 2026-10-10) accepted · Amendment 2 (one policy, runtime draws, app chooses the look, 2026-10-11) accepted · spec:
**Status:** Proposed (2026-10-06) · Phase 1 implemented · Phase 3a implemented (Metal) · spec v3 anchor modes (2026-10-07) · Amendment 1 (lifted content, 2026-10-10) accepted · Amendment 2 (one policy, runtime draws, app chooses the look, 2026-10-11) accepted · spec:
[XR_DXR_cursor_depth.md](../specs/extensions/XR_DXR_cursor_depth.md) · sibling of
[ADR-040](ADR-040-rear-depth-budget.md) (the same cue conflict, at the cursor instead of the desktop)

Expand Down Expand Up @@ -94,8 +94,9 @@ everything from the views that call has just produced:
- **E**, the cyclopean eye, is the midpoint of that pair.
- A point's depth is **t**: its distance in front of E along the display normal, divided by S's
distance. So t = 1 on the canvas, and t < 1 in front of it.
- The sprite goes at **C = E + t·(S − E)**, with height scaled by t. Because C stays on the
cyclopean ray, the cursor never slides sideways as it rises.
- The sprite's height is scaled by t, so its apparent size is constant. Where it goes
laterally is the anchor mode (§7). The default, HYBRID, puts it at **C = E + t·(S − E)**
while the pointer moves: on the cyclopean ray, so it never slides sideways as it rises.

Because only located views are used, the result is correct for:

Expand All @@ -120,7 +121,7 @@ The eye compares disparities, so the margin, clamp and slew rates are expressed

| Parameter | Default | Why |
|---|---|---|
| margin in front of content | 0.03 | about 2 mm crossed on screen at 65 mm IPD: reads as in front without looking detached |
| margin in front of content | 0.03 | about 2 mm crossed on screen at 65 mm IPD: reads as in front without looking detached. Spec v3 briefly lowered it to 0.005 after a head-tracked-panel test; Amendment 2 A.2 restores 0.03. Apps can override it (`XrCursorDepthOptionsDXR::margin`) |
| clamp | [−0.6, +0.6] — **superseded by Amendment 2: [−0.6, 0], never behind the glass** | t ≥ 0.625: never more than about 3/8 of the way to the eye |
| rise time constant | 30 ms | never lag behind content that comes forward, or the violation is visible |
| sink time constant | 250 ms | no flicker when the footprint crosses an edge |
Expand Down Expand Up @@ -154,7 +155,7 @@ checks, `DISPLAYXR_CURSOR_DEPTH_UV=u,v` scripts the cursor position. Atlas captu

| Cursor UV | Over | Cursor disparity between views | Meaning |
|---|---|---|---|
| 0.5, 0.5 | the cube | −23.4 px; the cube's nearest edge there is about −15 px | in front of the cube |
| 0.5, 0.5 | the cube | −23.4 px with the original 0.03 margin; the cube's nearest edge there is about −15 px (for spec v3's 0.005 margin, see §7) | in front of the cube |
| 0.5, 0.65 | the floor grid, which the app doesn't report as content | 0 px | on the display plane |
| 0.1, 0.1 | empty space | 0 px | on the display plane |

Expand Down Expand Up @@ -260,6 +261,82 @@ slightly nearer cube edge. A run without the request, in hint mode and with
`DISPLAYXR_CURSOR_DEPTH=0`, logs none of the Phase 3a one-time WARNs. A run with the request logs
each of them exactly once.

### 7. Spec v3: anchor modes, and why HYBRID is the default

**What the tester saw.** On a real head-tracked panel, the v1/v2 cursor read as "at the glass"
even when its disparity put it on the content. The cause is a cue conflict of its own. The sprite
sat on the cyclopean ray through the pointer's canvas point (§2), so each eye's image of it on the
glass was S ∓ (baseline/2)·d whatever the head did. That is disparity with **no motion parallax**.
On a head-tracked display everything else parallaxes as the head moves, so a cursor that doesn't
reads as lying on the one surface that also doesn't: the glass. A world-fixed cursor read
correctly, but sat a millimetre or two beside the click point.

**The modes** (`XrCursorDepthOptionsDXR::anchorMode`, a new struct chained on the hint; no
existing struct grows). With f the display normal away from the viewer and e = dot(S − E, f):

| Mode | Sprite centre C | Aim | Parallax |
|---|---|---|---|
| SCREEN (v1/v2) | E + t·(S − E) | exact, always | none: reads as at the glass |
| WORLD | S − f·(1 − t)·e | ~1–2 mm off for an off-axis viewer, always | correct |
| **HYBRID (default)** | moving: SCREEN, and store the line of sight (E₀, S₀, D₀ = e); still: E₀ + s·(S₀ − E₀), s = 1 − (1 − t)·e / D₀ | exact while moving, and on the click point while still unless the head moves | correct while still |

"Moving" means `cursorUV` changed by more than 1e-4 since the last moving placement. HYBRID
re-anchors on the first hinted locate and whenever the time filter re-primes; the anchor lives
beside the filter in the session and is touched only by a hinted locate, so §0 holds.

**The anchor is the line of sight at the last pointer move, not a point.** While the pointer is
still, C is the point on the anchor-time line E₀ → S₀ whose distance in front of the glass,
(1 − s)·D₀, equals the current depth (1 − t)·e. Three properties follow, all unit-tested:

- **Continuity:** on the frame the pointer stops, the geometry is the anchor's, so s = t and C is
the last moving placement exactly. No jump.
- **Parallax:** a lateral head move at fixed t leaves e, and so s and C, unchanged. The sprite is
world-fixed and parallaxes like content at its depth.
- **No drift from depth changes:** when the content under a still pointer changes depth, C slides
along the anchor-time line, so from the eye that aimed it, it stays on the click point.

An earlier draft of the rule stored the line's *foot* on the glass and lifted C from it along the
display normal. The headless captures below exposed the flaw: a depth change under a still
pointer then moved the sprite off the click point by about |Δd| × the viewer's lateral offset
(27 px for sim_display's viewer, 0.10 m above the canvas centre, when the scripted pointer
anchored at d = 0 on frame 1 and the cursor then rose to the cube). The web SDK takes the same
line-of-sight rule (inline3d 1.37.1).

**Why HYBRID beats both pure modes.** While the pointer moves, the user is aiming, and the eye
tracks the cursor, not the head; there SCREEN's exact aim is what matters and a parallax deficit
over a few frames is invisible. While it is still, the user is looking at the scene, often moving
their head to see around it, and the cursor must parallax like the content it rests on; there
WORLD's behaviour is right. The only cost is that the drawn sprite drifts from the click point as
the **head** moves while the pointer is still: its image on the glass moves by about |d| × the
head's lateral displacement. For content near the glass (d ≈ −0.005 with the new margin), 20 cm of
head motion moves it about 1 mm. The next mouse movement re-aims it exactly. Hit testing is
unchanged: the click is always at the pointer's canvas point.

**Evidence** (macOS, sim_display SBS, `cube_handle_metal_macos`, `DISPLAYXR_CURSOR_DEPTH_UV=0.5,0.5`,
atlas 1512×1646, two tiles of 1512×823 stacked; disparity = cursor x in the right-eye tile minus
the left-eye tile; the cube's nearest edge under the cursor is about −15 px). The scripted pointer
never moves, so HYBRID anchors on frame 1 at d = 0 and the cursor then rises to the cube with the
pointer still: the depth-change case.

| `DISPLAYXR_CURSOR_DEPTH_ANCHOR` | Target d | Cursor disparity | Cursor y in tile (pointer at 411) |
|---|---|---|---|
| unset (options not chained → HYBRID) | −0.040 | −16.6 px | 411 |
| `hybrid` | −0.040 | −16.6 px | 411 |
| `screen` | −0.040 | −16.6 px | 411 |
| `world` | −0.040 | −16.6 px | 438 |
| *foot-anchored draft of HYBRID* | *−0.040* | *−16.6 px* | *438* |
| *before v3: 0.03 margin, SCREEN (§5)* | *−0.066* | *−23.4 px* | — |

All sit just in front of the cube's edge, about 7 px nearer it than with the old margin. HYBRID
stays on the click point like SCREEN; WORLD sits 27 px below it for this off-axis viewer. In every
capture the midpoint of the two cursor images is at the requested u (0.4995).

**A deliberate default change.** An app that doesn't chain the options struct gets HYBRID and
the 0.005 margin, which is different from v2. It is tester-driven, both changes go the same way
(the cursor reads as on the content), and an app that wants the old placement chains
`XR_CURSOR_DEPTH_ANCHOR_MODE_SCREEN_DXR` with `margin = 0.03`. The web SDK's depth cursor
implements the identical rule with the same numbers.

## Consequences

- An app that already raycasts or reads depth gets a correct, consistent cursor for one struct
Expand All @@ -272,14 +349,20 @@ each of them exactly once.
that to the edge itself. Phase 3a as built does not remove this lag. It reads frame N's
submitted depth, but the result reaches the placement one or two frames later, because waiting
on the GPU would stall the pipeline (§6). So it has the same lag, and the same mitigation.
- **Head motion and look-around.** The sprite sits on the cyclopean ray through the cursor's
canvas point. So each eye sees it on the glass at S ∓ (baseline/2)·d, which doesn't depend on
where the head is (unit-tested across three head poses). The cursor's image on the panel
therefore cannot swim with tracking motion or jitter. Only the content under the line of sight
changes as the user looks around, and the hit test is redone every frame. This is deliberate
SCREEN anchoring, not world anchoring. A mouse is a 2D screen-space device, so the cursor slides
over a surface under head motion instead of sticking to it. If it stuck to the surface, the
click target would drift on the glass as the head moved.
- **Head motion and look-around (revised in spec v3, §7).** v1/v2 anchored the sprite to the
screen: on the cyclopean ray, so each eye saw it on the glass at S ∓ (baseline/2)·d whatever
the head did (still unit-tested, now as the SCREEN-mode property). The reasoning was that a
mouse is a 2D screen-space device and its click target must not drift. Testing on a real
head-tracked panel showed the price: a sprite with disparity but no motion parallax conflicts
with everything around it that does parallax, and reads as at the glass. The default is now
HYBRID: screen-anchored while the pointer moves (exact aim), and anchored to that line of
sight while it is still (correct parallax, unit-tested: the sprite does not move in locate
space as the head moves laterally).
The click target never drifts, because hit testing stays at the pointer's canvas point. Only
the drawn sprite drifts, by |d| × head displacement (about 1 mm for 20 cm of head motion over
content near the glass), and only until the mouse next moves. A depth change under a still
pointer does not drift it: the anchor is the line of sight, not a point (§7). SCREEN and WORLD remain
available through `XrCursorDepthOptionsDXR`.
- An app-drawn cursor shows the app's frame latency, not the hardware cursor's. Phase 2's late
cursor read narrows that but cannot remove it. This is inherent to any cursor that has
disparity.
Expand Down Expand Up @@ -476,6 +559,20 @@ sprite and the depth, and the runtime owns the drawing. Apps get the same contra
runtime as the sprite (B.1). Pages then style the depth cursor with plain CSS, and the SDK
needs no cursor code for it.

#### §A as built in the runtime (#1847, `XR_DXR_cursor_depth` spec v4)

- `u_cursor_depth_target()` is A.1 verbatim; the upper clamp is 0 whatever the tuning says
(`max_disparity` defaults to 0 and a value > 0 is treated as 0). Margin default 0.03 (A.2),
still overridable through `XrCursorDepthOptionsDXR::margin`.
- A.3 is `u_cursor_depth_should_draw()` (threshold −0.005). The state tracker reports it as
`XrCursorDepthPlacementDXR::isActive`, whose meaning was already "draw nothing, show the OS
cursor", so no struct changed. The filter and the HYBRID anchor step on every locate, drawn or
not, so the hand-back is smooth. `disparity` / `targetDisparity` stay filled as diagnostics.
- The 0.5 px floor is left to the drawing backend (Part B): the placement does not know the
panel's pixel pitch, and −0.005 baselines (~0.33 mm at 65 mm IPD) is above 0.5 px on every
current panel.
- A.5 (hybrid line-of-sight anchor) was already the v3 default and is unchanged.

### Rollout

1. **Groundwork:**
Expand Down
Loading
Loading