Editor crash: main-thread stack overflow, repeating 7,344-byte frame pair (dump Unity.exe.23896)

[dfattal]

Reported by a partner integrator (Amazon Lab126 / LeiaViewer port, internal ticket UNITY-133).

2026-09-02 — SCOPE CORRECTED. This issue originally pooled six crash dumps under one signature. Investigation with the partner's agent established that they are at least three distinct bugs. This issue is now scoped to one: the main-thread stack overflow in dump Unity.exe.23896. The others are split out — see Not this bug below.

This bug: main-thread stack overflow (dump Unity.exe.23896)

Independently proven from the dump by the partner's agent, not inferred:

• rsp == StackLimit — the stack is genuinely exhausted, not merely deep.
• A 7,344-byte frame pair repeating — unbounded recursion, one frame pair per iteration.
• Access violation in mono-2.0-bdwgc.dll writing the page just below rsp — mono's stack probe against the exhausted stack. A later dump re-faults inside mono's vectored exception handler, itself running on the already-dead stack.
• No DisplayXR native frames on the crashing stack, and no managed exception logged.

Attribution correction

An earlier revision of this issue presented that triage under the heading "Reporter's dump triage". The partner's agent did not write it and has asked for that on the record. Their own dump report stated only the faulting module; the proof above came from their later, independent analysis. The heading was wrong about the source — the content stands for dump 23896 and is not a valid description of "the #264 crash" generally, which is what the split below fixes.

Leading hypothesis: a stale SR window subclass (not yet confirmed)

From the runtime engineer, based on a prior recorded observation (2026-08-17), not on this incident:

The SR SDK subclasses the weaver's HWND via SetWindowLongPtr(GWLP_WNDPROC) and looks its context up by window.

• Benign form — the weaver context is destroyed at shutdown but the subclass is not removed. SR's wndproc stays in the chain; every message to that HWND hits it, finds no context, logs WeaverBaseImpl: no handler for this window, and passes on.
• Fatal form — a new weaver is created on the same HWND while the stale subclass is installed. The chain becomes self-referencing: SR wndproc -> CallWindowProcA -> SR wndproc -> ... until the stack is gone. Presents as c000041d FATAL_USER_CALLBACK_EXCEPTION with ~2500 SimulatedRealityDirectX frames, and no [TERMINATE]/[EXIT], because it dies inside the user callback.

This would make the repeating frame pair native SR wndproc frames, not managed ones — worth stating plainly, because the "managed recursion" reading in the original triage is an interpretation of the repeat, not an identification of it. The symbol on the repeating frame decides this and has not yet been read.

Runtime PR #1012 already applies the correct discipline on the service path (flush retired DPs bound to a window before creating a new one; destroy-before-recreate; prefer rebinding via setWindowHandle). The Unity provider uses the in-process _handle path, so the open question is whether that discipline is applied there at all.

Field evidence gathered 2026-09-02 (plugin v2.16.0)

Signature C reproduced post-shutdown, non-fatally. After 12 LoadScene Home/Viewer cycles with a live session and a normal Play-stop — no scripted resizes, so not a layout artefact:

provider:  GfxStop -> Lifecycle Stop -> Lifecycle Shutdown -> 0 further DisplayXR-PROV lines, ever
weaver:    WeaverBaseImpl: no handler for this window - 46 lines over ~6 min, THEN STOPPED
pattern:   groups of 3, ~3ms / ~3ms / ~100ms within a group, clusters every 5-30 s
editor:    alive and healthy throughout; no escalation

Message-driven rather than frame-driven (an idle editor is not rendering at 60 fps), surviving complete provider shutdown, and self-limiting — all consistent with the benign form above. Historical fatal runs reached 5,955 lines of the same string and ended with the editor's main window gone.

Ruled out on the plugin side (source-verified, no rig time):

Candidate	Result
Managed code driving native after teardown	None possible. DisplayXRGameViewRehostWatcher.Poll and DisplayXRPreviewClose.Poll both return on !isPlaying (PreviewClose's only native call is behind that gate). DisplayXRDisplayFramingPreview is the one thing that starts at Play-stop and hooks per-camera render callbacks — but makes zero native calls.
Our editor cadence matching the ~100 ms tail	No. Ours are 0.25 / 0.75 / 2.0 s.
Self-recursive managed members	None (5 grep hits, all false positives).
Camera.Render() inside a render callback	Zero live call sites.
Managed log callback recursing (logMessageReceived etc.)	Dead both sides — none in this plugin; partner grepped their Assets/, third-party, and PackageCache: zero hits.

Hypotheses tested and falsified

Recorded so they are not re-proposed:

1. Game-view resize during Play produces a texture realloc. 24 scripted resizes, 9 realloc cycles, no crash.
2. LoadScene during a live session reaching realloc from another direction. 12 loads, no crash — and a weak null: the pane size never changed, so reconcile_size() had nothing to reconcile and the path was hit once, not twelve times. The 2D/3D view-count switch alone (submit=1-view vs submit=2-view, same 1468x748) does not trigger a bridge reallocation.
3. Managed log-callback recursion. Zero hits in either tree.

Not this bug — split out

Dump	Signature	Issue
Crash_2026-09-01_0244	Unity analytics — GetRenderingResolution under DeviceInfoEvent::CollectExtraInfo under MainMessageLoop. 13 flat frames, no mono, no recursion; zero stack overflow/recursion hits in the whole log.	#282
LeiaViewer.35556	Player-side D3D12 shader library, CLibrary::Serialize.	#283
Crash_2026-08-31_0656	40000015 in AssetImportWorker6 — a different process; likely a fourth thing. Log pull pending.	TBD

Note for #282: the render-target layer is not ruled out there. It dies reading rendering resolution on a box where a DisplayXR session had been changing the render target. That is an observation, not a mechanism.

Next — in value order

• Read the symbol on the repeating frame in Unity.exe.23896. Decides managed-vs-native recursion outright and ends the guessing.
• GetWindowLongPtrW(hwnd, GWLP_WNDPROC) on the editor's main window after Lifecycle Shutdown, mapped to a module. If it lands in an SR module, the stale-subclass mechanism is confirmed. One call, non-invasive.
• Is the SR module (and DisplayXRClient.dll) still mapped post-shutdown? If the subclass survives but the code does not, that is a dangling wndproc and the next message is fatal, not the 2500th.
• Play, Stop, Play escalation test. If each cycle adds a subclass layer, the rate should escalate rather than reset. Cheap, reversible, and distinguishes stale-subclass from a stuck loop without needing the fatal event.
• Does the in-process _handle path apply PR #1012's destroy-before-recreate discipline?

Trap: do not read SRSession handle count as evidence. It leaks unconditionally on this platform — monotonic +2 per 10 s on a fully idle box with no SR app attached (7.8k vs ~400-650 for siblings). Use it to exclude, never to implicate.

Related: displayxr-leia-plugin#217 (the neighbouring unbounded-burst string, Weaving is not possible because window handle is invalid).

[dfattal]

Control test: the wedge does NOT reproduce without DisplayXR

The escape route is closed. This stays in our tracker.

Method — fresh-process condition verified, since a DisplayXR session leaves a window subclass behind that outlives its own shutdown by 25+ minutes (#284), so disabling the plugin in an already-contaminated editor would produce a false negative:

Unity fully quit first           verified: 0 Unity processes before launch
XR init disabled from the start  m_InitManagerOnStart: 1 -> 0 (backed up, restored after)
never entered a session          DisplayXR-PROV: 0 | GfxStart OK: 0 | Session start OK: 0
                                 SimulatedReality* modules loaded: 0

Sequence — the same one that wedged, with resizes folded in: Home.unity → Play → 6x (LoadScene Viewer → resize Game view → LoadScene Home) → Stop → CloseMainWindow(). 12 scene loads and 6 resizes with a live player loop.

Result:

WeaverBaseImpl / invalid-handle lines :  0 / 0
Play-stop                             :  clean, editor healthy
CloseMainWindow()                     :  EXITED CLEANLY in < 20s
Temp/UnityLockfile after exit         :  absent
crash dump                            :  none

Same rig, same day, plugin active: the identical close gesture produced main window gone, process alive, ~1.5 cores spinning, log ending mid-WeaverBaseImpl, lockfile stuck, no dump.

What this establishes, and what it does not

Established	The wedge does not occur without a DisplayXR session in the process. One sample, but a clean negative on the exact gesture that wedged.
NOT established	That the plugin's own code is at fault. Every measurement says the hang is mono-side, with zero SR frames and zero CallWindowProc across 221 threads. "Requires DisplayXR to have run" is not the same as "is DisplayXR's bug" — Unity or the SR SDK mishandling something the session leaves behind fits the evidence equally well.

This does not revive #284's refuted fatal attribution. The wedge stack still had no SR frames. It removes the "nothing to do with us" branch and nothing more.

Retracting an argument I was leaning on

I have repeatedly cited "no DisplayXR frames on the crashing stack" as evidence against our involvement. That argument is weaker than I was treating it.

The innermost resolved frame in the wedge was mono_runtime_invoke — managed code was executing and the managed frames below it did not resolve, which is why the managed stack was unrecoverable on that box. Our managed code would be equally invisible in that capture. The absence is a property of the unwinder, not of the callee.

So it rules out our native code being on the stack, and says nothing about our managed code. Candidates that survive Play-stop and keep running at editor shutdown, no longer excluded:

• DisplayXRGameViewRehostWatcher.Poll and DisplayXRPreviewClose.Poll — both [InitializeOnLoad] statics subscribed to EditorApplication.update. Both early-return on !isPlaying, so neither is known to do anything at shutdown — but they are running.
• DisplayXRRigManager — a static registry that rigs register into and out of.

No claim that any of these hangs. This is the withdrawal of an exclusion, not the proposal of a mechanism.

Next

• Repeat the wedge gesture under DISPLAYXR_PROV_EXTERNAL_WINDOW=1. If it does not wedge, the hang is tied to the docked child-glue/texture path specifically — which narrows it hard and makes that env var a workaround for the crash as well as for SR weaver subclass survives session teardown on Unity's container window (confirmed) — the editor-killer's mechanism #284's burst. If it does wedge, the hang is independent of the window path.
• The managed stack, if anyone has a box with a mono JIT map. It remains the single highest-value unread datum.

Closed on this issue

Unity.exe.23896's three reads all came back mono, not SR (see #284) — different pair return addresses, ~1 MB stack reserve. So 23896 and this wedge sit together, and neither belongs to #284.