Epic: PlayCanvas engine as a second splat backend (unified gsplat + LOD streaming), Streamed SOG in the pipeline, and upstream contributions

[dfattal]

Why

@displayxr/inline3d renders Gaussian splats through one backend, Spark (three.js). Measured on the gallery's photo-lifted .sog assets (1,179,648 gaussians, SH0), Spark is coverage-bound: 50 % / 25 % decimation changes frame time by ~0 %, the bit-exact quad-extent trick has nothing to cut (Spark already draws at √8σ), and the only headroom is lossy overdraw knobs (−11…−22 %). It also has no LOD/streaming path we can use (enableLod/lod were inert on our assets, and it does not read lod-meta.json).

PlayCanvas has, all MIT: the engine's unified gsplat system with LOD streaming and a per-frame splat budget balancer (src/scene/gsplat-unified/: octree, lod-table, budget-balancer), Streamed SOG (splat-transform writes multi-LOD chunked SOG + lod-meta.json), the supersplat-viewer player and the SuperSplat editor. The SOG container is already our interchange format, and the maintainer (slimbuck) is receptive to our camera block proposal (playcanvas/splat-transform#319).

A splat budget is exactly the lever our measurements say matters (native: ~half the frame is per-gaussian; a ~600 k budget is visually free on photo lifts, 25 % breaks). Large captured scenes (rooms, venues) are where streaming + budget pay and where Spark's coverage cost hurts most.

Goal

1. PlayCanvas engine as a second splat backend in @displayxr/inline3d, behind the same handle API (rig waterfall from the SOG camera block, setFocus/pick, focus gestures, perf presets), rendering N views from the SDK's eye poses into the tile canvas so the DisplayXR Browser weaves it unchanged (XR_DXR_weave is engine-agnostic: it takes the canvas atlas).
2. Streamed SOG through the pipeline: producer (gallery worker) emits it for large scenes; the adapter streams with a frame budget tuned for two views on a 3D display.
3. Upstream contributions, as small human-authored PRs from David's account: the camera block in splat-transform (writer preserves/accepts it, reader/MetaV2, spec §2/§5, a CLI option to seed it from the training cameras.json), engine exposes meta.camera, and a DXR display path (N-view render + weave submit when available) in supersplat-viewer so published players work on 3D displays without our SDK in the page.

Phases

• P0 — Investigation (read-only, 1–2 days). Engine ≥ 2.x gsplat-unified API surface (GSplatComponent, unified/octree/lod-table/budget-balancer, lodUpdateInterval-style params), multi-camera render into viewports / render targets of one canvas, WebGL2 vs WebGPU in the DisplayXR Browser (Chromium M150 fork, D3D11 weave), how XrManager must stay OFF, engine footprint next to three.js, lod-meta.json schema. Output: a design note in docs/ with a go/no-go on WebGPU and on vendoring vs npm.
• P1 — Adapter. addSplat(…, { engine: 'playcanvas' }) (default stays Spark): hosts an engine Application on the tile canvas, one camera per view, consumes the SDK's rig (display/camera, eye poses, off-axis frusta, focus.point = orbit centre = pivot = convergence), reuses the shared SOG reader (inline3d-sog.js), maps perf presets to engine equivalents, owns its pivot the way 1.7 does (see the _applyTransform trap). Parity gates: rest view MAE vs the source left photo on the gallery bench asset (Spark scores 3.05/255); interleaved-config perf table on the M1 (GPU timer query) at 720p/1080p; woven check on the Windows DisplayXR Browser box.
• P2 — Streamed SOG. Worker option to emit Streamed SOG (splat-transform lod pyramid) for large scenes; gallery/show serve it; adapter streams with a 2-view frame budget; measure on a large captured scene (not a photo lift) vs the flat SOG. Keep the camera block on the chunked container (check it survives write-sog/LOD output — today a re-encode drops unknown keys).
• P3 — Native viewer (optional). displayxr-demo-gaussiansplat reads the base LOD of a Streamed SOG as a budgeted load (≈500–600 k) instead of the full sheet.
• P4 — Upstream. (a) playcanvas/splat-transform: camera block preserved on re-encode + accepted by the reader + spec text + --camera-from cameras.json[:index]; (b) playcanvas/engine: expose meta.camera (and use it as the initial camera when present); (c) playcanvas/supersplat-viewer: DXR display mode (feature-detect, render N views, submit to XR_DXR_weave), or the minimal embed hook if they prefer. Continue on #319; one PR per repo; no AI trailers on upstream commits.

Exit criteria

• Gallery Spatial View can switch backends with a query flag; rest view within 1 MAE point of Spark on the bench asset; no visible regression on the wall.
• A large captured scene streams and holds the frame budget on the Windows DisplayXR Browser box.
• Upstream PRs opened (merged is a bonus); camera block survives a splat-transform re-encode.

Non-goals

Replacing Spark; changing the SOG geometry planes; anything requiring a private fork of PlayCanvas.

Refs

playcanvas/splat-transform#319 · @displayxr/inline3d 1.7.0 (perf presets, camera block waterfall, setFocus/pick) · displayxr-demo-gaussiansplat #117/#118/#120 (measurements, camera rig, fit) · gallery SpatialView.tsx + src/lib/spatialView/camera.ts (the reference camera model).

P5 — Automatic 3D for existing PlayCanvas sites (investigation → prototype)

Two tiers, cheapest first:

1. Pages that already support WebXR (PlayCanvas apps with XR enabled; likely SuperSplat's published viewer — verify): the DisplayXR Browser already exposes the runtime as a WebXR device (immersive-vr), and the device owns XRView.projectionMatrix, so the page renders off-axis, screen-converged views without knowing it. Missing piece: a browser-side "View in 3D" affordance that starts the session with a synthetic user activation instead of waiting for the page's VR button. Engine-agnostic, no injection.
2. PlayCanvas pages that never enabled XR: every app constructs app.xr (XrManager); usability only depends on navigator.xr, which our browser provides. A browser-injected content script locates pc.Application.getApplication(), takes the active camera and calls app.xr.start(camera, 'immersive-vr', 'local'); the engine's own XR path renders both eyes through our projections. Engine-specific; needs a per-site kill switch + allowlist; expect breakage on custom render passes, post-effects, screen-space UI, camera controllers fighting the pose.

Open questions (same ones the SOG camera block answers for content): which rig by default (first-person → camera rig at world scale; product viewer → display rig fitted to the window — heuristic on camera target distance), and the scale/convergence default when the page declares nothing. Generic WebGL interception for non-XR, non-PlayCanvas pages is out of scope (the browser cannot know the camera).

Verified 2026-09-22 — superspl.at/scene/<id> (the hosted supersplat-viewer) is TIER 1. supersplat-viewer 1.32.0 (playcanvas ^2.22.1) ships a WebXR path: src/xr.ts gates a VR/AR button on xr.isAvailable('immersive-vr' | 'immersive-ar') and starts with xr.start(camera, 'immersive-vr', 'local-floor', …); viewer.ts reconfigures the camera on app.xr.on('start'). So in the DisplayXR Browser the page needs no changes: our WebXR device must (a) advertise immersive-vr, (b) support the local-floor reference space, (c) work with PlayCanvas XrManager on WebGL2; then the browser adds the auto-enter affordance. P5 milestone 1 = open a superspl.at scene on the Windows box, press its VR button, confirm a woven stereo view; then auto-enter. The hosted assets (…/splat/<id>/v2/{scene.sog,settings.json}) are 403 from S3 — hosted-only, which is exactly why "no SDK in the page" matters.

[dfattal]

P0 — investigation done (2026-09-22)

Landed: design note docs/playcanvas-backend.md, draft PR #37 (one file, not merged). Every load-bearing claim below was re-run by the supervising session, not just read.

Decision: GO on WebGL2, playcanvas as an optional npm peer dependency (>=2.22.3 <3, like three/Spark), no vendoring, no WebGPU for P1.

What the spike showed (headless Chrome 153 on the real GPU — ANGLE Metal, Apple M1 Pro — engine 2.22.3, ports_25.sog, 1280×720 SBS canvas, synthetic off-axis matrices):

• N views into one canvas works two ways, bit-identical (MAE 0.000): N cameras with rect + calculateProjection, or one camera with N RenderViews (the engine's own WebXR stereo path, driven without XrManager). The latter is the design: one gsplat manager / sort / LOD pass / work buffer for all views (gsplatCount 294,912 vs 589,824 with two cameras). A 0.2-NDC skew lands as exactly 64 px between the halves; control = 0 px.
• Engine bug on non-square pixels: the splat footprint uses one focal from the viewport width for both axes (gsplatCorner.js:29,51-60, still at HEAD). On a 2:1 SBS buffer every splat is half height → visible stipple/cracks, MAE 11.3 vs a square-pixel reference; a one-line gsplatCornerVS chunk override (y focal from viewport_size.y·P[1][1]) gives 1.67, no-op on square pixels. P1 ships the override; upstream it (new P4d, engine PR).
• Custom projection honoured in the draw (per view) and the vertex-shader clip; CPU sort is directional along the camera node's forward (projection-independent; exact for all our views since they share one orientation); LOD uses camera.fov, so the adapter must set fov from P[1][1].
• XrManager stays off by construction: build on AppBase with our own device (app.xr === null, xrCompatible:false); the engine's only requestSession is reachable from app.xr.start().
• Footprint: minimal splat viewer bundle 357 KB gzip vs 1,089 KB for today's three + Spark path. Tree-shaking works (sideEffects:false); bundler trap: mark node:worker_threads external; workers are Blob URLs (CSP worker-src blob:).
• camera block: the engine keeps unknown meta.json keys (resource.gsplatData.meta.camera) and unknown lod-meta.json keys (resource.data.camera), both verified after 240 frames. splat-transform drops it on .sog → .sog re-encode and its Streamed-SOG writer builds lod-meta.json from a fixed shape (verified by run). On chunked output the block belongs at the top level of lod-meta.json.
• Streamed SOG: splatBudget (global count, default 1M) enforced: a 3-level pyramid at budget 150k rendered 137,706. Uniform-decimation pyramids open holes on a photo lift at low budgets, as expected → P2 gates on a large captured scene, never the bench asset.
• Perf-preset mapping: minAlpha → alphaClipForward (default already 1/255); minPixelRadius → minPixelSize (engine default 2 px culls fine grain Spark keeps — set 0 for parity runs); quad extent σ / per-splat alpha radius → chunk overrides; decimation/LOD → splatBudget.

Facts that drifted since the brief: npm @displayxr/inline3d is 1.7.1 (released), gallery pins 1.7.0; engine latest 2.22.3 (published today); superspl.at hosted assets are 200, not 403 — and the Trogir scene (14bac5b2) is a public Streamed SOG (splat-transform 3.3.3, 45.7M gaussians, 7 LOD levels, chunked under CloudFront): a ready-made P2 large-scene target.

Parity harness is ready (supervisor): headless Chrome on the real GPU via puppeteer-core; Spark rest view on ports_100 scores MAE 2.96 on the gallery /bench route and 2.94 on a plain SDK page (addSplat bytes, rig:'auto' from the injected camera block) vs the source left photo. Both agree with the recorded 3.05, so P1 gates engine-vs-engine on the SDK page and confirms on the bench route.

P5 reframed (David): letting a PlayCanvas / superspl.at page enter the browser's converted immersive-vr session is not inline 3D — it is a fullscreen takeover with the browser's rig controls, synthetic controllers and HUD, and (finding from the P5 read) a constant viewer pose that leaves the engine's radial splat sort never re-sorting on head motion. That path is demoted to an engine-compatibility smoke test. The P5 deliverable is the inline shape: the page keeps its own canvas, layout and controls and renders N views at the inline-3d eye poses — the P1 adapter for SDK pages, and for pages without our SDK an in-engine inline display path (P4c in supersplat-viewer). Tier-2 injection would then mean driving the inline session per view, not app.xr.start; investigation only, after P1.

Phase plan / estimates (agent sessions, supervisor-verified):

• P1 adapter — 2–3 sessions. addSplat(…, { engine:'playcanvas' }), Spark default untouched; AppBase + one camera + N RenderViews; footprint override; perf presets mapped; rig/focus/camera block via the existing pure modules; gates (a) rest MAE within 1 of Spark on the SDK page + bench route, (b) interleaved GPU-timer table 720p/1080p M1, (c) woven eyeball on the Windows box (David). Then gallery ?engine=playcanvas (no default change), sdk-v1.8.0, gallery pin PR.
• P2 Streamed SOG — 2 sessions. Worker option (large scenes only), camera block at lod-meta.json top level, two-view budget, measure on Trogir + our own large scan vs flat SOG, M1 + Windows box.
• P3 native base-LOD load — 1 session, optional.
• P4 upstream — (a) splat-transform camera pass-through + --camera-from, ~1 session incl. the #319 thread; (b) engine meta.camera, ½; (c) supersplat-viewer inline display path, 2; (d) new: engine footprint fix, ½. Drafts for David's review; he posts.
• P5 — after P1, as reframed above.

NOT tested in P0: anything on the DisplayXR Browser (weave join of a PlayCanvas canvas, the #24 race, real inline-3d views); rest-view parity vs Spark (P1); GPU timings (P1); WebGPU; the engine GPU picker; lodMode:'error'; multiple apps per page / teardown; the gallery's Next.js bundle; engine versions other than 2.22.3.

[dfattal]

P1 — adapter landed as draft PR #38 (2026-09-22); gates (a) and (b) re-measured by the supervisor

What landed (p1/playcanvas-adapter, 7 commits, npm test 213/213): addSplat(wall, canvas, src, { engine: 'playcanvas' }) in a new js/inline3d-splat-playcanvas.js, dynamically imported only when asked (a Spark page makes 0 requests matching playcanvas, verified via puppeteer); AppBase + one camera with N RenderViews, the footprint chunk override, perf presets mapped (alphaClipForward, minPixelSize 0 for every preset but false, quad-extent σ patch, splatBudget), the rig waterfall / SOG camera block / focus through the existing pure modules (cloud walkers made backend-neutral), cameraRigFromPose on ./three. inline3d-viewer.js and inline3d.js untouched (0-line diff). Optional peer dep playcanvas >=2.22.3 <3. Docs: docs/playcanvas-adapter.md (design, divergences, gates). Version NOT bumped yet.

Gate (a) — rest view vs the source left photo, grey MAE /255, SDK page, ports_100_cam.sog (1,179,648 gaussians, v2 camera block), 1280×720, real GPU (ANGLE Metal, M1 Pro):

render	MAE
Spark (baseline)	2.94
PlayCanvas	2.25
PlayCanvas vs Spark	1.49

Passes (limit Spark + 1 = 3.94). Engine tonemap set to NONE (0.004 effect); minPixelSize 0 (0.004); N-camera fallback path 0.007 from the RenderView path.

Gate (b) — GPU ms/frame, EXT_disjoint_timer_query_webgl2, rest pose pinned, 300 frames after 60 warm-up, each config loaded ALONE in one Chrome process (on ANGLE Metal a timer query sees other canvases' work, so the agent's interleaved-pairs numbers were contaminated and are superseded by these), idle GPU:

config	720p median / p90	1080p median / p90
Spark exact	60.5 / 66.2	75.4 / 80.7
Spark balanced	51.8 / 56.0	64.5 / 72.7
PlayCanvas default (minPixelSize 0)	7.4 / 9.5	10.0 / 10.5
PlayCanvas balanced	7.1 / 9.5	9.1 / 9.9
PlayCanvas splatBudget 600k	7.4 / 9.7 (all 1,179,648 drawn)	—
Spark exact, 2-view SBS (2560×720 / 3840×1080)	120.0 / 136.0	162.7 / 179.0
PlayCanvas default, 2-view SBS	12.7 / 13.2	35.5 / 38.4

Reading: on this photo-lift asset the PlayCanvas quad path is ~8× cheaper than Spark in mono and ~5–9× in stereo on the M1. Spark is coverage-bound as measured before; the engine's CPU-sorted quads with the 1/255 clip cost far less fill. The splatBudget does nothing on a flat SOG — every splat is drawn; it is a Streamed-SOG lever (P2). Stereo costs PlayCanvas ~2× its mono (two full passes), Spark ~2× as well here. Caveat: headless Chrome vsync-free timer queries on one GPU; treat as ranking + order of magnitude, re-check on the Windows box.

Divergences from the P0 design worth knowing: no setCanvasFillMode/setCanvasResolution calls (they zero the canvas and write inline styles; the engine default is already fixed-resolution); every entity is created against its own tile's app (the engine's global "current app" made a second tile steal the first's splat — found by running, fixed, test-pinned); fov set via camera.setXrProperties (with RenderViews the fov getter returns XR properties); pick = nearest-centre fallback; feather not implemented (warns once); the handle returns before the module loads (early calls queued); three + Spark are still statically imported by ./splat (a PlayCanvas page still pulls them — follow-up: split the entry).

Pending: gate (c) woven eyeball on the Windows box (David); gallery ?engine=playcanvas PR (in progress, depends on sdk-v1.8.0); then the 1.8.0 release + gallery pin.

NOT tested: anything on the DisplayXR Browser (real inline-3d views, weave join, #24 race, preserveDrawingBuffer); Windows/Android; Streamed SOG or .ply through the adapter; real pointer input / pick accuracy; remove() teardown leaks, context loss; the N-camera fallback beyond one rest view; engine versions other than 2.22.3; the gallery bundle.

[dfattal]

Correction to the P1 gate (b) numbers above — read these instead. The EXT_disjoint_timer_query figures overstate Spark. Measured by wall-clock rAF cadence, each engine alone on an idle GPU (M1 Pro, vsync on, two repeats), same asset ports_100_cam.sog, same SDK camera rig:

config	rAF Hz	ms/frame
Spark exact, mono 1280×720	35.5	28.2
PlayCanvas default, mono	60.2 (vsync cap)	≤ 16.6
Spark exact, 2-view SBS 2560×720	17.0–17.2	58.3–58.8
PlayCanvas default, 2-view SBS	59–60 (vsync cap)	≤ 17

Honest ratio on this asset: stereo ≥ 3.5× (PlayCanvas is at the cap; its timer-query value of ~13 ms suggests ~4.5×), mono ≥ 1.7×. The 8× in the earlier table came from Spark's timer bracket reading ~60 ms while the page actually runs at 35 Hz; the PlayCanvas query values are consistent with cadence. Method trap for anyone repeating this: any other headless Chrome on the machine (e.g. a parallel agent's run) contaminates both methods — check pgrep -f headless=new before every run.

[dfattal]

Decision (David, 2026-09-22): the SDK's splat viewer DEFAULTS to PlayCanvas. Spark stays as the explicit engine:'spark' kill switch. Reworking transitions and params for parity is the priority now, on p1/playcanvas-adapter (PR #38), before the 1.8.0 release:

• default flip + Spark path becomes a dynamic import (a default page loads neither three nor Spark);
• parity of every documented option/handle method: feather, setFocus easing / setPose snap / resetPose ease with SceneViewer's exact constants, idleSpin, fit/margin/fitSweep/depthLimit, zoom/pitchLimit, renderScale, frame, pick (exact or GPU), sortIntervalMs mapped or documented;
• new from downstream port audits: captureFit: 'height'|'cover' (both backends), handle.engine escape hatch ({app, root, camera} — a page can add its own PlayCanvas entities, glTF/skinned included), handle.setSource(src, {fadeMs}) crossfade.
  Gallery: ?engine=… in SpatialView becomes the opt-OUT to Spark; the three-specific overrides (viewer._eye + frustumFor, monoCamera override, Spark objectModifier reveal, tilt) are retired or re-expressed through the public handle in a follow-up to dfattal/displayxr-gallery-pvt#73, rebased over the mono-only camera change landing in parallel.
  Consumers notified: the gallery GS session, and the two sessions embedding the SDK's splat viewer (they will port Photos first behind the engine switch; a mixed glTF+splat scene needs the escape hatch).

[dfattal]

P4 posted (2026-09-22, David's account): playcanvas/splat-transform#328 (optional camera block through meta.json + lod-meta.json, --camera-from cameras.json[:n]; follow-up posted on #319) and playcanvas/engine#9486 (per-axis focal for the gsplat footprint — the P4d fix behind our chunk override; WebGL GLSL + WGSL + WebGPU compute projector). Spec paragraph for developer-site is drafted and waits on the shape being accepted. P4c (supersplat-viewer inline display path) is not started.