SOG meta.json: optional camera block (rest pose + pixel intrinsics) so a .sog is self-describing

[dfattal]

A .sog handed to a third-party viewer carries no viewpoint. Today the camera lives only in the viewer layer — settings.json / the sse-bootstrap inline block (supersplat-viewer/src/schemas/v2.ts, cameras[0].initial: {position, target, fov}) — which write-html.ts writes as a sibling of the .sog, never inside it. splat-transform in.ply out.sog (no --viewer-settings, which is HTML-only per the README) produces a file with no viewpoint at all, and so does SuperSplat's own SOG export (an editor-exported .sog has keys version,count,means,scales,quats,sh0,shN and no sibling settings.json).

Two different things are being conflated. settings.json is presentation — an authored framing, correctly kept out of the data. But a capture camera is intrinsic to the data: the gaussians were unprojected through it. Rendering a splat built at focal f_s through a frustum of f_v scales the image by f_v/f_s about the centre, so a viewer that guesses gets a silent zoom error. fov alone can't express it either — it has no principal point, so a deconverged or cropped capture is unrepresentable.

We generate SOGs from calibrated stereo pairs (single-photo 3DGS prediction), where the splat origin is the left capture camera and the correct rest view needs an off-axis frustum. We currently carry those numbers in our own database and re-attach them client-side, which means the .sog alone renders as intended for nobody else. The shape we need is the one INRIA's sidecar cameras.json already uses (utils/camera_utils.py::camera_to_JSON: position, rotation, pixel fx/fy, width/height) — established practice, just never inside a container.

Proposal: one optional top-level key in meta.json, sibling of count, version unchanged:

"camera": {
  "convention": "opencv",
  "rest": { "position": [0, 0, 0], "rotation": [0, 0, 0, 1] },
  "intrinsics": { "fx": 1194.666, "fy": 1194.666, "cx": 1024.0, "cy": 576.0, "width": 2048, "height": 1152 },
  "stereo": { "baseline_m": 0.063 }
}

• rest: position in splat units, rotation xyzw camera→world; convention names the camera axes (opencv = +x right, +y down, +z forward) since a viewer must know which half-turn to apply.
• intrinsics in pixels of the source image with explicit width/height, so the principal point survives.
• stereo optional (second eye at +x·baseline_m).
• ~250 bytes. Purely advisory; a viewer that doesn't want it ignores it.

This is already spec-legal — §5 says "readers should ignore unrecognized fields", read-sog.ts only checks version !== 2, the engine's sog.js cherry-picks fields — and we have shipped SOGs with an extra key for months without a reader complaint. The one thing that makes it need upstreaming rather than a private key: write-sog.ts rebuilds metaObj from scratch, so any re-encode silently drops it.

Happy to send the PR (writer preserves/accepts the block, MetaV2 type, spec page §2/§5) if the shape is acceptable. Related: #38 (SOG v2 format proposal); orthogonal to supersplat#924, which is the presentation layer.

[dfattal]

Follow-up after implementing this on our side (producer + two viewers) — two additions to the proposed block that fell out of real use, so that a viewer can open the file correctly and not just know the lens:

"camera": {
  "convention": "opencv",
  "rig": "camera",
  "rest": { "position": [0,0,0], "rotation": [0,0,0,1] },
  "intrinsics": { "fx": 1194.67, "fy": 1194.67, "cx": 1024, "cy": 576, "width": 2048, "height": 1152 },
  "stereo": { "baseline_m": 0.063 },
  "focus": { "point": [0, 0, 1.68], "subject_m": 2.14, "near_m": 0.73, "far_m": 66.2 }
}

• focus — the producer's scene facts a viewer cannot recover from the geometry alone: point is the focus / orbit centre in the rest camera's space (for a stereo or head-tracked viewer it is also the natural zero-parallax plane), subject_m the salient depth along the rest axis, near_m/far_m a robust depth range. Without it a viewer has to guess an orbit centre from the cloud, and on an open scene lifted from a photograph (a fixed grid of gaussians with a huge far sky) the AABB centre is tens of metres out; the median of 1/z works, but the producer knows better (it had the depth map).
• rig — "camera" (open at the capture camera, conserve its frustum) or "display" (object-centric: orbit the subject; rest is then just the preferred initial viewpoint). Presence of a camera does not by itself say which framing the author intended — a turntable object scan has cameras too.

Two things we verified that may be useful to the discussion: unknown top-level keys survive every current reader (engine, splat-transform, Spark), so the block is safe to carry today — the only loss is on re-encode through write-sog.ts; and when rest is the capture camera the intrinsics are latent in the geometry (the P1/P99 angular extent of the cloud about the origin recovered the true focal to 0.3 % on our test asset), so a viewer can estimate them — but a principal-point shift and the focus cannot be recovered that way, which is why the block still earns its place.

[slimbuck]

(Human writing here).

I am actually not averse to this, but wondering what you use the camera poses for at runtime?

It would definitely make sense to have poses available along with PLY data, but at runtime I don't really see the purpose?

[dfattal]

Fair question. To be clear, I'm not asking to carry the training trajectory at runtime, just one camera: its intrinsics and its pose relative to the scene, giving a hint developers / viewer apps can use to set the starting / default / rest view point.

You might be familiar with the new class of tools that lift a mono or stereo photograph into a splat. The associated point clouds are frustum shaped and only look right from near the camera that took the picture. You can see an example on displayxr-gallery.vercel.app with the "Spatial View" option, which shows SOGs generated from pictures. In that case the app knows the camera intrinsics and extrinsics because it keeps the pre-conversion data in its database. But if you want to export the SOG and view it somewhere else, that information needs to travel with the file, ideally without a sidecar.

But even for a regular surround capture, the author of a splat may want to convey a default view as a hint to viewer apps how to set the starting / rest viewing posiiton. SuperSplat's settings.json solves exactly this, but since it's a sidecar the answer is gone the moment the .sog travels on its own.

When the block is absent, readers already ignore unknown keys and every viewer keeps doing what it does today. About 400 bytes when present.

If you're open to it I'd send a PR: an optional camera key in meta.json, preserved on re-encode and exposed by the reader, plus a splat-transform option to write that block from one of the training cameras that come with the PLY (the cameras.json next to it).

[dfattal]

I've put the PR up: #328. It keeps an optional camera key through re-encode, same path as model. Streamed SOG output writes it once at the top of lod-meta.json. -t/-r/-s move it along with the scene. --camera-from cameras.json[:n] fills it in from one of the training cameras that come with the PLY. Files without a camera come out byte-identical. Once you're happy with the shape I can send the matching paragraph for the SOG spec page on developer-site.