Conversation
Core modelled one representation (TriangleMesh) and assumed any frame was reachable in one step. Adds Representation (mesh/points/gaussians/pixels) with PointCloud and GaussianCloud, and Dependency (independent/gop/sequential) so a consumer can plan a seek rather than discover mid-playback that a frame needed its key frame first. Streaming moves out of gs_tools into reconstruction/streamer, with a pluggable representation registry, a byte/request monitor, a static-file server and the browser playback client. Bundles name representations with core's vocabulary (view.json v2; v1 still loads).
QUEEN and 3DGStream already write the PLY every splat viewer reads, but nothing exported it, so the client's free camera showed Vega and nothing else. Adds a gaussian exporter over the three on-disk layouts those runs use, and fixes Detected.viewable, which called a 3DGS run viewable and then had export refuse it. Also adds .splat: 32 bytes a Gaussian against 248 for degree-3 PLY, measured 5.2x smaller on a QUEEN frame, at the cost of every SH band above degree 0. The PLY reader now handles mixed property types, since QUEEN writes an int column that made its output unreadable.
Both had geometry in core and were registered, but the client had no renderer, so Open4D's mesh sequences could only be seen through the Qt window that needs a display the GPU machine does not have. Adds a WebGL2 renderer for triangles and points, a general PLY reader for them, and streamer.export, which turns any open4d.Sequence into a bundle -- so a mesh clip gets the same viewport, camera and scrubbing a Gaussian one does. Shading comes from screen-space derivatives rather than stored normals, because Open4D's PLY writer refuses to store them. Frame writing needed the geometry to stop being assumed a TriangleMesh; a PointCloud has no connectivity. The client had no tests at all. Its parsers now run as shipped, cut out of viewer.html and executed under Node, which found the .splat quaternion coming back non-unit: q = 1 quantises to 256, clamps to 255 and decodes to 0.992.
Dependency was declared and tested but nothing consumed it, so the client asked for frame N directly whatever the codec needed first. Adds a Scheduler that plans the chain, tracks where a stateful decoder sits, and reuses that position on a forward seek while reporting a backward one in a non-rewindable stream as the full replay it is. The manifest now carries the declaration per clip. chain is transcribed into the client because a browser cannot run Python, so a test runs both implementations over the same 56 cases and asserts they agree. The Scheduler also takes over caching and look-ahead, which the two clip sources kept separately with different policies -- and only one of which evicted, so scrubbing a long pixel clip kept every frame it had ever shown. Images now decode through the fetched bytes with the served Content-Type, rather than by handing a URL to the browser where nothing could account for it. Every exporter still writes independent frames; the dependent paths are tested, not yet exercised.
Two regressions from the scheduler change, both of which stopped content playing rather than slowing it down. Images decoded to a blob URL that was revoked as soon as the decode finished, but a pane shows a pixel frame by copying the decoded frame's src onto its own img -- so it was copying a dead URL, which is not an error and so failed silently. Every pixel pane went blank while the Gaussian one kept working; in Compare that is captured, rerf and rerf-depth, i.e. everything but Vega. The URL now lives as long as the frame is cached and is revoked on eviction. The advance loop fired on a wall clock and did not wait for the frame it asked for. showFrame is async, so calls overlapped and resolved out of order: a request for 0..6 displayed 3,1,5,0,6,4,2. At most one advance is now in flight and the interval is timed from when a frame was actually shown, so fps is a ceiling rather than a promise and a clip too heavy to keep up plays in order at whatever rate it can manage.
A bundle could only be a directory of frames someone decoded earlier, which is the wrong shape for two cases already working here. ReRF cannot be decoded in a browser at all -- its entropy coder ships only as a CPython 3.8 binary -- so rendering where the GPU is and sending pixels is not a fallback but its only transport. And measured on this content one rendered view costs 47 kB a frame against 4.3 MB for a decoded Gaussian one, 1.4 MB/s against 129 MB/s at 30 fps, so for a fixed viewpoint pixels win by two orders of magnitude. A clip may now carry a stream URL instead of a frame list. The client points an img at it and stays out of the way, since multipart replacement is a browser feature and the four MJPEG servers here already speak it. Live clips get their own scene: a live renderer picks its own camera, and standing one next to a rig pose would break the guarantee Compare makes. Manifests are validated on write, because a clip with neither frames nor a stream, or a protocol the client does not know, otherwise renders an empty pane with no error anywhere a producer would see it.
The manifest was handing the browser the renderer's own URL, which asks the browser to reach that port. A browser on a laptop viewing a tunnelled page cannot: 127.0.0.1:8768 is the laptop, which has nothing there, so the pane stayed blank and nothing reported why. I had noted the reachability caveat in the clip's notes and left it at that, which was not a fix. The client is now handed live/<name> on the bundle server, which relays from the upstream recorded in the manifest. One forwarded port is enough, the page has a single origin, and a bundle stays portable. An unreachable renderer is a 502 naming the upstream, since that is the likeliest thing to be wrong and is not the bundle's fault.
All three streamed scenes were labelled live. Two are: Vega's wall and scene demos fetch bitstream chunks over HTTP per frame, reassemble, decode colour, cull, schedule and rasterise while you watch. The third is not -- NeVo's viewer loops PNGs rendered hours earlier, because a NeRF frame takes about half a second to ray-march, and its own status page has always said so. I dropped that context when wrapping it. MJPEG carries both equally, so the transport cannot be asked. live.mjpeg now requires an origin of 'rendered' or 'replay' with no default, since a default would let a slideshow be presented as live by omission, and the client labels the pane from it.
The first compressed format this thing actually streams. Until now every format the client could decode was an interchange dump of geometry or a picture, while every format Open4D's codecs produce was undecodable client-side -- the two sets did not intersect at all, which is why calling this a streaming platform was overclaiming. Measured on the mesh sequence the TVMC codec vendors: 761 kB of PLY a frame becomes 59 kB of Draco at 14-bit quantisation, 12.9x, and decodes in 17.8 ms against 20.1 ms for parsing the PLY in JavaScript. 1.8 MB/s at 30 fps rather than 23, which is the difference between a link and a LAN. The decoder is Google's WASM build, vendored under client/vendor/draco and served by the bundle server from its own origin rather than a CDN. That needed a route for client-package assets, which resolves and then checks containment because the path comes off a URL. Lossy in two bounded ways, both recorded in the clip's notes and measured: at 14 bits the worst vertex moved 0.0046% of the model's diagonal, and duplicate vertices are merged (20,672 to 19,747 here, since the source splits them at seams). Delivery form; the PLY stays the source of truth.
A representation says what a decoded frame is. Nothing said what a compressed one is or who could decode it, so the client carried a hand-written dispatcher per representation, each sniffing an extension, and a producer could only learn that its format was undecodable by watching a pane stay blank. Adds streamer.codecs, keyed by (representation, suffix) rather than suffix: .ply is claimed by three representations here and needs two parsers, since a 3DGS PLY and a mesh PLY share an extension and nothing else. Suffixes and media types move out of representations.py, which was listing them a second time and could disagree; they are derived now. The decodes axis is the point. codecs.client_decodable(representation) answers whether a module can be streamed to a browser at all, as a lookup instead of by reading the client's source, and ReRF is registered as server-decoded because that is permanent rather than pending. A lossy codec must state its cost or the constructor refuses it. The client's table is keyed the same way, so both dispatchers are gone, and a test holds the Python registry and the JavaScript one to each other.
Parsing was on the main thread, where it is the one expensive synchronous step in playback: 16.5 ms for a 3DGS PLY, 32.1 ms for a 439k-Gaussian .splat, 17.8 ms for a Draco mesh. Each of those is a missed requestAnimationFrame for every pane rather than only the one decoding, since there is one main thread -- four Gaussian panes blocked it for 66 ms a frame, two whole budgets at 30 fps. The codecs move into client/worker.js and live there alone. Sharing them with the page would mean duplicating them, which drifts, or a build step, which this client does not have. Replies transfer their buffers instead of copying: a 4.3 MB frame parses to several megabytes of typed arrays, and copying those back would return much of the saving. The transfer list is collected by inspecting the result, so a parser that grows a field does not have to declare it. decodeImage stays on the page. It needs Image, and a browser already decodes an image off the main thread. Removed the DOM-based Draco loader that the move left unreachable in the worker, and taught the Python/JavaScript drift guard about the split, since geometry now decodes in the worker while pixels do not.
A sibling of reconstruction/, holding the client side rather than the methods. system/ClientCore is platform-free streaming logic — segment loop, MCKP ABR, bandwidth estimator — shared by a Node and a browser client behind one ClientPlatform contract. system/WebClient has five pages: our adaptive mesh system, ViVo and NAVA point clouds over a WebSocket-to-TCP bridge, Vega splats and NeVo panels. Copied from the 4DVideoStreaming research repo, so the ladder solver, the baseline servers and the corpora still live there; see the README.
Drop system/Client, which held nothing but a captured viewpoint fixture, and move that under tests/fixtures; test_camera_pose now looks in both places so one file works here and in the research repo. Remove the unreferenced manifest.mpd.json that rode along with the server directory. Rewrite the README around getting the demo up and adding a method, instead of describing the layout.
# Conflicts: # README.md # docs/requirements.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Everything left on
streaming-abstractionsafter #49 and #50 took the browserclients. 191 files, all under
open4d/reconstruction/:nevo/— NeVo and its ORBIT wrapper, plus figuresrerf/— the vendored ReRF upstream (codec, CUDA lib) andrerf_streamstreamer/— the point-cloud streamer and its testsmainhas been merged in to resolve two documentation conflicts. Both weremechanical except one paragraph, flagged below.
Conflicts, and how they were resolved
README.md— three hunks, all of them this branch predating thestreaming/entries that #50 added. Resolved to
main. The branch had exactly one uniqueline, the old
└── reconstruction/tree connector, now superseded.docs/requirements.md— a real divergence: this branch keeps an introparagraph that
mainno longer has, removed by987e923 Update requirements.md.Resolved to
main, i.e. the paragraph stays dropped, on the grounds thatmains edit is the newer and deliberate one. Restore it if that was not theintent:
Note
open4d/streamingis byte-identical tomainhere, so nothing in this PRtouches the browser clients. The three streaming commits still appear in the
history because
mainreceived cherry-picked copies with different SHAs.One thing worth a look before merging: the layout block lists
reconstruction/ rgbd, queen, 3dgstream, vega, rerf, gs_tools, streamerandthis branch adds
nevo/, which is not in that list.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.