Repository navigation
perf(build): capture frames in parallel browsers and time-limit every step that can stall - #31
Merged
Merged
Conversation
2 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Renders MP4s about 4-5x faster by capturing frames in several browsers at once, and puts a time limit on every step that could stall, so a stuck film page or ffmpeg fails with its name instead of hanging.
The two go together: the time limits apply to every page the parallel capture opens, and if parallel capture had not paid off, both would have been dropped.
Closes #25. Closes #26. Part of #16.
Changes
Parallel capture (#25)
openFilm(dir, { pages })opens the film inpagesbrowsers, one page each, and hands each frame or layout sample to a free page.inOrder()(newscripts/inorder.ts) keeps up topagesframes in flight and yields them in film order, so ffmpeg still gets frames in order and at mostpagesframes are held in memory.buildencodes through it;check,stillsandbuildrun the layout check through it. Samples are judged in film order, so the "two samples in a row" rule is unchanged.MOTION_RENDER_PAGESoverrides it (README, usage).Why one browser per page: several pages in one browser were no faster than one (5.5 fps for 1 page, 5.6 for 2, 3.8 for 4); the browser kept about one core busy however many pages it had. Separate browsers scale with their number:
The first launch runs alone, since it may download Chromium; the rest start in parallel.
Time limits (#26)
scripts/limit.ts(new):limit(promise, ms, what, onTimeout)fails with<what> did not finish in <n>safter runningonTimeoutto stop what is stuck.seek(t), ready, screenshot, layout snapshot, Motion'sload/frame(t)/consoleLog) is limited to 60 s and names the call and time, e.g.the film page's __film.seek(12.3) did not finish in 60s. Playwright'sevaluatehas no timeout of its own.runFfmpeg/volumeStatskill ffmpeg after 5 min and name the file. The encoder inbuildis limited per frame (60 s to take each frame) and for the final mux (5 min), since it runs as long as capture.Measurements
Build of the WIF film (57.3 s, 1719 frames) on an Apple M4 Pro (14 cores), master and this branch alternately, while other jobs kept the load average at 9-22:
With the time limits alone (1 page) builds were no slower than master (251 s vs 270 s on average in an earlier alternating run).
Output: frames are not bit-identical to master's, but master is not bit-identical to itself either (#29). PSNR master vs master: average 59.0 dB, lowest frame 52.5 dB; master vs this branch: average 54.9 dB, lowest 40.8 dB. The lowest frames differ only by sub-pixel shifts along text and box edges during a push-in; a side-by-side of both MP4s was reviewed and looks the same. In tests, frames from 2 pages are byte-identical to 1 page.
Test plan
make check: lint, format, typecheck, 102 tests with the coverage gate, quality gatesinOrder(order, at mostaheadin flight, first failure passed on),capturePages(env var and its validation), 2 pages give the same frames and layout issues as 1 page,limit, a stuck ffmpeg is killed and named, a film whoseseekspins forever fails within the limit naming__film.seek(1.5)stillson a real film with a stuckseekexits after 61.8 s withthe film page's __film.seek(2) did not finish in 60s; no browser process is left behind