Repository navigation
Portfolio pass: fix Node/Chromium simulation drift, add tests, CI and a README built from real renders - #1
Open
matuskalis wants to merge 24 commits into
Open
matuskalis wants to merge 24 commits into
matuskalis wants to merge 24 commits into
Conversation
The project had typecheck but no test runner. vitest 5 covers the pure modules; @types/node was only present as a transitive dependency of webpack and is now declared. @remotion/bundler and @remotion/renderer are declared too because the parity script imports them directly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e scene The note times come from simulate() in Node and the frames from simulate() in the headless Chromium that Remotion renders with. The two must produce bit-identical states or the notes drift away from the bounces. They did not always. Node 24 (V8 13.6) and Chrome 149 differ by one ulp on about 3.2 percent of Math.cos and Math.sin inputs, and launch() turns a random angle straight into a ball velocity. Hashing the whole simulation result in both engines, 32 of the 34 shipped scenes agreed and two did not: scenes/swarm.json diverges at frame 70 and plays 380 notes where the picture implies 393, and scenes/infinite-loop.json diverges at frame 57. cosSin() works in exact BigInt fixed point and rounds once, so it returns the same bits in every engine and equals the correctly rounded value. It matches 66 arbitrary-precision references from bc. Chrome's own Math.cos is off the correctly rounded value on 0.15 percent of angles and Node's on 3.2 percent, so no existing picture changes: the new simulation equals the old Chromium one on all 34 scenes, and Node now equals Chromium on all 34. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…de.ts Both lived inside closures in simulate(), so the collision response, the part that would embarrass the author if wrong, could only be tested through a whole simulation. reflect() and collidePair() are the same statements in the same order, now exported and covered by 15 direct tests (elastic swap, momentum and energy conservation, restitution, frozen ball as wall, separating pair, coincident centres). No behaviour change: the SHA-256 of the complete simulation output and of the WAV for all 34 shipped scenes is identical before and after. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The free-flight tests check the integrator against the closed form of semi-implicit Euler at four substeps per frame (frame f holds the state after substep 4(f + 1)). A ball released from the exact centre with no gravity travels along a diameter, so the time of every bounce and note has a closed form, and restitution, the 55 px/s resting threshold, speed-up with its cap, growth, ring rotation and destruction are each checked against it. The 45 ms note merge, the substep grid the notes sit on, determinism and the short-scene-is-a-prefix property are covered too. Eleven deliberate mutations of the physics (substeps, thresholds, restitution, impulse, ring direction and others) each fail at least one of these tests. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…y scenes
validate printed VALID for a scene with a misspelt group ("phyiscs"), a
string where a number belongs ("gravity": "fast") and an instrument that
does not exist ("kazoo", silently played as piano). AGENTS.md admits that
unknown keys are merged and ignored and tells the agent to be careful; the
check that exists to catch the agent's mistakes now does.
- every field is compared with the defaults: unknown names get a "did you
mean" hint, wrong types and non-finite numbers are named
- arena.kind and music.instrument must be in their lists, which are now
runtime arrays that modes prints from the same source
- a duration under half a frame no longer prints VALID and then crashes with
a TypeError
- a rings scene with breakWalls off is no longer told the ball "never cleared
every ring": the preset keeps them on purpose (scenes/spiral.json)
All 34 shipped scenes still validate. 60 tests in test/scene.test.ts.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
make spawned `remotion render` with a fixed argument list, so a five second or quarter-size preview, which AGENTS.md tells the agent to render first, meant editing durationSeconds in the scene. Any flag make does not use itself now goes to Remotion: npm run make -- scenes/x.json --scale=0.25 --frames=0-299 The simulation still runs for the whole scene, so the preview is the exact first five seconds. The usage line said `--out out/<name>.mp4` while the parser wants `--out=out/<name>.mp4`; fixed. The melody walk (note n of the melody on bounce n, looping) was inline in the CLI, where nothing could test it. assignNotes() is that one line, exported next to frequency(), and the audio tests cover pitch, the 4 ms attack, tanh soft clipping, all five instruments, the WAV header and clamping, the MIDI reader (running status, meta and sysex events, several tracks, velocity-0 note-offs) and loadMelody. CLI tests run validate, modes and make as child processes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
For each of the 34 scene files the golden record holds the frame count, the note count, the bounce count and the first 16 hex digits of the SHA-256 of the whole simulation result (every frame, note time and ring burst) and of the WAV that make writes. A change that alters any shipped video or its audio fails here; `npm run golden` rewrites the record after a change you meant to make. The same test file checks that every scene validates, names its own file (the name decides the output path) and uses a melody that loads. The hashes were identical on Node 22.22 and 24.5 (macOS, arm64). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…he frames `npm run parity` hashes the complete simulation of every scene in Node and inside Remotion's own headless Chromium (bundled with @remotion/bundler, driven through @remotion/renderer's renderStill and its onBrowserLog callback), and exits 1 if a pair differs. It is the check that found the Math.cos and Math.sin disagreement fixed earlier in this branch. Result at this commit, macOS arm64, Node 24.5 against Chrome Headless Shell 149: 34 of 34 scenes identical, 75 s on a heavily loaded laptop. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`npm run probe:trig` pushes 200,000 angles in [0, 2 pi) through Math.cos and Math.sin in Node and in the headless Chromium Remotion renders with, and compares both with the exact values from src/sim/trig.ts. On this machine Node 24.5 is off by one ulp on 3.3 percent of cos and 3.2 percent of sin inputs, Chrome 149 on 0.14 and 0.13 percent. These are the numbers behind the trig fix, now checkable in about 20 seconds. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The contract's second example, prompt "ball escaping rotating rings, dark purple, no music, 15s". The plain escape preset at seed 9 leaves 5 of 10 rings standing at 15 s and validate says so; with arena.ringCount lowered to 6 the ball clears all six by 12.1 s. The scene is what the README walks through and what the docs tests check. Golden record extended to 35 scenes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Two jobs. `test` runs npm ci, typecheck and the vitest suite on Node 22 and 24. `render` installs the libraries Chrome Headless Shell needs (the list from Remotion's Ubuntu 24.04 documentation), renders one second of scenes/readme-demo.json at quarter size through `npm run make`, and runs `npm run parity`, which compares the simulation in Node with the one in the Chromium that draws the frames. Every command except the apt step, which is Linux only, was run locally first. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The ignore file covered node_modules, out, rendered WAVs, .DS_Store and logs. Adds .env and .env.* (keeping .env.example), CLAUDE.local.md, coverage and tsbuildinfo output. Nothing stray was tracked in the tree or in the history (checked with git log --all --name-only). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ism notes The README had 62 lines, a 300 px GIF and no account of the contract, the physics, the audio or the measurements. It now leads with a four-panel GIF cut from real renders (1.6 MB), a contact sheet of one frame per preset with scene name and seed captions, a worked prompt-to-frame example whose outputs are real, the pipeline, the real parameters, a determinism table, three design decisions with their costs, and a status and limits list. - docs/engine.md: every constant, the collision math, the instrument formulas, frame timing, and the measured 42.7 ms AAC delay in the MP4 - docs/determinism.md: what was hashed and compared, the cos and sin bug, how to reproduce, what is not claimed - scripts/readme-assets.mjs (npm run docs:assets) regenerates the pictures through `npm run make` and ffmpeg; the old docs/demo.gif is replaced - AGENTS.md: the missing growToFillAtEnd field, a table for reading validate output, the preview command, measured render costs in place of an unreproduced "7 seconds", and no em dashes - test/docs.test.ts: every scene example in README and AGENTS validates, AGENTS lists every schema field, every npm script and file the docs mention exists Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
npm audit reported two high advisories in dev tooling that Remotion pulls in: fast-uri 3.1.5 (through webpack and ajv) and js-yaml 4.3.1 (through svgr and cosmiconfig). `npm audit fix` bumps them to 3.1.8 and 4.3.2 and changes nothing else (six lines of the lockfile); audit now reports 0 vulnerabilities. After the bump the typecheck, all 270 tests and the parity check pass, and a full render of scenes/readme-demo.json is still byte-identical to the earlier ones. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Informational, to compare the Linux render with the one made on macOS arm64 (166,458 bytes, md5 d6cbbf39a1d8aa5e941d4d0a1af22a34 locally). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
GitHub announced that the ubuntu-latest label moves to Ubuntu 26 on 19 October 2026. The render job installs libasound2t64 and the other package names from Remotion's Ubuntu 24.04 list, so the runner is pinned to ubuntu-24.04 instead of letting the job change underneath it. The preview clip is uploaded as an artifact for seven days. The md5 printed by the previous commit differs from the macOS render, so the two files can now be compared frame by frame. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The first CI runs (ubuntu-24.04, x86_64, Node 22 and 24, Chrome Headless Shell 149.0.7790.0 for Linux) gave three results that belong in the notes. The golden record written on macOS arm64 passes on Linux: the simulation and WAV hashes of all 35 scenes are identical. The parity check passes on Linux too, 35 of 35. The rendered pixels are not identical across the two machines: the quarter-size preview of scenes/readme-demo.json differs in all 60 decoded frames (average PSNR 44.8 dB), while each machine reproduces its own bytes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The table gained Ubuntu rows, so the lead sentence now names both machines and the "last two rows" sentence points at the Node-against-Chromium row it means. Also: every numbered scene has variants, the test duration is stated for a quiet machine and for CI, probe:trig is listed with the other scripts, and the scene files are no longer described as hand-written (only the walk-through is). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The determinism notes now name both machines up front, and no longer assert which render stage differs between macOS and Linux (only that the pixels do, at 44.8 dB PSNR). The engine notes say that wall breaking applies to a circle arena, and the diagrams are marked as text. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The run for 648395c failed in `npm run make` with `ECONNRESET: socket hang up` fifteen seconds into the 93 MB Chrome Headless Shell download from remotion.media. The two earlier runs of the same step had passed, so it is a flaky host, not a broken step. The render job now restores a cached copy (keyed on the lockfile) and, on a miss, runs `npx remotion browser ensure` up to three times with a pause between attempts, before any render starts. `browser ensure` was run locally and exits 0 when the browser is present. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
docs/presets.png was still the first render of the sheet: the classic cell showed scene 02 instead of 02b, the growth cell frame 1200 instead of 1080, and the captions were the older layout. The script had been changed after that render and only hero and sound were re-run. The whole script now runs end to end, twice, and all four pictures come out byte-identical both times. Captions put the preset first, then scene and seed, and drop the scene name when it repeats the preset. Text is placed by baseline so captions line up whatever letters they contain, and a long scene name shrinks to fit its cell instead of running off it (02b-speed-demon-streak was clipped). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The scene name becomes out/<name>.props.json, public/<name>.wav and out/<name>.mp4. A scene written with name "../../x" would put those three files outside the project. The scene file is written by an agent acting on a user's prompt, so it is input from outside, and the validator is the place that already checks it. Names with spaces, dots and underscores still pass; only / and \ are refused. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review findings. npm run parity compares the Chromium simulation hash with the Node hash pinned in golden.json, so the docs now say that instead of calling it Node against Chromium. Simulating infinite-loop takes about 1.6 s against 0.8 s before the pair collision was split out; the exact trig is not the cause (about 3 microseconds per call). Renders of swarm and infinite-loop made before the trig change have different note times. cosSin is documented as checked only for the inputs the simulation feeds it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Most important first.
1. A determinism bug in the headline claim, found by measuring and fixed. The notes come from
simulate()in Node, the picture fromsimulate()in the Chromium that Remotion renders with. I hashed the complete simulation result in both. 32 of the 34 shipped scenes agreed.scenes/swarm.jsondid not (the two split at frame 70; Node counts 380 notes where the picture implies 393; only 33 of the first 380 note times agree) and neither didscenes/infinite-loop.json. Cause: Node 24 (V8 13.6) and Chrome 149 differ by one ulp on about 3 percent ofMath.cosandMath.sininputs, and every launched ball turns a random angle into a velocity with them.src/sim/trig.tsnow does those calls in exact BigInt arithmetic and rounds once. No existing picture changes: the new simulation equals the old Chromium one on all 34 scenes, and Node now equals Chromium on all 35.npm run parityandnpm run probe:trigreproduce the measurements.2.
validatecatches what the engine silently ignored. A scene with"phyiscs","gravity": "fast"and"instrument": "kazoo"used to print VALID. It now reportsunknown field "phyiscs", did you mean "physics"?, names the wrong type and lists the valid instruments. A scenenamecontaining/or\is refused, because it becomes the output file name and../../xwould write outside the project. A duration under half a frame no longer prints VALID and then crashes. Aspiralscene is no longer told its ball "never cleared every ring" (that preset keeps the rings on purpose).3. 273 tests (vitest), a typecheck script and golden output. The physics step is checked against the closed form of semi-implicit Euler; a ball released from the exact centre without gravity travels along a diameter, so every bounce and note time has a closed form to test against. Also covered: restitution and the 55 px/s resting threshold, every effect, rings, ball-to-ball collisions, the seeded generator, exact cos and sin against 66
bcreferences, pitch, the WAV header, the MIDI reader, validation, the CLIs, every JSON example in the docs, and a SHA-256 of the full simulation and WAV of all 35 scenes. Eleven deliberate mutations of the physics each fail at least one test.4. README and docs from real renders. A four-panel GIF (1.6 MB) and a contact sheet of one frame per preset, captioned with scene name, preset and seed, both cut from renders made with
npm run makeat 360x640; a 200 KB clip with sound; a worked prompt-to-frame example whose outputs are real;docs/engine.md(every constant, the collision math, the instrument formulas, frame timing) anddocs/determinism.md(what was hashed, what matched, what did not).npm run docs:assetsregenerates the pictures, and two full runs give byte-identical files.5. CI (
.github/workflows/ci.yml, pinned toubuntu-24.04): typecheck and tests on Node 22 and 24, and a job that renders one second ofscenes/readme-demo.jsonat quarter size (uploaded as an artifact) and runsnpm run parity. One of the five runs so far failed in the render job withECONNRESETwhile downloading Chrome Headless Shell from remotion.media; the job now caches the browser and retriesnpx remotion browser ensurethree times, and the run after that change passed with a cache hit.6. Smaller.
makenow hands unknown flags toremotion render(--scale=0.25 --frames=0-299is a 5 s preview; its usage line said--out out/x.mp4while the parser wants--out=out/x.mp4). AGENTS.md gained the missinggrowToFillAtEndfield, a table for readingvalidateoutput and measured render costs in place of an unreproduced "7 seconds".scenes/purple-escape.jsonis the contract's second worked example..gitignorecovers.env*,CLAUDE.local.mdand coverage. Two transitive dev advisories (fast-uri, js-yaml) are patched with the in-rangenpm audit fix(six lockfile lines), audit reports 0. The olddocs/demo.gifis replaced bydocs/hero.gif.Behaviour changes to review in code
src/sim/trig.ts,src/sim/simulate.ts: launch direction and initial placement usecosSin(). Audio forswarmandinfinite-loopchanges (it now matches their pictures); nothing else does.src/sim/collide.ts:reflect()andcollidePair()moved out ofsimulate()with the same statements in the same order. The SHA-256 of the complete output and of the WAV for all 34 scenes that existed is identical before and after.src/scene/resolve.ts:validateScenerejects unknown fields, wrong types, unknownarena.kindandmusic.instrument, and durations under one frame. All 35 shipped scenes pass. A scene with a typo that used to validate now fails, which is the intent.src/cli/make.ts: extra flags pass through to Remotion;assignNotes()replaces one inline line.How I checked it
npm run parity35 of 35 with the Linux build of Chrome Headless Shell 149.npm ci12.5 s,npm run typecheckclean,npm test270 passed (273 after the later name check),npm run make -- scenes/readme-demo.jsonwroteout/readme-demo.mp4(2,934,889 bytes).4459c552f450b5328e82542bcfc4b1d6, WAV md51ffb3531d42e1401a0ba9cdc7adfaa21). 60 PNG frames match across worker counts and between a range render and a full render.docs/determinism.md: the same quarter-size preview rendered on macOS and on Linux differs in all 60 frames (average PSNR 44.8 dB), and the audio in an MP4 trails the WAV by 42.7 ms (two AAC frames, constant, the same with ffmpeg and CoreAudio).npm run devstarts Remotion Studio (HTTP 200 on a private port, then stopped).Not checked
For Matúš
origin/mainis one commit ahead of the local checkout (ee33f54, 4 Sep, a README GIF). This PR replaces that GIF; pull before building on the branch.🤖 Generated with Claude Code