Skip to content

Portfolio pass: fix Node/Chromium simulation drift, add tests, CI and a README built from real renders - #1

Open
matuskalis wants to merge 24 commits into
mainfrom
polish/portfolio-2026-09
Open

matuskalis wants to merge 24 commits into
mainfrom
polish/portfolio-2026-09

Conversation

@matuskalis

@matuskalis matuskalis commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

What changed

Most important first.

1. A determinism bug in the headline claim, found by measuring and fixed. The notes come from simulate() in Node, the picture from simulate() in the Chromium that Remotion renders with. I hashed the complete simulation result in both. 32 of the 34 shipped scenes agreed. scenes/swarm.json did not (the two split at frame 70; Node counts 380 notes where the picture implies 393; only 33 of the first 380 note times agree) and neither did scenes/infinite-loop.json. Cause: Node 24 (V8 13.6) and Chrome 149 differ by one ulp on about 3 percent of Math.cos and Math.sin inputs, and every launched ball turns a random angle into a velocity with them. src/sim/trig.ts now does those calls in exact BigInt arithmetic and rounds once. No existing picture changes: the new simulation equals the old Chromium one on all 34 scenes, and Node now equals Chromium on all 35. npm run parity and npm run probe:trig reproduce the measurements.

2. validate catches what the engine silently ignored. A scene with "phyiscs", "gravity": "fast" and "instrument": "kazoo" used to print VALID. It now reports unknown field "phyiscs", did you mean "physics"?, names the wrong type and lists the valid instruments. A scene name containing / or \ is refused, because it becomes the output file name and ../../x would write outside the project. A duration under half a frame no longer prints VALID and then crashes. A spiral scene is no longer told its ball "never cleared every ring" (that preset keeps the rings on purpose).

3. 273 tests (vitest), a typecheck script and golden output. The physics step is checked against the closed form of semi-implicit Euler; a ball released from the exact centre without gravity travels along a diameter, so every bounce and note time has a closed form to test against. Also covered: restitution and the 55 px/s resting threshold, every effect, rings, ball-to-ball collisions, the seeded generator, exact cos and sin against 66 bc references, pitch, the WAV header, the MIDI reader, validation, the CLIs, every JSON example in the docs, and a SHA-256 of the full simulation and WAV of all 35 scenes. Eleven deliberate mutations of the physics each fail at least one test.

4. README and docs from real renders. A four-panel GIF (1.6 MB) and a contact sheet of one frame per preset, captioned with scene name, preset and seed, both cut from renders made with npm run make at 360x640; a 200 KB clip with sound; a worked prompt-to-frame example whose outputs are real; docs/engine.md (every constant, the collision math, the instrument formulas, frame timing) and docs/determinism.md (what was hashed, what matched, what did not). npm run docs:assets regenerates the pictures, and two full runs give byte-identical files.

5. CI (.github/workflows/ci.yml, pinned to ubuntu-24.04): typecheck and tests on Node 22 and 24, and a job that renders one second of scenes/readme-demo.json at quarter size (uploaded as an artifact) and runs npm run parity. One of the five runs so far failed in the render job with ECONNRESET while downloading Chrome Headless Shell from remotion.media; the job now caches the browser and retries npx remotion browser ensure three times, and the run after that change passed with a cache hit.

6. Smaller. make now hands unknown flags to remotion render (--scale=0.25 --frames=0-299 is a 5 s preview; its usage line said --out out/x.mp4 while the parser wants --out=out/x.mp4). AGENTS.md gained the missing growToFillAtEnd field, a table for reading validate output and measured render costs in place of an unreproduced "7 seconds". scenes/purple-escape.json is the contract's second worked example. .gitignore covers .env*, CLAUDE.local.md and coverage. Two transitive dev advisories (fast-uri, js-yaml) are patched with the in-range npm audit fix (six lockfile lines), audit reports 0. The old docs/demo.gif is replaced by docs/hero.gif.

Behaviour changes to review in code

  • src/sim/trig.ts, src/sim/simulate.ts: launch direction and initial placement use cosSin(). Audio for swarm and infinite-loop changes (it now matches their pictures); nothing else does.
  • src/sim/collide.ts: reflect() and collidePair() moved out of simulate() with the same statements in the same order. The SHA-256 of the complete output and of the WAV for all 34 scenes that existed is identical before and after.
  • src/scene/resolve.ts: validateScene rejects unknown fields, wrong types, unknown arena.kind and music.instrument, and durations under one frame. All 35 shipped scenes pass. A scene with a typo that used to validate now fails, which is the intent.
  • src/cli/make.ts: extra flags pass through to Remotion; assignNotes() replaces one inline line.

How I checked it

  • CI on the last commit, all three jobs green: typecheck and 273 tests on Node 22 and 24, and the render job (about a minute): a clip rendered on Linux and npm run parity 35 of 35 with the Linux build of Chrome Headless Shell 149.
  • Fresh clone of this branch on my laptop: npm ci 12.5 s, npm run typecheck clean, npm test 270 passed (273 after the later name check), npm run make -- scenes/readme-demo.json wrote out/readme-demo.mp4 (2,934,889 bytes).
  • Three full renders of that scene (Node 24 with 5 and with 2 workers, Node 22) and the fresh-clone render are byte-identical (MP4 md5 4459c552f450b5328e82542bcfc4b1d6, WAV md5 1ffb3531d42e1401a0ba9cdc7adfaa21). 60 PNG frames match across worker counts and between a range render and a full render.
  • The golden record (simulation and WAV hashes of 35 scenes) was written on macOS arm64 with Node 24.5.0 and passes on Node 22.22.2 there and on Ubuntu x86_64 in CI.
  • Not identical, and documented in docs/determinism.md: the same quarter-size preview rendered on macOS and on Linux differs in all 60 frames (average PSNR 44.8 dB), and the audio in an MP4 trails the WAV by 42.7 ms (two AAC frames, constant, the same with ffmpeg and CoreAudio).
  • npm run dev starts Remotion Studio (HTTP 200 on a private port, then stopped).

Not checked

  • The natural-language step: no model is called by any test or demo here.
  • Windows, Safari and Firefox previews, Node 18 and 20.
  • A listening test of the sound clip. It was never played; its level was measured (mean -19.1 dB, peak -8.8 dB).

For Matúš

  • No LICENSE file added.
  • origin/main is one commit ahead of the local checkout (ee33f54, 4 Sep, a README GIF). This PR replaces that GIF; pull before building on the branch.
  • Decision: the MP4 audio is 42.7 ms late because of Remotion's AAC encoder. Pre-shifting the WAV by 2048 samples at 48 kHz would cancel it and change all 35 golden hashes.

🤖 Generated with Claude Code

matuskalis and others added 24 commits September 30, 2026 20:27
The project had typecheck but no test runner. vitest 5 covers the pure
modules; @types/node was only present as a transitive dependency of
webpack and is now declared. @remotion/bundler and @remotion/renderer are
declared too because the parity script imports them directly.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e scene

The note times come from simulate() in Node and the frames from simulate()
in the headless Chromium that Remotion renders with. The two must produce
bit-identical states or the notes drift away from the bounces.

They did not always. Node 24 (V8 13.6) and Chrome 149 differ by one ulp on
about 3.2 percent of Math.cos and Math.sin inputs, and launch() turns a
random angle straight into a ball velocity. Hashing the whole simulation
result in both engines, 32 of the 34 shipped scenes agreed and two did not:
scenes/swarm.json diverges at frame 70 and plays 380 notes where the picture
implies 393, and scenes/infinite-loop.json diverges at frame 57.

cosSin() works in exact BigInt fixed point and rounds once, so it returns
the same bits in every engine and equals the correctly rounded value. It
matches 66 arbitrary-precision references from bc. Chrome's own Math.cos is
off the correctly rounded value on 0.15 percent of angles and Node's on 3.2
percent, so no existing picture changes: the new simulation equals the old
Chromium one on all 34 scenes, and Node now equals Chromium on all 34.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…de.ts

Both lived inside closures in simulate(), so the collision response, the
part that would embarrass the author if wrong, could only be tested through
a whole simulation. reflect() and collidePair() are the same statements in
the same order, now exported and covered by 15 direct tests (elastic swap,
momentum and energy conservation, restitution, frozen ball as wall,
separating pair, coincident centres).

No behaviour change: the SHA-256 of the complete simulation output and of
the WAV for all 34 shipped scenes is identical before and after.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The free-flight tests check the integrator against the closed form of
semi-implicit Euler at four substeps per frame (frame f holds the state after
substep 4(f + 1)). A ball released from the exact centre with no gravity
travels along a diameter, so the time of every bounce and note has a closed
form, and restitution, the 55 px/s resting threshold, speed-up with its cap,
growth, ring rotation and destruction are each checked against it. The 45 ms
note merge, the substep grid the notes sit on, determinism and the
short-scene-is-a-prefix property are covered too.

Eleven deliberate mutations of the physics (substeps, thresholds, restitution,
impulse, ring direction and others) each fail at least one of these tests.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…y scenes

validate printed VALID for a scene with a misspelt group ("phyiscs"), a
string where a number belongs ("gravity": "fast") and an instrument that
does not exist ("kazoo", silently played as piano). AGENTS.md admits that
unknown keys are merged and ignored and tells the agent to be careful; the
check that exists to catch the agent's mistakes now does.

- every field is compared with the defaults: unknown names get a "did you
  mean" hint, wrong types and non-finite numbers are named
- arena.kind and music.instrument must be in their lists, which are now
  runtime arrays that modes prints from the same source
- a duration under half a frame no longer prints VALID and then crashes with
  a TypeError
- a rings scene with breakWalls off is no longer told the ball "never cleared
  every ring": the preset keeps them on purpose (scenes/spiral.json)

All 34 shipped scenes still validate. 60 tests in test/scene.test.ts.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
make spawned `remotion render` with a fixed argument list, so a five second
or quarter-size preview, which AGENTS.md tells the agent to render first,
meant editing durationSeconds in the scene. Any flag make does not use
itself now goes to Remotion:

  npm run make -- scenes/x.json --scale=0.25 --frames=0-299

The simulation still runs for the whole scene, so the preview is the exact
first five seconds. The usage line said `--out out/<name>.mp4` while the
parser wants `--out=out/<name>.mp4`; fixed.

The melody walk (note n of the melody on bounce n, looping) was inline in
the CLI, where nothing could test it. assignNotes() is that one line,
exported next to frequency(), and the audio tests cover pitch, the 4 ms
attack, tanh soft clipping, all five instruments, the WAV header and
clamping, the MIDI reader (running status, meta and sysex events, several
tracks, velocity-0 note-offs) and loadMelody. CLI tests run validate, modes
and make as child processes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
For each of the 34 scene files the golden record holds the frame count, the
note count, the bounce count and the first 16 hex digits of the SHA-256 of
the whole simulation result (every frame, note time and ring burst) and of
the WAV that make writes. A change that alters any shipped video or its audio
fails here; `npm run golden` rewrites the record after a change you meant
to make. The same test file checks that every scene validates, names its own
file (the name decides the output path) and uses a melody that loads.

The hashes were identical on Node 22.22 and 24.5 (macOS, arm64).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…he frames

`npm run parity` hashes the complete simulation of every scene in Node and
inside Remotion's own headless Chromium (bundled with @remotion/bundler,
driven through @remotion/renderer's renderStill and its onBrowserLog
callback), and exits 1 if a pair differs. It is the check that found the
Math.cos and Math.sin disagreement fixed earlier in this branch. Result at
this commit, macOS arm64, Node 24.5 against Chrome Headless Shell 149: 34 of
34 scenes identical, 75 s on a heavily loaded laptop.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
`npm run probe:trig` pushes 200,000 angles in [0, 2 pi) through Math.cos and
Math.sin in Node and in the headless Chromium Remotion renders with, and
compares both with the exact values from src/sim/trig.ts. On this machine
Node 24.5 is off by one ulp on 3.3 percent of cos and 3.2 percent of sin
inputs, Chrome 149 on 0.14 and 0.13 percent. These are the numbers behind
the trig fix, now checkable in about 20 seconds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The contract's second example, prompt "ball escaping rotating rings, dark
purple, no music, 15s". The plain escape preset at seed 9 leaves 5 of 10
rings standing at 15 s and validate says so; with arena.ringCount lowered to
6 the ball clears all six by 12.1 s. The scene is what the README walks
through and what the docs tests check. Golden record extended to 35 scenes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Two jobs. `test` runs npm ci, typecheck and the vitest suite on Node 22 and
24. `render` installs the libraries Chrome Headless Shell needs (the list
from Remotion's Ubuntu 24.04 documentation), renders one second of
scenes/readme-demo.json at quarter size through `npm run make`, and runs
`npm run parity`, which compares the simulation in Node with the one in the
Chromium that draws the frames. Every command except the apt step, which is
Linux only, was run locally first.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The ignore file covered node_modules, out, rendered WAVs, .DS_Store and logs.
Adds .env and .env.* (keeping .env.example), CLAUDE.local.md, coverage and
tsbuildinfo output. Nothing stray was tracked in the tree or in the history
(checked with git log --all --name-only).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ism notes

The README had 62 lines, a 300 px GIF and no account of the contract, the
physics, the audio or the measurements. It now leads with a four-panel GIF
cut from real renders (1.6 MB), a contact sheet of one frame per preset with
scene name and seed captions, a worked prompt-to-frame example whose outputs
are real, the pipeline, the real parameters, a determinism table, three
design decisions with their costs, and a status and limits list.

- docs/engine.md: every constant, the collision math, the instrument
  formulas, frame timing, and the measured 42.7 ms AAC delay in the MP4
- docs/determinism.md: what was hashed and compared, the cos and sin bug,
  how to reproduce, what is not claimed
- scripts/readme-assets.mjs (npm run docs:assets) regenerates the pictures
  through `npm run make` and ffmpeg; the old docs/demo.gif is replaced
- AGENTS.md: the missing growToFillAtEnd field, a table for reading validate
  output, the preview command, measured render costs in place of an
  unreproduced "7 seconds", and no em dashes
- test/docs.test.ts: every scene example in README and AGENTS validates,
  AGENTS lists every schema field, every npm script and file the docs
  mention exists

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
npm audit reported two high advisories in dev tooling that Remotion pulls
in: fast-uri 3.1.5 (through webpack and ajv) and js-yaml 4.3.1 (through
svgr and cosmiconfig). `npm audit fix` bumps them to 3.1.8 and 4.3.2 and
changes nothing else (six lines of the lockfile); audit now reports 0
vulnerabilities. After the bump the typecheck, all 270 tests and the parity
check pass, and a full render of scenes/readme-demo.json is still
byte-identical to the earlier ones.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Informational, to compare the Linux render with the one made on macOS arm64
(166,458 bytes, md5 d6cbbf39a1d8aa5e941d4d0a1af22a34 locally).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
GitHub announced that the ubuntu-latest label moves to Ubuntu 26 on
19 October 2026. The render job installs libasound2t64 and the other
package names from Remotion's Ubuntu 24.04 list, so the runner is pinned
to ubuntu-24.04 instead of letting the job change underneath it.

The preview clip is uploaded as an artifact for seven days. The md5 printed
by the previous commit differs from the macOS render, so the two files can
now be compared frame by frame.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The first CI runs (ubuntu-24.04, x86_64, Node 22 and 24, Chrome Headless
Shell 149.0.7790.0 for Linux) gave three results that belong in the notes.
The golden record written on macOS arm64 passes on Linux: the simulation
and WAV hashes of all 35 scenes are identical. The parity check passes on
Linux too, 35 of 35. The rendered pixels are not identical across the two
machines: the quarter-size preview of scenes/readme-demo.json differs in all
60 decoded frames (average PSNR 44.8 dB), while each machine reproduces its
own bytes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The table gained Ubuntu rows, so the lead sentence now names both machines
and the "last two rows" sentence points at the Node-against-Chromium row it
means. Also: every numbered scene has variants, the test duration is stated
for a quiet machine and for CI, probe:trig is listed with the other scripts,
and the scene files are no longer described as hand-written (only the
walk-through is).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The determinism notes now name both machines up front, and no longer assert
which render stage differs between macOS and Linux (only that the pixels do,
at 44.8 dB PSNR). The engine notes say that wall breaking applies to a
circle arena, and the diagrams are marked as text.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The run for 648395c failed in `npm run make` with `ECONNRESET: socket hang
up` fifteen seconds into the 93 MB Chrome Headless Shell download from
remotion.media. The two earlier runs of the same step had passed, so it is
a flaky host, not a broken step. The render job now restores a cached copy
(keyed on the lockfile) and, on a miss, runs `npx remotion browser ensure`
up to three times with a pause between attempts, before any render starts.
`browser ensure` was run locally and exits 0 when the browser is present.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
docs/presets.png was still the first render of the sheet: the classic cell
showed scene 02 instead of 02b, the growth cell frame 1200 instead of 1080,
and the captions were the older layout. The script had been changed after
that render and only hero and sound were re-run. The whole script now runs
end to end, twice, and all four pictures come out byte-identical both times.

Captions put the preset first, then scene and seed, and drop the scene name
when it repeats the preset. Text is placed by baseline so captions line up
whatever letters they contain, and a long scene name shrinks to fit its cell
instead of running off it (02b-speed-demon-streak was clipped).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The scene name becomes out/<name>.props.json, public/<name>.wav and
out/<name>.mp4. A scene written with name "../../x" would put those three
files outside the project. The scene file is written by an agent acting on
a user's prompt, so it is input from outside, and the validator is the
place that already checks it. Names with spaces, dots and underscores still
pass; only / and \ are refused.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review findings. npm run parity compares the Chromium simulation hash
with the Node hash pinned in golden.json, so the docs now say that
instead of calling it Node against Chromium. Simulating infinite-loop
takes about 1.6 s against 0.8 s before the pair collision was split
out; the exact trig is not the cause (about 3 microseconds per call).
Renders of swarm and infinite-loop made before the trig change have
different note times. cosSin is documented as checked only for the
inputs the simulation feeds it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant