diff --git a/.claude/settings.local.json b/.claude/settings.local.json index 43dd5d096b..a2130dd2ec 100644 --- a/.claude/settings.local.json +++ b/.claude/settings.local.json @@ -1,3 +1,6 @@ { - "enabledMcpjsonServers": ["threenative-assets", "threenative-sculpt", "threenative-engine"] + "enabledMcpjsonServers": ["threenative-assets", "threenative-sculpt", "threenative-engine"], + "permissions": { + "allow": ["Bash(cd *)"] + } } diff --git a/.claude/skills/build-on-sandbox/SKILL.md b/.claude/skills/build-on-sandbox/SKILL.md index 8542f473a4..312066fc43 100644 --- a/.claude/skills/build-on-sandbox/SKILL.md +++ b/.claude/skills/build-on-sandbox/SKILL.md @@ -95,6 +95,9 @@ preference; it is the difference between the two outcomes. result matches, and the committed ledger path. The comparison and the friction record are the point of the exercise. +For performance work, use the existing stress scene instead of a new game: +[stress-scene.md](stress-scene.md) says where it is, what it loads, and how to measure it. + ## Rules - **Work only inside your game folder.** `../sandbox//` is yours; the sandbox root and diff --git a/.claude/skills/build-on-sandbox/stress-scene.md b/.claude/skills/build-on-sandbox/stress-scene.md new file mode 100644 index 0000000000..9e102301f2 --- /dev/null +++ b/.claude/skills/build-on-sandbox/stress-scene.md @@ -0,0 +1,56 @@ +# The stress scene (performance loop) + +The stress lab is a sandbox game whose second scene loads the engine to its limit with a realistic +open-world mix. It is not a game to play. It is the fixed bench for the inspect, fix, measure loop. + +## Where it is + +- Game folder: `../sandbox/stress-lab/`, a starter-kit sandbox game. The sandbox folder is its own git + repository: commit scene changes there. +- Scene: `src/scenes/Stress.ts`. Report: `src/stressReport.ts`. Look: `src/render/stress.ts`. +- Open it with `main.html?stress` (natively: `TN_DEBUG_STRESS=1`). Without `?stress` the starter course + opens, so the kit's own playtests still prove it. + +## What it loads + +The `real` preset: 4000 trees, 6000 rocks, 120 houses, 600 props, 120 animated rigs, 300 physics +bodies, 12 fires of 1500 particles, 16 lights, virtual shadow maps, and the engine's default look. +The camera orbits the town (radius 70 m, one turn in 40 s). + +| URL flag | Effect | +| --- | --- | +| `&max` | the `max` preset (2.5 to 3 times the `real` counts) | +| `&noTrees`, `&noRocks`, `&noHouses`, `&noProps`, `&noCrowd`, `&noBodies`, `&noFires`, `&noLights` | removes one layer, to price it | +| `&plainShadow` | regular shadow maps instead of virtual shadow maps | +| `&adaptive` | the adaptive quality tier instead of `high` | +| `&still` | stops the orbit and the walkers | +| `&closeup` | frames a sunlit walker, for shadow judging | +| `&freezeAt=` | holds the orbit at second `` after a real pan | + +## Put the latest engine in it + +1. Build the packages you changed (`pnpm --filter build`). +2. Copy the built files into `node_modules/` of the game folder, or make a fresh sandbox with + `pnpm sandbox --genre --name ` and copy `src/` across. +3. Move `node_modules/.vite` aside and restart Vite, or the browser keeps the old prebundle. + +## Measure + +1. Open `main.html?stress` in real Chrome through the Chrome DevTools MCP, with the cache ignored. + Read `adapter.info`: `swiftshader` means the numbers are not GPU numbers. +2. Wait 75 s. Startup, shader compiles and the world build are not stress. +3. Read `globalThis.__TN_STRESS__`: fps, frame, render, update and GPU p50, frame p95, presented p95, + JS-Wasm crossings and long tasks. +4. Record every run: `pnpm stress:loop record --label "" --decision keep|reject`. + The dashboard's Stress lab tab draws `docs/stress-loop/ledger.json`. + +## Compare two engines + +- Alternate the arms in one quiet window (A, B, B, A) with two or more runs per arm. Noise is about + ±7% between sessions and ±3% inside one. Compare render p50 and frame p95, not fps from different + sessions. +- Keep a change only on a paired gain. A count that falls (draws, crossings) is not a gain until frame + time falls with it. +- A change that can move pixels needs the visual judge from the repo's `AGENTS.md`. Capture the same + orbit time on both arms. +- Fix the engine in `packages/`, never the scene: the gain must reach every game. diff --git a/.github/actions/playwright-chromium/action.yml b/.github/actions/playwright-chromium/action.yml index 227bf0e786..9c39e4f027 100644 --- a/.github/actions/playwright-chromium/action.yml +++ b/.github/actions/playwright-chromium/action.yml @@ -24,12 +24,13 @@ runs: [ ! -f "$source" ] || sudo sed -i 's|http://azure.archive.ubuntu.com/ubuntu|https://archive.ubuntu.com/ubuntu|g; s|https://azure.archive.ubuntu.com/ubuntu|https://archive.ubuntu.com/ubuntu|g' "$source" done # The browser binary caches; its shared libraries are apt packages that do not, so - # `--with-deps` still runs. On a cache hit that is the only work left. + # `install-deps` still runs. On a cache hit that is the only work left. + # It takes 11-114 s; a mirror that trickles held it 29 minutes (run 41249368283), so each + # attempt is cut at 240 s and retried. Root runs the timeout, so it can kill apt's fetchers. - name: Install Chromium and its system dependencies shell: bash run: | - if [ "${{ steps.cache.outputs.cache-hit }}" = "true" ]; then - pnpm exec playwright install-deps chromium - else - pnpm exec playwright install --with-deps chromium + if [ "${{ steps.cache.outputs.cache-hit }}" != "true" ]; then + pnpm exec playwright install chromium fi + sh scripts/retry-download.sh sudo timeout --kill-after=15 240 env "PATH=$PATH" node_modules/.bin/playwright install-deps chromium diff --git a/.github/actions/scaffold-from-tarballs/action.yml b/.github/actions/scaffold-from-tarballs/action.yml index 83d1a6e21e..79976fe58f 100644 --- a/.github/actions/scaffold-from-tarballs/action.yml +++ b/.github/actions/scaffold-from-tarballs/action.yml @@ -42,7 +42,6 @@ runs: case "$package_name" in create-threenative) package_flag="--cli-package" ;; threenative-animation-mcp) package_flag="--animation-mcp-package" ;; - threenative-blender-mcp) package_flag="--blender-mcp-package" ;; threenative-engine-mcp) package_flag="--engine-mcp-package" ;; @threenative/*) package_flag="--threenative-${package_name#@threenative/}-package" ;; *) echo "unsupported workspace package $package_name"; exit 1 ;; diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 80d07a8ecd..b2d5274a27 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -929,8 +929,7 @@ jobs: # in, or this lane would quietly stop covering 21 scenarios it has always covered. extras="$GITHUB_WORKSPACE/packages/create-threenative/template-playtests/${{ matrix.template }}" if [ -d "$extras" ]; then cp -r "$extras/." "$target/playtests/"; fi - # PRD-748: the engine-cost contract row runs on every PR, for every kit that does not opt out. - pnpm exec tsx scripts/kit-contract-row.ts engine-cost "${{ matrix.template }}" "$target/playtests" + # The PRD-748 engine-cost row runs in `template-engine-cost`, beside this leg. scenario_list=$(node "$GITHUB_WORKSPACE/scripts/non-visual-scenarios.mjs" "$target") mapfile -t scenarios <<< "$scenario_list" test "${#scenarios[@]}" -gt 0 @@ -971,6 +970,72 @@ jobs: pnpm --dir "$target" exec threenative build --target web test -s "$target/dist/index.html" + # PRD-748's engine-cost contract row, every PR, per kit. It draws all 1260 ticks one by one for + # its four counter windows, and on hosted SwiftShader that alone took 14m51s (run 41170629638, + # adventure): inside a kit's 30-minute `template-nonvisual` leg it cancelled adventure and + # collectathon. It runs here in parallel, with the same scenario, bounds and ticks. A kit that + # opts out prints its reason and passes; `ci-required` gates this job like any other. + template-engine-cost: + needs: [scope, build-artifacts] + if: ${{ (needs.scope.outputs.selection == 'full' || needs.scope.outputs.selection == 'template') }} + strategy: + fail-fast: false + matrix: ${{ fromJSON(needs.scope.outputs.plan).templateMatrix }} + runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER || !contains(fromJSON(needs.scope.outputs.runner_admission || '[]'), format('ci:template-engine-cost:{0}', matrix.template))) && 'ubuntu-24.04' || vars.TN_RUNNER }} + # 45, owner-approved for this job only (2026-10-31): on 813c0166a the row step alone ran + # 964-1775+ s on hosted SwiftShader (VM speed spread 1.75x, 99% raster hostGap), and rain, + # adventure (twice) and shooter hit 30. Ticks, windows, bounds and every threshold are unchanged. + timeout-minutes: 45 + env: + TN_CI_SHA: ${{ needs.scope.outputs.candidate_sha }} + TN_PLAYTEST_ALLOW_SOFTWARE: "1" + steps: + - uses: actions/checkout@v7 + with: + ref: ${{ needs.scope.outputs.candidate_sha || github.sha }} + - uses: ./.github/actions/pnpm + - uses: actions/setup-node@v7 + with: + node-version: 20 + cache: pnpm + cache-dependency-path: | + pnpm-lock.yaml + packages/create-threenative/templates/*/package.json + - run: pnpm install --frozen-lockfile + # Before the row: its writer imports create-threenative, which needs this artifact's packages/*/dist. + - name: Take the packed workspace from build + uses: actions/download-artifact@v4 + with: + name: workspace-packages + - name: Write the kit's engine-cost row + id: row + run: | + set -euo pipefail + mkdir -p "$RUNNER_TEMP/engine-cost-row" + pnpm exec tsx scripts/kit-contract-row.ts engine-cost "${{ matrix.template }}" "$RUNNER_TEMP/engine-cost-row" + if [ -f "$RUNNER_TEMP/engine-cost-row/contract-engine-cost.playtest.json" ]; then echo "applies=true" >> "$GITHUB_OUTPUT"; fi + - name: Scaffold the template from local tarballs + if: steps.row.outputs.applies == 'true' + uses: ./.github/actions/scaffold-from-tarballs + with: + archives-directory: ${{ github.workspace }}/artifacts/workspace-packages + specs-file: ${{ github.workspace }}/artifacts/workspace-packages/specs + target: ${{ runner.temp }}/threenative-${{ matrix.template }} + template: ${{ matrix.template }} + - uses: ./.github/actions/playwright-chromium + if: steps.row.outputs.applies == 'true' + - name: Run the engine-cost row + if: steps.row.outputs.applies == 'true' + run: | + set -euo pipefail + target="$RUNNER_TEMP/threenative-${{ matrix.template }}" + pnpm --dir "$target" install + cp "$RUNNER_TEMP/engine-cost-row/contract-engine-cost.playtest.json" "$target/playtests/" + # `--server-timeout` as in template-nonvisual: the first asset cook holds the dev port. + pnpm --dir "$target" exec threenative-playtest --scenario playtests/contract-engine-cost.playtest.json \ + --browser-recipe webgpu --headed --no-screenshots --server-timeout 120000 \ + --server-command "pnpm dev --host 127.0.0.1 --port \$PORT --strictPort" + # `golden-path` is a required status check in the `main protection` ruleset, and required checks # match the reported context by exact string. A matrix job does not report its own name — it # reports `golden-path (starter)` and `golden-path (platformer)` — so the moment this lane became @@ -1491,7 +1556,7 @@ jobs: # The only required context after protected-develop cutover. Explicit exemptions are not passes. # Legacy main contexts stay in place until the owner changes protection after full qualification. ci-required: - needs: [scope, integration, native-platforms, native-engine, typecheck, lint, test, test-unit, test-native, test-browser, test-playtest, golden-path-template, template-nonvisual, golden-path, benchmark, build, build-artifacts, budgets, performance-contracts, supply-chain] + needs: [scope, integration, native-platforms, native-engine, typecheck, lint, test, test-unit, test-native, test-browser, test-playtest, golden-path-template, template-nonvisual, template-engine-cost, golden-path, benchmark, build, build-artifacts, budgets, performance-contracts, supply-chain] if: ${{ always() && !github.event.pull_request.draft }} runs-on: ${{ (github.event.pull_request.head.repo.fork || !vars.TN_RUNNER_LIGHT) && 'ubuntu-24.04' || vars.TN_RUNNER_LIGHT }} # 15, not 5: `fetch-depth: 0` pulls every branch and tag, which takes over five minutes on a @@ -1561,6 +1626,7 @@ jobs: - test-playtest - golden-path-template - template-nonvisual + - template-engine-cost - golden-path - benchmark - build diff --git a/.github/workflows/native-platforms.yml b/.github/workflows/native-platforms.yml index 704a1602ac..5f21d7508e 100644 --- a/.github/workflows/native-platforms.yml +++ b/.github/workflows/native-platforms.yml @@ -2624,7 +2624,8 @@ jobs: missing=() for config in packages/*/tsup.config.ts; do directory="${config%/tsup.config.ts}" - [ -f "$directory/dist/index.js" ] || missing+=("$directory/dist/index.js") + output="$directory/$(node -p "require('./$directory/package.json').main.replace(/^\\.\\//, '')")" + [ -f "$output" ] || missing+=("$output") done if [ "${#missing[@]}" -gt 0 ]; then echo "TN_IOS_PACKAGE_OUTPUTS_MISSING: ${missing[*]}" >&2 diff --git a/.github/workflows/npm-release.yml b/.github/workflows/npm-release.yml index 9589811bec..4edf130ee8 100644 --- a/.github/workflows/npm-release.yml +++ b/.github/workflows/npm-release.yml @@ -101,33 +101,6 @@ jobs: run: | sudo apt-get update sudo apt-get install -y libvulkan1 mesa-vulkan-drivers - - name: Install the pinned Blender used by the registry verifier - env: - BLENDER_VERSION: 5.2.0 - run: | - archive="${RUNNER_TEMP}/blender-${BLENDER_VERSION}-linux-x64.tar.xz" - checksums="${RUNNER_TEMP}/blender-${BLENDER_VERSION}.sha256" - install_root="${RUNNER_TEMP}/blender-${BLENDER_VERSION}-linux-x64" - curl --fail --location --retry 3 --retry-delay 2 \ - "https://download.blender.org/release/Blender5.2/blender-${BLENDER_VERSION}-linux-x64.tar.xz" \ - --output "$archive" - curl --fail --location --retry 3 --retry-delay 2 \ - "https://download.blender.org/release/Blender5.2/blender-${BLENDER_VERSION}.sha256" \ - --output "$checksums" - test -s "$archive" - ( - cd "$RUNNER_TEMP" - sha256sum --ignore-missing --check "$(basename "$checksums")" - ) - tar -xJf "$archive" -C "$RUNNER_TEMP" - blender="${install_root}/blender" - test -x "$blender" - actual_version="$("$blender" --version | awk 'NR == 1 { print $2 }')" - test "$actual_version" = "$BLENDER_VERSION" || { - echo "Expected Blender ${BLENDER_VERSION}, received ${actual_version}." >&2 - exit 1 - } - echo "THREENATIVE_BLENDER_PATH=${blender}" >> "$GITHUB_ENV" # The publish set, named one package per line so `pnpm publish:check` can assert that no # publishable package is missing from it. A package that exists, is not private, and is # absent here would be omitted from the release. @@ -148,6 +121,7 @@ jobs: pnpm tsx scripts/verify-native-release-commit.ts "$native_tag" # BEGIN GENERATED WORKSPACE PUBLISH SET # @threenative/assets + # @threenative/blender-bridge # @threenative/core # @threenative/metahuman # @threenative/physics @@ -161,7 +135,6 @@ jobs: # @threenative/ui # create-threenative # threenative-animation-mcp - # threenative-blender-mcp # threenative-engine-mcp # END GENERATED WORKSPACE PUBLISH SET - name: Run the single guarded release path @@ -228,33 +201,6 @@ jobs: run: | sudo apt-get update sudo apt-get install -y libvulkan1 mesa-vulkan-drivers libwebkit2gtk-4.1-0 - - name: Install the pinned Blender used by the MCP conversion proof - env: - BLENDER_VERSION: 5.2.0 - run: | - archive="${RUNNER_TEMP}/blender-${BLENDER_VERSION}-linux-x64.tar.xz" - checksums="${RUNNER_TEMP}/blender-${BLENDER_VERSION}.sha256" - install_root="${RUNNER_TEMP}/blender-${BLENDER_VERSION}-linux-x64" - curl --fail --location --retry 3 --retry-delay 2 \ - "https://download.blender.org/release/Blender5.2/blender-${BLENDER_VERSION}-linux-x64.tar.xz" \ - --output "$archive" - curl --fail --location --retry 3 --retry-delay 2 \ - "https://download.blender.org/release/Blender5.2/blender-${BLENDER_VERSION}.sha256" \ - --output "$checksums" - test -s "$archive" - ( - cd "$RUNNER_TEMP" - sha256sum --ignore-missing --check "$(basename "$checksums")" - ) - tar -xJf "$archive" -C "$RUNNER_TEMP" - blender="${install_root}/blender" - test -x "$blender" - actual_version="$("$blender" --version | awk 'NR == 1 { print $2 }')" - test "$actual_version" = "$BLENDER_VERSION" || { - echo "Expected Blender ${BLENDER_VERSION}, received ${actual_version}." >&2 - exit 1 - } - echo "THREENATIVE_BLENDER_PATH=${blender}" >> "$GITHUB_ENV" # Proves the registry path, which no other gate in this repository exercises: the sandbox, # the sweeps and verify-golden-path all resolve `file:` tarballs by design. - name: Install from the registry in a directory with no workspace above it diff --git a/.gitignore b/.gitignore index c870938b5d..0815bf2b3f 100644 --- a/.gitignore +++ b/.gitignore @@ -184,7 +184,7 @@ screenshot-*.png # scaffold the way a stranger's machine would; three of them were committed by accident in #37. *.tgz -# Python bytecode caches. `packages/blender-mcp/gpl/recipes` is executed by Blender, which +# Python bytecode caches. `packages/blender-bridge/gpl` is executed by Blender, which # writes a `__pycache__` beside the recipes on every run; one .pyc was committed by accident. __pycache__/ *.pyc diff --git a/AGENTS.md b/AGENTS.md index b5bd3ae076..ca40d5f3a6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,12 +26,13 @@ One source, two runtimes: browser WebGPU and an owned C++ host for desktop/Andro ## Pull requests - **Small changes skip the PR**: a tiny fix, a docs-only edit, a mirror regeneration — anything with no PRD — commits directly to local `develop` after its relevant checks pass. +- **Open a PR only when a PRD or the owner asks for one** (owner, 2026-10-28: "stop opening fucking PRs all the time"). Fix a red on an open PR by pushing to that PR's branch. Fix a red on `develop` by committing to the PR already in flight, or ask. A second PR for one release or one red is the failure, even when it looks small. - **One PR per PRD**, opened as a draft at the first commit — never one per phase, never several PRDs in one PR. Label it with the `prd:25/50/75/100` that `pnpm prd:progress` prints and update the label as phases land. - Branch from `origin/develop` (not a local branch that may be ahead of it), target `develop`, squash-merge. Set `git config threenative.integrationBranch develop` per checkout. - The commit that finishes a PRD also `git mv`s it to `docs/PRDs/done/`. - `main` takes only the ordinary full-checked `develop -> main` PR, merged with a merge commit, never squash or rebase. - **Push once per run**: while your PR's CI runs, commit locally; push to fix that run or after it finishes. A PRD tick rides with the next code push, never alone. -- **Short CI fix cycles.** Start a full CI run only when it is necessary: a release-candidate head or a required gate. A full run costs about an hour. Start the fix when the first red appears, and do not wait for the run to end. Prove the fix on the nearest lane (CI image, same software adapter, cold cache) before you push. Push all fixes for a run as one commit set and cancel the superseded run. `gh run rerun --failed` retries only a flake on the same commit. It cannot test a fix, and a partial rerun breaks the Integration receipts. Never trim the board or call a partial run green. +- **Short CI fix cycles.** Start a full CI run only when it is necessary: a release-candidate head or a required gate. A full run costs about an hour. Start the fix when the first red appears, and do not wait for the run to end. Prove the fix on the nearest lane (CI image, same software adapter, cold cache) before you push. Push all fixes for a run as one commit set and cancel the superseded run. `gh run rerun --failed` retries only a flake on the same commit. It cannot test a fix. Its green joins keep the receipts of the attempt that ran them, so it can turn `ci-required` green. Never trim the board or call a partial run green. - **No workflow file per feature**: add its CI proof as a job in an existing workflow. - Before retargeting, inventory `gh pr list` and `pnpm worktree:status`; retarget in-flight PRs one at a time and never rewrite another worktree. diff --git a/CHANGELOG.md b/CHANGELOG.md index fb3ab27cb2..dc900de129 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,15 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Breaking +- `@threenative/blender-bridge` (formerly `threenative-blender-mcp`) is now private and ships + no MCP server. The `threenative-blender` server is gone from `.mcp.json` and every other host + config, along with the `blender_status`, `blender_inspect`, `blender_convert`, + `blender_recipes`, `blender_export_world` and `blender_run_python` tools. Blender conversion at + build time is unchanged: `blenderImportPass` still converts `.fbx`, `.blend`, `.obj` and `.dae` + through the same bridge, and the bpy recipes stay in `packages/blender-bridge/gpl/` to run with + `blender --background --python`. Remove the stale `threenative-blender` entry from an existing + project's MCP config; a fresh scaffold no longer writes it. + - `@threenative/core#sceneWarning` takes an optional fourth `gpuClass` argument and returns no verdict for `"software"`: a CPU rasteriser puts the GPU's work inside the render phase, so "the GPU is idle" is false there. Existing three-argument callers diff --git a/CLAUDE.md b/CLAUDE.md index af9bac1885..5bd642d1ca 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,12 +28,13 @@ One source, two runtimes: browser WebGPU and an owned C++ host for desktop/Andro ## Pull requests - **Small changes skip the PR**: a tiny fix, a docs-only edit, a mirror regeneration — anything with no PRD — commits directly to local `develop` after its relevant checks pass. +- **Open a PR only when a PRD or the owner asks for one** (owner, 2026-10-28: "stop opening fucking PRs all the time"). Fix a red on an open PR by pushing to that PR's branch. Fix a red on `develop` by committing to the PR already in flight, or ask. A second PR for one release or one red is the failure, even when it looks small. - **One PR per PRD**, opened as a draft at the first commit — never one per phase, never several PRDs in one PR. Label it with the `prd:25/50/75/100` that `pnpm prd:progress` prints and update the label as phases land. - Branch from `origin/develop` (not a local branch that may be ahead of it), target `develop`, squash-merge. Set `git config threenative.integrationBranch develop` per checkout. - The commit that finishes a PRD also `git mv`s it to `docs/PRDs/done/`. - `main` takes only the ordinary full-checked `develop -> main` PR, merged with a merge commit, never squash or rebase. - **Push once per run**: while your PR's CI runs, commit locally; push to fix that run or after it finishes. A PRD tick rides with the next code push, never alone. -- **Short CI fix cycles.** Start a full CI run only when it is necessary: a release-candidate head or a required gate. A full run costs about an hour. Start the fix when the first red appears, and do not wait for the run to end. Prove the fix on the nearest lane (CI image, same software adapter, cold cache) before you push. Push all fixes for a run as one commit set and cancel the superseded run. `gh run rerun --failed` retries only a flake on the same commit. It cannot test a fix, and a partial rerun breaks the Integration receipts. Never trim the board or call a partial run green. +- **Short CI fix cycles.** Start a full CI run only when it is necessary: a release-candidate head or a required gate. A full run costs about an hour. Start the fix when the first red appears, and do not wait for the run to end. Prove the fix on the nearest lane (CI image, same software adapter, cold cache) before you push. Push all fixes for a run as one commit set and cancel the superseded run. `gh run rerun --failed` retries only a flake on the same commit. It cannot test a fix. Its green joins keep the receipts of the attempt that ran them, so it can turn `ci-required` green. Never trim the board or call a partial run green. - **No workflow file per feature**: add its CI proof as a job in an existing workflow. - Before retargeting, inventory `gh pr list` and `pnpm worktree:status`; retarget in-flight PRs one at a time and never rewrite another worktree. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 775ead2e9e..1733e9020e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -101,7 +101,7 @@ The **public surface** is every exported symbol, type, and export subpath of a p (`@threenative/core`, `@threenative/physics`, `@threenative/ui`, `@threenative/assets`, `@threenative/playtest`, `@threenative/runtime-native`, `@threenative/raw-unreal`, `@threenative/ueformat`, `create-threenative`, `threenative-engine-mcp`, -`threenative-blender-mcp`) — the API your game and its agents import. Everything else is internal +`threenative-animation-mcp`) — the API your game and its agents import. Everything else is internal and may change in any release. **Before 1.0** (`0.x.y`): `patch` is fixes only, `minor` may break. A break is always announced. diff --git a/PERFORMANCE-LEARNINGS.md b/PERFORMANCE-LEARNINGS.md index 4727e336fa..3201398a32 100644 --- a/PERFORMANCE-LEARNINGS.md +++ b/PERFORMANCE-LEARNINGS.md @@ -77,6 +77,81 @@ PRD's "Tried and rejected" or "Decisions" section, for example published state re-renders at the publish rate, not the rate of what it shows. `useUiState(selector)` compares the result by identity, so a selector that builds a new object or array renders as often as no selector. +## 2026-10-29 Stress lab (web Wasm engine, Chrome DevTools MCP, RTX 2080) + +The sandbox scene `sandbox/stress-lab` (`?stress`, real preset) holds houses, a forest, a skinned crowd, physics +bodies, fires and a virtual shadow map at the same time. Every row is in `docs/stress-loop/ledger.json` and on the +`pnpm dashboard` Stress lab tab. FPS went 1.36 -> 8.27 (six-window medians); the goal is 120. + +### 2026-10-29 — Virtual-shadow casters batch in the main pass — KEEP +- Where: `packages/runtime-native/src/engine/renderer/render_database.cpp`; `b4131e881`. +- Result: main draws 2795 -> 1836 at the same FPS (1.36 -> 1.3); visual judge NEUTRAL. A caster's shadow copy stays an + individual draw, so the per-page shadow cull still sees it. +- Insight: the draw count varied 1803-3168 between windows. A single-window draw claim is invalid; use medians. + +### 2026-10-29 — Rig shadow bounds come from bones, not CPU-skinned vertices — KEEP +- Where: `SkinnedMesh::boneReachBounds` (`animation/skinning/skeleton.cpp`), used by the VSM bounds loop in + `renderer.cpp`; `b95f4c8ef`. +- Result: 1.3 -> 4.8 FPS, GPU p95 125 ms, frame p95 223 ms. The crowd under VSM cost about 760 ms a frame because the + bounds loop skinned every vertex on the CPU. Now each bone has a reach radius measured once per geometry revision; + a call costs O(bones). Visual judge NEUTRAL on a close-up of a sunlit walker. +- Insight: UE 5.8.3 fits skinned bounds to bones. A bounds pass that skins vertices costs as much as skinning itself. + +### 2026-10-29 — A VSM pan keeps the shadow pages the window still covers — KEEP +- Where: `shadows/virtual/atlas.cpp`, `pages.cpp`; `8401791aa`. +- Result: 4.8 -> 8.27 FPS, frame p95 223 -> 159 ms, GPU p95 125 -> 72 ms, presented p95 533 -> 283 ms. The clipmap + re-centred at texel precision, so a camera orbit redrew every page of every level each frame. A pan now moves the + window by whole pages from its last origin (the first placement and a cut keep the exact texel origin, so the GPU + reference renders still match). Visual judge NEUTRAL; the orbit was frozen at 12 s after a real pan path. +- Insight: a world-grid snap broke the vsm-cut reference render (1.6% pixels). Snap relative to the previous origin + and only on a pan. + +### 2026-10-29 — Method: name web Wasm hot functions with `--profiling-funcs` — KEEP (method) +- A trace of the release Wasm shows `wasm-function[N]`. A symbol map from a relink does not match it, because + indices move. Relink with `-DCMAKE_EXE_LINKER_FLAGS=--profiling-funcs` and install it; Chrome then names every + function. Clear the flag and relink afterwards (a clean relink is byte-identical to the release build). +- Trap: any relink that changes the glue (`.mjs`) fails in the sandbox (`ASM_CONSTS[code] is not a function`, + `reading 'resolveQuerySet'`) until the Vite dev server restarts. Deleting `node_modules/.vite` is not enough: the + running server keeps the old prebundled glue in memory. +- Trap: removing one system (crowd, bodies or fires) cut `mainPass` from 112 ms to 21-38 ms each, so the costs do not + add. Bisect with all arms, then profile; do not sum single-system deltas. + +### 2026-10-29 — Method: the renderer's CPU stages are engine counters — KEEP (telemetry) +- Where: `Renderer::CpuStage` (`renderer.h`), `tnw_web_frame` 66-70, three-native `CPU_STAGE_FIELDS`; `5c6a695aa`. +- Use: set `window.__THREENATIVE_ENGINE_COST__ = true` (or debug); `TN_FRAME_BUDGET.counters.engine` then carries + `cpuVirtualPagesMs`, `cpuPlanMs`, `cpuShadowBuildMs`, `cpuPrepareMs`, `cpuEncodeMs` per frame (p50/p95). +- First reading, stress lab: virtual pages 11.3/64.4 ms, shadow build 1.9/89.1 ms, prepare 17/113 ms. The VSM + stages spike on page-redraw frames, so read p95, not p50. +- Insight: `renderPrepared` runs about 25 times a frame here; per-call checkpoints with a per-call threshold hid a + 70 ms total. Keep running totals and difference them per frame. + +### 2026-10-29 — A quaternion write syncs its object's rotation in JS — KEEP +- Where: three-native `eulerFromQuaternion` (`browser-backend.ts`), `d679189ff`. +- Cause: three's `Quaternion._onChangeCallback` called `Euler.setFromQuaternion` in the engine, one crossing per body + per frame; physics interpolation writes every body's quaternion (census: 1,732 a frame). +- Fix: the bridge runs Euler.cpp's formulas and writes the lanes. The engine's `ieee754.h` is V8's fdlibm, so + `Math.asin`/`Math.atan2` give the same bits; the spec checks all six orders and gimbal lock against three. +- Result: census 1,732 -> 0 a frame. Frame time not resolved: host load about 22 while a Codex lane linked. + +### 2026-10-29 — Animation reads are engine fields — KEEP +- Where: `registerAnimationAction`/`registerAnimationMixer` `b.fields` (`bindings_scene.cpp`), `09598f35a`. +- Cause: `AnimationPlayer.update` reads `action.timeScale`, `action.paused` and `mixer.timeScale` every frame; each + was a crossing (445 + 330 + 360 a frame). +- Fix: declare them as fields; the generic `fastGetter` reads Wasm memory. Count offsets from the first member of a + pointer-free block (`&time`): the registry is dumped by a 64-bit build and read by wasm32. A `static_assert` + pins the block. +- Result: census 3,696 -> 1,400 crossings a frame (plainShadow). Next on the list: `Mesh.updateMatrix` 300, + `SkinnedMesh.__tnVisibleInWorld` 240, `setEffectiveTimeScale` 174, `Group.getWorldPosition` 120. + +### 2026-10-29 — Worker threads forced on, web and native — KEEP (contract); no frame win yet +- Where: `web-engine.ts` (`4de07760e`), `job_pool.cpp` (`9634ebf03`). Owner order: "just force all to use worker + threads". +- Web pages load only `tn-native-engine-web-threads`; no isolation or no worker throws `TN_WEB_THREADS_OFF`, a missing + module `TN_WASM_THREADS_MISSING`. Native: `TN_JOB_WORKERS=0` throws `TN_JOB_WORKERS_OFF`; the default is 1-3 workers. +- Insight: the pool runs one job only, the world-matrix walk (`object3d.cpp`). Threads on does not move the frame + until the hot CPU loops (VSM pages, shadow build, per-pass culling) run on `JobPool::shared().forEach`. Lift the + 3-worker cap only after that work exists and an A/B shows it pays. + ## 2026-10-28 RTS kit probe (Chrome DevTools trace, web Wasm engine, real display) ### 2026-10-28 — RTS HUD panels re-render only when a shown value changes — KEEP @@ -505,3 +580,279 @@ no COOP/COEP); that lane is open. pixels. Count pixels where every AFTER run differs from every BEFORE run; a judge reading a sheet misses a thin stripe. - Revisit when: an opaque order that keeps the coplanar winner exists (depth order only among intersecting bounds). + +### 2026-10-28 — Web frame gate reads frame age, not a count of unfinished frames — KEEP +- Where: `packages/runtime-native/src/engine/renderer/frame_gate.h`, `wasm/web_host.cpp` (`tnw_web_frame` field 11); `597faf225`. +- Result: RTS kit, Chrome 153, RTX 2080 shared with other jobs. Frames with no GPU submit 1.52 % -> 0.35 % (1246 vs + 1155 frames, 20 s each); engine `renderSkips` per 5 s window 11-16 -> 0-6. A 3 ms GPU frame's work-done promise + resolved at p90 24 ms and p99 39 ms, so "two frames unfinished" declined frames the GPU had already drawn. +- Insight: on the web, completion arrives as a promise behind every earlier map; a gate on its count measures browser + scheduling, not the GPU. Decline only when the oldest unfinished frame is 4 loop intervals old (queue cap 6). +- Revisit when: presentation gets its own completion marker (Codex diagnosis: a per-frame mapped buffer), or skips + return; the timestamp readback still maps every frame until `trackTimestamp` reaches the native timer. + +### 2026-10-28 — AstraCraft (three r140, WebGL) as the RTS reference — KEEP (diagnostic) +- Result: same Chrome, same host load: AstraCraft holds rAF 59.8 with a main thread as busy as ours, and never skips a + frame. Our Wasm RTS: render phase p50 5.5-6.3 ms (three.js 7.4-9.1), GPU p50 2.6-2.8 ms (three.js 3.8), but update + phase p50 2.7-3.4 ms against three.js 0.6 ms: about 1,000 ABI crossings per `army.sync`, a 112x112x4 fog texture + widened to 401 KB of doubles per refresh, timestamp readbacks every frame. Fixes on `fix/wasm-update-crossings`. +- Insight: the existing bridge guard flags only redundant reads; writes and distinct reads passed silently + (telemetry lane `feat/web-engine-debug-telemetry`). + +### 2026-10-29 — Stress lab: virtual shadow pages pick their casters on the job pool (F2) — KEEP +- Where: `packages/runtime-native/src/engine/renderer/renderer.cpp` (`selectFace`), `128419fb5`. +- Result: `sandbox/stress-lab` `main.html?stress` (4000 trees, 6000 rocks, 120 rigs, 300 bodies, 18000 particles, + 16 lights), threaded Wasm, RTX 2080: 12.62 -> 16.33 fps on a loaded host, presented p95 150 -> 100 ms. Visual judge + NEUTRAL. Serial and pooled shadow plans are byte-identical over 20 frames (`native_engine_virtual_shadow_page_jobs`). +- Insight: per-page caster filtering is read-only on the atlas (`isMover`, `drawsStatic`, `casterGate`, `overlaps` are + `const`), so each page filters into its own output and the caller merges in page order. Split only when + `passCount * opaque >= 16384` and never inside a job (`JobPool::inJob()`). +- Revisit when: `kMaxWorkers` (3) rises; the page job is the first to scale with it. + +### 2026-10-29 — Stress lab: 660 bridge crossings a frame cut (hierarchy, visibility, matrix restores) — KEEP +- Where: `packages/three-native/src/browser-backend.ts`, `packages/core/src/render-camera-cull.ts`, + `packages/core/src/interpolation.ts`, `object3d.h` flag block; `d9bfd2aa4`. +- Result: same scene, develop core: 13.45 -> 14.03 fps, crossings p50 2520 -> 1838, frame p50 48.0 -> 45.4 ms. Judge + NEUTRAL. +- Insight: three cuts: `__hierarchyRevision` is answered from the kept hierarchy (its only writer is `markHierarchy`); + the camera cull walks visibility in JS instead of one `visibleInWorld()` call per object; the interpolation restore + writes the `matrixWorldNeedsUpdate` byte in place instead of `updateMatrix()` through the bridge. +- Revisit when: `setEffectiveTimeScale` (174 crossings) gets an engine "no interpolant" answer, and `getWorldPosition` + (120) has a field that is not one frame stale. + +### 2026-10-29 — Stress lab: most of the "12 fps" was host load, not the engine — KEEP (method) +- Result: develop f3215f431 read 12-16 fps while the host load average was 10-75 (a full `vitest run` in another lane, + two fixer C++ builds, a stray capture tab). After a reboot, at load 2.2 on 16 threads, the same build read + **25.54 fps** (frame p50 31.1, render p50 24.0, update p50 6.5, GPU p50 5.8 ms). +- Insight: the stress frame is CPU-bound on the main thread, so every busy core costs frame time. Measure only below a + 1-minute load of about 8, with one stress page open, and compare arms back to back. `nice -n 19` on lane builds. +- Revisit when: the frame becomes GPU-bound; then GPU sharing matters more than CPU load. + +### 2026-10-29 — Stress lab: shadow bounds on the job pool (F3) — REJECT, then root-caused +- Where: `perf/stress-fix-c` 5b5991370, `renderer.cpp` virtual caster bounds per ownership unit. +- Result: paired, same JS, only the Wasm differs, load 2.3: 25.54 -> 9.50 and 9.44 fps, render p50 24 -> 80 ms. The + pool itself ran in 1.5 ms. The cause (hunter H4): bone discovery ran `std::find` over the growing bone list for each + of 15,600 bone references (120 rigs x 65 bones, main and shadow copies): 60.84 million comparisons, 57-59 ms a frame. +- Insight: a byte-identical red-green test proves output, not cost. A lever that adds per-frame bookkeeping over every + bone or object needs a work-count assertion (comparisons or visits per frame) that fails when the work scales with + the square of the scene. Fix (F8): an `unordered_set` beside the ordered vector. +- Revisit when: F8 lands; measure F3+F8 against develop before F3 is kept. + +### 2026-10-29 — Stress lab: the virtual shadow LOD sync skips its scene walk while nothing it reads moved (FS1) — KEEP +- Where: `packages/three-native/src/shadow-node.ts` (`walkStillCurrent`), `browser-backend.ts` geometry setter, + `packages/core/src/model-lod.ts` (`bumpChains`); `0465c88e7`. +- Result: three paired runs, same Wasm: 22.43 -> 26.98 fps (+20%), render p50 26.3 -> 20.7 ms. Judge NEUTRAL. +- Insight: `syncVirtualShadowLods` ran from every `[DRAW_TARGET]` (each post stage re-enters the world draw), so one + JS scene walk cost 3.6-4 ms a frame through `accessor.get`. A per-frame JS walk is cheap to skip when its answer has + a small set of inputs: here the hierarchy revision, the chain registration count and the geometry assignment count. + The hunter's skip missed the third input (a `mesh.geometry` swap moves no hierarchy); list every writer of the + answer before you cache it. +- Revisit when: a fourth writer of a mesh's LOD chain appears (an engine-side geometry swap). + +### 2026-10-29 — Stress lab: F3 bounds on the pool + F8 bone hash set — REJECT; FS2 one caster record — REJECT +- Result: F3+F8 paired with develop: 21.98 against 22.77 fps. FS2 (`unordered_map` caster record per light) alone: + 21.65 against 22.43; with FS1: 24.44 against FS1 alone 26.98. +- Insight: F8 did remove the 61M-comparison bone dedupe, but pooled bounds still gave no gain on this scene. FS2 builds + its container every frame, and an `unordered_map` build (buckets plus one node per caster) costs more than the two + `std::map` lookups it saves, mostly on workers already. A per-frame container needs reuse or a flat array indexed by + draw ordinal before its lookups pay. +- Revisit when: the caster record lives in a flat array that survives frames. + +### 2026-10-29 — Stress lab: the first frame waits for shadows and exposure (F1+F5) — KEEP +- Where: web host ABI 8 `tnw_web_set_presentation_hold`, `packages/core/src/startup-readiness.ts`; `60de162f0`. +- Result: first revealed frame mean 106-110 with cast shadows, against 140-143 without shadows, a 1 s gap, then a dark + frame. Steady state unchanged (17.24 against 17.02 fps, paired on a loaded host). Judge IMPROVEMENT 4/4. +- Insight: a held frame does every pass (shadows, post, temporal history, the exposure meter) but takes no canvas + texture, so the engine warms itself up without presenting. Bound every such wait: 3 s from the frame window or from + any later cold picture, with one `TN_STARTUP_IMAGE_TIMEOUT` line naming the cold input. + +### 2026-10-30 — Stress lab: the job pool takes up to 8 workers (FT2) — KEEP +- Where: `packages/runtime-native/src/engine/foundation/job_pool.h` (`kMaxWorkers`), `job_pool.cpp`; `2ec6ed19d`. +- Result: develop head JS, paired rounds: 27.40 -> 30.30 and 23.96 -> 26.20 fps (+9 to +11%), long tasks down. +- Insight: a bench of the pooled job's shape (132 items x 0.165 ms) scales 3.89 lanes at 3 workers, 7.53 at 8, 8.76 at + 15 and 8.50 at 23: past 8 the item count is the wall, not the cores. The web count is + `navigator.hardwareConcurrency - 1` capped at 8, and Brave reports a masked value (15 on a 24-thread host), so read + the reported count before you claim a worker number. A pool where no worker starts now throws + `TN_JOB_WORKERS_START_FAILED` instead of running on one thread. +- Revisit when: a second pooled job lands, or the page job's items grow past about 8 x 0.2 ms. + +### 2026-10-30 — Stress lab: shadow page instancing (FM8) + caster page index (FS9) — KEEP +- Where: `renderer.cpp` virtual shadow selection and row planning, `atlas.cpp` `PageAtlas::pageRange`, + `standard.cpp` per-instance skin values; `04722134d`. +- Result: develop ~27.3 fps -> 31.1 fps (+14%) in alternating windows; FS9 alone halves long tasks (2469/1704 -> + 1166/1200) and cuts frame p95 by ~3 ms, FM8 alone cuts render p50 21.5 -> 19.5 ms. Judges NEUTRAL. +- Insight: on a shadow page, one row per caster was the cost, not the duplication across pages (FS7 measured + dedupe at 0.17 ms) and not the casters outside a mover page's window (FS8: the safe x/y test costs what the draw + costs). Group a page's selection into instanced rows and index the selection by page range; never widen a group + past the page's own selection. A crowd's shadow skin can reuse the main pass palette per instance. + +### 2026-10-30 — Stress lab: FU4 one engine call per rig step — REJECT +- Where: `AnimationMixer.updateAndSample` plus a `scheduled` field on `AnimationAction` (worktree only). +- Result: crossings 1276-1296 -> 994-1002 (-22%) but fps 25.99 -> 24.96 and update p50 7.36 -> 7.42 ms over 3 + runs per arm. Insight: the crossing count is not the cost; a returned list allocates per call, and the profiled + "glue" self time did not turn into frame time. FS3 (protected-page set built on grant frames only) measured inside + the noise in its own runs and is parked. + +### 2026-10-30 — Stress lab: main-pass plan cuts FM1+FM2+FM3 — KEEP +- Where: `render_database.cpp` (`refresh` guard, `convertedInputs`), `renderer.cpp` (per-program frame slab, + dirty-block upload, `program.probes`); `07703efaf`. +- Result: stacked on develop with FT2, alternating arms in one lock window: 27.07 -> 27.96 fps mean (+3.3%), render p50 + 22.1 -> 21.4 ms. Judge NEUTRAL. Solo runs had read +6 to +8% each: that was noise between lock windows. +- Insight: keep the material half of a draw on its record, guarded by the material's uniform signature plus the + geometry revision, and list every writer the guard misses before you trust it. The review found two: a converted + (non-float) attribute copy that `needsUpdate` replaces without a revision, and an `onBeforeRender` write after the + signature was taken. Both reproduced red in `draw_refresh_guard_holes`. Run-to-run noise on this frame is about +-7% + between sessions and +-3% inside one: compare render p50 across alternating arms, not fps across sessions, and + measure a lever under 1 ms by its function's self time in a profile. + +### 2026-10-30 — Stress lab: web shadow bundles (FM5), stride copy and unmoved-body skip (FU2+FU3) — REJECT +- Result: FM5 25.23 and FU2a+FU3 24.35 against develop head 26.62 / 26.12 fps in one lock window. +- Insight: FM5 turns PRD-754's recorded shadow bundles on for the web: +53 KB of Wasm and no gain on a moving camera, + as PRD-754 found for the shooter. FU2a replaced one `getWorldPosition` with `updateWorldMatrix(true, false)` plus a + matrix copy, and bridge crossings rose 1280 -> 1527: count crossings before you call a bridge read cheaper. FU2b's + per-frame `isScheduled()` cache was dropped in review: six paths start or stop actions, and only two cleared it. +- Revisit when: FU3 (skip the lerp for a body physics left unmoved) gets its own profile run; it is logic-safe and + about 0.3 ms, below what fps can show. + +### 2026-10-30 — Stress lab: shadow caster records indexed per draw + numeric page ids (FS11+FS12) — KEEP +- Where: `renderer.cpp`/`renderer.h` (`casterRecords_`, one vector per light, indexed by draw position, replaces the + per-light `std::map`s keyed by `item.key`), `shadows/virtual/pages.{h,cpp}` (`pageId(level,x,y)` replaces string + page keys), `atlas.cpp` (`protectedIds` as an `unordered_set`). +- Result: two clean lock windows, alternating arms, 1280x720, RTX 2080: 34.6 / 34.4 / 35.1 / 35.9 fps -> 46.0 / 45.5 / + 44.6 / 45.4 fps (+30%). GPU p95 62 -> 34 ms, frame p95 51 -> 28 ms, presented p95 50-59 -> 33 ms, long tasks 800-1100 + -> 72-96, main draws 150 -> 60 at equal triangles (7.07 M). Native tests 115/116 (material_props fails the same way + on develop: `tsx` missing in a worktree). Visual judge NEUTRAL: shadow outliers 0 of 4 per arm over 8 alternating + captures; the first review's 2 outliers (one per arm) were both slow runs, a flake that already existed. +- Insight: the fixer's profile estimate was 0.9 ms of CPU; the gain is ten times that. Read-only trace (verdict LEGIT): on the web build a batched draw group keeps its first member's + id as its key (`render_database.cpp:1432-1436`), so a casting group and its non-casting main-pass twin share one + key. The old `lightCasterBounds.count(item.key)` rule made the twin a caster state too, the two states alternated + under one key, `Motion::moving` never settled, and every mapped level got a mover page every frame + (`atlas.cpp:290-345`). 87 of the 147 counted draws were those shadow redraws: the web host reports main plus + shadow draws as `info.render.drawCalls` (`web_host.cpp:787-789`). `DrawItem::key` is not unique on the web build, + so any map keyed by it can merge two draws. A lever that + removes a per-frame string or map key can also fix what the key was wrong about: measure the whole frame, not only + the function the profile named. + +### 2026-10-30 — Stress lab: FS13 (plain-shadow lever) — REJECT; FP2 (one pass per re-encoded level) — PARKED +- Result: FS13 on `?stress&plainShadow` 59.59 / 59.81 vs develop 59.81 / 56.4 fps; render p50 11.65 / 11.71 vs 10.46 / + 12.5 ms: no paired gain, the arm already sits on vsync. FP2 37.4 / 37.4 vs develop 35.1 / 35.9 fps (+5%) but GPU p95 + 60.4 ms, unchanged: it does not touch the camera-motion burst it was written for. +- Insight: a lever for the camera-motion shadow burst must show GPU p95 falling, not only fps. FP2's worktree mixed two + workers' changes after the arm wrapper restarted a stopped worker: stop a lane by its wrapper, not its child. +- Revisit when: FP2 is split into its two changes and each is measured on top of FS11+FS12. + +### 2026-10-30 — Stress lab: mimalloc on the threaded web engine (FB1) — KEEP +- Where: `packages/runtime-native/cmake/NativeEngineCore.cmake`, the `TN_WEB_THREADS` link options (`-sMALLOC=mimalloc`); + lock: `packages/runtime-native/__tests__/web-threads-allocator.spec.ts`. +- Result: two fair uncapped windows, ABBA, develop `0974fd3b8` vs the same tree plus mimalloc. Window fb1c (load peak + 4.66): CPU frame p50 14.75 -> 13.76 and 14.65 -> 13.73 ms (-6.7%, -6.3%), fps 64.1 / 65.6 -> 70.7 / 70.7. Window fb1c2 + (load peak 5.26, ledger rows #65-#66): 14.64 -> 13.60 and 14.18 -> 13.63 ms (-7.1%, -3.9%). Pixels cannot move. +- Insight: Emscripten's default dlmalloc takes one global lock in a `-pthread` build, so the job pool's workers and the + main thread queue on every allocation. The single-thread build has no contention and keeps dlmalloc. Two earlier + profiled pairs read -5.5% and -23%; the second was voided (a stray stress page shared the GPU 07:30-09:20Z), which is + why the decision rests on the fair window. +- Revisit when: the engine stops allocating per frame on worker threads, or Emscripten changes its default allocator. + +### 2026-10-30 — Stress lab: a virtual shadow page draws only the instances that reach it (FG1, PRD-773) — KEEP +- Where: `renderer.cpp`/`renderer.h` (`DrawCache::shadowInstanceBoxes`, per-level instance index by page cell, compacted + row per static page), `atlas.cpp` + `waste_counters.h` (refresh-reason counters), lever `TN_SHADOW_INSTANCE_CULL=0`. +- Result: two clean pairs on `?stress`, 1280x720, RTX 2080, alternating arms: GPU p95 34.03 / 33.98 -> 7.54 / 7.33 ms + (-78%), frame p50 14.02 / 14.63 -> 14.09 / 14.86 (+0.5% / +1.6%, noise), presented p95 16.72 both. `?stress&still` + frame p50 13.12 / 13.17 -> 13.32 (+1.1 to +1.5%, inside the 3% gate). Native: the new case draws 1,118 instances over + 124 pages instead of 180,224 over 176 on develop, pixels byte-identical. +- Insight: the tail was the instanced static set, drawn whole on every refilled page (14,000 instances x up to 256 + pages). Counting pages or passes was the closed lever (FP2); culling the instances against the pages by their own + boxes (Unreal `VirtualShadowMapBuildPerPageDrawCommands.usf:299-309`) removes the work itself and costs nothing on a + still frame. After it, the remaining p95 frames are rebuild grants (0.16 per frame against 0.01 pans), 256 pages each: + a window that moves while its depth fit changes keeps no page. +- Judge: two Opus rounds; round 2 NEUTRAL (3 AFTER frames inside BEFORE-vs-BEFORE variation). A single odd frame was a + different camera pose at capture: take 3+ frames per arm before you read a shadow outlier as a cull defect. +- Cost: the fair windows show CPU frame p95 up about 1.3 ms on the granting frames (16.59 -> 17.85 ms vsync, 17.25 -> + 18.64 uncapped, rows #67-#70): the per-level index build, counting sort and 80 B per (page, instance) copy run on the + main thread when pages are granted. A p50 gate hid it; gate a shadow lever on CPU p95 too. +- Revisit when: GPU p95 on `?stress` rises above 16.7 ms again, or a game keeps casters that are not instanced (they + still draw whole on a rebuild); then FG2 (fixed depth band with a pan anchor). + +### 2026-10-30 — Method: the render phase splits into named stages on every window — KEEP (telemetry) +- Where: `packages/core/src/profiling/FrameStages.ts`, `FrameBudget.addStages` and `stagesAtP95`, `Renderer::CpuStage` + + `RenderDatabase::sceneMs/rendererMs` + `web_host.cpp` stage totals (`tnw_web_frame` 78). +- Use: every `TN_FRAME_BUDGET` window carries `stages`, `stageSum` and `stagesAtP95`; `?tnStages=0` turns the meters off + for the A/B of their own cost. +- Result (stress lab, `?stress`, develop Wasm, fair window): frame p50 14.70 ms = plan 4.00 + scene 2.83 + JS before the + call 2.65 + update 2.26 + engineJs 0.81 + render targets 0.72 + remainders 0.80 + post 0.25 + encode 0.12; submit, + poll, compute 0. The p95 tail (+5.3 ms) is 71% shadow refresh (encode +1.4, VSM pages +1.4, shadow lists +1.0), 22% + crowd and JS. Meters on vs off: +0.05 ms (5 valid pairs). +- Insight: publish the subtractions (`engineJs`, `wasmOther`, the gaps) as stages. A named remainder is a number to + chase; the first build had a 1.25 ms `wasmOther` that two extra meters split into 0.89 ms of render targets and a 0.39 + ms remainder. +- Insight: a meter that nobody reads is not free. Metering sibling renderers (render targets, reflections) cost 0.2 ms a + frame on Wasm (6 of 6 pairs); gate the meter to the renderer whose totals are published. +- Insight: read engine totals in place from linear memory. The renderer ABI census is pinned, and ten extra host calls a + frame broke it. +- Revisit when: a Chrome build with precise `performance.memory` runs the stress lab (`jsAllocKbP50` reads 0 under + Brave's bucketing). + +### 2026-10-30 — Stress lab: the sun map culls skinned casters by bone reach (add-on 5, PRD-770) — REJECT + +- Where: `RenderDatabase::skinnedShadowSphere` (patch kept in `~/.cache/tmp-tn/stress-loop/lanes/addon5.patch`). +- Result: plain sun shadow triangles 2.06 → 1.29 M (equal to vanilla three), pixels unchanged (native red-green + `skinned_caster_shadow_culling`). CPU frame p50 on `?stress&plainShadow` 9.09 → 10.17 ms vsync and 9.09 → 10.05 ms + uncapped (+11%, both pairs), GPU p95 unchanged; `?stress` (VSM) unchanged. Rows #77-#84. +- Insight: 120 posed bone-reach boxes a frame cost more CPU than the 0.77 M culled shadow triangles save on the GPU. + Price a cull by its per-frame test cost. The plain sun map's 6 ms GPU is not triangle-bound here. +- Revisit when: the skin update already publishes a per-rig bound this frame, so the test reads it in O(1). + +### 2026-10-30 — Stress lab: keep the virtual-shadow instance index across frames (FG1b, PRD-773) — REJECT + +- Where: `DrawCache::shadowInstanceIndices`, lattice-keyed (`PageAtlas::lattice`, `pageRangeOnLattice`); patch kept in + `~/.cache/tmp-tn/stress-loop/lanes/fg1b-on-stack.patch`. Native red-green: a pan regrants the level and builds 0 + indices (`virtual_shadow_page_draws_only_its_instances` section c); 197 native cases pass. +- Result on the stack (rows #89-#92): CPU p95 17.79 → 18.04 ms vsync, 18.95 → 18.83 ms uncapped; p50 flat; GPU flat. +- Insight: the mechanism works, but the index build was not FG1's CPU p95 cost. Look for that cost elsewhere in the + granting frames, and not in the per-level index. +- Revisit when: a per-stage p95 split (PRD-771 meters at p95) names the granting-frame stage that grew with FG1. + +### 2026-10-30 — Stress lab: unchanged instanced draws skip the per-frame copy and compare (TR1 + TR1b, PRD-774) — KEEP + +- Where: `Renderer::renderPrepared` stable per-draw slots plus a stamp (store epoch, writes, version, count and a + never-reused `BufferStore::id()`); the stamp commits only after the upload returns Ok. Counter + `lastFrame().instanceCopies`. +- Result on the stack (rows #85-#88): uncapped CPU frame p50 −4.9% and −7.6%, p95 18.13 → 17.64 ms, GPU p95 + 7.52 → 7.39 ms; vsync −2.5% and −3.7%. Upload bytes 3.32 → 2.10 MB a frame. Opus judge NEUTRAL. +- Insight: the stress lab's 14,260 static instances are user `InstancedMesh` batches, not the auto-batcher, so the + copy sat in `renderPrepared`. TR1 alone (stable slots) moved bytes but not time; skipping the unchanged draw is + the part that pays. +- Risk: a game that edits `instanceMatrix.array` without `needsUpdate` no longer sees the edit, which matches + three's own rule. + +### 2026-10-30 — Stress lab: image pixels cross into the engine as bytes, not boxed numbers (FL1, PRD-770) — KEEP + +- Where: `packages/three-native/src/browser-backend.ts` `toEngine`: a `Uint8ClampedArray` (an image's `getImageData` + pixels) now takes the typed-array fast path; `setImageBytes` in C++ already accepted it. +- Result (rows #95-#96, vsync): stress-lab `loadMs` 1550 → 542 and 1490 → 501 ms (−65%, both pairs), below vanilla + three's 548.5 ms; CPU frame p50 +0.5% (inside noise). Red-green spec in `browser-backend-coverage.spec.ts`. +- Insight: one 4096×2048 sky crossed as 33.5 M boxed numbers, written as 268 MB of float64: 1.56 s of a 1.98 s load + window. The rest of the scene built in 276 ms. Profile the load window before you suspect the scene build. +- Next: FL2, the same image is read back twice through an `OffscreenCanvas` (about 180 ms). + +### 2026-10-31 — Stress lab: a slow frame adds at most 1/20 s of simulated time (core FixedStepLoop) — KEEP + +- Where: `packages/core/src/loop.ts` `FixedStepLoop`, `maxPhysicsDeltaTime` (default 1/20 s, after Unreal's + `MaxPhysicsDeltaTime` 1/30, UE 5.8.3 `PhysicsSettings.cpp:22`, `PhysLevel.cpp:212`). Commit 24c5fe53a. +- Cause: from load 1 to load 4 (4x bodies) the physics step grew x24 and the frame fell to 12 fps, because each slow + frame ran up to 5 whole game updates to catch up, which made the next frame slower still. Profile + `lanes/prof-ramp/l1.json`, `l4.json`. +- Result at load 4, plain sun, fair ABBA against vanilla three.js (rows #128-#131): vsync 20.07 fps against 16.17, + presented p95 66.7 against 83.3 ms; uncapped 20.03 against 16.00 fps, p95 65.6 against 82.2 ms. Before the + clamp, ThreeNative read 15.79 fps and p95 100 ms at load 4 vsync (row #119), behind vanilla. +- Insight: under overload a fixed-step loop must slow the simulation, not catch it up; the catch-up is a spiral. + 1/30 s broke the 30 fps determinism spec (a 34 ms frame dropped a tick), so the default keeps 1/20 s. +- Risk: an overloaded game now runs its simulation slower than wall time (as Unreal does). Vanilla still catches up + with up to 4 Rapier steps, so the two compare frames delivered, not simulated seconds. Override: + `maxPhysicsDeltaTime: Infinity`. +- Ramp against vanilla three.js, fair ABBA, plain sun: load 2 vsync 54.3 against 35.3 fps, uncapped 53.2 against + 37.8 (rows #145-#148); load 8 vsync 9.65 against 7.6 fps, uncapped 9.5 against 7.9 (rows #149-#152). + ThreeNative leads at every load from 2 to 8. Against develop at load 4: vsync 19.1-19.6 against 17.1 fps + (+12-15%), uncapped 20.1 against 19.2-19.9 (+1-5%, inside noise) (rows #141-#144). +- Open: ThreeNative update p50 is still 28 ms against vanilla's 13 ms at load 4, and GPU p95 is 7.5 against 4.2 ms + at load 2, 21-24 against 14 ms at load 4 and 44 against 21 ms at load 8. diff --git a/README.md b/README.md index 869952c8c7..2eda6d5d4c 100644 --- a/README.md +++ b/README.md @@ -148,8 +148,8 @@ Details in [`create-threenative`](packages/create-threenative/README.md). | `@threenative/ui` | React bindings for HUD and game state | | `@threenative/ueformat` | UEFormat v10 `.uemodel` parser and Three.js loader for meshes exported from Unreal packages | | `threenative-animation-mcp` | Inspects, attaches, edits and verifies character animation through one MCP, as build-time edits of one GLB | +| `@threenative/blender-bridge` | Drives an installed Blender headlessly so a downloaded `.fbx` becomes a loadable GLB — ships no Blender | | `threenative-engine-mcp` | Offline MCP capability discovery for authoring agents | -| `threenative-blender-mcp` | Drives an installed Blender headlessly so a downloaded `.fbx` becomes a loadable GLB — ships no Blender | ## Platform support @@ -185,13 +185,13 @@ Every Unreal 5.8.3 system that sets the look or the frame cost, split into 380 s measured in lines of the Unreal source. Each one maps to the ThreeNative PRD that ports it. **20% done · 24% in progress · 33% not started · 16% deferred · 5% with no PRD** (share of 676K Unreal lines; the tool-heavy Niagara, Landscape, World Partition, RHI and RenderCore modules are left out) -· 82 open PRDs on this branch · 72 subsystems with no PRD +· 84 open PRDs on this branch · 72 subsystems with no PRD [![Unreal port progress map](docs/PRDs/Unreal/research/unreal-port-status.svg)](docs/PRDs/Unreal/research/unreal-visuals-and-performance-feature-map.html) -**Pace:** 118 PRD boxes ticked a week over the last 2 weeks. The port order runs waves 1-4 first, -so at that pace the default look is done about **Nov 17, 2026**, and everything filed today about -**Dec 8, 2026**. Scope grows as PRDs are filed, so read the dates as the earliest finish. +**Pace:** 102 PRD boxes ticked a week over the last 2 weeks. The port order runs waves 1-4 first, +so at that pace the default look is done about **Nov 24, 2026**, and everything filed today about +**Dec 18, 2026**. Scope grows as PRDs are filed, so read the dates as the earliest finish. [![Unreal port burn-up](docs/PRDs/Unreal/research/unreal-port-progress.svg)](docs/PRDs/Unreal/research/unreal-visuals-and-performance-feature-map.html) diff --git a/artifacts/engine-load-test/prd-449/holdout/README.md b/artifacts/engine-load-test/prd-449/holdout/README.md deleted file mode 100644 index 6e890b1858..0000000000 --- a/artifacts/engine-load-test/prd-449/holdout/README.md +++ /dev/null @@ -1,78 +0,0 @@ -# Real-game holdout — the projection perf changes, on the games people actually get - -Date: 2026-09-28. Machine: NVIDIA GeForce RTX 2080 (turing, 615.71.09), session display `:0`. -Record: `holdout-2026-09-28.json`. Captures: `*-{b,a}{1,2}-after.png`. - -## What was held out - -Two engine perf changes in `packages/core` on this branch against `origin/develop`: - -1. the projection reconcile skip (`2d856ac19`, `288198dc5`, `projection-stability.ts`) -2. batching meshes whose materials differ only in colour into instanced draws - (`7f8042277`, `projection-uniform.ts`) - -## Lane - -Each template's **own shipped** `playtests/performance.playtest.json`, served by the template's own -`pnpm dev`, driven by the repo's playtest CLI with `--browser-recipe webgpu` and -`TN_PLAYTEST_HOST_DISPLAY=1` so the run used the hardware adapter on the session's own display -(`adapter.vendor = nvidia`, never SwiftShader). One step was prepended to each scenario — a -3000-tick `scene-loaded` wait — so the template's own measured steps are steady state rather than -load. The engine's series is a 1024-sample ring holding the run's last 1024 frames; frames before the -world is up render nothing and are dropped. - -The runner freezes the game clock, so these are **uncapped fixed-step frame times, not presented -FPS**. Every run passed its template's own frozen budget (racing 330 draws / 105k tris, shooter 420 / -7.2k). - -## Templates - -Measured renderable counts across all ten kits (`TN_STARTUP_WARMUP` pipeline candidates): -shooter 100, racing 96, action-rpg 81, starter 70. The two heaviest are **shooter** and **racing**. - -## Arms - -- **BEFORE** — `packages/core/src` at `origin/develop` for exactly the six files the perf commits - touched (four rewritten, `projection-stability.ts` and `projection-uniform.ts` deleted), core - rebuilt, fresh `dist` copied into each scaffolded game. -- **AFTER** — `git checkout HEAD -- packages/core/src`, rebuilt, same copy. - -The two built `dist`s differ only in the projection code: AFTER adds 15 projection functions and the -`ProjectionStability` class, +627 lines / +24 KB in `index.js`. - -## Controls run because the first result looked too clean - -- Blocks 1-3 ran BEFORE first, blocks 4-6 ran AFTER first, so block position is separable from arm. -- `bb1`/`bb2`: the BEFORE arm run again after a second rebuild. -- `same1-3`: three back-to-back runs of one identical BEFORE dist, no rebuild. - -## Result - -| template | p50 before | p50 after | p95 before | p95 after | draws before | draws after | -| --- | --- | --- | --- | --- | --- | --- | -| racing | 7.4 ms | 6.8 ms | 13.0 ms | 11.75 ms | 134-181 | 134-181 | -| shooter | 7.5 ms | 7.8 ms | 13.35 ms | 13.05 ms | 132-184 | 132-184 | - -Null controls, same code as BEFORE: racing p50 `7.0, 6.9` (rebuild pair) and `6.8, 8.2, 8.1` -(no rebuild). That spread is wider than the racing arm difference, so the arm difference is inside -the lane's own run-to-run noise. - -**The projection declines in every window of every run** — `reasonCode: belowMeshFloor`, "fewer than -200 batchable meshes", 96 and 100 renderables against that 200 floor. On a declining scene the scan -declines, forgets any retained plan and hands the authored scene to the renderer, so neither the -stability skip nor the uniform batching is ever entered. The projection's own whole-run reconcile -total is 1.7-3.9 ms in both arms, and draw calls are identical in all 12 paired runs. - -## Visual - -`compareCaptures` (`packages/runtime-native/conformance/metrics.mjs`) on the after-frame at the same -tick, 1920x1080. Arm A/B: racing 0.671-0.675 mismatch / ΔE 0.446-0.451; shooter 0.765-0.788 / ΔE -0.603-0.626. The same-arm control pairs land in the same band (racing 0.550 and 0.673, shooter 0.769 -and 0.780), so the mismatch is the renderer's own run-to-run temporal post noise, not the change. -Nothing looks different. - -## Verdict - -**No effect on both templates, and no regression.** Not a gain either: on a real game the projection -never engages, so neither change has anything to do. A real-game holdout cannot confirm these two -changes help; it can only say they do nothing here and break nothing. diff --git a/artifacts/engine-load-test/prd-449/holdout/holdout-2026-09-28.json b/artifacts/engine-load-test/prd-449/holdout/holdout-2026-09-28.json deleted file mode 100644 index a8c4cb9126..0000000000 --- a/artifacts/engine-load-test/prd-449/holdout/holdout-2026-09-28.json +++ /dev/null @@ -1,1089 +0,0 @@ -{ - "capturedAt": "2026-09-28T07:57:19.832Z", - "dataset": "PRD-449 real-game holdout: the projection reconcile skip and the uniform material batching, measured on two scaffolded template games", - "lane": { - "adapter": "NVIDIA GeForce RTX 2080 (turing, driver 615.71.09) via Chromium --browser-recipe webgpu; no SwiftShader", - "command": "node packages/playtest/dist/runner/cli.js