CI caches stop missing on test-native and Android, and the GI harness takes Unreal's diffuse energy term (PRD-628, PRD-646) by jonit-dev · Pull Request #609 · ThreeNativeHQ/threenative-engine

jonit-dev · 2026-11-04T03:58:15Z

What is this PR about?

The follow-up to #605. It carries two pieces of lane work that finished after #605 entered the merge queue:

PRD-628 (every CI cache hits): the two cache misses that round 4 measured.
PRD-646 (ThreeNative recreates the GI calibration scenes): Unreal's diffuse energy term, ported into the calibration harness.

Why does it matter?

test-native restored its compiler cache 0.46% to 55.77% of the time, and the three Android download caches (about 790 MiB) were deleted between runs. Every run paid for those misses again.
The calibration harness took three.js's diffuse energy loss where Unreal takes its own. That made sunlit-room miss 4 of its 40 rows.

Done

✅ test-native ccache restore prefix stops at the lane. It no longer carries a hash of the CMake files, so a cache saved from another tree restores again. ccache keys every object by compiler, flags and source, so a wider prefix cannot serve a wrong object. Red-green: ci-cache-keys.spec.ts (3 cases).
✅ The cache janitor keeps the newest copy of each lane, not one copy per prefix. native-ccache- covers seven lanes, and before this change six of them were evictable. The Gradle, cargo-android and native-android-third-party prefixes are now hot. Red-green: ci-cache-prune.spec.ts (3 new cases).
✅ One native-android-third-party key had a space inside it, caused by a YAML >- fold. It is now one line.
✅ Harness: Unreal's diffuse energy loss (1 - E, the integrated GGX albedo, from UE 5.8.3: Engine/Shaders/Private/ShadingEnergyConservationTemplate.ush:126-132). It is the harness default at L1 for the gi: scenes, and ?diffuseEnergy=three brings back three's term. This PR changes no file under packages/ and no template.
- sunlit-room passes 36 → 40 of 40 L1 rows, and cornell passes 2 → 4 of 12. Each count is the same on 2 runs per arm.
- A fresh read-only Opus visual judge returned IMPROVEMENT on all 8 L1 frames, with no regression, red and green wall hue within 0.6° of Unreal, and run 2 identical to run 1.
- The sunlit-room Phase 2 box is ticked.
✅ Round 10 measured, then ruled out, cornell's lamp radius as the cause of the remaining gap. The gap is the direct specular lobe's share. No port follows, because the rules forbid a fitted constant.

Still open

⚠️ PRD-628 boxes stay unticked. Their proof is two CI runs after this merges: a test-native ccache hit of 80% or more, and the cache table.
⚠️ PRD-646 AC-2 needs cornell 12 of 12. The next measurement reads the level's Specular input from Unreal (round 11, paused while the owner uses the machine).

Checks

check:docs passes, and unreal:status is up to date.
Every workflow spec plus scripts/__tests__/unreal-look: 1,636 of 1,637 pass. The 1 failure is template.spec.ts starter build, which needs a Wasm build that this checkout does not have.

…e ccache restore prefix (PRD-628) Round 4 measured two misses and both have a cause in this tree. `test-native` reported `Hits: 229 / 1076 (21.28%)` in run 41688813459 while three sibling lanes reported 100% on the same board. Two reasons: - `restore-keys` repeated the key's own `hashFiles()` over `CMakeLists.txt` and every `*.cmake`, so a native change produced a prefix no saved entry matched (this tree hashed to `…-gcc-ccb139…`, the develop entry was `…-gcc-3ea15707…`). ccache keys each object by compiler, flags and source, so an object restored from another tree is identical or a miss; the prefix now stops at the lane. - the janitor protected the newest copy of the whole `native-ccache-` prefix per ref, so a ref that runs all seven ccache lanes kept exactly one of them and the budget deleted the rest. On the live list of 2026-11-03 it planned to delete `native-ccache-ci-test-native-…-ccb139…` (19.6 MiB) and `native-ccache-ci-native- engine-…` on develop (148.3 MiB) while keeping a third lane's copy. Protection is now per lane per ref, and an older copy of one lane stays evictable. The same rule protects the Android lane's three downloads (`gradle-` 499.6 MiB, `cargo-android-` 153.6 MiB, `native-android-third-party-` 139.5 MiB) and the 2.5 MiB web-wasm payload: the develop push 41703997856 restored none of the three and re-saved all three. Its `native-android-third-party` key also carried a space (`…-Linux-X64- 4c553d1e…`) from a folded `>-` block; it is one line now. Plan on the live list (87 entries, 9.43 GiB): 8 deletions keeping one lane per ref -> 11 deletions, all of them older copies of a lane another key replaced, 8.93 GiB. (cherry picked from commit 5b8113f05cef63444cb22e1e4a1becb8e0777fb3)

…es it touches (PRD-628) No box is ticked. The proof for all three is a CI run this branch cannot produce: two runs after merge. Each box now names `5b8113f05` and what changed there, plus the numbers the janitor plan produces on the live cache list of 2026-11-03 (9.43 GiB -> 8.93 GiB, 11 deletions, all of them older copies of a lane another key replaced). The Phase 2 "every cacheable input" box also records that the Android triad's save side already runs on develop (run 41703997856, job 126163329535), so the row's remaining miss is GitHub's cache scope for `workflow_dispatch` on a non-default branch and for `merge_group`, not a workflow rule. (cherry picked from commit 686de7b8baed89ebf6bfcb1bc43293fd35fb69f9)

…tree (PRD-628) `gradle-`, `cargo-android-`, `native-android-third-party-` and `native-engine-web-wasm-v2-` are hot keys now, so each has to be a literal head of a `key:` or `prefix=` that `.github/` actually writes. The existing case in `ci-cache-prune.spec.ts` checked three prefixes; a prefix with no declaration would protect nothing and read as protection. (cherry picked from commit 4ca56e2a373a42d988b72cee05dd6aaf36bb372a)

…d sunlit-room passes every L1 row Unreal multiplies a lit slab's diffuse by 1 - Luminance(E), the specular lobe's integrated directional albedo (ShadingEnergyConservationTemplate.ush:126-132, :55-62; applied at SubstrateEvaluation.ush:587-589, :836-837). three subtracts F_Schlick at the light's half angle instead (standard.cpp:1433-1442), which is 4 to 7 times as large at a grazing half angle. render/diffuseEnergy.ts carries Unreal's term on the material: the albedo scaled by (1 - E) / (1 - F) at the scene's one direct light, exact at L1. On by default; ?diffuseEnergy=three is the untouched BEFORE arm. Harness only: no packages/ file and no template changes. Measured, two runs per arm, 8 cells, engine threenative, adapter nvidia turing: cornell 2 of 12 rows pass before and 4 of 12 after, sunlit-room 36 of 40 then 40 of 40 (22 of 22 lit rows). (cherry picked from commit dc1da708712db68c16736ea00378c3168f236302)

…ows, judge IMPROVEMENT (PRD-646) (cherry picked from commit 3d9c8536dea96f7cc255f19289b1e6d9b7d07aa4)

…he specular lobe (PRD-646) (cherry picked from commit 85e17d46b9e0f6530dc4923d86c347edd4df134c)

github-actions · 2026-11-04T04:37:08Z

The merge queue stopped on test-unit (2/4) (failure).

First failing job: test-unit (2/4)
Merge-group run: 41771992038 (the rest of its jobs were cancelled)

Fix, push, re-enqueue.

…to be sampled Merge-group run 41771992038 failed 'fails a soak whose host stopped short of the requested frames' with 0 samples. The fake host wrote its two windows and exited at once; a soak reads memory from /proc/<pid>/status of the live host, and an exited child's entry is gone (ENOENT, checked with a spawned node child), so a runner that read the output late stopped with no sample. The host now stays up 3 s after its last window, then exits on its own as before. perf.spec.ts: 92 of 92 pass.

…PR sat in the queue)

github-actions · 2026-11-04T07:27:59Z

The merge queue stopped on template-nonvisual (sailing) (failure).

First failing job: template-nonvisual (sailing)
Merge-group run: 41789694036 (the rest of its jobs were cancelled)

Fix, push, re-enqueue.

github-actions · 2026-11-04T08:48:54Z

The merge queue stopped on test-native (failure).

First failing job: test-native
Merge-group run: 41794836052 (the rest of its jobs were cancelled)

Fix, push, re-enqueue.

…lowup # Conflicts: # README.md # docs/PRDs/Unreal/research/port-history.json # docs/PRDs/Unreal/research/unreal-port-progress.svg # docs/PRDs/Unreal/research/unreal-visuals-and-performance-feature-map.html # scripts/unreal-look/scaffold/Reference.ts.in

github-actions · 2026-11-04T10:09:31Z

The merge queue stopped on test-native (failure).

First failing job: test-native
Merge-group run: 41809404380 (the rest of its jobs were cancelled)

Fix, push, re-enqueue.

…s lane prefix (PRD-628) Merge-group run 41809404380 failed test-native on native-platform-workflow.test.mjs:73, which pinned a CMake hash in every native-ccache restore key. PRD-628 widened only test-native's restore prefix to its lane; the other lanes keep the hash, and the test now pins test-native's exact lane-only prefix.

github-actions · 2026-11-04T10:42:12Z

The merge queue stopped on test-native (failure).

First failing job: test-native
Merge-group run: 41813763183 (the rest of its jobs were cancelled)

Fix, push, re-enqueue.

…e the PR sat in the queue)

github-actions · 2026-11-04T12:11:04Z

The merge queue stopped on native-platforms / Android emulator visual parity (failure).

First failing job: native-platforms / Android emulator visual parity
Merge-group run: 41821401230 (the rest of its jobs were cancelled)

Fix, push, re-enqueue.

jonit-dev added 6 commits November 3, 2026 20:56

docs(prd): PRD-646 ticks the sunlit-room direct-light box: 40 of 40 r…

b91f251

…ows, judge IMPROVEMENT (PRD-646) (cherry picked from commit 3d9c8536dea96f7cc255f19289b1e6d9b7d07aa4)

docs(prd): PRD-646 measures cornell's lamp out of the gap and names t…

86ec34e

…he specular lobe (PRD-646) (cherry picked from commit 85e17d46b9e0f6530dc4923d86c347edd4df134c)

jonit-dev added the prd:25% PRD phase 1 verified label Nov 4, 2026

jonit-dev enabled auto-merge November 4, 2026 03:58