Streaming without holes 11/12: one frame budget, the partition reach follows zoom, scale and shear, rows grow in place (#404)

[pasquelin]

Closes #404

For the lead before opening: this branch delivers audit items 3, 4 and 2, the inCellFrame shear, and item 1 on every engine that follows the growth contract. It does not deliver the getObjectByName name buckets. It does not deliver item 1 on the WebGPU engine, which has no growPlacements. Nor does it deliver the other items carried in #404's comments (listed under Not proven). By AGENTS.md rule 5, Closes #404 holds until the CTO moves or splits them. The branch is 598 hand-written lines, so the name buckets (estimated at 400–500 more lines) could not fit.

What changed

• Dead code (audit item 3). The arrival queue no longer opens a budget itself: if (!opened) open() and its opened flag are gone (page/integration/arrivalQueue.ts). The queue takes the frame's budget as a required argument, and it only spends from it.
• One frame budget (audit item 4, Streaming without holes: rules and objectives for geometry, memory and shadows #483 rule 4). The session creates one integration budget per frame (world/session/prepare.ts, createFrameBudget(ARRIVAL_BUDGET_MS)), and hands it to three places:
  • the engines, as BackendContext.frameBudget;
  • the arrival queue;
  • the frame (world/render/render.ts), which opens it, lets the cells and the drain spend it, then pauses it.
    The WebGPU row records (webgpu/row/claims.ts) resume the budget, spend what is left, and pause it again. CLAIM's own clock (createWebgpuRowClaims().budget) is deleted. The budget counts integration time only: the engine's other work between the drain and the rows spends none of it (FrameBudget.pause/resume). With no session budget (an engine driven alone, or a barrier image), every owed row is written.
• The reach follows the zoom (audit item 2, Streaming without holes: rules and objectives for geometry, memory and shadows #483 rule 1).
  • For a perspective camera, cellReach reads the zoom through perspectiveSlope(fov, zoom) (scene/partition/plan.ts).
  • For an orthographic camera, the reach reaches the far corner of its zoomed box (orthographicView), and a negative near plane counts too. Before, it was infinite: every cell was read.
  • PartitionOptics is now a Pick of the engine's CameraOptics.
• Shear (decided in Streaming without holes 11/12: a large world loads its object table by distance, without crashing the tab #404, 25 Sept.). inCellFrame divides the reach by the root's least singular value, taken from stretchOf in boxes.ts (now exported) rather than its shortest column. A flattened root reads every cell.
• Scale-down reopens nothing (audit item 1, CONTRIBUTING §Streaming rule 10).
  • A reach or a parent's stretch past the rows sized at open now grows the rows in place when the engine has growPlacements (placement/growth.ts, the one growth contract). This goes through sizeRows(..., grown) and the new io.grow in scene/partition/cells.ts, wired in world/scene/partitionFrame.ts.
  • The growth doubles only what outgrew the rows (the reach, or the stretch), so an ongoing zoom or shrink resizes O(log) times.
  • An engine that grows no buffer in place still asks its owner to reopen, once, as before.
  • The touched-row ranges moved to createTouchedRows (rows.ts) to keep cells.ts within 200 lines, and the test helpers moved to cells.fixture.ts.
• docs/FORMAT.md describes the new reach, budget and growth.

Proof

Behaviour tests, on existing fixtures. Each one fails on develop's sources and passes on the branch; this was checked by restoring develop's plan.ts, boxes.ts, claims.ts, arrivalQueue.ts, frameBudget.ts, cells.ts, rows.ts and partitionFrame.ts and running the new tests:

• Scale-down reopens nothing:
  • cells.test.ts: "a parent scaled down grows the rows in place, on an engine that can, and reopens nothing". The rows grow in place, all cells are placed, outgrown is 0. Without growth it still asks once.
  • tests/integration/world-partition-parents.test.ts: "a page that shrinks the parent of placed nodes grows their rows in place: no reopen, no wait". This runs on the compiled synthetic open world (384², district shrunk 4×): renew is 0, waiting is 0, and the held cells grow.
• Zoom 0.5 loads what is in view:
  • partitionFrame.test.ts: "a camera zoomed out reads the cells its wider frustum sees". At zoom 1 nothing is asked; at zoom 0.5 the cell is asked at the visible priority.
  • plan.test.ts: "the reach follows the zoom", which covers perspective, orthographic and a negative near.
• One frame budget:
  • claims.test.ts: "the rows a frame writes spend what its arrivals left of its one integration budget". Arrivals spend the 2 ms, so one row goes through. The engine's 5 ms between drain and rows is not counted.
  • arrivalQueue.test.ts: "a drain spends what its frame left of the budget, and never opens it again".
• Shear: plan.test.ts, "a sheared root reads every cell the world reach holds, by its least singular value", including the flattened root.
• Gates:
  • pnpm run check:changed: 0.
  • pnpm run test:changed: 1418 pass, 0 fail, 1 skipped (waits for Streaming without holes 14/15: geometry written every frame is uploaded in place, never recut into pages #573), on the reviewer's head.
  • pnpm run validate --group quick: 0.
  • pnpm run validate --group typescript: 0.
  • node scripts/check-pr-size.ts: 598 hand-written lines (limit 600), after the reviewer's pass and a merge of develop.

Local review before push

• Simplification pass: (coder) The simplify skill ran with 4 agents.

  • Fixed:
    • world/session/backends.ts went past 200 lines.
    • The arrival queue fronted the budget (open/admits/spend). The session's frameBudget is now opened by render.ts and passed straight to the partition frame, and the queue's hard-coded createFrameBudget(2) default is gone (the budget is required).
    • serveClaims tested the budget twice.
    • In-place growth resized on every frame of an ongoing zoom or shrink. It now keeps geometric headroom.
  • Skipped:
    • The double inversion in inCellFrame: stretchOf is kept for reuse, about 250 flops per frame.
    • The per-frame bind of growPlacements: io is already rebuilt every frame.
    • The duplicate of worldPoses' range tracking in createTouchedRows: sharing it is out of scope.
    • zoom || 1 repeated from writeEngineCamera.

• Correctness review: (coder) The code-review --fix skill found 9 issues.

  • Fixed:
    • check:lines failed on cells.test.ts (230), partitionFrame.test.ts (220) and backend/types.ts (202). Helpers moved to cells.fixture.ts and a shared asks().
    • check:tools-types failed on the zoom/frameBudget literals and on the reference-queue cast.
    • docs/FORMAT.md still described the old rules.
    • Stale "arrival queue's budget" comments.
    • The zoom test never checked that zoom 1 reads nothing.
  • Then fixed by the coder after the review:
    • The WebGPU rows were spending a wall clock that included the engine's non-integration work, which could starve them to one row per frame. pause/resume now count integration time only.
    • Growth doubled both the reach and the stretch whatever outgrew. It now doubles only what outgrew.
    • The orthographic reach ignored a negative near.
  • Left: a capture rendered outside the session frame serves owed rows from what the last frame left of its budget (see Not proven).

• Simplification pass: (reviewer) The simplify skill ran with 4 agents (reuse, simplification, efficiency, altitude).

  • Fixed:
    • residentRows (sizing.ts) now runs inside a frame when rows grow, and it rebuilt each cell's centre and radius for every pair. Each cell's ball is now computed once per group, with plain dx/dy/dz.
    • pause/resume are balanced on every path, throws included: try/finally in serveClaims and around the cells and drain in render.ts.
    • The row slots take the budget per image: slots.apply(budget?) replaces a constructor parameter plus a bounded flag.
    • inCellFrame moves the eye with sdk-core's transformAffinePoint instead of a hand-written transform.
    • The orthographic reach takes the absolute half-size of the box, so a box given right to left or bottom up is as wide (tested).
    • Only world() moved to cells.fixture.ts; the other test helpers stay in cells.test.ts.
    • The arrival reference test takes built queues.
  • Skipped:
    • Merging createTouchedRows with worldPoses' range tracking: that code is older than this branch and out of scope.
    • A bracketed budget.stage(fn): try/finally gives the same balance.
    • The per-frame bind of growPlacements and the second inversion in stretchOf: each costs well under a microsecond per frame.

• Correctness review: (reviewer) The code-review --fix skill found 10 issues and fixed 7.

  • Fixed:
    • A scaled camera, or one under a scaled rig, draws up to far × scale, but the reach ignored that scale. partitionFrame.ts viewOf now multiplies the reach by the camera world matrix's maxStretch (tested at scale 2).
    • A root so nearly flat that its inverse overflows made maxStretch throw on every frame. stretchOf now returns a least stretch of 0 for it, and inCellFrame reads every cell (tested).
    • guides.follow() was charged to the integration budget. It now runs before open.
    • Two gate failures: no-useless-assignment in render.ts, and frameBudget required in backends.ts Inputs while a fixture omits it.
    • The render wiring had no test. triangles.test.ts now checks the order follow, open, drain, pause, draw, and that pause still runs when the drain throws.
    • claims.test.ts busy-waited on the wall clock. It now uses a clock the test moves.
    • The arrival bench compared a 2 ms budget with an oracle that has no time limit. Both now run with no limit.
  • Left for the lead:
    • Growth reaches the active engine only, like updatePlacements (see Not proven).
    • A capture drawn at another aspect than the camera's is not covered by the reach.
    • residentRows still checks every pair of cells under a parent when rows grow, a few times per zoom or shrink. The cost is for the measurer.
  • Acceptance list (reviewer):
    • The new behaviour tests fail on develop's sources and pass on the branch. Checked by restoring develop's plan.ts, boxes.ts, cells.ts, rows.ts, sizing.ts, claims.ts, arrivalQueue.ts, frameBudget.ts, partitionFrame.ts and render.ts.
    • The scale-down test on the compiled synthetic world ran (the compiler was present).
    • The tests wait on events or on a clock they drive; none uses a fixed delay.
    • docs/FORMAT.md follows the change. No format version changes.

#483 checklist

• Rule 1 (no hole): the reach now covers the zoomed frustum and a negative orthographic near. A scale-down places every cell within reach in the same frame on an engine that grows in place. On WebGPU a cell past the rows still waits until the reopen (see Not proven).
• Rule 2 (no image loss): no change to the cut. More WebGPU rows may be deferred to later frames when integration fills the 2 ms. Their pages stay non-resident and their resident parent covers them. Not measured here.
• Rule 3 (compiler first): not touched. The cook does not yet refuse a partition that breaks an invariant (carried, see Not proven).
• Rule 4 (one mechanism): one integration budget per frame across cells, arrivals and WebGPU row records. CLAIM's own clock is deleted in this PR. Texture tiles keep the host's maxTextureUploadMsPerFrame upload ceiling, which is a separate setting.
• Rule 5 (one memory budget): not touched. Grown rows are not counted in the global memory budget (carried).
• Rule 6 (bounded by the view):
  • The orthographic reach is now bounded by its box and far plane, where before it read every cell.
  • Growth follows the reach and the parents' stretch, not the world.
• Rule 7 (main thread): row growth runs residentRows on the main thread, O(log) times for an ongoing change. The per-frame cost is for the measurer.
• Rule 8 (WebGL2): the WebGL2 path (autonomous) grows rows in place, so it holds rule 10 on scale-down. No browser proof here.
• Rule 9 (two scenes): the scale-down is proven on the unit fixture and on the compiled synthetic open world. The browser A/A is the measurer's.
• CONTRIBUTING rule 10 (no reopen for moving content): held on engines with growPlacements. Not held on WebGPU (see Not proven).

Not proven / left out

• A flattened root is the one unbounded case. A root whose world matrix is degenerate (determinant 0, or so nearly flat its inverse overflows) has no finite reach in its frame. It reads every cell and grows the rows to every placement. That is correct, since the whole flattened world may be in view, but it is world-sized.

• Engine side of a grow. cells.test.ts checks that the engine is told of the grown buffers only (from → to, and every later update names a grown buffer). The develop test backend/autonomous/pages.test.ts ("a full instance buffer grows in place…") shows the WebGL2 engine reading the grown buffer after growPlacements ("the kept row reads the grown buffer"). No single test drives a partition into the real engine.

• WebGPU rows can be starved by a burst. When cells and arrivals spend the whole 2 ms, the WebGPU row records get one row per frame until the burst passes; before, they had their own 2 ms. Nothing is lost from the image: an unranked page is covered by its resident parent. The latency is for the measurer.

• Item 1 on WebGPU. The WebGPU engine implements no growPlacements. On it, a parent scaled down or stretched unevenly, or a reach past the rows, still asks the owner to reopen, once. Growing the WebGPU tables in place is the "extend growPlacements" of Streaming without holes 13/15: no content change reopens a session, and a reopen never shows an empty frame #572. Estimate: more than 400 lines, with test:gpu proof.

• getObjectByName name buckets (decided in Streaming without holes 11/12: a large world loads its object table by distance, without crashing the tab #404, 25 Sept.): not delivered. It needs three parts:

  • name-hash buckets written by the Rust compiler under the stream unit;
  • a format version bump with regenerated caches;
  • a runtime lookup that loads the cell and makes a host node, with sync/async forms and the 1×/16× bytes test.
    Estimate: 400–500 hand-written lines. It does not fit this PR (598 lines).

• Other items carried in Streaming without holes 11/12: a large world loads its object table by distance, without crashing the tab #404's comments, none moved to The compiler writes the manifest in pages a reader fetches by region #750/The runtime holds only the manifest pages its view needs, and the open world loads #751 by the split (those took only the paged manifest and the open-world crash):

  • the cook refusing a partition that breaks an invariant (Streaming without holes: rules and objectives for geometry, memory and shadows #483 rule 3);
  • rows counted in the one memory budget, with a "one level coarser" path (Streaming without holes: rules and objectives for geometry, memory and shadows #483 rule 5);
  • a partitioned scene on the WebGL2 path in a browser (Streaming without holes: rules and objectives for geometry, memory and shadows #483 rule 8, measurer);
  • the two-scene A/A still image (Streaming without holes: rules and objectives for geometry, memory and shadows #483 rule 9, measurer);
  • bare-explorer rows sized for the whole world (needs growth on every engine, like item 1);
  • placements under an animated parent staying in the core (rule 10 wants the dynamic path);
  • cells costing over 2 ms starving page drains (rule 7, measurer).

• Engines driven outside a session. An engine driven with no session frame budget (fixtures, tests/browser/support mounts, bench harnesses) writes every owed WebGPU row per image, unbounded. A capture rendered outside the session frame (renderForCapture) serves owed rows from what the last frame left of the budget.

• Comparison mode. Row growth and row updates reach the active engine only, the same shape as updatePlacements and sceneApi.growPlacements.

• For the measurer: main-thread ms per frame and row-record latency on the open world (WebGPU rows now share the 2 ms with cells and arrivals), and zoom-out and scale-down on the open world with no reopen on WebGL2.

Lead verification

Read by the geometry lead on head 9165dd3dd against #404 after the CTO's two splits (what moved is listed in #404's body: #750, #751, #572, #762, #745). The measurer proved it before this pull request opened (CTO rule of 26 Sept.).

• Audit, dead code: delivered in packages/sdk-browser/src/page/integration/arrivalQueue.ts (if (!opened) open() and the opened flag removed; the queue spends the session budget it is given). Proved by arrivalQueue.test.ts "a drain spends what its frame left of the budget, and never opens it again".
• Audit, one frame budget (rule 4): delivered by deleting CLAIM_BUDGET_MS (webgpu/row/claims.ts:71-91, which spends the session budget, balanced in try/finally) and by one budget per session shared by the arrival queue, the partition frame, render (world/render/render.ts:111) and the row claims. Proved by claims.test.ts "the rows a frame writes spend what its arrivals left of its one integration budget" and triangles.test.ts "each frame moves the followed guides once, and integrates within its budget, before it draws".
• Audit, cellReach and the zoom (rule 1): delivered in scene/partition/plan.ts:58 (perspectiveSlope(fov, zoom), the zoomed orthographic box, absolute, with a negative near plane) and world/scene/partitionFrame.ts:27 (times the camera's maxStretch). Proved by plan.test.ts "the reach follows the zoom…" and partitionFrame.test.ts "a camera zoomed out, or scaled up, reads the cells its wider frustum sees".
• Decided on Streaming without holes 11/12: a large world loads its object table by distance, without crashing the tab #404, the inCellFrame shear: delivered in plan.ts:79 (the least singular value, stretchOf; a nearly flat root reads every cell instead of throwing). Proved by plan.test.ts "a sheared root reads every cell the world reach holds, by its least singular value".
• Audit, reopen on scale-down (rule 10): delivered in scene/partition/cells.ts:149 (rows grown in place through io.grow, only what outgrew, doubled), on engines with growPlacements. WebGPU moved to Streaming without holes 13/15: no content change reopens a session, and a reopen never shows an empty frame #572. Proved by cells.test.ts "a parent scaled down grows the rows in place, on an engine that can, and reopens nothing" and tests/integration/world-partition-parents.test.ts on the compiled 384² district world.
• The name buckets: moved to Mesh pages carry their own tables and clusters.json stays the same size whatever the world #762 by the CTO.
• Proof, gates: check:changed 0, test:changed 1418 pass, 0 fail; validate quick and typescript 0 (reviewer, 9165dd3dd).
• Proof, measurer (against 5be70a2f7):
  • 0 px still and after an orbit, A/A 0, on observatory-streamed at full budget and memory-on-a-budget at 256 KiB;
  • CPU frame p50/p95 1.5/2.4 against 1.5/2.3, cut stages unchanged;
  • test:gpu 60/66, the 6 failures develop's known ones.
    The zoom-out and scale-down scenes cannot be driven in the browser until The runtime holds only the manifest pages its view needs, and the open world loads #751 (the compiled district world draws no cell in the harness on either side); the integration tests carry them. Row catch-up after a burst has no metric yet (null), which is declared.
• Before merge 1, whole promise: every item left on Streaming without holes 11/12: a large world loads its object table by distance, without crashing the tab #404 after the splits is delivered.
• Before merge 2, tests that bite: every new behaviour test fails with develop's sources restored (reviewer). Injected clocks, events, no fixed delay. The district-world test runs on the compiled world.
• Before merge 3, no image loss: 0 px, still and moved, on two scenes. The reach only widens, never narrows, so nothing in view is dropped.
• Before merge 4, reuse: one createFrameBudget per session (with pause/resume) replaces the claims' own clock. stretchOf, maxStretch, perspectiveSlope and transformAffinePoint are the existing ones. Rows grow through the existing growPlacements contract.
• Before merge 5, docs: docs/FORMAT.md follows. No public member changed. The body closes Streaming without holes 11/12: a large world loads its object table by distance, without crashing the tab #404.
• Before merge 6, measured first: not an optimisation. The CPU frame is re-measured unchanged.
• Before merge 7, path: the two splits are written on Streaming without holes 11/12: a large world loads its object table by distance, without crashing the tab #404 by the CTO. in review, audit ko and in progress are removed at hand-over; to measure stays.
• Streaming without holes: rules and objectives for geometry, memory and shadows #483 rules / CONTRIBUTING.md §Streaming, memory and shadows: 1 (the reach covers zoom, box, scale and shear); 2 (0 px); 4 (one integration budget); 7 (integration within the frame's budget); 10 (a scale-down reopens nothing where growPlacements exists); 9 (the tests above, and the measurer's proof). Rules 3, 5, 6 and 8 moved (Mesh pages carry their own tables and clusters.json stays the same size whatever the world #762, The runtime holds only the manifest pages its view needs, and the open world loads #751); 11 untouched.
• Size: 598 hand-written lines.
• CI fix after the reviewer's OK (4ecd8a1e8): scene/partition/cells.ts spells FrameBudget and Grow inline in the public frame signature, so both stay internal (public-types-audit.test.ts:115). It is two type lines with no behaviour change, re-read by the lead as the short re-review; validate --group native 3329 pass, 0 fail (coder).