fix(sdk): one HTTP loader and one retry policy for every file of a model (#765)

[pasquelin]

Closes #765

What changed

Every scene resource the engine fetches now goes through one loader, checked (packages/sdk-browser/src/cluster/pages.ts). It takes the most requests it makes (attempts, 2 by default), and optionalFile reads a file that may be absent:

• A network failure, a timeout (408), a rate limit (429) or a 5xx is asked once more, after the wait its Retry-After asks (seconds or an HTTP date); any other 4xx never (CTO decision C).
• For an optional file (optionalFile), a 404 or a 403 answers null (CTO decision A). A required read fails on either.
• An aborted signal rejects with its reason and makes no second request.
• Every refused body is cancelled.
• A caller that retries on its own asks for ONE_REQUEST (1).

Two helpers keep that rule in one place: refusedStatus(error) and retriableError(error). The GPU page cache, the page streamer and the physics tiles use them.

credentials is not an option. Its only use, same-origin for images, is already fetch's default, so the issue's "gains credentials" has no user and the field was removed.

Behaviour changes, by call site:

• gpu/page/pages.ts httpPageSource
  • A failed read now raises RESOURCE_HTTP_ERROR with details.url and details.status, instead of Error('PAGE_HTTP_<status>').
  • It makes one request per read.
  • The cache's diagnostics take the status from details.status. A host PageSource throwing the old undocumented PAGE_HTTP_<n> now reports status: null.
  • The cache (gpu/page/load.ts) no longer retries a 4xx: a missing page is asked once instead of twice. A 5xx or network failure is still retried once.
• streaming/fetch.ts, the page streamer (decision B)
  • It already read through checked with one request per attempt.
  • It now stops at a 4xx: one attempt instead of three, and the sticky failure reads after one attempt.
  • A 5xx, a network failure or a corrupt page still gets its three attempts.
• host/prepared/images.ts
  • A 5xx or network failure is now retried once; before, there was no retry.
  • The logged error is RESOURCE_HTTP_ERROR instead of a plain Error.
  • A failed image is still no image, and still logged.
• lighting/importedLights.ts
  • A 404 or 403 still means no lights.
  • Any other refusal (401, a 5xx after its retry, a network failure) now fails the load with RESOURCE_HTTP_ERROR; before, it quietly meant no lights.
  • An abort, including during the body read, now rejects; before, it was swallowed.
  • An unreadable body or an unknown version still means no lights.
• physics/tilePlace.ts cookedPhysics, the physics.json read
  • A 404 or 403 still means the model collides nowhere.
  • Any other refusal is now RESOURCE_HTTP_ERROR on world.physics.error. Before, a 401 or 500 was silent and a network failure raised a raw TypeError.
  • A 5xx is now retried once.
• Physics tiles (cookedBytes, ONE_REQUEST)
  • A failure is RESOURCE_HTTP_ERROR naming the address, not PHYSICS_FAILED.
  • A 5xx or network failure is asked again at the next update, one request per update, as before.
  • A 4xx is reported once, and the tile then leaves its opening until its model opens again, like a tile the worker refused. Before, it was asked and reported every frame.
  • Each model opening owns an AbortController. When the model leaves, the reads of its physics.json and tiles still in flight are aborted, and that is not reported as a failure. Before, they downloaded in full and the result was dropped.
  • The stale-opening checks now test that signal instead of comparing identities.
• Soft-body settings (physics/cookedSoft.ts)
  • A failure is RESOURCE_HTTP_ERROR, retried once after a 5xx, instead of PHYSICS_FAILED with no retry.
  • The read takes the opening's signal: a model that leaves lets it go, with no failure reported.
  • The settings cache is per opening, since each opening parses its own physics.json, so an aborted read is never reused.
  • A cooked file that lists no soft bodies opens as before (?? []).

Other fetch( calls in packages/ were not moved:

• page/decode/geometryPageWasm.ts:76 and physics/physicsWorker.ts:115 load the engine's own wasm, not scene resources.
• Importing cluster/pages.ts there would pull the decode pool into the physics worker, and for geometryPageWasm it would create an import cycle.

Documentation:

• The policy is written once, in docs/SDK.md under "Files over HTTP". The lights and physics sections and docs/ENGINE.md link to it.
• The RESOURCE_HTTP_ERROR meaning ("asked again first when the failure may pass") is updated in scripts/api-reference/errorCodeMeanings.ts, the generated reference, and all 14 translations.

The tileBody move from the previous round is reverted. tiles.ts stays within 200 lines because its body record is now written with grouped spreads, the fixture style.

Proof

The policy is tested once, in cluster/checked.test.ts:

• 503 then 200 is retried.
• A persisting 503 or a network failure is refused by address after two requests.
• A required 404 or 403 is refused after one request.
• An optional 404 or 403 gives null; an optional 401 is refused.
• An abort makes one request.

Each call site's own file tests only what that site adds:

• gpu/page/httpSource.test.ts: the cache asks a 404 once, and retries a 503 while reporting status: 503.
• streaming/pageRefusal.test.ts: a 404 or 403 page gets one attempt, a sticky failure after one attempt.
• host/prepared/imagesRead.test.ts: a 404 image is null and logged by address; a 503 then 200 image is decoded after two requests.
• lighting/importedLightsRead.test.ts: a 404 or 403 gives no lights; a 503 (asked twice) or a 401 (asked once) fails the read by address; an abort during the body read rejects.
• gpu/page/pages.test.ts: a host PageSource failing with a plain Error (no status) is still asked twice.
• physics/tileFetch.test.ts:
  • A physics.json 404 or 403 collides nowhere.
  • A tile 503 is asked once per update, and the next update brings it in.
  • A tile 404 is asked once over two updates.
  • A file with no softBodies still opens.
  • A model leaving mid-read, for a tile or for physics.json, aborts with no failure.

Fixtures:

• One fetch stub, cluster/answers.fixture.ts (answering, with untilAborted also used by cluster/pages.test.ts).
• One physics file router, modelFiles, shared by stubFetch.
• The image decode stub is split out of serveFiles (host/prepared/decodedImages.fixture.ts).

Gates, all green:

• pnpm run check:changed
• pnpm run test:changed (1620 pass, 0 fail)
• pnpm run validate --group quick, which includes check:i18n
• pnpm run validate --group typescript
• node scripts/check-pr-size.ts (592 hand-written lines added, limit 600)

The branch merges cleanly with origin/develop. Nothing was run in a browser.

Local review before push

• Simplification pass: round 1 used /simplify with 4 agents; round 2 applied the reviewer's four /simplify reports.
  • Fixed:
    • Tiles: one opening object per model; stale checks by signal; plain ifs instead of unlessLeft; ONE_REQUEST for tiles, which stops the doubled request per update; the soft-settings read takes the opening's signal.
    • checked: every refused body let go; transient inlined; content-type read once; credentials removed (default only).
    • One shared refusedStatus.
    • The GPU cache and the page streamer skip a 4xx.
    • Tests: the policy tests merged into checked.test.ts on refusedWith/answering; per-site tests reduced to what each site adds; one abort helper; one physics fetch stub; the decode stub split out; URL_ renamed LIGHTS.
    • The policy text is written once in SDK.md and linked from the other docs.
    • The tileBody move reverted.
  • Skipped: moving checked into a module with no dependencies so the wasm fetches could follow. That is outside Every engine fetch goes through the one HTTP loader: retry, abort and named errors #765.
• Correctness review: /code-review --fix, two rounds.
  • Fixed, round 1: one retriable rule shared by checked and the page cache; the code meaning reworded in every translation.
  • Reverted from round 1: its settings-eviction change, which was not asked for then.
  • Fixed, round 2:
    • softs.open lost its default for a file without softBodies, which would throw a raw TypeError. Now ?? [], with a test.
    • A 4xx tile was asked every frame. It now leaves its opening, with a test.
    • One retriableError for the three retry loops.
    • "Even asked again" was wrong for a 4xx; it now reads "asked again first when the failure may pass", in every language.
  • Skipped:
    • Parsing the old PAGE_HTTP_<n> messages.
    • A 5xx tile asked once per frame with no backoff. That is the stated policy.
    • Making stubFetch restore fetch through t.mock. It is test-only and would touch every physics test.
  • Reviewer round (/code-review --fix, 8 findings):
    • Fixed: SDK.md said every file the engine reads goes through the one loader. It now says every file of a model, because the engine's own wasm modules stay out.
    • Skipped then, the Every engine fetch goes through the one HTTP loader: retry, abort and named errors #765 policy of that round: a 429 or 408 is a 4xx and is not asked again. Decision C has since made both retriable (below).
    • Skipped, as before this change: a 5xx tile asked again at every update, with no backoff.
    • Skipped, diagnostics only: the maxAttempts announced before a 4xx stops the loop; a 503 then a network failure reports status: null; refusedStatus relies on instanceof EngineError.
    • Skipped: an unguarded indexOf in tiles.ts:110, not reachable; the spread writer.add literal kept to fit 200 lines.
• Second reviewer, Simplification pass: /simplify with 4 agents (reuse, simplification, efficiency, altitude).
  • Fixed: checked's two overloads and FetchPolicy replaced by an attempts number and one optionalFile wrapper (a 4xx is one request, so the count is unchanged); cookedBytes takes attempts; the test predicate refusedWith reads the status through refusedStatus.
  • Skipped: tried in the page streamer (not derivable once the loop runs out); the spread writer.add literal (200-line limit); stubFetch restore and a checkedBytes helper (low cost, outside the batch); a retry wrapper for the three loops (they also retry non-HTTP failures); the <img> fallback for a platform without createImageBitmap (unchanged code); a retire helper for two splice lines.
• Second reviewer, Correctness review: /code-review --fix, 7 findings.
  • Fixed: refusedStatus reads a status only from a RESOURCE_HTTP_ERROR (an INVALID_JSON_RESPONSE carrying the 200 it answered no longer counts as a 4xx); world.physics.error lists RESOURCE_HTTP_ERROR, in the source and all 14 translations; httpPageSource keeps the one-line summary its translations give (ENGINE.md holds the detail); tests for a 5xx image, a refused lights.json (503, 401) and a plain PageSource failure, the first two failing on develop.
  • Skipped, left to the lead: a tile failing on the network or with a 5xx is asked again at every update with no backoff, as on develop (the documented "next update" rule).
• Correctness review (decision C round): /code-review --fix over b287d6893..HEAD, 8 findings.
  • Delivered first: 408 and 429 join the one retriable rule; checked waits the Retry-After (seconds or an HTTP date) before its next request, abortable. checked.test.ts tests a 408 then a 200, a 429 then a 200, and that Retry-After is waited on, both forms, under t.mock.timers with no real delay. docs/SDK.md updated; the error-code meaning ("asked again first when the failure may pass") still holds, so no translation changed. The call-site tests were trimmed to stay under the size gate.
  • Fixed: the wait's abort listener is now removed once the timer fires; cookedBytes leaves the default request count to checked again; SDK.md no longer claims that the readers with their own retry (page streamer, GPU page cache, physics tiles) wait the Retry-After.
  • Skipped, each would add lines past the gate or change the decided policy:
    • Those readers retry a 408/429 without waiting its Retry-After: the header is not carried in the error. Left to the CTO.
    • No cap on the wait.
    • No backoff when there is no header.
    • Clock skew on the HTTP-date form, measured against the client's clock.
    • A shared fake device in httpSource.test.ts.
    • No test of an abort during the wait.
• Second reviewer, short re-review of the 408/429 and Retry-After round: the wait lets go of its listener, its tests use mock timers and fail without it, docs/SDK.md is true. Fixed: a read already aborted before its wait now rejects at once, and a test aborts a 3600 s wait (it fails in 1 s when the wait ignores the abort). Left to the CTO: Retry-After is uncapped, so a load with no signal waits as long as the server asks; no engine constant bounds it, and a new cap would take the diff over 600 lines.
• Auditor list: every To-do delivered, credentials left out by the path note; decisions A (403 absent for optional reads) and B (streaming stops at a 4xx) each tested; tests for every call site; codes documented in SDK.md, ENGINE.md, the API reference and all 14 translations; no caller matches PAGE_HTTP_; the PHYSICS_FAILED codes left are for other failures; the two wasm fetches rightly left out; Closes #765.

Not proven / left out

• Nothing was run in a browser; only the fast gates.
• A 5xx tile is asked once per update with no backoff, as before this change.
• Retry-After is honoured only by checked's own second request. The page streamer, the GPU page cache and the physics tiles retry a 408/429 without waiting it. The wait has no cap, and the date form is measured against the client's clock.
• A soft body's settings that fail with a 4xx stay failed for that opening. They are asked again when the model reopens.

Lead verification

• One loader, one policy (checked for the five call sites; the physics tiles take the model's abort signal): delivered in packages/sdk-browser/src/cluster/pages.ts:7-60 (checked(url, signal, attempts), optionalFile, one retriable rule: a network error, a 5xx, a 408 or a 429, the Retry-After waited, abortable), used by gpu/page/pages.ts:184, host/prepared/images.ts:30, lighting/importedLights.ts:35, physics/tilePlace.ts:50,58, and by streaming/fetch.ts:35 (CTO decision B); proved by a cache resource the server refuses once (503) is asked again and the cache opens, a resource still failing after its second request is refused by its address, an abort ends a read, even in its Retry-After wait, a timeout (408) or a rate limit (429) is asked again and answers, a refusal asking to wait (Retry-After, seconds or a date) is asked again once waited, and per site a page the server does not hold (404) is refused by its address, the cache asking it once, a page read a busy server refuses once (503) is asked again by the cache, its status reported, an image a busy server refuses once (503) is asked again and decoded, a lights read aborted while its body arrives rejects, never answering no lights, a tile a busy server refuses (503) is asked once per update: the next one brings it in, a model leaving while its physics.json or a tile is on its way lets the read go, no failure, a page the server refuses (404, 403) is asked once, never the three attempts of a 5xx.
• optional: delivered as optionalFile (a 404 or, per CTO decision A, a 403 answers null; a required read fails on either); proved by an optional file the server lacks (404) or hides (403) answers null, asked once, an optional file refused otherwise (401) is refused by its address, asked once, a cache without its lights — a 404, or the 403 of a store that hides it — has none and a model compiled before the cook — a 404, or the 403 of a store that hides it — collides nowhere.
• credentials: not delivered, by the path note on the issue (its only use was fetch's own default, so no user).
• The changed error codes documented (PAGE_HTTP_<n>, PHYSICS_FAILED for a file → RESOURCE_HTTP_ERROR): delivered in docs/SDK.md, docs/ENGINE.md, the API reference and its 14 translations (world.physics.error included); proved by check:i18n and the reference tests; no caller left on the old codes.
• Whole promise: every To-do delivered or left out by the recorded path note; three CTO decisions (403 absent for optional reads, streaming stops at a 4xx, 408/429 retriable with Retry-After) recorded on the issue.
• Tests that bite: the 503 image and the 503/401 lights tests fail on develop; the Retry-After tests fail without the wait (mock timers, no real delay).
• No image loss: no drawn path touched; a model's files read with one policy.
• Reuse: one loader and one retriable rule where there were six fetch policies; the engine's own wasm fetches stay out (not a model's file), stated in the body.
• Docs: docs/SDK.md, docs/ENGINE.md, API reference and translations.
• Path: in review set; to measure after the merge (CTO rule P15: no branch proof for this 🟠, no image or budget change). Two reviewers said OK (the CTO's second-reviewer rule). Streaming without holes: rules and objectives for geometry, memory and shadows #483 checklist and CONTRIBUTING.md §Streaming, memory and shadows: no new per-frame work; a 4xx tile is no longer asked every frame; one declared limit: Retry-After is not capped (for the CTO).