Add WebGPU wedge watchdog with one-way WASM fallback - #35
Merged
Merged
Conversation
Live smoke of v1.3.1 (M3 Pro under GPU contention): a WebGPU session.run stalled mid-page on Wikipedia and Amazon search, the offscreen document failed its 30 second init health check, and 104 badges stayed pending with no recovery path. src/webgpu-watchdog.js (new, pure, node:test-covered): - MODEL_RUN_WEDGE_TIMEOUT_MS 20s per single session.run, well under the 45s inference budget so the wedge verdict plus WASM retry fit inside it - isWedgeError / isWebGpuDeviceError / shouldFallBackToWasm decision logic, watchModelRun race, wrapSessionWithWatchdog session wrapper, createWebGpuHealth one-way latch src/offscreen.js: - WebGPU CF and DINO sessions are wrapped with the watchdog - GPUDevice.lost marks the health latch - On wedge or device loss: dispose both sessions, rebuild both on WASM, log the switch once, rescore the in-flight image, and let everything behind the inference lock and in the page queue run on WASM - One-way for the browser session; no flap back until the offscreen document restarts - An error after the WASM retry propagates so the badge gets its normal error state instead of staying pending tests/webgpu-watchdog.test.mjs: 20 new tests. Full suite 248 passing (was 228). No fusion, threshold, model, or probe changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Defect
Live smoke of v1.3.1 on an M3 Pro under GPU contention: the offscreen document's WebGPU inference stalled mid-page (Wikipedia article, Amazon search), the offscreen init health check failed after 30 seconds, and 104 badges stayed pending with no recovery. AGENTS.md covers init ("probe the adapter; do not latch a WebGPU error") but there was no runtime recovery once a WebGPU session.run hung or the device was lost: the exclusive inference lock stayed occupied by the wedged run and the whole queue starved silently.
Fix
New pure module
src/webgpu-watchdog.js, wired intosrc/offscreen.js:session.runon a WebGPU session (CF and DINO, viawrapSessionWithWatchdog) racesMODEL_RUN_WEDGE_TIMEOUT_MS(20 s, JSDoc-documented: generous vs sub-5 s normal runs, well under the 45 sINFERENCE_TIMEOUT_MSso the wedge verdict plus WASM rebuild plus retry fit in one inference budget). A run past the timer is treated as a wedged backend.GPUDevice.lost(viaort.env.webgpu.device) marks a one-way health latch; device-level errors thrown by a run (isWebGpuDeviceError) count the same way. Fetch, decode, and size-cap errors deliberately never match.console.warn "Clueside: WebGPU backend unhealthy (...)"), the in-flight image is rescored on WASM, and everything waiting behind the inference lock or in the page queue runs on the rebuilt WASM sessions. The latch is one-way for the browser session: no flap back to WebGPU until the offscreen document restarts.Not touched
Fusion policy, thresholds, bands, models, probe, benchmark numbers.
Evidence
npm test: 248 passing, 0 failing (was 228; 20 new tests intests/webgpu-watchdog.test.mjscovering the wedge timer, error classification, fallback decision, one-way latch, and offscreen wiring).esbuild src/offscreen.js --bundlecompiles clean (695 kb, same entry point the build uses).