From 401157e2fc87cb11fefd90bef3c8a10dd7373b92 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Wed, 14 Oct 2026 16:37:22 +0200 Subject: [PATCH] feat(effects): check an effect's code off the main thread, and price the tiers on an M1 A Metal compile takes 120 ms for a new source on an M1, up to 375 ms for the first in a process (#1206), against 10 ms for FXC. checkShaderEffect ran it on the main process's thread. - The addon's checkShaderEffect is an AsyncTask on the libuv pool, like segmentFrame. The service and the checker type return a promise; the service awaits it inside its try, so a rejecting addon still reads as "not checked", as a throwing one did. - runDocumentTool asks first, for the code addEffect or setEffectDefinition will store (shaderEffectToCheck), then hands the answer to the executor, which stays synchronous, as the cursor read and the auto-zoom plan already do. Edits off: nothing is asked. - addEffect's description and the MCP contract resource give each tier in ms on the RTX 4070 Ti and on the M1, as the spec's C.7 asks. --- crates/compositor-view-napi/src/lib.rs | 45 +++++--- docs/shader-effects.md | 8 +- electron/ai-edition/agent-tools.test.ts | 36 +++++++ electron/ai-edition/agent-tools.ts | 58 +++++++++- .../ai-edition/deep-agent/service.test.ts | 101 ++++++++++++++++++ electron/ai-edition/deep-agent/service.ts | 17 ++- electron/mcp/effects-contract.ts | 17 +-- .../services/compositorViewService.test.ts | 28 +++++ .../services/compositorViewService.ts | 8 +- electron/native/compositor-view/addon.d.ts | 7 +- src/lib/ai-edition/shaderEffects.ts | 11 +- .../engineering/rendering-performance.md | 2 +- 12 files changed, 300 insertions(+), 38 deletions(-) diff --git a/crates/compositor-view-napi/src/lib.rs b/crates/compositor-view-napi/src/lib.rs index 447df805c..b25b0dfe0 100644 --- a/crates/compositor-view-napi/src/lib.rs +++ b/crates/compositor-view-napi/src/lib.rs @@ -401,20 +401,41 @@ pub struct ShaderEffectCheck { pub iterations: Option, } +pub struct CheckShaderEffectTask { + code: String, + params_json: String, +} + +impl Task for CheckShaderEffectTask { + type Output = ShaderEffectCheck; + type JsValue = ShaderEffectCheck; + + fn compute(&mut self) -> Result { + Ok(match openscreen_compositor::shader_fx::check_for_agent_json(&self.code, &self.params_json) { + Ok(report) => ShaderEffectCheck { + error: None, + tier: Some(report.tier.name().to_string()), + frame_reads: Some(report.frame_reads as f64), + iterations: Some(report.iterations as f64), + }, + Err(error) => ShaderEffectCheck { error: Some(error), tier: None, frame_reads: None, iterations: None }, + }) + } + + fn resolve(&mut self, _env: Env, check: Self::Output) -> Result { + Ok(check) + } +} + /// Vérifie le WGSL d'un effet shader : validation, bornes des boucles, compilation réelle pour le -/// backend de la machine quand elle existe (FXC sur Windows), et palier de coût. `params_json` : -/// les paramètres déclarés, `[{ "name": "strength", "type": "number" }, …]`. +/// backend de la machine (FXC sur Windows, Metal sur macOS, SPIR-V sur Linux), et palier de coût. +/// `params_json` : les paramètres déclarés, `[{ "name": "strength", "type": "number" }, …]`. +/// +/// `AsyncTask`, comme `segment_frame` : la compilation Metal d'une source neuve prend 120 ms, et +/// jusqu'à 375 ms pour la première du processus. Ce temps n'a rien à faire sur le thread principal. #[napi] -pub fn check_shader_effect(code: String, params_json: String) -> ShaderEffectCheck { - match openscreen_compositor::shader_fx::check_for_agent_json(&code, ¶ms_json) { - Ok(report) => ShaderEffectCheck { - error: None, - tier: Some(report.tier.name().to_string()), - frame_reads: Some(report.frame_reads as f64), - iterations: Some(report.iterations as f64), - }, - Err(error) => ShaderEffectCheck { error: Some(error), tier: None, frame_reads: None, iterations: None }, - } +pub fn check_shader_effect(code: String, params_json: String) -> AsyncTask { + AsyncTask::new(CheckShaderEffectTask { code, params_json }) } /// Installe la scène de l'app (JSON `SceneDescription`) sur la vue : layout preset piloté par diff --git a/docs/shader-effects.md b/docs/shader-effects.md index bedb0af89..27e1b1cad 100644 --- a/docs/shader-effects.md +++ b/docs/shader-effects.md @@ -416,6 +416,9 @@ Il y a deux compilations, à deux moments : `setEffectDefinition`), le processus principal appelle `checkShaderEffect(code, params)` de l'addon. - **La compilation est réelle,** pour le backend de la machine, en plus de la validation naga. « Accepté » veut donc dire « se dessinera ». + - **Elle tourne sur le pool de libuv** (une `AsyncTask`) : la compilation Metal d'un code neuf + prend 120 ms, et jusqu'à 375 ms pour la première du processus. `runDocumentTool` attend la + réponse, puis la passe à l'exécuteur des outils, qui reste synchrone. - **Le coût est estimé** depuis le code (C.7), sans rendu ni mesure sur la machine. - **L'erreur revient à l'agent** avec la ligne et la colonne de SON code. 2. **À la pose de la scène** (`Compositor::set_scene`). Chaque code nouveau part compiler sur un @@ -743,9 +746,8 @@ module et pipeline compris ; les paliers en C.7. | Vue live contre MP4 (BT.709), image entière | 3,1 / 255 avec effet, 3,0 sans | | Vue live contre MP4, sur les pixels où l'effet agit | 13 à 16 / 255 avec effet, 14 à 16 sans effet sur les mêmes pixels : le bruit du codec sur des bords nets | -La compilation, dix fois plus lente qu'avec FXC, reste hors du rendu. `checkShaderEffect`, lui, -compile sur le thread qui l'appelle : le processus principal y passe ces 120 ms à chaque code -neuf. +La compilation, dix fois plus lente qu'avec FXC, reste hors du rendu. Elle tourne aussi à +l'écriture (`checkShaderEffect`), d'où le pool de libuv plutôt que le thread principal (D.4). Le détail est dans [rendering-performance.md](../technical-documentation/engineering/rendering-performance.md). diff --git a/electron/ai-edition/agent-tools.test.ts b/electron/ai-edition/agent-tools.test.ts index ca4261cf2..764467566 100644 --- a/electron/ai-edition/agent-tools.test.ts +++ b/electron/ai-edition/agent-tools.test.ts @@ -13,6 +13,8 @@ import { executeAgentTool, isMutatingTool, MUTATING_TOOL_NAMES, + shaderEffectCheckLookup, + shaderEffectToCheck, } from "./agent-tools"; import { measureTextInShippedFonts } from "./text-measure"; @@ -3924,3 +3926,37 @@ describe("shader effects", () => { expect(JSON.parse(added.resultJson)).toMatchObject({ validated: false, cost: null }); }); }); + +describe("the executor's view of a check asked beforehand", () => { + const source = { + code: "fn effect(uv: vec2) -> vec4 { return frame(uv); }", + params: [{ name: "level", type: "number" as const }], + }; + const check = { tier: "light" as const, frameReads: 1, iterations: 0 }; + + it("is the answer for the same code and layout, and nothing for any other", () => { + const lookup = shaderEffectCheckLookup(source, check); + expect(lookup(source.code, [{ name: "level", type: "number" }])).toBe(check); + expect(lookup(`${source.code}\n`, source.params)).toBeNull(); + expect(lookup(source.code, [{ name: "level", type: "color" }])).toBeNull(); + expect(lookup(source.code, [])).toBeNull(); + }); + + it("is asked only by the two tools that compile", () => { + const document = fixtureDocument(); + expect( + shaderEffectToCheck(document, "addEffect", { startSec: 0, endSec: 1, effectId: "fx_1" }), + ).toBeNull(); + expect(shaderEffectToCheck(document, "addTrim", { startSec: 0, endSec: 1 })).toBeNull(); + expect( + shaderEffectToCheck(document, "addEffect", { + startSec: 0, + endSec: 1, + effect: { name: "A", description: "", code: source.code }, + }), + ).toEqual({ code: source.code, params: [] }); + expect( + shaderEffectToCheck(document, "setEffectDefinition", { effectId: "fx_missing", code: "x" }), + ).toBeNull(); + }); +}); diff --git a/electron/ai-edition/agent-tools.ts b/electron/ai-edition/agent-tools.ts index 91c18d756..ffa381069 100644 --- a/electron/ai-edition/agent-tools.ts +++ b/electron/ai-edition/agent-tools.ts @@ -61,7 +61,7 @@ import { findShaderEffect, SHADER_EFFECT_CONTRACT, type ShaderEffectCheck, - type ShaderEffectChecker, + type ShaderEffectCheckLookup, shaderEffectCostAdvice, shaderEffectTierOf, storedShaderEffectValues, @@ -1526,7 +1526,7 @@ export function documentSnapshotForModel( * wired, its cost cached. An error reads as the agent must act on it. */ function checkedEffectDefinition( candidate: unknown, - checker: ShaderEffectChecker | undefined, + checker: ShaderEffectCheckLookup | undefined, ): { definition: AxcutShaderEffect; check: ShaderEffectCheck | null } | { error: string } { const parsed = shaderEffectSchema.safeParse(candidate); if (!parsed.success) { @@ -1546,6 +1546,53 @@ function checkedEffectDefinition( return { definition: { ...definition, cost }, check }; } +type ShaderEffectSource = { + code: string; + params: Array<{ name: string; type: AxcutShaderEffect["params"][number]["type"] }>; +}; + +/** The code and parameter layout `addEffect` or `setEffectDefinition` will have the compositor + * check, read from the same arguments the executor reads; `null` for a call that compiles nothing. + * The check is asynchronous (a Metal compile takes 120 ms and more) and the executor is not: the + * async wrapper asks first, then hands the answer over (`shaderEffectCheckLookup`). */ +export function shaderEffectToCheck( + document: AxcutDocument, + name: string, + args: unknown, +): ShaderEffectSource | null { + const layout = ( + params: ReadonlyArray>, + ) => params.map(({ name, type }) => ({ name, type })); + if (name === "addEffect") { + const parsed = addEffectArgs.safeParse(args); + if (!parsed.success || !parsed.data.effect) return null; + return { code: parsed.data.effect.code, params: layout(parsed.data.effect.params ?? []) }; + } + if (name === "setEffectDefinition") { + const parsed = setEffectDefinitionArgs.safeParse(args); + if (!parsed.success) return null; + const { effectId, code, params } = parsed.data; + const existing = findShaderEffect(document, effectId); + if (!existing || (code === undefined && params === undefined)) return null; + return { code: code ?? existing.code, params: layout(params ?? existing.params) }; + } + return null; +} + +/** The executor's view of an answer asked for `source`: that answer for the same code and + * layout, `null` (not checked) for anything else. */ +export function shaderEffectCheckLookup( + source: ShaderEffectSource, + check: ShaderEffectCheck | null, +): ShaderEffectCheckLookup { + return (code, params) => + code === source.code && + params.length === source.params.length && + params.every((p, i) => p.name === source.params[i].name && p.type === source.params[i].type) + ? check + : null; +} + /** Two effects of one lane cannot overlap: the refusal names what is in the way, and the way out. */ function effectLaneBlocked( ranges: AxcutShaderEffectRange[], @@ -1733,9 +1780,10 @@ export interface AgentToolOptions { /** Whether this machine can cut out the camera background, asked only by the two tools that * report it. Absent where nothing can ask the compositor: reported as not checked. */ cameraSegmentation?: () => SegmentationSupport; - /** Asks the compositor to check an effect's WGSL (`checkShaderEffect`). Absent where nothing - * can: the effect is stored unchecked, and the result says `validated: false`. */ - checkShaderEffect?: ShaderEffectChecker; + /** The compositor's check of the WGSL this call writes (`checkShaderEffect`), asked beforehand + * by the async wrapper (`shaderEffectToCheck`). Absent where nothing can: the effect is stored + * unchecked, and the result says `validated: false`. */ + checkShaderEffect?: ShaderEffectCheckLookup; /** `addAutoZooms`' plan, made by the async wrapper from the recorded clicks. Absent means * no cursor reader is wired here, which the tool reports as "could not look". */ autoZooms?: AutoZoomPlan; diff --git a/electron/ai-edition/deep-agent/service.test.ts b/electron/ai-edition/deep-agent/service.test.ts index 22d508fb7..fffbeb4c5 100644 --- a/electron/ai-edition/deep-agent/service.test.ts +++ b/electron/ai-edition/deep-agent/service.test.ts @@ -780,3 +780,104 @@ describe("addAutoZooms reads the clicks through the runtime's cursor reader", () expect(result.resultJson).toContain("could not be read"); }); }); + +describe("an effect's code is checked off the main thread, before the executor reads it", () => { + const LEVEL = { type: "number", name: "level", label: "Level", default: 1, min: 0, max: 2 }; + const effect = (code: string) => ({ + name: "Dim", + description: "Dims the picture.", + code, + params: [LEVEL], + }); + // The compositor's check takes a while (a Metal compile is 120 ms and more): a promise that + // settles later, refusing `nope` and pricing the rest at 20 reads. + const slowChecker = (asked: Array<{ code: string; params: unknown }>) => ({ + checkShaderEffect: async ( + code: string, + params: ReadonlyArray<{ name: string; type: string }>, + ) => { + asked.push({ code, params }); + await new Promise((resolve) => setTimeout(resolve, 5)); + return code.includes("nope") + ? { error: "error: no definition in scope for identifier: `nope`" } + : { tier: "medium" as const, frameReads: 20, iterations: 20 }; + }, + }); + + it("waits for the compositor's answer, accepting or refusing on it", async () => { + const asked: Array<{ code: string; params: unknown }> = []; + const runtime = slowChecker(asked); + const accepted = await runDocumentTool( + fixtureDocument(), + "addEffect", + { + startSec: 1, + endSec: 2, + effect: effect("fn effect(uv: vec2) -> vec4 { return frame(uv); }"), + }, + true, + runtime, + ); + expect(accepted.ok).toBe(true); + const result = JSON.parse(accepted.resultJson); + expect(result.validated).toBe(true); + expect(result.cost).toMatchObject({ tier: "medium", frameReads: 20 }); + expect(asked[0].params).toEqual([{ name: "level", type: "number" }]); + + const refused = await runDocumentTool( + fixtureDocument(), + "addEffect", + { startSec: 1, endSec: 2, effect: effect("fn effect(uv: vec2) -> vec4 { nope }") }, + true, + runtime, + ); + expect(refused.ok).toBe(false); + expect(refused.resultJson).toContain("does not compile"); + }); + + it("checks a new code against the definition's parameters when only the code changes", async () => { + const asked: Array<{ code: string; params: unknown }> = []; + const runtime = slowChecker(asked); + const added = await runDocumentTool( + fixtureDocument(), + "addEffect", + { + startSec: 1, + endSec: 2, + effect: effect("fn effect(uv: vec2) -> vec4 { return frame(uv); }"), + }, + true, + runtime, + ); + const { effectId } = JSON.parse(added.resultJson); + const document = added.document as AxcutDocument; + const code = "fn effect(uv: vec2) -> vec4 { return frame(uv) * params.level; }"; + const changed = await runDocumentTool( + document, + "setEffectDefinition", + { effectId, code }, + true, + runtime, + ); + expect(changed.ok).toBe(true); + expect(asked.at(-1)).toEqual({ code, params: [{ name: "level", type: "number" }] }); + }); + + it("asks nothing when edits are off, or for a tool that compiles nothing", async () => { + const asked: Array<{ code: string; params: unknown }> = []; + const runtime = slowChecker(asked); + await runDocumentTool( + fixtureDocument(), + "addEffect", + { + startSec: 1, + endSec: 2, + effect: effect("fn effect(uv: vec2) -> vec4 { return frame(uv); }"), + }, + false, + runtime, + ); + await runDocumentTool(fixtureDocument(), "addTrim", { startSec: 1, endSec: 2 }, true, runtime); + expect(asked).toEqual([]); + }); +}); diff --git a/electron/ai-edition/deep-agent/service.ts b/electron/ai-edition/deep-agent/service.ts index b3e2f978b..88b7333d0 100644 --- a/electron/ai-edition/deep-agent/service.ts +++ b/electron/ai-edition/deep-agent/service.ts @@ -89,6 +89,8 @@ import { setTrimArgs, setWordTextArgs, setZoomArgs, + shaderEffectCheckLookup, + shaderEffectToCheck, } from "../agent-tools"; import { getCaptionsArgs, @@ -263,7 +265,7 @@ export const TOOL_DESCRIPTIONS: Record = { addCameraFullscreen: "Add a camera-fullscreen region over a span of the edited timeline (virtual seconds): the webcam fills the frame for that span. This only does something when the footage under that span comes from an asset with a linked webcam — check assets[].hasCameraTrack (or hasAnyCamera) in getCurrentDocument first. On footage with no camera the call is refused rather than storing a region that would render nothing; say so instead of retrying.", addEffect: - "Write a visual effect as a shader and pose it over a span of the edited timeline (virtual seconds), or pose one of this project's effects again (effectId). Use it for a look the settings do not offer. `effect.code` is WGSL that defines `fn effect(uv: vec2) -> vec4`, called for every pixel; uv is 0..1 with (0,0) top-left. Available inside it: `frame(uv)` samples the composed picture; `params.` reads each declared parameter; `ctx.time` is programme seconds; `ctx.region.x` goes 0 to 1 across the region (y its length in s, z seconds since it began); `ctx.resolution` and `ctx.aspect`; `ctx.cursor` (xy in uv, z = 1 when on screen); `ctx.clicks[0..3]`, the last four clicks, newest first (xy in uv where it happened, z seconds since, w = 1 when the slot holds one); `ctx.footage` and `ctx.camera`, the recording's and the webcam's rects (x, y, w, h in uv; camera zero-sized when absent) — all already placed through zoom and layout; `fx_to_footage(uv)` and `fx_from_footage(p)` map between output uv and the recording's own surface (0..1 across the recording) through the warp the compositor draws it with, zoom, layout and 3D camera included, and `ctx.footage_aspect` is its width over height: the effect works in whichever of the two spaces fits it; `fx_px(n)`, n pixels of a 1080p frame as a fraction of the height; `fx_hash(vec2)`, a random number in [0,1) identical on every GPU. Rules: the effect is a pure function of those inputs (no state between frames), so the preview and the export show the same pixels; reason in uv or fx_px, never raw pixels; every loop is a `for` with literal bounds (`for (var i = 0; i < 16; i++)`), at most 4096 iterations per pixel — the one hard rule. `params` declares up to 8 (recommended) parameters, each `{type: number|toggle|color|point, name, label, default}` (number also `min`, `max`); declare one only when moving it visibly changes the effect over its whole range. `target`: `image` (default: the picture under the annotations and captions), `text` (the annotations and captions alone, premultiplied alpha), `both` (everything); use text or both only when asked. Effects of one `lane` (0–7) cannot overlap; lanes apply in order, lane 0 first. The result gives the cost tier (frame() reads per pixel: light ≤ 8, about 0.06 ms per 1080p frame on a fast GPU; medium ≤ 32, 0.14 ms; heavy ≤ 128, 0.54 ms; very heavy beyond) and a recommendation when the tier runs long: light anywhere, medium up to a minute, heavy up to 10 s, very heavy a few seconds. It is a recommendation, never a refusal. Code that does not compile is refused with the compiler's error at the line of your code: fix it and call again. Check the result with getFrames.", + "Write a visual effect as a shader and pose it over a span of the edited timeline (virtual seconds), or pose one of this project's effects again (effectId). Use it for a look the settings do not offer. `effect.code` is WGSL that defines `fn effect(uv: vec2) -> vec4`, called for every pixel; uv is 0..1 with (0,0) top-left. Available inside it: `frame(uv)` samples the composed picture; `params.` reads each declared parameter; `ctx.time` is programme seconds; `ctx.region.x` goes 0 to 1 across the region (y its length in s, z seconds since it began); `ctx.resolution` and `ctx.aspect`; `ctx.cursor` (xy in uv, z = 1 when on screen); `ctx.clicks[0..3]`, the last four clicks, newest first (xy in uv where it happened, z seconds since, w = 1 when the slot holds one); `ctx.footage` and `ctx.camera`, the recording's and the webcam's rects (x, y, w, h in uv; camera zero-sized when absent) — all already placed through zoom and layout; `fx_to_footage(uv)` and `fx_from_footage(p)` map between output uv and the recording's own surface (0..1 across the recording) through the warp the compositor draws it with, zoom, layout and 3D camera included, and `ctx.footage_aspect` is its width over height: the effect works in whichever of the two spaces fits it; `fx_px(n)`, n pixels of a 1080p frame as a fraction of the height; `fx_hash(vec2)`, a random number in [0,1) identical on every GPU. Rules: the effect is a pure function of those inputs (no state between frames), so the preview and the export show the same pixels; reason in uv or fx_px, never raw pixels; every loop is a `for` with literal bounds (`for (var i = 0; i < 16; i++)`), at most 4096 iterations per pixel — the one hard rule. `params` declares up to 8 (recommended) parameters, each `{type: number|toggle|color|point, name, label, default}` (number also `min`, `max`); declare one only when moving it visibly changes the effect over its whole range. `target`: `image` (default: the picture under the annotations and captions), `text` (the annotations and captions alone, premultiplied alpha), `both` (everything); use text or both only when asked. Effects of one `lane` (0–7) cannot overlap; lanes apply in order, lane 0 first. The result gives the cost tier, in frame() reads per pixel, with what a 1080p frame pays on a fast discrete GPU (RTX 4070 Ti) and on an integrated one (Apple M1): light ≤ 8, up to 0.06 ms and 1.3 ms; medium ≤ 32, 0.14 ms and 4.3 ms; heavy ≤ 128, 0.54 ms and 16 ms; very heavy beyond, 2 ms and 49 ms at 512 reads. It also gives a recommendation when the tier runs long: light anywhere, medium up to a minute, heavy up to 10 s, very heavy a few seconds. It is a recommendation, never a refusal. Code that does not compile is refused with the compiler's error at the line of your code: fix it and call again. Check the result with getFrames.", setEffect: "Move, resize or re-tune an effect region by id (virtual seconds): its span, its parameter `values`, its `target` or its `lane`. Only the fields you pass change; a value equal to the default is dropped. Refused if the new span runs into another effect of the same lane.", setEffectDefinition: @@ -405,11 +407,22 @@ export async function runDocumentTool( : undefined; const autoZooms = name === "addAutoZooms" && editsAllowed ? await planAutoZooms(document, runtime) : undefined; + // The compositor compiles an effect's code for real, off the main thread: asked here, before + // the synchronous executor reads the answer. + const effectSource = + editsAllowed && runtime.checkShaderEffect ? shaderEffectToCheck(document, name, args) : null; + const checkShaderEffect = + effectSource && runtime.checkShaderEffect + ? shaderEffectCheckLookup( + effectSource, + await runtime.checkShaderEffect(effectSource.code, effectSource.params), + ) + : undefined; return executeAgentTool(document, name, JSON.stringify(args ?? {}), { editsAllowed, cursorTelemetry: { availableByAssetId: runtime.availableByAssetId, load }, cameraSegmentation: runtime.cameraSegmentation, - checkShaderEffect: runtime.checkShaderEffect, + checkShaderEffect, autoZooms, }); } diff --git a/electron/mcp/effects-contract.ts b/electron/mcp/effects-contract.ts index 47c28e37d..96c24d764 100644 --- a/electron/mcp/effects-contract.ts +++ b/electron/mcp/effects-contract.ts @@ -91,16 +91,17 @@ two lanes. ## Cost The result of addEffect gives the cost in frame() reads per pixel, counted from the code (loops -multiplied by their bound), and its tier. Measured on a fast GPU (RTX 4070 Ti) at 1080p: +multiplied by their bound), and its tier. Per 1080p frame, measured on a fast discrete GPU (RTX +4070 Ti) and on an integrated one (Apple M1): -| Tier | Reads | Per frame | Recommended for | -|---|---|---|---| -| light | ≤ 8 | ≤ 0.06 ms | anywhere, the whole video included | -| medium | ≤ 32 | ≤ 0.14 ms | up to a minute | -| heavy | ≤ 128 | ≤ 0.54 ms | up to 10 s: a transition, a key moment | -| very heavy | more | 2 ms at 512 | a few seconds | +| Tier | Reads | Per frame, RTX 4070 Ti | Per frame, M1 | Recommended for | +|---|---|---|---|---| +| light | ≤ 8 | ≤ 0.06 ms | ≤ 1.3 ms | anywhere, the whole video included | +| medium | ≤ 32 | ≤ 0.14 ms | ≤ 4.3 ms | up to a minute | +| heavy | ≤ 128 | ≤ 0.54 ms | ≤ 16 ms | up to 10 s: a transition, a key moment | +| very heavy | more | 2 ms at 512 | 49 ms at 512 | a few seconds | -An integrated GPU pays several times more. The tiers are recommendations, never refusals. +A 4K frame costs about four times as much. The tiers are recommendations, never refusals. ## Example: a ripple on the recording, on every click diff --git a/electron/native-bridge/services/compositorViewService.test.ts b/electron/native-bridge/services/compositorViewService.test.ts index 01817e826..40987fe62 100644 --- a/electron/native-bridge/services/compositorViewService.test.ts +++ b/electron/native-bridge/services/compositorViewService.test.ts @@ -271,6 +271,34 @@ function writeCargoConfig(root: string, body: string): void { ); } +describe("CompositorViewService checks an effect's code off the main thread", () => { + const service = (checkShaderEffect: CompositorViewAddon["checkShaderEffect"]) => + new CompositorViewService({ addon: { checkShaderEffect } as unknown as CompositorViewAddon }); + + it("passes the addon's answer through", async () => { + const check = { tier: "light" as const, frameReads: 1, iterations: 0 }; + const checker = vi.fn(async () => check); + expect( + await service(checker).checkShaderEffect("code", [{ name: "level", type: "number" }]), + ).toEqual(check); + expect(checker).toHaveBeenCalledWith( + "code", + JSON.stringify([{ name: "level", type: "number" }]), + ); + }); + + it("reads an addon that fails, whether it throws or rejects, as not checked", async () => { + const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined); + const rejects = vi.fn(() => Promise.reject(new Error("native"))); + const throws = vi.fn(() => { + throw new Error("native"); + }); + expect(await service(rejects).checkShaderEffect("code", [])).toBeNull(); + expect(await service(throws).checkShaderEffect("code", [])).toBeNull(); + warn.mockRestore(); + }); +}); + describe("ffmpegSharedBinCandidates", () => { let tmpRoot: string; diff --git a/electron/native-bridge/services/compositorViewService.ts b/electron/native-bridge/services/compositorViewService.ts index fd0a25b52..95bff7ad0 100644 --- a/electron/native-bridge/services/compositorViewService.ts +++ b/electron/native-bridge/services/compositorViewService.ts @@ -592,14 +592,16 @@ export class CompositorViewService { /** Checks an agent-written effect's WGSL in the compositor (`docs/shader-effects.md`, E.3): the * compile error, or the cost tier. `null` when this addon cannot answer, which the agent tools * report as "not checked" rather than as a compile failure. */ - checkShaderEffect( + async checkShaderEffect( code: string, params: ReadonlyArray<{ name: string; type: string }>, - ): ShaderEffectCheck | null { + ): Promise { const addon = this.ensureAddon(); if (!addon || typeof addon.checkShaderEffect !== "function") return null; try { - return addon.checkShaderEffect( + // `await`, not a bare return: the check is a promise now, and a rejection must land in + // this catch rather than escape it. + return await addon.checkShaderEffect( code, JSON.stringify(params.map(({ name, type }) => ({ name, type }))), ); diff --git a/electron/native/compositor-view/addon.d.ts b/electron/native/compositor-view/addon.d.ts index de1bb08d7..501a41c73 100644 --- a/electron/native/compositor-view/addon.d.ts +++ b/electron/native/compositor-view/addon.d.ts @@ -156,9 +156,10 @@ export interface CompositorViewAddon { * needs the answer before any preview exists. Cached native-side. */ probeBackend(): CompositorBackend; /** Checks an effect's WGSL: validation, literal loop bounds, the real compile for this machine's - * backend where there is one (FXC on Windows), and the cost tier. `paramsJson` is the declared - * parameters, `[{ "name": "strength", "type": "number" }, …]`. Absent from older builds. */ - checkShaderEffect?(code: string, paramsJson: string): ShaderEffectCheck; + * backend (FXC on Windows, Metal on macOS, SPIR-V on Linux), and the cost tier. `paramsJson` + * is the declared parameters, `[{ "name": "strength", "type": "number" }, …]`. Runs on the + * libuv pool: a Metal compile takes 120 ms and more. Absent from older builds. */ + checkShaderEffect?(code: string, paramsJson: string): Promise; /** Whether the ONNX Runtime library is where the app staged it, and therefore whether a * segmentation mask can be produced at all. Asked of the machine rather than guessed from diff --git a/src/lib/ai-edition/shaderEffects.ts b/src/lib/ai-edition/shaderEffects.ts index 8aa973128..6d0f29515 100644 --- a/src/lib/ai-edition/shaderEffects.ts +++ b/src/lib/ai-edition/shaderEffects.ts @@ -25,10 +25,19 @@ export interface ShaderEffectCheck { iterations?: number | null; } -/** Asks the compositor about a code; `null` where nothing can answer (no addon, a test). */ +/** Asks the compositor about a code; `null` where nothing can answer (no addon, a test). A + * promise: the compositor compiles for real, which takes 120 ms and more on Metal, off the main + * thread. */ export type ShaderEffectChecker = ( code: string, params: ReadonlyArray>, +) => Promise; + +/** The checker's answer as the tool executor reads it, synchronously: `runDocumentTool` asked + * first (`shaderEffectToCheck`). `null` for a code that was not asked about. */ +export type ShaderEffectCheckLookup = ( + code: string, + params: ReadonlyArray>, ) => ShaderEffectCheck | null; export function findShaderEffect( diff --git a/technical-documentation/engineering/rendering-performance.md b/technical-documentation/engineering/rendering-performance.md index b88bedc35..78c68bf18 100644 --- a/technical-documentation/engineering/rendering-performance.md +++ b/technical-documentation/engineering/rendering-performance.md @@ -474,7 +474,7 @@ An agent-written effect (`crates/compositor/src/shader_fx.rs`, [docs/shader-effe - **About 0.12 ms per read at 1080p and 0.5 ms at 4K,** over a fixed 0.3 ms: thirty times the RTX. The fixed part is the copy into `ann_copy` and the full-frame pass, which are bandwidth on a 68 GB/s unified memory. - **The reads dominate here too.** The bench effect computes a `sin` and a `cos` per read; with integer offsets instead, 128 reads cost 10.5 ms rather than 16.2. - **naga's `force_loop_bounding` is off** (`shader_fx::to_msl`): its guard, a `volatile` read per iteration, cost 7 to 10 % of every row above. The validation has already proved each loop bounded. -- **Compiling is ten times slower than FXC.** naga, `new_library_with_source` and the pipeline state take 120 ms in the median for a source Metal has not seen (the C.6 ripple), and 160 to 375 ms for the first one in a process. A source Metal has seen comes back in 0.4 ms in the same process, 3 ms in another. All of it runs on the background thread; the preview shows the effect once it lands. `checkShaderEffect` compiles on the calling thread, so an agent's `addEffect` took 152 ms through the `mcp` CLI. +- **Compiling is ten times slower than FXC.** naga, `new_library_with_source` and the pipeline state take 120 ms in the median for a source Metal has not seen (the C.6 ripple), and 160 to 375 ms for the first one in a process. A source Metal has seen comes back in 0.4 ms in the same process, 3 ms in another. All of it runs on the background thread; the preview shows the effect once it lands. An agent's `addEffect` took 152 ms through the `mcp` CLI, the compile included; `checkShaderEffect` runs on the libuv pool, off the main thread. - **The export does not notice an effect:** 180 frames at 1752×1080 (`exportMulti`, three alternated runs after a warm-up) took 2469 ms without, 2428 ms with the ripple over the whole video, and 2375 ms with the spike's heavier effect. The M1 export is encoder-bound. The live view held 29.7 fps against a 30 fps source with and without the effect. - **Not measured yet:** the Radeon 610M.