diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 807c91d..bfc5b04 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,6 +1,6 @@ # 架构总览 -这个仓库当前已经从模板收敛成一个本地 `computer-use` 项目。macOS 主线是 Swift 实现的 `maka.cu/2` executor：它只对 Maka Electron host 说话，不再自带任何面向模型的 tool 层。Windows 和 Linux 仍是实验性的 Go runtime，暴露原先那组 9 个 Computer Use tools，尚未迁到 host protocol。 +这个仓库当前已经从模板收敛成一个本地 `computer-use` 项目。macOS 主线是 Swift 实现的 `maka.cu/3` executor：它只对 Maka Electron host 说话，不再自带任何面向模型的 tool 层。Windows 和 Linux 仍是实验性的 Go runtime，暴露原先那组 9 个 Computer Use tools，尚未迁到 host protocol。 ## 当前目录结构 @@ -14,7 +14,7 @@ 实验性 Linux runtime。它不依赖 Swift 或 `.app` bundle，Go CLI/MCP 入口会嵌入 Python AT-SPI bridge，构建产物是 `open-computer-use`，并随已有 npm 包的 `dist/linux//` bundled artifacts 分发。 - `packages/OpenComputerUseKit` 核心库，包含： - - `maka.cu/2` host protocol server（`HostProtocol/`）：line-framed JSON-RPC、session/snapshot 生命周期、element token 绑定、dispatch 路径裁决 + - `maka.cu/3` host protocol server（`HostProtocol/`）：line-framed JSON-RPC、session/snapshot 生命周期、element token 绑定、dispatch 路径裁决 - app discovery - Accessibility / 窗口 snapshot - 键鼠输入模拟 @@ -43,7 +43,7 @@ - 辅助 drag panel 会跳转到对应的 `System Settings` 页面；点击 `Allow` 后，panel 会从主窗口里的按钮位置做一段 spring + curved frame 的入场，再落到 `System Settings` 内容区下沿。panel 默认保持在窗口右侧内容区下方居中并固定贴近窗口底边，不再依赖实时扫描权限页内部 `+ / -` 控件行；窗口层级上会显式排在当前 `System Settings` 窗口之上，避免被权限列表内容盖住，同时尽量减少对系统设置自身滚动区域的干扰。panel 内也补了显式返回按钮，允许用户中断当前 guidance、回到 onboarding 主窗口重新选择权限步骤。 - 权限状态会合并 TCC 持久授权记录与当前 app 进程的 runtime preflight：TCC 中任一匹配 client 已授权即可视为 granted，避免 CLI 子进程与 GUI app 对授权状态看到不一致的结果；如果当前 `.app` 进程已经通过 `AXIsProcessTrusted()` / `CGPreflightScreenCaptureAccess()`，也会立即视为 granted，避免 stale 或不匹配的 TCC path 记录让 onboarding 浮层继续停留。正式 release 仍以 CI 打出来的 `Open Computer Use.app` 为准，而本地 debug/dev 打包现在显式命名为 `Open Computer Use (Dev).app`，并在 dev bundle 运行时优先认当前 dev 副本，避免系统设置里出现两个完全同名的条目。 -### 2. Host Protocol 层（`maka.cu/2`） +### 2. Host Protocol 层（`maka.cu/3`） - transport 是 stdio 上的 line-framed JSON-RPC 2.0：一行一个 JSON value，UTF-8，`\n` 结尾，没有 `Content-Length`。stdout 只承载 JSON-RPC，诊断一律走 stderr。 - 两层错误刻意分开：JSON-RPC `error` 只表示请求本身不可用（`-32000` 版本不匹配、`-32001` 未握手、`-32002` 未知 session、`-32003` 正在退出）；世界的状态一律是 `result` 里的 `{ ok: false, error: { code, message, detail } }`，`code` 是闭集，`message` 由 `code` 决定，`detail` 只有枚举和数字。 @@ -58,8 +58,8 @@ - host protocol 自己实现 observe 与 dispatch：`observe` 产出结构化的 AX 树 + element token，可执行 mutation 通过 `dispatch.element` 或绑定已验证焦点的 `dispatch.key` 完成。生产动作面不接受坐标输入，也不把 semantic action 降级成 point event。它不复用 `ComputerUseService` 的 index 定址入口——把协议接到 index 上就等于把协议存在的理由接回来了。 - mutation 消费 snapshot 的 dispatch authority，但 spent snapshot 的 revision 仍可作为下一次 post-action observation 的差分基线；expired/evicted snapshot 不参与。这样动作后的模型输入可以只写有效变化，又不放宽 single-use snapshot 合同。 -- `{kind: "app"}` 按 window inventory 的前到后顺序选择当前 sheet/窗口；`{kind: "window"}` 严格按 PID + window ID。AppKit sheet 在 CGWindow 侧是独立窗口、在 AX 侧是主窗口的 `AXSheet` / `AXDrawer` child，匹配顺序固定为 direct AXWindow 后 child sheet。 -- CGWindow 已出现但对应 AXWindow/AXSheet 尚未发布时，会对同一 PID/window/frame 最多重读 250ms；window inventory 本身不存在目标时仍立即 `window_gone`，不会回退到别的窗口。 +- `target.resolve` 按 window inventory 的前到后顺序为应用选择当前 sheet/窗口；后续 `observe` 严格绑定 app identity + PID + process generation + window ID。AppKit sheet 在 CGWindow 侧是独立窗口、在 AX 侧是主窗口的 `AXSheet` / `AXDrawer` child；executor 通过 `_AXUIElementGetWindow` 将 direct AXWindow 与 child sheet 精确映射到获批的 `CGWindowID`，不再以 frame 猜测身份。 +- CGWindow 已出现但对应 AXWindow/AXSheet 尚未发布时，会对同一 PID/window ID/bounds 最多重读 250ms；window inventory 本身不存在目标、SPI 不可用、匹配不唯一或 bounds 已变化时都 fail closed，不会回退到别的窗口。 - host protocol 会同时绑定宿主 app 与真实 input-owner 的 PID/start time。WKWebView/WebContent 通过动态解析 `_AXUIElementGetActualPid` 识别；冷启动时用 XNU resource + jetsam coalition 的唯一 WebContent 关系做 readiness gate，首轮没有 `AXWebArea` 时等待 250ms 后重读一次。 - observation 会删除被唯一真实 WebContent 元素遮蔽的叶子 accessibility mirror；歧义或非叶子 mirror 保留。WebContent 左键点击使用 host window 的精确 `CGWindowID` 和单通道 private SkyLight 事件，WindowServer 完成 renderer hop，不再走会产生 `isTrusted=false` 的 AX/JavaScript mirror。 - retained AX 引用失效时允许同一进程世代内的唯一 identity-preserving refetch；只有 renderer-owned live 引用可在仅发生 `frame` reflow 时进入同一恢复路径。原生 AX 的 frame 变化和其它 digest 字段变化仍立即 fail closed，missing 与 ambiguous 分别拒绝为原始 stale code / `element_changed`。 @@ -135,11 +135,11 @@ ## 关键边界 - 开源版当前不复刻官方闭源实现里的 caller signing、私有 IPC、完整 overlay choreography 和 plugin 自安装逻辑。 -- 因为官方 `SkyComputerUseClient` 带有宿主侧 launch constraints，普通 stdio MCP client 在本机上可能被系统直接杀掉；如果要探测官方 bundled `computer-use`，`scripts/computer-use-cli` 的 app-server 模式现在只适合做工具清单和协议面观察。官方 `1.0.755` 的真实 tool call 还会经过 service-side sender authorization / active IPC client 追踪，外部 raw helper 即使走已签名 Codex binary，也可能返回 `Sender process is not authenticated`；需要真实使用官方工具时应走正常 Codex agent/tool 调用链，本仓库这条线现在只对 Maka host 说 `maka.cu/2`，不再提供可直连的 MCP server。 +- 因为官方 `SkyComputerUseClient` 带有宿主侧 launch constraints，普通 stdio MCP client 在本机上可能被系统直接杀掉；如果要探测官方 bundled `computer-use`，`scripts/computer-use-cli` 的 app-server 模式现在只适合做工具清单和协议面观察。官方 `1.0.755` 的真实 tool call 还会经过 service-side sender authorization / active IPC client 追踪，外部 raw helper 即使走已签名 Codex binary，也可能返回 `Sender process is not authenticated`；需要真实使用官方工具时应走正常 Codex agent/tool 调用链，本仓库这条线现在只对 Maka host 说 `maka.cu/3`，不再提供可直连的 MCP server。 - 当前权限引导已经具备可运行 app、深链、拖拽辅助，以及一版更接近官方的 accessory panel 入场动画和返回 affordance；点击链路也已经补上独立 visual cursor、官方 asset fallback 和相对目标 window 的排序逻辑，并且在 overlay 可见期间会持续重申“排在目标 window 之上”，避免用户手动激活目标 app 后 cursor 被目标窗口重新盖住；但整体还没有完全复刻官方那套嵌入式 choreography / host 集成 / session approval 体验。 - host protocol 的截图一律以文件路径返回，写在握手声明的 `imageDir` 里，生命周期与 snapshot 绑定；line-framed 通道上内联 base64 是 4/3 膨胀，而且一条 8 MB 的行会把其它待回的响应全部堵住。调试命令仍走 `ScreenCaptureKit` 捕获目标窗口，不再把普通 app 截图落盘到仓库或临时目录；编码前会按最大尺寸和目标字节数自适应缩小，避免复杂页面的大 PNG 触发 host 侧 MCP result 降级，同时 coordinate tools 继续按实际返回的 screenshot pixel 尺寸映射坐标；单次 ScreenCaptureKit capture 会设置超时，超时后省略 image block 而不是卡住整个 `get_app_state`。 - host protocol 的会话状态是进程内内存态：每个 session 持有自己的 snapshot 集合、element token 字典和保留的 `AXUIElement` 引用；`session.end` 会一次性释放 snapshot、删除本会话写出的图片，并清掉 executor 画的 cursor，同时把释放计数报回去，好让这类回归有断言可写。 -- 本仓库旧 MCP/CLI 产品面仍有历史坐标 API；它和 Maka 的 `maka.cu/2` host protocol 是不同边界。Windows、macOS 和后续平台接入 Maka 时必须共享 semantic-only host contract，不能从旧 MCP/CLI schema 派生第二套 model action space 或 fallback ladder。 +- 本仓库旧 MCP/CLI 产品面仍有历史坐标 API；它和 Maka 的 `maka.cu/3` host protocol 是不同边界。Windows、macOS 和后续平台接入 Maka 时必须共享 semantic-only host contract，不能从旧 MCP/CLI schema 派生第二套 model action space 或 fallback ladder。 ## 主要验证路径 @@ -162,4 +162,4 @@ - `open-computer-use doctor --json` - `open-computer-use snapshot ` - `open-computer-use list-apps` - - `open-computer-use host`（读 stdin 的 `maka.cu/2` host protocol server） + - `open-computer-use host`（读 stdin 的 `maka.cu/3` host protocol server） diff --git a/docs/HOST_PROTOCOL.md b/docs/HOST_PROTOCOL.md index c81c60f..9061879 100644 --- a/docs/HOST_PROTOCOL.md +++ b/docs/HOST_PROTOCOL.md @@ -1,4 +1,4 @@ -# maka-cu Host Protocol (`maka.cu/2`) +# maka-cu Host Protocol (`maka.cu/3`) The wire contract between the Maka Electron host (TypeScript) and `maka-cu`, the native macOS executor (Swift). Both sides are ours. This protocol answers to @@ -13,7 +13,7 @@ Two engineers who cannot talk to each other should be able to build the two ends from this document and have them interoperate. Where a rule exists because of a specific bug or measurement, the rule says so. -### Why this is version 2 +### Why this is version 3 `maka.cu/1` was built twice from this document, once in Swift and once in TypeScript, by engineers who could not talk to each other. The two ends did not @@ -29,6 +29,12 @@ them move the wire, so the version string moves with them: a `maka.cu/1` peer is not compatible and must fail the handshake rather than degrade (§2). There is no `maka.cu/1` peer worth interoperating with — no two of them agreed. +Version 3 adds side-effect-free target resolution and binds every observation to +the resolved app identity, PID, process generation, and window. The host can now +obtain approval for one native target and prove that every later read or effect +still addresses that exact target. Version 2 cannot express this proof and is +therefore not compatible. + --- ## 1. Framing @@ -128,7 +134,7 @@ message-redaction pass to get wrong. ### 1.3 One way to write a hash Every hash on this wire is the string `":"`. In -`maka.cu/2` the algorithm is always `sha256`, so every hash begins `sha256:`. +`maka.cu/3` the algorithm is always `sha256`, so every hash begins `sha256:`. This applies without exception to `element.digest`, `snapshot.windowDigest`, `image.sha256` — including the images in `screen.capture` (§6.6) and in the @@ -164,7 +170,7 @@ executor is already correct; the host is the side that changes. { "jsonrpc": "2.0", "id": 1, "method": "host.hello", "params": { - "protocol": "maka.cu/2", + "protocol": "maka.cu/3", "host": { "name": "maka", "version": "0.9.3" }, "hostPid": 8123, "imageDir": "/var/folders/…/maka-cu-images-8123", @@ -191,7 +197,7 @@ executor is already correct; the host is the side that changes. ```json { "ok": true, - "protocol": "maka.cu/2", + "protocol": "maka.cu/3", "executor": { "name": "maka-cu", "version": "0.4.0", "commit": "1747868" }, "pid": 8140, "capabilities": { @@ -228,7 +234,7 @@ executor implements, it MUST answer ```json { "jsonrpc": "2.0", "id": 1, "error": { "code": -32000, "message": "protocol_version_mismatch", - "data": { "supported": ["maka.cu/2"] } } } + "data": { "supported": ["maka.cu/3"] } } } ``` then flush stdout and exit with status `78` (`EX_CONFIG`). The host MUST classify @@ -237,7 +243,7 @@ already treats `service_mismatch` as non-retryable (`cua-driver-service.ts:200-203`). Silent degradation to a subset is forbidden in both directions. -`supported` lists `maka.cu/2` and nothing else. `maka.cu/1` is withdrawn, not +`supported` lists `maka.cu/3` and nothing else. `maka.cu/1` is withdrawn, not deprecated: the parts of it that moved are exactly the parts the two `maka.cu/1` implementations disagreed about, so a peer still speaking it is a peer whose behaviour on those points is unknown. Accepting it back would reintroduce every @@ -474,10 +480,9 @@ There is exactly one string that names an app on this wire, and it is called moved to the side that knows. - `apps.list`, `window.list`, `snapshot.target` and the `apps.launch` result all carry it, spelled the same way, for the same process. -- `{ "kind": "app", "app": … }` (§5.2) takes an `appId` and nothing else. The - executor resolves it by exact string match against `appId`. It MUST NOT match - against `appName`, against `snapshot.target.title`, or against any prefix or - case-folded form of either. +- `target.resolve` is the only operation that accepts a display name. It returns + a canonical `appId` and exact execution selector; later reads never resolve a + display name again. - `appName` is a display string. It is untrusted application content (§1.2), it is localised, two apps may share one, and it is never a key. - There is no `bundleId` field anywhere on this wire. It was a second spelling of @@ -507,16 +512,25 @@ one namespace, the app *name*, in both places (`appIdForWindow` feeds both the better one to standardise on, because a display name is neither unique nor stable across locales. -Which side changes: both. The executor adds `appId` to `window.list` and -`snapshot.target` and resolves `{kind: "app"}` on it. The host stops matching on -`appName`/`title` and passes `appId` through unaltered. +### 5.1.1 `target.resolve` + +`target.resolve` is read-only. It MUST NOT launch or activate an application. -When the host is given both an app string and a window id, it resolves the window -id — exact, numeric — and then requires that window's `appId` to equal the app -string. Disagreement is `target_missing`, because no window satisfies the pair; -honouring one input and discarding the other would be acting on a target the -caller did not name. This is not the old over-strict rule, which required both to -match when the caller had sent only one (§5.2). +```json +{ "method": "target.resolve", + "params": { "target": { + "kind": "application", "app": "TextEdit", "intent": "operate" + } } } +``` + +`intent: "operate"` returns exactly one running window or `missing` / `ambiguous`. +`intent: "launch"` returns exactly one installed bundle identifier without +starting it. An exact window request is `{ "kind": "window", "windowId": 90210 }`. + +A resolved running target contains `appId`, `pid`, `processGeneration` and +`windowId`. `processGeneration` is canonical `pst:` and is taken +from the process start time. A resolved installed target contains only its bundle +identifier as `appId`. ### 5.2 `observe` @@ -524,7 +538,13 @@ match when the caller had sent only one (§5.2). { "method": "observe", "params": { "session": "s-01J…", - "target": { "kind": "window", "pid": 4711, "windowId": 90210 }, + "target": { + "kind": "window", + "appId": "com.apple.Notes", + "pid": 4711, + "processGeneration": "pst:912345678", + "windowId": 90210 + }, "includeImage": true, "maxElements": 1500, "maxDepth": 64, @@ -532,29 +552,19 @@ match when the caller had sent only one (§5.2). } } ``` -`target` is a **tagged union**, never a bag of optional fields: +`target` is the exact running selector returned by `target.resolve`; `observe` +does not perform target selection. Before reading, and again before returning, +the executor requires the same app identity, PID generation and window. A failed +post-check discards the snapshot and image. -```json -{ "kind": "app", "app": "com.apple.Notes" } -{ "kind": "window", "pid": 4711, "windowId": 90210 } -``` - -`app` is an `appId` (§5.1). `{ "kind": "app" }` resolves to the app's frontmost -usable window and is ambiguous by design; `{ "kind": "window" }` is exact. The frontmost inventory entry may be an AppKit sheet. `CGWindowList` reports a sheet as a window, while Accessibility exposes it as an `AXSheet`/`AXDrawer` -child of the main `AXWindow`; the executor first matches ordinary AX windows by -frame, then matches those child roles by the same frame. It does not query a -fictional `AXSheets` attribute. Exact window targeting never falls back to the -main window when the requested secondary or sheet window cannot be matched. -Optional `app` *and* optional `windowId` in one object is how a real-machine -failure happened: the contract said "app **or** window\_id" while the harness -required both to match, so a compliant model could not pass. A tagged union -cannot express that disagreement. - -The **executor** resolves `{ "kind": "app" }`. It owns the window inventory and -the z-order, and the host that tried to pre-resolve an app string against -`window.list` is the host that invented title matching to make it work (§5.1). +child of the main `AXWindow`. The executor walks both ordinary AX windows and +those child roles, then uses `_AXUIElementGetWindow` to require the exact +`CGWindowID` plus the expected bounds. It does not query a fictional `AXSheets` +attribute or fall back to frame-only matching. Exact window targeting never +falls back to the main window when the requested secondary or sheet window +cannot be matched uniquely. Omitted `maxElements` / `maxDepth` / `maxTextChars` mean the values in `limits`. A value above the limit is `-32602`, not a silent clamp. @@ -670,7 +680,7 @@ the window list is read. other. They are not merged into one field because §4.3 digests them separately and - §6.2 reports `detail.changed: ["title"]`. `maka.cu/2` carried the digest input + §6.2 reports `detail.changed: ["title"]`. `maka.cu/3` carried the digest input without carrying the field, so the executor could tell a host *the title changed* about something it had never sent — and the host's only possible reply was to re-observe and compare nothing. @@ -790,10 +800,12 @@ fact `move_window` rests on. Measured across seventeen applications on this machine, `AXPosition` equalled `CGWindowListCopyWindowInfo`'s origin to the point on every one, including the four windows with a negative origin — iTerm2 at `(80, -1049)`, a Chrome window at `(-193, -1049)`, Terminal at `(775, -964)`, -Music at `(277, -931)`. The executor was already relying on this without saying -so: `HostAX.window(pid:windowId:bounds:)` matches AX windows against the window -list's frame to within one point, because there is no public AX attribute -carrying a `CGWindowID`. +Music at `(277, -931)`. The executor uses that agreement as a consistency check, +not as identity: `HostAX.window(pid:windowId:bounds:)` obtains the AX element's +exact `CGWindowID` through `_AXUIElementGetWindow` and separately requires its +bounds to match the window inventory. If the SPI is unavailable, the id is +missing, the match is ambiguous, or the bounds changed, lookup fails closed +rather than guessing by frame. `image.scale` is `image.widthPx / target.bounds.width`, computed by the executor from the image it actually captured — not from `NSScreen.backingScaleFactor`. @@ -854,9 +866,9 @@ statements shipped. Ordered front-to-back. `zIndex` is monotonically decreasing along the array; the executor MUST NOT emit ties. `appId` is required and is the same namespace `apps.list` returns (§5.1); its absence here is what made an app string -unresolvable against this list. The host uses this list for occlusion decisions -and for joining a window id to its pid — not for resolving `{ "kind": "app" }`, -which the executor does (§5.2). +unresolvable against this list. The host uses this list for occlusion decisions; +target selection belongs to the executor's side-effect-free `target.resolve` +operation (§5.1.1). ### 5.5 `apps.list` @@ -1015,6 +1027,19 @@ target kind, no new dispatch kind and no new method: a menu item exposes `AXPress`, which normalises to `press`, which is what `{ "kind": "click", "button": "left" }` already requires. +Menu bindings are application-level, not window-level. macOS attaches the menu +bar to the application responder chain, so a menu item is validated against the +snapshot's app identity, PID generation, input owner and element digest, but it +is not required to descend from the snapshot window. Window-tree elements do +have that additional requirement: immediately before every effect they must still +belong to the exact `CGWindowID` frozen by the snapshot. +Because menu effects have no authoritative window root, `settle: "quiesce"` +does not run a window-tree settle for them; the response keeps the action's own +verification and reports the settle as `not_requested`. A requested +`observeAfter` may still return a fresh view of the original window when it +exists, or a typed post-observation error without changing the delivered menu +outcome. + **Why this exists.** Before it, no observation this executor produced contained a single menu element — not a truncated one, not a filtered one, *none*. `observe` roots its walk at a window element and `kAXMenuBarAttribute` hangs off the @@ -1457,12 +1482,12 @@ one write per row, polling the window list at 5 ms: | Obsidian | 16 ms | 172 ms | Everything else in this executor reads the window server: `observe` resolves its -target out of `CGWindowListCopyWindowInfo`, and matches the AX window against -that frame to within one point because there is no public AX attribute carrying a -`CGWindowID`. So an executor that returned the instant the write was +target out of `CGWindowListCopyWindowInfo`, then requires both the exact +`CGWindowID` from `_AXUIElementGetWindow` and matching bounds. So an executor +that returned the instant the write was acknowledged would answer with a frame in which its own `observeAfter` cannot find the window — the list still reporting the old origin, the application -already reporting the new one, no candidate within a point, and +already reporting the new one, no candidate with the expected bounds, and `postObservationError: window_gone` for a window in plain sight. The host's next `observe` would race the same way. @@ -2438,14 +2463,14 @@ finer and remains safe because the host's queue is still upstream of it. --- -## 10. Capture stream (reserved, not implemented in `maka.cu/2`) +## 10. Capture stream (reserved, not implemented in `maka.cu/3`) Maka's picture-in-picture mirror repaints from the screenshot each action returns. A live mirror needs a stream, and this channel is request/response — so the stream is long-polling, the way Codex does it on its privileged channel (`AppStartCapture` then repeated `AppNextCaptureUpdate`). -The method space is reserved now. In `maka.cu/2` all three return +The method space is reserved now. In `maka.cu/3` all three return `{ "ok": false, "error": { "code": "not_implemented" } }` — a **domain** result, not `-32601`, so feature detection is a stable field read and the names can never be taken by something else. @@ -2468,7 +2493,7 @@ be taken by something else. → { "ok": true, "released": { "frames": 30 } } ``` -What `maka.cu/2` already provides so this needs no protocol change: +What `maka.cu/3` already provides so this needs no protocol change: - **Images are already references.** A 10 fps stream is path churn, not stdout churn. @@ -2642,14 +2667,14 @@ Naming an app (§5.1): 30. `apps.list`, `window.list`, `snapshot.target` and the `apps.launch` result report the same `appId` for the same process. -31. `observe` with `{ "kind": "app", "app": "" }` - resolves; the reviewer's reproduction — an `{app, windowId}` observation of a - bundle-identified app — succeeds instead of being refused. -32. `observe` with an `app` that is a display name, not an `appId`, is - `app_not_found`; the executor does not fall back to matching `appName` or - `title`. -33. `apps.launch` by display name returns the resolved `appId`, and a subsequent - `observe` with that `appId` resolves the launched window. +31. `target.resolve` with a bundled app's display name returns the canonical + bundle `appId`, PID, process generation, and window without launching or + activating anything. +32. An ambiguous display name returns `ambiguous`; it does not pick a match by + title, prefix, locale, or inventory order. +33. `apps.launch` receives the exact installed target returned by + `target.resolve`, and the subsequent `observe` receives the exact running + selector returned by a new resolution. Keys (§6.4): diff --git a/docs/QUALITY_SCORE.md b/docs/QUALITY_SCORE.md index 500134c..23fb510 100644 --- a/docs/QUALITY_SCORE.md +++ b/docs/QUALITY_SCORE.md @@ -11,10 +11,10 @@ | 区域 | 评分 | 原因 | 下一步 | | --- | --- | --- | --- | -| 产品面 | B | macOS `maka.cu/2` 已具备 snapshot/token/digest、跨 revision stable ID 与 post-action AX diff、唯一 stale refetch、WebContent trusted click、数值 slider 和语义 scroll；旧 MCP/CLI 产品面仍保留 9 tools。 | 完成签名/notarization 与 Maka release 集成，并继续收敛复杂 AX 场景。 | +| 产品面 | B | macOS `maka.cu/3` 已具备 snapshot/token/digest、跨 revision stable ID 与 post-action AX diff、唯一 stale refetch、WebContent trusted click、数值 slider 和语义 scroll；旧 MCP/CLI 产品面仍保留 9 tools。 | 完成签名/notarization 与 Maka release 集成，并继续收敛复杂 AX 场景。 | | Windows runtime | C | 已新增独立 Go `.exe`，通过 Windows UI Automation + Win32 window message 暴露同样 9 个 tools、MCP server 和 `call --calls`；默认不再自动启动 app、执行 `SetFocus`，或让 `type_text` 走可能抢前台的 UIA text fallback，并已接入 npm bundled artifact 分发，但仍是功能性第一版。 | 补交互式桌面 smoke、Windows fixture、installer/signing，以及更原生的 Go UIA 实现或更稳定的 bridge。 | | Linux runtime | C | 已新增独立 Go binary，通过 Python GI / AT-SPI2 暴露同样 9 个 tools、MCP server 和 `call --calls`；Ubuntu GNOME VM 已跑通 `list_apps`、MCP tools list 和 Text Editor 8-tool sequence，并已接入 npm bundled artifact 分发，但截图在 GNOME Wayland 下仍只能 best-effort，coordinate input 也不是通用后台模型。 | 补 Linux fixture、可重复 smoke runner、portal/compositor screenshot 路径，以及更原生的 Go D-Bus/libatspi bridge。 | | 架构文档 | B | 顶层结构、fixture bridge、app 模式和验证路径已经落文档。 | 后续补 release artifact、code signing / notarization 和 host 集成方式。 | | 测试 | B | `swift test` 覆盖绑定/refetch/path/readback/doctor/stable revision diff；共享 CUA Lab 已验证 WebContent、slider、scroll、stale，以及 modal/secondary open-button-scroll-close 5 轮真机闭环且目标始终后台。 | 把共享矩阵收进可选 CI/live runner，并继续扩展跨应用 modal/window 样本。 | | 可观测性 | B | `doctor --json` 已覆盖协议/版本、TCC、锁屏、SkyLight、actual-PID SPI、coalition、签名/hardened runtime 与 readiness；另有 snapshot、smoke 和对比样本。 | 补统一日志级别、notarization/staple 诊断与 release artifact 自检。 | -| 安全 | B | 已明确本地-only、权限边界和 fixture test bridge 的作用域，并将内置 denylist 收缩到密码管理器。 | 增加 session approval 和更清楚的敏感 app policy，避免策略长期硬编码在仓库里。 | +| 安全 | B | 已明确本地-only、权限边界和 fixture test bridge 的作用域，并将内置 denylist 收缩到密码管理器；v3 snapshot/effect 通过精确 `CGWindowID`、应用身份与进程代际绑定获批目标。 | 增加 session approval 和更清楚的敏感 app policy，避免策略长期硬编码在仓库里。 | diff --git a/docs/exec-plans/completed/20260902-maka-cu-v3-target-binding.md b/docs/exec-plans/completed/20260902-maka-cu-v3-target-binding.md new file mode 100644 index 0000000..9ff85b7 --- /dev/null +++ b/docs/exec-plans/completed/20260902-maka-cu-v3-target-binding.md @@ -0,0 +1,35 @@ +# maka.cu/3 trusted target binding + +## Goal + +Provide Maka with a side-effect-free target resolver and process-generation evidence so managed Computer Use admission and native execution use the same application target. + +## Scope + +- Bump the host protocol directly from `maka.cu/2` to `maka.cu/3`. +- Add a read-only target-resolution method for running apps, exact windows, and installed launch targets. +- Carry canonical app identity and process generation in running targets and snapshots. +- Validate the snapshot root before and after evidence reads and before every real effect; discard evidence when the process generation changes. +- Preserve a known successful effect when only the post-action observation loses its root. +- Keep launch, TCC prompts, Accessibility reads, and application content out of target resolution. +- Update protocol documentation, focused tests, and history. + +## Non-goals + +- No compatibility fallback for v2. +- No Session Grant or user-approval UI in this repository. +- No Windows or Linux protocol changes. +- No new action-result causal semantics. + +## Steps + +1. Define the v3 wire types and side-effect-free resolver. +2. Bind snapshots and evidence to canonical root identity and process generation. +3. Add deterministic protocol, replacement, per-effect binding, and post-observation tests. +4. Update docs/history and run `swift test` plus focused host protocol tests. + +## Status + +Completed, including the per-effect binding and post-observation follow-up. Source parsing succeeds locally; the full Swift build and test suite +must run on the repository's matching macOS CI toolchain because this machine's +Swift compiler and SDK versions differ. diff --git a/docs/histories/2026-09/20260902-1216-bind-native-targets.md b/docs/histories/2026-09/20260902-1216-bind-native-targets.md new file mode 100644 index 0000000..e665369 --- /dev/null +++ b/docs/histories/2026-09/20260902-1216-bind-native-targets.md @@ -0,0 +1,33 @@ +## [2026-09-02 12:16] | Task: Bind native execution targets + +### 🤖 Execution Context +* **Agent ID**: `OpenAI Codex` +* **Base Model**: `GPT-5` +* **Runtime**: `Codex Desktop` + +### 📥 User Query +> Add the native target identity required for Maka's managed Computer Use Session Grants. + +### 🛠 Changes Overview +**Scope:** macOS `maka.cu` host protocol and executor + +**Key Actions:** +- **Resolve targets**: Added a side-effect-free resolver for installed applications and exact running windows. +- **Bind execution**: Added process-generation evidence and exact app, PID, and window validation around observations and effects. +- **Bind AX windows exactly**: Resolve direct windows and attached sheets through their exact `CGWindowID`, reject ambiguous/frame-only matches, and re-check the retained element immediately before effects. +- **Revalidate compound actions**: Re-check the approved root, binding, and derived-element window before every Accessibility action or write; stop repeated actions with `outcome_unknown` if an earlier effect already landed. +- **Preserve completed effects**: Keep a known successful outcome and its evidence when only post-action observation reports `window_gone`. +- **Clarify menus**: Keep menu dispatch application-bound while window-tree dispatch remains bound to the frozen window. +- **Advance protocol**: Replaced `maka.cu/2` with the incompatible `maka.cu/3` contract and updated focused tests and documentation. +- **Remove stale harnesses**: Deleted the two hard-coded v2 proof scripts now superseded by the v3 tests and protocol tooling. + +### 🧠 Design Intent (Why) +Maka must approve the same canonical application target that the native executor later reads or changes. A PID or display name alone cannot prove that binding because applications can restart and PIDs can be reused. + +### 📁 Files Modified +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSnapshotRegistry.swift` +- `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSystemEnvironment.swift` +- `packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTests.swift` +- `docs/HOST_PROTOCOL.md` diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift index d511ab3..657719b 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift @@ -241,6 +241,46 @@ enum AppDiscovery { ) } + /// Resolve an installed application without starting it. Managed hosts use + /// this before approval, so this path must remain free of launch or focus + /// side effects. + static func matchingInstalledBundleIdentifiers(_ query: String) throws -> [String] { + let normalizedQuery = query.trimmingCharacters(in: .whitespacesAndNewlines) + guard !normalizedQuery.isEmpty else { return [] } + + if let blocked = blockedBundleIdentifier(forQuery: normalizedQuery) { + throw AppSafetyPolicy.permissionDenied(bundleIdentifier: blocked) + } + + if isBundleIdentifierQuery(normalizedQuery) { + guard + let url = NSWorkspace.shared.urlForApplication( + withBundleIdentifier: normalizedQuery + ), + let bundleIdentifier = Bundle(url: url)?.bundleIdentifier, + !AppSafetyPolicy.isBlocked(bundleIdentifier: bundleIdentifier) + else { + return [] + } + return [bundleIdentifier] + } + + return Array( + Set( + applicationURLs(named: normalizedQuery).compactMap { url in + let bundleIdentifier = Bundle(url: url)?.bundleIdentifier + guard + let bundleIdentifier, + !AppSafetyPolicy.isBlocked(bundleIdentifier: bundleIdentifier) + else { + return nil + } + return bundleIdentifier + } + ) + ).sorted() + } + private static func resolvedRunningApp(in descriptors: [RunningAppDescriptor], matching query: String) -> RunningAppDescriptor? { if isBundleIdentifierQuery(query) { return descriptors.first(where: { descriptor in @@ -357,14 +397,19 @@ enum AppDiscovery { } private static func applicationURL(named query: String) -> URL? { + applicationURLs(named: query).first + } + + private static func applicationURLs(named query: String) -> [URL] { let targetName = stripAppSuffix(from: query).trimmingCharacters(in: .whitespacesAndNewlines) guard !targetName.isEmpty else { - return nil + return [] } let fileManager = FileManager.default let resourceKeys: [URLResourceKey] = [.isApplicationKey, .isDirectoryKey, .nameKey] var visitedPaths: Set = [] + var matches: [URL] = [] for root in standardApplicationSearchRoots where fileManager.fileExists(atPath: root.path) { guard let enumerator = fileManager.enumerator( @@ -387,12 +432,12 @@ enum AppDiscovery { let candidateName = stripAppSuffix(from: candidateURL.lastPathComponent) if candidateName.caseInsensitiveCompare(targetName) == .orderedSame { - return candidateURL + matches.append(candidateURL) } } } - return nil + return matches.sorted { $0.path < $1.path } } /// §5.7 — the configuration every launch is asked for. diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift index 7d01358..c2b7eff 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift @@ -418,7 +418,7 @@ public final class ComputerUseService { public init() {} /// A human-readable catalogue for `open-computer-use list-apps`. The - /// `maka.cu/2` executor never emits this: `apps.list` answers with structured + /// `maka.cu/3` executor never emits this: `apps.list` answers with structured /// data, because rendered text is model-facing prose and Maka's runtime owns /// every word the model reads. public func listApps() -> String { diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostAccessibilityAdapter.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostAccessibilityAdapter.swift index 817abab..c0eca80 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostAccessibilityAdapter.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostAccessibilityAdapter.swift @@ -9,6 +9,11 @@ private typealias HostAXGetActualPid = @convention(c) ( UnsafeMutablePointer ) -> AXError +private typealias HostAXGetWindowId = @convention(c) ( + AXUIElement, + UnsafeMutablePointer +) -> AXError + private let hostAXGetActualPid: HostAXGetActualPid? = { guard let handle = dlopen( "/System/Library/Frameworks/ApplicationServices.framework/ApplicationServices", @@ -22,10 +27,38 @@ private let hostAXGetActualPid: HostAXGetActualPid? = { return unsafeBitCast(symbol, to: HostAXGetActualPid.self) }() +private let hostAXGetWindowId: HostAXGetWindowId? = { + guard let handle = dlopen( + "/System/Library/Frameworks/ApplicationServices.framework/ApplicationServices", + RTLD_LAZY | RTLD_LOCAL + ) else { + return nil + } + guard let symbol = dlsym(handle, "_AXUIElementGetWindow") else { + return nil + } + return unsafeBitCast(symbol, to: HostAXGetWindowId.self) +}() + public func hostActualPidSPIAvailable() -> Bool { hostAXGetActualPid != nil } +public func hostWindowIdSPIAvailable() -> Bool { + hostAXGetWindowId != nil +} + +public func hostWindowId(of element: AXUIElement) -> CGWindowID? { + guard let hostAXGetWindowId else { + return nil + } + var windowId = CGWindowID(0) + guard hostAXGetWindowId(element, &windowId) == .success, windowId != 0 else { + return nil + } + return windowId +} + /// Everything in this file talks to macOS. It is kept apart from the protocol /// logic so the lifecycle, binding and path rules stay testable without a /// desktop, and so this is the only place to look when Accessibility behaviour @@ -64,19 +97,23 @@ public struct HostWindowInfo: Equatable, Sendable { public let displayId: String? } -func hostFirstWindowCandidate( - _ candidates: [(element: Element, frame: CGRect?)], - matching bounds: CGRect +func hostWindowBoundsMatch(_ lhs: CGRect, _ rhs: CGRect) -> Bool { + abs(lhs.origin.x - rhs.origin.x) < 1 + && abs(lhs.origin.y - rhs.origin.y) < 1 + && abs(lhs.width - rhs.width) < 1 + && abs(lhs.height - rhs.height) < 1 +} + +func hostUniqueWindowCandidate( + _ candidates: [(element: Element, windowId: CGWindowID?, frame: CGRect?)], + matching windowId: CGWindowID, + bounds: CGRect ) -> Element? { - candidates.first { candidate in - guard let frame = candidate.frame else { - return false - } - return abs(frame.origin.x - bounds.origin.x) < 1 - && abs(frame.origin.y - bounds.origin.y) < 1 - && abs(frame.width - bounds.width) < 1 - && abs(frame.height - bounds.height) < 1 - }?.element + let matches = candidates.filter { candidate in + candidate.windowId == windowId + && candidate.frame.map { hostWindowBoundsMatch($0, bounds) } == true + } + return matches.count == 1 ? matches[0].element : nil } public enum HostWindowInventory { @@ -854,24 +891,13 @@ enum HostAX { windows = array(application, kAXWindowsAttribute) } - // There is no public AX attribute carrying a CGWindowID, so the window is - // matched by its frame against the one the window list reported. Bounds - // are compared at whole-point resolution because AX and CGWindowList - // disagree in the sub-pixel digits on scaled displays. - if let matched = hostFirstWindowCandidate( - windows.map { ($0, frame($0)) }, - matching: bounds - ) { - return matched - } - // A sheet is a window to CGWindowList and a child to accessibility. It // is never in `AXWindows` — it is a child of its parent window whose // role is `AXSheet`, and a drawer is the same shape. Alerts, save // panels, print panels and permission prompts are all sheets, so an // observer that reads only `AXWindows` goes blind exactly when the app - // has stopped to ask a question — and `{ "kind": "app" }` resolves to - // the frontmost window, which while a sheet is up is the sheet. + // has stopped to ask a question — and `target.resolve` chooses the + // frontmost window, which while a sheet is up is the sheet. // // There is no `AXSheets` attribute, which is the trap. AppleScript // offers `sheets of window` and that reads like one, but System Events @@ -886,9 +912,10 @@ enum HostAX { sheetLikeRoles.contains(string($0, kAXRoleAttribute) ?? "") } } - return hostFirstWindowCandidate( - sheets.map { ($0, frame($0)) }, - matching: bounds + return hostUniqueWindowCandidate( + (windows + sheets).map { ($0, hostWindowId(of: $0), frame($0)) }, + matching: windowId, + bounds: bounds ) } @@ -896,6 +923,25 @@ enum HostAX { /// accessibility reports them as a child of one. static let sheetLikeRoles: Set = ["AXSheet", "AXDrawer"] + static func containingWindow(of element: AXUIElement) -> AXUIElement? { + var current: AXUIElement? = element + var visited = Set() + var examined = 0 + while let candidate = current, examined < 128 { + let key = HostAXElementKey(candidate) + guard visited.insert(key).inserted else { + return nil + } + examined += 1 + let role = string(candidate, kAXRoleAttribute) ?? "" + if role == kAXWindowRole as String || sheetLikeRoles.contains(role) { + return candidate + } + current = parent(of: candidate) + } + return nil + } + static func focusedElement(pid: pid_t) -> AXUIElement? { let application = AXUIElementCreateApplication(pid) guard let value = attribute(application, kAXFocusedUIElementAttribute) else { @@ -1079,10 +1125,12 @@ enum HostAX { /// copy of that list — and the two copies have drifted twice now, each time /// refusing dispatches against elements nothing had touched. final class HostAXBindingProbe: HostElementBindingProbe { + let windowId: CGWindowID let windowBounds: CGRect private let limits = HostLimits() - init(windowBounds: CGRect) { + init(windowId: CGWindowID, windowBounds: CGRect) { + self.windowId = windowId self.windowBounds = windowBounds } @@ -1104,6 +1152,20 @@ final class HostAXBindingProbe: HostElementBindingProbe { return HostAX.actualPid(of: element) } + func belongsToBoundTarget(_ binding: HostElementBinding) -> Bool { + if binding.isMenu { + return true + } + guard + let element = binding.element, + let window = HostAX.containingWindow(of: element), + hostWindowId(of: window) == windowId + else { + return false + } + return true + } + func currentDigestInput(_ binding: HostElementBinding) -> HostElementDigestInput? { guard let element = binding.element else { return nil @@ -1185,7 +1247,7 @@ final class HostAXBindingProbe: HostElementBindingProbe { dropsAppleMenu: true ) } else { - guard let window = HostAX.window(pid: binding.pid, windowId: 0, bounds: windowBounds) else { + guard let window = HostAX.window(pid: binding.pid, windowId: windowId, bounds: windowBounds) else { return [] } root = HostAXNode( diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostElementBinding.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostElementBinding.swift index fc319c3..5523dad 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostElementBinding.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostElementBinding.swift @@ -263,6 +263,10 @@ public protocol HostElementBindingProbe { /// The input owner right now. This differs from `binding.pid` for /// out-of-process WebContent/renderer elements. func actualPid(_ binding: HostElementBinding) -> pid_t? + /// The retained element still belongs to the target the snapshot froze. + /// Window-tree bindings prove the exact CGWindowID; application-menu + /// bindings prove their application identity through E2 and `actualPid`. + func belongsToBoundTarget(_ binding: HostElementBinding) -> Bool /// E3 — the digest inputs as they are right now, or `nil` when the element /// can no longer be read at all. func currentDigestInput(_ binding: HostElementBinding) -> HostElementDigestInput? @@ -279,6 +283,10 @@ public extension HostElementBindingProbe { binding.dispatchPid } + func belongsToBoundTarget(_ binding: HostElementBinding) -> Bool { + true + } + func uniqueRefetch(_ binding: HostElementBinding) -> HostBindingRefetchResult { .missing } @@ -348,10 +356,10 @@ public final class HostElementBinding { } } -/// Verifies E1, E2 and E3 in that order. The order matters: a dead reference and -/// a recycled pid are different diagnoses, and reporting a digest mismatch for a -/// process that no longer exists would send the host down the wrong retry path. -public func hostVerifyBinding( +/// Verifies the retained reference still belongs to the same process generation +/// and exact approved window. Compound actions repeat this check after their +/// first effect, when the effect itself may legitimately have changed E3. +public func hostVerifyBindingTarget( _ binding: HostElementBinding, probe: HostElementBindingProbe ) -> HostDomainError? { @@ -372,6 +380,24 @@ public func hostVerifyBinding( return HostDomainError(.processReplaced) } + guard probe.belongsToBoundTarget(binding) else { + return HostDomainError(.windowChanged) + } + + return nil +} + +/// Verifies E1, E2 and E3 in that order. The order matters: a dead reference and +/// a recycled pid are different diagnoses, and reporting a digest mismatch for a +/// process that no longer exists would send the host down the wrong retry path. +public func hostVerifyBinding( + _ binding: HostElementBinding, + probe: HostElementBindingProbe +) -> HostDomainError? { + if let failure = hostVerifyBindingTarget(binding, probe: probe) { + return failure + } + guard let current = probe.currentDigestInput(binding) else { return HostDomainError(.elementReleased) } diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostObservation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostObservation.swift index ba711d6..26136ac 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostObservation.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostObservation.swift @@ -19,7 +19,7 @@ public struct HostObservedElement: Codable, Equatable, Sendable { /// a wire that merged them could not report `detail.changed: ["title"]` /// against anything the host had been shown. /// - /// It was missing from `maka.cu/2` until the menu bar needed it, and it was + /// It was missing from `maka.cu/3` until the menu bar needed it, and it was /// already a hole before that. §4.3 lists `title` among the digest inputs and /// §6.2 reports it in `detail.changed`, so the protocol could tell a host /// *the title changed* about a field it had never sent — the host's only @@ -79,6 +79,7 @@ public struct HostDisplayInfo: Codable, Equatable, Sendable { public struct HostWindowTarget: Codable, Equatable, Sendable { public let pid: Int32 + public let processGeneration: String public let windowId: UInt32 /// §5.1 — the one namespace, spelled the same way `apps.list`, `window.list` /// and the `apps.launch` result spell it for the same process. There is no diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift index 54719ae..beb5d46 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer+Observe.swift @@ -95,6 +95,61 @@ extension HostProtocolServer { return true } + private func validateRoot( + appId: String, + pid: pid_t, + processGeneration: String, + windowId: CGWindowID, + expectedBounds: CGRect? = nil + ) -> Result { + guard + let startTime = environment.processStartTime(pid: pid), + hostProcessGeneration(startTime) == processGeneration + else { + return .failure(HostDomainError(.processReplaced)) + } + guard let window = environment.onScreenWindows().first(where: { + $0.pid == pid && $0.windowId == windowId + }) else { + return .failure(HostDomainError(.windowGone)) + } + guard window.appId == appId else { + return .failure(HostDomainError(.windowChanged)) + } + if let expectedBounds, !hostWindowBoundsMatch(window.bounds, expectedBounds) { + return .failure(HostDomainError(.windowChanged)) + } + return .success(window) + } + + private func validateRoot( + _ snapshot: HostSnapshot, + requireFrozenBounds: Bool = true + ) -> Result { + validateRoot( + appId: snapshot.appId, + pid: snapshot.pid, + processGeneration: snapshot.processGeneration, + windowId: snapshot.windowId, + expectedBounds: requireFrozenBounds ? snapshot.windowBounds : nil + ) + } + + private func validateApplicationRoot(_ snapshot: HostSnapshot) -> HostDomainError? { + guard + let startTime = environment.processStartTime(pid: snapshot.pid), + hostProcessGeneration(startTime) == snapshot.processGeneration + else { + return HostDomainError(.processReplaced) + } + guard let app = environment.runningApps().first(where: { + $0.pid == snapshot.pid && $0.running + }) else { + return HostDomainError(.processReplaced) + } + return app.appId == snapshot.appId ? nil : HostDomainError(.windowChanged) + } + /// The one place a snapshot is produced. Both `observe` and every /// `observeAfter` go through it, so a post-dispatch frame is the same shape, /// under the same bounds, as one the host asked for directly. @@ -116,34 +171,30 @@ extension HostProtocolServer { return .failure(HostDomainError(.permissionMissing, detail: .missingPermission(.accessibility))) } - let windows = environment.onScreenWindows() - let resolved: HostWindowInfo + let appId: String + let pid: pid_t + let processGeneration: String + let windowId: CGWindowID switch target { - case .window(let pid, let windowId): - guard let match = windows.first(where: { $0.pid == pid && $0.windowId == windowId }) else { - return .failure(HostDomainError(.windowGone)) - } - resolved = match - case .app(let appId): - // §5.2 — the executor resolves `{ "kind": "app" }`, and it resolves it - // by exact `appId` against what is *already running*. `observe` is a - // read: it never launches and never activates. - let app: HostRunningApp - switch hostResolveAppTarget(appId: appId, in: environment.runningApps()) { - case .success(let match): - app = match - case .failure(let error): - return .failure(error) - } - - // `{ "kind": "app" }` resolves to the app's frontmost usable window - // and is ambiguous by design; `{ "kind": "window" }` is exact. The - // window list is front-to-back, so the first match is the frontmost. - guard let match = windows.first(where: { $0.pid == app.pid && $0.layer == 0 }) else { - return .failure(HostDomainError(.windowGone)) - } - resolved = match + case .window(let expectedAppId, let expectedPid, let expectedGeneration, let expectedWindowId): + appId = expectedAppId + pid = expectedPid + processGeneration = expectedGeneration + windowId = expectedWindowId + } + let resolved: HostWindowInfo + switch validateRoot( + appId: appId, + pid: pid, + processGeneration: processGeneration, + windowId: windowId + ) { + case .success(let window): + resolved = window + case .failure(let error): + return .failure(error) } + let windows = environment.onScreenWindows() var windowElement = environment.windowElement( pid: resolved.pid, @@ -166,7 +217,7 @@ extension HostProtocolServer { return .failure(HostDomainError(.windowGone)) } - guard let startTime = hostProcessStartTime(pid: resolved.pid) else { + guard let startTime = environment.processStartTime(pid: resolved.pid) else { return .failure(HostDomainError(.processReplaced)) } @@ -289,7 +340,9 @@ extension HostProtocolServer { let displays = HostWindowInventory.displays() let previousSnapshot = currentRegistry().latestDifferenceBaseline( session: session, + appId: resolved.appId, pid: resolved.pid, + processGeneration: hostProcessGeneration(startTime), windowId: resolved.windowId ) @@ -326,6 +379,7 @@ extension HostProtocolServer { capturedAt: capturedAt, target: HostWindowTarget( pid: resolved.pid, + processGeneration: hostProcessGeneration(startTime), windowId: resolved.windowId, appId: resolved.appId, appName: resolved.appName, @@ -358,13 +412,28 @@ extension HostProtocolServer { } return .failure(error) case .success(let fit): + if case .failure(let error) = validateRoot( + appId: appId, + pid: pid, + processGeneration: processGeneration, + windowId: windowId, + expectedBounds: resolved.bounds + ) { + if let path = image?.path { + currentImageStore().delete(path: path) + } + return .failure(error) + } let (payload, walkResult, observationRevision) = fit.payload return .success( HostSnapshot( id: snapshotId, session: session, pid: resolved.pid, + appId: resolved.appId, + processGeneration: hostProcessGeneration(startTime), windowId: resolved.windowId, + windowBounds: resolved.bounds, capturedAt: capturedAt, windowDigest: payload.windowDigest, payload: payload, @@ -486,13 +555,28 @@ extension HostProtocolServer { return } - let windows = environment.onScreenWindows() - guard let window = windows.first(where: { $0.pid == snapshot.pid && $0.windowId == snapshot.windowId }) else { - refuse(HostDomainError(.windowGone)) - return + let window: HostWindowInfo? + if binding.isMenu { + if let error = validateApplicationRoot(snapshot) { + refuse(error) + return + } + window = nil + } else { + switch validateRoot(snapshot) { + case .success(let current): + window = current + case .failure(let error): + refuse(error) + return + } } + let windows = environment.onScreenWindows() - let probe = environment.bindingProbe(windowBounds: window.bounds) + let probe = environment.bindingProbe( + windowId: snapshot.windowId, + windowBounds: window?.bounds ?? snapshot.windowBounds + ) var effectiveBinding = binding var promotedToWebContent = false if let failure = hostVerifyBinding(binding, probe: probe) { @@ -575,7 +659,7 @@ extension HostProtocolServer { // §6.1 `strictness: "window"` — the only defence against recycled row // views, at the cost of refusing on any change anywhere in the window. - if params.strictness == .window { + if params.strictness == .window, !binding.isMenu, let window { let current = hostRecomputeWindowDigest(snapshot: snapshot, window: window, probe: probe) guard current == snapshot.windowDigest else { refuse(HostDomainError(.windowChanged)) @@ -590,7 +674,9 @@ extension HostProtocolServer { // Applying the check here would also refuse window management on every // application `apps.launch` started, because those begin at the bottom of // the z-order — the defect §6.1 already had to fix once for `same_app`. - if !params.action.addressesTheWindowItself, let frame = effectiveBinding.observed.frame?.cgRect { + if !params.action.addressesTheWindowItself, + let window, + let frame = effectiveBinding.observed.frame?.cgRect { let center = CGPoint( x: window.bounds.minX + frame.midX, y: window.bounds.minY + frame.midY @@ -636,13 +722,41 @@ extension HostProtocolServer { .filter { $0.pid == snapshot.pid } .map(\.windowId) ) - cancellations.markDispatched(id: id) + let effectGate: EffectGate = { effectElement, requireOriginalDigest in + if effectiveBinding.isMenu { + if let error = self.validateApplicationRoot(snapshot) { + return error + } + } else if case .failure(let error) = self.validateRoot(snapshot) { + return error + } + let bindingFailure = requireOriginalDigest + ? hostVerifyBinding(effectiveBinding, probe: probe) + : hostVerifyBindingTarget(effectiveBinding, probe: probe) + if let failure = bindingFailure { + return failure + } + if !effectiveBinding.isMenu, + !CFEqual(effectElement, element), + ( + HostAX.containingWindow(of: effectElement).flatMap { hostWindowId(of: $0) } + != snapshot.windowId + ) { + return HostDomainError(.windowChanged) + } + if self.cancellations.isCancelledBeforeDispatch(id: id) { + return HostDomainError(.aborted) + } + self.cancellations.markDispatched(id: id) + return nil + } let attempted = performElementAction( params.action, on: element, binding: effectiveBinding, window: window, - settle: settleMode + settle: settleMode, + effectGate: effectGate ) var performed = attempted.outcome == .unknown @@ -690,6 +804,7 @@ extension HostProtocolServer { fallbackVerdict: performed.verdict, settleMode: settleMode, observeAfter: params.observeAfter, + applicationLevel: binding.isMenu, window: window ) } @@ -702,16 +817,20 @@ extension HostProtocolServer { let failure: HostDomainError? } + private typealias EffectGate = (AXUIElement, Bool) -> HostDomainError? + private func performElementAction( _ action: HostElementAction, on element: AXUIElement, binding: HostElementBinding, - window: HostWindowInfo, - settle: HostSettleMode + window: HostWindowInfo?, + settle: HostSettleMode, + effectGate: EffectGate ) -> PerformedAction { switch action { case .click(.left, let count) where binding.dispatchPid != binding.pid: - guard (1...2).contains(count), + guard let window, + (1...2).contains(count), let frame = binding.observed.frame?.cgRect else { return refused(.unsupportedAction) @@ -721,6 +840,9 @@ extension HostProtocolServer { x: window.bounds.minX + windowPoint.x, y: window.bounds.minY + windowPoint.y ) + if let failure = effectGate(element, true) { + return refused(failure) + } do { try environment.postWebContentClick( at: screenPoint, @@ -754,25 +876,53 @@ extension HostProtocolServer { // a coordinate click it did not ask for. return refused(.elementNotActionable) } + if binding.isMenu { + return performAXAction( + required, + on: element, + binding: binding, + repeatCount: count, + settle: settle, + effectGate: effectGate + ) + } + guard let window else { + return refused(.windowGone) + } return performAXActionWithSyntheticFocus( required, on: element, binding: binding, window: window, repeatCount: count, - settle: settle + settle: settle, + effectGate: effectGate ) case .secondaryAction(let name): // §6.5 — `secondary_action` gets `action_result` only. There is // nothing generic to read back. + if binding.isMenu { + return performAXAction( + name, + on: element, + binding: binding, + repeatCount: 1, + settle: .none, + effectGate: effectGate + ) + } + guard let window else { + return refused(.windowGone) + } return performAXActionWithSyntheticFocus( name, on: element, binding: binding, window: window, repeatCount: 1, - settle: .none + settle: .none, + effectGate: effectGate ) case .scroll(let direction, let pages): @@ -794,7 +944,8 @@ extension HostProtocolServer { on: element, binding: binding, repeatCount: Int(pages), - settle: settle + settle: settle, + effectGate: effectGate ) // `path: none` proves the OS refused before one page landed. // Only that side-effect-free arm may try the declared pid-bound @@ -812,17 +963,21 @@ extension HostProtocolServer { binding: binding, repeatCount: Int(pages), settle: settle, - requireAdvertisedAction: false + requireAdvertisedAction: false, + effectGate: effectGate ) } - guard let frame = binding.observed.frame?.cgRect else { + guard let window, let frame = binding.observed.frame?.cgRect else { return refused(.elementNotActionable) } let point = CGPoint( x: window.bounds.minX + frame.midX, y: window.bounds.minY + frame.midY ) + if let failure = effectGate(element, true) { + return refused(failure) + } do { try environment.postPointEvent( .scroll(direction: direction, pages: pages), @@ -876,11 +1031,15 @@ extension HostProtocolServer { requestedNumber: requestedNumber, direction: direction, opposite: opposite, - on: element + on: element, + effectGate: effectGate ) } } + if let failure = effectGate(element, true) { + return refused(failure) + } let result = AXUIElementSetAttributeValue( element, kAXValueAttribute as CFString, @@ -923,6 +1082,9 @@ extension HostProtocolServer { return refused(.unsupportedAction) } + if let failure = effectGate(element, true) { + return refused(failure) + } let result = AXUIElementSetAttributeValue(element, kAXSelectedTextRangeAttribute as CFString, axRange) guard result == .success else { return failed(.dispatchRefused, path: .none) @@ -940,7 +1102,16 @@ extension HostProtocolServer { ) case .moveWindow, .resizeWindow, .minimizeWindow: - return performWindowAction(action, on: element, binding: binding, window: window) + guard let window else { + return refused(.windowGone) + } + return performWindowAction( + action, + on: element, + binding: binding, + window: window, + effectGate: effectGate + ) } } @@ -958,13 +1129,19 @@ extension HostProtocolServer { requestedNumber: Double, direction: HostElementActionName, opposite: HostElementActionName, - on element: AXUIElement + on element: AXUIElement, + effectGate: EffectGate ) -> PerformedAction { - guard AXUIElementPerformAction(element, direction.rawAXAction as CFString) == .success, - let firstReadback = HostAX.stringLikeValue(element, kAXValueAttribute), + if let failure = effectGate(element, true) { + return refused(failure) + } + guard environment.performAccessibilityAction(direction, on: element) == .success else { + return failed(.dispatchRefused, path: .axAction) + } + guard let firstReadback = HostAX.stringLikeValue(element, kAXValueAttribute), let firstNumber = Double(firstReadback) else { - return failed(.dispatchRefused, path: .axAction) + return unknownOutcome() } let step = firstNumber - currentNumber @@ -979,7 +1156,10 @@ extension HostProtocolServer { && roundedRemaining <= 99 guard reachesTarget else { - let reversed = AXUIElementPerformAction(element, opposite.rawAXAction as CFString) + if effectGate(element, false) != nil { + return unknownOutcome() + } + let reversed = environment.performAccessibilityAction(opposite, on: element) let restored = HostAX.stringLikeValue(element, kAXValueAttribute) if reversed != .success || !hostNumericStringsEqual(restored, previous) { return unknownOutcome() @@ -988,7 +1168,10 @@ extension HostProtocolServer { } for _ in 0.. PerformedAction { // §5.3 — the snapshot's tree is rooted at the window, so `depth == 0` is // the window and nothing else is. @@ -1063,6 +1247,9 @@ extension HostProtocolServer { } let previous = subject.read(element) + if let failure = effectGate(element, true) { + return refused(failure) + } guard subject.write(element) == .success else { return failed(.dispatchRefused, path: .none) } @@ -1129,7 +1316,9 @@ extension HostProtocolServer { binding: HostElementBinding, repeatCount: Int, settle: HostSettleMode, - requireAdvertisedAction: Bool = true + requireAdvertisedAction: Bool = true, + firstEffectRequiresOriginalDigest: Bool = true, + effectGate: EffectGate ) -> PerformedAction { guard !requireAdvertisedAction || binding.observed.actions.contains(name) else { return refused(.elementNotActionable) @@ -1137,7 +1326,10 @@ extension HostProtocolServer { var delivered = 0 for _ in 0.. PerformedAction { + if let failure = effectGate(element, true) { + return refused(failure) + } let context = environment.beginSyntheticTargetFocus( pid: binding.pid, windowId: window.windowId @@ -1200,7 +1396,9 @@ extension HostProtocolServer { on: element, binding: binding, repeatCount: repeatCount, - settle: settle + settle: settle, + firstEffectRequiresOriginalDigest: false, + effectGate: effectGate ) guard let context else { return result @@ -1217,12 +1415,16 @@ extension HostProtocolServer { /// `path: none`; the tier travels anyway, because it is the tier the executor /// would have used. private func refused(_ code: HostDomainErrorCode) -> PerformedAction { + refused(HostDomainError(code)) + } + + private func refused(_ error: HostDomainError) -> PerformedAction { PerformedAction( outcome: .refused, path: .none, tier: .ax, verdict: hostEffectNotChecked(), - failure: HostDomainError(code) + failure: error ) } @@ -1322,18 +1524,72 @@ extension HostProtocolServer { fallbackVerdict: HostEffectVerdict, settleMode: HostSettleMode, observeAfter: HostObserveAfter?, - window: HostWindowInfo + applicationLevel: Bool, + window: HostWindowInfo? ) { let digestBefore = snapshot.windowDigest currentRegistry().spend(snapshot) + func emitRootReplacement(_ error: HostDomainError) { + emit( + id: id, + payload: HostDispatchResult( + toolCallId: toolCallId, + outcome: outcome, + tier: path.tier ?? .ax, + path: path, + effect: fallbackVerdict.effect, + verification: fallbackVerdict.verification, + settle: HostSettleReport( + waitedMs: 0, + quiesced: false, + reason: .notRequested + ), + snapshot: nil, + postObservationError: HostDomainErrorPayload(error) + ) + ) + } + + func rootFailure() -> HostDomainError? { + if applicationLevel { + return validateApplicationRoot(snapshot) + } + guard window != nil else { + return HostDomainError(.windowGone) + } + if case .failure(let error) = validateRoot(snapshot, requireFrozenBounds: false) { + return error + } + return nil + } + + if let error = rootFailure() { + emitRootReplacement(error) + return + } + var settleReport = HostSettleReport(waitedMs: 0, quiesced: false, reason: .notRequested) var digestAfter: String? - if settleMode == .quiesce { - let settled = quiesce(snapshot: snapshot, window: window) - settleReport = settled.report - digestAfter = settled.digest + if settleMode == .quiesce, !applicationLevel { + // A move or resize intentionally changes the window bounds. Settle + // the same frozen window identity at its current geometry rather + // than handing the pre-effect HostWindowInfo back to the probe. + switch validateRoot(snapshot, requireFrozenBounds: false) { + case .success(let currentWindow): + let settled = quiesce(snapshot: snapshot, window: currentWindow) + settleReport = settled.report + digestAfter = settled.digest + case .failure(let error): + emitRootReplacement(error) + return + } + } + + if let error = rootFailure() { + emitRootReplacement(error) + return } let verdict = verificationIsTreeDelta @@ -1351,7 +1607,12 @@ extension HostProtocolServer { if let observeAfter { switch buildSnapshot( session: snapshot.session, - target: .window(pid: snapshot.pid, windowId: snapshot.windowId), + target: .window( + appId: snapshot.appId, + pid: snapshot.pid, + processGeneration: snapshot.processGeneration, + windowId: snapshot.windowId + ), includeImage: observeAfter.includeImage, menuScope: observeAfter.menu, maxElements: limits.maxElements, @@ -1394,11 +1655,24 @@ extension HostProtocolServer { snapshot: HostSnapshot, window: HostWindowInfo ) -> (report: HostSettleReport, digest: String) { - let probe = environment.bindingProbe(windowBounds: window.bounds) + let probe = environment.bindingProbe(windowId: snapshot.windowId, windowBounds: window.bounds) return hostSettle( ceilingMs: limits.settleCeilingMs, pollMs: hostSettlePollMs, - sample: { hostRecomputeWindowDigest(snapshot: snapshot, window: window, probe: probe) } + sample: { + guard case .success = self.validateRoot(snapshot, requireFrozenBounds: false) else { + return "root-invalid-before-read" + } + let digest = hostRecomputeWindowDigest( + snapshot: snapshot, + window: window, + probe: probe + ) + guard case .success = self.validateRoot(snapshot, requireFrozenBounds: false) else { + return "root-invalid-after-read" + } + return digest + } ) } } @@ -1597,13 +1871,20 @@ extension HostProtocolServer { return } - let windows = environment.onScreenWindows() - guard let window = windows.first(where: { $0.pid == snapshot.pid && $0.windowId == snapshot.windowId }) else { - refuse(HostDomainError(.windowGone)) + let window: HostWindowInfo + switch validateRoot(snapshot) { + case .success(let current): + window = current + case .failure(let error): + refuse(error) return } - if let failure = hostVerifyBinding(binding, probe: environment.bindingProbe(windowBounds: window.bounds)) { + let probe = environment.bindingProbe(windowId: snapshot.windowId, windowBounds: window.bounds) + if let failure = hostVerifyBinding( + binding, + probe: probe + ) { refuse(failure) return } @@ -1622,9 +1903,19 @@ extension HostProtocolServer { } var focused = environment.focusedElement(pid: snapshot.pid) + var acquiredFocus = false if (params.focusPolicy ?? .require) == .acquire, !(focused.map { CFEqual($0, element) } ?? false) { + if case .failure(let error) = validateRoot(snapshot) { + refuse(error) + return + } + if let failure = hostVerifyBinding(binding, probe: probe) { + refuse(failure) + return + } + cancellations.markDispatched(id: id) guard environment.setFocusedElement(element, pid: snapshot.pid) else { // No fallback: an element that refused focus is not an element to // post keys at and hope. §6.4 — the code is the same @@ -1633,6 +1924,7 @@ extension HostProtocolServer { refuse(HostDomainError(.focusChanged)) return } + acquiredFocus = true // The write's own success is not proof. Applications accept // `kAXFocused` and leave focus where it was, so the only evidence @@ -1655,6 +1947,17 @@ extension HostProtocolServer { ? HostAX.stringLikeValue(element, kAXValueAttribute) : nil let settleMode = params.observeAfter?.settle ?? HostSettleMode.none + if case .failure(let error) = validateRoot(snapshot) { + refuse(error) + return + } + let bindingFailure = acquiredFocus + ? hostVerifyBindingTarget(binding, probe: probe) + : hostVerifyBinding(binding, probe: probe) + if let failure = bindingFailure { + refuse(failure) + return + } cancellations.markDispatched(id: id) do { @@ -1708,6 +2011,7 @@ extension HostProtocolServer { fallbackVerdict: verdict, settleMode: settleMode, observeAfter: params.observeAfter, + applicationLevel: false, window: window ) } diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer.swift index caaae7e..e4d36b3 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolServer.swift @@ -26,25 +26,79 @@ struct HostSessionParams: Decodable { let session: String } -enum HostTargetSelector: Decodable, Equatable { - case app(String) - case window(pid: pid_t, windowId: CGWindowID) +enum HostTargetResolutionSelector: Decodable { + case application(app: String, intent: Intent) + case window(windowId: CGWindowID) + + enum Intent: String, Decodable { + case operate + case launch + } private enum Key: String, CodingKey { case kind case app + case intent + case windowId + } + + init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: Key.self) + switch try container.decode(String.self, forKey: .kind) { + case "application": + self = .application( + app: try container.decode(String.self, forKey: .app), + intent: try container.decode(Intent.self, forKey: .intent) + ) + case "window": + self = .window( + windowId: CGWindowID(try container.decode(UInt32.self, forKey: .windowId)) + ) + default: + throw DecodingError.dataCorruptedError( + forKey: .kind, + in: container, + debugDescription: "target.kind" + ) + } + } +} + +struct HostTargetResolveParams: Decodable { + let target: HostTargetResolutionSelector +} + +enum HostTargetSelector: Decodable, Equatable { + case window(appId: String, pid: pid_t, processGeneration: String, windowId: CGWindowID) + + private enum Key: String, CodingKey { + case kind + case appId case pid + case processGeneration case windowId } init(from decoder: Decoder) throws { let container = try decoder.container(keyedBy: Key.self) switch try container.decode(String.self, forKey: .kind) { - case "app": - self = .app(try container.decode(String.self, forKey: .app)) case "window": + let processGeneration = try container.decode(String.self, forKey: .processGeneration) + guard + processGeneration.hasPrefix("pst:"), + let startTime = UInt64(processGeneration.dropFirst(4)), + hostProcessGeneration(startTime) == processGeneration + else { + throw DecodingError.dataCorruptedError( + forKey: .processGeneration, + in: container, + debugDescription: "target.processGeneration" + ) + } self = .window( + appId: try container.decode(String.self, forKey: .appId), pid: try container.decode(Int32.self, forKey: .pid), + processGeneration: processGeneration, windowId: CGWindowID(try container.decode(UInt32.self, forKey: .windowId)) ) default: @@ -396,6 +450,30 @@ struct HostAppsListResult: Encodable { } } +struct HostTargetResolveResult: Encodable { + enum Resolution: String, Encodable { + case resolved + case missing + case ambiguous + } + + struct Target: Encodable { + enum Kind: String, Encodable { + case installed + case running + } + + let kind: Kind + let appId: String + let pid: Int32? + let processGeneration: String? + let windowId: UInt32? + } + + let resolution: Resolution + let target: Target? +} + struct HostPermissionsResult: Encodable { let accessibility: Bool let screenRecording: Bool @@ -453,7 +531,7 @@ struct HostScreenCaptureResult: Encodable { // MARK: - Server -/// The `maka.cu/2` executor. One reader, serial lanes, one response per request +/// The `maka.cu/3` executor. One reader, serial lanes, one response per request /// id. Everything the host needs to reason about is a declared field; nothing is /// inferred from a message string on either side. public final class HostProtocolServer { @@ -591,6 +669,8 @@ public final class HostProtocolServer { } ifShuttingDown: { [weak self] in self?.emit(id: id, failure: HostDomainError(.aborted)) } + case "target.resolve": + enqueueControl(id: id, data: data, handler: handleTargetResolve) case "window.list": lanes.enqueue(.control) { [weak self] in self?.handleWindowList(id: id) @@ -692,7 +772,8 @@ public final class HostProtocolServer { } private func targetLane(for params: Params) -> HostLaneScheduler.Lane { - if let observe = params as? HostObserveParams, case let .window(pid, windowId) = observe.target { + if let observe = params as? HostObserveParams, + case let .window(_, pid, _, windowId) = observe.target { return .target(pid: pid, windowId: windowId) } @@ -1028,4 +1109,92 @@ public final class HostProtocolServer { emit(id: id, payload: HostAppsListResult(apps: apps)) } + + private func handleTargetResolve(id: Int, params: HostTargetResolveParams) { + switch params.target { + case .application(let app, .launch): + switch environment.installedBundleIdentifiers(matching: app) { + case .failure(let error): + emit(id: id, failure: error) + case .success(let bundleIds): + let unique = Array(Set(bundleIds)).sorted() + guard unique.count == 1, let bundleId = unique.first else { + emit( + id: id, + payload: HostTargetResolveResult( + resolution: unique.isEmpty ? .missing : .ambiguous, + target: nil + ) + ) + return + } + emit( + id: id, + payload: HostTargetResolveResult( + resolution: .resolved, + target: .init( + kind: .installed, + appId: bundleId, + pid: nil, + processGeneration: nil, + windowId: nil + ) + ) + ) + } + + case .application(let app, .operate): + let running = environment.runningApps().filter(\.running) + let exact = running.filter { $0.appId == app } + let matchingApps = exact.isEmpty + ? running.filter { $0.name.caseInsensitiveCompare(app) == .orderedSame } + : exact + let matchingPids = Set(matchingApps.map(\.pid)) + resolveRunningTarget( + id: id, + windows: environment.onScreenWindows().filter { + matchingPids.contains($0.pid) && $0.layer == 0 && $0.onScreen + } + ) + + case .window(let windowId): + resolveRunningTarget( + id: id, + windows: environment.onScreenWindows().filter { $0.windowId == windowId } + ) + } + } + + private func resolveRunningTarget(id: Int, windows: [HostWindowInfo]) { + guard windows.count == 1, let window = windows.first else { + emit( + id: id, + payload: HostTargetResolveResult( + resolution: windows.isEmpty ? .missing : .ambiguous, + target: nil + ) + ) + return + } + guard let startTime = environment.processStartTime(pid: window.pid) else { + emit( + id: id, + payload: HostTargetResolveResult(resolution: .missing, target: nil) + ) + return + } + emit( + id: id, + payload: HostTargetResolveResult( + resolution: .resolved, + target: .init( + kind: .running, + appId: window.appId, + pid: window.pid, + processGeneration: hostProcessGeneration(startTime), + windowId: window.windowId + ) + ) + ) + } } diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolWire.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolWire.swift index ea5e8d9..1e30133 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolWire.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostProtocolWire.swift @@ -1,13 +1,13 @@ import Foundation -/// Wire vocabulary for `maka.cu/2`, the protocol spoken between the Maka Electron +/// Wire vocabulary for `maka.cu/3`, the protocol spoken between the Maka Electron /// host and this executor. See `docs/HOST_PROTOCOL.md`. /// /// Every enum here is a closed set on purpose. The protocol exists because the /// previous surface answered with free-form prose that the host had to pattern /// match; nothing in this file may grow an "other" case. -public let makaCuProtocolVersion = "maka.cu/2" +public let makaCuProtocolVersion = "maka.cu/3" /// `EX_CONFIG`. §2 requires this exact status after a version mismatch so the /// host can classify the start as `service_mismatch` and refuse to retry. @@ -232,6 +232,12 @@ public func hostAppId(bundleIdentifier: String?, pid: pid_t) -> String { return bundleIdentifier } +/// A monotonic process identity used to prevent PID reuse from changing an +/// already approved target. +public func hostProcessGeneration(_ startTime: UInt64) -> String { + "pst:\(startTime)" +} + // MARK: - Dispatch vocabulary /// §6.5. `tier` uses Maka's exact `COMPUTER_USE_DISPATCH_TIERS` vocabulary so the diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSnapshotRegistry.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSnapshotRegistry.swift index 981bd13..c410952 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSnapshotRegistry.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSnapshotRegistry.swift @@ -34,7 +34,10 @@ public final class HostSnapshot { public let id: String public let session: String public let pid: pid_t + public let appId: String + public let processGeneration: String public let windowId: CGWindowID + public let windowBounds: CGRect public let capturedAt: Int64 public let windowDigest: String public let payload: HostSnapshotPayload @@ -49,7 +52,10 @@ public final class HostSnapshot { id: String, session: String, pid: pid_t, + appId: String, + processGeneration: String, windowId: CGWindowID, + windowBounds: CGRect, capturedAt: Int64, windowDigest: String, payload: HostSnapshotPayload, @@ -60,7 +66,10 @@ public final class HostSnapshot { self.id = id self.session = session self.pid = pid + self.appId = appId + self.processGeneration = processGeneration self.windowId = windowId + self.windowBounds = windowBounds self.capturedAt = capturedAt self.windowDigest = windowDigest self.payload = payload @@ -301,13 +310,17 @@ public final class HostSnapshotRegistry { func latestDifferenceBaseline( session: String, + appId: String, pid: pid_t, + processGeneration: String, windowId: CGWindowID ) -> HostSnapshot? { lock.lock() defer { lock.unlock() } return sessions[session]?.snapshots.last { - $0.pid == pid + $0.appId == appId + && $0.pid == pid + && $0.processGeneration == processGeneration && $0.windowId == windowId && $0.state != .expired && $0.state != .evicted diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSystemEnvironment.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSystemEnvironment.swift index aaab474..771aa54 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSystemEnvironment.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/HostProtocol/HostSystemEnvironment.swift @@ -33,6 +33,11 @@ public protocol HostSystemEnvironment { func screenIsLocked() -> Bool func permissions() -> PermissionDiagnostics func runningApps() -> [HostRunningApp] + /// Bundle identifiers matching an installed application, without launching + /// it or changing foreground state. + func installedBundleIdentifiers(matching query: String) -> Result<[String], HostDomainError> + /// Stable process birth identity used to bind an approved running target. + func processStartTime(pid: pid_t) -> UInt64? /// The pid holding the foreground, or `nil` when nothing ordinary does. /// /// It is behind the seam because `apps.launch` reports `foregroundTaken` by @@ -82,7 +87,13 @@ public protocol HostSystemEnvironment { /// an application may return success and leave focus where it was, so the /// caller re-reads `focusedElement(pid:)` before posting anything. func setFocusedElement(_ element: AXUIElement, pid: pid_t) -> Bool - func bindingProbe(windowBounds: CGRect) -> HostElementBindingProbe + func bindingProbe(windowId: CGWindowID, windowBounds: CGRect) -> HostElementBindingProbe + /// Performs one Accessibility action after the protocol handler has + /// revalidated the exact approved target for this individual effect. + func performAccessibilityAction( + _ action: HostElementActionName, + on element: AXUIElement + ) -> AXError /// Posts an executor-derived, PID-bound event for an already bound semantic /// element action. Model-provided point dispatch never reaches this seam. func postPointEvent( @@ -129,6 +140,20 @@ public struct HostLiveEnvironment: HostSystemEnvironment { } } + public func installedBundleIdentifiers( + matching query: String + ) -> Result<[String], HostDomainError> { + do { + return .success(try AppDiscovery.matchingInstalledBundleIdentifiers(query)) + } catch { + return .failure(hostAppLaunchFailure(error)) + } + } + + public func processStartTime(pid: pid_t) -> UInt64? { + hostProcessStartTime(pid: pid) + } + public func frontmostApplicationPid() -> pid_t? { LiveApplicationInventory.frontmostApplicationPid() } @@ -214,8 +239,15 @@ public struct HostLiveEnvironment: HostSystemEnvironment { AXUIElementSetAttributeValue(element, kAXFocusedAttribute as CFString, kCFBooleanTrue) == .success } - public func bindingProbe(windowBounds: CGRect) -> HostElementBindingProbe { - HostAXBindingProbe(windowBounds: windowBounds) + public func bindingProbe(windowId: CGWindowID, windowBounds: CGRect) -> HostElementBindingProbe { + HostAXBindingProbe(windowId: windowId, windowBounds: windowBounds) + } + + public func performAccessibilityAction( + _ action: HostElementActionName, + on element: AXUIElement + ) -> AXError { + AXUIElementPerformAction(element, action.rawAXAction as CFString) } public func postPointEvent( @@ -345,14 +377,14 @@ public func hostAppLaunchFailure(_ error: Error) -> HostDomainError { } } -// MARK: - Resolving `{ "kind": "app" }` +// MARK: - Resolving a running app identity /// §5.1 / §5.2 — the executor resolves an `appId` by exact string match, against /// applications that are **already running**, and refuses everything else. /// /// Two rules are load-bearing and neither is defensive: /// -/// - No launch. `observe` is a read. The previous implementation went through +/// - No launch. `target.resolve` is a read. The previous implementation went through /// `AppDiscovery.resolve`, which falls through to `NSWorkspace.openApplication` /// with a configuration that activates, and then polls for five seconds — so /// observing a not-running app started it and took the user's foreground, with diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift index 58d9b49..ca65da8 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift @@ -319,7 +319,7 @@ enum InputSimulation { /// The CLI surface's key press, which parses an xdotool-flavoured string and /// posts the event the key code alone produces. /// - /// **Not the `maka.cu/2` path.** `dispatch.key` goes through + /// **Not the `maka.cu/3` path.** `dispatch.key` goes through /// `pressKeyStroke` and the closed table above, because a key posted this /// way carries the keyboard layout's translation of the key code — which is /// the wrong character for 22 of the wire's 26 named keys — and does not diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift index 956fb93..5bde97c 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift @@ -2,7 +2,7 @@ import Foundation public enum OpenComputerUseCLICommand: Equatable { case launchOnboarding - /// Speak `maka.cu/2` over stdio. This is the only automation entry point: + /// Speak `maka.cu/3` over stdio. This is the only automation entry point: /// the Maka host owns every model-facing word, so there is no second, /// model-shaped surface here to drift from it. case host @@ -84,7 +84,7 @@ public func openComputerUseHelpText(command: String? = nil) -> String { open-computer-use Commands: - host Speak the maka.cu/2 host protocol over stdio. + host Speak the maka.cu/3 host protocol over stdio. doctor Print runtime diagnostics and launch onboarding if needed. list-apps Print running or recently used apps. snapshot Print the current accessibility snapshot for an app. @@ -105,7 +105,7 @@ public func openComputerUseHelpText(command: String? = nil) -> String { Usage: open-computer-use host - Speak the maka.cu/2 host protocol over stdio. line-delimited JSON-RPC 2.0, + Speak the maka.cu/3 host protocol over stdio. line-delimited JSON-RPC 2.0, one JSON value per line. The Maka host drives it; it is not interactive. See docs/maka-cu-host-protocol.md in the Maka repository. """ diff --git a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift index a5c4edd..b27c2e6 100644 --- a/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift +++ b/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseVersion.swift @@ -11,7 +11,7 @@ public func resolvedOpenComputerUseVersion(bundle: Bundle = .main) -> String { return openComputerUseVersion } -/// Reported in the `maka.cu/2` handshake so a host trace can name the exact +/// Reported in the `maka.cu/3` handshake so a host trace can name the exact /// executor build. Populated at package time through the app bundle; a source /// checkout has no commit to claim, and saying `unknown` is better than a value /// the host would trust. diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostDispatchTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostDispatchTests.swift index 717fe73..ee8e9d7 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostDispatchTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostDispatchTests.swift @@ -102,6 +102,114 @@ final class HostDispatchTests: XCTestCase { ) } + func testElementReparentedToAnotherWindowIsRefusedBeforeAnyEffect() throws { + var environment = FakeEnvironment() + var probe = FakeBindingProbe() + probe.belongsToTarget = false + environment.probe = probe + environment.windows = [hostTestWindow()] + + let harness = ServerHarness(environment: environment) + try harness.begin() + let snapshot = hostTestSnapshot(registry: harness.server.currentRegistry(), session: "s1") + harness.install(snapshot) + + harness.send(dispatchElement(snapshot: snapshot)) + + XCTAssertEqual(try errorCode(harness.awaitResult()), "window_changed") + XCTAssertTrue(harness.environment.pointEvents.posted.isEmpty) + } + + func testRepeatedActionStopsWhenTheApprovedTargetChangesAfterTheFirstEffect() throws { + var environment = FakeEnvironment() + environment.windows = [hostTestWindow()] + environment.probe = ActionReactiveBindingProbe(actions: environment.accessibilityActions) + + let harness = ServerHarness(environment: environment) + try harness.begin() + let snapshot = hostTestSnapshot( + registry: harness.server.currentRegistry(), + session: "s1", + element: hostTestElement(), + elementActions: [.press] + ) + harness.install(snapshot) + + harness.send( + dispatchElement( + snapshot: snapshot, + action: #"{"kind":"click","button":"left","count":2}"# + ) + ) + let result = try harness.awaitResult() + + XCTAssertEqual(result["ok"] as? Bool, false) + XCTAssertEqual(result["outcome"] as? String, "unknown") + XCTAssertEqual(try errorCode(result), "outcome_unknown") + XCTAssertEqual(harness.environment.accessibilityActions.performed, [.press]) + XCTAssertEqual( + harness.server.currentRegistry().snapshotState(session: "s1", snapshotId: snapshot.id), + .spent, + "the first effect landed before the target changed" + ) + } + + func testPostObservationRootLossDoesNotEraseAKnownSuccessfulEffect() throws { + let harness = ServerHarness() + try harness.begin() + let snapshot = hostTestSnapshot(registry: harness.server.currentRegistry(), session: "s1") + harness.install(snapshot) + + harness.server.finishDispatch( + id: 81, + toolCallId: "call_1", + snapshot: snapshot, + outcome: .ok, + path: .axAttribute, + verificationIsTreeDelta: false, + fallbackVerdict: HostEffectVerdict( + effect: .confirmed, + verification: HostVerification(method: .valueReadback, observedChange: true) + ), + settleMode: .none, + observeAfter: HostObserveAfter(includeImage: false, settle: .none, menu: nil), + applicationLevel: false, + window: hostTestWindow() + ) + let result = try harness.awaitResult() + + XCTAssertEqual(result["ok"] as? Bool, true) + XCTAssertEqual(result["outcome"] as? String, "ok") + XCTAssertEqual(result["path"] as? String, "ax_attribute") + XCTAssertEqual(result["effect"] as? String, "confirmed") + XCTAssertEqual( + (result["postObservationError"] as? [String: Any])?["code"] as? String, + "window_gone" + ) + } + + func testMenuBindingUsesApplicationIdentityWhenItsSnapshotWindowIsGone() throws { + var environment = FakeEnvironment() + environment.apps = [ + HostRunningApp(appId: hostTestAppId, pid: hostTestPid, name: "Notes", running: true), + ] + environment.windows = [] + + let harness = ServerHarness(environment: environment) + try harness.begin() + let snapshot = hostTestSnapshot( + registry: harness.server.currentRegistry(), + session: "s1", + enabled: false, + isMenu: true + ) + harness.install(snapshot) + + harness.send(dispatchElement(snapshot: snapshot)) + + XCTAssertEqual(try errorCode(harness.awaitResult()), "element_disabled") + } + // MARK: - Element binding (§4.3, §6.2) func testTheEchoedDigestSeparatesAnUnknownTokenFromAMismatchedEcho() throws { diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostKeyDeliveryLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostKeyDeliveryLiveTests.swift index ce86a15..d13cf96 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostKeyDeliveryLiveTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostKeyDeliveryLiveTests.swift @@ -352,7 +352,7 @@ final class HostKeyDeliveryLiveTests: XCTestCase { window: HostWindowInfo ) throws -> [String: Any] { server.handle(line: #""" - {"jsonrpc":"2.0","id":\#(id),"method":"observe","params":{"session":"s1","target":{"kind":"window","pid":\#(window.pid),"windowId":\#(window.windowId)},"includeImage":false}} + {"jsonrpc":"2.0","id":\#(id),"method":"observe","params":{"session":"s1","target":{"kind":"window","appId":"\#(window.appId)","pid":\#(window.pid),"processGeneration":"\#(hostProcessGeneration(hostProcessStartTime(pid: window.pid) ?? 0))","windowId":\#(window.windowId)},"includeImage":false}} """#) let response = try XCTUnwrap(try inbox.next(timeout: 30)["result"] as? [String: Any]) guard response["ok"] as? Bool == true else { diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostKeyDispatchLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostKeyDispatchLiveTests.swift index c75dcaf..f16a27a 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostKeyDispatchLiveTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostKeyDispatchLiveTests.swift @@ -294,7 +294,7 @@ final class HostKeyDispatchLiveTests: XCTestCase { window: HostWindowInfo ) throws -> [String: Any] { server.handle(line: #""" - {"jsonrpc":"2.0","id":\#(id),"method":"observe","params":{"session":"s1","target":{"kind":"window","pid":\#(window.pid),"windowId":\#(window.windowId)},"includeImage":false}} + {"jsonrpc":"2.0","id":\#(id),"method":"observe","params":{"session":"s1","target":{"kind":"window","appId":"\#(window.appId)","pid":\#(window.pid),"processGeneration":"\#(hostProcessGeneration(hostProcessStartTime(pid: window.pid) ?? 0))","windowId":\#(window.windowId)},"includeImage":false}} """#) let response = try XCTUnwrap(try inbox.next(timeout: 30)["result"] as? [String: Any]) guard response["ok"] as? Bool == true else { diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObservationTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObservationTests.swift index 5c50ee5..761761e 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObservationTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObservationTests.swift @@ -437,6 +437,7 @@ final class HostMenuObservationTests: XCTestCase { HostRunningApp(appId: hostTestAppId, pid: livePid, name: "Notes", running: true), ] environment.windows = [hostTestWindow(pid: livePid)] + environment.processStartTimes[livePid] = hostProcessStartTime(pid: livePid) // §4.3 E2 — these snapshots are minted by the real `buildSnapshot`, so // their bindings record the *machine's* start time for `livePid`. The // fixture's constant would be a different number and every dispatch below @@ -504,7 +505,7 @@ final class HostMenuObservationTests: XCTestCase { let field = menu.map { ",\"menu\":\($0)" } ?? "" return """ {"jsonrpc":"2.0","id":\(id),"method":"observe","params":{"session":"s1",\ - "target":{"kind":"window","pid":\(livePid),"windowId":1},"includeImage":false\(field)}} + "target":{"kind":"window","appId":"\(hostTestAppId)","pid":\(livePid),"processGeneration":"\(hostProcessGeneration(hostProcessStartTime(pid: livePid) ?? 0))","windowId":1},"includeImage":false\(field)}} """ } diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObserveLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObserveLiveTests.swift index 35ed826..f21050c 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObserveLiveTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostMenuObserveLiveTests.swift @@ -82,7 +82,7 @@ final class HostMenuObserveLiveTests: XCTestCase { let started = Date() server.handle(line: #""" - {"jsonrpc":"2.0","id":3,"method":"observe","params":{"session":"s1","target":{"kind":"window","pid":\#(window.pid),"windowId":\#(window.windowId)},"includeImage":false,"menu":{"scope":"all"}}} + {"jsonrpc":"2.0","id":3,"method":"observe","params":{"session":"s1","target":{"kind":"window","appId":"\#(window.appId)","pid":\#(window.pid),"processGeneration":"\#(hostProcessGeneration(hostProcessStartTime(pid: window.pid) ?? 0))","windowId":\#(window.windowId)},"includeImage":false,"menu":{"scope":"all"}}} """#) let observed = try XCTUnwrap(try inbox.next(timeout: 30)["result"] as? [String: Any]) let observeMs = Date().timeIntervalSince(started) * 1000 @@ -166,7 +166,7 @@ final class HostMenuObserveLiveTests: XCTestCase { ), isMenu: true ) - let probe = HostAXBindingProbe(windowBounds: .zero) + let probe = HostAXBindingProbe(windowId: window.windowId, windowBounds: .zero) var disagreed: [String] = [] for binding in walk.bindings { guard let now = probe.currentDigestInput(binding) else { continue } @@ -213,6 +213,7 @@ final class HostMenuObserveLiveTests: XCTestCase { server: server, inbox: inbox, id: 4, + appId: window.appId, pid: window.pid, windowId: window.windowId, titles: basic, @@ -222,6 +223,7 @@ final class HostMenuObserveLiveTests: XCTestCase { server: server, inbox: inbox, id: 6, + appId: window.appId, pid: window.pid, windowId: window.windowId, titles: scientific, @@ -252,6 +254,7 @@ final class HostMenuObserveLiveTests: XCTestCase { server: server, inbox: inbox, id: 8, + appId: window.appId, pid: window.pid, windowId: window.windowId, titles: sizeAtStart == afterBasic ? basic : scientific, @@ -327,7 +330,7 @@ final class HostMenuObserveLiveTests: XCTestCase { _ = try inbox.next() server.handle(line: #""" - {"jsonrpc":"2.0","id":3,"method":"observe","params":{"session":"s1","target":{"kind":"window","pid":\#(window.pid),"windowId":\#(window.windowId)},"includeImage":false}} + {"jsonrpc":"2.0","id":3,"method":"observe","params":{"session":"s1","target":{"kind":"window","appId":"\#(window.appId)","pid":\#(window.pid),"processGeneration":"\#(hostProcessGeneration(hostProcessStartTime(pid: window.pid) ?? 0))","windowId":\#(window.windowId)},"includeImage":false}} """#) let observed = try XCTUnwrap(try inbox.next(timeout: 30)["result"] as? [String: Any]) let snapshot = try XCTUnwrap(observed["snapshot"] as? [String: Any]) @@ -369,13 +372,14 @@ final class HostMenuObserveLiveTests: XCTestCase { server: HostProtocolServer, inbox: ResponseInbox, id: Int, + appId: String, pid: pid_t, windowId: CGWindowID, titles: [String], size: () -> CGSize? ) throws -> CGSize { server.handle(line: #""" - {"jsonrpc":"2.0","id":\#(id),"method":"observe","params":{"session":"s1","target":{"kind":"window","pid":\#(pid),"windowId":\#(windowId)},"includeImage":false,"menu":{"scope":"all"}}} + {"jsonrpc":"2.0","id":\#(id),"method":"observe","params":{"session":"s1","target":{"kind":"window","appId":"\#(appId)","pid":\#(pid),"processGeneration":"\#(hostProcessGeneration(hostProcessStartTime(pid: pid) ?? 0))","windowId":\#(windowId)},"includeImage":false,"menu":{"scope":"all"}}} """#) let observed = try XCTUnwrap(try inbox.next(timeout: 30)["result"] as? [String: Any]) let snapshot = try XCTUnwrap(observed["snapshot"] as? [String: Any]) diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveContractTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveContractTests.swift index de98dcf..d6897cd 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveContractTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveContractTests.swift @@ -7,124 +7,36 @@ import XCTest final class HostObserveContractTests: XCTestCase { // MARK: - `observe` is a read (§5.2) - func testObserveResolvesAnAppFromWhatIsAlreadyRunningAndNeverLaunchesIt() throws { - // The executor used to resolve `{ "kind": "app" }` through - // `AppDiscovery.resolve`, which falls through to - // `NSWorkspace.openApplication` — with a configuration that activates — - // and then polls for five seconds. Observing something started it and - // took the user's foreground, with nothing on the wire saying so. The - // environment here cannot launch anything at all, and resolution answers - // from its inventory. - var environment = FakeEnvironment() - environment.apps = [ - HostRunningApp(appId: hostTestAppId, pid: hostTestPid, name: "Notes", running: true), - ] - environment.windows = [hostTestWindow()] - - let harness = ServerHarness(environment: environment) - try harness.begin() - - harness.send(observe(app: hostTestAppId)) - let resolved = try harness.awaitResult() - // Resolution succeeded: the refusal that follows is about the window's - // Accessibility element, which no fake can produce. - XCTAssertEqual(try errorCode(resolved), "window_gone") - - harness.send(observe(app: "com.example.not-running")) - XCTAssertEqual(try errorCode(harness.awaitResult()), "app_not_found") - } - - func testObserveWithADisplayNameIsAppNotFoundRatherThanAGuess() throws { - // Vector 32 — `appName` is untrusted, localised display text and two apps - // may share one, so it is never a key. - var environment = FakeEnvironment() - environment.apps = [ - HostRunningApp(appId: hostTestAppId, pid: hostTestPid, name: "Notes", running: true), - ] - environment.windows = [hostTestWindow()] - - let harness = ServerHarness(environment: environment) - try harness.begin() - - harness.send(observe(app: "Notes")) - XCTAssertEqual(try errorCode(harness.awaitResult()), "app_not_found") - - harness.send(observe(app: "COM.APPLE.NOTES")) - XCTAssertEqual(try errorCode(harness.awaitResult()), "app_not_found", "exact match, not case-folded") - } - - func testAppTargetChoosesFrontmostSheetAndWindowTargetRemainsExact() throws { - let pid = getpid() - let appId = "pid:\(pid)" - let sheet = hostTestWindow( - windowId: 72, - pid: pid, - appId: appId, - bounds: CGRect(x: 100, y: 100, width: 360, height: 150), - title: "Sheet", - zIndex: 9 - ) - let main = hostTestWindow( - windowId: 71, - pid: pid, - appId: appId, - bounds: CGRect(x: 50, y: 50, width: 800, height: 600), - title: "Main", - zIndex: 3 - ) - - var environment = FakeEnvironment() - environment.apps = [ - HostRunningApp(appId: appId, pid: pid, name: "Fixture", running: true), - ] - environment.windows = [sheet, main] - environment.windowElement = hostTestElement(pid: pid) - - let harness = ServerHarness(environment: environment) - try harness.begin() - - harness.send(observe(app: appId)) - let appSnapshot = try XCTUnwrap( - try harness.awaitResult()["snapshot"] as? [String: Any] - ) - let appTarget = try XCTUnwrap(appSnapshot["target"] as? [String: Any]) - XCTAssertEqual(appTarget["windowId"] as? UInt32, sheet.windowId) - - harness.send( - """ - {"jsonrpc":"2.0","id":22,"method":"observe","params":{\ - "session":"s1","target":{"kind":"window","pid":\(pid),"windowId":\(main.windowId)},\ - "includeImage":false}} - """ - ) - let exactSnapshot = try XCTUnwrap( - try harness.awaitResult()["snapshot"] as? [String: Any] - ) - let exactTarget = try XCTUnwrap(exactSnapshot["target"] as? [String: Any]) - XCTAssertEqual(exactTarget["windowId"] as? UInt32, main.windowId) - } - - func testWindowFrameMatchingFindsSheetCandidateWithoutChangingDirectPriority() { + func testWindowIdentitySelectsOnlyOneExactWindowIdAndMatchingFrame() { let target = CGRect(x: 100, y: 100, width: 360, height: 150) - let direct = [ - (element: "main", frame: CGRect(x: 50, y: 50, width: 800, height: 600)), - ] - let sheets = [ - (element: "sheet", frame: CGRect(x: 100.4, y: 99.6, width: 360, height: 150)), - ] - - XCTAssertNil(hostFirstWindowCandidate(direct, matching: target)) XCTAssertEqual( - hostFirstWindowCandidate(direct + sheets, matching: target), + hostUniqueWindowCandidate( + [ + (element: "same-frame-wrong-id", windowId: 40, frame: target), + (element: "sheet", windowId: 41, frame: CGRect(x: 100.4, y: 99.6, width: 360, height: 150)), + ], + matching: 41, + bounds: target + ), "sheet" ) - XCTAssertEqual( - hostFirstWindowCandidate( - [(element: "direct", frame: target)] + sheets, - matching: target + XCTAssertNil( + hostUniqueWindowCandidate( + [ + (element: "first", windowId: 41, frame: target), + (element: "second", windowId: 41, frame: target), + ], + matching: 41, + bounds: target ), - "direct", - "an ordinary AXWindow match remains preferred over a child sheet" + "duplicate AX candidates for one CGWindowID must fail closed" + ) + XCTAssertNil( + hostUniqueWindowCandidate( + [(element: "moved", windowId: 41, frame: CGRect(x: 101, y: 100, width: 360, height: 150))], + matching: 41, + bounds: target + ) ) } @@ -150,14 +62,14 @@ final class HostObserveContractTests: XCTestCase { let harness = ServerHarness(environment: environment) try harness.begin() - harness.send(observe(app: appId)) + harness.send(observe(appId: appId, pid: pid, windowId: 81)) let first = try XCTUnwrap( try harness.awaitResult()["snapshot"] as? [String: Any] ) XCTAssertNil(first["difference"]) let firstId = try XCTUnwrap(first["snapshotId"] as? String) - harness.send(observe(app: appId)) + harness.send(observe(appId: appId, pid: pid, windowId: 81)) let second = try XCTUnwrap( try harness.awaitResult()["snapshot"] as? [String: Any] ) @@ -184,7 +96,7 @@ final class HostObserveContractTests: XCTestCase { let harness = ServerHarness(environment: environment) try harness.begin() - harness.send(observe(app: appId)) + harness.send(observe(appId: appId, pid: pid, windowId: 91)) let snapshot = try XCTUnwrap( try harness.awaitResult()["snapshot"] as? [String: Any] @@ -206,7 +118,7 @@ final class HostObserveContractTests: XCTestCase { let harness = ServerHarness(environment: environment) try harness.begin() - harness.send(observe(app: "com.example.absent")) + harness.send(observe(appId: "com.example.absent", pid: hostTestPid, windowId: 999)) let result = try harness.awaitResult() XCTAssertEqual(result["ok"] as? Bool, false) XCTAssertNil(result["outcome"]) @@ -374,11 +286,13 @@ final class HostObserveContractTests: XCTestCase { private var nextId = 200 - private func observe(app: String) -> String { + private func observe(appId: String, pid: pid_t, windowId: CGWindowID) -> String { nextId += 1 return """ {"jsonrpc":"2.0","id":\(nextId),"method":"observe","params":{"session":"s1",\ - "target":{"kind":"app","app":"\(app)"},"includeImage":false}} + "target":{"kind":"window","appId":"\(appId)","pid":\(pid),\ + "processGeneration":"\(hostProcessGeneration(UInt64(pid)))","windowId":\(windowId)},\ + "includeImage":false}} """ } diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveLiveTests.swift index 717e13b..f487406 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveLiveTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObserveLiveTests.swift @@ -68,7 +68,7 @@ final class HostObserveLiveTests: XCTestCase { // has a ceiling of its own that would be mixed into the measurement. let started = Date() server.handle(line: #""" - {"jsonrpc":"2.0","id":\#(10 + offset),"method":"observe","params":{"session":"s1","target":{"kind":"window","pid":\#(window.pid),"windowId":\#(window.windowId)},"includeImage":false}} + {"jsonrpc":"2.0","id":\#(10 + offset),"method":"observe","params":{"session":"s1","target":{"kind":"window","appId":"\#(window.appId)","pid":\#(window.pid),"processGeneration":"\#(hostProcessGeneration(hostProcessStartTime(pid: window.pid) ?? 0))","windowId":\#(window.windowId)},"includeImage":false}} """#) // The wait is longer than the host's own deadline on purpose: an diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObservePerformanceLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObservePerformanceLiveTests.swift index 9b7b530..bc4ee7c 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObservePerformanceLiveTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostObservePerformanceLiveTests.swift @@ -277,7 +277,7 @@ final class HostObservePerformanceLiveTests: XCTestCase { let menuClause = menu ? #","menu":{"scope":"bar"}"# : "" let started = DispatchTime.now().uptimeNanoseconds server.handle(line: #""" - {"jsonrpc":"2.0","id":\#(identifier),"method":"observe","params":{"session":"bench","target":{"kind":"window","pid":\#(window.pid),"windowId":\#(window.windowId)},"includeImage":\#(image)\#(menuClause)}} + {"jsonrpc":"2.0","id":\#(identifier),"method":"observe","params":{"session":"bench","target":{"kind":"window","appId":"\#(window.appId)","pid":\#(window.pid),"processGeneration":"\#(hostProcessGeneration(hostProcessStartTime(pid: window.pid) ?? 0))","windowId":\#(window.windowId)},"includeImage":\#(image)\#(menuClause)}} """#) let response = try inbox.next(timeout: 120) let ms = Double(DispatchTime.now().uptimeNanoseconds - started) / 1_000_000 diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTestSupport.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTestSupport.swift index c220832..75a830f 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTestSupport.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTestSupport.swift @@ -3,7 +3,7 @@ import Foundation import XCTest @testable import OpenComputerUseKit -// Shared doubles for the `maka.cu/2` conformance vectors. Everything here stands +// Shared doubles for the `maka.cu/3` conformance vectors. Everything here stands // in for the live machine so the §4 binding rules can be asserted without a // desktop; nothing here fakes protocol logic. @@ -77,6 +77,7 @@ struct FakeBindingProbe: HostElementBindingProbe { var startTime: UInt64? = hostTestProcessStartTime var startTimes: [pid_t: UInt64] = [:] var actualPidOverride: pid_t? + var belongsToTarget = true var override: HostElementDigestInput? var refetch: HostBindingRefetchResult = .missing var webContentEquivalent: HostElementBinding? @@ -90,6 +91,7 @@ struct FakeBindingProbe: HostElementBindingProbe { func actualPid(_ binding: HostElementBinding) -> pid_t? { actualPidOverride ?? binding.dispatchPid } + func belongsToBoundTarget(_ binding: HostElementBinding) -> Bool { belongsToTarget } func currentDigestInput(_ binding: HostElementBinding) -> HostElementDigestInput? { override ?? binding.digestInput } @@ -356,6 +358,38 @@ final class WindowElementLog { } } +final class AccessibilityActionLog { + private let lock = NSLock() + private(set) var performed: [HostElementActionName] = [] + var result: AXError = .success + + func perform(_ action: HostElementActionName) -> AXError { + lock.lock() + defer { lock.unlock() } + performed.append(action) + return result + } +} + +/// Keeps the original binding valid until the first Accessibility action lands, +/// then reports that the retained element moved to another window. +final class ActionReactiveBindingProbe: HostElementBindingProbe { + private let actions: AccessibilityActionLog + + init(actions: AccessibilityActionLog) { + self.actions = actions + } + + func isReferenceAlive(_ binding: HostElementBinding) -> Bool { true } + func processStartTime(pid: pid_t) -> UInt64? { hostTestProcessStartTime } + func belongsToBoundTarget(_ binding: HostElementBinding) -> Bool { + actions.performed.isEmpty + } + func currentDigestInput(_ binding: HostElementBinding) -> HostElementDigestInput? { + binding.digestInput + } +} + struct FakeEnvironment: HostSystemEnvironment { var locked = false var accessibilityTrusted = true @@ -385,7 +419,10 @@ struct FakeEnvironment: HostSystemEnvironment { var pointEvents = PointEventLog() var keyEvents = KeyEventLog() var focusRequests = FocusRequestLog() + var accessibilityActions = AccessibilityActionLog() var launches = AppLaunchLog() + var installedBundleIds: [String] = [] + var processStartTimes: [pid_t: UInt64] = [:] func screenIsLocked() -> Bool { locked } @@ -397,6 +434,10 @@ struct FakeEnvironment: HostSystemEnvironment { } func runningApps() -> [HostRunningApp] { inventory.read() } + func installedBundleIdentifiers( + matching query: String + ) -> Result<[String], HostDomainError> { .success(installedBundleIds) } + func processStartTime(pid: pid_t) -> UInt64? { processStartTimes[pid] ?? UInt64(pid) } func frontmostApplicationPid() -> pid_t? { frontmost.read() } func restoreFrontmostApplication(pid: pid_t) -> Bool { frontmost.restore(pid) } func beginSyntheticTargetFocus( @@ -417,7 +458,11 @@ struct FakeEnvironment: HostSystemEnvironment { func menuBarNode(pid: pid_t) -> HostAccessibilityNode? { menuBar } func focusedElement(pid: pid_t) -> AXUIElement? { focusRequests.currentFocus ?? focused } func setFocusedElement(_ element: AXUIElement, pid: pid_t) -> Bool { focusRequests.record(element) } - func bindingProbe(windowBounds: CGRect) -> HostElementBindingProbe { probe } + func bindingProbe(windowId: CGWindowID, windowBounds: CGRect) -> HostElementBindingProbe { probe } + func performAccessibilityAction( + _ action: HostElementActionName, + on element: AXUIElement + ) -> AXError { accessibilityActions.perform(action) } func postPointEvent( _ action: HostPointAction, @@ -602,6 +647,7 @@ func hostTestBinding( frame: HostRect? = nil, element: AXUIElement? = nil, actions: [HostElementActionName] = [.press], + isMenu: Bool = false, dispatchPid: pid_t = hostTestPid, dispatchProcessStartTime: UInt64 = hostTestProcessStartTime ) -> HostElementBinding { @@ -634,7 +680,8 @@ func hostTestBinding( actions: actions, digest: hostElementDigest(digestInput), truncated: [] - ) + ), + isMenu: isMenu ) } @@ -650,6 +697,7 @@ func hostTestSnapshot( elementFrame: HostRect? = nil, element: AXUIElement? = nil, elementActions: [HostElementActionName] = [.press], + isMenu: Bool = false, dispatchPid: pid_t? = nil, dispatchProcessStartTime: UInt64? = nil ) -> HostSnapshot { @@ -665,6 +713,7 @@ func hostTestSnapshot( frame: elementFrame, element: element, actions: elementActions, + isMenu: isMenu, dispatchPid: dispatchPid ?? hostTestPid, dispatchProcessStartTime: dispatchProcessStartTime ?? hostTestProcessStartTime @@ -681,6 +730,7 @@ func hostTestSnapshot( capturedAt: capturedAt, target: HostWindowTarget( pid: window.pid, + processGeneration: hostProcessGeneration(hostTestProcessStartTime), windowId: window.windowId, appId: window.appId, appName: window.appName, @@ -706,7 +756,10 @@ func hostTestSnapshot( id: id, session: session, pid: window.pid, + appId: window.appId, + processGeneration: hostProcessGeneration(hostTestProcessStartTime), windowId: window.windowId, + windowBounds: window.bounds, capturedAt: capturedAt, windowDigest: windowDigest, payload: payload, @@ -747,6 +800,7 @@ func hostTestWalkedSnapshot( capturedAt: capturedAt, target: HostWindowTarget( pid: window.pid, + processGeneration: hostProcessGeneration(hostTestProcessStartTime), windowId: window.windowId, appId: window.appId, appName: window.appName, @@ -772,7 +826,10 @@ func hostTestWalkedSnapshot( id: id, session: session, pid: window.pid, + appId: window.appId, + processGeneration: hostProcessGeneration(hostTestProcessStartTime), windowId: window.windowId, + windowBounds: window.bounds, capturedAt: capturedAt, windowDigest: windowDigest, payload: payload, diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTests.swift index 50a5172..0e18e2f 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostProtocolTests.swift @@ -5,7 +5,7 @@ import Foundation import XCTest @testable import OpenComputerUseKit -/// Conformance vectors for `maka.cu/2` (§12). Each test here fails without the +/// Conformance vectors for `maka.cu/3` (§12). Each test here fails without the /// rule it names; the rules that need a live desktop (real Accessibility /// invalidation, real capture) are called out in the commit rather than faked. final class HostProtocolTests: XCTestCase { @@ -202,9 +202,16 @@ final class HostProtocolTests: XCTestCase { let failure = hostVerifyBinding(binding, probe: probe) XCTAssertEqual(failure?.code, .elementChanged) XCTAssertEqual(failure?.detail, .changed([.label])) + XCTAssertNil( + hostVerifyBindingTarget(binding, probe: probe), + "an earlier effect may change E3 without changing its approved target" + ) probe = FakeBindingProbe() XCTAssertNil(hostVerifyBinding(binding, probe: probe)) + + probe.belongsToTarget = false + XCTAssertEqual(hostVerifyBinding(binding, probe: probe)?.code, .windowChanged) } func testWindowDigestChangesWhenAnUnrelatedElementChanges() { @@ -752,7 +759,7 @@ final class HostProtocolTests: XCTestCase { func testAnyMethodBeforeHostHelloIsHandshakeRequired() throws { let harness = ServerHarness() - harness.send(#"{"jsonrpc":"2.0","id":1,"method":"observe","params":{"session":"s","target":{"kind":"app","app":"Notes"}}}"#) + harness.send(#"{"jsonrpc":"2.0","id":1,"method":"observe","params":{"session":"s","target":{"kind":"window","appId":"com.apple.Notes","pid":42,"processGeneration":"pst:1","windowId":7}}}"#) let error = try XCTUnwrap(try harness.awaitResponse()["error"] as? [String: Any]) XCTAssertEqual(error["code"] as? Int, -32001) @@ -770,7 +777,7 @@ final class HostProtocolTests: XCTestCase { XCTAssertEqual(error["code"] as? Int, -32000) XCTAssertEqual(error["message"] as? String, "protocol_version_mismatch") let data = try XCTUnwrap(error["data"] as? [String: Any]) - XCTAssertEqual(data["supported"] as? [String], ["maka.cu/2"]) + XCTAssertEqual(data["supported"] as? [String], ["maka.cu/3"]) // §2 — `EX_CONFIG`, so the host classifies the start as `service_mismatch` // and does not retry. @@ -783,7 +790,7 @@ final class HostProtocolTests: XCTestCase { let result = try XCTUnwrap(try harness.awaitResponse()["result"] as? [String: Any]) XCTAssertEqual(result["ok"] as? Bool, true) - XCTAssertEqual(result["protocol"] as? String, "maka.cu/2") + XCTAssertEqual(result["protocol"] as? String, "maka.cu/3") let limits = try XCTUnwrap(result["limits"] as? [String: Any]) for key in [ @@ -870,7 +877,7 @@ final class HostProtocolTests: XCTestCase { harness.sendHello() _ = try harness.awaitResponse() - harness.send(#"{"jsonrpc":"2.0","id":6,"method":"observe","params":{"session":"never-begun","target":{"kind":"app","app":"Notes"}}}"#) + harness.send(#"{"jsonrpc":"2.0","id":6,"method":"observe","params":{"session":"never-begun","target":{"kind":"window","appId":"com.apple.Notes","pid":42,"processGeneration":"pst:1","windowId":7}}}"#) let error = try XCTUnwrap(try harness.awaitResponse()["error"] as? [String: Any]) XCTAssertEqual(error["code"] as? Int, -32002) } @@ -891,7 +898,7 @@ final class HostProtocolTests: XCTestCase { harness.send(#"{"jsonrpc":"2.0","id":2,"method":"session.begin","params":{"session":"s1","captureScope":"window"}}"#) _ = try harness.awaitResponse() - harness.send(#"{"jsonrpc":"2.0","id":3,"method":"observe","params":{"session":"s1","target":{"kind":"window","pid":1,"windowId":1},"maxElements":999999}}"#) + harness.send(#"{"jsonrpc":"2.0","id":3,"method":"observe","params":{"session":"s1","target":{"kind":"window","appId":"pid:1","pid":1,"processGeneration":"pst:1","windowId":1},"maxElements":999999}}"#) let error = try XCTUnwrap(try harness.awaitResponse()["error"] as? [String: Any]) XCTAssertEqual(error["code"] as? Int, -32602) XCTAssertEqual((error["data"] as? [String: Any])?["field"] as? String, "maxElements") @@ -1779,6 +1786,59 @@ final class HostProtocolTests: XCTestCase { XCTAssertEqual(harness.environment.frontmost.reads, 2, "before and after, not once") } + // MARK: - Trusted target resolution (§5.2) + + func testTargetResolveFreezesRunningWindowAndProcessGeneration() throws { + var environment = FakeEnvironment() + environment.apps = [ + HostRunningApp( + appId: hostTestAppId, + pid: hostTestPid, + name: "Fixture", + running: true + ) + ] + environment.windows = [hostTestWindow()] + environment.processStartTimes[hostTestPid] = hostTestProcessStartTime + let harness = ServerHarness(environment: environment) + try harness.begin() + + harness.send(#"{"jsonrpc":"2.0","id":10,"method":"target.resolve","params":{"target":{"kind":"window","windowId":7}}}"#) + let result = try harness.awaitResult() + XCTAssertEqual(result["resolution"] as? String, "resolved") + let target = try XCTUnwrap(result["target"] as? [String: Any]) + XCTAssertEqual(target["kind"] as? String, "running") + XCTAssertEqual(target["appId"] as? String, hostTestAppId) + XCTAssertEqual(target["pid"] as? Int, Int(hostTestPid)) + XCTAssertEqual( + target["processGeneration"] as? String, + hostProcessGeneration(hostTestProcessStartTime) + ) + XCTAssertEqual(target["windowId"] as? Int, 7) + } + + func testTargetResolveDoesNotLaunchInstalledApplication() throws { + var environment = FakeEnvironment() + environment.installedBundleIds = ["com.apple.TextEdit"] + let harness = ServerHarness(environment: environment) + try harness.begin() + + harness.send(#"{"jsonrpc":"2.0","id":11,"method":"target.resolve","params":{"target":{"kind":"application","app":"TextEdit","intent":"launch"}}}"#) + let result = try harness.awaitResult() + XCTAssertEqual(result["resolution"] as? String, "resolved") + XCTAssertEqual((result["target"] as? [String: Any])?["kind"] as? String, "installed") + XCTAssertEqual(environment.launches.requests.count, 0) + } + + func testObserveRejectsNonCanonicalProcessGeneration() throws { + let harness = ServerHarness() + try harness.begin() + + harness.send(#"{"jsonrpc":"2.0","id":12,"method":"observe","params":{"session":"s1","target":{"kind":"window","appId":"com.apple.TextEdit","pid":42,"processGeneration":"pst:01","windowId":7}}}"#) + let error = try XCTUnwrap(try harness.awaitResponse()["error"] as? [String: Any]) + XCTAssertEqual(error["code"] as? Int, -32602) + } + // MARK: - Seeing the machine change (§5.5, §5.7) func testTheApplicationListIsAskedOfTheMachineOnEveryCall() throws { diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostSettleLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostSettleLiveTests.swift index 54dcfc4..67614ba 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostSettleLiveTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostSettleLiveTests.swift @@ -68,7 +68,14 @@ final class HostSettleLiveTests: XCTestCase { // capture has a ceiling of its own that would be mixed into it. guard case .success(let snapshot) = server.buildSnapshot( session: "s1", - target: .window(pid: window.pid, windowId: window.windowId), + target: .window( + appId: window.appId, + pid: window.pid, + processGeneration: hostProcessGeneration( + hostProcessStartTime(pid: window.pid) ?? 0 + ), + windowId: window.windowId + ), includeImage: false, menuScope: nil, maxElements: limits.maxElements, @@ -82,7 +89,7 @@ final class HostSettleLiveTests: XCTestCase { continue } - let probe = environment.bindingProbe(windowBounds: window.bounds) + let probe = environment.bindingProbe(windowId: window.windowId, windowBounds: window.bounds) var looks = 0 var costliestLook: TimeInterval = 0 diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostWindowDispatchLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostWindowDispatchLiveTests.swift index 6bdf73d..2845cb3 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostWindowDispatchLiveTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostWindowDispatchLiveTests.swift @@ -511,7 +511,7 @@ final class HostWindowDispatchLiveTests: XCTestCase { func observe(_ window: HostWindowInfo) throws -> LiveFrame { nextId += 1 server.handle(line: #""" - {"jsonrpc":"2.0","id":\#(nextId),"method":"observe","params":{"session":"s1","target":{"kind":"window","pid":\#(window.pid),"windowId":\#(window.windowId)},"includeImage":false}} + {"jsonrpc":"2.0","id":\#(nextId),"method":"observe","params":{"session":"s1","target":{"kind":"window","appId":"\#(window.appId)","pid":\#(window.pid),"processGeneration":"\#(hostProcessGeneration(hostProcessStartTime(pid: window.pid) ?? 0))","windowId":\#(window.windowId)},"includeImage":false}} """#) let result = try XCTUnwrap(try inbox.next(timeout: 30)["result"] as? [String: Any]) diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostWindowIdentityLiveTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostWindowIdentityLiveTests.swift new file mode 100644 index 0000000..43e6882 --- /dev/null +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/HostWindowIdentityLiveTests.swift @@ -0,0 +1,67 @@ +import AppKit +import ApplicationServices +import CoreGraphics +import Foundation +import XCTest +@testable import OpenComputerUseKit + +/// Verifies the private AX-to-CGWindowID contract against real AppKit windows. +/// This test is opt-in because it requires a logged-in WindowServer session and +/// Accessibility access for the test runner. +@MainActor +final class HostWindowIdentityLiveTests: XCTestCase { + func testProductionResolverMatchesAWindowAndAttachedSheet() throws { + guard ProcessInfo.processInfo.environment["OPEN_COMPUTER_USE_RUN_WINDOW_ID_LIVE_TEST"] == "1" else { + throw XCTSkip("Set OPEN_COMPUTER_USE_RUN_WINDOW_ID_LIVE_TEST=1 to run the live identity test") + } + XCTAssertTrue(hostWindowIdSPIAvailable(), "_AXUIElementGetWindow is required") + + let window = NSWindow( + contentRect: CGRect(x: 240, y: 240, width: 520, height: 360), + styleMask: [.titled, .closable], + backing: .buffered, + defer: false + ) + window.title = "Maka Window Identity Contract" + window.makeKeyAndOrderFront(nil) + + let sheet = NSWindow( + contentRect: CGRect(x: 300, y: 300, width: 360, height: 180), + styleMask: [.titled], + backing: .buffered, + defer: false + ) + window.beginSheet(sheet) + defer { + window.endSheet(sheet) + sheet.orderOut(nil) + window.orderOut(nil) + } + + RunLoop.current.run(until: Date(timeIntervalSinceNow: 0.2)) + try assertResolved(windowNumber: window.windowNumber, expectedRole: kAXWindowRole as String) + try assertResolved(windowNumber: sheet.windowNumber, expectedRole: "AXSheet") + } + + private func assertResolved(windowNumber: Int, expectedRole: String) throws { + let windowId = CGWindowID(windowNumber) + let deadline = Date(timeIntervalSinceNow: 1) + var info: HostWindowInfo? + repeat { + info = HostWindowInventory.onScreenWindows().first { + $0.pid == getpid() && $0.windowId == windowId + } + if info == nil { + RunLoop.current.run(until: Date(timeIntervalSinceNow: 0.02)) + } + } while info == nil && Date() < deadline + + let resolvedInfo = try XCTUnwrap(info, "WindowServer did not publish window \(windowId)") + let element = try XCTUnwrap( + HostAX.window(pid: getpid(), windowId: windowId, bounds: resolvedInfo.bounds), + "Accessibility did not publish exact window \(windowId)" + ) + XCTAssertEqual(hostWindowId(of: element), windowId) + XCTAssertEqual(HostAX.string(element, kAXRoleAttribute), expectedRole) + } +} diff --git a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift index d3d87a7..499f5d5 100644 --- a/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift +++ b/packages/OpenComputerUseKit/Tests/OpenComputerUseKitTests/OpenComputerUseKitTests.swift @@ -87,7 +87,7 @@ final class OpenComputerUseKitTests: XCTestCase { from: Data(encoded.utf8) ) XCTAssertEqual(decoded, report) - XCTAssertTrue(encoded.contains(#""protocolVersion" : "maka.cu/2""#)) + XCTAssertTrue(encoded.contains(#""protocolVersion" : "maka.cu/3""#)) } func testDoctorTrustedWebReadinessRequiresEveryNativeFence() { diff --git a/scripts/maka-cu-binding-proof.mjs b/scripts/maka-cu-binding-proof.mjs deleted file mode 100644 index bfc86f5..0000000 --- a/scripts/maka-cu-binding-proof.mjs +++ /dev/null @@ -1,239 +0,0 @@ -#!/usr/bin/env node -// Does the frame binding actually bind? -// -// A unit test can be deleted and stay green; this cannot. It drives the real -// executor against a real window and asks the three questions that decide -// whether the binding is a mechanism or a decoration: -// -// 1. a correct token + correct digest dispatches -// 2. the SAME snapshot, used twice, is refused (single-use) -// 3. a corrupted digest is refused (the digest is actually read) -// -// and, throughout, that driving a background window does not steal the user's -// foreground — the invariant the whole executor exists to keep. -// -// Read-only with respect to permissions: `permissions.check` does not prompt. -// The target is the CUA Lab fixture, whose Reset button exists to be pressed. -import { spawn, execFileSync } from 'node:child_process'; -import { mkdtempSync, rmSync } from 'node:fs'; -import { tmpdir } from 'node:os'; -import { join } from 'node:path'; - -// Resolved from this repository rather than hardcoded, so the script works -// from any checkout. `swift build -c release` produces it. -const BINARY = new URL('../.build/release/OpenComputerUse', import.meta.url).pathname; -const TARGET_NAME = process.argv[2] ?? 'Codex CUA Lab'; -const imageDir = mkdtempSync(join(tmpdir(), 'maka-cu-binding-')); - -const child = spawn(BINARY, ['host'], { stdio: ['pipe', 'pipe', 'pipe'] }); -let nextId = 1; -const pending = new Map(); -let buffered = ''; -const stderrLines = []; - -child.stdout.on('data', (chunk) => { - buffered += chunk.toString('utf8'); - let index; - while ((index = buffered.indexOf('\n')) >= 0) { - const line = buffered.slice(0, index).trim(); - buffered = buffered.slice(index + 1); - if (!line) continue; - try { - const message = JSON.parse(line); - const waiter = pending.get(message.id); - if (waiter) { - pending.delete(message.id); - waiter(message); - } - } catch { - /* non-JSON on stdout is a protocol violation; the first-light probe reports it */ - } - } -}); -child.stderr.on('data', (c) => stderrLines.push(c.toString('utf8').trim())); - -function call(method, params) { - const id = nextId++; - return new Promise((resolve, reject) => { - const timer = setTimeout(() => { - pending.delete(id); - reject(new Error(`${method} timed out`)); - }, 30_000); - pending.set(id, (message) => { - clearTimeout(timer); - resolve(message); - }); - child.stdin.write(`${JSON.stringify({ jsonrpc: '2.0', id, method, params })}\n`); - }); -} - -function frontmost() { - try { - return execFileSync( - 'osascript', - [ - '-e', - 'tell application "System Events" to get unix id of first process whose frontmost is true', - ], - { encoding: 'utf8', timeout: 5000 }, - ).trim(); - } catch { - return 'unavailable'; - } -} - -let failures = 0; -const check = (label, pass, detail) => { - if (!pass) failures += 1; - console.log(`[${pass ? 'PASS' : 'FAIL'}] ${label}${detail ? ` — ${detail}` : ''}`); -}; -const refusalOf = (message) => { - const result = message.result ?? {}; - if (message.error) return `rpc:${message.error.code}`; - if (result.ok === false) return result.error?.code ?? 'unknown'; - return null; -}; - -const SESSION = 'binding-proof'; - -try { - await call('host.hello', { - protocol: 'maka.cu/2', - hostPid: process.pid, - imageDir, - allowGlobalPointer: false, - }); - const perms = (await call('permissions.check', {})).result ?? {}; - if (!perms.accessibility) { - console.log('Accessibility not granted for this process; stopping before anything else.'); - process.exit(2); - } - await call('session.begin', { session: SESSION, captureScope: 'window' }); - - const apps = (await call('apps.list', { session: SESSION })).result?.apps ?? []; - const target = apps.find((a) => a.name === TARGET_NAME); - if (!target) { - console.log(`fixture "${TARGET_NAME}" is not running; launch it and re-run.`); - process.exit(2); - } - - const beforePid = frontmost(); - console.log(`frontmost before: pid ${beforePid} (target is pid ${target.pid})`); - check( - 'the target is NOT frontmost, so this is a background test', - String(beforePid) !== String(target.pid), - `frontmost=${beforePid} target=${target.pid}`, - ); - - const observe = async (tag) => { - const message = await call('observe', { - session: SESSION, - target: { kind: 'app', app: target.appId }, - includeImage: false, - }); - const snapshot = message.result?.snapshot; - if (!snapshot) throw new Error(`${tag}: observe failed — ${JSON.stringify(message).slice(0, 300)}`); - return snapshot; - }; - const pressable = (snapshot) => - // Deliberately NOT "CUA Lab Reset". Measured on 2026-07-29 against the same - // executor, the same action and the same code path: Primary Button and Diff - // Probe leave the foreground alone, Reset raises the fixture. The fixture's - // Reset handler activates its own app, so pressing it would fail the - // foreground invariant for a reason that has nothing to do with the - // executor — and did, on the first run of this script. - snapshot.elements.find((e) => e.label === 'CUA Lab Primary Button') ?? - snapshot.elements.find((e) => e.role === 'AXButton' && e.label !== 'CUA Lab Reset'); - - // ── 1. a correct token and digest dispatches ────────────────────────────── - const first = await observe('first'); - const button = pressable(first); - check('the fixture exposes a pressable element', Boolean(button), button?.label ?? 'none'); - - const good = await call('dispatch.element', { - session: SESSION, - snapshotId: first.snapshotId, - toolCallId: 'proof-1', - elementToken: button.token, - expectElementDigest: button.digest, - action: { kind: 'click', button: 'left', count: 1 }, - }); - const goodResult = good.result ?? {}; - check( - 'a correct token and digest dispatches', - goodResult.ok === true && goodResult.outcome === 'ok', - `outcome=${goodResult.outcome} path=${goodResult.path} tier=${goodResult.tier} ${refusalOf(good) ?? ''}`, - ); - - // ── 2. the same snapshot, used twice ────────────────────────────────────── - const replay = await call('dispatch.element', { - session: SESSION, - snapshotId: first.snapshotId, - toolCallId: 'proof-2', - elementToken: button.token, - expectElementDigest: button.digest, - action: { kind: 'click', button: 'left', count: 1 }, - }); - check( - 'a spent snapshot is refused', - replay.result?.ok === false || Boolean(replay.error), - refusalOf(replay) ?? 'IT WENT THROUGH — the snapshot is not single-use', - ); - - // ── 3. a corrupted digest ───────────────────────────────────────────────── - // The token is real and current; only the digest is wrong. If this dispatches, - // the digest is being carried but never compared. - const fresh = await observe('fresh'); - const freshButton = pressable(fresh); - const corrupted = `${String(freshButton.digest).slice(0, -4)}0000`; - const tampered = await call('dispatch.element', { - session: SESSION, - snapshotId: fresh.snapshotId, - toolCallId: 'proof-3', - elementToken: freshButton.token, - expectElementDigest: corrupted, - action: { kind: 'click', button: 'left', count: 1 }, - }); - check( - 'a corrupted digest is refused', - tampered.result?.ok === false || Boolean(tampered.error), - refusalOf(tampered) ?? 'IT WENT THROUGH — the digest is decoration', - ); - - // ── 4. an unknown token ─────────────────────────────────────────────────── - const bogus = await observe('bogus'); - const unknown = await call('dispatch.element', { - session: SESSION, - snapshotId: bogus.snapshotId, - toolCallId: 'proof-4', - elementToken: 'el_not_a_real_token', - expectElementDigest: pressable(bogus).digest, - action: { kind: 'click', button: 'left', count: 1 }, - }); - check( - 'an unknown token is refused', - unknown.result?.ok === false || Boolean(unknown.error), - refusalOf(unknown) ?? 'IT WENT THROUGH', - ); - - // ── the invariant that outranks all of them ─────────────────────────────── - const afterPid = frontmost(); - check( - 'driving a background window did not steal the foreground', - String(afterPid) === String(beforePid), - `${beforePid} → ${afterPid}`, - ); - - await call('session.end', { session: SESSION }); -} catch (error) { - failures += 1; - console.log(`\n[ERROR] ${error.message}`); -} finally { - child.stdin.end(); - child.kill('SIGTERM'); - rmSync(imageDir, { recursive: true, force: true }); -} - -if (stderrLines.length > 0) console.log(`\nstderr:\n ${stderrLines.join('\n ').slice(0, 700)}`); -console.log(failures === 0 ? '\nBINDING PROVEN' : `\n${failures} CHECK(S) FAILED`); -process.exit(failures === 0 ? 0 : 1); diff --git a/scripts/maka-cu-first-light.mjs b/scripts/maka-cu-first-light.mjs deleted file mode 100644 index ed95510..0000000 --- a/scripts/maka-cu-first-light.mjs +++ /dev/null @@ -1,175 +0,0 @@ -#!/usr/bin/env node -// First light for the maka-cu host protocol: talk to the executor directly, -// with no TypeScript backend in between, so a failure is unambiguously the -// executor's rather than the client's. -// -// Read-only. `permissions.check` is the only permission touch and it does not -// prompt, so this cannot create a TCC grant for a bare node process — the trap -// this repository has hit before. -// -// node scripts/maka-cu-first-light.mjs "Codex CUA Lab" -import { spawn } from 'node:child_process'; -import { mkdtempSync, rmSync } from 'node:fs'; -import { tmpdir } from 'node:os'; -import { join } from 'node:path'; - -// Resolved from this repository rather than hardcoded, so the script works -// from any checkout. `swift build -c release` produces it. -const BINARY = new URL('../.build/release/OpenComputerUse', import.meta.url).pathname; -const TARGET = process.argv[2] ?? 'Codex CUA Lab'; -const imageDir = mkdtempSync(join(tmpdir(), 'maka-cu-first-light-')); - -const child = spawn(BINARY, ['host'], { stdio: ['pipe', 'pipe', 'pipe'] }); -let nextId = 1; -const pending = new Map(); -let buffered = ''; - -child.stdout.on('data', (chunk) => { - buffered += chunk.toString('utf8'); - let index; - while ((index = buffered.indexOf('\n')) >= 0) { - const line = buffered.slice(0, index).trim(); - buffered = buffered.slice(index + 1); - if (!line) continue; - let message; - try { - message = JSON.parse(line); - } catch { - console.log(` [stdout non-JSON] ${line.slice(0, 160)}`); - continue; - } - const waiter = pending.get(message.id); - if (waiter) { - pending.delete(message.id); - waiter(message); - } - } -}); -const stderrLines = []; -child.stderr.on('data', (c) => stderrLines.push(c.toString('utf8').trim())); - -function call(method, params) { - const id = nextId++; - return new Promise((resolve, reject) => { - const timer = setTimeout(() => { - pending.delete(id); - reject(new Error(`${method} timed out`)); - }, 30_000); - pending.set(id, (message) => { - clearTimeout(timer); - resolve(message); - }); - child.stdin.write(`${JSON.stringify({ jsonrpc: '2.0', id, method, params })}\n`); - }); -} - -let failures = 0; -function check(label, pass, detail) { - if (!pass) failures += 1; - console.log(`[${pass ? 'PASS' : 'FAIL'}] ${label}${detail ? ` — ${detail}` : ''}`); -} - -function summarize(message) { - if (message.error) return `rpc error ${message.error.code} ${message.error.message ?? ''}`; - const result = message.result ?? {}; - if (result.ok === false) return `refused ${result.error?.code}: ${result.error?.message ?? ''}`; - return 'ok'; -} - -try { - // ── handshake ───────────────────────────────────────────────────────────── - const hello = await call('host.hello', { - protocol: 'maka.cu/2', - hostPid: process.pid, - imageDir, - allowGlobalPointer: false, - }); - check('handshake accepted', !hello.error && hello.result?.ok !== false, summarize(hello)); - if (hello.error) { - console.log(JSON.stringify(hello.error, null, 2)); - process.exit(1); - } - console.log(` executor: ${JSON.stringify(hello.result).slice(0, 220)}`); - - // ── permissions, non-prompting ──────────────────────────────────────────── - const perms = await call('permissions.check', {}); - console.log(` permissions: ${JSON.stringify(perms.result ?? perms.error)}`); - - // ── session ─────────────────────────────────────────────────────────────── - const begun = await call('session.begin', { session: 'first-light', captureScope: 'window' }); - check('session.begin', !begun.error && begun.result?.ok !== false, summarize(begun)); - - // ── discovery ───────────────────────────────────────────────────────────── - const apps = await call('apps.list', { session: 'first-light' }); - const appList = apps.result?.apps ?? []; - check('apps.list returns apps', appList.length > 0, `${appList.length} apps`); - const target = appList.find( - (a) => a.appId === TARGET || a.name === TARGET || a.appId?.includes(TARGET), - ); - console.log(` target lookup "${TARGET}" → ${target ? JSON.stringify(target) : 'NOT FOUND'}`); - if (!target) { - console.log(` available: ${appList.map((a) => `${a.name}=${a.appId}`).slice(0, 8).join(', ')}`); - } - - const windows = await call('window.list', { session: 'first-light' }); - const windowList = windows.result?.windows ?? []; - check('window.list returns windows', windowList.length > 0, `${windowList.length} windows`); - - // ── the real question: does observe produce a bound snapshot ────────────── - if (target) { - const observed = await call('observe', { - session: 'first-light', - // §5.2: the field is `app` and its value is an appId. The two spellings - // are deliberate — one namespace, and a tagged union so "app or window" - // cannot be read as "app and window", which is how a compliant model was - // once made to fail against a harness that required both. - target: { kind: 'app', app: target.appId }, - includeImage: false, - }); - const snapshot = observed.result?.snapshot; - check('observe returns a snapshot', Boolean(snapshot), summarize(observed)); - if (snapshot) { - const elements = snapshot.elements ?? []; - console.log(` snapshotId=${snapshot.snapshotId}`); - console.log(` window="${snapshot.target?.title ?? ''}" appId=${snapshot.target?.appId}`); - console.log(` elements=${elements.length}`); - const withToken = elements.filter((e) => typeof e.token === 'string' && e.token.length > 0); - check( - 'every element carries a binding token', - withToken.length === elements.length, - `${withToken.length}/${elements.length}`, - ); - const withDigest = elements.filter((e) => typeof e.digest === 'string'); - check( - 'every element carries a digest', - withDigest.length === elements.length, - `${withDigest.length}/${elements.length}`, - ); - for (const element of elements.slice(0, 6)) { - console.log( - ` ${element.role}${element.label ? ` "${element.label}"` : ''}` + - `${element.value !== undefined ? ` =${JSON.stringify(element.value)}` : ''}` + - ` token=${String(element.token).slice(0, 12)}…`, - ); - } - if (elements.length > 6) console.log(` … ${elements.length - 6} more`); - } else { - console.log(JSON.stringify(observed).slice(0, 800)); - } - } - - await call('session.end', { session: 'first-light' }); -} catch (error) { - failures += 1; - console.log(`\n[ERROR] ${error.message}`); -} finally { - child.stdin.end(); - child.kill('SIGTERM'); - rmSync(imageDir, { recursive: true, force: true }); -} - -if (stderrLines.length > 0) { - console.log(`\nstderr:\n ${stderrLines.join('\n ').slice(0, 900)}`); -} -console.log(failures === 0 ? '\nFIRST LIGHT OK' : `\n${failures} CHECK(S) FAILED`); -process.exit(failures === 0 ? 0 : 1);