diff --git a/skills/cua-driver/README.md b/skills/cua-driver/README.md new file mode 100644 index 0000000..a60996b --- /dev/null +++ b/skills/cua-driver/README.md @@ -0,0 +1,128 @@ +# cua-driver — Claude Code skill + +A [Claude Code](https://code.claude.com) skill that teaches Claude to +drive native macOS apps via the +[`cua-driver`](https://github.com/trycua/cua/tree/main/libs/cua-driver) +CLI — snapshot an app's accessibility tree, click/type/scroll by +`element_index`, and verify via re-snapshot. Backgrounded-first: no +focus steal, no cursor warp, no Space follow. + +## What the skill covers + +- The snapshot-before-AND-after invariant that keeps the agent honest + about whether an action actually landed. +- The backgrounded-click recipe (yabai focus-without-raise + stamped + SLEventPostToPid) that lets synthetic clicks land on Chrome web + content without raising the window or pulling the user across Spaces. +- Web-app quirks (`WEB_APPS.md`) — Chromium/WebKit/Electron/Tauri, + including the minimized-Chrome keyboard-commit caveat and the + `set_value` workaround. +- Trajectory recording (`RECORDING.md`) — optional per-session + recording + replay for demos and regressions. +- Canvas/viewport apps (Blender, Unity, GHOST, Qt, wxWidgets) — + HID-tap fallback when AX is empty. + +See `SKILL.md` for the main body. + +## Prerequisites + +1. **macOS 14 or newer** — the driver depends on SkyLight private SPIs + that were stabilized in Sonoma. +2. **`cua-driver` CLI + `CuaDriver.app`** — installable one-liner: + ```bash + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)" + ``` + Or from a clone of `trycua/cua`: + ```bash + cd libs/cua-driver + scripts/install-local.sh # builds + installs + symlinks for dev use + ``` + The driver runs as an `.app` bundle because macOS TCC grants are + tied to a stable bundle id (`com.trycua.driver`). The CLI symlink + lets Claude invoke tools via plain shell. +3. **TCC grants on `CuaDriver.app`** — **Accessibility** and + **Screen Recording** in System Settings → Privacy & Security. + Verify with: + ```bash + cua-driver check_permissions + ``` + Both fields must be `true`. If not, the app appears in the + relevant panes of System Settings after first use; toggle it on + there. + +## Install + +The skill is two drop-in directories. + +**Personal scope** (all Claude Code sessions on your machine): + +```bash +mkdir -p ~/.claude/skills +cp -R Skills/cua-driver ~/.claude/skills/ +``` + +Or symlink if you want edits-in-place: + +```bash +ln -s "$PWD/Skills/cua-driver" ~/.claude/skills/cua-driver +``` + +**Project scope** (committed alongside a specific repo): + +```bash +mkdir -p .claude/skills +cp -R /path/to/cua/libs/cua-driver/Skills/cua-driver .claude/skills/ +``` + +## Invoking the skill + +Claude Code auto-invokes the skill when you ask for macOS GUI +automation — e.g. "open the Downloads folder in Finder", "click the +Save button in Numbers", "navigate to trycua.com in Chrome". You can +also invoke it explicitly: + +``` +/cua-driver +``` + +## Files + +- `SKILL.md` — the main skill body (~500 lines). Loaded on first + invocation; stays in context for the session. +- `WEB_APPS.md` — browsers, Electron, Tauri (Chromium + WebKit). Loaded + on demand when SKILL.md's pointer is followed. +- `RECORDING.md` — trajectory recording / replay. Loaded on demand. +- `TESTS.md` — manual test scripts for end-to-end skill verification. + +## Troubleshooting + +- `cua-driver: command not found` → re-run the installer or add + `.build/CuaDriver.app/Contents/MacOS/` to `$PATH`. +- `No cached AX state for pid X window_id W` → element_index was + reused across turns, or across different windows of the same app. + Call `get_window_state({pid, window_id})` first in the same turn, + with the same window_id you're about to act against. +- Empty `tree_markdown` → `capture_mode` is set to `vision`, which + skips the AX walk by design. Flip back to the default `som` + (`cua-driver config set capture_mode som`) to get the tree. + Tiny screenshot → likely a stale window capture. See "Behavior + matrix" in SKILL.md for the full mode table. +- System-alert beep when pressing Return on a minimized Chrome + omnibox → the keyboard-commit-on-minimized limitation. Use + `set_value` on the field instead, or AX-click a Go/Submit button. + See `WEB_APPS.md`. + +## Updates + +The skill evolves alongside the driver. To update: + +```bash +cd /path/to/cua && git pull +# if you copied: re-copy +cp -R libs/cua-driver/Skills/cua-driver ~/.claude/skills/ +# if you symlinked: nothing needed +``` + +## License + +MIT. Same license as the parent `trycua/cua` repo. diff --git a/skills/cua-driver/RECORDING.md b/skills/cua-driver/RECORDING.md new file mode 100644 index 0000000..79fc526 --- /dev/null +++ b/skills/cua-driver/RECORDING.md @@ -0,0 +1,114 @@ +# Recording & replaying trajectories + +Session-scoped capture of action sequences + pre/post state, suitable +for demos, regression diffs, and training data. Invoked only when the +user explicitly asks to record — the skill does not auto-enable this. + +`set_recording` turns on a session-scoped trajectory recorder. While +enabled, every action-tool call (`click`, `right_click`, `scroll`, +`type_text`, `type_text_chars`, `press_key`, `hotkey`, `set_value`) +writes a numbered turn folder under a caller-chosen output +directory. Read-only tools (`get_window_state`, `list_windows`, +`screenshot`, `list_apps`, permission probes, agent-cursor getters / +setters, and `set_recording` itself) are not recorded. + +## Enable / disable + +Two equivalent surfaces: the `set_recording` MCP tool, or the +friendlier `cua-driver recording` subcommand group (wraps +`set_recording` + `get_recording_state` with human-readable output). + +``` +cua-driver recording start ~/cua-trajectories/run-1 +# … run the workflow … +cua-driver recording status # -> enabled / disabled, next_turn, output_dir +cua-driver recording stop # -> "Recording disabled (N turns captured in …)" +``` + +Raw-tool equivalent: + +``` +cua-driver set_recording '{"enabled":true,"output_dir":"~/cua-trajectories/run-1"}' +cua-driver get_recording_state +cua-driver set_recording '{"enabled":false}' +``` + +The `recording` subcommands require a running daemon (`cua-driver +serve &`) because recording state is per-process. `output_dir` expands +`~` and is created (with intermediates) if missing. Turn numbering +starts at `1` every time recording is (re-)enabled, regardless of any +existing contents in the directory. State lives in memory only — a +daemon restart resets to disabled. + +## What each turn folder contains + +Each action writes to `turn-NNNNN/` (five-digit zero-padded counter): + +- `app_state.json` — post-action AX snapshot for the target pid, same + shape `get_window_state` returns (tree_markdown, element_count, + turn_id, etc.) minus the screenshot fields. The recorder resolves a + frontmost window internally (visible + on-current-Space preferred, + max-area fallback) since individual action tools carry a + window_id but the recorder has no caller-supplied anchor. +- `screenshot.png` — post-action capture of the same window the + recorder just snapshotted. Omitted when the pid has no visible + window. +- `action.json` — the tool name, full input arguments, result + summary, pid, click point (when applicable), ISO-8601 timestamp. +- `click.png` — only for click-family actions (`click`, + `right_click`): a copy of `screenshot.png` with a red dot drawn at + the click point (screen-absolute point → window-local pixels via + the screenshot's `scale_factor`). Absent for other tools and for + clicks whose point falls outside the captured window. + +## When to use it + +- Demos and screen recordings — play the turn folder back to show + exactly what the agent saw and what it did. +- Replay for regression — re-run the same sequence against a future + build and diff the new trajectory against the saved one. +- Training data collection — each turn is a + `(state, action, next_state)` triple ready for offline learning. + +## When to invoke it + +This skill does **not** auto-enable recording. The client invokes +`set_recording` explicitly when the user asks to capture a session. +If the user says "record this session" or similar, call +`set_recording({enabled:true, output_dir:…})` before the first +action, and `set_recording({enabled:false})` when done. + +## Replaying a recorded trajectory + +`replay_trajectory({dir})` walks `/turn-NNNNN/` folders in +lexical order, reads each `action.json`, and re-invokes the recorded +tool with its recorded `arguments`. Optional knobs: `delay_ms` +(pacing between turns, default 500) and `stop_on_error` (halt on +first failure, default true). + +``` +cua-driver recording start ~/cua-trajectories/demo1 +# … run the workflow … +cua-driver recording stop +# Later: replay against a new build. +cua-driver replay_trajectory '{"dir":"~/cua-trajectories/demo1","delay_ms":500}' +``` + +Important caveat: **element_index doesn't survive across sessions**. +Indices are assigned fresh on every `get_window_state` snapshot, +keyed on `(pid, window_id)`, so a recorded +`click({pid, window_id, element_index: 14})` from yesterday won't +resolve today — the pid is usually different, the window_id always +is. The call returns `Invalid element_index` or `No cached AX +state`. Pixel clicks (`click({pid, x, y})`) and keyboard tools +(`press_key`, `type_text_chars`, `hotkey`, `type_text` without +element_index) replay cleanly; element-indexed actions require a +live snapshot that replay doesn't currently re-emit (read-only tools +like `get_window_state` aren't recorded). For a reliable replay, either +compose the trajectory from pixel + keyboard primitives, or capture +it as a regression artifact (compare the failure/success pattern +across builds) rather than a re-driving script. + +If recording is still enabled while replay runs, the replay is +itself recorded into the current output directory — that's the +intended regression-diff workflow. diff --git a/skills/cua-driver/SKILL.md b/skills/cua-driver/SKILL.md new file mode 100644 index 0000000..69885cc --- /dev/null +++ b/skills/cua-driver/SKILL.md @@ -0,0 +1,816 @@ +--- +name: cua-driver +description: Drive a native macOS app via the cua-driver CLI (default) or MCP server — snapshot its AX tree, click/type/scroll by element_index, verify via re-snapshot. Use when the user asks you to operate, drive, automate, or perform a GUI task in a real macOS application on the host (e.g. "open a file in TextEdit", "navigate to /Applications in Finder", "click the Save button in Numbers"). +--- + +# cua-driver + +Orchestrates macOS app automation via `cua-driver`. Whenever a user +asks to drive a native macOS app, follow the loop in this skill rather +than calling tools ad-hoc — the snapshot-before-action invariant is not +optional and silently breaks if you skip it. + +## The no-foreground contract — read this first + +**The user's frontmost app MUST NOT change.** This is the whole +reason cua-driver exists. Users pay for the right to keep typing in +their editor while an agent drives another app in the background. +Violate this rule and every other nice property the driver gives +you (no cursor warp, no Space switch, no window raise) stops +mattering — you just shipped the Accessibility Inspector with extra +steps. + +Before running any shell command, ask: **"does this raise, +activate, foreground, or make-key any app?"** If yes, don't run it. +Every one of the commands below activates the target on macOS and +is therefore forbidden unless the user **explicitly** asked for +frontmost state: + +- **Every form of the `open` CLI — `open -a `, `open -b + `, `open `, `open `, `open + ` — always activates.** macOS routes all forms through + LaunchServices, which unhides and foregrounds the target + regardless of whether you passed an app name, a bundle id, a + document, a URL, or the bundle path itself. The activation + happens even when the only intent was "start the process." + **Never use `open` for any app launch.** This includes launching + a just-built .app from a local build dir (e.g. `open + build/Build/Products/Debug/MyApp.app`) — resolve the + `CFBundleIdentifier` from `Info.plist` and use `launch_app` + with that id. See "The narrow carve-out" below for why + `launch_app` is safe even when the app internally calls + `NSApp.activate`. +- `osascript -e 'tell application "X" to activate'` — + activates by design. Same for `... to open `, + `... to launch`, and anything with `activate` in the tell block. +- `osascript -e 'tell application "System Events" to ... frontmost'` + in a mutating form (setting `frontmost` rather than reading it). +- AppleScript files that invoke `activate`, `launch`, or `open` + against the target app. +- `cliclick` (moves the user's real cursor to the target coords + before clicking — a focus-steal-equivalent even if the app's + window state is unchanged). +- `CGEventPost` with `cghidEventTap` targeting a coordinate over + a different app's window (warps the cursor, possibly activates + on hit). +- `AppleScriptTask`, `NSAppleScript`, `Process` wrapping `osascript` + that contains any of the above. +- `NSRunningApplication.activate(options:)` called from your own + helper binary — same class. +- Dock clicks and any `open` invocation (see the first bullet — + every form of `open` goes through LaunchServices which + activates, full stop). +- **Keyboard shortcuts that semantically mean "focus here" — + most notably Chrome / Safari / Arc's `⌘L` (focus omnibox) and + Finder's `⌘⇧G` (Go to Folder).** These aren't pure key events — + the receiving app interprets "user wants to type here" as + activation intent and raises its window to be key. Even when + delivered to a backgrounded pid via `hotkey`, the downstream app + pulls focus. **For omnibox navigation specifically**, don't + `hotkey ⌘L`; instead find the omnibox AX element via `som` snapshot + (`AXTextField` id=something like `location_bar` / `omnibox`) and + either AX-click it by `element_index` or dispatch `set_value` with + the URL directly — both stay backgrounded. The general principle: + a shortcut that says "put my cursor inside this app" is a + focus-steal; a shortcut that says "do this thing" (copy, save, + quit) is fine. +- **Tab-switching shortcuts in browsers (`⌘1..⌘9`, `⌘]`, `⌘[`, + `⌘⇧[`, `⌘⇧]`) are visibly disruptive even when delivered to a + backgrounded pid.** The app's key handler processes the shortcut, + the window re-renders the new tab's content, the user sees their + tabs flipping. There is no AX-only workaround: page content (HTML, + form state, `AXWebArea`) populates only for the focused tab; + inspecting a background tab requires activating it, which is the + visible flip. Observed with Dia; the same mechanic applies to every + Chromium-family browser (Chrome, Arc, Brave, Edge). + + **Prefer the windows-over-tabs pattern**: for each URL you need to + drive backgrounded, use `launch_app({bundle_id, urls: [url]})` — + browsers open each URL in a new **window**. Each window has its own + `window_id`, its own AX tree, and can be inspected / interacted with + via `element_index` without activating or switching anything. Tabs + are a UX grouping for humans; cua-driver workflows should default to + windows. See `WEB_APPS.md` → "Tabs vs windows" for the full pattern. + + Tab-title enumeration (read-only) IS safe — walk a window's toolbar + AX tree for `AXTab` / `AXRadioButton` children and read their + `AXTitle`s. Tab switching (activating one) is not. + +Reading frontmost state is fine (`osascript -e 'tell application +"System Events" to get name of first application process whose +frontmost is true'`). Mutating it is not. + +**Corollary — the AXMenuBar rule.** `AXMenuBarItem` + AXPick +dispatches at the AX layer regardless of which app is frontmost, +but macOS's on-screen menu bar always belongs to the frontmost +app. If you drive a *backgrounded* app's menu bar, the AX call +succeeds but the viewer sees the dispatch rendered over the +*frontmost* app's menu bar — confusing in any observed session and +routinely a silent no-op too, because action menu items go +`DISABLED` when their owning app isn't the key window. **So: only +use menu-bar navigation when the target is already frontmost.** For +backgrounded targets, read state via in-window AX (window title, +toolbar `AXStaticText`) and dispatch via in-window `element_index` +or pixel clicks — both paths are frontmost-insensitive. Full +rationale in "Navigating native menu bars" below. + +**"Open \" in user speech means launch, not activate.** +`cua-driver launch_app` is the one correct path for process +startup — it's idempotent (no-op on a running app), returns the +pid, and has an internal `FocusRestoreGuard` that catches +`NSApp.activate(ignoringOtherApps:)` calls the target makes during +`application(_:open:)` and clobbers the frontmost back to what it +was before the launch. That guard is why `launch_app` with `urls` +(e.g. `{"bundle_id": "com.colliderli.iina", "urls": ["~/video.mp4"]}`) +is safe even for apps that normally foreground on media-load +(Chrome, Electron, media players). + +## Defaults — always prefer cua-driver over shell shims + +**Default transport is the `cua-driver` CLI** — `Bash` shelling out +to `cua-driver ''`. MCP tools (prefix +`mcp__cua-driver__*`) only when the user explicitly asks for them. +CLI wins because it picks up rebuilds instantly, failures are +easier to diagnose, and there's no per-tool schema-load overhead. + +Every reference to `click(...)`, `get_window_state(...)` etc. in this +skill means `cua-driver click '{...}'` — translate to MCP form only +when MCP is requested. + +Intent → tool mapping. If you find yourself reaching for the right +column, something has gone wrong — re-read "The no-foreground +contract" above: + +| Intent | Use | Don't use | +|---|---|---| +| Open / launch an app | `launch_app({bundle_id})` or `launch_app({bundle_id, urls:[...]})` | `open -a`, `osascript 'tell app … to launch/activate/open'` | +| Find a pid | `list_apps` or `launch_app`'s return | `pgrep`, `ps`, `osascript frontmost` | +| Enumerate an app's windows | `list_windows({pid})` — or read the `windows` array `launch_app` already returns | `osascript 'every window of app …'` | +| Click / type / scroll / keys | `click`, `type_text`, `scroll`, `press_key`, `hotkey` | `osascript`, `cliclick`, raw `CGEvent`, `open ` | +| Screenshot | `screenshot` or the PNG in `get_window_state` | `screencapture` | +| Quit an app | ask the user first, then `hotkey({pid, keys:["cmd","q"]})` | `kill`, `killall`, `pkill` | +| Hand a file/URL to an app | `launch_app({bundle_id, urls:[]})` | `open -a `, `open ` | + +### The narrow carve-out + +The **only** legitimate use of `osascript -e 'tell app X to +activate'` is when the user **explicitly** asked for frontmost +state ("bring Chrome to the front", "make it frontmost", "I want +to see X"). Reaching for it because a tool call returned something +confusing is wrong — that's the skill's classic foot-in-the-door +failure mode and it steals focus every time. + +When a cua-driver call surprises you, diagnose cua-driver first: + +- **Tiny screenshot / empty `tree_markdown`?** Check + `cua-driver get_config` → `capture_mode`. Default `"vision"` omits + the AX tree (PNG only), `"ax"` omits the PNG, `"som"` returns + both. If a snapshot lacks a tree, `capture_mode` is almost + certainly `"vision"` — either reason purely from the PNG or flip + to `"som"` / `"ax"` via `set_config`. +- **`has_screenshot: false`?** The window capture failed (transient + race against a close, or the window has no backing store yet). + Re-snapshot; if persistent, pick a different `window_id` via + `list_windows`. +- **`Invalid element_index` / `No cached AX state`?** You either + skipped `get_window_state` this turn or passed a different + `window_id` than the one the snapshot cached against. The cache + is keyed on `(pid, window_id)` — indices don't carry across + windows of the same app. Re-snapshot with the same window_id + you're about to click in. +- **Sparse Chromium AX tree?** Retry `get_window_state` once — the + tree populates on second call. + +Only after those are ruled out, and only if the user's action +genuinely needs frontmost state, fall through to the activate +fallback. Always name the focus steal in your response ("I'll +briefly bring Chrome to the front because …"). + +### Self-check pattern + +Before every `Bash` call whose command line touches any macOS app +(launching, opening, clicking, typing, scripting, screenshotting), +run the self-check: + +1. **Does this command foreground the target?** If yes — stop and + translate to the cua-driver equivalent from the mapping table. +2. **Does this command move the user's real cursor?** (`cliclick`, + any `CGEventPost` at `cghidEventTap` over another app's window). + If yes — stop; use `click({pid, x, y})` which routes per-pid + via SkyLight and never warps the cursor. +3. **Does this command bypass cua-driver entirely?** (`osascript` + mutating GUI state, AppleScript files, external helpers.) If + yes — stop; find the cua-driver tool that does the intent. + +If all three are "no," the command is safe. If you can't answer, +default to stop and ask rather than proceed. A single `open -a` +run by accident kills the demo, the trust, and the user's in-flight +editor state. + +## Prerequisites — check before starting + +1. `cua-driver` is on `$PATH` (`which cua-driver`). If not, point the + user at `scripts/install-local.sh` and stop. +2. Run `cua-driver check_permissions`. If either grant is `false`, tell + the user to open System Settings → Privacy & Security and grant + Accessibility and Screen Recording to `CuaDriver.app`, then stop. + (`cua-driver check_permissions '{"prompt":true}'` raises the system + dialogs, but only do that if the user asks — it steals focus.) +3. Start the daemon with `open -n -g -a CuaDriver --args serve` (the + recommended form — goes through LaunchServices so TCC attributes + the process to CuaDriver.app). `cua-driver serve &` also works; + the CLI auto-relaunches through `open -n -g -a CuaDriver` when it + detects a wrong-TCC context (any IDE-spawned shell: Claude Code, + Cursor, VS Code, Conductor). Verify with `cua-driver status`. + +## Using cua-driver from the shell + +Tool names are `snake_case`, management subcommands are +`kebab-case` — no ambiguity. Tools invoked as `cua-driver + ''`. Management subcommands: + +- `open -n -g -a CuaDriver --args serve` — start persistent daemon + (**required** for `element_index` workflows; without it each CLI + invocation spawns a fresh process and the per-pid element cache + dies between calls). `cua-driver serve &` also works — the CLI + auto-relaunches via `open` when the shell's TCC context is wrong. + Pass `--no-relaunch` / `CUA_DRIVER_NO_RELAUNCH=1` to opt out. +- `cua-driver stop` / `status` +- `cua-driver list-tools`, `describe ` +- `cua-driver recording start|stop|status` — see `RECORDING.md` + +Canonical multi-step workflow: + +``` +open -n -g -a CuaDriver --args serve +cua-driver launch_app '{"bundle_id":"com.apple.calculator"}' +# → {pid: 844, windows: [{window_id: 10725, ...}]} +cua-driver get_window_state '{"pid":844,"window_id":10725}' +cua-driver click '{"pid":844,"window_id":10725,"element_index":14}' +cua-driver stop +``` + +## Agent cursor overlay + +Visual cursor overlay for demos and screen recordings. Default: +enabled. Toggle with `cua-driver set_agent_cursor_enabled +'{"enabled":true|false}'`. A triangle pointer Bezier-glides to each +click target, ring-ripples on landing, idle-hides after ~1.5s. +Motion knobs: `set_agent_cursor_motion` takes any subset of +`start_handle`, `end_handle`, `arc_size`, `arc_flow`, `spring` — +tuneable at runtime, persisted to config. + +Requires an AppKit runloop, which `cua-driver serve` / `mcp` +bootstraps. One-shot CLI invocations skip the overlay entirely. + +## The core invariant — snapshot before AND after every action + +**Every action MUST be bracketed by `get_window_state(pid, window_id)`**: + +- **Before** — the pre-action snapshot resolves the `element_index` + you're about to use. Indices from previous turns are stale; the + server replaces the element index map on every snapshot, keyed + on `(pid, window_id)`. Indices from turn N don't resolve in turn + N+1, and indices from window A don't resolve against window B of + the same app. Skip this and element-indexed actions fail with + `No cached AX state`. +- **After** — the post-action snapshot verifies the action actually + landed. Without it you can't tell a silent no-op from a real + effect. The AX tree change (new value, new window, disappeared + menu, disabled button, etc.) is your evidence that the action + fired. If nothing changed, the action probably failed silently — + say so, don't assume success. + +This applies to pixel clicks too — re-snapshot after to confirm the +click landed on the intended target. + +### Why window selection is the caller's job now + +`get_app_state` used to pick a window for you via a max-area heuristic +that returned the wrong surface on apps with large off-screen utility +panels. Concrete reproducer: IINA's OpenSubtitles helper (600×432 +off-screen) out-area'd the visible 320×240 player window, so +`get_app_state(pid)` screenshot'd the invisible panel and clicks landed +there silently. The new `get_window_state(pid, window_id)` makes the +caller name the window explicitly — the driver validates that the +window belongs to the pid and is on the current Space, then snapshots +exactly what was asked for. Enumerate candidates via `list_windows` or +read the `windows` array `launch_app` already returns. + +## Behavior matrix + +Two orthogonal axes shape what the agent can do. + +**capture_mode → addressing mode** + +| `capture_mode` | `get_window_state` returns | Use for actions | +|---|---|---| +| **`som`** (default) | tree + screenshot | `element_index` preferred; pixel fallback | +| **`ax`** | tree only (no PNG) | `element_index` only | +| **`vision`** | PNG only (no tree) | pixel only — see [SCREENSHOT.md](./SCREENSHOT.md) | + +`vision` was renamed from `screenshot` — the old name still decodes +as a deprecated alias, so an on-disk `"capture_mode": "screenshot"` +keeps working. Default is `som` so element_index clicks work the +first time a user calls `get_window_state`; the other modes are +opt-in when the caller specifically doesn't want one half of the +work. Note the tool named `screenshot` is separate (raw PNG, no AX +walk) and unrelated to the capture mode. + +When a snapshot looks wrong (tiny screenshot / empty tree), check +`cua-driver get_config` for `capture_mode` before anything else. + +Pure-vision mode has its own caveats — Claude Code's vision +pipeline downsamples dense text aggressively, so pixel grounding +takes multiple correction cycles on text-heavy UIs. Read +[SCREENSHOT.md](./SCREENSHOT.md) before driving anything in that +mode; it documents the iterate/annotate/verify recipe plus the +JPEG-over-PNG finding. + +**Window state → what works** + +| state | `get_window_state` | `click`/`set_value` (AX) | `press_key` commit (Return/Space/Tab) | pixel click | +|---|---|---|---|---| +| frontmost | ✅ | ✅ | ✅ | ✅ | +| backgrounded / visible | ✅ | ✅ | ✅ | ✅ | +| **minimized** (Dock genie) | ✅ | ✅ (no deminiaturize — AX actions fire on the minimized window in place) | ❌ silent no-op / system beep — use `set_value` or click equivalent | ❌ no on-screen bounds | +| hidden (`hides=true` / `NSApp.hide`) | ✅ | ✅ | depends | ❌ | +| on another Space | ⚠️ AX tree often stripped to menu-bar-only on SwiftUI apps (System Settings) — AppKit apps usually fine. Response carries `off_space: true` + `window_space_ids` so you can detect it | ✅ | ✅ | ❌ window not in current-Space list | + +**Critical cell — minimized + keyboard commit.** The keystroke +reaches the app but AX focus doesn't propagate to renderer focus on +a minimized window. Workarounds in order of preference: +`set_value` to write the field's entire value directly, or AX-click +a commit-equivalent button (Go, Submit, checkbox). Tell the user +the window needs to un-minimize only as a last resort. + +## The canonical loop + +``` +launch_app(target) + → pick window_id from the returned `windows` array + (or call list_windows(pid) separately) + → get_window_state(pid, window_id) + → [act] # every action also takes (pid, window_id) + → get_window_state(pid, window_id) → verify +``` + +`launch_app` now returns a `windows` array alongside the pid, so the +common case collapses to two calls (`launch_app` → `get_window_state`) +without a separate `list_windows` hop. + +### 1. Resolve target pid — always via `launch_app` + +**Always start with `launch_app`**, whether or not the target is already +running. It's idempotent (relaunching returns the existing pid with no +side effects) and gives you the pid in one call — no `list_apps` hop. + +- `launch_app({bundle_id: "com.apple.finder"})` — preferred, unambiguous. +- `launch_app({name: "Calculator"})` — when bundle_id isn't known. + +`launch_app` is a **hidden-launch primitive by design** — that's the +entire point of cua-driver: agents drive apps in the background while +the user keeps typing in their real foreground app. The target's +window is initialized (AX tree fully populated, clickable via +`element_index`, the pid appears in `list_apps`) but not drawn on +screen. The driver never activates or unhides apps on its own; that +would violate the no-foreground contract the whole driver exists to +protect. + +If the user explicitly wants the window visible (usually for a demo +or recording), they unhide it themselves — Dock click, Cmd-Tab, or +Spotlight. Do not reach for `open` / `osascript activate` as a +shortcut to make the window visible; those paths break the backgrounded +invariant on every call, not just the call that "needed" the +foreground. Say out loud what the user needs to do ("click the +Todo app in your Dock to bring it forward") and let them do it. + +Never shell out to **any** form of `open` (including `open +` for a just-built binary — resolve the bundle id +from `Info.plist` and use `launch_app` with that), `osascript 'tell +app … to launch/open'`, or similar. Those paths activate the target, +bypass the driver's focus-restore guard, and require a Bash +permission prompt the agent loop shouldn't be burning on app launch. +See "Prefer cua-driver tools over shell shims" above for the full +intent → tool mapping. + +`list_apps` is for app-level discovery (answering "what's installed / +running / frontmost?") — not part of the core action loop. Skip it in +the loop. For **window-level** questions — "does this app have a +visible window?", "which Space is this window on?", "which of this +pid's windows is the main one?" — call `list_windows` instead; the +app record doesn't carry window state on purpose. In the common +single-window case you can skip `list_windows` entirely and read the +`windows` array that `launch_app` already returned. + +### 2. Snapshot and act by element_index + +Call `get_window_state({pid, window_id})` with the `window_id` from +`launch_app`'s `windows` array (or a fresh `list_windows({pid})` if +you're interacting with a long-lived process). In the default +`vision` capture_mode the response carries **only the screenshot** +— no AX tree — so the canonical loop is `list_windows → +get_window_state → reason over PNG → pixel click`. When you need +`element_index` dispatch (AX-addressable elements, backgrounded +clicks), flip to `som` first: `cua-driver set_config '{"key": +"capture_mode", "value": "som"}'`, or call `get_accessibility_tree` +directly. The rest of this section walks through `som` mode, which +is what you want once you've decided element-indexed addressing is +required. + +In `som` mode the response carries: + +- `tree_markdown` — every actionable element tagged `[N]`. That `N` + is the `element_index`. The tree can be very large (Finder is + ~1600 elements, ~190 KB); when it exceeds token limits the MCP + harness saves it to a file and returns the path. Use `Bash` + + `jq -r '.tree_markdown'` + `grep` to pull the section you need. +- `screenshot_png_b64` + `screenshot_width` / `_height` / + `_scale_factor` — the window screenshot (actually JPEG-85 despite + the `_png_` field name, hard-coded in + `WindowCapture.captureFrontmostWindow`). Present in `som` mode + (spliced into the structured JSON alongside the tree). In `vision` + mode the image arrives as a native MCP image content block with no + structured wrapper. Omitted when the target has no on-screen + window. +- `has_screenshot: bool` — **gate on this before piping the PNG**. + Otherwise `jq -r '.screenshot_png_b64'` emits the literal + `"null"`, base64-decodes into 3 bytes of garbage, and downstream + vision APIs reject it with an opaque "Could not process image" + error. + +``` +# canonical, works in every capture mode — writes the image bytes +# wherever you point, stdout stays readable (tree in som, summary +# in vision). stderr warns (exit 0) if the response had no image. +cua-driver get_window_state '{"pid":N,"window_id":W}' --image-out /tmp/shot.png + +# som-only legacy path: pull the spliced base64 out of structuredContent. +# Prefer --image-out above — it's one flag vs a probe + pipe. +if [ "$(cua-driver get_window_state '{"pid":N,"window_id":W}' | jq -r '.has_screenshot')" = "true" ]; then + cua-driver get_window_state '{"pid":N,"window_id":W}' | jq -r '.screenshot_png_b64' | base64 -d > shot.png +fi +``` + +**Reason over both the tree AND the screenshot — they're +complementary, not redundant.** In `som` mode every +turn's `get_window_state` gives you both halves and you should pull +signal from each: + +- The **AX tree** tells you *what's clickable* — roles, labels, + `element_index` handles, advertised actions, parent-child + structure. This is the ground truth for dispatching. +- The **screenshot** tells you *which one* — the tree often has + many buttons with similar or empty labels ("Delete", "OK", + anonymous UUID-labeled buttons, five `AXStaticText = " "`), and + visual context disambiguates. Captions, colors, layout relationships + visible in pixels often don't show up in the AX tree at all + (especially in Chromium / Electron / web content). + +Canonical pattern: look at the screenshot to decide "the blue +Subscribe button on the top-right of the video card", then walk the +tree to find the matching `AXButton` and dispatch by its +`element_index`. Don't try to do it from just the tree — you'll +pick the wrong element when labels repeat. Don't try to do it from +just the screenshot — you lose the reliable AX-action path and the +safe backgrounded-dispatch. + +Reach for pixel coordinates only when the target is a canvas / +video / WebGL / custom-drawn surface that isn't in the AX tree +(see Pixel-coordinate clicks below). + +The `actions=[...]` list on each element is **advisory**, not +authoritative. cua-driver does not pre-flight check against it — +`click({pid, element_index})` always attempts `AXPress` (or the +action you pass) and surfaces whatever the target returns. Many +apps accept `AXPress` on elements that don't advertise it — Chrome's +omnibox suggestion `AXMenuItem` is a live example. **Try the click +first** — pivot only on the returned AX error code. + +Dispatch table (every row assumes a `(pid, window_id)` pair from the +last `get_window_state`; `window_id` is required alongside +`element_index`, ignored on pixel-only forms unless you want to +anchor the conversion against a specific window): + +| Intent | Tool | Notes | +|---|---|---| +| List an app's windows | `list_windows({pid})` | returns `window_id`, `title`, `bounds`, `z_index`, `is_on_screen`, `on_current_space`. Already included in `launch_app`'s response — only call this for long-lived pids | +| Snapshot a window | `get_window_state({pid, window_id})` | returns `tree_markdown` + `screenshot_*`; populates the `(pid, window_id)` element_index cache | +| Left click | `click({pid, window_id, element_index})` | default `action: "press"`. Pixel form: `click({pid, x, y})` (window_id optional — when supplied, pinpoints the anchor window) — `modifier: ["cmd"]` | +| Double-click / open | `double_click({pid, window_id, element_index})` | AXOpen when advertised (Finder items, openable rows); else stamped pixel double-click at the element's center. Pixel form: `double_click({pid, x, y})` — primer-gated recipe lands on backgrounded Chromium web content (YouTube fullscreen, Finder open-on-dbl). `click({..., count: 2})` still works and routes through the same recipe; `double_click` is the intent-first spelling | +| Right click / context menu | `right_click({pid, window_id, element_index})` or `click({pid, window_id, element_index, action: "show_menu"})` | Chromium web-content coerces pixel right-click to left — see `WEB_APPS.md` | +| Type at cursor | `type_text({pid, text, window_id, element_index})` | `AXSelectedText` write; focuses first | +| Set whole field value | `set_value({pid, window_id, element_index, value})` | sliders, steppers, text fields; **use for keyboard-commit workarounds on minimized windows** | +| Scroll | `scroll({pid, direction, amount, by, window_id, element_index})` | synthesizes PageUp/PageDown/arrows via SLEventPostToPid | +| Focus + send key | `press_key({pid, key, window_id, element_index, modifiers})` | element_index sets AXFocused, then posts key | +| Send key to pid | `press_key({pid, key, modifiers})` | no focus change; key goes to pid's current focus | +| Modifier combo | `hotkey({pid, keys})` | e.g. `["cmd","c"]`; posted per-pid, not HID tap | +| Unicode keystrokes | `type_text_chars({pid, text, delay_ms})` | CGEvent-to-pid; reaches Chromium/Electron inputs | + +**All keyboard/text primitives require `pid`.** There is no +frontmost-routed variant — every key goes to the named target via +`CGEvent.postToPid`, so the driver cannot leak keystrokes into the +user's foreground app. + +**Why `element_index` is the primary path:** works on hidden / +occluded / off-Space windows, no focus steal, stable across +rebuilds, labels tell you what you're clicking. Reach for pixel +coordinates only when AX can't. + +### Pixel-coordinate clicks + +The pixel path (`click({pid, x, y})`) is for surfaces the AX tree +doesn't reach — canvases, video players, WebGL, custom-drawn controls. +Coords are **window-local screenshot pixels** (same space as the PNG +`get_window_state` returns). Top-left origin, y-down. The driver +handles screen-point conversion internally. Passing `window_id` +alongside `x, y` is optional but recommended — it pins the +coordinate conversion to the window whose screenshot produced the +pixel, rather than the driver's heuristic choice. + +#### Reading coordinates from the PNG + +PNGs returned by `get_window_state` are capped at **1568 px +long-side by default** (`max_image_dimension` config), matching +Anthropic's multimodal-vision downsampling limit. That means the +image the model reasons over and the image the click tool's +coordinate system lives in are the **same resolution** — just look +at the PNG, pick a pixel, click at that pixel. No scaling math. + +This is the default because the mismatch between "rendered +thumbnail" and "native PNG" was a recurring coord-estimation +footgun. If you opt out (explicit `max_image_dimension=0` for +pixel-perfect verification flows), the old rule applies: don't +eyeball coords from whatever your client renders — it may be +2-4× smaller than the PNG on disk, and a 2% error in thumbnail +space becomes ~80 px in the real image. Use the crosshair recipe +below against the full-resolution file in that case. + +1. `get_window_state({pid, window_id})` returns an image capped + at 1568 long-side (default) plus its dimensions + (`screenshot_width` / `screenshot_height`). Write the bytes to + disk with `--image-out ` in any capture mode — works + identically in `vision` (where it's the only way) and `som` + (where it sidesteps the jq + base64 dance on the spliced + `screenshot_png_b64` field). +2. You are a multimodal model — look at the PNG. Since the PNG + matches what you see, pick the target pixel directly. No + fractional math needed. +3. When precision matters (small targets, dense UIs), draw a + crosshair on the image (do **not** crop — cropping loses the + coordinate system and requires error-prone offset math) and + show it before clicking: + +```python +from PIL import Image, ImageDraw +img = Image.open('/tmp/shot.png') +draw = ImageDraw.Draw(img) +x, y = +r = 18 +draw.ellipse([x-r, y-r, x+r, y+r], outline='red', width=4) +draw.line([x-30, y, x+30, y], fill='red', width=3) +draw.line([x, y-30, x, y+30], fill='red', width=3) +img.save('/tmp/shot_annotated.png') +``` + +4. Only dispatch the click after the user (or your own re-read of + the annotated image) confirms the crosshair is on target. + +#### Addressing variants + +- `click({pid, x, y})` — single left-click. +- `click({pid, x, y, count: 2})` — double-click. +- `click({pid, x, y, modifier: ["cmd"]})` — cmd-click. Accepts any + subset of `cmd/shift/option/ctrl`. +- `right_click({pid, x, y})` — also takes `modifier`. + +The pixel path animates the agent cursor overlay but never warps +the real cursor. If the pid has no on-screen window the call errors +with `pid X has no on-screen window` — you need a visible window to +anchor the conversion. + +#### How the pixel click is dispatched + +The recipe is the backgrounded "noraise" sequence: yabai's +focus-without-raise SLPS event records followed by an off-screen +user-activation primer and the real click, all stamped via +`SLEventPostToPid`. The target app becomes AppKit-active for event +routing but its window does **not** rise to the front of the +z-stack, and macOS's "switch to Space with windows for app" follow +is suppressed. Full mechanics in +`Sources/CuaDriverCore/Input/MouseInput.swift` (`clickViaAuthSignedPost`) +and the companion `FocusWithoutRaise.swift`. + +#### Known limits + +- **Chromium `