diff --git a/.changeset/ax-box-model.md b/.changeset/ax-box-model.md new file mode 100644 index 0000000..eb2a947 --- /dev/null +++ b/.changeset/ax-box-model.md @@ -0,0 +1,30 @@ +--- +"macos-vision": minor +--- + +feat: `axTree()` — box model of a live application from the accessibility tree + +Returns element boxes, hierarchy, roles and labels for a running app, with +optional colours sampled from a capture and typography from the AX attributed +string. Geometry is measured rather than inferred from OCR, so it is exact. + +Shipped as a fourth prebuilt native helper (`ax-helper`) through the existing +pipeline, so nothing needs compiling on the user's machine. + +Cost is bounded deliberately, because every attribute read is a synchronous IPC +round trip into the target app and that app's implementation — not tree size — +dominates: the same 4000 elements measured 1.6s in Safari and 11s in Finder. +Attribute reads are batched, offscreen subtrees are culled, `maxElements` and +`maxDepth` cap the walk, and `budget` reports what happened including +`capped: true` — a truncated tree is never presented as a complete one. + +`detail: 'content'` (default) drops unlabelled structural containers and +re-parents their children, halving the payload on a Finder window (600 → 289 +nodes). Boxes are encoded as `[x, y, w, h]` and default-valued fields are +omitted, for the same reason. + +Also fixes `captureScreen()` silently accepting an invalid region: given a +negative or fully offscreen rect, `screencapture` clamps and returns a tiny +image with exit 0 on an unlocked Mac while failing elsewhere, so a caller's +mistake surfaced as corrupt output. The rect is now validated against the actual +display bounds before capture. diff --git a/README.md b/README.md index f55d00c..92af843 100644 --- a/README.md +++ b/README.md @@ -261,6 +261,86 @@ All coordinates are **global screen points with a top-left origin** — the same --- +## API — Box model (accessibility tree) + +`axTree()` returns the on-screen layout of a running application: element boxes, +hierarchy, roles and labels from the accessibility API, optionally with colours +sampled from a capture and typography from the AX attributed string. Geometry is +**measured**, not inferred from OCR bounding boxes. + +```ts +import { axTree, captureScreen } from 'macos-vision'; + +const tree = await axTree({ app: 'Safari' }); +// { app, pid, window: [x,y,w,h], source: 'ax', budget: {...}, nodes: [...] } + +// With colours and fonts, from a capture of the same window: +const shot = await captureScreen({ app: 'Safari' }); +const full = await axTree({ + app: 'Safari', + colors: { path: shot.path, frame: shot.frame }, + typography: true, +}); +``` + +A node: + +```jsonc +{ + "id": 42, "parent": 7, "depth": 5, + "role": "Button", "label": "Zapisz", + "box": [812, 540, 96, 32], // [x, y, w, h] in screen points + "style": { "bg": "#2F6FEB", "border": "#1B4FC4", "borderWidth": 1 }, + "text": { "font": "SFPro-Semibold", "family": "SF Pro", "size": 13, "align": "center" } +} +``` + +`box` is an array rather than a keyed object because the same four numbers repeat +on every node; keys would cost roughly four times the tokens. `enabled` appears +only when `false` and `focused` only when `true`, for the same reason. + +### Cost, and how it is bounded + +Every attribute read is a synchronous IPC round trip into the target app, so cost +tracks that app's accessibility implementation rather than tree size — the same +4000 elements measured **1.6 s in Safari and 11 s in Finder**. Three things keep +it bounded: attribute reads are batched, subtrees outside the window's visible +rect are culled, and `maxElements` / `maxDepth` cap the walk. `budget` reports +what happened, including `capped: true`, so a truncated tree is never mistaken +for a complete one. + +`detail: 'content'` (the default) drops unlabelled structural containers and +re-parents their children. Boxes are absolute, so the nesting adds little for a +reader and roughly halves the payload — measured 600 → 289 nodes on a Finder +window. + +> **A full tree is not a token saving over a screenshot.** A dense window runs to +> thousands of tokens either way; measured on Finder, a pruned 289-node tree is +> ~7.3k tokens against ~6.9k for the image. The reason to use it is what a +> screenshot cannot give — exact boxes, roles, enabled state, hierarchy — and the +> fact that you can take a slice (`maxElements`, one window, one subtree) instead +> of the whole thing. + +### Limits, stated plainly + +- **This is not the CSS box model.** CSS has four nested boxes; AX has one. + `borderWidth` is inferred by an edge scan and there is no padding or margin. +- **Colours come from pixels**, so an occluded element reports whatever is drawn + on top of it, and gradients or shadows are approximations. +- **Typography depends on the app.** `AXAttributedStringForRange` returns real + font data where it is implemented (TextEdit gives `Menlo-Regular` 11 pt); web + content in Safari exposes alignment but no font. +- **Requires Accessibility permission** for the host process, separately from + Screen Recording. Without it `axTree()` throws with that reason. +- For web pages, Chrome DevTools `DOM.getBoxModel` and `CSS.getComputedStyleForNode` + return the real thing and are strictly better. This is for native apps, Electron, + canvas/WebGL, games and mockups. + +See [`docs/BOX-MODEL.md`](docs/BOX-MODEL.md) for the measurements behind these +numbers. + +--- + ## API — Markdown pipeline (VisionScribe) `VisionScribe` converts an image or PDF to Markdown by combining Apple Vision OCR with a local LLM (via Ollama). The LLM never sees the image — it only formats text that Vision already extracted. This keeps image processing local and reduces the risk of vision-model hallucinations, but Markdown reconstruction is still best-effort and depends on the local model and document complexity. diff --git a/docs/BOX-MODEL.md b/docs/BOX-MODEL.md index 16c0229..881f3f6 100644 --- a/docs/BOX-MODEL.md +++ b/docs/BOX-MODEL.md @@ -4,7 +4,7 @@ Goal: hand an LLM a compact JSON description of what is on screen — element bo colours, borders, typography — complete enough that the model can reconstruct the layout, review it, or write assertions against it, **without ever seeing the screenshot**. -Status: design only, nothing implemented. Every number below was measured on this machine +Status: **phases 1–3 implemented** as `axTree()` (see the README). Phase 4 — merging with OCR to populate `unresolved` — is still open. Every number below was measured on this machine (Apple M1 Pro, 16 GB, macOS 26.5.2) with throwaway probes, not estimated. --- @@ -160,7 +160,29 @@ a web page. --- -## 6. Suggested order of work +## 6. What implementation actually cost + +Built as `ax-helper` + `src/ax.ts`. Findings that only appeared once it ran: + +- **The naive JSON was more expensive than the screenshot it replaces.** A + 250-node Safari tree came to ~12.8k tokens against ~6.9k for the image. + Encoding `box` as `[x, y, w, h]` instead of a keyed object, and omitting + `enabled: true` / `focused: false`, cut that by 44%. Pruning unlabelled + containers (`detail: 'content'`) took another 48% — 600 → 289 nodes on Finder. + Net: ~25 tokens per node instead of ~51. +- **A full tree still is not a token win over a screenshot** — a pruned Finder + window is ~7.3k tokens against ~6.9k for the image. The case for it is + capability (exact boxes, roles, enabled state, hierarchy) and the ability to + take a slice, not raw token count. The README says so rather than implying a + saving that does not exist. +- **Swift omits `nil` rather than encoding `null`**, so the root node has no + `parent` key at all. The TypeScript type said `number | null`; a consumer + checking `=== null` would have been wrong. Caught by a test, fixed in the type. +- **`AXAttributedStringForRange` is app-dependent.** TextEdit returns + `Menlo-Regular` 11 pt; Safari's web `StaticText` returns alignment but no font. + That is a limit of the source, not of the reader. + +## 7. Suggested order of work 1. **`ax-helper` with tree walk only** — role, frame, hierarchy, label, enabled. Batched reads, viewport culling, explicit budget, messaging timeout. This alone is ~80% of the value, because diff --git a/scripts/build-native-cross.js b/scripts/build-native-cross.js index a5d9829..2fec805 100644 --- a/scripts/build-native-cross.js +++ b/scripts/build-native-cross.js @@ -11,7 +11,7 @@ const TARGETS = [ { arch: 'arm64', swift: 'arm64-apple-macos12' }, { arch: 'x64', swift: 'x86_64-apple-macos12' }, ]; -const HELPERS = ['vision-helper', 'pdf-helper', 'ui-helper']; +const HELPERS = ['vision-helper', 'pdf-helper', 'ui-helper', 'ax-helper']; // Symbols added in newer SDKs are absent when building against an older one, so // the helper gates them on -DSDK_nn. Detect what this machine's SDK provides. diff --git a/scripts/install-native.js b/scripts/install-native.js index 3bc37e0..841f4d6 100644 --- a/scripts/install-native.js +++ b/scripts/install-native.js @@ -17,7 +17,7 @@ const root = path.resolve(__dirname, '..'); const pkg = JSON.parse(readFileSync(path.join(root, 'package.json'), 'utf8')); const binDir = path.join(root, 'bin'); -const HELPERS = ['vision-helper', 'pdf-helper', 'ui-helper']; +const HELPERS = ['vision-helper', 'pdf-helper', 'ui-helper', 'ax-helper']; // Symbols added in newer SDKs are absent when building against an older one, so // the helper gates them on -DSDK_nn. Detect what this machine's SDK provides. diff --git a/src/ax.ts b/src/ax.ts new file mode 100644 index 0000000..4a29be0 --- /dev/null +++ b/src/ax.ts @@ -0,0 +1,133 @@ +// Accessibility tree of a running application, shaped as a box model. +// +// Geometry and semantics come from the AX API — measured, not inferred from OCR. +// Colours are sampled from a capture you supply; typography comes from the AX +// attributed string where the app exposes it. See docs/BOX-MODEL.md for what is +// fact and what is estimate. + +import { AX_BIN, runHelper } from './helper.js'; +import type { ScreenFrame } from './ui.js'; + +/** + * `[x, y, w, h]` in global screen points, top-left origin — the space click + * drivers use. An array rather than a keyed object: the same four numbers cost + * roughly 4x fewer tokens when repeated across a whole tree. + */ +export type AxBox = [x: number, y: number, w: number, h: number]; + +export interface AxStyle { + /** Dominant fill colour of the element's interior, as #RRGGBB. */ + bg?: string; + /** Outline colour, present only when it differs from the fill. */ + border?: string; + /** **Inferred** by walking inward from the edge — an estimate, not a measured value. */ + borderWidth?: number; +} + +export interface AxTypography { + /** PostScript name, e.g. `Menlo-Regular` */ + font?: string; + family?: string; + size?: number; + align?: 'natural' | 'left' | 'right' | 'center' | 'justified'; +} + +export interface AxNode { + id: number; + /** Absent on the root of the walk; otherwise the id of the nearest kept ancestor. */ + parent?: number; + depth: number; + /** AX role without the `AX` prefix: `Button`, `StaticText`, `WebArea`… */ + role: string; + subrole?: string; + /** Title, falling back to the accessibility description. */ + label?: string; + value?: string; + /** Present only when `false` — enabled is the overwhelming default. */ + enabled?: boolean; + /** Present only when `true`. */ + focused?: boolean; + box: AxBox; + style?: AxStyle; + text?: AxTypography; +} + +export interface AxBudget { + elements: number; + /** True when the walk stopped at `maxElements`/`maxDepth` — the tree is incomplete. */ + capped: boolean; + maxElements: number; + maxDepth: number; + elapsedMs: number; + /** Elements skipped because they fell outside the window's visible rect. */ + culled: number; +} + +export interface AxTree { + app: string; + pid: number; + /** Frame of the window that was walked. */ + window?: AxBox; + /** `ax` when geometry only, `ax+px` once colours were sampled. */ + source: 'ax' | 'ax+px'; + budget: AxBudget; + nodes: AxNode[]; +} + +export interface AxTreeOptions { + /** Application name — exact, else case-insensitive prefix. */ + app?: string; + pid?: number; + /** Which window of the app, in front-to-back order. Default 0 (frontmost). */ + window?: number; + /** + * `content` (default) drops unlabelled structural containers and re-parents + * their children — boxes are absolute, so the nesting adds little for a reader + * and roughly halves the payload. `full` returns the raw tree. + */ + detail?: 'content' | 'full'; + /** Stop after this many elements. Default 1500. */ + maxElements?: number; + /** Default 40. */ + maxDepth?: number; + /** Keep elements whose frame falls outside the window. Default false. */ + includeOffscreen?: boolean; + /** + * Sample colours from this PNG. Pass the `path` and `frame` of a `captureScreen()` + * result taken of the same window, close in time. + */ + colors?: { path: string; frame: ScreenFrame }; + /** Read font and alignment for text elements. One extra IPC round trip each. */ + typography?: boolean; +} + +/** + * Walks the accessibility tree of a running application and returns its box model. + * + * Cost is dominated by the target app's AX responsiveness, not by tree size — the + * same 4000 elements measured 1.6s in Safari and 11s in Finder. Attribute reads are + * batched and offscreen subtrees are culled; `budget` reports what that cost and + * whether the result was truncated, so a capped tree is never mistaken for a + * complete one. + * + * Requires Accessibility permission for the host process; throws otherwise. + */ +export async function axTree(options: AxTreeOptions = {}): Promise { + const args: string[] = []; + if (options.pid !== undefined) args.push('--pid', String(options.pid)); + else if (options.app) args.push('--app', options.app); + else throw new Error('axTree requires app or pid'); + + if (options.window !== undefined) args.push('--window', String(options.window)); + if (options.detail) args.push('--detail', options.detail); + if (options.maxElements !== undefined) args.push('--max-elements', String(options.maxElements)); + if (options.maxDepth !== undefined) args.push('--max-depth', String(options.maxDepth)); + if (options.includeOffscreen) args.push('--include-offscreen'); + if (options.typography) args.push('--typography'); + if (options.colors) { + const f = options.colors.frame; + args.push('--colors', options.colors.path, '--frame', `${f.x},${f.y},${f.w},${f.h}`); + } + // Walking a slow application can legitimately take many seconds. + return runHelper(AX_BIN, args, { timeout: 60_000 }); +} diff --git a/src/helper.ts b/src/helper.ts index 7483edc..f2fce12 100644 --- a/src/helper.ts +++ b/src/helper.ts @@ -1,4 +1,4 @@ -// Shared plumbing for the native helpers (vision-helper, pdf-helper, ui-helper). +// Shared plumbing for the native helpers (vision-helper, pdf-helper, ui-helper, ax-helper). // // Wire contract: a helper writes exactly one payload to stdout (the Swift side // diverts framework noise to stderr), exits 1 on failure and 2 when a feature is @@ -17,6 +17,7 @@ const BIN_DIR = resolve(dirname(fileURLToPath(import.meta.url)), '../bin'); export const VISION_BIN = join(BIN_DIR, 'vision-helper'); export const PDF_BIN = join(BIN_DIR, 'pdf-helper'); export const UI_BIN = join(BIN_DIR, 'ui-helper'); +export const AX_BIN = join(BIN_DIR, 'ax-helper'); /** Helper exit status meaning "not supported on this macOS". */ export const EXIT_UNSUPPORTED = 2; diff --git a/src/index.ts b/src/index.ts index 349ce09..a466f06 100644 --- a/src/index.ts +++ b/src/index.ts @@ -356,6 +356,17 @@ export type { export { inferLayout, sortBlocksByReadingOrder } from './layout.js'; // ─── Markdown pipeline (VisionScribe) ────────────────────────────────────────── +export { axTree } from './ax.js'; +export type { + AxTree, + AxTreeOptions, + AxNode, + AxBox, + AxStyle, + AxTypography, + AxBudget, +} from './ax.js'; + export { VisionScribe, OllamaUnavailableError } from './markdown/index.js'; export type { VisionScribeOptions, ParagraphGroup } from './markdown/index.js'; diff --git a/src/native/ax-helper.swift b/src/native/ax-helper.swift new file mode 100644 index 0000000..54bfff3 --- /dev/null +++ b/src/native/ax-helper.swift @@ -0,0 +1,419 @@ +import AppKit +import ApplicationServices +import Foundation + +// ax-helper — accessibility tree of a running application, as a box model. +// +// Read-only: reads attributes, never sets them and never synthesises input. +// +// Every attribute read is a synchronous IPC round trip into the target app, so a +// naive walk is unusably slow (measured: 4001 elements cost 1.57s in Safari but +// 26s in Finder, one attribute at a time). Three things keep it bounded: +// batched reads, viewport culling, and a hard element/depth budget that is +// reported rather than applied silently. + +// ─── Wire types ────────────────────────────────────────────────────────────── + +/// Encoded as [x, y, w, h] — the same four numbers cost ~4x fewer tokens than +/// a keyed object repeated across a whole tree, and read just as clearly. +typealias Box = [Double] + +struct Style: Codable { + let bg: String? + let border: String? + let borderWidth: Int? +} + +struct Typography: Codable { + let font: String? + let family: String? + let size: Double? + let align: String? +} + +struct Node: Codable { + let id: Int + let parent: Int? + let depth: Int + let role: String + let subrole: String? + let label: String? + let value: String? + /// Present only when false — enabled is the overwhelming default. + let enabled: Bool? + /// Present only when true. + let focused: Bool? + /// Global screen points, top-left origin — the space click drivers use. + let box: Box + let style: Style? + let text: Typography? +} + +struct Budget: Codable { + let elements: Int + let capped: Bool + let maxElements: Int + let maxDepth: Int + let elapsedMs: Int + /// Elements skipped because their frame fell outside the culling rect. + let culled: Int +} + +struct TreeResult: Codable { + let app: String + let pid: Int + let window: Box? + /// "ax" for geometry only, "ax+px" once colours were sampled. + let source: String + let budget: Budget + let nodes: [Node] +} + +func encodeJSON(_ value: T) -> String { + let enc = JSONEncoder() + guard let data = try? enc.encode(value), let s = String(data: data, encoding: .utf8) else { return "{}" } + return s +} + +func fail(_ message: String, code: Int32 = 1) -> Never { + fputs("ERROR: \(message)\n", stderr) + exit(code) +} + +// ─── Argument parsing ──────────────────────────────────────────────────────── + +let args = CommandLine.arguments +func opt(_ flag: String) -> String? { + guard let i = args.firstIndex(of: flag), i + 1 < args.count else { return nil } + return args[i + 1] +} +func intOpt(_ flag: String, _ fallback: Int) -> Int { Int(opt(flag) ?? "") ?? fallback } + +let maxElements = intOpt("--max-elements", 1500) +let maxDepth = intOpt("--max-depth", 40) +let visibleOnly = !args.contains("--include-offscreen") +let wantTypography = args.contains("--typography") +let colorsPath = opt("--colors") +// "content" (default) drops unlabelled structural containers; "full" keeps the +// raw tree. Boxes are absolute, so nesting adds little for a reader — but it +// adds a lot of tokens: on a Safari window it is roughly half the payload. +let detailFull = (opt("--detail") ?? "content") == "full" + +// ─── Target application ────────────────────────────────────────────────────── + +let running = NSWorkspace.shared.runningApplications +var target: NSRunningApplication? +if let pidStr = opt("--pid"), let pid = Int32(pidStr) { + target = running.first { $0.processIdentifier == pid } + if target == nil { fail("no running application with pid \(pidStr)") } +} else if let name = opt("--app") { + let q = name.lowercased() + target = running.first { ($0.localizedName ?? "").lowercased() == q } + ?? running.first { ($0.localizedName ?? "").lowercased().hasPrefix(q) } + if target == nil { + let visible = running.compactMap { $0.activationPolicy == .regular ? $0.localizedName : nil } + fail("no running application matches \"\(name)\". Running: \(visible.joined(separator: ", "))") + } +} else { + fail("usage: ax-helper --app | --pid [--window ] [--detail content|full] [--max-elements N] [--max-depth N] [--include-offscreen] [--typography] [--colors --frame x,y,w,h]") +} + +guard AXIsProcessTrusted() else { + fail("Accessibility permission missing. Grant it to the host application in System Settings → Privacy & Security → Accessibility, then restart it.", code: 3) +} + +let app = target! +let axApp = AXUIElementCreateApplication(app.processIdentifier) +// An unresponsive app must degrade, never hang the caller. +AXUIElementSetMessagingTimeout(axApp, 2.0) + +// ─── Attribute plumbing ────────────────────────────────────────────────────── + +let wanted: [CFString] = [ + kAXRoleAttribute as CFString, + kAXSubroleAttribute as CFString, + kAXTitleAttribute as CFString, + kAXValueAttribute as CFString, + kAXDescriptionAttribute as CFString, + kAXPositionAttribute as CFString, + kAXSizeAttribute as CFString, + kAXEnabledAttribute as CFString, + kAXFocusedAttribute as CFString, +] + +/// One IPC round trip for every attribute we need. Measured 1.4–2.3x faster than +/// reading them one at a time, and the gap widens on slow apps. +func readAttributes(_ el: AXUIElement) -> [Any?] { + var out: CFArray? + let err = AXUIElementCopyMultipleAttributeValues(el, wanted as CFArray, AXCopyMultipleAttributeOptions(rawValue: 0), &out) + guard err == .success, let arr = out as? [Any] else { return Array(repeating: nil, count: wanted.count) } + // Unset attributes come back as AXValue of type .axError; map those to nil. + return arr.map { v -> Any? in + if CFGetTypeID(v as CFTypeRef) == AXValueGetTypeID(), + AXValueGetType(v as! AXValue) == .axError { return nil } + return v + } +} + +func point(_ v: Any?) -> CGPoint? { + guard let v = v, CFGetTypeID(v as CFTypeRef) == AXValueGetTypeID() else { return nil } + var p = CGPoint.zero + return AXValueGetValue(v as! AXValue, .cgPoint, &p) ? p : nil +} +func size(_ v: Any?) -> CGSize? { + guard let v = v, CFGetTypeID(v as CFTypeRef) == AXValueGetTypeID() else { return nil } + var s = CGSize.zero + return AXValueGetValue(v as! AXValue, .cgSize, &s) ? s : nil +} +func text(_ v: Any?) -> String? { + guard let s = v as? String else { return nil } + let t = s.trimmingCharacters(in: .whitespacesAndNewlines) + return t.isEmpty ? nil : t +} + +func children(_ el: AXUIElement) -> [AXUIElement] { + var v: CFTypeRef? + guard AXUIElementCopyAttributeValue(el, kAXChildrenAttribute as CFString, &v) == .success, + let kids = v as? [AXUIElement] else { return [] } + return kids +} + +// ─── Typography (text elements only) ───────────────────────────────────────── + +/// AX carries no styling on ordinary elements, but AXAttributedStringForRange on +/// a text element does return real font data. Costs an extra round trip, so it is +/// opt-in and limited to roles that can actually answer. +func typography(_ el: AXUIElement) -> Typography? { + var r = CFRange(location: 0, length: 1) + guard let rangeValue = AXValueCreate(.cfRange, &r) else { return nil } + var out: CFTypeRef? + guard AXUIElementCopyParameterizedAttributeValue(el, "AXAttributedStringForRange" as CFString, rangeValue, &out) == .success, + let s = out as? NSAttributedString, s.length > 0 else { return nil } + let attrs = s.attributes(at: 0, effectiveRange: nil) + var family: String?, name: String?, pt: Double? + if let f = attrs[NSAttributedString.Key("AXFont")] as? [String: Any] { + family = f["AXFontFamily"] as? String + name = f["AXFontName"] as? String + pt = (f["AXFontSize"] as? NSNumber)?.doubleValue + } + var align: String? + if let a = attrs[NSAttributedString.Key("AXATextAlignmentValue")] as? NSNumber { + align = ["natural", "left", "right", "center", "justified"][safe: a.intValue] ?? nil + } + if family == nil && name == nil && pt == nil && align == nil { return nil } + return Typography(font: name, family: family, size: pt, align: align) +} + +extension Array { + subscript(safe i: Int) -> Element? { indices.contains(i) ? self[i] : nil } +} + +// ─── Pixels: background and border colour ──────────────────────────────────── + +final class Pixels { + private let buf: [UInt8] + private let w: Int, h: Int + /// Screen rect the image covers, and pixels per point. + private let frame: CGRect, scale: Double + + init?(path: String, frame: CGRect) { + guard let img = NSImage(contentsOfFile: path), + let cg = img.cgImage(forProposedRect: nil, context: nil, hints: nil) else { return nil } + w = cg.width; h = cg.height + var data = [UInt8](repeating: 0, count: w * h * 4) + guard let ctx = CGContext(data: &data, width: w, height: h, bitsPerComponent: 8, + bytesPerRow: w * 4, space: CGColorSpaceCreateDeviceRGB(), + bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue) else { return nil } + ctx.draw(cg, in: CGRect(x: 0, y: 0, width: w, height: h)) + buf = data + self.frame = frame + scale = frame.width > 0 ? Double(w) / Double(frame.width) : 1 + } + + private func rgb(_ x: Int, _ y: Int) -> (Int, Int, Int) { + let i = (y * w + x) * 4 + return (Int(buf[i]), Int(buf[i + 1]), Int(buf[i + 2])) + } + + private func hex(_ c: (Int, Int, Int)) -> String { String(format: "#%02X%02X%02X", c.0, c.1, c.2) } + + /// Screen points → image pixels, clamped to the bitmap. + private func toPixels(_ b: Box) -> (Int, Int, Int, Int)? { + let px = Int((b[0] - Double(frame.origin.x)) * scale) + let py = Int((b[1] - Double(frame.origin.y)) * scale) + let pw = Int(b[2] * scale), ph = Int(b[3] * scale) + let x0 = max(0, px), y0 = max(0, py) + let x1 = min(w - 1, px + pw), y1 = min(h - 1, py + ph) + if x1 - x0 < 2 || y1 - y0 < 2 { return nil } + return (x0, y0, x1, y1) + } + + /// Most common colour inside the rect, on a coarse grid — the element's fill. + private func dominant(_ x0: Int, _ y0: Int, _ x1: Int, _ y1: Int) -> (Int, Int, Int) { + var hist: [Int: Int] = [:] + let sx = max(1, (x1 - x0) / 16), sy = max(1, (y1 - y0) / 16) + var y = y0 + while y <= y1 { + var x = x0 + while x <= x1 { + let c = rgb(x, y) + hist[(c.0 / 16) << 10 | (c.1 / 16) << 5 | (c.2 / 16), default: 0] += 1 + x += sx + } + y += sy + } + guard let k = hist.max(by: { $0.value < $1.value })?.key else { return (0, 0, 0) } + return (((k >> 10) & 31) * 16, ((k >> 5) & 31) * 16, (k & 31) * 16) + } + + /// Fill colour, plus a border when the outline differs from the fill. + /// Border width is *inferred* by walking inward until the colour settles — + /// it is an estimate, not a measured CSS value. + func style(_ b: Box) -> Style? { + guard let (x0, y0, x1, y1) = toPixels(b) else { return nil } + let inset = max(2, min((x1 - x0) / 4, (y1 - y0) / 4)) + let fill = dominant(x0 + inset, y0 + inset, x1 - inset, y1 - inset) + let edge = dominant(x0, y0, x1, min(y0 + 1, y1)) + func near(_ a: (Int, Int, Int), _ c: (Int, Int, Int)) -> Bool { + abs(a.0 - c.0) + abs(a.1 - c.1) + abs(a.2 - c.2) < 48 + } + if near(edge, fill) { return Style(bg: hex(fill), border: nil, borderWidth: nil) } + var width = 1 + let midX = (x0 + x1) / 2 + var y = y0 + 1 + while y < y1 && width < 12 && near(rgb(midX, y), edge) { width += 1; y += 1 } + return Style(bg: hex(fill), border: hex(edge), borderWidth: max(1, Int((Double(width) / scale).rounded()))) + } +} + +// ─── Walk ──────────────────────────────────────────────────────────────────── + +var nodes: [Node] = [] +var nextId = 0 +var culled = 0 +var capped = false +let started = Date() + +// Window frame, used both as the culling rect and as output metadata. +var windowBox: Box? +var cullRect: CGRect? + +let windows = children(axApp).filter { el in + let a = readAttributes(el) + return (a[0] as? String) == "AXWindow" +} +let windowIndex = intOpt("--window", 0) +if let win = windows[safe: windowIndex] { + let a = readAttributes(win) + if let p = point(a[5]), let s = size(a[6]) { + windowBox = [Double(p.x), Double(p.y), Double(s.width), Double(s.height)] + if visibleOnly { cullRect = CGRect(origin: p, size: s) } + } +} + +var pixels: Pixels? +if let cp = colorsPath { + guard let fs = opt("--frame") else { fail("--colors requires --frame x,y,w,h (the capture's screen rect)") } + let f = fs.split(separator: ",").compactMap { Double($0) } + guard f.count == 4 else { fail("--frame expects x,y,w,h") } + pixels = Pixels(path: cp, frame: CGRect(x: f[0], y: f[1], width: f[2], height: f[3])) + if pixels == nil { fail("cannot read image for --colors: \(cp)") } +} + +func walk(_ el: AXUIElement, parent: Int?, depth: Int) { + if depth > maxDepth { capped = true; return } + if nodes.count >= maxElements { capped = true; return } + + let a = readAttributes(el) + guard let role = a[0] as? String else { return } + guard let p = point(a[5]), let s = size(a[6]), s.width > 0, s.height > 0 else { + // No geometry of its own (menus before they open, transient containers) — + // still descend, since children often do have frames. + for c in children(el) { walk(c, parent: parent, depth: depth + 1) } + return + } + + let rect = CGRect(origin: p, size: s) + if let cull = cullRect, !cull.intersects(rect) { + culled += 1 + return + } + + let id = nextId + nextId += 1 + let box: Box = [Double(p.x), Double(p.y), Double(s.width), Double(s.height)] + let style = pixels?.style(box) + let isTextRole = role == "AXStaticText" || role == "AXTextField" || role == "AXTextArea" + + nodes.append(Node( + id: id, + parent: parent, + depth: depth, + role: String(role.dropFirst(2)), // AXButton → Button + subrole: (a[1] as? String).map { String($0.dropFirst(2)) }, + label: text(a[2]) ?? text(a[4]), + value: text(a[3]), + enabled: (a[7] as? Bool) == false ? false : nil, + focused: (a[8] as? Bool) == true ? true : nil, + box: box, + style: style, + text: (wantTypography && isTextRole) ? typography(el) : nil + )) + + for c in children(el) { walk(c, parent: id, depth: depth + 1) } +} + +if let win = windows[safe: windowIndex] { + walk(win, parent: nil, depth: 0) +} else { + walk(axApp, parent: nil, depth: 0) +} + +/// Roles worth keeping even with no label — they are actionable or structural +/// landmarks a reader needs. +let keepRoles: Set = [ + "Button", "Link", "CheckBox", "RadioButton", "TextField", "TextArea", "ComboBox", + "PopUpButton", "MenuItem", "MenuButton", "Slider", "Stepper", "Tab", "Window", + "Sheet", "Toolbar", "Image", "Table", "Outline", +] + +if !detailFull { + // Keep anything that carries meaning; re-parent survivors onto their nearest + // surviving ancestor so the hierarchy stays walkable. + var keep = Set() + for n in nodes where n.label != nil || n.value != nil || keepRoles.contains(n.role) || n.parent == nil { + keep.insert(n.id) + } + var parentOf: [Int: Int?] = [:] + for n in nodes { parentOf[n.id] = n.parent } + func nearestKept(_ id: Int?) -> Int? { + var cur = id + while let c = cur { + if keep.contains(c) { return c } + cur = parentOf[c] ?? nil + } + return nil + } + nodes = nodes.filter { keep.contains($0.id) }.map { n in + Node(id: n.id, parent: nearestKept(n.parent), depth: n.depth, role: n.role, + subrole: n.subrole, label: n.label, value: n.value, enabled: n.enabled, + focused: n.focused, box: n.box, style: n.style, text: n.text) + } +} + +let result = TreeResult( + app: app.localizedName ?? "", + pid: Int(app.processIdentifier), + window: windowBox, + source: pixels == nil ? "ax" : "ax+px", + budget: Budget( + elements: nodes.count, + capped: capped, + maxElements: maxElements, + maxDepth: maxDepth, + elapsedMs: Int(Date().timeIntervalSince(started) * 1000), + culled: culled + ), + nodes: nodes +) +print(encodeJSON(result)) diff --git a/src/ui.ts b/src/ui.ts index 5142b3b..8bc02b4 100644 --- a/src/ui.ts +++ b/src/ui.ts @@ -167,6 +167,23 @@ export async function captureScreen(opts: CaptureOptions = {}): Promise 0) || !(h > 0) || ![x, y].every(Number.isFinite)) { + throw new Error(`capture rect must have positive width and height, got ${w}×${h}`); + } + const displays = await listDisplays(); + const intersects = displays.some( + (d) => x < d.x + d.w && x + w > d.x && y < d.y + d.h && y + h > d.y + ); + if (!intersects) { + const bounds = displays.map((d) => `${d.w}×${d.h} at ${d.x},${d.y}`).join('; '); + throw new Error( + `capture rect ${x},${y} ${w}×${h} does not intersect any displays (${bounds})` + ); + } args.push(`-R${x},${y},${w},${h}`); frame = { x, y, w, h }; targetDesc = `region ${x},${y} ${w}×${h}`; diff --git a/test/ax.test.ts b/test/ax.test.ts new file mode 100644 index 0000000..92af1a8 --- /dev/null +++ b/test/ax.test.ts @@ -0,0 +1,105 @@ +import { describe, it, expect } from 'vitest'; +import { axTree } from '../src/index.js'; + +// Finder is always running on a Mac and has a deep, geometry-rich tree, which +// makes it the least flaky target available without shipping a fixture app. +const APP = 'Finder'; +const T = 60_000; + +describe('axTree()', () => { + it( + 'returns a tree with geometry for a running app', + async () => { + const tree = await axTree({ app: APP, maxElements: 120 }); + expect(tree.app).toBe(APP); + expect(tree.pid).toBeGreaterThan(0); + expect(tree.nodes.length).toBeGreaterThan(0); + expect(tree.source).toBe('ax'); + }, + T + ); + + it( + 'gives every node a four-number box and a role', + async () => { + const { nodes } = await axTree({ app: APP, maxElements: 120 }); + for (const n of nodes) { + expect(n.box).toHaveLength(4); + for (const v of n.box) expect(Number.isFinite(v)).toBe(true); + expect(n.box[2]).toBeGreaterThan(0); // width + expect(n.box[3]).toBeGreaterThan(0); // height + expect(typeof n.role).toBe('string'); + expect(n.role.startsWith('AX')).toBe(false); // prefix stripped + } + }, + T + ); + + it( + 'reports the budget honestly instead of truncating silently', + async () => { + const tree = await axTree({ app: APP, maxElements: 10 }); + expect(tree.budget.elements).toBe(tree.nodes.length); + expect(tree.budget.elements).toBeLessThanOrEqual(10); + expect(tree.budget.capped).toBe(true); + expect(tree.budget.elapsedMs).toBeGreaterThanOrEqual(0); + }, + T + ); + + it( + 'keeps parent ids resolvable within the returned set', + async () => { + const { nodes } = await axTree({ app: APP, maxElements: 200 }); + const ids = new Set(nodes.map((n) => n.id)); + // The root carries no `parent` key at all — Swift omits nil rather than + // encoding null, and that saves a key on every root. + const roots = nodes.filter((n) => n.parent === undefined); + expect(roots.length).toBeGreaterThan(0); + for (const n of nodes) { + if (n.parent !== undefined) expect(ids.has(n.parent)).toBe(true); + } + }, + T + ); + + it( + 'detail:content is a subset of detail:full', + async () => { + const full = await axTree({ app: APP, maxElements: 300, detail: 'full' }); + const content = await axTree({ app: APP, maxElements: 300, detail: 'content' }); + expect(content.nodes.length).toBeLessThanOrEqual(full.nodes.length); + // Pruning drops unlabelled structure, so what survives should be + // overwhelmingly nodes that carry meaning. + const meaningful = content.nodes.filter((n) => n.label || n.value || n.role === 'Window'); + expect(meaningful.length).toBeGreaterThan(content.nodes.length / 2); + }, + T + ); + + it( + 'omits enabled when true and focused when false, to keep the payload small', + async () => { + const { nodes } = await axTree({ app: APP, maxElements: 200 }); + for (const n of nodes) { + expect(n.enabled).not.toBe(true); // present only when false + expect(n.focused).not.toBe(false); // present only when true + } + }, + T + ); + + it('rejects a call with neither app nor pid', async () => { + await expect(axTree({})).rejects.toThrow(/app or pid/); + }); + + it( + 'reports a missing application clearly', + async () => { + await expect(axTree({ app: 'NoSuchApplication12345' })).rejects.toThrow( + /no running application/ + ); + }, + T + ); +}); diff --git a/test/vision.test.ts b/test/vision.test.ts index 6fefbff..f10f140 100644 --- a/test/vision.test.ts +++ b/test/vision.test.ts @@ -296,11 +296,19 @@ describe('helper error reporting', () => { }); }); - it('captureScreen surfaces screencapture stderr', async () => { + it('captureScreen rejects a rect with no area', async () => { + // screencapture itself is inconsistent here — it clamps and succeeds on an + // unlocked Mac, fails elsewhere — so the check has to be ours to be reliable. await expect(captureScreen({ rect: { x: 0, y: 0, w: -5, h: -5 } })).rejects.toThrow( - /does not intersect any displays/ + /positive width and height/ ); }); + + it('captureScreen rejects a rect that is off every display', async () => { + await expect( + captureScreen({ rect: { x: 900_000, y: 900_000, w: 10, h: 10 } }) + ).rejects.toThrow(/does not intersect any displays/); + }); }); describe('capability gating', () => {