feat: axTree() — box model of a live application

.changeset/ax-box-model.md

@@ -0,0 +1,30 @@
+---
+"macos-vision": minor
+---
+
+feat: `axTree()` — box model of a live application from the accessibility tree
+
+Returns element boxes, hierarchy, roles and labels for a running app, with
+optional colours sampled from a capture and typography from the AX attributed
+string. Geometry is measured rather than inferred from OCR, so it is exact.
+
+Shipped as a fourth prebuilt native helper (`ax-helper`) through the existing
+pipeline, so nothing needs compiling on the user's machine.
+
+Cost is bounded deliberately, because every attribute read is a synchronous IPC
+round trip into the target app and that app's implementation — not tree size —
+dominates: the same 4000 elements measured 1.6s in Safari and 11s in Finder.
+Attribute reads are batched, offscreen subtrees are culled, `maxElements` and
+`maxDepth` cap the walk, and `budget` reports what happened including
+`capped: true` — a truncated tree is never presented as a complete one.
+
+`detail: 'content'` (default) drops unlabelled structural containers and
+re-parents their children, halving the payload on a Finder window (600 → 289
+nodes). Boxes are encoded as `[x, y, w, h]` and default-valued fields are
+omitted, for the same reason.
+
+Also fixes `captureScreen()` silently accepting an invalid region: given a
+negative or fully offscreen rect, `screencapture` clamps and returns a tiny
+image with exit 0 on an unlocked Mac while failing elsewhere, so a caller's
+mistake surfaced as corrupt output. The rect is now validated against the actual
+display bounds before capture.

README.md

@@ -261,6 +261,86 @@ All coordinates are **global screen points with a top-left origin** — the same
 
 ---
 
+## API — Box model (accessibility tree)
+
+`axTree()` returns the on-screen layout of a running application: element boxes,
+hierarchy, roles and labels from the accessibility API, optionally with colours
+sampled from a capture and typography from the AX attributed string. Geometry is
+**measured**, not inferred from OCR bounding boxes.
+
+```ts
+import { axTree, captureScreen } from 'macos-vision';
+
+const tree = await axTree({ app: 'Safari' });
+// { app, pid, window: [x,y,w,h], source: 'ax', budget: {...}, nodes: [...] }
+
+// With colours and fonts, from a capture of the same window:
+const shot = await captureScreen({ app: 'Safari' });
+const full = await axTree({
+  app: 'Safari',
+  colors: { path: shot.path, frame: shot.frame },
+  typography: true,
+});
+```
+
+A node:
+
+```jsonc
+{
+  "id": 42, "parent": 7, "depth": 5,
+  "role": "Button", "label": "Zapisz",
+  "box": [812, 540, 96, 32],                 // [x, y, w, h] in screen points
+  "style": { "bg": "#2F6FEB", "border": "#1B4FC4", "borderWidth": 1 },
+  "text": { "font": "SFPro-Semibold", "family": "SF Pro", "size": 13, "align": "center" }
+}
+```
+
+`box` is an array rather than a keyed object because the same four numbers repeat
+on every node; keys would cost roughly four times the tokens. `enabled` appears
+only when `false` and `focused` only when `true`, for the same reason.
+
+### Cost, and how it is bounded
+
+Every attribute read is a synchronous IPC round trip into the target app, so cost
+tracks that app's accessibility implementation rather than tree size — the same
+4000 elements measured **1.6 s in Safari and 11 s in Finder**. Three things keep
+it bounded: attribute reads are batched, subtrees outside the window's visible
+rect are culled, and `maxElements` / `maxDepth` cap the walk. `budget` reports
+what happened, including `capped: true`, so a truncated tree is never mistaken
+for a complete one.
+
+`detail: 'content'` (the default) drops unlabelled structural containers and
+re-parents their children. Boxes are absolute, so the nesting adds little for a
+reader and roughly halves the payload — measured 600 → 289 nodes on a Finder
+window.
+
+> **A full tree is not a token saving over a screenshot.** A dense window runs to
+> thousands of tokens either way; measured on Finder, a pruned 289-node tree is
+> ~7.3k tokens against ~6.9k for the image. The reason to use it is what a
+> screenshot cannot give — exact boxes, roles, enabled state, hierarchy — and the
+> fact that you can take a slice (`maxElements`, one window, one subtree) instead
+> of the whole thing.
+
+### Limits, stated plainly
+
+- **This is not the CSS box model.** CSS has four nested boxes; AX has one.
+  `borderWidth` is inferred by an edge scan and there is no padding or margin.
+- **Colours come from pixels**, so an occluded element reports whatever is drawn
+  on top of it, and gradients or shadows are approximations.
+- **Typography depends on the app.** `AXAttributedStringForRange` returns real
+  font data where it is implemented (TextEdit gives `Menlo-Regular` 11 pt); web
+  content in Safari exposes alignment but no font.
+- **Requires Accessibility permission** for the host process, separately from
+  Screen Recording. Without it `axTree()` throws with that reason.
+- For web pages, Chrome DevTools `DOM.getBoxModel` and `CSS.getComputedStyleForNode`
+  return the real thing and are strictly better. This is for native apps, Electron,
+  canvas/WebGL, games and mockups.
+
+See [`docs/BOX-MODEL.md`](docs/BOX-MODEL.md) for the measurements behind these
+numbers.
+
+---
+
 ## API — Markdown pipeline (VisionScribe)
 
 `VisionScribe` converts an image or PDF to Markdown by combining Apple Vision OCR with a local LLM (via Ollama). The LLM never sees the image — it only formats text that Vision already extracted. This keeps image processing local and reduces the risk of vision-model hallucinations, but Markdown reconstruction is still best-effort and depends on the local model and document complexity.

docs/BOX-MODEL.md

@@ -4,7 +4,7 @@ Goal: hand an LLM a compact JSON description of what is on screen — element bo
 colours, borders, typography — complete enough that the model can reconstruct the layout, review
 it, or write assertions against it, **without ever seeing the screenshot**.
 
-Status: design only, nothing implemented. Every number below was measured on this machine
+Status: **phases 1–3 implemented** as `axTree()` (see the README). Phase 4 — merging with OCR to populate `unresolved` — is still open. Every number below was measured on this machine
 (Apple M1 Pro, 16 GB, macOS 26.5.2) with throwaway probes, not estimated.
 
 ---
@@ -160,7 +160,29 @@ a web page.
 
 ---
 
-## 6. Suggested order of work
+## 6. What implementation actually cost
+
+Built as `ax-helper` + `src/ax.ts`. Findings that only appeared once it ran:
+
+- **The naive JSON was more expensive than the screenshot it replaces.** A
+  250-node Safari tree came to ~12.8k tokens against ~6.9k for the image.
+  Encoding `box` as `[x, y, w, h]` instead of a keyed object, and omitting
+  `enabled: true` / `focused: false`, cut that by 44%. Pruning unlabelled
+  containers (`detail: 'content'`) took another 48% — 600 → 289 nodes on Finder.
+  Net: ~25 tokens per node instead of ~51.
+- **A full tree still is not a token win over a screenshot** — a pruned Finder
+  window is ~7.3k tokens against ~6.9k for the image. The case for it is
+  capability (exact boxes, roles, enabled state, hierarchy) and the ability to
+  take a slice, not raw token count. The README says so rather than implying a
+  saving that does not exist.
+- **Swift omits `nil` rather than encoding `null`**, so the root node has no
+  `parent` key at all. The TypeScript type said `number | null`; a consumer
+  checking `=== null` would have been wrong. Caught by a test, fixed in the type.
+- **`AXAttributedStringForRange` is app-dependent.** TextEdit returns
+  `Menlo-Regular` 11 pt; Safari's web `StaticText` returns alignment but no font.
+  That is a limit of the source, not of the reader.
+
+## 7. Suggested order of work
 
 1. **`ax-helper` with tree walk only** — role, frame, hierarchy, label, enabled. Batched reads,
    viewport culling, explicit budget, messaging timeout. This alone is ~80% of the value, because

scripts/build-native-cross.js

@@ -11,7 +11,7 @@ const TARGETS = [
   { arch: 'arm64', swift: 'arm64-apple-macos12' },
   { arch: 'x64', swift: 'x86_64-apple-macos12' },
 ];
-const HELPERS = ['vision-helper', 'pdf-helper', 'ui-helper'];
+const HELPERS = ['vision-helper', 'pdf-helper', 'ui-helper', 'ax-helper'];
 
 // Symbols added in newer SDKs are absent when building against an older one, so
 // the helper gates them on -DSDK_nn. Detect what this machine's SDK provides.

scripts/install-native.js

@@ -17,7 +17,7 @@ const root = path.resolve(__dirname, '..');
 const pkg = JSON.parse(readFileSync(path.join(root, 'package.json'), 'utf8'));
 
 const binDir = path.join(root, 'bin');
-const HELPERS = ['vision-helper', 'pdf-helper', 'ui-helper'];
+const HELPERS = ['vision-helper', 'pdf-helper', 'ui-helper', 'ax-helper'];
 
 // Symbols added in newer SDKs are absent when building against an older one, so
 // the helper gates them on -DSDK_nn. Detect what this machine's SDK provides.

src/ax.ts

@@ -0,0 +1,133 @@
+// Accessibility tree of a running application, shaped as a box model.
+//
+// Geometry and semantics come from the AX API — measured, not inferred from OCR.
+// Colours are sampled from a capture you supply; typography comes from the AX
+// attributed string where the app exposes it. See docs/BOX-MODEL.md for what is
+// fact and what is estimate.
+
+import { AX_BIN, runHelper } from './helper.js';
+import type { ScreenFrame } from './ui.js';
+
+/**
+ * `[x, y, w, h]` in global screen points, top-left origin — the space click
+ * drivers use. An array rather than a keyed object: the same four numbers cost
+ * roughly 4x fewer tokens when repeated across a whole tree.
+ */
+export type AxBox = [x: number, y: number, w: number, h: number];
+
+export interface AxStyle {
+  /** Dominant fill colour of the element's interior, as #RRGGBB. */
+  bg?: string;
+  /** Outline colour, present only when it differs from the fill. */
+  border?: string;
+  /** **Inferred** by walking inward from the edge — an estimate, not a measured value. */
+  borderWidth?: number;
+}
+
+export interface AxTypography {
+  /** PostScript name, e.g. `Menlo-Regular` */
+  font?: string;
+  family?: string;
+  size?: number;
+  align?: 'natural' | 'left' | 'right' | 'center' | 'justified';
+}
+
+export interface AxNode {
+  id: number;
+  /** Absent on the root of the walk; otherwise the id of the nearest kept ancestor. */
+  parent?: number;
+  depth: number;
+  /** AX role without the `AX` prefix: `Button`, `StaticText`, `WebArea`… */
+  role: string;
+  subrole?: string;
+  /** Title, falling back to the accessibility description. */
+  label?: string;
+  value?: string;
+  /** Present only when `false` — enabled is the overwhelming default. */
+  enabled?: boolean;
+  /** Present only when `true`. */
+  focused?: boolean;
+  box: AxBox;
+  style?: AxStyle;
+  text?: AxTypography;
+}
+
+export interface AxBudget {
+  elements: number;
+  /** True when the walk stopped at `maxElements`/`maxDepth` — the tree is incomplete. */
+  capped: boolean;
+  maxElements: number;
+  maxDepth: number;
+  elapsedMs: number;
+  /** Elements skipped because they fell outside the window's visible rect. */
+  culled: number;
+}
+
+export interface AxTree {
+  app: string;
+  pid: number;
+  /** Frame of the window that was walked. */
+  window?: AxBox;
+  /** `ax` when geometry only, `ax+px` once colours were sampled. */
+  source: 'ax' | 'ax+px';
+  budget: AxBudget;
+  nodes: AxNode[];
+}
+
+export interface AxTreeOptions {
+  /** Application name — exact, else case-insensitive prefix. */
+  app?: string;
+  pid?: number;
+  /** Which window of the app, in front-to-back order. Default 0 (frontmost). */
+  window?: number;
+  /**
+   * `content` (default) drops unlabelled structural containers and re-parents
+   * their children — boxes are absolute, so the nesting adds little for a reader
+   * and roughly halves the payload. `full` returns the raw tree.
+   */
+  detail?: 'content' | 'full';
+  /** Stop after this many elements. Default 1500. */
+  maxElements?: number;
+  /** Default 40. */
+  maxDepth?: number;
+  /** Keep elements whose frame falls outside the window. Default false. */
+  includeOffscreen?: boolean;
+  /**
+   * Sample colours from this PNG. Pass the `path` and `frame` of a `captureScreen()`
+   * result taken of the same window, close in time.
+   */
+  colors?: { path: string; frame: ScreenFrame };
+  /** Read font and alignment for text elements. One extra IPC round trip each. */
+  typography?: boolean;
+}
+
+/**
+ * Walks the accessibility tree of a running application and returns its box model.
+ *
+ * Cost is dominated by the target app's AX responsiveness, not by tree size — the
+ * same 4000 elements measured 1.6s in Safari and 11s in Finder. Attribute reads are
+ * batched and offscreen subtrees are culled; `budget` reports what that cost and
+ * whether the result was truncated, so a capped tree is never mistaken for a
+ * complete one.
+ *
+ * Requires Accessibility permission for the host process; throws otherwise.
+ */
+export async function axTree(options: AxTreeOptions = {}): Promise<AxTree> {
+  const args: string[] = [];
+  if (options.pid !== undefined) args.push('--pid', String(options.pid));
+  else if (options.app) args.push('--app', options.app);
+  else throw new Error('axTree requires app or pid');
+
+  if (options.window !== undefined) args.push('--window', String(options.window));
+  if (options.detail) args.push('--detail', options.detail);
+  if (options.maxElements !== undefined) args.push('--max-elements', String(options.maxElements));
+  if (options.maxDepth !== undefined) args.push('--max-depth', String(options.maxDepth));
+  if (options.includeOffscreen) args.push('--include-offscreen');
+  if (options.typography) args.push('--typography');
+  if (options.colors) {
+    const f = options.colors.frame;
+    args.push('--colors', options.colors.path, '--frame', `${f.x},${f.y},${f.w},${f.h}`);
+  }
+  // Walking a slow application can legitimately take many seconds.
+  return runHelper<AxTree>(AX_BIN, args, { timeout: 60_000 });
+}

src/helper.ts

@@ -1,4 +1,4 @@
-// Shared plumbing for the native helpers (vision-helper, pdf-helper, ui-helper).
+// Shared plumbing for the native helpers (vision-helper, pdf-helper, ui-helper, ax-helper).
 //
 // Wire contract: a helper writes exactly one payload to stdout (the Swift side
 // diverts framework noise to stderr), exits 1 on failure and 2 when a feature is
@@ -17,6 +17,7 @@ const BIN_DIR = resolve(dirname(fileURLToPath(import.meta.url)), '../bin');
 export const VISION_BIN = join(BIN_DIR, 'vision-helper');
 export const PDF_BIN = join(BIN_DIR, 'pdf-helper');
 export const UI_BIN = join(BIN_DIR, 'ui-helper');
+export const AX_BIN = join(BIN_DIR, 'ax-helper');
 
 /** Helper exit status meaning "not supported on this macOS". */
 export const EXIT_UNSUPPORTED = 2;

src/index.ts

@@ -356,6 +356,17 @@ export type {
 export { inferLayout, sortBlocksByReadingOrder } from './layout.js';
 
 // ─── Markdown pipeline (VisionScribe) ──────────────────────────────────────────
+export { axTree } from './ax.js';
+export type {
+  AxTree,
+  AxTreeOptions,
+  AxNode,
+  AxBox,
+  AxStyle,
+  AxTypography,
+  AxBudget,
+} from './ax.js';
+
 export { VisionScribe, OllamaUnavailableError } from './markdown/index.js';
 export type { VisionScribeOptions, ParagraphGroup } from './markdown/index.js';