Repository navigation
macOS native app backend: follow-ups (guarantees, evidence, persistent helper, session backend) #3213
Description
Activity
Pointer event delivery measurements (2026-10-05, item 13)
Setup: macOS 26.6.2. A fixture app with an
NSButton, a SwiftUIButton, an AppKit view (double, secondary, drag) and aWKWebView(button,dblclick,contextmenu, drag). The fixture stays in the background behind Ghostty. It records each event and whether it was active when the event arrived. Events are posted to its pid. Activity is checked withlsappinfo frontand withNSApplicationactivation notifications.Delivery mode × target, no activation (9 cases per mode):
Mode AppKit click / double / secondary SwiftUI click WKWebView double / secondary AppKit drag, WKWebView button, WKWebView drag Bare CGEventPostToPid(ADR 0031)no no no no Public post + cua fields yes yes yes no SLEventPostToPid+ cua fieldsyes yes yes no Both posts (cua default) NSButtonclick lost; rest yesyes yes no With focus-without-raise (
SLPSPostEventRecordTo), the public and SkyLight posts deliver all 9 cases. Each time, the fixture becomes AppKit-active, butlsappinfo frontstays on Ghostty, and a real HID keystroke still goes to Ghostty, not to the fixture. Bare posts deliver nothing even with activation.Required fields (public post, no activation): removing field 51 (window number) breaks every case. Field 1 (click state) is needed for double clicks. The rest of what cua sets is optional (mouse-move primer, fields 3, 7, 40, 91 and 92). Field 58 and the window-local location are interchangeable, but one of them is needed.
Public API only: setting fields 51 and 58 with
CGEventSetIntegerValueFieldand posting withCGEventPostToPiddelivered 54 of 54 (6 cases × 3 runs × 3 variants). The variants were: a nil event source, a HID-state source, and the SkyLight field setter. No private symbol was linked.Gaps: Electron, Catalyst and wheel events were not measured. Fields 51 and 58 are undocumented. A dropped post still returns success.
Minimal public-only poster
import CoreGraphics import Foundation // poster <pid> <x> <y> <click|double|right>; x/y in global display points let a = CommandLine.arguments let pid = pid_t(a[1])!, p = CGPoint(x: Double(a[2])!, y: Double(a[3])!) let windows = CGWindowListCopyWindowInfo([.optionOnScreenOnly], kCGNullWindowID) as! [[String: Any]] let wid = windows.first { ($0[kCGWindowOwnerPID as String] as! pid_t) == pid && ($0[kCGWindowLayer as String] as! Int) == 0 && CGRect(dictionaryRepresentation: $0[kCGWindowBounds as String] as! CFDictionary)!.contains(p) }![kCGWindowNumber as String] as! Int let windowNumber = unsafeBitCast(UInt32(51), to: CGEventField.self) let field58 = unsafeBitCast(UInt32(58), to: CGEventField.self) func post(_ t: CGEventType, _ b: CGMouseButton = .left, _ state: Int64 = 1) { let e = CGEvent(mouseEventSource: nil, mouseType: t, mouseCursorPosition: p, mouseButton: b)! e.setIntegerValueField(windowNumber, value: Int64(wid)) e.setIntegerValueField(field58, value: 12345) e.setIntegerValueField(.mouseEventClickState, value: state) e.postToPid(pid) } switch a[4] { case "click": post(.leftMouseDown); usleep(28_000); post(.leftMouseUp) case "double": for s in [Int64(1), 2] { post(.leftMouseDown, .left, s); usleep(28_000); post(.leftMouseUp, .left, s); usleep(80_000) } default: post(.rightMouseDown, .right); usleep(28_000); post(.rightMouseUp, .right) } usleep(200_000)
- added a commit that references this issue
on Oct 6, 2026
Purpose
Follow-ups for the opt-in macOS native app backend (
AGENT_DEVICE_MACOS_APP_BACKEND=native): #3188 (helper app surface), #3189 (routing, admission facts, ADR 0031), #3195 (ghost cursor). The backend works: accessibility actions, no XCTest, the app may stay in the background. These items make it provably correct, evidence-backed, then faster and broader.The plan was reviewed by Claude and by Codex (
codex exec, read-only). The order below is the reconciled one: correctness and evidence come before persistence.Work items (in order)
Guarantee classification and fallback correctness (M, no deps).
resolvePressTarget→smallestElement(BackgroundInteraction.swift) may act on a different element than the snapshot node the shared guards evaluated. That contradicts ADR 0031's "ADR 0011 guarantees unchanged". Repeated presses also stop early and report fewer clicks without a typed outcome.interaction-guarantees.ts, and either bound the fallback to the target node's own subtree or report the acted element (role, label, frame, window) so the daemon can verify it. Partial outcomes are typed, never replayed automatically. ADR 0031 is updated.Snapshot quality and typed AX failures (S–M).
SnapshotQualityVerdict(as iOS does), plus typed accessibility and permission failures.Native fixture lane (M, needs 1–2).
macos.ymlhost lane, run withAGENT_DEVICE_MACOS_APP_BACKEND=nativeset explicitly.Capture-frame contract (M). Snapshots and screenshots carry explicit frame metadata: window identity, origin, scale and display. Native snapshots traverse several windows while the screenshot captures one; XCTest subtracts the window origin at dispatch.
Keys, menu and value actions (S–M, needs 3). Separate increments, each with fixture evidence:
AXShowMenuwhere a control supports it;Not general double-click via
AXOpen, and not Cmd+[ as a universalback.Persistent helper transport (L, needs 3).
helper.ts), AX work and cursor time.helper.ts(install / transport / clients) andmain.swiftbefore adding behavior.Element-identity dispatch (L, needs 1 and 6). Act on cached accessibility handles only with helper, app and snapshot-generation tokens, ref-epoch authorization, live validation and bounded retention. Raw
x ycommands keep hit-testing with window ownership.Session-scoped backend (L, independent design).
open --backend native|xctest, with the env var only as a default.devicetoday, and bindings rebuild device facts.prepare, reopen and recovery semantics.Native recording (L). A ScreenCaptureKit window stream as its own lifecycle, replacing the runner refusal.
Helper tree presentation (M). Collapse only proven structural wrapper groups (Chromium), keeping raw topology, selector meaning and clip ownership. Use a reproducible fixture, not ad-hoc counts.
Parity cleanup (S).
MACOS_CLICK_*↔MouseClickSchedule.swift) with a fixture.Framework matrix and default decision (L, last). Cover AppKit, SwiftUI, Catalyst, Electron and WebKit, including unsupported actions and sparse apps. Native stays opt-in until this evidence exists.
Pointer events with window fields (M, needs 1 and 3).
CGEventPostToPidwith the target window number in event field 51, plus field 58 or a window-local location, delivers click, double-click and secondary click to a background AppKit, SwiftUI and WKWebView fixture: 54 of 54, no activation, frontmost app unchanged. A bare post, the ADR 0031 measurement, delivers nothing. Posting through SkyLight adds nothing. Drags and WKWebView button clicks need the private focus-without-raise activation.interaction-guarantees.tsand fixture-lane evidence (item 3), including Electron and Catalyst. Fields 51 and 58 are undocumented, so the fixture pins them and a macOS change fails the lane, not users.Non-goals
{facts, interactor, snapshotRoute}facade: the parts have different lifetimes.frontmost-app/menubarpresses into accessibility-only presses before their surface guarantees are defined. Synthetic menu extras need pointer delivery.Completion
Each item lands as its own PR that names this issue. The umbrella closes when items 1–3 have landed, and items 4–13 are either done or split into their own issues.