Repository navigation
macOS: drive a local app without activating it or moving the real pointer #3254
Description
Activity
State of the five proposals, verified against
origin/mainat24b2ce638and measured on a macOS host (macOS 26 A28 / Xcode 27.1) with the built runner. Landed in #3339 unless stated.Proposal 1 —
open -g. Contested; owned by #3323.
#3323 already changesopentoopen -g, editing exactly the lines this proposal needs:buildMacOpenArgs's caller inpackages/platform-apple/src/os/macos/host-provider.ts(it addsopenActivationArgs(options)returning['-g']) pluspackages/platform-apple/src/os/macos/apps.tsandcore/tool-provider-types.ts(MacOsOpenOptions = { background?: boolean }). I did not touch those files. Residual gap to triage: #3323 setsconst openOptions = { background: hostMacOsAppBackend() === 'native' }, so non-activation follows the daemon backend rather than the caller's request — on the default XCTest backend there is still no way to askopento stay in the background. The flag that would express it is undecided: #3338.Proposal 2 —
postToPidwith window fields. Gated on #3213.
This is #3213 item 13, which its own text gates on items 1 and 3, because event fields 51 and 58 are undocumented and the fixture lane is what would catch a macOS change breaking them. Implementing it outside that lane ships undocumented fields unguarded. Data point for #3213: keyboard text already uses barepostToPidwith no window fields and works in the background —BackgroundInteraction.swift:529-530(postKey) and550-551(postText). Note also thatpostMouseClick's.cghidEventTapposts (MouseClickDelivery.swift:53, 86, 108, 113) are not reachable from a native app session:main.swift:419routes--surface apptopressInBackground(accessibility actions), so those posts serve onlyfrontmost-app/menubar, where "act on whatever is frontmost" is the intent. The proposal's premise that the native coordinate path needs window fields does not hold for app sessions today.Proposal 3 — skip
target.activate()unless--foreground. Partially landed (#3339); the rest is not supported as written.
What landed: on macOS a.existingAppread (snapshot,querySelector,gestureViewport,findText,readText) of a session app that is not running now answers the typedAPP_NOT_RUNNINGinstead of bare-launching it.notRunningRefusalwas#if os(iOS), so the guard never ran on macOS; the claim it rests on ("iOS … the platform that can read an app's state without launching it") is false — measured, a freshXCUIApplication(bundleIdentifier:)for a stopped macOS app answers.notRunning(1) and no process appears, and for a running background app it answers3, not4. Two host-lane tests pin it, including a canary that fails without the fix with the app left at.runningForeground.What did not land, and why the proposal needs amending: activation cannot simply be skipped for the commands that name a running app.
screenshot'sactivate()(RunnerTests+CommandExecution.swift:434) is load-bearing. With a known window frame and a second window laid over it, a backgroundedscreenshotRoot(app:).screenshot()returned that rect at the right scale with the occluding app's windows in it; afteractivate()it returned the app's own content. This backend captures the screen region, not the window buffer, so removing the activation would silently report another app's screen as the session app's window. The helper's window capture is a different mechanism,SCContentFilter(desktopIndependentWindow:)(AppWindowScreenshot.swift:29), on a different backend — not reachable from the runner.- Reads answer a tree whose rects move with time, and I could not separate that from foreground state: capturing the same app foreground → background → foreground gave 38 name-matched rows with a moved rect for foreground-vs-background and 39 for foreground-vs-later-foreground (node counts 146/143/143). So "a background tree is wrong" is not something I can claim; equally, nothing measured supports a non-activating read being equivalent either. Since rects feed the occlusion/offscreen decisions, this needs its own evidence before it is default behavior.
- The
--foregroundhalf is undecided: on main--foregroundalready means "include an initial interactive snapshot" (flag-definitions-action.ts,recorded: false), andsrc/mcp/server-guide.ts:20tells agents to always pass it. macOS: decide the public surface for "do not activate this app" (the --foreground axis collision) #3338 puts both readings with their costs.
Proposal 4 — ScreenCaptureKit window capture. Already landed; verified.
captureAppWindowScreenshot(AppWindowScreenshot.swift) filtersSCContentFilter(desktopIndependentWindow:)and is wired atmain.swift:519for--surface app; ADR 0031 rule 6 states it andwebsite/docs/docs/commands.mdalready saysscreenshotcaptures the app's front window "even when other windows cover it". It applies to the native backend; the XCTest backend still activates for the reason above.Proposal 5 — documentation. Landed in #3339.
AGENT_DEVICE_MACOS_APP_BACKENDwas already inwebsite/docs/docs/configuration.mdandcommands.md, so #3339 adds the missing half: an "Activation and the pointer" block in versionedhelp macosand a bullet list naming which macOS commands still bring the app forward or move the real pointer, including thatpress/clickonfrontmost-appandmenubarpost.cghidEventTapevents that move the real pointer and activate the clicked app.Also fixed as a side effect: three stale declarations that said the
APP_NOT_RUNNINGrefusal was iOS-only (RunnerTests+Models.swift,runner-contract.ts).Correction to my previous comment — an independent review found four errors in what I reported as landed. This comment is authoritative; my earlier one is superseded.
What my earlier comment claimed had landed in #3339 — that the
APP_NOT_RUNNINGrefusal was extended to macOS so reads of a stopped app refuse — was withdrawn.notRunningRefusalstays#if os(iOS)and returns nil off iOS. Refusing a stopped-app read on macOS was the wrong target: macOS callers rely onopen(a host-side launch) followed bysnapshot, and the hazard #3254 actually reports is different: a read of an app that IS running, sitting behind other windows, was raised first, stealing the user's frontmost app.What #3339 now actually changes (head
2df89243c):- The
.existingApparm ofprepareActiveCommandContext(the existinglaunchPolicyaxis — no new flag, no second axis): on macOS, a read whose app answers.runningBackgroundis served in place viaresolveAppWithoutActivation, booking no activation fact. A stopped app answers.notRunningto the state probe and keeps today's route, including the launch it performs there. - Interactions are untouched: the macOS XCTest path drives events through the foreground window, so they must keep activating.
screenshoton the XCTest backend: window-level capture keeps itsactivate()— measured, a backgroundedscreenshotRoot(app:).screenshot()returns the occluding window's content inside the session window's own rect, so the raise is load-bearing for the pixels. A--fullscreencapture of a running app no longer raises it (XCUIScreen.mainanswers from any foreground), and the 0.5 s activation settle is skipped with the raise. A stopped app still launches for either shape.- Live verification on the default backend with Finder frontmost:
snapshot/getanswered from the background app, Finder stayed frontmost,runner.logbooked zeroACTIVATE_FACTlines; the following click activated withpriorState=3as it must. Host lane: 294/294, canary fails without the arm.
What survives from my earlier comment, unchanged: the dispositions for proposals 1, 2, and 4 (native/accessibility/
SCContentFilterpaths already cover them), and the #3338 flag decision — now with an explicit recommendation against overloading--foreground.- The
- added 4 commits that reference this issue
on Oct 8, 2026
Problem
Driving a local macOS app with agent-device takes the app to the front and moves the user's real pointer, even when the app could be driven in the background. For an agent running on the user's own Mac this interrupts whatever the user is doing in another app. The opt-in native backend (
AGENT_DEVICE_MACOS_APP_BACKEND=native, ADR 0031, #3213) already acts through accessibility actions without XCTest and without the "Automation Running" overlay, butopen, the XCTest runner and the coordinate click path still bring the target app forward or use real pointer events. This issue is about those remaining activation points. It is related to #3213 (item 13, pointer events with window fields) and #3106 (non-activating observations), and does not replace either.Measured behavior
Fixture: a small AppKit app launched in the background (it logs
NSApp.isActiveitself and the test recordsNSWorkspace.frontmostApplication). agent-device 0.21.12, with 0.21.20 checked for the points below.agent-device open <bundleId> --platform macosactivates the app: the fixture loggeddidBecomeActive active=true key=trueandfrontmostApplicationchanged from the previously front app to the fixture. The macOS open path runsopen -b <bundleId>(buildMacOpenArgsinpackages/platform-apple/src/os/macos/host-provider.ts), without-g.AXError -25204(cannotComplete) fromAXUIElementCopyElementAtPositionon the first call after the app has been idle in the background. A retry succeeds.Read from the source on main, not measured:
RunnerTests+Lifecycle.swiftcallstarget.activate()unless the target is already.runningForeground, so each runner command on an inactive app activates it.screenshotwith an app bundle id callstargetApp.activate()inRunnerTests+CommandExecution.swift.MouseClickDelivery.swiftposts clicks withCGEvent.post(tap: .cghidEventTap)at screen coordinates. That moves the real pointer, lands on whatever window is topmost at that point, and the system activates the clicked app. It is not delivered to the target process.Related measurement from #3213:
CGEventPostToPidwith the window number set delivers clicks to a background app, while a bare post delivers nothing.Proposal
open: do not activate the app unless--foregroundis passed (open -g, orNSWorkspace.OpenConfiguration.activates = false). An app that is already running is attached to, not raised.--foregroundkeeps today's behavior.AXPress,AXSetValue, scroll) addressed by element ref. For raw coordinates and keys, useCGEvent.postToPidwith the window fields from macOS native app backend: follow-ups (guarantees, evidence, persistent helper, session backend) #3213 item 13 instead of.cghidEventTap. In my fixtures,postToPidwith the window set delivered clicks on buttons, text fields and table rows, text, Backspace and Tab, Command shortcuts and scroll while the app was inactive. Clicks on views that reject the first mouse (custom views, SwiftUIonTapGesture) were dropped, so those need an explicit activation step (--foreground, or afocuscommand).target.activate()unless--foreground.screenshotof an app: capture the target window (ScreenCaptureKit window capture) instead of activating the app first.Not in scope
Drags and WKWebView button clicks, which #3213 measured as needing the private focus-without-raise call.