feat(macos): opt-in native app backend beside XCTest

[thymikee]

Summary

AGENT_DEVICE_MACOS_APP_BACKEND=native drives macOS app sessions through the helper's accessibility actions (#3188) instead of the XCTest runner. XCTest stays the default. ADR 0031 records the decision.

AGENT_DEVICE_MACOS_APP_BACKEND=native agent-device open TextEdit --platform macos
agent-device click 'label=Format'   # app may stay behind your windows

• A native daemon never starts XCTest on macOS: macOsNativeBackendFacts refuses record, prepare, back, press-and-hold and gestures at admission (UNSUPPORTED_OPERATION, reason: unsupported-device-backend, dispatched: no).
• macOsNativeAppInteractor is assembled member by member; it does not spread the runner-backed interactor. Double/secondary clicks refuse before the helper runs; helper refusals keep helperReason.
• The backend is read once by the runtime owner (facts and bindings) and once per interactor; the daemon no longer reads it. Snapshot routing passes the routed surface explicitly.

27 files. Over the 700-line production threshold; independently reviewed, findings addressed.

Validation

• pnpm check:affected --run passed on 4b5291259 (3,889 tests).
• Live CLI output (run on the previous stack head; facts, admission and interactor unchanged since) (record refusal, click with ax-press/windowTitle, no runner process) is in the review reply below.

🤖 Generated with Claude Code

[cubic-dev-ai]

All reported issues were addressed across 19 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	5.00 MB	5.01 MB	+4.1 kB
Package (unpacked)	5.00 MB	5.01 MB	+4.1 kB
Package (download)	1.50 MB	1.50 MB	+1.4 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	17.5 ms	20.2 ms	+2.8 ms
CLI --help	50.8 ms	53.9 ms	+3.1 ms

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://callstack.github.io/agent-device/pr-preview/pr-3189/
Built to branch gh-pages at 2026-10-04 08:37 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[thymikee]

Thanks for the PR. At 87340b8 there is one problem to fix before merge: record start on a native macOS session still starts the XCTest runner.

With AGENT_DEVICE_MACOS_APP_BACKEND=native, startAppleRecording sends the local Mac (kind 'device') to startAppleRunnerRecording, not through withMacOsNativeAppBackend. That calls host.screenRecording.apple.runRunner with the session's bundle id. appleScreenRecordingFacts still reports recording as available on macOS whatever the backend is. The user gets no typed refusal. The host can go into Automation Mode with the overlay, and later native actions run beside a live runner. This breaks ADR 0031 rule 3 and the fail-closed claim. Could you make this rule hold: under the native backend, any operation that needs the Apple runner refuses before dispatch with UNSUPPORTED_OPERATION and reason macos-native-backend-unsupported? Please enforce it where capabilities are admitted, not with another wrapper override. appleScreenRecordingFacts already does this for the physical-iOS XCTest backend with unavailable('unsupported-device-backend', …). The set of runner callers the PR lists (alert.ts, app-device-io.ts, screenshot.ts, recording) shows where to check the same rule. A test that sets the env to native and asserts the facts refusal and its typed reason would cover it.

Could this be simpler? If the backend is resolved once in the Apple runtime binding and native limits are runtime facts, admission would refuse the runner-only capabilities (back, orientation, gestures, keyboard dismiss/enter, recording) before dispatch. The wrapper would then need no hand-kept override list. The native interactor could be built from helper-backed methods plus explicitly delegated ones, not by spreading the runner-backed base, so a runner method added later cannot leak in. Both cells in screenshot-crop-target.ts are rejected with the same PENDING reason, so that change may not be needed. platform-runtime-operation-host could take the surface the Apple owner already routed, not read the env again. The catch is that runtime facts are per device, not per session. The backend would first have to become an input to admission in src/daemon/runtime-admission.ts and src/platform-runtime-gateway.ts, or be recorded on the session at open, which you already named as a follow-up. After that, contracts, platform-apple and the daemon would stop reading AGENT_DEVICE_MACOS_APP_BACKEND at five sites.

I traced the recording route by reading the code and did not run it on a device. I did not read host.screenRecording.apple.runRunner to confirm it starts a runner lazily instead of refusing when none is live. I also did not audit setSetting, readSetting and readClipboard on macOS for runner use. The PR body says live CLI runs passed on this commit, but no output is attached, and I could not verify them or the check:affected result. After the fix, please run AGENT_DEVICE_MACOS_APP_BACKEND=native agent-device open TextEdit --platform macos and then agent-device record start <out> on the PR head. The output should show the UNSUPPORTED_OPERATION response with reason macos-native-backend-unsupported, and that no runner or xcodebuild process started (process list, or no runner.log). Please also paste a click response that shows mechanism 'ax-press'. The Swift helper's hit-test fallback comes from #3188 in the base branch. Whether it can press an element other than the one the daemon resolved is outside this diff, and I did not verify it.

The failed Smoke Tests check looks unrelated. An iOS simulator open … --relaunch --launch-url hit the 90s daemon request timeout during cold launch. This diff does not touch the iOS open route, and every new branch is guarded by isMacOs(device).

On the earlier review threads, one open thread on the repeated env reads still applies, and the same design cost is what the simplicity question above is about. The thread on findText gating does not apply, because findText is gated by admitAppleNativeFind in runtime-snapshot.ts before any interactor is resolved. The screenshot-crop-target thread is fixed at this head, so please resolve it. The signal-passing thread is fixed too, because interactor.ts now passes both signals to the helper calls.

Before merge, record start and any other runner-only capability must refuse at fact admission under the native backend, with a test and the live refusal run above.

[thymikee]

Thanks. Addressed in 168af661c.

Runner-only capabilities refuse at fact admission (fixed).

• macOsNativeBackendFacts (os/macos/native-backend-facts.ts) declares the macOS operations only the runner serves: screenRecordingStart/Reattach, prepareAppleRunner, back, longPressPoint, and the five gesture cells.
• createApplePlatformRuntime spreads it last over the leaf facts.
• I reused the existing reason unsupported-device-backend, the one the physical-iOS XCTest backend uses, instead of adding a sixth value to the closed reason union.
• Navigation bound its operations from its own recomputed facts, so a refused back stayed bound. It now binds from the admitted facts.
• runtime.test.ts sets the env to native and asserts both the refused facts and the unbound operations.

Native interactor built explicitly. macOsNativeAppInteractor is assembled member by member:

• The helper backs tap, press, focus, type, fill and scroll.
• These are taken from base because on macOS they use local tooling or the helper: open, close, screenshot, snapshot, readTextAtPoint, clipboard, setSetting, alerts, and pressPoint on other surfaces.
• No member reaches the runner, and a runner method added later cannot leak in.
• doubleTap is omitted, so the shared rule refuses it. Double and secondary clicks, and hold, refuse before any helper process starts.

Env reads. The daemon no longer reads AGENT_DEVICE_MACOS_APP_BACKEND:

• The crop classifier goes by surface alone; an app screenshot is its window on either backend.
• The snapshot loader receives the surface the Apple owner routed, and the owner passes it explicitly.
• Remaining reads: the Apple runtime owner reads the backend once at creation for facts and bindings; createAppleInteractor reads it once per interactor.

Live run on the stack head (c91f43079, built from this layer plus the cursor). T3 Code was frontmost throughout:

$ AGENT_DEVICE_MACOS_APP_BACKEND=native agent-device open Calculator --platform macos
Opened: Calculator
$ agent-device record start /tmp/ev.mp4
Error (UNSUPPORTED_OPERATION): record is not supported on this device
Hint: The native macOS app backend never starts the XCTest runner, which this command needs. Unset AGENT_DEVICE_MACOS_APP_BACKEND to use XCTest.
  (--json: details {"reason":"unsupported-device-backend","dispatched":"no"})
$ agent-device prepare ios-runner --platform macos --json
{"code":"UNSUPPORTED_OPERATION","message":"prepare is not supported on this device","details":{"reason":"unsupported-device-backend"}}
$ agent-device back --json      -> UNSUPPORTED_OPERATION {"reason":"unsupported-device-backend","dispatched":"no"}
$ agent-device click label=7 --json
{"mechanism":"ax-press","windowTitle":"Calculator","message":"Tapped label=7 (340, 663)"}
$ agent-device click label=7 --double-tap
Error (UNSUPPORTED_OPERATION): double-click is not supported by the native macOS app backend.
runner/xcodebuild processes from this daemon: 0

pgrep matched one process, a tvOS-simulator build-for-testing from another worktree that was already running. Nothing in this run started a runner.

Audited, no runner use on macOS:

• setSetting: setMacOsAppearance and the helper.
• readClipboard / writeClipboard: host pasteboard.
• readSetting: unavailable on macOS.

Smoke Tests failure. I agree it is unrelated: an iOS simulator open --relaunch hit the daemon timeout. It did not recur on the new head.

[cubic-dev-ai]

4 issues found across 17 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/platform-apple/src/os/macos/native-backend-facts.test.ts">

<violation number="1" location="packages/platform-apple/src/os/macos/native-backend-facts.test.ts:24">
P2: This test omits `performMultiTouchGesturePlan` and `performTargetAuthoredDrag`, so either could become available for the native backend without this “every runner-only” assertion failing. Add both operation names to the cases.</violation>
</file>

<file name="packages/platform-apple/src/runtime-snapshot.test.ts">

<violation number="1" location="packages/platform-apple/src/runtime-snapshot.test.ts:46">
P2: This test only proves that `xctest` skips the helper; it never verifies that the selected interactor actually captures, or that native avoids the XCTest interactor. Assert the selected owner is called and the other is not in each case.</violation>
</file>

<file name="docs/adr/0031-macos-native-app-backend.md">

<violation number="1" location="docs/adr/0031-macos-native-app-backend.md:60">
P3: The helper does not always identify the acted-on window: `windowTitle` is optional and comes from an optional accessibility title. Qualify this as being returned when available.</violation>
</file>

<file name="packages/platform-apple/src/os/macos/native-backend-facts.ts">

<violation number="1" location="packages/platform-apple/src/os/macos/native-backend-facts.ts:22">
P2: Also mark `screenRecordingCleanup` unavailable: cleanup of a macOS runner-backed recording calls `runRunner`, contradicting this backend's no-XCTest guarantee.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[thymikee]

The earlier concerns on 87340b8 are now fixed at 168af66, and I found no code problems that need a change before merge. The backend choice now comes from the owner and is read once, the screenshot crop target classifies by surface only, the native interactor no longer spreads the base interactor, and the abort signals reach the macOS screenshot and text reads. The required live evidence is in. It ran on c91f430, which is this head plus a ghost-cursor commit. That commit does not touch facts, admission or the interactor, so I accepted the run, but I did not rerun it. The open run started no runner, but I did not read close or setSetting on macOS, and those were not run live.

CI is green: 21 checks, none failing at 168af66. There are no conflicts. I did not run pnpm check:affected myself, so the 3,880-test count is the author's claim and I relied on CI.

Not blocking, and you can take or leave it: hostMacOsAppBackend() now runs when the Apple runtime loads, for every Apple device, so a misspelled AGENT_DEVICE_MACOS_APP_BACKEND throws INVALID_ARGS there. Before, it was read only per interactor and only on macOS. You could resolve it lazily for macOS devices only, or note in ADR rule 1 that an invalid value fails the whole Apple runtime.

On the existing threads, two Cubic test-gap threads still apply: the missing multi-touch and drag cells in the facts test (r4176451591), and the weak snapshot routing assertions (r4176451594). One lower-priority ADR wording thread still applies, about the window title (r4176451599). The recording cleanup thread does not apply, because reattach is refused under native, so that cleanup cannot run (r4176451597). These are fixed at this head and can be resolved: r4174630088, r4174630111, r4174630063, r4174630105, r4174630103, r4174630083. I did not check threads r4174630072, r4174630095 and r4174630098 against this head.

Nothing blocks merge. Please close the two test-gap threads and fix the ADR wording before merging into feat/macos-native-helper.

[thymikee]

Thanks. Addressed at 4b5291259. The stack is rebased onto main fae04c392, and pnpm check:affected --run passed on all three heads.

• Test gaps. The facts test covers all five gesture cells. The routing tests assert both owners in each direction. Threads resolved.
• ADR wording. windowTitle is qualified as "when the window has an accessibility title".
• Lazy backend (taken). The runtime owner reads AGENT_DEVICE_MACOS_APP_BACKEND on its first macOS device only and keeps it for its lifetime; every other Apple device uses xctest without reading it. A new test binds an iOS leaf with AGENT_DEVICE_MACOS_APP_BACKEND=nativ (succeeds) and the macOS leaf (INVALID_ARGS). ADR rule 1 states the scope.
• Recording cleanup. Not applied, for the reason in the review: reattach and start are refused under native, so no runner recording exists to clean up.

[thymikee]

The code looks good at 4b52912, and I found no new problems. The invalid-backend read is now lazy, and the new unit test covers it. The earlier findings from 168af66 (#3189 (comment)) are fixed.

The Smoke Tests check was still running when I looked, so it is not green yet. It drives iOS and Android simulators. The change touches the Apple bind route at packages/platform-apple/src/runtime.ts#L284-L286 (https://github.com/callstack/agent-device/blob/4b52912/packages/platform-apple/src/runtime.ts#L284-L286). For non-macOS devices it still returns xctest without reading the env, so I expect no change on that route.

I did not run the lazy-read change live. The live evidence from the earlier round covers valid values, and this change only affects when an invalid value is read. I also did not run pnpm check:affected or any tests myself.

There are no conflicts. Before merge, Smoke Tests needs to finish green on 4b52912.

The cubic-dev-ai threads that no longer apply can be resolved. Fixed at this commit: the test coverage for performMultiTouchGesturePlan and performTargetAuthoredDrag (#3189 (comment)), the helper-vs-runner owner assertions (#3189 (comment)), the windowTitle wording in ADR 0031 (#3189 (comment)), the click/press/fill, type, and scroll route docs (#3189 (comment)), the ADR 0031 click refusal wording (#3189 (comment)), and the full scroll result assertion (#3189 (comment)). Benign: native-backend-facts.ts refuses start and reattach under native, so no runner recording exists for cleanup to run on (#3189 (comment)). No open threads still apply.