diff --git a/docs/adr/0025-foldable-apple-panels.md b/docs/adr/0025-foldable-apple-panels.md index 8ab29aa07b..a841d26b5c 100644 --- a/docs/adr/0025-foldable-apple-panels.md +++ b/docs/adr/0025-foldable-apple-panels.md @@ -3,8 +3,8 @@ ## Status Accepted (2026-09-20; the pose-settle rule amended 2026-09-21 under #2730; the touch-overlay -export diagnosis corrected 2026-09-22 under #2707). Covers iPhone Duo (iOS 27.1, `iPhone19,4`) and -any Apple device that reports more than one integrated CoreDevice display. +export diagnosis corrected 2026-09-22 under #2707; the Android emulator pose added 2026-10-10). Covers iPhone Duo (iOS 27.1, `iPhone19,4`) and +any Apple device that reports more than one integrated CoreDevice display, and, through the amendment below, foldable Android emulators. An iPhone Duo carries two integrated panels — Apple's **outer display** and **inner display** — and lights one of them at a time. Which one is lit is the device pose. Two independent facts @@ -536,3 +536,21 @@ incorrect viewport, not an inner-panel delivery prohibition. hidden-sidebar recovery were checked; the simultaneous same-name simulator matrix remains open. - **Physical foldables.** Device Hub poses simulators only; the leaf fact refuses a physical device, and the hinge stream on one was not exercised. + +## Android emulators (amendment, 2026-10-10) + +The same `fold` command poses a foldable Android emulator, with the two facts of this ADR kept +apart in the same way. There is no host display authority to consult: the emulator console's +`posture ` moves the hinge sensor to the angle the AVD profile defines for the posture (0°, 90°, +180° on the Pixel folds; 15°, 90°, 165° on the generic "7.6in Foldable"), and the pose is verified +by polling `cmd device_state print-state` until the guest commits the posture's own device state +(`CLOSED`, `HALF_OPENED`, `OPENED`), which is what `WindowManager`'s `FoldingFeature` derives from; +the hinge sensor is read back for the reported angle, not judged. A `half-open` Android pose is +therefore 90° on every profile seen so far, where the Duo's is 130°. + +Folding to the cover display can raise the keyguard (Android's "continue using apps on fold" +setting, which emulator images do not honour); `fold` dismisses it, because every later capture +would otherwise read the lock screen instead of the app. A profile that lists none of those device +states has no hinge and is refused as `single-panel-device`, the typed reason a single-panel +simulator gets. Keyframes and panel geometry stay Apple-only: the console poses only fixed +postures, and Android reports the app window on its next snapshot rather than a panel. diff --git a/docs/adr/README.md b/docs/adr/README.md index f6dd2a76a5..621ab75908 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -26,7 +26,7 @@ | [0022 Daemon — Platform Runtime Coupling Audit and Ownership Ratchets](0022-daemon-platform-runtime-coupling.md) | daemon imports of root `platform-runtime-*` modules, the R76 edge classification inventory, R75 session-authority ratchet, entry-to-platform hop routes and roles | | [0023 End-State Entry-to-Platform Hop Trace](0023-end-state-hop-trace.md) | the hop counting definition, the ordered `press`/Android and dual-arm `snapshot`/iOS chains, per-hop roles, and the deletion test behind the numbers ADR 0019 and ADR 0022 quote | | [0024 Screen Recording — One Coordinator, Two Facts (Proposed)](0024-screen-recording-provable-signal.md) | `record start`/`stop` lifecycle, the export-exists / recorder-stopped split, backend stop-and-collect contract, recovery through the coordinator, the ADR 0019 §5 amendment and the failed-finish cleanup fix | -| [0025 Foldable Apple Panels — Capture the Lit Panel](0025-foldable-apple-panels.md) | iPhone Duo outer/inner panels, CoreDevice display info as the only screen authority, why `simctl`'s implicit display default captures a black panel, derived (never asserted) pose, and why no pose-control command exists | +| [0025 Foldable Apple Panels — Capture the Lit Panel](0025-foldable-apple-panels.md) | iPhone Duo outer/inner panels, CoreDevice display info as the only screen authority, why `simctl`'s implicit display default captures a black panel, derived (never asserted) pose, the `fold` pose control, and the Android emulator posture amendment | | [0026 Scroll Clip Authority — Ownership Is the Parent Edge](0026-scroll-clip-authority.md) | iOS scroll indicators and the visible band, why ownership is read from the parent edge instead of an ancestor walk, what may remove a node from an interactive snapshot, and ejection dispositions | | [0027 Descriptor Root Size vs Eager-Closure Budget](0027-descriptor-root-vs-eager-closure-budget.md) | accepted: the eager-closure budget tolerates a statement-neutral hub split only (containment + added-module novelty + flat top-level-statement weight) and nothing else; splitting `packages/command-registry/src/registry.ts` is unblocked | | [0028 Capability-Family Cell Vocabulary — One Runtime Source (Proposed)](0028-capability-family-cell-vocabulary.md) | adding a capability operation family, `UnavailablePlatformRuntimeFacts` / `UNAVAILABLE_CELLS`, `INTERACTOR_OPERATIONS`, and why an eight-package fan-out recurs | diff --git a/packages/platform-android/src/foldable/pose.test.ts b/packages/platform-android/src/foldable/pose.test.ts new file mode 100644 index 0000000000..acd493ed56 --- /dev/null +++ b/packages/platform-android/src/foldable/pose.test.ts @@ -0,0 +1,286 @@ +import { beforeEach, expect, test, vi } from 'vitest'; + +vi.mock('../adb.ts', () => ({ runAndroidAdb: vi.fn(), runAndroidShell: vi.fn() })); +vi.mock('@agent-device/host-kit/retry', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, sleep: vi.fn(async () => {}) }; +}); +vi.mock('@agent-device/host-kit/diagnostics', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, emitDiagnostic: vi.fn() }; +}); + +import { sleep } from '@agent-device/host-kit/retry'; + +import { runAndroidAdb, runAndroidShell } from '../adb.ts'; +import { ANDROID_EMULATOR } from '../runtime.fixtures.ts'; +import { setAndroidFoldPose } from './pose.ts'; + +const mockAdb = vi.mocked(runAndroidAdb); +const mockShell = vi.mocked(runAndroidShell); +const mockSleep = vi.mocked(sleep); + +/** `cmd device_state print-states` on a Pixel fold: the posture states are numbered from 0. */ +const PIXEL_FOLD_STATES = `Supported states: [ + DeviceState{identifier=0, name='CLOSED', app_accessible=true, cancel_when_requester_not_on_top=false}, + DeviceState{identifier=1, name='HALF_OPENED', app_accessible=true, cancel_when_requester_not_on_top=false}, + DeviceState{identifier=2, name='OPENED', app_accessible=true, cancel_when_requester_not_on_top=false}, + DeviceState{identifier=3, name='REAR_DISPLAY_MODE', app_accessible=true, cancel_when_requester_not_on_top=false}, +] +`; +/** The generic "7.6in Foldable" profile numbers the same states from 1. */ +const GENERIC_FOLDABLE_STATES = `Supported states: [ + DeviceState{identifier=1, name='CLOSED', app_accessible=true, cancel_when_requester_not_on_top=false}, + DeviceState{identifier=2, name='HALF_OPENED', app_accessible=true, cancel_when_requester_not_on_top=false}, + DeviceState{identifier=3, name='OPENED', app_accessible=true, cancel_when_requester_not_on_top=false}, +] +`; +/** A profile with a hinge that lists only the two end stops. */ +const END_STOPS_ONLY_STATES = `Supported states: [ + DeviceState{identifier=0, name='CLOSED', app_accessible=true, cancel_when_requester_not_on_top=false}, + DeviceState{identifier=2, name='OPENED', app_accessible=true, cancel_when_requester_not_on_top=false}, +] +`; +/** A phone profile has one state and no hinge. */ +const PHONE_STATES = `Supported states: [ + DeviceState{identifier=0, name='DEFAULT', app_accessible=true, cancel_when_requester_not_on_top=false}, +] +`; + +function answer(stdout: string) { + return { stdout, stderr: '', exitCode: 0 }; +} + +/** The last entry of a read sequence repeats, as a device holding its state would. */ +function nextRead(reads: T[]): T { + return reads.length > 1 ? reads.shift()! : reads[0]!; +} + +/** + * One emulator, answered by command: `stateReads` are the successive `print-state` replies, + * `keyguardReads` the successive `isKeyguardShowing` values, `posture` the console's reply. + */ +function stubEmulator(options: { + states?: string; + stateReads: string[]; + keyguardReads?: boolean[]; + posture?: string; + hinge?: string; +}) { + const stateReads = [...options.stateReads]; + const keyguardReads = [...(options.keyguardReads ?? [false])]; + mockShell.mockImplementation(async (_device, words) => { + const command = words.join(' '); + if (command === 'cmd device_state print-states') { + return answer(options.states ?? PIXEL_FOLD_STATES); + } + if (command === 'cmd device_state print-state') return answer(`${nextRead(stateReads)}\n`); + if (command === 'dumpsys window') { + return answer(` isKeyguardShowing=${nextRead(keyguardReads)}\n`); + } + if (command === 'wm dismiss-keyguard') return answer(''); + throw new Error(`unexpected shell command: ${command}`); + }); + mockAdb.mockImplementation(async (_device, args) => { + const command = args.join(' '); + if (command.startsWith('emu posture ')) return answer(options.posture ?? 'OK\n'); + if (command === 'emu sensor get hinge-angle0') { + return answer(options.hinge ?? 'hinge-angle0 = 0\nOK\n'); + } + throw new Error(`unexpected adb command: ${command}`); + }); +} + +function adbCommands(): string[] { + return mockAdb.mock.calls.map(([, args]) => args.join(' ')); +} + +function shellCommands(): string[] { + return mockShell.mock.calls.map(([, words]) => words.join(' ')); +} + +beforeEach(() => { + mockAdb.mockReset(); + mockShell.mockReset(); + mockSleep.mockClear(); +}); + +test('sets the console posture, waits for the device state, and dismisses the lock screen the fold raised', async () => { + // No keyguard before the fold; the first state read still sees the open posture; the lock + // screen shows once and is gone for three reads after the dismissal. + stubEmulator({ stateReads: ['2', '0'], keyguardReads: [false, true, false, false, false] }); + + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'closed' })).resolves.toEqual({ + pose: 'closed', + hingeAngleDegrees: 0, + }); + + expect(adbCommands()).toEqual(['emu posture 1', 'emu sensor get hinge-angle0']); + expect(shellCommands()).toEqual([ + 'cmd device_state print-states', + 'dumpsys window', + 'cmd device_state print-state', + 'cmd device_state print-state', + 'dumpsys window', + 'wm dismiss-keyguard', + 'dumpsys window', + 'dumpsys window', + 'dumpsys window', + ]); +}); + +test('waits before the first lock-screen read, so a keyguard landing after the state is still caught', async () => { + // No keyguard before the fold or at the moment the state commits; it shows up on the next read. + stubEmulator({ stateReads: ['0'], keyguardReads: [false, false, true, false, false, false] }); + + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'closed' })).resolves.toEqual({ + pose: 'closed', + hingeAngleDegrees: 0, + }); + expect(shellCommands().filter((command) => command === 'wm dismiss-keyguard')).toHaveLength(1); + // One poll precedes every keyguard read: five reads, five sleeps, none from the state settle. + expect(mockSleep).toHaveBeenCalledTimes(5); +}); + +test('looks the device state up by name, so a profile numbered from 1 verifies the same pose', async () => { + stubEmulator({ + states: GENERIC_FOLDABLE_STATES, + stateReads: ['2'], + hinge: 'hinge-angle0 = 90\nOK\n', + }); + + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'half-open' })).resolves.toEqual({ + pose: 'half-open', + hingeAngleDegrees: 90, + }); + expect(adbCommands()[0]).toBe('emu posture 2'); + // Unfolding lights the inner display and raises no keyguard, so no lock-screen read is paid. + expect(shellCommands()).toEqual([ + 'cmd device_state print-states', + 'cmd device_state print-state', + ]); + expect(mockSleep).not.toHaveBeenCalled(); +}); + +test('refuses a posture the profile cannot commit with its own reason, not as single-panel', async () => { + stubEmulator({ states: END_STOPS_ONLY_STATES, stateReads: ['0'] }); + + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'half-open' })).rejects.toMatchObject({ + code: 'UNSUPPORTED_OPERATION', + details: { + reason: 'fold-posture-unsupported', + supportedPoses: ['closed', 'open'], + deviceStates: ['CLOSED', 'OPENED'], + hint: 'This profile poses only closed and open; request one of those.', + }, + }); + expect(mockAdb).not.toHaveBeenCalled(); +}); + +test('a failed print-states read is an adb failure, never an empty state list read as a phone', async () => { + // The managed scoped transport hands back a nonzero result instead of throwing. + mockShell.mockResolvedValue({ + stdout: '', + stderr: "cmd: Can't find service: device_state", + exitCode: 1, + }); + + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'closed' })).rejects.toMatchObject({ + code: 'COMMAND_FAILED', + details: { exitCode: 1, processExitError: true }, + }); + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'closed' })).rejects.not.toMatchObject({ + details: { reason: 'single-panel-device' }, + }); + expect(mockAdb).not.toHaveBeenCalled(); +}); + +test('refuses a phone profile as single-panel-device before touching the console', async () => { + stubEmulator({ states: PHONE_STATES, stateReads: ['0'] }); + + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'open' })).rejects.toMatchObject({ + code: 'UNSUPPORTED_OPERATION', + details: { reason: 'single-panel-device' }, + }); + expect(mockAdb).not.toHaveBeenCalled(); +}); + +test('refuses keyframes before touching the device', async () => { + stubEmulator({ stateReads: ['0'] }); + + await expect( + setAndroidFoldPose(ANDROID_EMULATOR, { + keyframes: [ + { atMs: 0, angle: 0 }, + { atMs: 1000, angle: 180 }, + ], + }), + ).rejects.toMatchObject({ + code: 'UNSUPPORTED_OPERATION', + details: { reason: 'fold-keyframes-unsupported' }, + }); + expect(mockAdb).not.toHaveBeenCalled(); + expect(mockShell).not.toHaveBeenCalled(); +}); + +test('reads a KO reply as a dispatch failure even though the console exits 0', async () => { + stubEmulator({ stateReads: ['0'], posture: 'KO: Posture 1 not supported\n' }); + + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'closed' })).rejects.toMatchObject({ + code: 'COMMAND_FAILED', + details: { reason: 'fold-posture-dispatch-failed' }, + }); +}); + +test('fails as fold-pose-unverified when the guest never commits the device state', async () => { + stubEmulator({ stateReads: ['2'] }); + + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'closed' })).rejects.toMatchObject({ + code: 'COMMAND_FAILED', + details: { reason: 'fold-pose-unverified', observedDeviceState: 'OPENED' }, + }); + // The settle is bounded: it ended, every retry was preceded by one poll, and the budget ended + // with a read rather than an unobserved interval. + const stateReads = shellCommands().filter( + (command) => command === 'cmd device_state print-state', + ); + expect(stateReads.length).toBeGreaterThan(1); + expect(mockSleep).toHaveBeenCalledTimes(stateReads.length - 1); + expect(shellCommands().at(-1)).toBe('cmd device_state print-state'); +}); + +test('reports the sensor angle as read, without judging it against the pose', async () => { + // Profiles park the hinge at their own angles (the generic "7.6in Foldable" closes at 15°, + // the midpoint of its range), so the committed device state is the pose and the angle is + // information: even one no profile would give for closed is reported, not refused. + stubEmulator({ + states: GENERIC_FOLDABLE_STATES, + stateReads: ['1'], + hinge: 'hinge-angle0 = 77\nOK\n', + }); + + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'closed' })).resolves.toEqual({ + pose: 'closed', + hingeAngleDegrees: 77, + }); +}); + +test('leaves a keyguard that was already showing before the fold alone', async () => { + stubEmulator({ stateReads: ['0'], keyguardReads: [true] }); + + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'closed' })).resolves.toEqual({ + pose: 'closed', + hingeAngleDegrees: 0, + }); + expect(shellCommands()).not.toContain('wm dismiss-keyguard'); + expect(shellCommands().filter((command) => command === 'dumpsys window')).toHaveLength(1); +}); + +test('fails when the lock screen the fold raised keeps coming back', async () => { + stubEmulator({ stateReads: ['0'], keyguardReads: [false, true] }); + + await expect(setAndroidFoldPose(ANDROID_EMULATOR, { pose: 'closed' })).rejects.toMatchObject({ + code: 'COMMAND_FAILED', + details: { reason: 'fold-lock-screen-persists' }, + }); +}); diff --git a/packages/platform-android/src/foldable/pose.ts b/packages/platform-android/src/foldable/pose.ts new file mode 100644 index 0000000000..56b050de38 --- /dev/null +++ b/packages/platform-android/src/foldable/pose.ts @@ -0,0 +1,312 @@ +import type { FoldPose, SetFoldPoseInput } from '@agent-device/contracts/device'; +import type { SetFoldPoseResult } from '@agent-device/contracts/fold-runtime'; +import { emitDiagnostic } from '@agent-device/host-kit/diagnostics'; +import { sleep } from '@agent-device/host-kit/retry'; +import type { DeviceInfo } from '@agent-device/kernel/device'; +import { AppError } from '@agent-device/kernel/errors'; + +import { runAndroidAdb, runAndroidShell } from '../adb.ts'; +import { androidAdbResultError } from '../adb-failure.ts'; +import type { AndroidAdbExecutorResult } from '../adb-transport.ts'; + +/** + * The emulator console's posture ids (`adb emu posture `) and the device state Android + * derives from each. The console moves the hinge sensor to the angle the AVD profile defines for + * the posture (0°, 90°, 180° on the Pixel folds; 15°, 90°, 165° on the generic "7.6in Foldable", + * the midpoints of its posture ranges), and `DeviceStateManager` commits the matching state once + * the guest has seen it. The state *ids* differ per profile too (the Pixel folds number `CLOSED` + * 0, the generic foldables 1), so they are looked up by name, and the committed state, which is + * what `WindowManager`'s `FoldingFeature` derives from, is the verification; the angle is reported + * as the sensor reads it. + */ +const EMULATOR_POSTURES = Object.freeze({ + closed: { id: '1', state: 'CLOSED' }, + 'half-open': { id: '2', state: 'HALF_OPENED' }, + open: { id: '3', state: 'OPENED' }, +} as const); +/** Reads until the guest commits the posture's device state; `attempts × poll` is the settle budget. */ +const ANDROID_FOLD_SETTLE_ATTEMPTS = 60; +const ANDROID_FOLD_SETTLE_POLL_MS = 250; +/** + * The lock screen can land after the device state did, so every read waits a poll first and the + * keyguard must stay away for this many consecutive reads (about 750 ms of quiet) before the fold + * is done. + */ +const LOCK_SCREEN_CLEAR_READS = 3; + +const FOLDABLE_REQUIRED_HINT = + 'fold poses the hinge of a foldable emulator such as the Pixel 9 Pro Fold; this emulator lists no CLOSED, HALF_OPENED, or OPENED device state, so it has no hinge to pose.'; +const KEYFRAMES_HINT = + 'The emulator console poses the hinge at fixed angles only; request closed, half-open, or open.'; +const POSTURE_DISPATCH_HINT = + 'The emulator console refused the posture. Check that the AVD is a foldable profile (hw.sensor.hinge=yes), then retry.'; +const POSE_UNVERIFIED_HINT = + 'The emulator console accepted the posture, but the guest did not commit the matching device state. Read it directly with `adb shell cmd device_state print-state` to see which state the emulator holds.'; +const HINGE_ANGLE_HINT = + 'The emulator console answered the posture but exposes no hinge-angle0 sensor to read back; check that the AVD is a foldable profile (hw.sensor.hinge=yes) and that the emulator build supports `sensor get hinge-angle0`.'; +const LOCK_SCREEN_HINT = + 'The fold lit a panel Android keeps locked and `wm dismiss-keyguard` did not clear it; a keyguard with a PIN or password has to be unlocked by hand.'; + +/** Poses a foldable emulator's hinge through the emulator console and verifies the device state the guest commits. */ +export async function setAndroidFoldPose( + device: DeviceInfo, + input: SetFoldPoseInput, + options: { signal?: AbortSignal } = {}, +): Promise { + const { signal } = options; + signal?.throwIfAborted(); + const pose = requirePosePreset(device, input); + const states = await readDeviceStates(device, signal); + const posture = EMULATOR_POSTURES[pose]; + const expectedState = requirePostureState(device, pose, states); + + // Only a fold to the cover display raises the keyguard, and a keyguard already showing is not + // the fold's to clear: only one that `closed` raises is dismissed. + const lockedBefore = pose === 'closed' && (await isKeyguardShowing(device, signal)); + await sendEmulatorPosture(device, pose, posture.id, signal); + emitDiagnostic({ + level: 'info', + phase: 'android_fold_pose_dispatched', + data: { deviceId: device.id, pose }, + }); + + await awaitDeviceState(device, pose, expectedState, states, signal); + if (pose === 'closed' && !lockedBefore) await dismissFoldLockScreen(device, signal); + return { pose, hingeAngleDegrees: await readHingeAngle(device, signal) }; +} + +/** + * The id of the requested posture's device state. A profile that lists none of the posture states + * has no hinge at all; one that lists some but not the requested one has a hinge the console + * cannot pose this way, which is its own refusal rather than a single-panel one. + */ +function requirePostureState( + device: DeviceInfo, + pose: FoldPose, + states: ReadonlyMap, +): string { + const posture = EMULATOR_POSTURES[pose]; + const expectedState = states.get(posture.state); + if (expectedState !== undefined) return expectedState; + const supportedPoses = (Object.keys(EMULATOR_POSTURES) as FoldPose[]).filter((candidate) => + states.has(EMULATOR_POSTURES[candidate].state), + ); + const listed = supportedPoses.map((candidate) => EMULATOR_POSTURES[candidate].state); + if (listed.length === 0) { + throw new AppError( + 'UNSUPPORTED_OPERATION', + `${device.name} is not a foldable emulator: it lists no CLOSED, HALF_OPENED, or OPENED device state`, + { deviceId: device.id, reason: 'single-panel-device', hint: FOLDABLE_REQUIRED_HINT }, + ); + } + throw new AppError( + 'UNSUPPORTED_OPERATION', + `${device.name} has no ${posture.state} device state to commit for the ${pose} pose; it lists ${listed.join(', ')}`, + { + deviceId: device.id, + reason: 'fold-posture-unsupported', + requestedPose: pose, + supportedPoses, + deviceStates: listed, + hint: `This profile poses only ${supportedPoses.join(' and ')}; request one of those.`, + }, + ); +} + +/** The console poses fixed postures only, so a keyframe trajectory has no Android driver. */ +function requirePosePreset(device: DeviceInfo, input: SetFoldPoseInput): FoldPose { + if (input.pose === undefined) { + throw new AppError('UNSUPPORTED_OPERATION', 'fold keyframes are not supported on Android', { + deviceId: device.id, + reason: 'fold-keyframes-unsupported', + hint: KEYFRAMES_HINT, + }); + } + return input.pose; +} + +/** The device states the guest can commit, by name, as `cmd device_state print-states` lists them. */ +async function readDeviceStates( + device: DeviceInfo, + signal: AbortSignal | undefined, +): Promise> { + const { stdout } = await shell(device, ['cmd', 'device_state', 'print-states'], signal); + const states = new Map(); + for (const match of stdout.matchAll(/identifier=(\d+), name='([A-Z_]+)'/g)) { + states.set(match[2]!, match[1]!); + } + return states; +} + +/** The console answers `OK` or `KO: …` on stdout and exits 0 either way, so the reply is the verdict. */ +async function sendEmulatorPosture( + device: DeviceInfo, + pose: FoldPose, + postureId: string, + signal: AbortSignal | undefined, +): Promise { + const sent = await emulatorConsole(device, ['emu', 'posture', postureId], signal); + if (/^KO\b/m.test(sent.stdout)) { + throw new AppError('COMMAND_FAILED', `Unable to set the emulator posture for ${pose}`, { + deviceId: device.id, + reason: 'fold-posture-dispatch-failed', + stdout: sent.stdout.trim(), + hint: POSTURE_DISPATCH_HINT, + }); + } +} + +/** + * The console returns before the guest reacts, and a posture the console accepted is not yet a + * pose the app sees, so the device state is the verification: it is what `WindowManager`'s + * `FoldingFeature` is derived from. + */ +async function awaitDeviceState( + device: DeviceInfo, + pose: FoldPose, + expectedState: string, + states: ReadonlyMap, + signal: AbortSignal | undefined, +): Promise { + let observed: string | undefined; + // The poll precedes each retry, so the budget ends with a read and a guest that commits in + // the last interval is still seen. + for (let attempt = 1; attempt <= ANDROID_FOLD_SETTLE_ATTEMPTS; attempt += 1) { + signal?.throwIfAborted(); + if (attempt > 1) await sleep(ANDROID_FOLD_SETTLE_POLL_MS, signal); + observed = await readDeviceState(device, signal); + if (observed === expectedState) return; + } + const observedName = [...states].find(([, id]) => id === observed)?.[0]; + throw new AppError( + 'COMMAND_FAILED', + `${device.name} did not reach the ${pose} pose: the emulator still reports device state ${observedName ?? observed}`, + { + deviceId: device.id, + requestedPose: pose, + observedDeviceState: observedName ?? observed, + reason: 'fold-pose-unverified', + hint: POSE_UNVERIFIED_HINT, + }, + ); +} + +async function readDeviceState( + device: DeviceInfo, + signal: AbortSignal | undefined, +): Promise { + const { stdout } = await shell(device, ['cmd', 'device_state', 'print-state'], signal); + return /\d+/.exec(stdout)?.[0]; +} + +/** + * Folding to the cover display raises the keyguard on Pixel images ("Swipe up to continue"): + * Android's continue-using-apps-on-fold setting, which the emulator images do not honour. Left + * alone, every later capture reads the lock screen instead of the app, so it is dismissed the + * way the swipe would. The keyguard can land after the device state did, so it has to stay away + * for consecutive reads before the fold is done. Runs only for `closed` (unfolding lights the + * inner display, which raises nothing) and only when the keyguard was not showing before the + * fold. + */ +async function dismissFoldLockScreen( + device: DeviceInfo, + signal: AbortSignal | undefined, +): Promise { + let clearReads = 0; + let dismissed = false; + for (let attempt = 1; attempt <= ANDROID_FOLD_SETTLE_ATTEMPTS; attempt += 1) { + await sleep(ANDROID_FOLD_SETTLE_POLL_MS, signal); + if (await isKeyguardShowing(device, signal)) { + clearReads = 0; + dismissed = true; + await shell(device, ['wm', 'dismiss-keyguard'], signal); + } else { + clearReads += 1; + if (clearReads >= LOCK_SCREEN_CLEAR_READS) break; + } + } + if (clearReads < LOCK_SCREEN_CLEAR_READS) { + throw new AppError('COMMAND_FAILED', `${device.name} stays locked after the fold`, { + deviceId: device.id, + reason: 'fold-lock-screen-persists', + hint: LOCK_SCREEN_HINT, + }); + } + if (dismissed) { + emitDiagnostic({ + level: 'info', + phase: 'android_fold_lock_screen_dismissed', + data: { deviceId: device.id }, + }); + } +} + +async function isKeyguardShowing( + device: DeviceInfo, + signal: AbortSignal | undefined, +): Promise { + const { stdout } = await shell(device, ['dumpsys', 'window'], signal); + return stdout.includes('isKeyguardShowing=true'); +} + +/** The hinge sensor the guest reads, as the console reports it: `hinge-angle0 = `. */ +async function readHingeAngle( + device: DeviceInfo, + signal: AbortSignal | undefined, +): Promise { + const { stdout } = await emulatorConsole( + device, + ['emu', 'sensor', 'get', 'hinge-angle0'], + signal, + ); + const match = /hinge-angle0\s*=\s*(-?\d+(?:\.\d+)?)/.exec(stdout); + if (!match) { + throw new AppError('COMMAND_FAILED', `${device.name} reports no hinge angle`, { + deviceId: device.id, + reason: 'fold-hinge-angle-unreadable', + stdout: stdout.trim(), + hint: HINGE_ANGLE_HINT, + }); + } + return Number(match[1]); +} + +/** + * An executor may hand back a failed result instead of throwing (the managed scoped transport + * does), so every read checks the exit code itself before trusting stdout: a failed + * `print-states` must be an adb failure, never an empty state list read as a phone. + */ +async function shell( + device: DeviceInfo, + words: readonly string[], + signal: AbortSignal | undefined, +): Promise { + const result = await runAndroidShell(device, words, { signal, allowFailure: true }); + if (result.exitCode !== 0) { + throw androidAdbResultError( + `adb shell ${words.join(' ')} exited with code ${result.exitCode}`, + result, + { deviceId: device.id }, + ); + } + return result; +} + +/** The emulator console through adb, with the same exit-code check as {@link shell}. */ +async function emulatorConsole( + device: DeviceInfo, + args: readonly string[], + signal: AbortSignal | undefined, +): Promise { + const result = await runAndroidAdb(device, args, { signal, allowFailure: true }); + if (result.exitCode !== 0) { + throw androidAdbResultError( + `adb ${args.join(' ')} exited with code ${result.exitCode}`, + result, + { + deviceId: device.id, + }, + ); + } + return result; +} diff --git a/packages/platform-android/src/foldable/runtime.test.ts b/packages/platform-android/src/foldable/runtime.test.ts new file mode 100644 index 0000000000..f73a5e229a --- /dev/null +++ b/packages/platform-android/src/foldable/runtime.test.ts @@ -0,0 +1,102 @@ +import { beforeEach, expect, test, vi } from 'vitest'; + +vi.mock('./pose.ts', () => ({ setAndroidFoldPose: vi.fn() })); + +import { + localRuntimeOwner, + narrowDeviceBinding, + type DeviceBinding, + type RuntimeFacts, +} from '@agent-device/contracts/platform-runtime'; +import { + foldRuntimeUse, + type PlatformRuntimeOperations, +} from '@agent-device/contracts/platform-runtime-operations'; +import { deviceShape, type DeviceInfo } from '@agent-device/kernel/device'; + +import { ANDROID_EMULATOR, UNKNOWN_KIND_DEVICE } from '../runtime.fixtures.ts'; +import { androidFoldableFacts, createAndroidFoldableOperations } from './runtime.ts'; +import { setAndroidFoldPose } from './pose.ts'; + +const mockPose = vi.mocked(setAndroidFoldPose); + +const physical: DeviceInfo = { ...ANDROID_EMULATOR, kind: 'device' }; + +/** A full owner binding whose fold cell is derived only from the fact under test. */ +function foldBinding(device: DeviceInfo): DeviceBinding { + const { setFoldPose } = androidFoldableFacts(device); + return { + device, + owner: localRuntimeOwner('android'), + facts: { + device: { ...deviceShape(device), providerMode: 'local' }, + operations: { setFoldPose } as RuntimeFacts['operations'], + }, + operations: createAndroidFoldableOperations({ device, signal: new AbortController().signal }), + [Symbol.asyncDispose]: async () => {}, + }; +} + +function thrownBy(run: () => T): unknown { + try { + run(); + } catch (error) { + return error; + } + throw new Error('expected the routed fold binding to refuse'); +} + +beforeEach(() => { + mockPose.mockReset(); +}); + +test('an emulator admits setFoldPose and binds the pose operation to the bound signal', async () => { + expect(androidFoldableFacts(ANDROID_EMULATOR).setFoldPose).toEqual({ available: true }); + const { signal } = new AbortController(); + const operations = createAndroidFoldableOperations({ device: ANDROID_EMULATOR, signal }); + expect(operations).toHaveProperty('setFoldPose', expect.any(Function)); + + mockPose.mockResolvedValueOnce({ pose: 'closed', hingeAngleDegrees: 0 }); + await expect(operations.setFoldPose?.({ pose: 'closed' })).resolves.toEqual({ + pose: 'closed', + hingeAngleDegrees: 0, + }); + expect(mockPose).toHaveBeenCalledWith(ANDROID_EMULATOR, { pose: 'closed' }, { signal }); +}); + +test('a physical device and an unknown kind refuse setFoldPose with the typed kind fact', () => { + for (const device of [physical, UNKNOWN_KIND_DEVICE]) { + expect(androidFoldableFacts(device).setFoldPose).toMatchObject({ + available: false, + reason: 'unsupported-device-kind', + hint: expect.stringContaining('folded by hand'), + }); + const operations = createAndroidFoldableOperations({ + device, + signal: new AbortController().signal, + }); + expect(operations).not.toHaveProperty('setFoldPose'); + } + expect(mockPose).not.toHaveBeenCalled(); +}); + +test('the narrowed fold use refuses a physical device with the typed fact reason and hint', () => { + const refusal = thrownBy(() => narrowDeviceBinding(foldBinding(physical), foldRuntimeUse)) as { + code?: string; + details?: { reason?: string; hint?: string }; + }; + expect(refusal.code).toBe('UNSUPPORTED_OPERATION'); + expect(refusal.details?.reason).toBe('unsupported-device-kind'); + expect(refusal.details?.hint).toContain('folded by hand'); +}); + +test('an aborted scope never reaches the emulator console', async () => { + const controller = new AbortController(); + const operations = createAndroidFoldableOperations({ + device: ANDROID_EMULATOR, + signal: controller.signal, + }); + controller.abort(); + await expect(operations.setFoldPose?.({ pose: 'open' })).rejects.toThrow(); + expect(mockPose).not.toHaveBeenCalled(); +}); diff --git a/packages/platform-android/src/foldable/runtime.ts b/packages/platform-android/src/foldable/runtime.ts new file mode 100644 index 0000000000..b5438bae38 --- /dev/null +++ b/packages/platform-android/src/foldable/runtime.ts @@ -0,0 +1,44 @@ +import type { SetFoldPoseInput } from '@agent-device/contracts/device'; +import { foldRuntimeOperationFacts } from '@agent-device/contracts/fold-runtime'; +import type { RuntimeOperationFact } from '@agent-device/contracts/platform-runtime'; +import { whenAdmitted } from '@agent-device/contracts/platform-runtime'; +import type { DeviceInfo } from '@agent-device/kernel/device'; + +import { setAndroidFoldPose } from './pose.ts'; + +const available = Object.freeze({ available: true } as const); + +const foldKindUnavailable = Object.freeze({ + available: false, + reason: 'unsupported-device-kind', + hint: 'fold poses the hinge of a foldable Android emulator through the emulator console; a physical device is folded by hand.', +} as const); + +/** + * The emulator console is the only Android posture control, so the kind is the whole admission. + * Whether the emulator *profile* carries a hinge is not in `DeviceInfo`; the operation answers + * that from the device states the guest lists and refuses a phone profile with a typed + * `UNSUPPORTED_OPERATION`, the way the Apple owner answers from CoreDevice's display table. + */ +function androidFoldFact(device: DeviceInfo): RuntimeOperationFact { + return device.kind === 'emulator' ? available : foldKindUnavailable; +} + +/** The foldable cell: `setFoldPose`. */ +export function androidFoldableFacts(device: DeviceInfo) { + return foldRuntimeOperationFacts({ fold: androidFoldFact(device) }); +} + +/** Binds `setFoldPose` when {@link androidFoldableFacts} admitted it. */ +export function createAndroidFoldableOperations(params: { + device: DeviceInfo; + signal: AbortSignal; +}) { + const { device, signal } = params; + return whenAdmitted(androidFoldableFacts(device).setFoldPose, () => ({ + setFoldPose: async (input: SetFoldPoseInput) => { + signal.throwIfAborted(); + return await setAndroidFoldPose(device, input, { signal }); + }, + })); +} diff --git a/packages/platform-android/src/runtime.test.ts b/packages/platform-android/src/runtime.test.ts index 011afb24f9..1be6b47804 100644 --- a/packages/platform-android/src/runtime.test.ts +++ b/packages/platform-android/src/runtime.test.ts @@ -215,22 +215,27 @@ test('Android refuses the action-button fact on every kind', async () => { } }); -test('Android refuses the fold fact on every kind', async () => { - for (const runtimeDevice of [ +test('Android admits the fold fact on emulators and refuses every other kind', async () => { + const emulator = await bindOrdinary( + createAndroidPlatformRuntime(androidNavigationHost()), ANDROID_EMULATOR, + ); + expect(emulator.facts.operations.setFoldPose).toEqual({ available: true }); + expect(emulator.operations.setFoldPose).toEqual(expect.any(Function)); + for (const runtimeDevice of [ { ...ANDROID_EMULATOR, kind: 'device' as const }, + { ...ANDROID_EMULATOR, kind: 'simulator' as const }, UNKNOWN_KIND_DEVICE, ]) { const binding = await bindOrdinary( createAndroidPlatformRuntime(androidNavigationHost()), runtimeDevice, ); - // fold drives only foldable iPhone simulators; the Android emulator posture control is not - // driven yet. - expect(binding.facts.operations.setFoldPose).toEqual({ + // The emulator console is the only Android posture control; a physical foldable is folded + // by hand, so this is the kind refusal rather than the touch gate `orientation` rides. + expect(binding.facts.operations.setFoldPose).toMatchObject({ available: false, - reason: 'unsupported-platform-leaf', - hint: 'fold drives the hinge of a foldable iPhone simulator; the Android emulator posture control is not driven by agent-device yet.', + reason: 'unsupported-device-kind', }); expect(binding.operations.setFoldPose).toBeUndefined(); } diff --git a/packages/platform-android/src/runtime.ts b/packages/platform-android/src/runtime.ts index c5ae12aa49..fe691b14ed 100644 --- a/packages/platform-android/src/runtime.ts +++ b/packages/platform-android/src/runtime.ts @@ -50,6 +50,7 @@ import { orientationRuntimeOperationFacts } from '@agent-device/contracts/orient import { tvRemoteRuntimeOperationFacts } from '@agent-device/contracts/tv-remote-runtime'; import type { DeviceInfo } from '@agent-device/kernel/device'; import { createHostAudioProbeCaptureOperations } from '@agent-device/capture-kit'; +import { androidFoldableFacts, createAndroidFoldableOperations } from './foldable/runtime.ts'; import { androidAudioProbeCaptureFact } from './audio-runtime.ts'; import { ANDROID_CLIPBOARD_SHELL_COMMAND_UNAVAILABLE_HINT } from './clipboard-shell-response.ts'; import { createAndroidPerfOperations } from './perf/runtime.ts'; @@ -91,15 +92,6 @@ const systemButtonUnavailable = unavailableFact( 'unsupported-platform-leaf', 'Android has no key event for this system button.', ); -/** - * Foldable Android emulators do carry a posture control (the emulator console's `fold` and - * `posture` commands), but nothing in this project drives it yet, so the cell refuses on every - * kind rather than advertising a pose it cannot set. - */ -const foldUnavailable = unavailableFact( - 'unsupported-platform-leaf', - 'fold drives the hinge of a foldable iPhone simulator; the Android emulator posture control is not driven by agent-device yet.', -); const headlessUnavailable = unavailableFact( 'unsupported-device-kind', 'Headless boot is supported only for Android emulators.', @@ -348,7 +340,7 @@ export function createAndroidPlatformRuntime(host: PlatformRuntimeHost): Platfor home: androidTouchFact(device), appSwitcher: androidTouchFact(device), }), - setFoldPose: foldUnavailable, + ...androidFoldableFacts(device), // The deep link opens through `am start`, admitted wherever the retired `ANDROID_ALL` // bucket admitted it. ...appEventRuntimeOperationFacts({ triggerAppEvent: androidTouchFact(device) }), @@ -463,6 +455,10 @@ export function createAndroidPlatformRuntime(host: PlatformRuntimeHost): Platfor }), ), ...androidInteractionOperations(host, request, facts), + ...createAndroidFoldableOperations({ + device: request.device, + signal: request.scope.signal, + }), ensureReady: async (input: EnsureReadyInput) => await ensureAndroidReady( host, diff --git a/src/commands/system/index.ts b/src/commands/system/index.ts index e0a593d588..12eed6fcd4 100644 --- a/src/commands/system/index.ts +++ b/src/commands/system/index.ts @@ -112,7 +112,9 @@ export const SYSTEM_COMMAND_OUTPUT_SCHEMAS = { { action: constSchema('fold'), pose: enumSchema(FOLD_POSES), - hingeAngleDegrees: numberSchema('Hinge angle CoreDevice read back after the pose settled.'), + hingeAngleDegrees: numberSchema( + 'Hinge angle read back after the pose settled: from CoreDevice on iOS, from the emulator hinge sensor on Android.', + ), screen: objectSchema( { display: stringSchema('CoreDevice name of the panel the device now lights.'), @@ -236,10 +238,10 @@ const foldGuidance = { unsupportedScope: 'UNSUPPORTED_OPERATION and reason unsupported-device-scope', } as const; -const foldCommandDescription = `Fold or unfold a foldable iPhone simulator (iPhone Duo) into the closed, half-open, or open pose, or follow timestamped angle keyframes, by sending a ${foldGuidance.hingeEvent}, then read the hinge angle back from CoreDevice to confirm it. A pose change moves the app to a different panel with a different point size, so every ref and coordinate from before it is stale: re-snapshot after this command. Taps, long presses, and scrolling target the app window on its current panel in closed, half-open, and open poses. Simulator-only; requires the iOS simulator SDK; ${foldGuidance.permissions} A simulator scoped to a non-default simulator set is refused with ${foldGuidance.unsupportedScope}; run fold against a simulator in the default set.`; +const foldCommandDescription = `Fold or unfold a foldable iPhone simulator (iPhone Duo) or a foldable Android emulator into the closed, half-open, or open pose. iOS sends a ${foldGuidance.hingeEvent}, then reads the hinge angle back from CoreDevice to confirm it, and can follow timestamped angle keyframes; Android sets the emulator console posture, then confirms the device state the guest commits and reports the hinge angle the AVD profile defines for that posture, and dismisses the lock screen a fold to the cover display can raise. On iOS a pose change moves the app to a different panel with a different point size; on Android a fold to or from closed moves the app between the cover and inner displays, and any pose change can reshape the app window, whose size the next snapshot reports. Either way every ref and coordinate from before it is stale: re-snapshot after this command. Taps, long presses, and scrolling target the app window on its current panel in closed, half-open, and open poses. Simulator and emulator only; iOS requires the iOS simulator SDK; ${foldGuidance.permissions} A simulator scoped to a non-default simulator set is refused with ${foldGuidance.unsupportedScope}; run fold against a simulator in the default set.`; export const foldableHelpTopic = { - summary: 'Foldable Apple devices: panels, pose, and which screen you are on', + summary: 'Foldable devices: panels, pose, and which screen you are on', body: `agent-device help foldable A foldable Apple device (iPhone Duo) carries two integrated panels, which Apple calls the outer display and the inner display. Only one is lit at a time, and which one is lit is the device pose. @@ -256,6 +258,7 @@ Changing the pose: Timed motion: fold --keyframes '[{"atMs":0,"angle":0},{"atMs":5000,"angle":180}]'. Use 2–64 frames starting at 0ms, increasing integer timestamps up to 60000ms, and angles from 0 to 180. The last timestamp sets motion duration, excluding setup and verification. Equal angles hold; cancellation stops motion. See the fold examples in the command and Node API documentation for trajectories. Requirements: an iOS simulator session on a foldable device and an Xcode toolchain with the iOS simulator SDK and foldable HID support (verified on Xcode 27.1). ${foldGuidance.permissions} The command runs a small helper through simctl spawn for the session UDID; the helper is built once per Fold.m source hash and Xcode toolchain, cached under ~/.agent-device/fold-helper, and rebuilt only when the source or the toolchain changes. Build or dispatch failures are reported without a UI fallback. The app under test reads the resulting pose as UIHinge.status. + Android emulators: fold sets the posture through the emulator console (adb emu posture 1|2|3), then reads adb shell cmd device_state print-state until the guest commits CLOSED, HALF_OPENED, or OPENED, and reports the hinge sensor's angle, which the AVD profile defines per posture: 0, 90 and 180 degrees on the Pixel folds, 15, 90 and 165 on the generic 7.6in Foldable. Folding to the cover display can raise Android's "Swipe up to continue" lock screen; fold dismisses it so the app stays reachable. An emulator profile without those device states (a phone) fails with UNSUPPORTED_OPERATION and reason single-panel-device, and a physical device is refused. Keyframes are not supported on Android, and the response carries no panel geometry: re-snapshot after every fold. If a task asserts behavior for more than one pose, fold to each pose and re-snapshot, and report which poses the run covered.`, } as const; @@ -507,8 +510,8 @@ const orientationCommandFacet = defineCommandFacet({ const foldCommandFacet = defineCommandFacet({ name: FOLD_COMMAND_NAME, text: { - summary: 'Fold or unfold a foldable iPhone simulator', - cliDetail: `iPhone Duo simulators only. Sends a ${foldGuidance.hingeEvent} and confirms the hinge angle through CoreDevice; refs and coordinates do not survive a pose change.`, + summary: 'Fold or unfold a foldable iPhone simulator or Android emulator', + cliDetail: `iPhone Duo simulators and foldable Android emulators. iOS sends a ${foldGuidance.hingeEvent} and confirms the hinge angle through CoreDevice; Android sets the emulator console posture and confirms the device state. Refs and coordinates do not survive a pose change.`, }, metadata: foldCommandMetadata, run: (client, input) => client.command.fold(input), diff --git a/test/integration/command-coverage/declarations.ts b/test/integration/command-coverage/declarations.ts index b3adbb26c4..0646bdd3ea 100644 --- a/test/integration/command-coverage/declarations.ts +++ b/test/integration/command-coverage/declarations.ts @@ -1333,7 +1333,7 @@ const COMMAND_COVERAGE_DECLARATIONS = { [C.fold]: { androidEmulator: androidEmulator.contract( ANDROID_FOLD_RUNTIME_CONTRACT_EVIDENCE, - 'the Android runtime fact refuses a foldable hinge pose on every kind', + 'the Android emulator advertises the hinge pose and binds it; a phone profile is refused by the operation itself, a physical device by the fact', ), iosSimulator: iosSimulator.contract( 'packages/platform-apple/src/runtime.test.ts', diff --git a/test/integration/command-coverage/evidence.ts b/test/integration/command-coverage/evidence.ts index 42aed9865b..ccd78dc1e5 100644 --- a/test/integration/command-coverage/evidence.ts +++ b/test/integration/command-coverage/evidence.ts @@ -48,7 +48,7 @@ export const ANDROID_FOLD_RUNTIME_CONTRACT_EVIDENCE: AndroidContractEvidence = defineAndroidContractEvidence( 'packages/platform-android/src/runtime.test.ts', [C.fold], - 'Android refuses the fold fact on every kind', + 'Android admits the fold fact on emulators and refuses every other kind', ); export const ANDROID_VIEWPORT_RUNTIME_CONTRACT_EVIDENCE: AndroidContractEvidence = defineAndroidContractEvidence( diff --git a/website/docs/docs/client-api.md b/website/docs/docs/client-api.md index 9b4ea68bd6..1c8b19732b 100644 --- a/website/docs/docs/client-api.md +++ b/website/docs/docs/client-api.md @@ -304,7 +304,7 @@ await client.command.fold({ }); ``` -`fold` accepts either `pose` or `keyframes`. Keyframes use linear interpolation at roughly 60 updates per second; repeat an angle to hold it. Timestamps must start at zero and increase strictly, with 2–64 frames and a final timestamp no greater than 60,000ms. Angles must be finite and between 0° and 180°. The final timestamp bounds motion, excluding helper preparation and final hinge verification. A custom final angle is verified within 0.5°; interior angles must also settle. Cancellation stops the motion at its current angle. Re-snapshot afterwards, including after interrupted motion. +`fold` accepts either `pose` or `keyframes`. Keyframes use linear interpolation at roughly 60 updates per second; repeat an angle to hold it. Timestamps must start at zero and increase strictly, with 2–64 frames and a final timestamp no greater than 60,000ms. Angles must be finite and between 0° and 180°. The final timestamp bounds motion, excluding helper preparation and final hinge verification. A custom final angle is verified within 0.5°; interior angles must also settle. Cancellation stops the motion at its current angle. Re-snapshot afterwards, including after interrupted motion. Android emulators accept `pose` only. `press`, `click`, and `longpress` take `readinessTimeoutMs`. With it, the command waits up to that many milliseconds for a target that is not on screen yet, then performs the requested interaction. Without it, the command looks once and fails at once, which is the right choice for an agent that most often misses because the selector is wrong. Use it in scripted flows, where a step can run before the screen finishes rendering: diff --git a/website/docs/docs/commands.md b/website/docs/docs/commands.md index 146bf6ba67..47a659d89d 100644 --- a/website/docs/docs/commands.md +++ b/website/docs/docs/commands.md @@ -120,9 +120,10 @@ agent-device fold open - `action-button` is slow to loop. On an iPhone 17 Pro Simulator the press itself spent about five seconds inside XCUITest, while `home` and `app-switcher` on the same session took under two seconds each. - `fold ` puts a foldable iPhone simulator (iPhone Duo) into a hinge pose. The command sends a hinge event inside the selected simulator, then reads the hinge angle back with `devicectl device motion hinge-angle` and reports the pose only when that reading agrees: `closed` is 0°, `open` is 180°, and `half-open` is any angle between them (requested at 130°). An angle inside that interval only proves the category, so `half-open` is reported once two consecutive readings both fall inside it and agree within 0.5°. The response names the panel the device now lights and its native panel point size, marked `coordinateSpace: "native-panel"`; that size is the panel's own geometry, not the next snapshot's viewport (a 669x951 inner panel can host a 951x669 app window), so it cannot place a tap. Re-snapshot afterwards, and never carry refs or coordinates across a `fold`. After that snapshot, taps, long presses, and scrolling follow the app window on the active panel in closed, half-open, and open poses. - For timed motion, use `fold --keyframes '[{"atMs":0,"angle":0},{"atMs":1667,"angle":160},{"atMs":3333,"angle":100},{"atMs":5000,"angle":180}]'`. This runs the opening/reversal/reopening sequence over five seconds. Supply either a preset or keyframes, never both. Use 2–64 keyframes starting at 0ms with strictly increasing integer timestamps up to 60,000ms and finite angles in 0–180°. Linear interpolation runs at roughly 60Hz; equal consecutive angles hold the hinge. Motion duration excludes preparation and final-angle verification. Cancellation stops at the current angle; re-snapshot even after an interrupted trajectory. -- `fold` is simulator-only and requires an Xcode toolchain with the iOS simulator SDK and foldable HID support (verified on Xcode 27.1). It runs a helper against the session UDID; the helper is built once per `Fold.m` source hash and Xcode toolchain, cached under `~/.agent-device/fold-helper`, and rebuilt only when the source or the toolchain changes. Device Hub and host Accessibility permission are not required. Build failures report `fold-helper-build-failed`; dispatch failures report `fold-hid-dispatch-failed`. There is no UI fallback. Single-panel simulators and physical devices are refused. +- On iOS, `fold` is simulator-only and requires an Xcode toolchain with the iOS simulator SDK and foldable HID support (verified on Xcode 27.1). It runs a helper against the session UDID; the helper is built once per `Fold.m` source hash and Xcode toolchain, cached under `~/.agent-device/fold-helper`, and rebuilt only when the source or the toolchain changes. Device Hub and host Accessibility permission are not required. Build failures report `fold-helper-build-failed`; dispatch failures report `fold-hid-dispatch-failed`. There is no UI fallback. Single-panel simulators and physical devices are refused. - A simulator scoped to a non-default set with `--ios-simulator-device-set` is refused before any hinge is touched with `UNSUPPORTED_OPERATION` and `details.reason: "unsupported-device-scope"`, because `devicectl` cannot read the hinge pose of a simulator outside the default set. Run `fold` against a simulator in the default set. - `fold` costs one bounded hinge stream per read, and devicectl's smallest stream is five seconds: `closed` and `open` take about ten seconds, `half-open` about sixteen, because the hinge animates and the command waits for it to stop. A hinge whose last reading is some other pose fails with `COMMAND_FAILED` and `reason: fold-pose-unverified`, naming the angle CoreDevice still reports. A hinge seen `half-open` but never at rest fails with `reason: fold-pose-unsettled`, naming the observed and previous angles: the requested category was observed, and what is missing is a pose the hinge holds. +- On a foldable Android emulator (`pixel_9_pro_fold`, `pixel_10_pro_fold`, the generic `7.6in Foldable`), `fold` sets the posture through the emulator console (`adb emu posture`), waits until the guest commits the matching device state (`adb shell cmd device_state print-state`), and reports the hinge sensor's angle, which the AVD profile defines per posture: 0°, 90° and 180° on the Pixel folds, 15°, 90° and 165° on the generic 7.6in Foldable. The committed device state is the verification; it is what `WindowManager`'s `FoldingFeature` derives from. Folding to the cover display can raise Android's "Swipe up to continue" lock screen; `fold` dismisses it so the app stays reachable. The response carries no panel geometry and keyframes are not supported. A phone profile, which lists none of those device states, fails with `UNSUPPORTED_OPERATION` and `reason: single-panel-device`; a physical device is refused, since foldable hardware is folded by hand. - On iOS devices, `http(s)://` URLs open in Safari when no app is active. Custom scheme URLs require an active app in the session. - Commands that need one concrete device refuse to guess: if no `--device`/`--udid`/`--serial` is given and several candidates are equally preferred (for example two booted emulators), the command fails with `AMBIGUOUS_MATCH` and lists them, rather than picking one and returning a successful answer about a device you did not select. Preferences still apply first — virtual over physical, booted over offline — so one booted emulator beside offline ones resolves normally, as does any command running inside an existing session. `devices` still lists every device. - Commands that omit `--session` use an implicit session scoped to the caller's git worktree or working directory and to the selected platform (`ios`, `android`; the platform-less `default` session only without `--platform`), so independent local agents do not attach to each other's sessions. Use `--session ` or `AGENT_DEVICE_SESSION` to share a named session, and `--session-lock reject|strip` (or `AGENT_DEVICE_SESSION_LOCK`) to set its lock mode. See [Sessions](/docs/sessions#lock-a-named-session-to-a-device).