Skip to content

Apple driver: SwiftUI's identifiers, multi-line fields, dismiss, turned windows, late presentations, scene phase; launchers fresh, prompt to stop, with arguments - #231

Merged
logan-shire-agent[bot] merged 11 commits into
mainfrom
apple-driver-gaps
Nov 16, 2026

Conversation

@logan-shire-agent

@logan-shire-agent logan-shire-agent Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Gaps an app's UI tests found driving its app headful on an iPhone simulator through Isomer's Apple driver, and on an Android emulator. For 0.56.

The Apple driver (drive/apple)

  • Identifiers of SwiftUI's own elements. A List's or a Form's rows, the controls in them, text fields and a principal toolbar item are SwiftUI AccessibilityNodes, which answer accessibilityIdentifier without declaring UIAccessibilityIdentification, so the driver's cast found nothing where XCUITest, which asks the selector, finds them all. The driver now asks the selector (AccessibilityScreen.identifier).
  • A multi-line TextField (a UITextView) is labelled by its placeholder: accessibility's placeholder value while it is empty, else the placeholder label it keeps hidden once there is text, else its identifier.
  • dismiss closes the topmost presentation: escape first; else a sheet a person may swipe away, closed as the swipe closes it (its presentation controller's delegate asked and told, so SwiftUI's isPresented and onDismiss follow; not when isModalInPresentation); else its own Cancel, Close or Done. back at a sheet's first page closes the sheet, as Isomer's driver does.
  • Taps in a turned window. A tap that falls back to where the element stands now maps accessibility's screen point into the window by UIKit's own mapping (UIAccessibility.convertToScreenCoordinates, inverted), not through UIScreen.coordinateSpace; where that point reaches nothing that takes a tap, the element's accessibility containers lead to its cell or control.
  • info says how the interface stands (Info.orientation, new in headless.proto).
  • Settling. After an action, a screen still as it was gets up to 1.5 s more to change (a sheet presented from a sheet, Quick Look's controls), and a transition that starts while it settles is waited out. ls, info, inspect and quit answer with the reading they settled on instead of settling twice.
  • Attaching a tenth of a second after launch, not half a second.
  • ISO_DRIVE_TRACE=1 prints the driver's walk of the tree.

Launchers and HeadlessApp

  • Stop at once. Every launcher watches for quit()'s stop file (requests.sh, browser_check), and quit() also sends SIGTERM at once off Windows, so it no longer waits 10 s.
  • Fresh by default. HeadlessApp(launcher:environment:arguments:fresh:timeout:), fresh: true by default (the launcher's ISOMER_FRESH=1): a test's app starts with none of its data from before, as an app built headless does. The default is on because that is what the same test gets headless and on the web, and a test that wants data kept is the exception. iPhone: the app's defaults are deleted through the simulator's cfprefsd by each plist's path (deleting them by name reaches the simulator's own domain, and a file removed under cfprefsd comes back), then its files and its app groups', and the simulator's keychain is reset. Android: pm clear. Mac: its defaults are deleted (cfprefsd keeps them in the user's real home whatever CFFIXED_USER_HOME says, so the Mac launcher's home of its own never isolated them) and its home is emptied. Linux: its home (HOME, XDG_*) is emptied. Web: a new browser profile every launch anyway. fresh: false keeps the data. Windows keeps it for now.
  • The app's arguments. <launcher> <hub> <name> [KEY=VALUE…] [-- <argument>…]: the iPhone's through simctl, the Mac's and Linux's on the command line, Android's as ISO_ARGUMENTS; HeadlessApp(…, arguments:).
  • Pictures by orientation. picture names Android's phone-landscape in a wide window, as the iPhone's. The iPhone's launcher turns a picture of an app that turned itself to landscape on an upright simulator (the request carries the driver's orientation; sips, only when the frame is upright).
  • Faster pictures. The requests are polled every 50 ms, not 200 ms, and picture no longer asks for the screen twice: about 0.6 s a picture on the iPhone, down from 1.7 s.
  • The iPhone launcher keeps the app's output with the test's outputs (<name>.log).
  • A HeadlessApp released without quit() (a test that threw first) stops its launcher.
  • iso launch <launcher> [--name <name>] [--keep-data] [KEY=VALUE…] [-- <argument>…] and iso launch stop <name>: a launcher run in the background, its app dialling the CLI's hub.

Scene phase: scene <active|inactive|background>

A driver command (headless.proto's scene, its phase in value), iso scene <phase> and HeadlessApp.scene(.background | .inactive | .active): the app's scene phase moved as leaving the app and coming back does, so what an app does on .onChange(of: scenePhase) (a sign-in prompt on return) runs under test.

  • Isomer's driver (headless, the web, Linux, Windows): the runtime's phase (Runtime.setScenePhase).
  • iOS (the Apple driver, in process): SwiftUI's own scene delegate is told each step UIKit tells it (sceneWillResignActive, sceneDidEnterBackground, sceneWillEnterForeground, sceneDidBecomeActive), with UIKit's scene and application notifications posted after each. Tried on an iPhone 17 simulator (iOS 27.2) with a one-file SwiftUI app: the delegate calls move scenePhase through inactive, background, inactive, active, and the app stays on screen and driven; posting the notifications alone moves nothing; a real trip home (another app launched by simctl, then simctl launch again, which brings the same process back) works too but takes about 7 s to reach background and suspends the app, so its driver can't answer meanwhile.
  • The Mac (the Apple driver): background is the app hidden (NSApp.hide), inactive its windows in the Dock, active shown and brought to the front. Tried the same way: hiding gives background, miniaturizing inactive; deactivating the app alone leaves SwiftUI's phase active.
  • Android: HeadlessApp.scene (and iso scene for an app iso launch started) asks the launcher first (requests.sh's scene:<phase> request, answered in <n>.answer): android_launcher.sh sends the app home (KEYCODE_HOME: paused, stopped) and brings it back with am start (the singleTask activity's task brought to the front, the same activity and process: onRestart, onStart, onResume), waiting until it is stopped or resumed. Inactive has no such way, nor has any other launcher: the driver's command does it (a launcher that doesn't take the request removes it unanswered, or answers why not).
  • Tests. The driver probe's scene screen (its phase, each change, and "Welcome back" with a Sign in button when it comes back from the background) and DriverProbeTests.sceneLeavesAndComesBack (headless, and headful on the iPhone through the iOS launcher); StorybookHeadlessTests.comesBackFromTheBackground (a story's state kept across a trip to the background, the app still driven) on every platform the Storybook's tests run on, Android's launcher way included.

CI: GitHub's xcode-27 image 20261112.0361.1

The first verify run's apple (ios) and apple (pictures) failed because CoreSimulator went down under them partway through: every simctl after that answered CoreSimulatorService: Connection interrupted. On the iPhone, 6 of the Storybook's 17 tests and all 16 of the probe's then failed at launch; on the pictures job, 9 stories didn't launch. Both jobs ran on image 20261112.0361.1, and earlier runs on 20261104.0337.1 passed. The new image adds Xcode 27.3 beta and macOS 27.2, and has simdiskimaged unmount an iOS runtime it mounted twice while a simulator runs from it (actions/runner-images#15334, #14946). apple (macos) and apple (hosted) don't depend on a simulator surviving long, and passed.

tools/ci/hosted_simulator.sh, sourced by hosted_apple_test.sh and hosted_headful_test.sh:

  • waits for every runtime to be mounted (simctl runtime list -j) before it makes the simulator;
  • when a story or a test run fails with the simulator gone, starts CoreSimulator again, boots the simulator again (a new one if its runtime is gone), prepares it as before (apps installed, appearance and status bar), and tries again what failed: the story, or the failed tests (Swift Testing's --filter), at most twice;
  • keeps CoreSimulator's crash reports in the job's artifact.

A test that fails while the simulator is up still fails.

Tests

  • The driver probe on Apple's SwiftUI. //apple/driver_probe_ios_application builds the probe twice: upright, and landscape-only (:driver_probe_landscape_ios_application). New screens: a Form (identified controls in rows, a field, a multi-line field, a principal title button) and sheets (one with only Done, one with nothing but a swipe, one over another). DriverProbeTests run headful through the iOS launcher (:driver_probe_headful_apple_runner). New tests: form identifiers, the multi-line field's label, dismiss, a sheet over a sheet, launch arguments, fresh and kept launches, an app turned to landscape (taps, back, info, the picture turned), and a selection row's tap in landscape.
  • CI's container builds the probe apps and runner beside the Storybook's headful apps (.ci/platforms/headful/probe). hosted_headful_test.sh apple installs them on the job's simulator and runs them in its ios part. No workflow change is needed: the existing headful-apps-apple artifact carries the folder.
  • StorybookHeadlessTests.aLaunchStartsFreshUnlessAskedToKeep (state.appstorage's stars) runs on every headful platform that has a launcher.

Verified

On a macOS VM with Xcode 27.2, on an iPhone 17 simulator (iOS 27.2), with the apps built there with --@isomer//drive:headful:

  • The probe's 16 tests pass headful through the iOS launcher. Before the fixes, identifiers, dismiss, sheet-at-root back and fresh launches failed.
  • The Storybook's 17 headful tests pass on the iPhone through the iOS launcher and on the Mac through the Mac launcher (in the VM's GUI session), the new fresh test included. That test caught that the Mac's defaults outlived a fresh launch, fixed here.
  • The Mac launcher passes arguments after -- to the app's command line (checked with ps), and ends the app on the stop file alone.
  • The landscape picture is 2622×1206, the right way up.
  • Timings: attach about 3 s including the fresh clearing, a picture 0.6 s, a tap 0.75 s, info and ls 0.3 s, quit 0.45 s.

The selection row's tap in landscape passed with the old coordinate mapping as well, so that failure isn't reproduced by the probe. The new mapping is exact by construction, and the container fallback doesn't depend on coordinates.

…wn elements, multi-line fields' labels, dismiss and back closing sheets, taps in a turned window, settling for a late presentation; launchers that start fresh, stop at once, pass arguments and turn pictures
… Mac, deletes its defaults through cfprefsd (CFFIXED_USER_HOME doesn't move them); a kept launch sees the data a fresh one left; taps fall back to the element's own cell
…ain when GitHub's image takes it down (actions/runner-images#15334), then try again what failed
…aving it and coming back does (headless.proto, iso, HeadlessApp.scene); Isomer's runtime's phase, SwiftUI's scene delegate on iOS, hidden or in the Dock on the Mac, Android's launcher through the activity's lifecycle; the probe's scene screen and its test
@logan-shire-agent logan-shire-agent Bot changed the title Apple driver: SwiftUI's identifiers, multi-line fields, dismiss, turned windows, late presentations; launchers fresh, prompt to stop, with arguments Apple driver: SwiftUI's identifiers, multi-line fields, dismiss, turned windows, late presentations, scene phase; launchers fresh, prompt to stop, with arguments Nov 16, 2026
@logan-shire-agent
logan-shire-agent Bot merged commit 2620982 into main Nov 16, 2026
25 checks passed
@logan-shire-agent
logan-shire-agent Bot deleted the apple-driver-gaps branch November 16, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant