Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 51 additions & 9 deletions docs/drive.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ driving is something a build opts into, never an API an app calls.
## The protocol and its transports

shared/wire/headless.proto is the driver's protocol (ls, info, tap, type,
set, select, swipe, longpress, back, dismiss, wait, quit); shared/wire/session.proto a
set, select, swipe, longpress, back, dismiss, scene, wait, quit); shared/wire/session.proto a
head's session (drawing the app). An app serves both as an `IsomerServer`
(drive/serving), apart from any transport; adapters carry it:

Expand Down Expand Up @@ -105,8 +105,12 @@ iso swipe "Weekly report" left # a list row's trailing swipe actions, listed as
iso swipe "Weekly report" left Delete # or the one named, performed
iso longpress "Weekly report" # its context menu: its items are the screen's buttons
iso back | dismiss
iso scene background # the app left (its scene phase), then come back to:
iso scene active # .onChange(of: scenePhase) sees each step
iso wait "Taps: 1" --timeout 10
iso drive # JSON lines on standard input
iso launch …/drive/launchers/ios_launcher.sh # a headful app of a launcher's (below)
iso launch stop ios
```
An element's name is its role and its place among that role's on the
Expand Down Expand Up @@ -158,9 +162,29 @@ Apple's frameworks): with the flag, @isomer//apple/libraries/swift_ui links
drive/apple, which a constructor starts (the app calls nothing). It answers
the same commands from the app's accessibility tree, with the same names:
on iOS it walks UIKit's accessibility containers once the app's
accessibility is on, scrolls to what is off screen as a person would, waits
out navigation transitions, and taps what takes no activation (a list's
cell, a control) as a tap would; on the Mac it asks its own process as an
accessibility is on (SwiftUI's own elements, a List's or a Form's rows and
the controls in them, give their identifiers to the selector, as XCUITest
reads them, not through UIAccessibilityIdentification; a multi-line
`TextField` is labelled by its placeholder), scrolls to what is off screen
as a person would, waits out navigation transitions and, after an action,
what it starts a moment later (a sheet presented from a sheet), and taps
what takes no activation (a list's cell, a control) as a tap would, at its
place in the window however the app is turned. `dismiss` closes what is
presented on top: escape, else a sheet a person may swipe away, as the
swipe does (its `onDismiss` runs), else its own Cancel, Close or Done.
`scene` moves SwiftUI's scene phase as leaving the app and coming back
does: on iOS, SwiftUI's own scene delegate is told each step UIKit tells it
(will resign active, did enter background, will enter foreground, did become
active) and UIKit's notifications are posted after each, the app staying on
screen and driven (posting the notifications alone moves nothing, and a
simulator's real trip to the home screen takes seconds and suspends the
app); on the Mac the app is hidden (background), its windows put in the
Dock (inactive) or shown and brought to the front (active). Isomer's own
driver sets its runtime's phase; a test's `HeadlessApp.scene` asks the
launcher first, and Android's sends the app home and brings it back, its
activity stopped and resumed, as `iso scene` does for an app `iso launch`
started. ISO_DRIVE_TRACE=1 in the app's environment prints each reading's walk of
the tree; on the Mac it asks its own process as an
accessibility client would (AXUIElement), on the main thread. A list's row
of texts is a button, as Isomer's renderers give it. Launch with simctl's
`SIMCTL_CHILD_ISO_DRIVER=…`, or a Mac app with `ISO_DRIVER` in its
Expand All @@ -180,11 +204,29 @@ try app.tap("Tap")
try app.wait(for: "Taps: 1")
```
`HeadlessApp(launcher:environment:)` drives a headful app with the same
test: the test holds a hub of its own (IsomerServing's `IsomerHub`, on a
loopback WebSocket), and the launcher, run as `<launcher> <hub URL> <name>
[KEY=VALUE…]`, starts the app dialling out to it (ISO_DRIVER,
ISO_DRIVER_NAME) and ends it when stopped. A launcher a platform, beside the
`HeadlessApp(launcher:environment:arguments:fresh:)` drives a headful app
with the same test: the test holds a hub of its own (IsomerServing's
`IsomerHub`, on a loopback WebSocket), and the launcher, run as
`<launcher> <hub URL> <name> [KEY=VALUE…] [-- <argument>…]`, starts the app
dialling out to it (ISO_DRIVER, ISO_DRIVER_NAME), its environment the
pairs and its command line the arguments, and ends it when stopped (SIGTERM,
or a file named `stop` among its requests, which `quit()` leaves, then
signals: drive/launchers/requests.sh).
- **Fresh by default.** A launch starts with none of the app's data from
before, as an app built headless does (`fresh: true`, the launcher's
ISOMER_FRESH=1): the iPhone's launcher clears the app's defaults (through
the simulator's cfprefsd), its files and its app groups' and the
simulator's keychain; Android's runs `pm clear`; the Mac's and Linux's
give the app a new home; the web's page has a new browser profile every
launch anyway. `fresh: false` (`iso launch --keep-data`) keeps it, for a
test of what lasts between launches. Windows' launcher keeps it, for now.
- **Pictures** (`picture`) are named by the platform's device as the app
stands (`ios/phone-landscape/…`, `android/phone-landscape/…` in a wide
window). An iPhone app that turns itself to landscape on an upright
simulator is drawn on its side in the simulator's frame; its driver says
how it stands (Info's `orientation`), and the iPhone's launcher turns the
picture to match. A launcher a platform, beside the
Storybook's tests (examples' headless/storybook_headless_application): the
web's (headless Chrome under browser_check `--hold`), the iPhone's (simctl
on a booted simulator), the Mac's, Android's (`am start` with the extras,
Expand Down
24 changes: 24 additions & 0 deletions drive/apple/AXClientAccessibility.swift
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,30 @@ extension AccessibilityDriver {
return await activate(button)
}

/// Moves SwiftUI's scene phase as the Mac moves it: background when the
/// app is hidden (NSApp.hide, as ⌘H), inactive when its windows are in
/// the Dock (miniaturized), active when shown and frontmost again. The
/// app's deactivation alone (another app frontmost) leaves it active.
func scene(_ phase: String) async -> String? {
let windows = NSApp.windows.filter { $0.isMiniaturized || $0.isVisible }
switch phase {
case "background":
NSApp.hide(nil)
case "inactive":
if NSApp.isHidden { NSApp.unhide(nil) }
for window in windows where !window.isMiniaturized && window.canBecomeMain { window.miniaturize(nil) }
case "active":
if NSApp.isHidden { NSApp.unhide(nil) }
for window in windows where window.isMiniaturized { window.deminiaturize(nil) }
NSApp.activate(ignoringOtherApps: true)
default:
return "no scene phase \(phase)"
}
// The window's animation into the Dock or out of it.
try? await Task.sleep(nanoseconds: 600_000_000)
return nil
}

/// A sheet's cancel, as Escape gives it; else its Cancel button.
func dismiss(_ screen: AccessibilityScreen) async -> Bool {
if let cancel = screen.elements.last(where: { $0.role == "button" && ["Cancel", "Done", "OK"].contains($0.label) }) {
Expand Down
61 changes: 48 additions & 13 deletions drive/apple/AccessibilityDriver.swift
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ struct AccessibilityScreen {
/// Drives the app running in this process through its accessibility tree.
/// Each platform's file gives it `screen()`, `transitioning()`, `reveal`, and
/// the actions (`activate`, `type`, `set`, `select`, `swipe`, `back`,
/// `dismiss`).
/// `dismiss`, `scene`).
@MainActor
final class AccessibilityDriver {
var info: Wire_Info
Expand All @@ -205,26 +205,42 @@ final class AccessibilityDriver {

/// Frames until the screen holds still (five readings alike, or the
/// timeout): animations, transitions and the app's tasks run meanwhile.
func settle(timeout: Double = 3, after action: Bool = false) async -> AccessibilityScreen {
/// After an action (`unchanged`: the screen before it), a screen still as
/// it was is given a while longer to change: what a tap starts may begin
/// a few turns of the run loop later (a sheet presented from a sheet,
/// which SwiftUI presents once the one below has settled; Quick Look's
/// controls), and the answer is the screen it leads to.
func settle(timeout: Double = 3, after unchanged: AccessibilityScreen? = nil) async -> AccessibilityScreen {
// What an action starts (a navigation's push, a sheet) begins on the
// next turns of the run loop: a moment before reading the screen.
if action { try? await Task.sleep(nanoseconds: 300_000_000) }
let deadline = Date().addingTimeInterval(timeout)
// A navigation's push or pop, a sheet's coming or going: the screen
// is the one it ends on.
while Date() < deadline, transitioning() {
try? await Task.sleep(nanoseconds: 50_000_000)
if unchanged != nil { try? await Task.sleep(nanoseconds: 150_000_000) }
let start = Date()
let deadline = start.addingTimeInterval(timeout)
let before = unchanged?.wire
func busy() async {
// A navigation's push or pop, a sheet's coming or going: the
// screen is the one it ends on.
while Date() < deadline, transitioning() {
try? await Task.sleep(nanoseconds: 50_000_000)
}
}
await busy()
var current = await screen()
var last = current.wire
var still = 0
while Date() < deadline {
try? await Task.sleep(nanoseconds: 50_000_000)
if transitioning() {
await busy()
still = 0
}
current = await screen()
let reading = current.wire
if reading == last {
still += 1
if still >= 5 { break }
// Still as it was before the action: up to a second and a
// half for what it started to show.
if still >= 5, before != reading || Date().timeIntervalSince(start) > 1.5 { break }
} else {
still = 0
last = reading
Expand All @@ -235,6 +251,7 @@ final class AccessibilityDriver {

func handle(_ command: Wire_Command) async -> Wire_Answer {
var error: String?
var waited: AccessibilityScreen?
let target = command.target
let before = await settle()
switch command.cmd {
Expand Down Expand Up @@ -289,6 +306,14 @@ final class AccessibilityDriver {
}
case "longpress":
error = await longPress(before, target)
case "scene":
// The app's scene phase, as leaving it and coming back moves it
// (each platform's file says how).
if ["active", "inactive", "background"].contains(command.value) {
error = await scene(command.value)
} else {
error = "scene: active, inactive or background, not \(command.value)"
}
case "back":
if !(await back(before)) { error = "nothing to go back from" }
case "dismiss":
Expand All @@ -301,13 +326,16 @@ final class AccessibilityDriver {
let text = command.text
let deadline = Date().addingTimeInterval(command.timeout > 0 ? command.timeout : 5)
var shown = false
while Date() < deadline {
let frame = await settle(timeout: 0.5)
var frame = before
while true {
if frame.texts.contains(where: { $0.text.contains(text) })
|| frame.elements.contains(where: { $0.label.contains(text) || ($0.value ?? "").contains(text) }) {
shown = true
waited = frame
break
}
if Date() >= deadline { break }
frame = await settle(timeout: 0.5)
}
if !shown { error = "\(text) didn't show" }
default:
Expand All @@ -316,8 +344,15 @@ final class AccessibilityDriver {
var answer = Wire_Answer()
answer.ok = error == nil
if let error { answer.error = error }
let after = await settle(after: !["ls", "info", "quit", "wait"].contains(command.cmd))
answer.screen = after.wire
// What only reads the screen answers with the reading it made; an
// action, with the screen it leads to.
if ["ls", "info", "inspect", "quit"].contains(command.cmd) {
answer.screen = before.wire
} else if command.cmd == "wait", let waited {
answer.screen = waited.wire
} else {
answer.screen = await settle(after: before).wire
}
if command.cmd == "info" { answer.info = info }
return answer
}
Expand Down
Loading
Loading