Skip to content

fix: waitToSettleTimeoutMs was silently broken on iOS — 2x faster flows with tuned timeouts#3141

Open
qwertey6 wants to merge 4 commits intomobile-dev-inc:mainfrom
ReverentPeer:pr/4-settle-timeout
Open

fix: waitToSettleTimeoutMs was silently broken on iOS — 2x faster flows with tuned timeouts#3141
qwertey6 wants to merge 4 commits intomobile-dev-inc:mainfrom
ReverentPeer:pr/4-settle-timeout

Conversation

@qwertey6
Copy link
Copy Markdown

@qwertey6 qwertey6 commented Apr 4, 2026

Proposed changes

waitToSettleTimeoutMs was silently broken on iOS. The per-command config existed in the YAML spec and was documented, but on iOS it only controlled the fallback tier — the primary 3-second screenshot hash check always ran regardless. Users who tried to optimize their flows with waitToSettleTimeoutMs on iOS got no benefit.

Impact

  • 2x faster flow execution with tuned timeouts: Wikipedia e2e flow goes from 53s → 25s
  • waitToSettleTimeoutMs now actually works on iOS: setting waitToSettleTimeoutMs: 500 on a swipe genuinely caps settle time at 500ms
  • Partially addresses [Feature Request] how to reduce delay between actions globally? #1528 — "how to reduce delay between actions globally"
  • Zero behavior change for existing flows: default (no waitToSettleTimeoutMs set) uses unchanged 3000ms budget

Root cause

iOS waitForAppToSettle has two tiers:

  1. Tier 1 (server-side): XCTest runner takes two screenshots, SHA-256 hashes both, compares. Polls until identical or timeout. Hardcoded 3000ms.
  2. Tier 2 (client-side fallback): If Tier 1 times out, compare view hierarchy. Uses waitToSettleTimeoutMs as timeout.

The waitToSettleTimeoutMs parameter only controlled Tier 2. Tier 1 always burned its full 3000ms budget regardless. So waitToSettleTimeoutMs: 100 meant: wait 3000ms for screenshots, THEN wait 100ms for hierarchy comparison. Total: 3100ms, not 100ms.

Fix

Use waitToSettleTimeoutMs as the total settle budget:

  • Tier 1 runs with totalTimeout (either waitToSettleTimeoutMs or default 3000ms)
  • If Tier 1 doesn't settle in time, remaining budget goes to Tier 2
  • If no budget remains, return immediately
# Before: this still waited 3000ms+ on iOS
- swipe:
    direction: LEFT
    waitToSettleTimeoutMs: 500

# After: genuinely capped at 500ms total
- swipe:
    direction: LEFT
    waitToSettleTimeoutMs: 500

Measured impact (Wikipedia e2e flow)

Command Default (3000ms) Tuned (500ms) Savings
Swipe + settle 3172ms 2670ms 16%
WaitForAnimationToEnd 3203ms 1535ms 52%
Repeat block (3 swipes) 14028ms 12386ms 12%
Full flow 53s 25s 53%

Changes

  • IOSDriver.kt — waitForAppToSettle() uses timeoutMs as total budget for both tiers, with remaining time passed to the hierarchy fallback

Testing

Manually verified with Wikipedia e2e flow on iOS simulator. All existing tests pass. Default behavior (no waitToSettleTimeoutMs) unchanged.

Depends on: #3140 (version safety) → #3139 (performance) → #3138 (parallel execution)

Issues fixed

Partially addresses #1528

qwertey6 and others added 4 commits April 4, 2026 15:37
@qwertey6 @claude
feat: enable parallel iOS simulator execution
dd74302 
iOS simulators share the host's localhost, causing port collisions when
multiple Maestro processes target different sims simultaneously. Session
tracking was per-platform, so two processes on different devices would
interfere with each other's sessions.

Changes:
- Per-device session tracking: SessionStore keys are now
  "{platform}_{deviceId}_{sessionId}" instead of "{platform}_{sessionId}"
- Add --driver-host-port CLI flag for explicit XCTest server port
- Auto-select available ports with isPortAvailable() check
- Refactor SessionStore from singleton to injectable class (DI)
- Add shouldCloseSession(platform, deviceId) for per-device shutdown
  instead of global activeSessions().isEmpty()
- Add cross-process file locking to KeyValueStore (~/.maestro/sessions)
- Append PID to debug log directory to prevent parallel race
- Enable useJUnitPlatform() in maestro-cli (was missing)
- Add SessionStoreTest with 8 tests covering isolation and lifecycle

Verified: 3 iOS simulators + Android emulator running simultaneously,
all passing. Both --driver-host-port (explicit) and auto-port-selection
work correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@qwertey6 @claude
perf: skip unnecessary driver reinstall, cache build products
41de7ee 
- Default --reinstall-driver to false: reuse a healthy running driver
  instead of killing and reinstalling on every run (~40s saved on iOS)
- XCTestDriverClient checks isChannelAlive() before reinstalling —
  if the user explicitly passes --reinstall-driver, honor it
- Cache extracted iOS build products per-device in
  ~/.maestro/build-products/<deviceId>/ with SHA-256 hash validation:
  skips extraction when source matches cache, re-extracts on upgrade
- Reduce XCTest status check HTTP read timeout from 100s to 3s
- Remove Thread.sleep(1000) heartbeat delay hack (no longer needed
  with per-device session tracking)

Single device: ~52s → ~10-12s. Three devices parallel: ~54s → ~18s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@qwertey6 @claude
fix: version checks for drivers and app binary cache staleness
86a5353 
- iOS XCTest runner: add isVersionMatch() to XCTestInstaller interface.
  LocalXCTestInstaller compares SHA-256 hash of build products against
  a .running-hash marker written at startup. restartXCTestRunner now
  checks both isChannelAlive() AND isVersionMatch() — stale runners
  from a previous Maestro version are replaced automatically.

- Android driver: add isDriverVersionCurrent() that hashes the bundled
  maestro-app.apk and maestro-server.apk, compares against stored hash
  in ~/.maestro/android-driver-hash. On mismatch, APKs are reinstalled
  even when reinstallDriver=false.

- App binary cache (clearAppState): getCachedAppBinary now compares
  Info.plist of cached vs installed app. Stale cache from app updates
  is detected and refreshed before reinstall. Per-device cache dirs
  (~/.maestro/app-cache/<deviceId>/) prevent parallel races.

- Add XCTestDriverClientTest (4 tests) and LocalSimulatorUtilsTest
  (3 tests) covering version mismatch, reuse, and cache behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@qwertey6 @claude
fix: make waitToSettleTimeoutMs control total iOS settle budget
e61fb0e 
On iOS, waitForAppToSettle has two tiers: a server-side screenshot
hash check (Tier 1, hardcoded 3000ms) and a client-side hierarchy
comparison fallback (Tier 2). The per-command waitToSettleTimeoutMs
config only controlled Tier 2, so even waitToSettleTimeoutMs: 100
would still burn up to 3 seconds in Tier 1.

Fix: use waitToSettleTimeoutMs as the total settle budget. Tier 1
runs with this timeout, and any remaining time goes to Tier 2:

  - swipe with waitToSettleTimeoutMs: 500 → capped at 500ms total
  - default (no config) → unchanged 3000ms behavior

Wikipedia e2e flow with tuned timeouts: 25s vs 53s baseline.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This was referenced Apr 10, 2026
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@qwertey6