[okwasniewski]

What and why

• agent-device throws a retriable COMMAND_FAILED with details.androidSnapshotHelperFailureReason (empty-helper-output, content-poor-app-window, system-window-only) when an Android snapshot has no usable app content, e.g. while the app is still loading.
• translateError mapped it to non-retryable ENGINE_FAILURE, so resolveOnce stopped polling and toBeVisible failed immediately instead of waiting out its timeout.
• locate() now treats an unreadable capture as zero matches and keeps polling. If the capture overruns the time budget, the original error is rethrown.
• Tradeoff: while the screen is unreadable, not.toBeVisible(), count() and isVisible() see 0 matches. Alternative is a retryable NODE_STALE; open to switching.
• Seen as flakes in react-navigation Android CI: test: migrate e2e tests to e2e react-navigation/react-navigation#13299

Verified

Ran it locally: yes - unit tests and pnpm check only, not on a real emulator.

• 4 new unit tests in packages/mobile/tests/unit/engine.test.ts.

Checklist

• Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
• Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).
• Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
• Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
• Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile.

[cubic-dev-ai]

All reported issues were addressed across 5 files

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[github-actions]

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 35 model calls · 175.5k tokens (87% cached) · $0.0100

All 20 tests in 1 file

	Test	Agent	Time
🟢	tests-agent/scenarios.e2e.ts · 20 passed	24 steps · 35 calls	8m 7s
🟢	act completes Login Form	1 step	33.8s
🟢	act completes Infinite Scroll List	1 step	25.7s
🟢	act completes Modal Flow	1 step	12.3s
🟢	act completes Bottom Tabs	1 step	10.3s
🟢	act completes Text Input Variations	1 step	8.8s
🟢	act completes Broken Accessibility	2 steps · 5 calls	49.3s
🟢	act completes Vision Only	2 steps · 1 call	20.4s
🟢	act completes Gestures	1 step · 9 calls	1m 5s
🟢	act completes Async States	1 step · 8 calls	47.0s
🟢	act completes Debounced Search	1 step	7.1s
🟢	act completes Huge Virtualized List	1 step	1m 9s
🟢	act completes Flattened Registration Form	2 steps · 1 call	12.0s
🟢	act completes Flattened Login	2 steps · 6 calls	47.0s
🟢	act completes Sticky Chrome Target	1 step	9.5s
🟢	act completes WebView Accessibility	1 step · 5 calls	19.8s
🟢	act completes Permission Prompt	1 step	7.4s
🟢	act completes Product Catalog	1 step	10.3s
🟢	act completes Error Recovery	1 step	9.5s
🟢	act completes Choice Controls	1 step	12.0s
🟢	act completes Sequential Onboarding	1 step	11.1s

_{e2e 0.21.1 · 8m 8s · android-emulator · run artifacts}

[github-actions]

🟢 e2e ios: 53 passed, 1 skipped

All 54 tests in 7 files

	Test	Time
⏭️	tests/controls.e2e.ts · 23 passed, 1 skipped	3m 58s
🟢	control inventory › a label query resolves a labeled text field	11.8s
🟢	control inventory › a labeled field answers name, display value, value reads, fill, and clear	33.4s
🟢	control inventory › a placeholder query and attribute reads find the field	5.9s
🟢	control inventory › a secure field takes a fill and denies value reads	11.8s
🟢	control inventory › press sends a character and Enter through the soft keyboard	9.6s
🟢	control inventory › a field autofocuses on mount and focus() moves the focus	12.5s
🟢	control inventory › toBeFocused follows the focus	10.4s
🟢	control inventory › a switch checks, unchecks, and reports its state	13.1s
🟢	control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap	10.3s
🟢	control inventory › checkbox and radio roles check, uncheck, and report their state	16.5s
🟢	control inventory › a radio group reports the checked option	11.6s
🟢	control inventory › buttons report disabled, enabled, and a state that arrives later	9.8s
🟢	control inventory › a display: none element is absent rather than hidden, its visible twin attached	9.8s
🟢	control inventory › a header answers its accessible name	5.6s
🟢	control inventory › a tab row answers count, all, texts, filters, positions, and switching	8.7s
🟢	control inventory › a list answers counts, list-form text, reads, and boxes in order	9.2s
⏭️	control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢	control inventory › a long press lands as one	9.7s
🟢	control inventory › pointer verbs land at the point asked for	12.7s
🟢	control inventory › an ambiguous locator fails at once	7.5s
🟢	control inventory › screenshot, restart, and clearState reopen the app on its home list	16.2s
🟢	refusals › verbs a device never declares are refused before the engine	3ms
🟢	refusals › keys the soft keyboard cannot send are refused by the engine	497ms
🟢	refusals › a missing locator fails as not found once its timeout passes	1.3s
🟢	tests/device-fixture.e2e.ts · 10 passed	1m 51s
🟢	device fixture › setAppearance flips the color scheme the app reads	14.2s
🟢	device fixture › setOrientation rotates the window the app measures	8.9s
🟢	device fixture › the clipboard reads back what was written	9.7s
🟢	device fixture › home leaves the app, openApp brings it back where it was	9.4s
🟢	device fixture › closeApp then openApp relaunches on the home list	11.4s
🟢	device fixture › closeApp leaves the next commands on the same device	14.7s
🟢	device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN	7.6s
🟢	device fixture › device.back pops the scenario	6.8s
🟢	device fixture › setLocation and clearLocation change what the app reads	13.5s
🟢	device fixture › enrollBiometrics and setBiometrics answer a biometric prompt	14.4s
🟢	tests/device.e2e.ts · 2 passed	59.1s
🟢	sequential onboarding echoes the exact values	44.2s
🟢	onboarding rejects a malformed email before advancing	14.9s
🟢	tests/flows-lists.e2e.ts · 7 passed	4m 2s
🟢	infinite scroll reaches the target item	29.9s
🟢	product catalog adds exactly two of the right variant	12.3s
🟢	gestures: long-press, then swipe left	10.8s
🟢	sticky chrome: accept below the fold, then continue	13.2s
🟢	async states: load, pull to refresh, claim through the toast	14.5s
🟢	huge virtualized list reaches row 512	2m 26s
🟢	web view coupon form applies the code shown on the page	14.9s
🟢	tests/flows.e2e.ts · 7 passed	1m 25s
🟢	text inputs unlock the submit button	16.8s
🟢	filled text inputs report their values	12.2s
🟢	modal flow confirms through the native alert	13.5s
🟢	bottom tabs act inside the last tab	9.1s
🟢	error recovery retries, then confirms the delete	10.6s
🟢	choice controls place the exact order	10.2s
🟢	debounced search selects the target once results arrive	12.4s
🟢	tests/login-form.e2e.ts · 3 passed	52.2s
🟢	login form › rejects a malformed email	14.1s
🟢	login form › rejects an unknown account	14.3s
🟢	login form › signs in and logs out	23.7s
🟢	tests/smoke.e2e.ts · 1 passed	12.9s
🟢	home lists the scenarios and opens one	12.9s

_{e2e 0.21.1 · 6m 52s · ios-simulator · run artifacts}

[github-actions]

🟢 e2e android: 55 passed, 1 skipped

All 56 tests in 7 files

	Test	Time
⏭️	tests/controls.e2e.ts · 24 passed, 1 skipped	3m 38s
🟢	control inventory › a label query resolves a labeled text field	30.3s
🟢	control inventory › a labeled field answers name, display value, value reads, fill, and clear	15.9s
🟢	control inventory › a placeholder query and attribute reads find the field	8.8s
🟢	control inventory › a secure field takes a fill and denies value reads	9.5s
🟢	control inventory › press sends a character and Enter through the soft keyboard	11.4s
🟢	control inventory › a field autofocuses on mount and focus() moves the focus	13.6s
🟢	control inventory › toBeFocused follows the focus	11.1s
🟢	control inventory › a switch checks, unchecks, and reports its state	11.3s
🟢	control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap	7.8s
🟢	control inventory › checkbox and radio roles check, uncheck, and report their state	12.8s
🟢	control inventory › a tab role answers getByRole	5.4s
🟢	control inventory › a radio group reports the checked option	10.1s
🟢	control inventory › buttons report disabled, enabled, and a state that arrives later	8.7s
🟢	control inventory › a display: none element is absent rather than hidden, its visible twin attached	7.2s
🟢	control inventory › a header answers its accessible name	4.4s
🟢	control inventory › a tab row answers count, all, texts, filters, positions, and switching	5.6s
🟢	control inventory › a list answers counts, list-form text, reads, and boxes in order	5.9s
⏭️	control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢	control inventory › a long press lands as one	8.1s
🟢	control inventory › a double tap lands inside the 300 ms window	9.7s
🟢	control inventory › an ambiguous locator fails at once	5.9s
🟢	control inventory › screenshot, restart, and clearState reopen the app on its home list	12.9s
🟢	refusals › verbs a device never declares are refused before the engine	5ms
🟢	refusals › keys the soft keyboard cannot send are refused by the engine	473ms
🟢	refusals › a missing locator fails as not found once its timeout passes	1.1s
🟢	tests/device-fixture.e2e.ts · 9 passed	1m 27s
🟢	device fixture › setOrientation rotates the window the app measures	6.2s
🟢	device fixture › home leaves the app, openApp brings it back where it was	7.3s
🟢	device fixture › foregroundApp sees the home screen after home()	5.0s
🟢	device fixture › closeApp then openApp relaunches on the home list	9.1s
🟢	device fixture › closeApp leaves the next commands on the same device	9.6s
🟢	device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN	5.6s
🟢	device fixture › device.back pops the scenario	6.1s
🟢	device fixture › setNetwork and setAirplaneMode change what the app sees	17.9s
🟢	device fixture › setLocation and clearLocation change what the app reads	20.5s
🟢	tests/device.e2e.ts · 4 passed	46.8s
🟢	permission prompt › allowing the system dialog enables the microphone	8.6s
🟢	permission prompt › a denied prompt recovers once the permission is granted outside	16.6s
🟢	sequential onboarding echoes the exact values	13.2s
🟢	onboarding rejects a malformed email before advancing	8.5s
🟢	tests/flows-lists.e2e.ts · 7 passed	2m 28s
🟢	infinite scroll reaches the target item	20.7s
🟢	product catalog adds exactly two of the right variant	11.6s
🟢	gestures: long-press, then swipe left	9.4s
🟢	gestures: double-tap completes the flow	9.9s
🟢	sticky chrome: accept below the fold, then continue	13.4s
🟢	async states: load, pull to refresh, claim through the toast	13.5s
🟢	huge virtualized list reaches row 512	1m 10s
🟢	tests/flows.e2e.ts · 7 passed	57.7s
🟢	text inputs unlock the submit button	7.6s
🟢	filled text inputs report their values	6.7s
🟢	modal flow confirms through the native alert	9.4s
🟢	bottom tabs act inside the last tab	7.1s
🟢	error recovery retries, then confirms the delete	8.7s
🟢	choice controls place the exact order	9.4s
🟢	debounced search selects the target once results arrive	8.7s
🟢	tests/login-form.e2e.ts · 3 passed	31.2s
🟢	login form › rejects a malformed email	9.1s
🟢	login form › rejects an unknown account	9.4s
🟢	login form › signs in and logs out	12.7s
🟢	tests/smoke.e2e.ts · 1 passed	6.2s
🟢	home lists the scenarios and opens one	6.2s

_{e2e 0.21.1 · 9m 56s · android-emulator · run artifacts}

[github-actions]

🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped

25 agent steps · 9 replayed from cache · 68 model calls · 486.4k tokens (83% cached) · $0.0329

⚠️ 1 flaky test passed on a retry

⚠️ act completes Gestures
tests-agent/scenarios.e2e.ts:257

ASSERTION_FAILED at step 4 of 4: agent.act "Complete this scenario as the screen instructs: long-press the first target, swipe the card left, then double-tap the last target", after 1m 18s and 10 model calls

agent.act failed: The cached long-press was already complete, and swiping the card left advanced the screen to Step 3 of 3. I double-tapped “Double-tap me” twice, but the screen remained on Step 3 with no visible completion or other outcome.

• Turn 7: screenshot({}) → [screenshot] The screen changed substantially since revision b10. Current screen (revision b12, path dev.e2e.benchmark / Benchmark Examples, 11 nodes):
• Turn 8: double_tap({"target":"n3511"}) → [double_tap] Double-tapped #n3511.
• Turn 9: double_tap({"target":"n3541"}) → [double_tap] Double-tapped #n3541.
• Screen: dev.e2e.benchmark / Benchmark Examples

Evidence: screenshot, log

All 22 tests in 1 file

	Test	Agent	Time
⚠️	tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped	25 steps · 68 calls	16m 11s
🟢	act completes Login Form	1 step · 7 calls	1m 24s
🟢	act completes Infinite Scroll List	1 step	1m 2s
🟢	act completes Modal Flow	1 step	17.8s
🟢	act completes Bottom Tabs	1 step	11.9s
🟢	act completes Text Input Variations	1 step	21.1s
🟢	act completes Broken Accessibility	2 steps · 5 calls	54.7s
🟢	act completes Vision Only	2 steps · 6 calls	1m 0s
⚠️	act completes Gestures (1 failed attempt first)	1 step · 8 calls	1m 11s
🟢	act completes Async States	1 step · 6 calls	39.2s
🟢	act completes Debounced Search	1 step · 3 calls	20.6s
🟢	act completes Flattened Registration Form	2 steps · 1 call	38.9s
🟢	act completes Flattened Login	2 steps · 13 calls	2m 33s
🟢	act completes Sticky Chrome Target	1 step	27.3s
🟢	act completes WebView Accessibility	1 step	21.2s
🟢	act completes Permission Prompt	1 step · 2 calls	45.3s
🟢	act completes Photo Picker	1 step · 2 calls	40.9s
🟢	act completes Product Catalog	1 step	17.5s
🟢	act completes Error Recovery	1 step · 1 call	28.4s
🟢	act completes Choice Controls	1 step · 7 calls	1m 17s
🟢	act completes Stripe PaymentSheet	1 step · 7 calls	1m 1s
🟢	act completes Apple Pay	1 step	18.3s
⏭️	act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

_{e2e 0.21.1 · 17m 38s · ios-simulator · run artifacts}