Repository navigation
fix(cache): a row of a windowed list is not keyed by its list's first rendered row; mobile benchmark re-recorded, Android strict - #998
Merged
Conversation
Contributor
🟢 e2e web: 91 passedAll 91 tests in 41 files
e2e 0.18.0 · 2m 3s · web · run artifacts |
Contributor
🟢 e2e web agent: 33 passed34 agent steps · 26 replayed from cache · 20 model calls · 76.4k tokens (91% cached) · $0.0036 All 33 tests in 5 files
e2e 0.18.0 · 3m 1s · web · run artifacts |
Contributor
🟢 e2e android agent: 1 flaky, 19 passed24 agent steps · 15 replayed from cache · 34 model calls
|
| Test | Agent | Time | |
|---|---|---|---|
| tests-agent/scenarios.e2e.ts · 1 flaky, 19 passed | 24 steps · 34 calls | 5m 31s | |
| act completes Login Form (1 failed attempt first) | 1 step | 9.1s | |
| 🟢 | act completes Infinite Scroll List | 1 step | 22.4s |
| 🟢 | act completes Modal Flow | 1 step | 8.0s |
| 🟢 | act completes Bottom Tabs | 1 step | 6.5s |
| 🟢 | act completes Text Input Variations | 1 step | 6.1s |
| 🟢 | act completes Broken Accessibility | 2 steps · 5 calls | 31.4s |
| 🟢 | act completes Vision Only | 2 steps · 1 call | 10.8s |
| 🟢 | act completes Gestures | 1 step · 5 calls | 22.6s |
| 🟢 | act completes Async States | 1 step · 11 calls | 36.2s |
| 🟢 | act completes Debounced Search | 1 step | 6.6s |
| 🟢 | act completes Huge Virtualized List | 1 step | 1m 3s |
| 🟢 | act completes Flattened Registration Form | 2 steps · 1 call | 10.4s |
| 🟢 | act completes Flattened Login | 2 steps · 6 calls | 28.3s |
| 🟢 | act completes Sticky Chrome Target | 1 step | 8.4s |
| 🟢 | act completes WebView Accessibility | 1 step · 5 calls | 16.4s |
| 🟢 | act completes Permission Prompt | 1 step | 6.1s |
| 🟢 | act completes Product Catalog | 1 step | 9.4s |
| 🟢 | act completes Error Recovery | 1 step | 9.0s |
| 🟢 | act completes Choice Controls | 1 step | 11.0s |
| 🟢 | act completes Sequential Onboarding | 1 step | 10.0s |
e2e 0.18.0 · 6m 9s · android-emulator · run artifacts
Contributor
🟢 e2e ios: 52 passed, 1 skippedAll 53 tests in 7 files
e2e 0.18.0 · 5m 48s · ios-simulator · run artifacts |
Contributor
🟢 e2e android: 54 passed, 1 skippedAll 55 tests in 7 files
e2e 0.18.0 · 8m 16s · android-emulator · run artifacts |
Contributor
🟢 e2e ios agent: 2 flaky, 19 passed, 1 skipped25 agent steps · 10 replayed from cache · 73 model calls · 700.4k tokens (77% cached) · $0.0576
|
| Test | Agent | Time | |
|---|---|---|---|
| tests-agent/scenarios.e2e.ts · 2 flaky, 19 passed, 1 skipped | 25 steps · 73 calls | 11m 26s | |
| act completes Login Form (1 failed attempt first) | 1 step · 9 calls | 59.0s | |
| 🟢 | act completes Infinite Scroll List | 1 step | 48.0s |
| 🟢 | act completes Modal Flow | 1 step | 13.1s |
| 🟢 | act completes Bottom Tabs | 1 step | 11.2s |
| 🟢 | act completes Text Input Variations | 1 step | 20.3s |
| 🟢 | act completes Broken Accessibility | 2 steps · 6 calls | 37.0s |
| 🟢 | act completes Vision Only | 2 steps · 6 calls | 37.1s |
| 🟢 | act completes Gestures | 1 step · 10 calls | 59.0s |
| 🟢 | act completes Async States | 1 step · 9 calls | 44.2s |
| 🟢 | act completes Debounced Search | 1 step · 3 calls | 24.0s |
| 🟢 | act completes Flattened Registration Form | 2 steps · 1 call | 39.7s |
| act completes Flattened Login (1 failed attempt first) | 2 steps · 11 calls | 1m 6s | |
| 🟢 | act completes Sticky Chrome Target | 1 step | 20.3s |
| 🟢 | act completes WebView Accessibility | 1 step | 16.6s |
| 🟢 | act completes Permission Prompt | 1 step · 2 calls | 30.7s |
| 🟢 | act completes Photo Picker | 1 step · 2 calls | 30.3s |
| 🟢 | act completes Product Catalog | 1 step | 12.8s |
| 🟢 | act completes Error Recovery | 1 step | 12.0s |
| 🟢 | act completes Choice Controls | 1 step · 5 calls | 46.7s |
| 🟢 | act completes Stripe PaymentSheet | 1 step · 9 calls | 44.0s |
| 🟢 | act completes Apple Pay | 1 step | 14.0s |
| ⏭️ | act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there) |
e2e 0.18.0 · 15m 1s · ios-simulator · run artifacts
… drop orphan web entries The 39 mobile recordings dated from Sep 23-24, before the Oct 3 replay policy bump (#732) and the key change in #929, so every CI run missed all of them and called the model. Both platforms are re-recorded on a Mac (iPhone 17 Pro on iOS 26.5, Pixel 7 API 36). The Android agentic job runs with --strict-cache like the web one; iOS stays lenient until the three recordings named in the README replay on a simulator. The web benchmark cache kept 30 entries under the pre-#929 key next to their re-keyed twins; the orphans go.
okwasniewski
force-pushed
the
oskar/cache-orphans-mobile-strict
branch
from
October 9, 2026 07:56
9ae76bc to
e0ef197
Compare
… rendered row The container key of a row inside an unnamed list was the list's first leaf text, which on Android is whichever row the window rendered first: the Huge Virtualized List tap replayed as 'Row 0512 in Row 0500' and handed off whenever the window started at 0501, once locally and once in CI. A container whose first text sits in a twin of the row the node came up through is a list, and keys nothing by that text; its own label still keys it. The four Android entries that carried such a key are re-recorded.
okwasniewski
changed the title
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
20 missed/21 missedon the last three merge queue runs) and called the model for every step..github/actions/expo-appmakes them. 20 Android entries, 21 iOS.--strict-cachelike the web one: a recording that no longer replays fails its step asREPLAY_STALEinstead of spending model calls on every run.containerKey,packages/e2e/src/agent/actions.ts). On Android a windowed list renders whichever rows are near the viewport, so that text is a scroll position: the Huge Virtualized List tap was recorded astap group "Row 0512" in "Row 0500"and handed off whenever the window started at 0501, once locally and once in CI. A container whose first text sits in a twin of the row the node came up through is a list and keys nothing by that text; its own label still keys it. The four Android entries that carried such a key (Infinite Scroll List, Huge Virtualized List, Sticky Chrome Target, Product Catalog) are re-recorded. iOS entries carry no placement. Web entries are unaffected: strict web replays the same 26.flowchart LR A[mobile agentic job] --> B{platform} B -- android --> C[e2e run --strict-cache] C --> D{entry replays?} D -- yes --> E[replayed, no model call] D -- no --> F[REPLAY_STALE, job fails] B -- ios --> G[e2e run] G --> H{entry replays?} H -- yes --> E H -- no --> I[hand off to the model]Verified
Ran it locally: yes
e2e run --target android-emulator --config e2e.agent.config.ts --strict-cacheon Pixel_7_API_36 with the committed entries:20 passed,15 replayed · 2 handed off · 3 missed. The hand-offs and misses are swipes and run-time values, which the grammar does not record.--strict-cacheon two iPhone 17 Pro (26.5) simulators:22 passed, 18 replayed; the three named above hand off. With--strict-cachethose three fail asREPLAY_STALEon every run, which is why iOS is not strict yet.e2e run --config e2e.agent.config.ts --strict-cachewith the pruned cache:33 passed,26 replayed · 2 handed off · 4 missed, the same summary main's web agent job has printed since Oct 5.CI on this branch
REPLAY_STALE, action-failed, typing on the loaded emulator), and the--last-failedpass replayed it under strict and passed. An earlier run flaked on Gestures the same way: the replayed long-press completed, then the live agent's swipes were not accepted.Login Form, a Mac reads the back button'sBenchmark Examples), Photo Picker (the recorded tap names the photo by its seed time), Permission Prompt (end stateMicrophone enableddoes not show after replay). All six are engine or cache follow-ups; none is a stale recording.The first strict Android run on this branch failed on
Huge Virtualized Listboth passes; that was the placement key above, not scroll-until. With the fix, Android strict replays locally:20 passed,15 replayed · 2 handed off · 3 missed.