Repository navigation
fix(e2e): identify recorded controls by test id, labels only by shape - #637
Closed
okwasniewski wants to merge 9 commits into
Closed
okwasniewski wants to merge 9 commits into
okwasniewski wants to merge 9 commits into
Conversation
Contributor
🟢 e2e web: 90 passedAll 90 tests in 41 files
e2e 0.15.0-canary-20260928184528 · 1m 50s · web · run artifacts |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Devin Review found 3 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
Contributor
🟢 e2e web agent: 32 passed33 agent steps · 25 replayed from cache · 19 model calls · 71.5k tokens (91% cached) · $0.0034 All 32 tests in 5 files
e2e 0.15.0-canary-20260928184528 · 2m 51s · web · run artifacts |
Contributor
🟢 e2e ios: 50 passed, 1 skippedAll 51 tests in 7 files
e2e 0.15.0-canary-20260928184528 · 6m 51s · ios-simulator · run artifacts |
Contributor
🟢 e2e android: 54 passed, 1 skippedAll 55 tests in 7 files
e2e 0.15.0-canary-20260928184528 · 8m 1s · android-emulator · run artifacts |
Contributor
🟢 e2e android agent: 1 flaky, 19 passed24 agent steps · 110 model calls · 601.1k tokens (85% cached) · $0.0365
|
| Test | Agent | Time | |
|---|---|---|---|
| tests-agent/scenarios.e2e.ts · 1 flaky, 19 passed | 24 steps · 110 calls | 8m 51s | |
| 🟢 | act completes Login Form | 1 step · 3 calls | 32.0s |
| 🟢 | act completes Infinite Scroll List | 1 step · 3 calls | 26.6s |
| 🟢 | act completes Modal Flow | 1 step · 4 calls | 12.5s |
| 🟢 | act completes Bottom Tabs | 1 step · 3 calls | 14.6s |
| 🟢 | act completes Text Input Variations | 1 step · 5 calls | 12.9s |
| 🟢 | act completes Broken Accessibility | 2 steps · 6 calls | 27.7s |
| 🟢 | act completes Vision Only | 2 steps · 6 calls | 29.7s |
| act completes Gestures (1 failed attempt first) | 1 step · 8 calls | 29.0s | |
| 🟢 | act completes Async States | 1 step · 8 calls | 29.6s |
| 🟢 | act completes Debounced Search | 1 step · 4 calls | 9.3s |
| 🟢 | act completes Huge Virtualized List | 1 step · 3 calls | 1m 13s |
| 🟢 | act completes Flattened Registration Form | 2 steps · 8 calls | 42.7s |
| 🟢 | act completes Flattened Login | 2 steps · 6 calls | 29.4s |
| 🟢 | act completes Sticky Chrome Target | 1 step · 4 calls | 16.3s |
| 🟢 | act completes WebView Accessibility | 1 step · 5 calls | 18.6s |
| 🟢 | act completes Permission Prompt | 1 step · 3 calls | 9.1s |
| 🟢 | act completes Product Catalog | 1 step · 6 calls | 15.0s |
| 🟢 | act completes Error Recovery | 1 step · 7 calls | 14.3s |
| 🟢 | act completes Choice Controls | 1 step · 6 calls | 28.7s |
| 🟢 | act completes Sequential Onboarding | 1 step · 12 calls | 1m 1s |
e2e 0.15.0-canary-20260928184528 · 9m 12s · android-emulator · run artifacts
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 19 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Contributor
🟢 e2e ios agent: 2 flaky, 19 passed, 1 skipped25 agent steps · 11 replayed from cache · 62 model calls · 158.9k tokens (88% cached) · $0.0094
|
| Test | Agent | Time | |
|---|---|---|---|
| tests-agent/scenarios.e2e.ts · 2 flaky, 19 passed, 1 skipped | 25 steps · 62 calls | 11m 30s | |
| act completes Login Form (1 failed attempt first) | 1 step · 10 calls | 1m 26s | |
| 🟢 | act completes Infinite Scroll List | 1 step | 51.3s |
| 🟢 | act completes Modal Flow | 1 step | 13.1s |
| 🟢 | act completes Bottom Tabs | 1 step | 11.3s |
| 🟢 | act completes Text Input Variations | 1 step | 25.0s |
| 🟢 | act completes Broken Accessibility | 2 steps · 5 calls | 41.4s |
| 🟢 | act completes Vision Only | 2 steps · 1 call | 17.8s |
| act completes Gestures (3 failed attempts first) | 1 step · 8 calls | 34.9s | |
| 🟢 | act completes Async States | 1 step · 8 calls | 32.3s |
| 🟢 | act completes Debounced Search | 1 step · 3 calls | 20.5s |
| 🟢 | act completes Flattened Registration Form | 2 steps · 1 call | 39.3s |
| 🟢 | act completes Flattened Login | 2 steps · 10 calls | 1m 14s |
| 🟢 | act completes Sticky Chrome Target | 1 step | 23.2s |
| 🟢 | act completes WebView Accessibility | 1 step | 16.5s |
| 🟢 | act completes Permission Prompt | 1 step · 2 calls | 30.7s |
| 🟢 | act completes Photo Picker | 1 step · 2 calls | 31.6s |
| 🟢 | act completes Product Catalog | 1 step | 13.9s |
| 🟢 | act completes Error Recovery | 1 step | 14.2s |
| 🟢 | act completes Choice Controls | 1 step · 6 calls | 42.9s |
| 🟢 | act completes Stripe PaymentSheet | 1 step · 6 calls | 57.2s |
| 🟢 | act completes Apple Pay | 1 step | 13.7s |
| ⏭️ | act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there) |
e2e 0.15.0-canary-20260928184528 · 15m 18s · ios-simulator · run artifacts
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 11 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Member
Author
|
Bot findings, and what changed for each (commit 2bd1068):
Also in that commit, per review of the direction: relocation degrades through an explicit ladder, test id, authored element id (web now projects |
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 71 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
A recorded test id is the identity of a control; twins sharing one resolve by recorded position, never by label. Controls without an id, or whose id churned, match by label exactly and then by shape (digits, plurals, relative times folded). End anchors follow the same rule, are leaves ranked by durability, and skip relative times and social tallies. An end-mismatch hand-off names the anchors the screen did not show (step.cache.missingAnchors, agent notice). @e2e-dev/mobile: every settings request names the worker's device through one settings() helper, so a second booted device no longer fails clear-app-state with an ambiguous match.
…eport and the prefix
…findings Relocation degrades from the strongest recorded identity: test id, then an authored element id (web now projects the id attribute), then the exact label, then the label's shape, then place among unnamed twins. A position recorded among id twins holds on the label rungs. A bare count compares exactly, so an unchanged counter is not the recorded effect; counted nouns stem the same in singular and plural; an unreadable end screen reports every anchor missing; stable containers outrank volatile leaves as anchors.
…escriptor now carries The testbed's controls have authored ids, so anchors and targets record an elementId; the assertions match on the fields they mean. The mid-step divergence sabotage churns the ids as well as the label, since an id would otherwise still find the button.
…nservative/6 The policy bump keyed every committed recording out, so the web and iOS agentic jobs ran live and paid for it in model calls and flakiness. Both suites re-recorded locally (web: 32 passed, iOS simulator: 22 passed); the policy/5 entries are gone. Android has no committed recordings, as before.
…ey read, skip counter ids A stress probe through the CLI found three places where replay passed a step clean that main hands off, each acting on the wrong control or accepting an effect that did not happen: - `labelShape` folded every digit run, so `Delete item 3` found `Delete item 4`, `Page 2` found `Page 3`, `Order #1234` found another order. It now folds a count only where it governs a noun (`0 replies`, `3 followers`) and a relative time; a number that names something stays as it reads. `now` is a time only on its own, not in `Buy now`, a one-letter unit is lower case only (`3 M` is a size), and `-es` plurals stem with their singular (`match`/`matches`, `box`/`boxes`). - End anchors compared by the same shape, so a recorded `Count: 1` passed on a screen left at `Count: 0`, `Step 2 of 3` at `Step 1 of 3`, `Cart (2 items)` at `Cart (1 item)`. Anchors now fold relative times only: a count is usually the very effect the anchor stands for. - `stableElementId` let framework counters through (`mat-input-2`, `mui-3`, `headlessui-menu-button-1`), and the id rung outranks the label, so a field added above shifted the counter and the recorded `Email` was typed into `Phone` while `Email` sat beside it. Any trailing digit now disqualifies an id. A recorded position now holds on every rung, not only below an id rung: a rung that finds another number of twins than were recorded passes to the next, and a lone survivor of a recorded set is ambiguous with its candidates rather than found, since among label twins it is whichever one still reads as recorded, the wrong one after the step acted on it. Also the rebase onto main: the `settings.update` expectation main added carries the device selection, and the orphaned JSDoc above `settings()` in the mobile surface is back on `command()`. The invalid report fixture exercises `missingAnchors`.
okwasniewski
force-pushed
the
oskar/cache-relocate-label-drift
branch
from
September 30, 2026 11:31
c72cb4c to
cc6fe85
Compare
… the replay policy to conservative/7 The element id went into the descriptor straight from the attribute; it now passes through the same redaction and bounding as the other fields, and an id redaction touched is no id. The matching rules changed since the recordings under conservative/6 were made, so the policy version moves and those entries cold-start. The `describeAnchors` docstring names the order the code follows: stable leaves, stable containers, then whatever appeared.
…nservative/7 Web: 29 entries, replays 25 of 31 steps under --strict-cache, the rest designed gaps. iOS, on an iPhone 17 Pro simulator: 22 entries, a read-only replay pass then runs 13 steps from cache and hands 8 off, as the previous entries did in CI. No Android entries exist yet.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 54 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
…, as a triage dismissal shape
okwasniewski
commented
Sep 30, 2026
okwasniewski
left a comment
okwasniewski
left a comment
Member
Author
There was a problem hiding this comment.
On 6b3cddc: one issue with end anchors, inline on anchors.ts:181.
When every anchor a step produced is volatile, so describeAnchors keeps them anyway, anchorMatches folds relative times and ages with sameLabels(..., 'times'). The pre step state can then satisfy its own anchor. A step that changes Due today to Due tomorrow, or Snooze: 5 minutes to Snooze: 10 minutes, records the new label as its anchor. Both labels fold to the same shape (due <age>), so if the step silently does nothing on replay, the screen left at the old value still passes and nothing hands off to the model. Before this change the comparison was exact, so this case handed off. The smallest fix may be to compare exactly for anchors that were only kept because no stable anchor existed, or to refuse a match whose label was already on the start screen. Is this reachable in practice, or does something upstream stop a date or duration change from being the only evidence of a step?
reviewed 6b3cddc by Hermes on behalf of @okwasniewski
| const semantic = withoutIds(anchor); | ||
| const sameId = (anchor.testId !== undefined && candidate.testId === anchor.testId) || (anchor.elementId !== undefined && candidate.elementId === anchor.elementId); | ||
| const identified = sameId || !isAnonymous(semantic); | ||
| return identified && fieldsEqual(semantic, candidate, SEMANTIC_ID_FIELDS) && sameLabels(anchor, candidate, ['name', 'text'], 'times'); |
Member
Author
There was a problem hiding this comment.
sameLabels(..., 'times') folds Due today and Due tomorrow to the same due <age>. When only volatile anchors were kept, the pre step screen matches its own anchor and a silently failed step passes from cache. Should volatile only anchors compare exactly?
Member
Author
|
[claude-opus-5-5] responding on behalf of okwasniewski Superseded by #805, which keeps this PR's label fold (narrowed to leading counts that agree with their noun) and twins rule, and leaves out the element id rung. Closing as discussed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Like (0 likes), a row named after its contents and· now) that handed every second step back to the model. Relocation now degrades from the strongest identity it recorded: test id, then an authored element id (web now projects theidattribute), then the exact label, then the label's shape, then place among unnamed twins.mat-input-2,mui-3,headlessui-menu-button-1) or reads like a hash (:r3:,input-1739) is skipped. A counter names whichever control rendered in that place this time, and an id rung that trusted it typed into the field beside the right one.0 replies,3 followers,2 matches) andnow,2m,yesterday. A number that names something stays as it reads, soDelete item 3never findsDelete item 4,Page 2never findsPage 3,Order #1234never findsOrder #9876.nowis a time only on its own (Bob · now), not the adverb ofBuy now, and a one-letter unit is lower case only (3 Mis a size).Like). Any rung that finds another number of them than were recorded passes to the next rung, and a lone survivor of a recorded set is ambiguous, not found: among label twins it is whichever one still reads as recorded, which after the step acted on the recorded one is exactly the wrong one.Count: 1,Step 2 of 3, orCart (2 items)does not pass on a screen left atCount: 0,Step 1 of 3,Cart (1 item). Anchors are the leaves that appeared, ranked by durability, and skip relative times and social tallies; an unreadable end screen reports every anchor as missing; a stable container outranks volatile leaves.end-mismatchhand-off names the anchors the screen did not show, instep.cache.missingAnchorsand in the notice the agent gets. That is how the shell-chrome anchors were found.@e2e-dev/mobile: every settings request names the worker's device through onesettings()helper. Without it the daemon resolvedclear app stateagainst every booted device, and one engine failure failed every later test of a run with "2 devices match this request equally".REPLAY_POLICY_VERSIONgoes toconservative/7; recordings re-record once on their next passing run.flowchart LR R[recorded target] --> T{test id recorded?} T -- yes --> I[nodes with that test id] T -- no --> E I -- none --> E{element id recorded?} E -- yes --> J[nodes with that element id] E -- no --> L J -- none --> L[label equal] L -- none --> S[label shape equal] S -- none --> N[not found: hand off] I -- one --> F[found] J -- one --> F L -- one --> F S -- one --> F I -- several --> P{same count as recorded?} J -- several --> P L -- several --> P S -- several --> P P -- yes --> F P -- no --> A[ambiguous: hand off]packages/e2e/src/cache/relocate.ts(matchingTiers,labelShape) andpackages/e2e/src/cache/anchors.ts(anchorMatches) shareSEMANTIC_ID_FIELDSandsameLabels, so the two can no longer drift; anchors ask for thetimesfold, relocation forcounts.Verified
Ran it locally: yes
home.e2e.tsswipetool; a grammar action for it is a separate change.Incrementclick is a no-op on replay, status readsCount: 0Count: 1taken by shapeDelete item 3recorded, onlyDelete item 4on screenDelete item 4target-not-found, agent runs live and leaves item 4Emailrecorded with idmat-input-2; on replay that id isPhone,Emailismat-input-3Emailby labelPhoneEmailby label, no model callconservative/7recordings under--strict-cache: 32 passed, 25 replayed, noREPLAY_STALE; the hand-offs and misses are the designedgapand unique-param cases.conservative/7, then a read-only replay pass: 22 passed, 13 steps from cache, 8 handed off (tool gaps, native sheets), the same profile the previous entries had in CI.missingAnchorsin a real hand-off, before the leaf-only rule:["other \"Test harness control, not part of the app\"", "other \"Home\"", "scroll-view \"Following\"", "other \"New post\""]with the post text itself on screen.Checklist
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.!, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).docs/**/*.mdxpage for the behavior, andskills/e2e/if the skill describes it..d.tsreviewed,tests/types/sdk-types.tsupdated; wire change edits the schema and both fixtures.e2e/engine): changesets fore2e,@e2e-dev/web, and@e2e-dev/mobile.