Repository navigation
feat: write a failure page per failed test with each step's actions, polls, cache decision, and app log - #823
Closed
okwasniewski wants to merge 4 commits into
Closed
okwasniewski wants to merge 4 commits into
okwasniewski wants to merge 4 commits into
Conversation
Contributor
🟢 e2e web: 91 passedAll 91 tests in 41 files
e2e 0.17.0 · 1m 52s · web · run artifacts |
Contributor
🟢 e2e web agent: 33 passed34 agent steps · 26 replayed from cache · 20 model calls · 76.2k tokens (92% cached) · $0.0035 All 33 tests in 5 files
e2e 0.17.0 · 2m 54s · web · run artifacts |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 99 files
Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.
Re-trigger cubic
Contributor
🟢 e2e ios: 1 flaky, 51 passed, 1 skipped
|
| Test | Time | |
|---|---|---|
| tests/controls.e2e.ts · 1 flaky, 22 passed, 1 skipped | 3m 37s | |
| 🟢 | control inventory › a label query resolves a labeled text field | 10.0s |
| 🟢 | control inventory › a labeled field answers name, display value, value reads, fill, and clear | 18.3s |
| 🟢 | control inventory › a placeholder query and attribute reads find the field | 6.1s |
| 🟢 | control inventory › a secure field takes a fill and denies value reads | 9.5s |
| 🟢 | control inventory › press sends a character and Enter through the soft keyboard | 9.0s |
| 🟢 | control inventory › a field autofocuses on mount and focus() moves the focus | 11.8s |
| control inventory › toBeFocused follows the focus (2 failed attempts first) | 13.5s | |
| 🟢 | control inventory › a switch checks, unchecks, and reports its state | 14.5s |
| 🟢 | control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap | 9.8s |
| 🟢 | control inventory › checkbox and radio roles check, uncheck, and report their state | 16.9s |
| 🟢 | control inventory › a radio group reports the checked option | 15.3s |
| 🟢 | control inventory › buttons report disabled, enabled, and a state that arrives later | 9.0s |
| 🟢 | control inventory › a display: none element is absent rather than hidden, its visible twin attached | 8.6s |
| 🟢 | control inventory › a header answers its accessible name | 5.2s |
| 🟢 | control inventory › a tab row answers count, all, texts, filters, positions, and switching | 8.1s |
| 🟢 | control inventory › a list answers counts, list-form text, reads, and boxes in order | 8.5s |
| ⏭️ | control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…) | |
| 🟢 | control inventory › a long press lands as one | 9.6s |
| 🟢 | control inventory › pointer verbs land at the point asked for | 11.9s |
| 🟢 | control inventory › an ambiguous locator fails at once | 6.9s |
| 🟢 | control inventory › screenshot, restart, and clearState reopen the app on its home list | 13.0s |
| 🟢 | refusals › verbs a device never declares are refused before the engine | 3ms |
| 🟢 | refusals › keys the soft keyboard cannot send are refused by the engine | 636ms |
| 🟢 | refusals › a missing locator fails as not found once its timeout passes | 1.3s |
| 🟢 | tests/device-fixture.e2e.ts · 9 passed | 1m 20s |
| 🟢 | device fixture › setAppearance flips the color scheme the app reads | 11.8s |
| 🟢 | device fixture › setOrientation rotates the window the app measures | 7.3s |
| 🟢 | device fixture › the clipboard reads back what was written | 5.7s |
| 🟢 | device fixture › home leaves the app, openApp brings it back where it was | 7.6s |
| 🟢 | device fixture › closeApp then openApp relaunches on the home list | 8.0s |
| 🟢 | device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN | 5.3s |
| 🟢 | device fixture › device.back pops the scenario | 5.8s |
| 🟢 | device fixture › setLocation and clearLocation change what the app reads | 13.3s |
| 🟢 | device fixture › enrollBiometrics and setBiometrics answer a biometric prompt | 14.6s |
| 🟢 | tests/device.e2e.ts · 2 passed | 1m 7s |
| 🟢 | sequential onboarding echoes the exact values | 51.3s |
| 🟢 | onboarding rejects a malformed email before advancing | 15.4s |
| 🟢 | tests/flows-lists.e2e.ts · 7 passed | 3m 53s |
| 🟢 | infinite scroll reaches the target item | 30.3s |
| 🟢 | product catalog adds exactly two of the right variant | 11.1s |
| 🟢 | gestures: long-press, then swipe left | 9.9s |
| 🟢 | sticky chrome: accept below the fold, then continue | 11.9s |
| 🟢 | async states: load, pull to refresh, claim through the toast | 13.7s |
| 🟢 | huge virtualized list reaches row 512 | 2m 20s |
| 🟢 | web view coupon form applies the code shown on the page | 15.8s |
| 🟢 | tests/flows.e2e.ts · 7 passed | 1m 20s |
| 🟢 | text inputs unlock the submit button | 19.2s |
| 🟢 | filled text inputs report their values | 10.6s |
| 🟢 | modal flow confirms through the native alert | 10.7s |
| 🟢 | bottom tabs act inside the last tab | 8.4s |
| 🟢 | error recovery retries, then confirms the delete | 10.8s |
| 🟢 | choice controls place the exact order | 9.1s |
| 🟢 | debounced search selects the target once results arrive | 11.4s |
| 🟢 | tests/login-form.e2e.ts · 3 passed | 49.2s |
| 🟢 | login form › rejects a malformed email | 13.1s |
| 🟢 | login form › rejects an unknown account | 13.8s |
| 🟢 | login form › signs in and logs out | 22.3s |
| 🟢 | tests/smoke.e2e.ts · 1 passed | 11.8s |
| 🟢 | home lists the scenarios and opens one | 11.8s |
e2e 0.17.0 · 6m 37s · ios-simulator · run artifacts
Contributor
🟢 e2e android: 54 passed, 1 skippedAll 55 tests in 7 files
e2e 0.17.0 · 8m 30s · android-emulator · run artifacts |
Contributor
🟢 e2e android agent: 20 passed24 agent steps · 112 model calls · 603.2k tokens (85% cached) · $0.0361 All 20 tests in 1 file
e2e 0.17.0 · 9m 27s · android-emulator · run artifacts |
Contributor
🟢 e2e ios agent: 21 passed, 1 skipped25 agent steps · 139 model calls · 873.1k tokens (81% cached) · $0.0614 All 22 tests in 1 file
e2e 0.17.0 · 13m 35s · ios-simulator · run artifacts |
okwasniewski
force-pushed
the
oskar/failure-pages
branch
from
October 4, 2026 07:32
62a3104 to
b8e4852
Compare
okwasniewski
force-pushed
the
oskar/drop-playwright-trace
branch
from
October 4, 2026 07:50
5e10e1f to
0be324d
Compare
okwasniewski
force-pushed
the
oskar/failure-pages
branch
from
October 4, 2026 07:57
b8e4852 to
c71a9c8
Compare
Member
Author
|
[claude-opus-5-5] responding on behalf of okwasniewski Same status as #822: everything is green except |
okwasniewski
force-pushed
the
oskar/drop-playwright-trace
branch
from
October 4, 2026 09:07
0be324d to
bbbd984
Compare
okwasniewski
force-pushed
the
oskar/failure-pages
branch
from
October 4, 2026 09:15
c71a9c8 to
c801779
Compare
okwasniewski
force-pushed
the
oskar/drop-playwright-trace
branch
from
October 5, 2026 14:38
bbbd984 to
9373624
Compare
…polls, cache decision, and app log Every run writes .e2e/failures/<test>.md and the terminal names it. Steps record the node an action hit, assertion and waitFor readings, the cache's decision and write outcome, and what the app logged (new engine appLog hook, implemented by the web engine). REPLAY_STALE for a rekeyed recording names the key parts that changed.
…nd share one node describer A failure page that cannot be written is a stderr line, as junit.xml and summary.md are. cache.write moves to the report's step cache, since it is decided after act() returns. Kept entries are no longer rewritten only to gain keyedBy, so committed caches stay clean. Node and clip helpers reuse targetLabel and bound.
…ps in their video segment, and record point actions
… entry they now report
okwasniewski
force-pushed
the
oskar/failure-pages
branch
from
October 5, 2026 14:45
c801779 to
da9b383
Compare
6 tasks done
Member
Author
|
[claude-opus-5-5] responding on behalf of okwasniewski Superseded by #869: one PR, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #822. When a test fails, a coding agent (or a person) should read one file and know what happened. Today that file is opt-in (
--reporter markdown), deterministic steps record nothing (events: []), the cache's decision never leavesreport.json, and console errors and failed requests exist only inside the trace zip the parent PR removes.Every run now writes a page per failed or flaky test under
.e2e/failures/, and each step on it says what it did.EngineAttemptContext.appLog({ source, level, text }). The session adapter checks each entry, holds it until a step recorder takes it, and files it under the step that started last, redacted then clipped (200 per attempt). The web engine reports console errors and warnings, uncaught exceptions, failed requests, and 4xx/5xx responses (packages/web/src/app-log.ts).engineevent per action names the node from the resolve's own cache, no engine call (tap button "Add" (appeared after 1200ms, went stale 1x)); a point action names its point (tap at (120, 48)).expectandwaitForrecord apollevent with the readings in order: one value the whole time is a wrong expectation, changing values are a race.step.cache.entry(the file),detail(at action 2 of 3, tap button "Apply": ..., or the recorded end state that never showed), and the report-onlywrite(saved,kept,unconfirmed,evicted,no-change). New entries store their key's context, so a rekeyedREPLAY_STALEnames what changed:since the app's identity (app.identity, else its URL) or environment changed. Existing entries are not rewritten to gain it.step.phase), loop-guard and wind-down notes on the agent's turns, and the message of a failed agent action.run-finished(named<file>-<title slug>-<16 hex of the result id>.md) and names them in the event andFinishedRun.failurePages. Themarkdownreporter keeps onlysummary.md, linked to them. A page that cannot be written is a stderr line, likejunit.xml.Verified
Ran it locally: yes
e2e run:The page's steps:
and for the agent step that passed before a wrong expectation:
cache: replayed all 2 recorded actions, no model call; recording kept. Changing the app port and running with--strict-cache:REPLAY_STALE ... since the app's identity (app.identity, else its URL) or environment changed after it was recorded.CI=1 --strict-cachewith committed recordings and no key: 26 replayed. The 7 failures are steps that call the model by design, and the failure pages said so (no replay: a retry never replays,at action 2 of 3, tool type (run-time value)).