Skip to content

fix(agent): name the status and the body of a provider error the SDK could not parse - #922

Merged
okwasniewski merged 1 commit into
mainfrom
oskar/provider-error-body
Oct 8, 2026
Merged

fix(agent): name the status and the body of a provider error the SDK could not parse#922
okwasniewski merged 1 commit into
mainfrom
oskar/provider-error-body

Conversation

@okwasniewski

@okwasniewski okwasniewski commented Oct 6, 2026 •
edited by cubic-dev-ai Bot
Loading

Copy link
Copy Markdown
Member

Fixes ENG-875

  • When an error body doesn't match the provider's error schema, the AI SDK's JSON error handler (createJsonErrorResponseHandler) uses response.statusText as the message. statusText is empty over HTTP/2. The status and body stay on the APICallError, and data stays undefined. The ChatGPT backend's 400 {"detail":"Unsupported service_tier: fast"} hits this path.
  • We only read cause.message, so the step failed MODEL_PROVIDER_FAILED with an empty message and the --ai-trace step error was "".
  • New providerFailureMessage (packages/e2e/src/internal/errors.ts): for an AI_APICallError with status >= 400, a string body, and no parsed data, it returns HTTP <status>[ <statusText>]: <body>.
    • The body is redacted with the process secret redactor first, then cut at 1 KB with a … marker, so a secret that straddles the cut can't leak a prefix.
    • Errors the SDK parsed keep their message unchanged. Gateway errors are not AI_APICallError, so they are untouched too.
    • A spent RetryError is described by its last attempt, the same way judgments already unwrap it. Before this, a retryable HTTP/2 502 with an HTML body still read Failed after 6 attempts. Last error: with nothing after it.
  • Used in agent/model/sdk.ts (judgments, including the CONTEXT_OVERFLOW message), agent/tool-loop.ts (act), and internal/ai-trace.ts (onError).
  • docs/reference/errors.mdx has one added sentence on the MODEL_PROVIDER_FAILED row.

Verified

Ran it locally: yes

  • testbed, built CLI against a local openai-compatible fake server that answers every request with 400 {"detail":"Unsupported service_tier: fast"} and an empty status text. Command: e2e run --config <scratch config> --ai-trace, one agent.act step. The scratch config is not committed.
  • The new unit tests fail on main. The ai-trace one gets '', and the act and judgment ones get an empty provider message. They pass here and cover the act and judgment paths through a real @ai-sdk/openai-compatible model with a stubbed fetch, redaction of a secret straddling the cut, and a 502 retry chain.
main this branch
step error MODEL_PROVIDER_FAILED: the model provider failed: MODEL_PROVIDER_FAILED: the model provider failed: HTTP 400: {"detail":"Unsupported service_tier: fast"}
.e2e/ai-trace.json step error "" "HTTP 400: {\"detail\":\"Unsupported service_tier: fast\"}"

Not verified: the real ChatGPT backend (no stored login locally). The fake server reproduces the reported response shape.

Review in cubic Turn on auto-fix

@github-actions

github-actions Bot commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 e2e web agent: 33 passed

34 agent steps · 26 replayed from cache · 21 model calls · 83.6k tokens (90% cached) · $0.0042

All 33 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 7 passed 7 steps 7.6s
🟢 control inventory › check sets the checkbox and the radio 1 step 548ms
🟢 control inventory › check picks a radio the pick replaces with its summary 1 step 260ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 982ms
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 2.6s
🟢 control inventory › upload attaches the named project files 1 step 2.5s
🟢 control inventory › back returns from the details view to the inventory 1 step 326ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 368ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 464ms
🟢 act signs in with the benchmark account 1 step 464ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 10.5s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 5.2s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 1.5s
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 3.9s
🟢 tests-agent/scenarios.e2e.ts · 21 passed 21 steps · 13 calls 2m 36s
🟢 act completes shadow-dom-form 1 step · 3 calls 4.3s
🟢 act completes lying-labels 1 step · 4 calls 6.7s
🟢 act completes iframe-form 1 step · 3 calls 4.9s
🟢 act completes infinite-scroll 1 step 5.4s
🟢 act completes virtualized-table 1 step 1m 44s
🟢 act completes overlay-trap 1 step 862ms
🟢 act completes sticky-chrome 1 step 896ms
🟢 act completes async-states 1 step · 3 calls 7.7s
🟢 act completes debounced-search 1 step 2.3s
🟢 act completes hover-menu 1 step 504ms
🟢 act completes drag-and-drop 1 step 458ms
🟢 act completes kanban-board 1 step 667ms
🟢 act completes context-menu-trap 1 step 2.6s
🟢 act completes stale-dom 1 step 508ms
🟢 act completes otp-auto-advance 1 step 753ms
🟢 act completes date-picker 1 step 881ms
🟢 act completes rich-text-editor 1 step 8.2s
🟢 act completes playbook-cleanup 1 step 589ms
🟢 act completes promo-storefront 1 step 1.3s
🟢 act completes gift-card-purchase 1 step 2.4s
🟢 act filters the list and the URL carries the filter 1 step 239ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 327ms
🟢 act adds an address with a run-unique label 1 step 327ms

e2e 0.18.0 · 2m 56s · web · run artifacts

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Oct 6, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 8 files

Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic

@github-actions

github-actions Bot commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 e2e web: 91 passed

All 91 tests in 41 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 499ms
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 348ms
🟢 aria hidden flow › another plan shows the error and keeps the picker open 151ms
🟢 tests/async-states.e2e.ts · 2 passed 7.0s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.2s
🟢 async states › a wrong code is rejected after the toast has gone 4.8s
🟢 tests/canvas-only.e2e.ts · 2 passed 252ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 135ms
🟢 canvas only › a shape out of order paints the error and resets progress 117ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 440ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 255ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 185ms
🟢 tests/cart-totals.e2e.ts · 2 passed 358ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 208ms
🟢 cart totals › a quantity never drops below one 150ms
🟢 tests/checkout-review.e2e.ts · 1 passed 179ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 179ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 594ms
🟢 context menu trap › right-click opens nothing; the row menu renames the file 315ms
🟢 context menu trap › the decoy menu items refuse and keep the file 279ms
🟢 tests/control-inventory.e2e.ts · 8 passed 3.8s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 253ms
🟢 control inventory › check picks a radio the pick replaces with its summary 175ms
🟢 control inventory › a long press and a double click are told apart from a tap 1.0s
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 277ms
🟢 control inventory › setInputFiles attaches files named from the project root 175ms
🟢 control inventory › the details view is left through the browser history 201ms
🟢 control inventory › scrollIntoView brings the footnote into view 297ms
🟢 control inventory › every exercise done shows the success message 1.4s
🟢 tests/css-content-ui.e2e.ts · 2 passed 305ms
🟢 css content ui › the tile labelled Ship it advances and confirming ships 170ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 135ms
🟢 tests/date-picker.e2e.ts · 2 passed 611ms
🟢 date picker › navigates three months ahead and books the required date 364ms
🟢 date picker › confirming without the exact date shows the error 247ms
🟢 tests/debounced-search.e2e.ts · 2 passed 2.8s
🟢 debounced search › a decoy one character off is refused 1.4s
🟢 debounced search › picking the exact item adds it to the cart 1.4s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.1s
🟢 one submit opens one confirm, even when double-clicked 1.1s
🟢 tests/div-soup.e2e.ts · 2 passed 673ms
🟢 div soup › a wrong code is rejected 316ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 357ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 598ms
🟢 drag and drop › an out-of-order item resets the dropzone 297ms
🟢 drag and drop › dropping Banana then Cherry passes 301ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 242ms
🟢 filter deep link › filtering writes the query to the URL 132ms
🟢 filter deep link › opening a shared link restores the filter 110ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 522ms
🟢 gift card purchase › refuses to buy without an amount 155ms
🟢 gift card purchase › refuses a recipient without an at sign 180ms
🟢 gift card purchase › sends the gift card 187ms
🟢 tests/hover-menu.e2e.ts · 2 passed 429ms
🟢 hover menu › a decoy leaf is refused 197ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 232ms
🟢 tests/iframe-form.e2e.ts · 2 passed 494ms
🟢 iframe form › a wrong code is refused by the host page 244ms
🟢 iframe form › the code read from one frame applies through the other 250ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 442ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 99ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 161ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 182ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 4.8s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 4.8s
🟢 tests/kanban-board.e2e.ts · 3 passed 769ms
🟢 kanban board › pointer drags move both cards into the goal layout 422ms
🟢 kanban board › a swipe along a path drags a card as well 164ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 183ms
🟢 tests/login-form.e2e.ts · 4 passed 812ms
🟢 login form › rejects a malformed email 216ms
🟢 login form › rejects a short password 198ms
🟢 login form › rejects an unknown account 179ms
🟢 login form › signs in and logs out 219ms
🟢 tests/lying-labels.e2e.ts · 4 passed 665ms
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 147ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 99ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 188ms
🟢 lying labels › visible text drives the payment through 231ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 568ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 206ms
🟢 native dialogs › dismissing the confirm cancels the order 198ms
🟢 native dialogs › a wrong gift code is rejected 164ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.3s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.3s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 5.9s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 300ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 402ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 1.6s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 1.8s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 1.9s
🟢 tests/order-history.e2e.ts · 2 passed 1.4s
🟢 order history › overview and settings render their panels 179ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.3s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.4s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 518ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 128ms
🟢 otp auto-advance › a wrong code clears every box and starts over 760ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 701ms
🟢 continues only after the banner and the overlay are gone 701ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 568ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 273ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 295ms
🟢 tests/price-sorting.e2e.ts · 3 passed 463ms
🟢 price sorting › opens in the featured order with sale prices struck through 110ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 141ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 212ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.2s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.2s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.4s
🟢 buys a deal once the promo chrome is out of the way 1.4s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 542ms
🟢 rich text editor › bolding exactly the required word passes the check 273ms
🟢 rich text editor › bold that spills into the previous word fails the check 269ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 644ms
🟢 saved addresses › adds an address and clears the form 224ms
🟢 saved addresses › ignores an add with an empty field 218ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 202ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 617ms
🟢 shadow DOM form › rejects a wrong access code 305ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 312ms
🟢 tests/smoke.e2e.ts · 1 passed 394ms
🟢 home lists the scenarios and opens one 394ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.1s
🟢 taps the moving target three times across rebuilds of the list 2.1s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 206ms
🟢 accepts the terms once the button clears the sticky footer 206ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 1m 11s
🟢 scrolls the windowed table to the Golden Row and claims it 1m 11s
🟢 tests/wishlist.e2e.ts · 2 passed 1.5s
🟢 wishlist › saving toggles the button and the tab badge 241ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.3s

e2e 0.18.0 · 2m 2s · web · run artifacts

@github-actions

github-actions Bot commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 52 passed, 1 skipped

All 53 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 7m 4s
🟢 control inventory › a label query resolves a labeled text field 27.4s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 36.6s
🟢 control inventory › a placeholder query and attribute reads find the field 14.8s
🟢 control inventory › a secure field takes a fill and denies value reads 21.6s
🟢 control inventory › press sends a character and Enter through the soft keyboard 18.2s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 21.4s
🟢 control inventory › toBeFocused follows the focus 17.5s
🟢 control inventory › a switch checks, unchecks, and reports its state 19.4s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 18.8s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 24.6s
🟢 control inventory › a radio group reports the checked option 20.8s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 15.8s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 16.5s
🟢 control inventory › a header answers its accessible name 12.0s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 17.9s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 20.9s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 21.7s
🟢 control inventory › pointer verbs land at the point asked for 21.0s
🟢 control inventory › an ambiguous locator fails at once 15.3s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 39.5s
🟢 refusals › verbs a device never declares are refused before the engine 8ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 839ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.5s
🟢 tests/device-fixture.e2e.ts · 9 passed 3m 6s
🟢 device fixture › setAppearance flips the color scheme the app reads 32.0s
🟢 device fixture › setOrientation rotates the window the app measures 15.3s
🟢 device fixture › the clipboard reads back what was written 18.5s
🟢 device fixture › home leaves the app, openApp brings it back where it was 16.1s
🟢 device fixture › closeApp then openApp relaunches on the home list 22.7s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 13.7s
🟢 device fixture › device.back pops the scenario 14.1s
🟢 device fixture › setLocation and clearLocation change what the app reads 26.4s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 27.7s
🟢 tests/device.e2e.ts · 2 passed 1m 38s
🟢 sequential onboarding echoes the exact values 1m 12s
🟢 onboarding rejects a malformed email before advancing 26.1s
🟢 tests/flows-lists.e2e.ts · 7 passed 6m 37s
🟢 infinite scroll reaches the target item 47.8s
🟢 product catalog adds exactly two of the right variant 19.3s
🟢 gestures: long-press, then swipe left 16.1s
🟢 sticky chrome: accept below the fold, then continue 23.6s
🟢 async states: load, pull to refresh, claim through the toast 19.0s
🟢 huge virtualized list reaches row 512 3m 57s
🟢 web view coupon form applies the code shown on the page 33.4s
🟢 tests/flows.e2e.ts · 7 passed 2m 25s
🟢 text inputs unlock the submit button 29.4s
🟢 filled text inputs report their values 23.3s
🟢 modal flow confirms through the native alert 19.3s
🟢 bottom tabs act inside the last tab 15.6s
🟢 error recovery retries, then confirms the delete 17.1s
🟢 choice controls place the exact order 16.6s
🟢 debounced search selects the target once results arrive 24.2s
🟢 tests/login-form.e2e.ts · 3 passed 1m 51s
🟢 login form › rejects a malformed email 30.0s
🟢 login form › rejects an unknown account 28.0s
🟢 login form › signs in and logs out 53.1s
🟢 tests/smoke.e2e.ts · 1 passed 26.4s
🟢 home lists the scenarios and opens one 26.4s

e2e 0.18.0 · 11m 49s · ios-simulator · run artifacts

@github-actions

github-actions Bot commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 105 model calls · 545.9k tokens (85% cached) · $0.0333

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 105 calls 8m 18s
🟢 act completes Login Form 1 step · 3 calls 28.9s
🟢 act completes Infinite Scroll List 1 step · 3 calls 25.6s
🟢 act completes Modal Flow 1 step · 4 calls 11.8s
🟢 act completes Bottom Tabs 1 step · 3 calls 8.0s
🟢 act completes Text Input Variations 1 step · 3 calls 10.5s
🟢 act completes Broken Accessibility 2 steps · 6 calls 27.0s
🟢 act completes Vision Only 2 steps · 6 calls 28.8s
🟢 act completes Gestures 1 step · 11 calls 37.6s
🟢 act completes Async States 1 step · 6 calls 21.0s
🟢 act completes Debounced Search 1 step · 4 calls 10.0s
🟢 act completes Huge Virtualized List 1 step · 3 calls 1m 13s
🟢 act completes Flattened Registration Form 2 steps · 8 calls 39.1s
🟢 act completes Flattened Login 2 steps · 6 calls 28.4s
🟢 act completes Sticky Chrome Target 1 step · 4 calls 19.9s
🟢 act completes WebView Accessibility 1 step · 4 calls 15.3s
🟢 act completes Permission Prompt 1 step · 3 calls 8.1s
🟢 act completes Product Catalog 1 step · 5 calls 13.4s
🟢 act completes Error Recovery 1 step · 6 calls 12.4s
🟢 act completes Choice Controls 1 step · 5 calls 18.9s
🟢 act completes Sequential Onboarding 1 step · 12 calls 59.6s

e2e 0.18.0 · 8m 18s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 e2e android: 54 passed, 1 skipped

All 55 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 2m 55s
🟢 control inventory › a label query resolves a labeled text field 22.2s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 13.0s
🟢 control inventory › a placeholder query and attribute reads find the field 5.3s
🟢 control inventory › a secure field takes a fill and denies value reads 7.2s
🟢 control inventory › press sends a character and Enter through the soft keyboard 7.7s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.4s
🟢 control inventory › toBeFocused follows the focus 9.1s
🟢 control inventory › a switch checks, unchecks, and reports its state 10.0s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 7.3s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 11.5s
🟢 control inventory › a tab role answers getByRole 3.9s
🟢 control inventory › a radio group reports the checked option 9.1s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 7.8s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 6.3s
🟢 control inventory › a header answers its accessible name 3.8s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 5.0s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 4.9s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 6.9s
🟢 control inventory › a double tap lands inside the 300 ms window 7.7s
🟢 control inventory › an ambiguous locator fails at once 4.8s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 9.4s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 98ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.4s
🟢 tests/device-fixture.e2e.ts · 8 passed 1m 6s
🟢 device fixture › setOrientation rotates the window the app measures 5.0s
🟢 device fixture › home leaves the app, openApp brings it back where it was 5.7s
🟢 device fixture › foregroundApp sees the home screen after home() 3.9s
🟢 device fixture › closeApp then openApp relaunches on the home list 6.1s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.1s
🟢 device fixture › device.back pops the scenario 4.7s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 16.0s
🟢 device fixture › setLocation and clearLocation change what the app reads 20.5s
🟢 tests/device.e2e.ts · 4 passed 35.9s
🟢 permission prompt › allowing the system dialog enables the microphone 6.1s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 12.7s
🟢 sequential onboarding echoes the exact values 10.4s
🟢 onboarding rejects a malformed email before advancing 6.7s
🟢 tests/flows-lists.e2e.ts · 7 passed 2m 13s
🟢 infinite scroll reaches the target item 18.1s
🟢 product catalog adds exactly two of the right variant 9.7s
🟢 gestures: long-press, then swipe left 7.8s
🟢 gestures: double-tap completes the flow 9.6s
🟢 sticky chrome: accept below the fold, then continue 11.0s
🟢 async states: load, pull to refresh, claim through the toast 12.2s
🟢 huge virtualized list reaches row 512 1m 5s
🟢 tests/flows.e2e.ts · 7 passed 53.3s
🟢 text inputs unlock the submit button 6.9s
🟢 filled text inputs report their values 5.5s
🟢 modal flow confirms through the native alert 8.4s
🟢 bottom tabs act inside the last tab 6.5s
🟢 error recovery retries, then confirms the delete 8.7s
🟢 choice controls place the exact order 8.7s
🟢 debounced search selects the target once results arrive 8.6s
🟢 tests/login-form.e2e.ts · 3 passed 30.2s
🟢 login form › rejects a malformed email 8.9s
🟢 login form › rejects an unknown account 8.9s
🟢 login form › signs in and logs out 12.3s
🟢 tests/smoke.e2e.ts · 1 passed 5.7s
🟢 home lists the scenarios and opens one 5.7s

e2e 0.18.0 · 8m 20s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped

25 agent steps · 129 model calls · 41.3k tokens (88% cached) · $0.0023

⚠️ 1 flaky test passed on a retry

⚠️ act completes Gestures
tests-agent/scenarios.e2e.ts:257

ACTION_FAILED at step 4 of 4: agent.act "Complete this scenario as the screen instructs: long-press the first target, swipe the card left, then double-tap the last target", after 56.4s and 9 model calls

agent.act failed: Long-pressed the first target successfully. Tried swiping the card left several times, but it remained in place and the screen still shows Step 2 of 3, so I could not proceed to double-tap the last target.

  • Failed the same way on 2 attempts: ACTION_FAILED at step 4. The retry that passed is the exception.
  • Turn 6: screenshot({}) → [screenshot] The screen changed substantially since revision b7. Current screen (revision b9, path dev.e2e.benchmark / Benchmark Examples, 12 nodes):
  • Turn 7: swipe({"from":{"x":230,"y":460},"to":{"x":20,"y":460}}) → [swipe] Swiped from (230, 460) to (20, 460).
  • Turn 8: screenshot({}) → [screenshot] The screen changed substantially since revision b9. Current screen (revision b11, path dev.e2e.benchmark / Benchmark Examples, 12 nodes):
  • Screen: dev.e2e.benchmark / Benchmark Examples

Evidence: screenshot, log

All 22 tests in 1 file
Test Agent Time
⚠️ tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped 25 steps · 129 calls 21m 17s
🟢 act completes Login Form 1 step · 7 calls 2m 9s
🟢 act completes Infinite Scroll List 1 step · 3 calls 1m 22s
🟢 act completes Modal Flow 1 step · 4 calls 30.6s
🟢 act completes Bottom Tabs 1 step · 3 calls 24.8s
🟢 act completes Text Input Variations 1 step · 4 calls 59.5s
🟢 act completes Broken Accessibility 2 steps · 8 calls 1m 13s
🟢 act completes Vision Only 2 steps · 6 calls 57.9s
⚠️ act completes Gestures (2 failed attempts first) 1 step · 8 calls 52.1s
🟢 act completes Async States 1 step · 8 calls 1m 0s
🟢 act completes Debounced Search 1 step · 5 calls 35.4s
🟢 act completes Flattened Registration Form 2 steps · 14 calls 2m 29s
🟢 act completes Flattened Login 2 steps · 9 calls 1m 34s
🟢 act completes Sticky Chrome Target 1 step · 8 calls 56.8s
🟢 act completes WebView Accessibility 1 step · 6 calls 1m 16s
🟢 act completes Permission Prompt 1 step · 4 calls 33.4s
🟢 act completes Photo Picker 1 step · 5 calls 33.4s
🟢 act completes Product Catalog 1 step · 5 calls 29.0s
🟢 act completes Error Recovery 1 step · 4 calls 26.9s
🟢 act completes Choice Controls 1 step · 9 calls 1m 3s
🟢 act completes Stripe PaymentSheet 1 step · 5 calls 1m 12s
🟢 act completes Apple Pay 1 step · 4 calls 39.0s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

e2e 0.18.0 · 24m 26s · ios-simulator · run artifacts

@okwasniewski

okwasniewski commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

[claude-opus-5-5] responding on behalf of okwasniewski

web agent fails here with 30 REPLAY_STALE. That isn't caused by this PR: main fails the same way since the version bump in #825 (b2a28d4, run 37466109473). Run 37465480924 on the commit before it passed. The committed web-benchmark recordings need re-recording on main. Every other check is green. I left the Ready for Human Review label off until web agent passes.

@okwasniewski
okwasniewski force-pushed the oskar/provider-error-body branch from 72cceb5 to 109c5a0 Compare October 6, 2026 13:51
@okwasniewski
okwasniewski force-pushed the oskar/provider-error-body branch from 109c5a0 to 3ce9fc2 Compare October 7, 2026 09:53
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

🟢 e2e android: 54 passed, 1 skipped

All 55 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 3m 2s
🟢 control inventory › a label query resolves a labeled text field 21.8s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 13.3s
🟢 control inventory › a placeholder query and attribute reads find the field 5.2s
🟢 control inventory › a secure field takes a fill and denies value reads 7.5s
🟢 control inventory › press sends a character and Enter through the soft keyboard 8.1s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 11.3s
🟢 control inventory › toBeFocused follows the focus 10.1s
🟢 control inventory › a switch checks, unchecks, and reports its state 9.9s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 6.9s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 11.1s
🟢 control inventory › a tab role answers getByRole 3.7s
🟢 control inventory › a radio group reports the checked option 9.1s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 8.0s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 6.7s
🟢 control inventory › a header answers its accessible name 4.8s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 5.3s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 5.3s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 7.3s
🟢 control inventory › a double tap lands inside the 300 ms window 8.6s
🟢 control inventory › an ambiguous locator fails at once 5.2s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 11.0s
🟢 refusals › verbs a device never declares are refused before the engine 6ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 154ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 8 passed 1m 14s
🟢 device fixture › setOrientation rotates the window the app measures 5.5s
🟢 device fixture › home leaves the app, openApp brings it back where it was 6.4s
🟢 device fixture › foregroundApp sees the home screen after home() 4.5s
🟢 device fixture › closeApp then openApp relaunches on the home list 8.1s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 5.0s
🟢 device fixture › device.back pops the scenario 5.6s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 16.9s
🟢 device fixture › setLocation and clearLocation change what the app reads 21.4s
🟢 tests/device.e2e.ts · 4 passed 43.7s
🟢 permission prompt › allowing the system dialog enables the microphone 7.9s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 15.7s
🟢 sequential onboarding echoes the exact values 11.8s
🟢 onboarding rejects a malformed email before advancing 8.3s
🟢 tests/flows-lists.e2e.ts · 7 passed 2m 17s
🟢 infinite scroll reaches the target item 21.1s
🟢 product catalog adds exactly two of the right variant 11.4s
🟢 gestures: long-press, then swipe left 8.3s
🟢 gestures: double-tap completes the flow 9.8s
🟢 sticky chrome: accept below the fold, then continue 11.9s
🟢 async states: load, pull to refresh, claim through the toast 11.3s
🟢 huge virtualized list reaches row 512 1m 4s
🟢 tests/flows.e2e.ts · 7 passed 51.3s
🟢 text inputs unlock the submit button 6.0s
🟢 filled text inputs report their values 5.8s
🟢 modal flow confirms through the native alert 8.3s
🟢 bottom tabs act inside the last tab 6.9s
🟢 error recovery retries, then confirms the delete 8.3s
🟢 choice controls place the exact order 8.0s
🟢 debounced search selects the target once results arrive 8.0s
🟢 tests/login-form.e2e.ts · 3 passed 30.0s
🟢 login form › rejects a malformed email 8.8s
🟢 login form › rejects an unknown account 8.7s
🟢 login form › signs in and logs out 12.5s
🟢 tests/smoke.e2e.ts · 1 passed 5.7s
🟢 home lists the scenarios and opens one 5.7s

e2e 0.18.0 · 8m 44s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 113 model calls · 600.3k tokens (84% cached) · $0.0372

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 113 calls 9m 27s
🟢 act completes Login Form 1 step · 3 calls 34.4s
🟢 act completes Infinite Scroll List 1 step · 3 calls 27.4s
🟢 act completes Modal Flow 1 step · 4 calls 13.2s
🟢 act completes Bottom Tabs 1 step · 3 calls 8.4s
🟢 act completes Text Input Variations 1 step · 5 calls 14.5s
🟢 act completes Broken Accessibility 2 steps · 6 calls 29.1s
🟢 act completes Vision Only 2 steps · 6 calls 31.4s
🟢 act completes Gestures 1 step · 11 calls 40.9s
🟢 act completes Async States 1 step · 10 calls 39.0s
🟢 act completes Debounced Search 1 step · 4 calls 10.7s
🟢 act completes Huge Virtualized List 1 step · 3 calls 1m 17s
🟢 act completes Flattened Registration Form 2 steps · 8 calls 48.1s
🟢 act completes Flattened Login 2 steps · 6 calls 33.1s
🟢 act completes Sticky Chrome Target 1 step · 4 calls 15.8s
🟢 act completes WebView Accessibility 1 step · 5 calls 17.6s
🟢 act completes Permission Prompt 1 step · 3 calls 9.3s
🟢 act completes Product Catalog 1 step · 5 calls 14.1s
🟢 act completes Error Recovery 1 step · 6 calls 13.8s
🟢 act completes Choice Controls 1 step · 6 calls 24.1s
🟢 act completes Sequential Onboarding 1 step · 12 calls 1m 6s

e2e 0.18.0 · 9m 28s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 52 passed, 1 skipped

All 53 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 4m 8s
🟢 control inventory › a label query resolves a labeled text field 13.5s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 21.4s
🟢 control inventory › a placeholder query and attribute reads find the field 7.4s
🟢 control inventory › a secure field takes a fill and denies value reads 12.8s
🟢 control inventory › press sends a character and Enter through the soft keyboard 10.3s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 13.5s
🟢 control inventory › toBeFocused follows the focus 11.4s
🟢 control inventory › a switch checks, unchecks, and reports its state 13.6s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 11.6s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 16.1s
🟢 control inventory › a radio group reports the checked option 11.0s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 10.5s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 10.6s
🟢 control inventory › a header answers its accessible name 7.3s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 10.4s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 10.4s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 11.3s
🟢 control inventory › pointer verbs land at the point asked for 15.2s
🟢 control inventory › an ambiguous locator fails at once 8.0s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 19.1s
🟢 refusals › verbs a device never declares are refused before the engine 3ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 692ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.4s
🟢 tests/device-fixture.e2e.ts · 9 passed 1m 40s
🟢 device fixture › setAppearance flips the color scheme the app reads 15.3s
🟢 device fixture › setOrientation rotates the window the app measures 9.1s
🟢 device fixture › the clipboard reads back what was written 8.5s
🟢 device fixture › home leaves the app, openApp brings it back where it was 9.1s
🟢 device fixture › closeApp then openApp relaunches on the home list 10.7s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 11.2s
🟢 device fixture › device.back pops the scenario 7.6s
🟢 device fixture › setLocation and clearLocation change what the app reads 15.0s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 13.4s
🟢 tests/device.e2e.ts · 2 passed 1m 4s
🟢 sequential onboarding echoes the exact values 47.3s
🟢 onboarding rejects a malformed email before advancing 16.9s
🟢 tests/flows-lists.e2e.ts · 7 passed 4m 48s
🟢 infinite scroll reaches the target item 37.2s
🟢 product catalog adds exactly two of the right variant 12.8s
🟢 gestures: long-press, then swipe left 11.3s
🟢 sticky chrome: accept below the fold, then continue 14.7s
🟢 async states: load, pull to refresh, claim through the toast 14.7s
🟢 huge virtualized list reaches row 512 2m 58s
🟢 web view coupon form applies the code shown on the page 18.9s
🟢 tests/flows.e2e.ts · 7 passed 1m 35s
🟢 text inputs unlock the submit button 20.0s
🟢 filled text inputs report their values 14.5s
🟢 modal flow confirms through the native alert 12.6s
🟢 bottom tabs act inside the last tab 9.7s
🟢 error recovery retries, then confirms the delete 12.1s
🟢 choice controls place the exact order 12.1s
🟢 debounced search selects the target once results arrive 14.0s
🟢 tests/login-form.e2e.ts · 3 passed 1m 1s
🟢 login form › rejects a malformed email 16.4s
🟢 login form › rejects an unknown account 16.6s
🟢 login form › signs in and logs out 27.8s
🟢 tests/smoke.e2e.ts · 1 passed 17.4s
🟢 home lists the scenarios and opens one 17.4s

e2e 0.18.0 · 7m 32s · ios-simulator · run artifacts

@github-actions

github-actions Bot commented Oct 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped

25 agent steps · 140 model calls · 69.6k tokens (86% cached) · $0.0043

⚠️ 1 flaky test passed on a retry

⚠️ act completes Gestures
tests-agent/scenarios.e2e.ts:257

ASSERTION_FAILED at step 4 of 4: agent.act "Complete this scenario as the screen instructs: long-press the first target, swipe the card left, then double-tap the last target", after 50.5s and 9 model calls

agent.act failed: Long-pressed the first target and swiped the card left, advancing to Step 3 of 3. Double-tapped the last target twice, but the screen remained on “Double-tap the button below” with no completion indication.

  • Failed the same way on 2 attempts: ASSERTION_FAILED at step 4. The retry that passed is the exception.
  • Turn 6: screenshot({}) → [screenshot] The screen changed substantially since revision b7. Current screen (revision b9, path dev.e2e.benchmark / Benchmark Examples, 11 nodes):
  • Turn 7: double_tap({"target":"n3900"}) → [double_tap] Double-tapped #n3900.
  • Turn 8: double_tap({"target":"n3930"}) → [double_tap] Double-tapped #n3930.
  • Screen: dev.e2e.benchmark / Benchmark Examples

Evidence: screenshot, log

All 22 tests in 1 file
Test Agent Time
⚠️ tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped 25 steps · 140 calls 15m 9s
🟢 act completes Login Form 1 step · 8 calls 1m 2s
🟢 act completes Infinite Scroll List 1 step · 3 calls 1m 2s
🟢 act completes Modal Flow 1 step · 4 calls 20.1s
🟢 act completes Bottom Tabs 1 step · 3 calls 14.2s
🟢 act completes Text Input Variations 1 step · 5 calls 33.4s
🟢 act completes Broken Accessibility 2 steps · 7 calls 56.6s
🟢 act completes Vision Only 2 steps · 6 calls 40.8s
⚠️ act completes Gestures (2 failed attempts first) 1 step · 12 calls 1m 3s
🟢 act completes Async States 1 step · 6 calls 34.6s
🟢 act completes Debounced Search 1 step · 5 calls 27.5s
🟢 act completes Flattened Registration Form 2 steps · 16 calls 1m 57s
🟢 act completes Flattened Login 2 steps · 12 calls 1m 27s
🟢 act completes Sticky Chrome Target 1 step · 8 calls 45.0s
🟢 act completes WebView Accessibility 1 step · 5 calls 27.7s
🟢 act completes Permission Prompt 1 step · 3 calls 17.4s
🟢 act completes Photo Picker 1 step · 4 calls 17.3s
🟢 act completes Product Catalog 1 step · 5 calls 28.4s
🟢 act completes Error Recovery 1 step · 6 calls 18.1s
🟢 act completes Choice Controls 1 step · 8 calls 47.7s
🟢 act completes Stripe PaymentSheet 1 step · 9 calls 1m 1s
🟢 act completes Apple Pay 1 step · 5 calls 27.6s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

e2e 0.18.0 · 17m 45s · ios-simulator · run artifacts

@okwasniewski
okwasniewski merged commit c10dc00 into main Oct 8, 2026
30 checks passed
@okwasniewski
okwasniewski deleted the oskar/provider-error-body branch October 8, 2026 06:43
@github-actions github-actions Bot mentioned this pull request Oct 8, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cubic-dev-ai cubic-dev-ai[bot] cubic-dev-ai[bot] left review comments
@szymonrybczak szymonrybczak Awaiting requested review from szymonrybczak szymonrybczak is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@okwasniewski