chore: version packages

[github-actions]

This PR was opened by the Changesets release GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated.

Releases

e2e@0.17.0

Minor Changes

• #779 6639bfa Thanks @fecolinhares! - copilot() reaches the models Copilot serves only over its Responses API, such as gpt-6-luna, gpt-5.3-codex, and grok-4.5. On the first call it reads the plan's model listing and picks chat completions or the Responses API from the model's supported_endpoints; a model served over chat completions, or one the listing does not place, stays on chat completions. A listing that cannot be read sends that call over chat completions without remembering it, and a revoked login fails on the listing with the sign-in command. Responses models run through @ai-sdk/openai, which e2e init now adds for Copilot; without it the call names the package to install. Responses turns carry the same initiator and vision headers as chat turns. e2e models github-copilot marks what copilot() cannot call with no chat or responses in place of no chat completions, and a disabled model as not enabled only.

• #794 c66176b Thanks @neriousy! - Sign in with OpenCode Console to use OpenCode Zen and OpenCode Go models for agent steps. e2e login opencode-console runs Console's device flow: approve the code, pick the workspace, and the login is stored and refreshed like the other subscriptions. opencodeConsole('<id>') from e2e/oauth/opencode-console calls a Zen model for a bare id and a Go model for a go/ id (go/deepseek-v4.1-flash); on its first call it reads the workspace config to pick the model's API (OpenAI chat completions, OpenAI Responses, Anthropic Messages, or Google) and loads only that AI SDK package, so @ai-sdk/anthropic and @ai-sdk/google join the optional peers. Once the workspace config is read, a model it does not serve, or a Go model without the subscription, fails with MISCONFIGURED naming the fix. Each model sends one x-opencode-session, so a worker's calls stay on one upstream cache, and prompts cache the way e2e caches them for OpenAI and Anthropic directly: models served over Anthropic Messages get a breakpoint on the system prompt and, in agent steps, on the newest message, and Responses models get the system prompt's cache key. Reports name the model opencode/<id> or opencode.go/<id>, and chat models of both plans read provider options under opencode. e2e models opencode-console lists the enabled models tagged Zen or Go, e2e init offers OpenCode Console among the subscriptions, and OPENCODE_API_KEY, the variable OpenCode reads, takes a Console service account key in place of the stored login.

• #741 bfa58c7 Thanks @Marve10s! - Keep soft assertion failures in the report when a test skips itself, and show
  them in the CLI under Skipped After Failure. A failed attempt followed by a
  skip on retry also appears there with its original error.

  Add failOnSkippedFailure: true to fail the run with exit code 1 for these
  tests. The default is false, preserving the existing exit code. Tests retain
  their skipped status and reason, teardown still runs, and a skip does not
  trigger another retry.

• #713 7ab80bc Thanks @okwasniewski! - Breaking: an interrupted test is no longer counted as failed. report.json gains run.summary.interrupted, and run.summary.failed counts only failed and timed-out tests. run.summary.skipped now counts only selected tests, so passed + failed + interrupted + flaky + skipped equals selected; the tests a filter left out are discovered - selected. The list reporter prints 3 interrupted in its own column, summary.md shows them with ⏹️ and gives them no failure block or page, and junit.xml writes each one as a <skipped> whose message starts interrupted:. A test that failed and whose retry an interrupt cut short stays failed, and its failure is the one reported. A --repeat-each run the interrupt stopped is listed as interrupted, not as a flake. The telemetry event gains tests_interrupted. In @e2e-dev/github, a test that was interrupted and then passed on a --last-failed rerun shows as passed, not flaky.

Patch Changes

• #711 a3da00d Thanks @okwasniewski! - device.installApp() no longer pins the build's file path as the app when agent-device reports no bundle id or package for it, which on Android made app.open() fail with "Android runtime hints require an installed package name". It fails at the install with ENGINE_FAILURE naming app.bundleId or installApp's app option; the mobile docs' Troubleshooting covers the Android cause (agent-device 0.21.18 reads an aapt2-built APK's package only through the SDK's aapt, which it does not find in the macOS default SDK location, or when the install added the package). app.open() on a device now keeps the engine's own reason it cannot launch, such as a build not installed yet with device.installApp(), instead of a generic "pin one with app.bundleId or app.appPath".

• #732 3bf295e Thanks @okwasniewski! - The replay cache no longer passes a step whose effect did not happen. Entries are keyed by the agent and a hash of its redacted context, so one persona never replays another's steps and a changed agentContext records again. A route now includes the origin and the query (ids, tokens, and timestamps aside), and the end route must match exactly. A replay checks what the step made appear, with checked or selected state, and what it made disappear, needs at least one of those changes to happen during the replay, and stops on an alert the recording never saw. A count the step made appear is part of the effect, and an unnamed control with twins needs a named row or group to replay. Every existing entry is re-recorded on the first run after the upgrade.

• #713 7ab80bc Thanks @okwasniewski! - A forced interrupt (a second Ctrl-C, or a CI cancel that sends SIGINT and then SIGTERM) still writes junit.xml and summary.md. The runner used to abandon them along with every other reporter, which left the previous run's files beside the new report.json. Custom reporters are still abandoned on force.

• #799 96ff11b Thanks @okwasniewski! - A recorded step that removes a control while something else on screen keeps its text, such as a radio labelled "Express" replaced by a status reading "Express", now replays. Its recording used to fail its own end check on every replay (end-mismatch, or REPLAY_STALE under --strict-cache). Re-record such a step once to pick up the fix.

• #726 f7c0756 Thanks @okwasniewski! - locator.waitFor adds Playwright's attached and detached states and refuses any other state, or an option key besides state and timeout, with INVALID_ARGUMENT. Before, every state but visible ran as hidden, so waitFor({ state: 'attached' }) and a misspelled state passed at once on an absent node. toBeHidden, toBeVisible({ visible: false }), toBeAttached({ attached: false }), and waitFor detached and hidden read a locator under a frame missing from the document as zero matches instead of timing out waiting for the frame. toHaveProperty reads a primitive on the path through its wrapper, as Jest does, so toHaveProperty('label.length', 3) passes. expect(browser).toHaveURL takes ignoreCase, and urlMatches in e2e/engine takes it as an optional fourth argument.

• #784 cc54c6b Thanks @okwasniewski! - Every model call now identifies e2e, whichever provider serves it: the User-Agent starts with e2e/<version> (<platform>; <arch>) ahead of the AI SDK's own, and HTTP-Referer: https://tester.army/e2e with X-Title: e2e attribute the traffic on the Vercel AI Gateway and OpenRouter. These replace an HTTP-Referer or X-Title set on the provider instance. Before, only subscription logins sent the e2e user agent.

• #722 f0f9c8d Thanks @okwasniewski! - Explicit navigation now uses an allowlist instead of a denylist: app.open, browser.goto, and the agent's navigate verb admit http:, https:, and the exact about:blank, and every other scheme (chrome:, blob:, about:srcdoc, ...) is POLICY_DENIED. A wrapped scheme such as view-source:file:///... no longer loads a local file; it is POLICY_DENIED like file: itself. device.openLink and device.openApp also refuse view-source:, blob:, and filesystem: links. browser.setCookies refuses an about:blank cookie URL with POLICY_DENIED.

• #729 4ce01b2 Thanks @okwasniewski! - A registered secret no longer leaks through the observed screen: a test id, an iframe name in a frame path, a selector, or an attribute holding one is masked in the model's text, an executor's tree, and cache entries, and a secret with a line break, tab, CRLF, no-break space, or repeated spaces that an engine collapsed and then cut at the name or text limit no longer leaves its leading part in observations or end anchors.

• #798 02ad48a Thanks @okwasniewski! - Breaking: @e2e-dev/web depends on playwright-core pinned to an exact version instead of peering on playwright. Projects no longer install Playwright themselves, and the engine always runs the Playwright it was tested against. Remove playwright from your dependencies unless your app uses it for its own tests:

  npm uninstall playwright

  Install browsers in CI with the package's new command, which runs the engine's Playwright: npx @e2e-dev/web install chromium --with-deps (pnpm exec e2e-web install chromium --with-deps under pnpm) replaces npx playwright install chromium --with-deps. Open traces with npx playwright-core@1.63.0 show-trace <file> (pnpm dlx in a pnpm project). An app that also depends on @playwright/test keeps its own copy; the two share a browser cache only when their versions match. e2e init no longer adds playwright to a new project.

• #725 9b93575 Thanks @okwasniewski! - An expect.poll that the test body, an afterEach hook, or a fixture teardown returns without awaiting is now cancelled and fails that phase with STEP_NOT_AWAITED at the line of the call; in a beforeAll or afterAll hook it fails the hook with HOOK_FAILED. Before, the test passed and the poll's timeout failed whichever test ran next, or nothing at all.

  The negation window of a negated expect(locator) or expect(browser) matcher now starts when the first read that saw the negation was issued, not when it returned, so a slow read counts toward it. With a timeout under a second, a negation that held for the whole budget now passes at the deadline; before, a slow first read made a true negation time out.

• #777 79931d0 Thanks @DeryFerd! - The replay cache abstracts short prefixed record ids as minted segments. A route segment of letters, a dash, and a digit tail of two or more (PROJ-016, INV-2041) now reduces to :id, so a step recorded on one record replays on the next instead of missing with wrong-context and running live every time. A single trailing digit (page-2) stays a route word.

• #724 3a9667c Thanks @okwasniewski! - --last-failed no longer goes green on tests it never ran. It reruns tests --max-failures skipped, and a test or failed beforeAll/afterAll another filter leaves out stays owed in the report's new run.carried until a rerun runs it. A rerun also keeps the artifacts of the run it reruns and writes its own under artifacts/rerun-<n>/ in the output directory (.e2e by default), so the folded pull request comment keeps its evidence and stays red while anything is owed.

• #721 94ddbfe Thanks @okwasniewski! - Two credentials or two secrets whose names map to the same override variable (api-key and api_key both read E2E_SECRET_API_KEY) now fail the config load with INVALID_CONFIG instead of both silently taking one value.

• #800 b5dc0f8 Thanks @okwasniewski! - --strict-cache now fails a step with REPLAY_STALE when the cache directory holds its recording under another key, for example after an e2e or engine upgrade or a change to the agent's context. Before, such a step ran live and spent model calls with no error. The message names the old entry file. A step with no recording still runs live.

@e2e-dev/web@0.12.0

Minor Changes

• #798 02ad48a Thanks @okwasniewski! - Breaking: @e2e-dev/web depends on playwright-core pinned to an exact version instead of peering on playwright. Projects no longer install Playwright themselves, and the engine always runs the Playwright it was tested against. Remove playwright from your dependencies unless your app uses it for its own tests:

  npm uninstall playwright

  Install browsers in CI with the package's new command, which runs the engine's Playwright: npx @e2e-dev/web install chromium --with-deps (pnpm exec e2e-web install chromium --with-deps under pnpm) replaces npx playwright install chromium --with-deps. Open traces with npx playwright-core@1.63.0 show-trace <file> (pnpm dlx in a pnpm project). An app that also depends on @playwright/test keeps its own copy; the two share a browser cache only when their versions match. e2e init no longer adds playwright to a new project.

• #795 3b026fa Thanks @okwasniewski! - Init scripts, like Playwright's addInitScript: web({ initScripts }) for every test, browser.addInitScript(script, arg?) for one. A script is source, a { path }, or a function.

• #786 f88233f Thanks @okwasniewski! - web({ locale, timezoneId }) sets the language and time zone every attempt's context runs in, as Playwright's use.locale and use.timezoneId do: navigator.language, Intl, the Accept-Language header, and local Date time. A tag Intl rejects is INVALID_CONFIG at config load, and so is an accept-language entry in headers beside locale, and either option on a persistent remote context (connect.reconnectEndpoint, or a provider with scope: 'attempt').

Patch Changes

• #788 baa32c6 Thanks @okwasniewski! - check() and uncheck(), and the agent's check tool, pass when the click removes the control or navigates away from it, such as a radio the app replaces with its selected view. The click landed, so the action is recorded and replays. What replaced the control is not read, so a test that needs it asserts on it. They used to fail: an agent step reported LOCATOR_NOT_FOUND and left the action out of its recording, so the next run handed off with end-mismatch, and a locator check() timed out. Unchecking a checked radio and a click that leaves the state unchanged now fail with NOT_ACTIONABLE instead of ENGINE_FAILURE.

• #726 f7c0756 Thanks @okwasniewski! - locator.waitFor adds Playwright's attached and detached states and refuses any other state, or an option key besides state and timeout, with INVALID_ARGUMENT. Before, every state but visible ran as hidden, so waitFor({ state: 'attached' }) and a misspelled state passed at once on an absent node. toBeHidden, toBeVisible({ visible: false }), toBeAttached({ attached: false }), and waitFor detached and hidden read a locator under a frame missing from the document as zero matches instead of timing out waiting for the frame. toHaveProperty reads a primitive on the path through its wrapper, as Jest does, so toHaveProperty('label.length', 3) passes. expect(browser).toHaveURL takes ignoreCase, and urlMatches in e2e/engine takes it as an optional fourth argument.

• #722 f0f9c8d Thanks @okwasniewski! - Explicit navigation now uses an allowlist instead of a denylist: app.open, browser.goto, and the agent's navigate verb admit http:, https:, and the exact about:blank, and every other scheme (chrome:, blob:, about:srcdoc, ...) is POLICY_DENIED. A wrapped scheme such as view-source:file:///... no longer loads a local file; it is POLICY_DENIED like file: itself. device.openLink and device.openApp also refuse view-source:, blob:, and filesystem: links. browser.setCookies refuses an about:blank cookie URL with POLICY_DENIED.

• #785 9844c30 Thanks @okwasniewski! - A string role name with a quote (getByRole('button', { name: "Yes, I'm sure" })) no longer fails with InvalidSelectorError once the locator is chained (.first(), nth, a filter, or the read itself). The same goes for a u or v flag RegExp with a quote or >> in getByRole, getByText, getByLabel, getByPlaceholder, and getByTestId.

• #723 cdc3a53 Thanks @okwasniewski! - Route decisions behave like Playwright's. route.continue() now sends the request to the network instead of handing it to an earlier route, and still carries the configured headers to the app's site; the new route.fallback() keeps the old chaining. continue takes url, method, headers, and postData, with url checked by the same rule as goto, and fulfill takes path (relative to the project root) and contentType. An option a decision does not take, such as abort(errorCode) or a misspelled key, is INVALID_ARGUMENT instead of being ignored. browser.waitForResponse resolves once a matching response's headers arrive, so a slow body no longer fails the match with a timeout; text() and json() wait for the body up to the action timeout. browser.setCookies with a relative url sets the cookie on the base URL instead of failing with ENGINE_FAILURE.

• #728 7f9ee10 Thanks @okwasniewski! - Web visibility and reads follow what renders, as Playwright reads them: an aria-hidden node that paints is visible, so toBeHidden() on a decorative spinner waits until it leaves; content under content-visibility: hidden is hidden; a child that sets visibility: visible under a hidden parent is visible and in the agent's observation; text under a display: contents element with visibility: hidden is hidden, where Playwright calls it visible, since it paints nothing. boundingBox() is null for a node with no layout box. inputValue() and toHaveValue read a checkbox's or radio's value (on by default) whatever its checked state. An accessible name reads an embedded control's value: <button>Flash the screen <input value="3"> times</button> is "Flash the screen 3 times".

@e2e-dev/github@0.3.3

Patch Changes

• #713 7ab80bc Thanks @okwasniewski! - Breaking: an interrupted test is no longer counted as failed. report.json gains run.summary.interrupted, and run.summary.failed counts only failed and timed-out tests. run.summary.skipped now counts only selected tests, so passed + failed + interrupted + flaky + skipped equals selected; the tests a filter left out are discovered - selected. The list reporter prints 3 interrupted in its own column, summary.md shows them with ⏹️ and gives them no failure block or page, and junit.xml writes each one as a <skipped> whose message starts interrupted:. A test that failed and whose retry an interrupt cut short stays failed, and its failure is the one reported. A --repeat-each run the interrupt stopped is listed as interrupted, not as a flake. The telemetry event gains tests_interrupted. In @e2e-dev/github, a test that was interrupted and then passed on a --last-failed rerun shows as passed, not flaky.

• #724 3a9667c Thanks @okwasniewski! - --last-failed no longer goes green on tests it never ran. It reruns tests --max-failures skipped, and a test or failed beforeAll/afterAll another filter leaves out stays owed in the report's new run.carried until a rerun runs it. A rerun also keeps the artifacts of the run it reruns and writes its own under artifacts/rerun-<n>/ in the output directory (.e2e by default), so the folded pull request comment keeps its evidence and stays red while anything is owed.

@e2e-dev/mobile@0.9.2

Patch Changes

• #711 a3da00d Thanks @okwasniewski! - device.installApp() no longer pins the build's file path as the app when agent-device reports no bundle id or package for it, which on Android made app.open() fail with "Android runtime hints require an installed package name". It fails at the install with ENGINE_FAILURE naming app.bundleId or installApp's app option; the mobile docs' Troubleshooting covers the Android cause (agent-device 0.21.18 reads an aapt2-built APK's package only through the SDK's aapt, which it does not find in the macOS default SDK location, or when the install added the package). app.open() on a device now keeps the engine's own reason it cannot launch, such as a build not installed yet with device.installApp(), instead of a generic "pin one with app.bundleId or app.appPath".

• #791 b3e1fd1 Thanks @DimaMishchenko! - Update agent-device to 0.21.20 so sibling-sized iOS toolbar containers no longer falsely mark nearby controls as covered.

• #801 ae3dc64 Thanks @thymikee! - Recognize agent-device's stale-ref refusals by their details.reason instead of matching their message, so a reworded message in a later agent-device release cannot turn a stale ref into an engine failure. Every ref-frame reason agent-device 0.21.20 sends is covered, including the three whose messages the old pattern did not match.

• #722 f0f9c8d Thanks @okwasniewski! - Explicit navigation now uses an allowlist instead of a denylist: app.open, browser.goto, and the agent's navigate verb admit http:, https:, and the exact about:blank, and every other scheme (chrome:, blob:, about:srcdoc, ...) is POLICY_DENIED. A wrapped scheme such as view-source:file:///... no longer loads a local file; it is POLICY_DENIED like file: itself. device.openLink and device.openApp also refuse view-source:, blob:, and filesystem: links. browser.setCookies refuses an about:blank cookie URL with POLICY_DENIED.

• #793 4db7d6d Thanks @DeryFerd! - device.openLink no longer echoes the string it refused in its INVALID_ARGUMENT message. A malformed link can carry a magic-link token in its query, path, or userinfo, so the message names no part of the input, matching what device.openApp already does.

[github-actions]

🟢 e2e web: 91 passed

All 91 tests in 41 files

	Test	Time
🟢	tests/aria-hidden-flow.e2e.ts · 2 passed	423ms
🟢	aria hidden flow › picks the Pro plan by its visible text and confirms it	307ms
🟢	aria hidden flow › another plan shows the error and keeps the picker open	116ms
🟢	tests/async-states.e2e.ts · 2 passed	7.0s
🟢	async states › reads the code off the transient toast and verifies it once the button enables	2.2s
🟢	async states › a wrong code is rejected after the toast has gone	4.8s
🟢	tests/canvas-only.e2e.ts · 2 passed	213ms
🟢	canvas only › taps square, circle, then diamond at their pixel coordinates	110ms
🟢	canvas only › a shape out of order paints the error and resets progress	103ms
🟢	tests/canvas-whiteboard.e2e.ts · 2 passed	322ms
🟢	canvas whiteboard › drags both shapes into their slots and saves	187ms
🟢	canvas whiteboard › bounces a shape off the wrong slot and refuses to save early	135ms
🟢	tests/cart-totals.e2e.ts · 2 passed	296ms
🟢	cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug)	179ms
🟢	cart totals › a quantity never drops below one	117ms
🟢	tests/checkout-review.e2e.ts · 1 passed	146ms
🟢	shows a raw greeting token and an undefined delivery date (planted bug), yet places the order	146ms
🟢	tests/context-menu-trap.e2e.ts · 2 passed	446ms
🟢	context menu trap › right-click opens nothing; the row menu renames the file	230ms
🟢	context menu trap › the decoy menu items refuse and keep the file	216ms
🟢	tests/control-inventory.e2e.ts · 8 passed	3.2s
🟢	control inventory › check sets the checkbox and the radio without flipping a set one	214ms
🟢	control inventory › check picks a radio the pick replaces with its summary	122ms
🟢	control inventory › a long press and a double click are told apart from a tap	931ms
🟢	control inventory › the file menu opens on right-click only and Rename renames the file	202ms
🟢	control inventory › setInputFiles attaches files named from the project root	130ms
🟢	control inventory › the details view is left through the browser history	146ms
🟢	control inventory › scrollIntoView brings the footnote into view	237ms
🟢	control inventory › every exercise done shows the success message	1.2s
🟢	tests/css-content-ui.e2e.ts · 2 passed	266ms
🟢	css content ui › the tile labelled Ship it advances and confirming ships	146ms
🟢	css content ui › a wrong tile shows the CSS-rendered error and keeps the picker	120ms
🟢	tests/date-picker.e2e.ts · 2 passed	508ms
🟢	date picker › navigates three months ahead and books the required date	294ms
🟢	date picker › confirming without the exact date shows the error	214ms
🟢	tests/debounced-search.e2e.ts · 2 passed	2.8s
🟢	debounced search › a decoy one character off is refused	1.4s
🟢	debounced search › picking the exact item adds it to the cart	1.4s
🟢	tests/deferred-dialog.e2e.ts · 1 passed	1.0s
🟢	one submit opens one confirm, even when double-clicked	1.0s
🟢	tests/div-soup.e2e.ts · 2 passed	561ms
🟢	div soup › a wrong code is rejected	244ms
🟢	div soup › the right code grants access after the delete tile fixes a typo	317ms
🟢	tests/drag-and-drop.e2e.ts · 2 passed	580ms
🟢	drag and drop › an out-of-order item resets the dropzone	298ms
🟢	drag and drop › dropping Banana then Cherry passes	282ms
🟢	tests/filter-deep-link.e2e.ts · 2 passed	221ms
🟢	filter deep link › filtering writes the query to the URL	120ms
🟢	filter deep link › opening a shared link restores the filter	101ms
🟢	tests/gift-card-purchase.e2e.ts · 3 passed	473ms
🟢	gift card purchase › refuses to buy without an amount	144ms
🟢	gift card purchase › refuses a recipient without an at sign	178ms
🟢	gift card purchase › sends the gift card	151ms
🟢	tests/hover-menu.e2e.ts · 2 passed	398ms
🟢	hover menu › a decoy leaf is refused	167ms
🟢	hover menu › hovering Account, then Billing, reaches Redeem voucher	231ms
🟢	tests/iframe-form.e2e.ts · 2 passed	387ms
🟢	iframe form › a wrong code is refused by the host page	190ms
🟢	iframe form › the code read from one frame applies through the other	197ms
🟢	tests/image-only-ui.e2e.ts · 3 passed	397ms
🟢	image only ui › the icons have no names, so the bare svg locator is ambiguous	83ms
🟢	image only ui › a wrong icon shows the error image and keeps the icons	115ms
🟢	image only ui › tapping the star, then the heart, verifies the icons	199ms
🟢	tests/infinite-scroll.e2e.ts · 1 passed	3.9s
🟢	scrolls the feed until the Golden Ticket loads, then claims it	3.9s
🟢	tests/kanban-board.e2e.ts · 3 passed	599ms
🟢	kanban board › pointer drags move both cards into the goal layout	316ms
🟢	kanban board › a swipe along a path drags a card as well	135ms
🟢	kanban board › a plain tap moves nothing and the untouched board is rejected	148ms
🟢	tests/login-form.e2e.ts · 4 passed	616ms
🟢	login form › rejects a malformed email	132ms
🟢	login form › rejects a short password	135ms
🟢	login form › rejects an unknown account	149ms
🟢	login form › signs in and logs out	200ms
🟢	tests/lying-labels.e2e.ts · 4 passed	530ms
🟢	lying labels › the button whose accessible name says Pay now is the wrong one	113ms
🟢	lying labels › a text match for Pay now catches the decoy phrase too	96ms
🟢	lying labels › the field whose accessible name says Amount is the wrong one	171ms
🟢	lying labels › visible text drives the payment through	150ms
🟢	tests/native-dialogs.e2e.ts · 3 passed	367ms
🟢	native dialogs › answers the confirm, prompt, and alert chain	117ms
🟢	native dialogs › dismissing the confirm cancels the order	132ms
🟢	native dialogs › a wrong gift code is rejected	118ms
🟢	tests/newsletter-signup.e2e.ts · 1 passed	2.2s
🟢	subscribe renders no feedback and logs a TypeError (planted bug)	2.2s
🟢	tests/onboarding-wizard.e2e.ts · 5 passed	4.3s
🟢	onboarding wizard › back keeps what was typed and the brief comes back with it	290ms
🟢	onboarding wizard › the address step takes the brief through its bare-div dropdown	301ms
🟢	onboarding wizard › the profile step steers the calendar three decades back	1.1s
🟢	onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off	1.3s
🟢	onboarding wizard › the review step repeats the brief and finishes with the PIN	1.3s
🟢	tests/order-history.e2e.ts · 2 passed	1.4s
🟢	order history › overview and settings render their panels	161ms
🟢	order history › the orders tab never leaves its loading state (planted bug)	1.2s
🟢	tests/otp-auto-advance.e2e.ts · 3 passed	1.3s
🟢	otp auto-advance › focus advances with each digit and the full code verifies	453ms
🟢	otp auto-advance › backspace on an empty box clears and refocuses the previous one	130ms
🟢	otp auto-advance › a wrong code clears every box and starts over	761ms
🟢	tests/overlay-trap.e2e.ts · 1 passed	731ms
🟢	continues only after the banner and the overlay are gone	731ms
🟢	tests/playbook-cleanup.e2e.ts · 2 passed	407ms
🟢	playbook cleanup › deleting the draft removes it from an exact-name search	208ms
🟢	playbook cleanup › cancelling the dialog keeps the draft searchable	199ms
🟢	tests/price-sorting.e2e.ts · 3 passed	330ms
🟢	price sorting › opens in the featured order with sale prices struck through	98ms
🟢	price sorting › low to high sorts by the original price, not the paid one (planted bug)	118ms
🟢	price sorting › high to low is the same wrong order reversed (planted bug)	114ms
🟢	tests/product-reviews.e2e.ts · 1 passed	1.1s
🟢	the running-shoe page lists coffee-grinder reviews (planted bug)	1.1s
🟢	tests/promo-storefront.e2e.ts · 1 passed	1.4s
🟢	buys a deal once the promo chrome is out of the way	1.4s
🟢	tests/rich-text-editor.e2e.ts · 2 passed	414ms
🟢	rich text editor › bolding exactly the required word passes the check	195ms
🟢	rich text editor › bold that spills into the previous word fails the check	219ms
🟢	tests/saved-addresses.e2e.ts · 3 passed	462ms
🟢	saved addresses › adds an address and clears the form	153ms
🟢	saved addresses › ignores an add with an empty field	147ms
🟢	saved addresses › delete shows its toast but keeps the address, on retry too (planted bug)	162ms
🟢	tests/shadow-dom-form.e2e.ts · 2 passed	501ms
🟢	shadow DOM form › rejects a wrong access code	249ms
🟢	shadow DOM form › grants access with the code from the hint inside the root	252ms
🟢	tests/smoke.e2e.ts · 1 passed	377ms
🟢	home lists the scenarios and opens one	377ms
🟢	tests/stale-dom.e2e.ts · 1 passed	2.1s
🟢	taps the moving target three times across rebuilds of the list	2.1s
🟢	tests/sticky-chrome.e2e.ts · 1 passed	164ms
🟢	accepts the terms once the button clears the sticky footer	164ms
🟢	tests/virtualized-table.e2e.ts · 1 passed	1m 5s
🟢	scrolls the windowed table to the Golden Row and claims it	1m 5s
🟢	tests/wishlist.e2e.ts · 2 passed	1.5s
🟢	wishlist › saving toggles the button and the tab badge	233ms
🟢	wishlist › the wishlist tab renders the empty state despite the badge count (planted bug)	1.3s

_{e2e 0.17.0 · 1m 50s · web · run artifacts}

[github-actions]

🟢 e2e web agent: 33 passed

33 agent steps · 120 model calls · 548.7k tokens (91% cached) · $0.0260

All 33 tests in 5 files

	Test	Agent	Time
🟢	tests-agent/control-inventory.e2e.ts · 7 passed	7 steps · 19 calls	28.1s
🟢	control inventory › check sets the checkbox and the radio	1 step · 2 calls	3.1s
🟢	control inventory › check picks a radio the pick replaces with its summary	1 step · 2 calls	2.4s
🟢	control inventory › a long press and a double click are chosen over a plain tap	1 step · 3 calls	3.8s
🟢	control inventory › a right-click opens the file menu and Rename renames the file	1 step · 5 calls	8.8s
🟢	control inventory › upload attaches the named project files	1 step · 2 calls	2.9s
🟢	control inventory › back returns from the details view to the inventory	1 step · 3 calls	3.6s
🟢	control inventory › scroll_to brings the footnote into view	1 step · 2 calls	3.5s
🟢	tests-agent/login-form.e2e.ts · 1 passed	1 step · 3 calls	3.9s
🟢	act signs in with the benchmark account	1 step · 3 calls	3.9s
🟢	tests-agent/planted-bugs.e2e.ts · 3 passed	3 steps · 6 calls	9.6s
🟢	cart totals: a judgment catches the stale order summary	1 step · 2 calls	2.6s
🟢	checkout review: a judgment catches the broken copy	1 step · 1 call	1.6s
🟢	wishlist: a judgment catches the lost saved items	1 step · 3 calls	5.4s
🟢	tests-agent/scenarios.e2e.ts · 21 passed	21 steps · 89 calls	3m 50s
🟢	act completes shadow-dom-form	1 step · 3 calls	3.0s
🟢	act completes lying-labels	1 step · 4 calls	6.2s
🟢	act completes iframe-form	1 step · 3 calls	3.8s
🟢	act completes infinite-scroll	1 step · 3 calls	8.6s
🟢	act completes virtualized-table	1 step · 3 calls	1m 43s
🟢	act completes overlay-trap	1 step · 4 calls	4.8s
🟢	act completes sticky-chrome	1 step · 4 calls	5.9s
🟢	act completes async-states	1 step · 6 calls	8.5s
🟢	act completes debounced-search	1 step · 5 calls	5.7s
🟢	act completes hover-menu	1 step · 4 calls	5.4s
🟢	act completes drag-and-drop	1 step · 3 calls	3.9s
🟢	act completes kanban-board	1 step · 4 calls	5.8s
🟢	act completes context-menu-trap	1 step · 7 calls	12.4s
🟢	act completes stale-dom	1 step · 4 calls	5.1s
🟢	act completes otp-auto-advance	1 step · 3 calls	5.4s
🟢	act completes date-picker	1 step · 7 calls	8.1s
🟢	act completes rich-text-editor	1 step · 6 calls	10.5s
🟢	act completes playbook-cleanup	1 step · 6 calls	7.0s
🟢	act completes promo-storefront	1 step · 3 calls	6.2s
🟢	act completes gift-card-purchase	1 step · 5 calls	8.6s
🟢	act filters the list and the URL carries the filter	1 step · 2 calls	2.5s
🟢	tests-agent/unique-cache.e2e.ts · 1 passed	1 step · 3 calls	3.8s
🟢	act adds an address with a run-unique label	1 step · 3 calls	3.8s

_{e2e 0.17.0 · 4m 36s · web · run artifacts}

[github-actions]

🟢 e2e android: 54 passed, 1 skipped

All 55 tests in 7 files

	Test	Time
⏭️	tests/controls.e2e.ts · 24 passed, 1 skipped	2m 44s
🟢	control inventory › a label query resolves a labeled text field	19.8s
🟢	control inventory › a labeled field answers name, display value, value reads, fill, and clear	11.4s
🟢	control inventory › a placeholder query and attribute reads find the field	4.6s
🟢	control inventory › a secure field takes a fill and denies value reads	7.3s
🟢	control inventory › press sends a character and Enter through the soft keyboard	7.2s
🟢	control inventory › a field autofocuses on mount and focus() moves the focus	9.5s
🟢	control inventory › toBeFocused follows the focus	8.6s
🟢	control inventory › a switch checks, unchecks, and reports its state	9.6s
🟢	control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap	6.4s
🟢	control inventory › checkbox and radio roles check, uncheck, and report their state	11.1s
🟢	control inventory › a tab role answers getByRole	3.6s
🟢	control inventory › a radio group reports the checked option	8.6s
🟢	control inventory › buttons report disabled, enabled, and a state that arrives later	7.7s
🟢	control inventory › a display: none element is absent rather than hidden, its visible twin attached	5.9s
🟢	control inventory › a header answers its accessible name	3.4s
🟢	control inventory › a tab row answers count, all, texts, filters, positions, and switching	5.2s
🟢	control inventory › a list answers counts, list-form text, reads, and boxes in order	4.7s
⏭️	control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢	control inventory › a long press lands as one	6.9s
🟢	control inventory › a double tap lands inside the 300 ms window	7.6s
🟢	control inventory › an ambiguous locator fails at once	4.5s
🟢	control inventory › screenshot, restart, and clearState reopen the app on its home list	7.7s
🟢	refusals › verbs a device never declares are refused before the engine	2ms
🟢	refusals › keys the soft keyboard cannot send are refused by the engine	1.3s
🟢	refusals › a missing locator fails as not found once its timeout passes	1.0s
🟢	tests/device-fixture.e2e.ts · 8 passed	59.7s
🟢	device fixture › setOrientation rotates the window the app measures	3.9s
🟢	device fixture › home leaves the app, openApp brings it back where it was	5.3s
🟢	device fixture › foregroundApp sees the home screen after home()	3.9s
🟢	device fixture › closeApp then openApp relaunches on the home list	5.3s
🟢	device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN	3.7s
🟢	device fixture › device.back pops the scenario	4.4s
🟢	device fixture › setNetwork and setAirplaneMode change what the app sees	14.5s
🟢	device fixture › setLocation and clearLocation change what the app reads	18.7s
🟢	tests/device.e2e.ts · 4 passed	35.1s
🟢	permission prompt › allowing the system dialog enables the microphone	6.2s
🟢	permission prompt › a denied prompt recovers once the permission is granted outside	12.3s
🟢	sequential onboarding echoes the exact values	9.6s
🟢	onboarding rejects a malformed email before advancing	7.0s
🟢	tests/flows-lists.e2e.ts · 7 passed	2m 6s
🟢	infinite scroll reaches the target item	17.1s
🟢	product catalog adds exactly two of the right variant	9.5s
🟢	gestures: long-press, then swipe left	7.8s
🟢	gestures: double-tap completes the flow	9.1s
🟢	sticky chrome: accept below the fold, then continue	11.6s
🟢	async states: load, pull to refresh, claim through the toast	11.4s
🟢	huge virtualized list reaches row 512	59.3s
🟢	tests/flows.e2e.ts · 7 passed	49.8s
🟢	text inputs unlock the submit button	6.3s
🟢	filled text inputs report their values	5.3s
🟢	modal flow confirms through the native alert	7.7s
🟢	bottom tabs act inside the last tab	6.1s
🟢	error recovery retries, then confirms the delete	8.1s
🟢	choice controls place the exact order	8.2s
🟢	debounced search selects the target once results arrive	8.2s
🟢	tests/login-form.e2e.ts · 3 passed	27.8s
🟢	login form › rejects a malformed email	7.2s
🟢	login form › rejects an unknown account	8.4s
🟢	login form › signs in and logs out	12.2s
🟢	tests/smoke.e2e.ts · 1 passed	5.2s
🟢	home lists the scenarios and opens one	5.2s

_{e2e 0.17.0 · 7m 47s · android-emulator · run artifacts}

[github-actions]

🟢 e2e ios: 1 flaky, 51 passed, 1 skipped

⚠️ 1 flaky test passed on a retry

⚠️ control inventory › toBeFocused follows the focus
tests/controls.e2e.ts:101

ASSERTION_FAILED at step 5 of 5: expect.toBeFocused getByRole("textbox", name: "Autofocus field"), after 5.3s

• Expected: focused
• Observed: default states (1 match)
• Failed the same way on 2 attempts: ASSERTION_FAILED at step 5. The retry that passed is the exception.
• Screen: dev.e2e.benchmark / Control Inventory

Evidence: screenshot, log

All 53 tests in 7 files

	Test	Time
⚠️	tests/controls.e2e.ts · 1 flaky, 22 passed, 1 skipped	3m 58s
🟢	control inventory › a label query resolves a labeled text field	13.4s
🟢	control inventory › a labeled field answers name, display value, value reads, fill, and clear	20.2s
🟢	control inventory › a placeholder query and attribute reads find the field	8.9s
🟢	control inventory › a secure field takes a fill and denies value reads	11.7s
🟢	control inventory › press sends a character and Enter through the soft keyboard	9.7s
🟢	control inventory › a field autofocuses on mount and focus() moves the focus	13.6s
⚠️	control inventory › toBeFocused follows the focus (2 failed attempts first)	15.6s
🟢	control inventory › a switch checks, unchecks, and reports its state	13.8s
🟢	control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap	10.2s
🟢	control inventory › checkbox and radio roles check, uncheck, and report their state	17.0s
🟢	control inventory › a radio group reports the checked option	11.0s
🟢	control inventory › buttons report disabled, enabled, and a state that arrives later	9.7s
🟢	control inventory › a display: none element is absent rather than hidden, its visible twin attached	9.1s
🟢	control inventory › a header answers its accessible name	5.5s
🟢	control inventory › a tab row answers count, all, texts, filters, positions, and switching	9.1s
🟢	control inventory › a list answers counts, list-form text, reads, and boxes in order	10.4s
⏭️	control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢	control inventory › a long press lands as one	11.0s
🟢	control inventory › pointer verbs land at the point asked for	13.4s
🟢	control inventory › an ambiguous locator fails at once	7.8s
🟢	control inventory › screenshot, restart, and clearState reopen the app on its home list	15.0s
🟢	refusals › verbs a device never declares are refused before the engine	3ms
🟢	refusals › keys the soft keyboard cannot send are refused by the engine	486ms
🟢	refusals › a missing locator fails as not found once its timeout passes	1.3s
🟢	tests/device-fixture.e2e.ts · 9 passed	1m 45s
🟢	device fixture › setAppearance flips the color scheme the app reads	16.3s
🟢	device fixture › setOrientation rotates the window the app measures	7.9s
🟢	device fixture › the clipboard reads back what was written	6.6s
🟢	device fixture › home leaves the app, openApp brings it back where it was	13.5s
🟢	device fixture › closeApp then openApp relaunches on the home list	10.9s
🟢	device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN	6.7s
🟢	device fixture › device.back pops the scenario	7.4s
🟢	device fixture › setLocation and clearLocation change what the app reads	19.4s
🟢	device fixture › enrollBiometrics and setBiometrics answer a biometric prompt	16.3s
🟢	tests/device.e2e.ts · 2 passed	1m 3s
🟢	sequential onboarding echoes the exact values	47.0s
🟢	onboarding rejects a malformed email before advancing	15.6s
🟢	tests/flows-lists.e2e.ts · 7 passed	4m 18s
🟢	infinite scroll reaches the target item	31.9s
🟢	product catalog adds exactly two of the right variant	14.3s
🟢	gestures: long-press, then swipe left	11.9s
🟢	sticky chrome: accept below the fold, then continue	13.6s
🟢	async states: load, pull to refresh, claim through the toast	12.6s
🟢	huge virtualized list reaches row 512	2m 37s
🟢	web view coupon form applies the code shown on the page	16.8s
🟢	tests/flows.e2e.ts · 7 passed	1m 30s
🟢	text inputs unlock the submit button	20.0s
🟢	filled text inputs report their values	12.9s
🟢	modal flow confirms through the native alert	11.9s
🟢	bottom tabs act inside the last tab	9.6s
🟢	error recovery retries, then confirms the delete	11.8s
🟢	choice controls place the exact order	10.8s
🟢	debounced search selects the target once results arrive	13.3s
🟢	tests/login-form.e2e.ts · 3 passed	55.8s
🟢	login form › rejects a malformed email	14.9s
🟢	login form › rejects an unknown account	14.7s
🟢	login form › signs in and logs out	26.2s
🟢	tests/smoke.e2e.ts · 1 passed	11.9s
🟢	home lists the scenarios and opens one	11.9s

_{e2e 0.17.0 · 7m 26s · ios-simulator · run artifacts}

[github-actions]

🟢 e2e android agent: 20 passed

24 agent steps · 117 model calls · 648.3k tokens (84% cached) · $0.0404

All 20 tests in 1 file

	Test	Agent	Time
🟢	tests-agent/scenarios.e2e.ts · 20 passed	24 steps · 117 calls	9m 17s
🟢	act completes Login Form	1 step · 4 calls	39.7s
🟢	act completes Infinite Scroll List	1 step · 3 calls	25.7s
🟢	act completes Modal Flow	1 step · 4 calls	11.4s
🟢	act completes Bottom Tabs	1 step · 3 calls	10.0s
🟢	act completes Text Input Variations	1 step · 5 calls	12.3s
🟢	act completes Broken Accessibility	2 steps · 6 calls	28.0s
🟢	act completes Vision Only	2 steps · 6 calls	28.7s
🟢	act completes Gestures	1 step · 12 calls	43.6s
🟢	act completes Async States	1 step · 10 calls	42.3s
🟢	act completes Debounced Search	1 step · 4 calls	9.8s
🟢	act completes Huge Virtualized List	1 step · 3 calls	1m 15s
🟢	act completes Flattened Registration Form	2 steps · 9 calls	50.9s
🟢	act completes Flattened Login	2 steps · 6 calls	26.8s
🟢	act completes Sticky Chrome Target	1 step · 4 calls	14.5s
🟢	act completes WebView Accessibility	1 step · 5 calls	21.0s
🟢	act completes Permission Prompt	1 step · 3 calls	8.3s
🟢	act completes Product Catalog	1 step · 5 calls	12.6s
🟢	act completes Error Recovery	1 step · 7 calls	12.9s
🟢	act completes Choice Controls	1 step · 6 calls	23.8s
🟢	act completes Sequential Onboarding	1 step · 12 calls	59.5s

_{e2e 0.17.0 · 9m 18s · android-emulator · run artifacts}

[github-actions]

🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped

25 agent steps · 129 model calls · 54.6k tokens (85% cached) · $0.0035

⚠️ 1 flaky test passed on a retry

⚠️ act completes Gestures
tests-agent/scenarios.e2e.ts:257

ACTION_FAILED at step 4 of 4: agent.act "Complete this scenario as the screen instructs: long-press the first target, swipe the card left, then double-tap the last target", after 1m 8s and 12 model calls

agent.act failed: Long-pressed the first target and swiped the card left; both advanced to the next instruction. Tried double-tapping the final target with the double-tap action and two taps, but the screen stayed on Step 3 of 3, so the final gesture did not complete.

• Failed the same way on 2 attempts: ACTION_FAILED at step 4. The retry that passed is the exception.
• Turn 9: double_tap({"target":"n3855"}) → [double_tap] Double-tapped #n3855.
• Turn 10: tap({"target":"n3885"}) → [tap] Tapped #n3885.
• Turn 11: tap({"target":"n3915"}) → [tap] Tapped #n3915.
• Screen: dev.e2e.benchmark / Benchmark Examples

Evidence: screenshot, log

All 22 tests in 1 file

	Test	Agent	Time
⚠️	tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped	25 steps · 129 calls	14m 2s
🟢	act completes Login Form	1 step · 7 calls	1m 9s
🟢	act completes Infinite Scroll List	1 step · 3 calls	56.8s
🟢	act completes Modal Flow	1 step · 4 calls	17.3s
🟢	act completes Bottom Tabs	1 step · 3 calls	13.9s
🟢	act completes Text Input Variations	1 step · 3 calls	28.8s
🟢	act completes Broken Accessibility	2 steps · 7 calls	41.1s
🟢	act completes Vision Only	2 steps · 6 calls	53.7s
⚠️	act completes Gestures (2 failed attempts first)	1 step · 10 calls	56.8s
🟢	act completes Async States	1 step · 8 calls	39.7s
🟢	act completes Debounced Search	1 step · 5 calls	23.2s
🟢	act completes Flattened Registration Form	2 steps · 14 calls	1m 43s
🟢	act completes Flattened Login	2 steps · 12 calls	1m 29s
🟢	act completes Sticky Chrome Target	1 step · 7 calls	40.8s
🟢	act completes WebView Accessibility	1 step · 4 calls	25.8s
🟢	act completes Permission Prompt	1 step · 3 calls	21.5s
🟢	act completes Photo Picker	1 step · 4 calls	19.4s
🟢	act completes Product Catalog	1 step · 5 calls	17.7s
🟢	act completes Error Recovery	1 step · 6 calls	17.3s
🟢	act completes Choice Controls	1 step · 8 calls	37.9s
🟢	act completes Stripe PaymentSheet	1 step · 6 calls	41.4s
🟢	act completes Apple Pay	1 step · 4 calls	27.8s
⏭️	act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

_{e2e 0.17.0 · 16m 14s · ios-simulator · run artifacts}