diff --git a/.changeset/atomic-pinned-locate.md b/.changeset/atomic-pinned-locate.md deleted file mode 100644 index 7e7f53864..000000000 --- a/.changeset/atomic-pinned-locate.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -"@e2e-dev/web": patch ---- - -A locator whose match the page replaces on every animation frame (a field re-rendered each frame) no longer fails with `LOCATOR_NOT_FOUND` "node became stale" on a loaded machine. Locate read each match in a protocol call after the one that found it, so when a frame passed between the two, which a busy runner makes likely every time, the read found the match detached and the locate retried until the action timeout. Each match is now read in the same in-page task that finds it, so the read never sees a replaced element. A `getByLabel` or `getByDisplayValue` match is still pinned to the element that was read. - -A `browser.locator` selector with a `*` capture before its last part (`*css=article >> text=Hello`) now fails with `INVALID_LOCATOR` instead of resolving; use `filter({ has })`. A capture rewrites the chain so the elements it returns are never the ones the rest of the locate reads, and composing anything onto such a selector already matched the captured element instead of its descendants. diff --git a/.changeset/codex-client-version.md b/.changeset/codex-client-version.md deleted file mode 100644 index 928f80757..000000000 --- a/.changeset/codex-client-version.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'e2e': patch ---- - -`e2e models openai` asks the ChatGPT backend for its model list as Codex CLI 0.160.0 (was 0.155.1). The backend hides models from clients it considers too old, so models unlocked since 0.155.1 now show up and can be passed to `chatgpt()`. diff --git a/.changeset/copilot-unusable-models.md b/.changeset/copilot-unusable-models.md deleted file mode 100644 index 8263c6245..000000000 --- a/.changeset/copilot-unusable-models.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'e2e': patch ---- - -`e2e models github-copilot` marks the models `copilot()` cannot call: `not enabled` for a model your plan has not turned on, and `no chat completions` for a model Copilot serves only over other APIs such as Responses. These used to be listed like any other model and then failed the first agent step with "The requested model is not supported". diff --git a/.changeset/decision-executor.md b/.changeset/decision-executor.md new file mode 100644 index 000000000..9c7941594 --- /dev/null +++ b/.changeset/decision-executor.md @@ -0,0 +1,11 @@ +--- +"@e2e-dev/decision": minor +--- + +Add `@e2e-dev/decision`, an executor that runs `agent.act` and `agent.assert` +through a decision model instead of an LLM. It takes any AI SDK evaluation +model with choice distributions and decides through `experimental_evaluate`, +one call per action with the operation and target questions fanned out; a +small language model writes field values when the choice is `type`. Tests +stay plain natural language with no params. Probability and confidence +gates are off by default. diff --git a/.changeset/download-full-ledger-pass.md b/.changeset/download-full-ledger-pass.md new file mode 100644 index 000000000..7e27a9243 --- /dev/null +++ b/.changeset/download-full-ledger-pass.md @@ -0,0 +1,5 @@ +--- +"e2e": patch +--- + +A text download is rewritten with the full ledger pass — fragments and base64-encoded forms of a secret included — the way trace text entries already are. The download path only matched whole values while still labeling the file `redaction: "complete"`, so a cut or base64 copy of a secret could sit in a download the report vouched for as clean. diff --git a/.changeset/failure-evidence-budget.md b/.changeset/failure-evidence-budget.md new file mode 100644 index 000000000..7e2d44bad --- /dev/null +++ b/.changeset/failure-evidence-budget.md @@ -0,0 +1,5 @@ +--- +'e2e': patch +--- + +A failed attempt on an app that stops answering no longer waits 5 s for failure evidence at every capture point, or forever when the engine ignores cancellation. The runner captures the screen and screenshot once per attempt, stops waiting after 5 s, and keeps the original error. diff --git a/.changeset/matcher-options-honored-or-rejected.md b/.changeset/matcher-options-honored-or-rejected.md deleted file mode 100644 index 209e481d9..000000000 --- a/.changeset/matcher-options-honored-or-rejected.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -'e2e': patch ---- - -Locator matchers honor Playwright's `{ checked: false }`, `{ enabled: false }`, `{ visible: false }`, and `{ attached: false }`, which flip the matcher, and `{ ignoreCase }` on `toHaveText`, `toContainText`, `toHaveAccessibleName`, and `toHaveAttribute(name, value)`. Before, these were ignored at runtime, so `toBeChecked({ checked: false })` passed on a checked box and `not.toContainText('error', { ignoreCase: true })` passed on `Error`. Any other option key, or a flag that is not a boolean, is now `INVALID_ARGUMENT` before the first read, from JavaScript too. - -`e2e/engine` exports `rejectUnknownOptions(api, options, known, code?)`, the check the runner's own fixtures use, so an engine refuses an option its fixture methods and matchers do not take the same way. diff --git a/.changeset/mcp-without-ai.md b/.changeset/mcp-without-ai.md deleted file mode 100644 index 803307ffe..000000000 --- a/.changeset/mcp-without-ai.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'e2e': patch ---- - -`e2e mcp` opens a session in a project without the optional `ai` package. Without the AI SDK, the catalog and the argument checks read each tool's Standard Schema (zod's), so a deterministic-only project (no `agents`, `e2e init` with no model gateway) can `open_session`, `observe`, `locate`, and drive the page. diff --git a/.changeset/oxc-typescript-loader.md b/.changeset/oxc-typescript-loader.md new file mode 100644 index 000000000..671f714f4 --- /dev/null +++ b/.changeset/oxc-typescript-loader.md @@ -0,0 +1,12 @@ +--- +'e2e': minor +'@e2e-dev/web': minor +'@e2e-dev/mobile': minor +'@e2e-dev/kernel': minor +'@e2e-dev/eas': minor +'@e2e-dev/github': minor +--- + +**Breaking: e2e and every `@e2e-dev` package now need Node.js 22.22.3 or newer on Node.js 22, or 24.8.0 or newer** (`engines.node` is `^22.22.3 || >=24.8.0`). Node.js 22.12 through 22.22.2, 23, and 24.0 through 24.7 are no longer supported, and the CLI refuses to start on them, naming the versions to upgrade to. Those releases fail when a CommonJS file a test imports requires TypeScript through the new loader's `module.registerHooks` (nodejs/node#59679 fixed it in 22.22.3 and 24.8.0). + +e2e now loads TypeScript with its own loader, built on oxc, instead of tsx. tsx brought in esbuild, whose postinstall script made every `pnpm install` on pnpm 11 and later fail with `ERR_PNPM_IGNORED_BUILDS` until the project approved the build; nothing e2e installs runs an install script now. Config and tests load as before: ESM whatever `package.json` says, `./x.js` and `./x` resolve to `x.ts`, directory index imports, tsconfig `paths` and `baseUrl`, workspace packages exporting `.ts` source, JSX in `.tsx` and `.jsx`, enums, namespaces, `experimentalDecorators`, and source-mapped stack traces. JSON imported without `with { type: 'json' }` still loads. A test file whose name holds `%`, `#`, or a space now collects with its source location instead of failing with "URI malformed" or losing the location. Each file reads its nearest `tsconfig.json` rather than the one in the working directory. What changes: `import` statements in a `.cts` file are no longer compiled into `require` calls (write `import x = require()`); standard decorators without `experimentalDecorators`, auto-accessors, and ES module `import` and value `export` declarations in `.cts` are refused with their line (`import type`, `export type`, `import x = require()`, and `export =` work); the CLI refuses the Bun (`bun --bun`) and Deno runtimes (install with any package manager, run on Node.js); a type imported without `import type` that stays in the output (a decorated member's type under `emitDecoratorMetadata`) fails to load, with an error naming the import; a `.ts` file a CommonJS file requires runs as an ES module like every other `.ts`, so one written with `module.exports` needs the `.cts` extension; and an import written in a JavaScript file resolves as Node.js resolves it, without TypeScript's extension rules or tsconfig `paths`. TypeScript inside an installed package compiles without the project's `tsconfig.json`. diff --git a/.changeset/plain-string-secret-trace.md b/.changeset/plain-string-secret-trace.md new file mode 100644 index 000000000..754bb39b2 --- /dev/null +++ b/.changeset/plain-string-secret-trace.md @@ -0,0 +1,5 @@ +--- +'e2e': patch +--- + +A secret value a test passes as a plain string (spelled into an `app.open` URL, typed with `fill`) no longer stays in the Playwright trace. Traces and text downloads are now rewritten whenever the session knows a secret value (every static one, and a provider's once resolved), not only after a secret fill or an engine-held secret, so they are labeled `redaction: "complete"` instead of `not-required` or `incomplete`. Screencast frames are still dropped only after a secret fill. Like any secret trace, a rewritten one masks unrelated text that shares 8 characters with a value. diff --git a/.changeset/replay-lost-list-once.md b/.changeset/replay-lost-list-once.md new file mode 100644 index 000000000..cca3d4911 --- /dev/null +++ b/.changeset/replay-lost-list-once.md @@ -0,0 +1,5 @@ +--- +"e2e": patch +--- + +Replaying a repeated scroll on a list that filled the screen, when the list can no longer be found, waits for it once and then scrolls the screen for every remaining repeat. It used to wait out the full relocation backoff (about 14 seconds) before each repeat, so a recorded 4x scroll took about a minute to replay. diff --git a/.changeset/serial-group-verdicts.md b/.changeset/serial-group-verdicts.md new file mode 100644 index 000000000..36ee965e0 --- /dev/null +++ b/.changeset/serial-group-verdicts.md @@ -0,0 +1,5 @@ +--- +"e2e": patch +--- + +A serial group whose session fails to launch, or whose file fails to load, now fails the run. Before, every member was skipped and the run passed with exit code 0. The first member now fails with the launch error and the rest skip with cause `serial-predecessor-failed`, so the `list` reporter, `summary.md`, and `junit.xml` show the failure. A retry that fails to launch keeps the members' verdicts from the attempt before it, so a member that failed there still fails. An infrastructure error in a serial group, at launch or in a member, now exits 3 and sets the run status to `error`, as it does for any other test, instead of exiting 1. When a worker crashes during a serial group, the member that was running fails with `WORKER_CRASH` instead of being skipped as if it never started. diff --git a/.changeset/slim-install.md b/.changeset/slim-install.md deleted file mode 100644 index d21572e7d..000000000 --- a/.changeset/slim-install.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -'e2e': patch -'@e2e-dev/web': patch -'@e2e-dev/mobile': patch -'@e2e-dev/github': patch -'@e2e-dev/kernel': patch -'@e2e-dev/eas': patch ---- - -Installing `e2e` pulls in 29 packages instead of 117 and takes about 31MB instead of 36MB. `e2e mcp` now runs on the split MCP SDK (`@modelcontextprotocol/server` 2.2.0) in place of `@modelcontextprotocol/sdk`, which brought in express, hono, and the rest of an HTTP server stack that stdio never used. The server keeps the same protocol version, so existing clients connect as before. Packages are built and published without sourcemaps, which pointed at a `src/` that was never shipped. Stack traces show `dist/` positions. diff --git a/.changeset/telemetry-mcp-explore-config.md b/.changeset/telemetry-mcp-explore-config.md deleted file mode 100644 index 1775d22dd..000000000 --- a/.changeset/telemetry-mcp-explore-config.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'e2e': minor ---- - -Anonymous telemetry covers `e2e mcp` and `e2e explore`. Each MCP session sends one `e2e_mcp_session` event when it closes or fails to open: the client's self-reported name and version, the platform and engine, how it ended, and its tool calls counted by the runner's own tool names with the error codes they failed with. `e2e_run_completed` adds `command`, counts and option ids of the config features the run used, and for `e2e explore` why it stopped and its steps and findings by kind and severity. Beyond the client's own name, no project names, paths, tool arguments or results, goals, or findings are sent. The notice shows again once; `E2E_TELEMETRY_DEBUG=1` prints every event, and the opt-outs are unchanged. diff --git a/.changeset/web-matcher-options-rejected.md b/.changeset/web-matcher-options-rejected.md deleted file mode 100644 index 69c70e823..000000000 --- a/.changeset/web-matcher-options-rejected.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'@e2e-dev/web': patch ---- - -`expect(browser).toHaveURL`, `toHaveTitle`, `toHaveClass`, and `browser.waitForURL` refuse an option other than `timeout` with `INVALID_ARGUMENT` instead of ignoring it, so Playwright's `ignoreCase` on `toHaveURL` or `waitUntil` on `waitForURL` no longer runs as if absent. diff --git a/.claude/skills/verify/SKILL.md b/.claude/skills/verify/SKILL.md index 03767e001..42484baad 100644 --- a/.claude/skills/verify/SKILL.md +++ b/.claude/skills/verify/SKILL.md @@ -16,7 +16,7 @@ repeat. ```bash pnpm install --frozen-lockfile -pnpm --filter @e2e-dev/web exec playwright install chromium +pnpm --filter @e2e-dev/web exec playwright-core install chromium pnpm build # every consumer below runs dist; rebuild after each change ``` diff --git a/.fallowrc.json b/.fallowrc.json index 779d9cb90..7df1d7782 100644 --- a/.fallowrc.json +++ b/.fallowrc.json @@ -16,8 +16,12 @@ "apps/mobile-benchmark/index.ts", "packages/e2e/tests/types/sdk-types.ts", "packages/web/tests/types/connection.ts", + "packages/web/tests/types/init-scripts.ts", + "packages/web/tests/types/routes.ts", "packages/mobile/tests/types/options.ts", - "packages/e2e/tests/live/*.ts" + "packages/decision/tests/types/options.ts", + "packages/e2e/tests/live/*.ts", + "packages/e2e/src/config/register.ts" ], "ignorePatterns": [ "docs/**", @@ -30,7 +34,8 @@ "@e2e-dev/web", "@openrouter/ai-sdk-provider", "@expo/fingerprint", - "@expo/repack-app" + "@expo/repack-app", + "@oxc-project/runtime" ], "rules": { "unused-class-member": "warn", diff --git a/.github/workflows/agent.yml b/.github/workflows/agent.yml index a1a4e4081..cb09710d7 100644 --- a/.github/workflows/agent.yml +++ b/.github/workflows/agent.yml @@ -96,7 +96,7 @@ jobs: fi - run: pnpm install --frozen-lockfile - name: Install browsers - run: pnpm --filter @e2e-dev/web exec playwright install chromium --with-deps + run: pnpm --filter @e2e-dev/web exec playwright-core install chromium --with-deps - name: Build run: pnpm run build - name: Run ${{ matrix.script }} diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index 977bb94ba..67078d505 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -67,7 +67,7 @@ jobs: cache: pnpm - run: pnpm install --frozen-lockfile - name: Install browsers - run: pnpm --filter @e2e-dev/web exec playwright install chromium --with-deps + run: pnpm --filter @e2e-dev/web exec playwright-core install chromium --with-deps - name: Build run: pnpm run build - name: Run test @@ -132,7 +132,7 @@ jobs: fi - run: pnpm install --frozen-lockfile - name: Install browsers - run: pnpm --filter @e2e-dev/web exec playwright install chromium --with-deps + run: pnpm --filter @e2e-dev/web exec playwright-core install chromium --with-deps - name: Build run: pnpm run build - name: Run test:agent diff --git a/.github/workflows/spec.yml b/.github/workflows/spec.yml index 95d1a08e1..a1a0fa8a5 100644 --- a/.github/workflows/spec.yml +++ b/.github/workflows/spec.yml @@ -38,12 +38,14 @@ jobs: run: pnpm docs:check-errors - name: Peer ranges on the runner run: pnpm check:peer-ranges + - name: No install scripts in published dependency trees + run: pnpm check:install-scripts - name: Agent skill scripts run: | pnpm exec tsc -p .claude/skills/babysit/scripts node --test .claude/skills/babysit/scripts/pr-status.test.ts - name: Install browsers - run: pnpm --filter @e2e-dev/web exec playwright install chromium --with-deps + run: node packages/web/dist/bin.js install chromium --with-deps - name: Testbed run: pnpm --filter @e2e-dev/testbed test @@ -53,16 +55,41 @@ jobs: strategy: fail-fast: false matrix: - node: [22, 24, 26] + # The newest release of each line, and the floors in engines.node. + node: [22, 24, 26, 22.22.3, 24.8.0] steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: pnpm/action-setup@9fd676a19091d4595eefd76e4bd31c97133911f1 # v4.2.0 - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: node-version: ${{ matrix.node }} + # A major alone resolves to the runner's cached release, which can + # sit below the patch floor in engines.node (22.22.3). + check-latest: true cache: pnpm - run: pnpm install --frozen-lockfile - name: Install browsers - run: pnpm --filter @e2e-dev/web exec playwright install chromium --with-deps + run: pnpm --filter @e2e-dev/web exec playwright-core install chromium --with-deps - name: Unit and integration tests run: pnpm test + + # A new user's first install of the packed packages, which pnpm 11 and + # later fail on any dependency build script (scripts/install-smoke.ts). + install: + runs-on: blacksmith-4vcpu-ubuntu-2404 + timeout-minutes: 15 + strategy: + fail-fast: false + matrix: + pnpm: [11, 12] + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: pnpm/action-setup@9fd676a19091d4595eefd76e4bd31c97133911f1 # v4.2.0 + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: 26 + cache: pnpm + - run: pnpm install --frozen-lockfile + - run: pnpm run build + - name: Fresh install with pnpm ${{ matrix.pnpm }} + run: node scripts/install-smoke.ts ${{ matrix.pnpm }} diff --git a/AGENTS.md b/AGENTS.md index c12d1fee0..4f761c8c8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,7 +28,6 @@ There is no separate spec. The code is the contract, pinned in three places: There are no RFCs or design documents in the repo. The why lives in PR descriptions and commit bodies; `git log` and `gh pr view` are the archive. - ## Layout `packages/` holds what publishes to npm; `apps/` holds the private apps and @@ -52,8 +51,10 @@ suites that consume the built packages the way a user would. the `act` socket with a queue executor so every MCP call is a harness action), `src/oauth/` subscription sign-in (the `e2e login`, `logout`, and `models` commands and the `e2e/oauth/*` model constructors; each - constructor subpath is the only place its `@ai-sdk/*` optional peer is - imported, so the CLI boots without them). The constructors and the CLI + constructor subpath is the place its `@ai-sdk/*` optional peers are + imported, and the Copilot constructor loads `@ai-sdk/openai` lazily, only + for the Responses models, so the CLI boots without any of them). The + constructors and the CLI are the whole public surface: the flows, stores, and fetch behind them are module-private, not a library for other products. `tests/live/` holds hand-run checks that need a stored login and are never part of `pnpm test`. @@ -136,7 +137,7 @@ suites that consume the built packages the way a user would. Build first — nearly everything downstream consumes `dist`. ```bash -pnpm check # lint -> check:dead-code -> typecheck -> docs:check-errors -> check:peer-ranges -> docs:check (full gate) +pnpm check # lint -> check:dead-code -> typecheck -> docs:check-errors -> check:peer-ranges -> check:install-scripts -> docs:check (full gate) pnpm test # builds, then vitest unit + integration pnpm test:testbed # builds, then runs the real CLI against the playground app pnpm test:web-benchmark # builds, then runs the real CLI against the benchmark scenarios @@ -162,7 +163,7 @@ pnpm --filter @e2e-dev/testbed run test:headed package `typecheck` covers `tests/**`, which is what makes `tests/types/sdk-types.ts` a test. - Integration tests need Chromium: `pnpm --filter @e2e-dev/web exec - playwright install chromium`. The web engine's `prepare` hook + playwright-core install chromium`. The web engine's `prepare` hook also installs a missing browser once per run, in the runner, before `plan` is emitted and the run's clock starts. @@ -373,6 +374,13 @@ trees, on both platforms, without a device. commit the changed entries in the same pull request as the scenario change. The web benchmark's agent job runs with `--strict-cache`, so a recording a change broke fails with `REPLAY_STALE` instead of quietly calling the model. + That includes a change to the cache key (`REPLAY_POLICY_VERSION`, a new key + field, an engine minor): strict lists the file store and fails a step whose + key misses while an entry recorded for the same step sits under another key + (`cache/rekeyed.ts`). Only entries whose `recordedFor` names the whole step + (params digest, occurrence, agent) count; a `read-write` replay completes + an older one. Such a change re-records every entry and deletes the old ones + in the same pull request. A step that was never recorded still runs live. The web benchmark's entries are in. The mobile benchmark's iOS entries are recorded on a Mac; nobody has recorded on an Android emulator yet, so the Android side spends model calls until an emulator recording is committed. @@ -394,8 +402,9 @@ trees, on both platforms, without a device. secret an engine resolves for an option the app sees (basic auth) is protected as text only: redacted everywhere text goes, pixels untouched. One exposure level per session (`SecretExposure` in `run/secrecy.ts`) - decides pixels, trace and download rewriting, and the taint a saved - session carries. What + decides pixels and the taint a saved session carries; traces and text + downloads are rewritten whenever the session's ledger holds a value + (`redactsRecordings`), since a plain string reaches the app unseen. What an executor keeps in `attempt.memory` is its own; the harness never reports it. - An agent's secret fill is authorized by the runner, not the model. @@ -417,10 +426,15 @@ trees, on both platforms, without a device. denied destinations, forbidden fills, and tainted pixels. Model text is never evaluated as code, selectors, shell, or config. App content, ledger text, and pixels are quoted as untrusted evidence with no policy authority. - - Every navigation a test or the agent asks for (`app.open`, the `navigate` - verb, `device.openLink`) goes through one rule: `file:`, `data:`, and - `javascript:` destinations and malformed URLs are `POLICY_DENIED`. There - is no origin or host allowlist; PR #290 removed them on purpose, since a + - Every navigation a test or the agent asks for (`app.open`, + `browser.goto`, the `navigate` verb) goes through one rule, + `resolveNavigationUrl`: only `http:`, `https:`, and the exact + `about:blank` pass; every other scheme (`view-source:file:` included) and + a malformed URL is `POLICY_DENIED` (a relative URL on a target with no + `app.url` is `APP_URL_REQUIRED`). `device.openLink` takes custom app + schemes, so it refuses a list instead (`packages/mobile/src/links.ts`): + `file:`, `data:`, `javascript:`, `view-source:`, `blob:`, `filesystem:`. + There is no origin or host allowlist; PR #290 removed them on purpose, since a click reaches any origin a typed URL could. - Sessions are per-run, target-bound, AES-256-GCM encrypted with a memory-only key, and deleted at cleanup; payloads never enter diagnostics. @@ -431,7 +445,9 @@ trees, on both platforms, without a device. sandbox with no secrets or write tokens. - CI: `.github/workflows/spec.yml` runs lint, typecheck, and the testbed on - Node 26 and `pnpm test` on Node 22, 24, and 26; `benchmark.yml` runs the + Node 26, `pnpm test` on the newest Node 22, 24, and 26 and on the + `engines.node` floors (22.22.3, 24.8.0), and `scripts/install-smoke.ts`, a + fresh install of the packed packages with pnpm 11 and 12; `benchmark.yml` runs the web benchmark's two suites; `mobile.yml` runs the mobile benchmark's on an iOS simulator and an Android emulator (KVM on x64 Linux). The two benchmark workflows gate on paths: a `changes` job (dorny/paths-filter @@ -508,7 +524,7 @@ trees, on both platforms, without a device. needs `node scripts/restore-peer-ranges.ts` after it, or `pnpm check` fails on the pin. - The runner publishes as the unscoped `e2e` (entry points `e2e`, `e2e/agent`, - `e2e/engine`, `e2e/oauth/chatgpt`, `e2e/oauth/copilot`, `e2e/oauth/grok`; the bin is `e2e` too); engines, reporters, and integrations publish public + `e2e/engine`, `e2e/oauth/chatgpt`, `e2e/oauth/copilot`, `e2e/oauth/grok`, `e2e/oauth/opencode-console`; the bin is `e2e` too); engines, reporters, and integrations publish public under the `@e2e-dev` scope. The `@e2edev` scope (moved to `@e2e-dev` on 2026-09-28), `@e2edev/e2e`, `@e2edev/oauth` (folded into `e2e/oauth` on 2026-09-21), and `@e2e-dev/integrations` (moved to `@e2e-dev/kernel` on diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 93827362c..07abd007d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -43,6 +43,7 @@ This project is a pnpm monorepo. `packages/` holds what publishes to npm, - `packages/e2e`: the published `e2e` package (SDK, runner, CLI, engine contract) - `packages/web`: the published `@e2e-dev/web` browser engine - `packages/mobile`: the published `@e2e-dev/mobile` mobile engine +- `packages/decision`: the decision-model executor on the public `StepExecutor` contract - `apps/testbed`: private dogfood suite that consumes the built packages - `apps/web-benchmark`: private Next.js app of hard-surface scenarios plus the e2e suites written against them - `apps/mobile-benchmark`: private Expo app of hard mobile surfaces plus the e2e suites written against them @@ -71,7 +72,7 @@ pnpm test # unit + integration Integration tests need Chromium: ```sh -pnpm --filter @e2e-dev/web exec playwright install chromium +pnpm --filter @e2e-dev/web exec playwright-core install chromium ``` Docs are part of the change, not a follow-up. A behavior change updates its @@ -117,8 +118,8 @@ We use [changesets](https://github.com/changesets/changesets) to publish new versions. It handles bumping the version based on semver, writing the changelog, and creating tags and GitHub releases. -If your change affects `e2e`, `@e2e-dev/web`, or -`@e2e-dev/mobile`, add a changeset: +If your change affects `e2e`, `@e2e-dev/web`, `@e2e-dev/mobile`, or +`@e2e-dev/decision`, add a changeset: ```sh pnpm changeset diff --git a/README.md b/README.md index 0bcb1f5a2..65bf73b19 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,7 @@ config and an example test. The | [`@e2e-dev/github`](https://www.npmjs.com/package/@e2e-dev/github) | Reporter that posts results as a pull request comment. | | [`@e2e-dev/kernel`](https://www.npmjs.com/package/@e2e-dev/kernel) | Kernel hosted browsers for the web engine. | | [`@e2e-dev/eas`](https://www.npmjs.com/package/@e2e-dev/eas) | EAS Simulators hosted iOS simulators and Android emulators for the mobile engine. | +| [`@e2e-dev/decision`](https://e2e.tester.army/docs/decision-models) | Decision-model executors for bounded semantic actions and assertions. | | [`@e2e-dev/testmu`](https://www.npmjs.com/package/@e2e-dev/testmu) | TestMu AI (formerly LambdaTest) hosted Android emulators, iOS simulators, and real devices for the mobile engine. | ## Documentation diff --git a/SECURITY.md b/SECURITY.md index 3126c7125..75bbae0c7 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -55,8 +55,11 @@ with a memory-only key, and deleted at cleanup. ## Navigation and origins -A test or the agent may open any http(s) URL; `file:`, `data:`, and -`javascript:` are denied. There is no origin allowlist, on navigation or on +On the web a test or the agent may open any http(s) URL, and `about:blank`; +every other scheme (`file:`, `data:`, `javascript:`, `view-source:`, `blob:`) +is denied. A device link may use an app's custom scheme, so +`device.openLink` refuses a list instead: `file:`, `data:`, `javascript:`, +`view-source:`, `blob:`, and `filesystem:`. There is no origin allowlist, on navigation or on secret fills. A click, a redirect, or a popup reaches another origin just as a typed URL would, so a gate on typed navigation guarded nothing; and a secret is only ever typed into a field the test itself handed to the step, a diff --git a/apps/testbed/app/pages/controls.mjs b/apps/testbed/app/pages/controls.mjs index c9da66864..47b178f37 100644 --- a/apps/testbed/app/pages/controls.mjs +++ b/apps/testbed/app/pages/controls.mjs @@ -219,6 +219,7 @@ const pages = { + `, }), diff --git a/apps/testbed/package.json b/apps/testbed/package.json index d1b7b889d..715e90198 100644 --- a/apps/testbed/package.json +++ b/apps/testbed/package.json @@ -24,7 +24,6 @@ "@ai-sdk/devtools": "1.0.22", "@types/node": "26.6.2", "ai": "7.0.107", - "playwright": "1.63.0", "typescript": "7.0.2", "zod": "4.6.1" } diff --git a/apps/testbed/tests/browser.e2e.ts b/apps/testbed/tests/browser.e2e.ts index 933419ba5..4e49391ab 100644 --- a/apps/testbed/tests/browser.e2e.ts +++ b/apps/testbed/tests/browser.e2e.ts @@ -33,6 +33,13 @@ test.describe('browser fixture', { requires: ['browser'], tags: ['browser'] }, ( await expect(screen.getByLabel('Loads')).toHaveText('loads: 2'); }); + test('an init script runs before the page reads it, from the next load on', async ({ screen, browser }) => { + await expect(screen.getByLabel('Random')).toHaveText('random: unseeded'); + await browser.addInitScript((value) => { Math.random = () => value; }, 0.5); + await browser.reload(); + await expect(screen.getByLabel('Random')).toHaveText('random: seeded'); + }); + test('the viewport size is what the page measures', async ({ screen, browser }) => { await browser.setViewport({ width: 500, height: 700 }); await expect(screen.getByLabel('Viewport')).toHaveText('500x700'); diff --git a/apps/web-benchmark/.e2e/cache/262851ad51db31a58d9bc17bea285445b7ce2b9a260883a89737a083c4dd8bd1.json b/apps/web-benchmark/.e2e/cache/02156375f0838e2b8dc5966f63159826d1953e0248c4bb1d49243a7fed208aba.json similarity index 64% rename from apps/web-benchmark/.e2e/cache/262851ad51db31a58d9bc17bea285445b7ce2b9a260883a89737a083c4dd8bd1.json rename to apps/web-benchmark/.e2e/cache/02156375f0838e2b8dc5966f63159826d1953e0248c4bb1d49243a7fed208aba.json index 45f244540..f2b38f32d 100644 --- a/apps/web-benchmark/.e2e/cache/262851ad51db31a58d9bc17bea285445b7ce2b9a260883a89737a083c4dd8bd1.json +++ b/apps/web-benchmark/.e2e/cache/02156375f0838e2b8dc5966f63159826d1953e0248c4bb1d49243a7fed208aba.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-24T19:35:15.679Z", + "createdAt": "2026-10-04T08:29:27.731Z", "payload": { "actions": [ { @@ -24,9 +24,12 @@ "recordedFor": { "testId": "tests-agent/control-inventory.e2e.ts::control%20inventory::back%20returns%20from%20the%20details%20view%20to%20the%20inventory", "targetId": "web", - "instructionDigest": "b654f62f1b06969b737e6bc5900524da39b012965a0d82aef45fce7af099ba6c" + "instructionDigest": "b654f62f1b06969b737e6bc5900524da39b012965a0d82aef45fce7af099ba6c", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Opened the Details view, then used browser history to return to the inventory. The inventory is back on screen and Navigation state reads “back on the inventory.”.", + "summary": "Opened Details, then used browser history to return to /e/control-inventory. The Navigation state now reads “back on the inventory.”", "startPath": "/e/control-inventory", "endPath": "/e/control-inventory", "endAnchors": [ @@ -36,6 +39,13 @@ "text": "back on the inventory" } ], - "endWaitMs": 12026 + "goneAnchors": [ + { + "role": "status", + "name": "Navigation state", + "text": "not opened" + } + ], + "endWaitMs": 12570 } } diff --git a/apps/web-benchmark/.e2e/cache/5637caec136ab6deac26176da39d3dffa164a00e43f1576cc62708becbfad6e3.json b/apps/web-benchmark/.e2e/cache/037d5252e59b546953ccf4a4baf4623fb4ba6df76f87bcd3a6a99f25ab88b060.json similarity index 57% rename from apps/web-benchmark/.e2e/cache/5637caec136ab6deac26176da39d3dffa164a00e43f1576cc62708becbfad6e3.json rename to apps/web-benchmark/.e2e/cache/037d5252e59b546953ccf4a4baf4623fb4ba6df76f87bcd3a6a99f25ab88b060.json index 7ea51f1b5..620707c1a 100644 --- a/apps/web-benchmark/.e2e/cache/5637caec136ab6deac26176da39d3dffa164a00e43f1576cc62708becbfad6e3.json +++ b/apps/web-benchmark/.e2e/cache/037d5252e59b546953ccf4a4baf4623fb4ba6df76f87bcd3a6a99f25ab88b060.json @@ -1,17 +1,16 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:18.310Z", + "createdAt": "2026-10-04T08:31:18.423Z", "payload": { "actions": [ { - "name": "scroll", - "summary": "scroll down on node \"scroll-container\"", - "direction": "down", + "name": "scrollTo", + "summary": "scroll button \"Accept terms\" into view", "target": { - "testId": "scroll-container" - }, - "times": 10, - "spans": 0.3083 + "role": "button", + "name": "Accept terms", + "testId": "accept-button" + } }, { "name": "tap", @@ -30,9 +29,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20sticky-chrome", "targetId": "web", - "instructionDigest": "507883318a8655d47afe9f680ee91cde1ae64055fbef530129cc289101a30c23" + "instructionDigest": "507883318a8655d47afe9f680ee91cde1ae64055fbef530129cc289101a30c23", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Scrolled the terms container until the Accept terms button was clear of the sticky footer, then clicked it. The page now shows “Terms accepted.”", + "summary": "Scrolled until the Accept terms button was clear above the sticky footer, clicked it once, and verified the “Terms accepted” confirmation.", "startPath": "/e/sticky-chrome", "endPath": "/e/sticky-chrome", "endAnchors": [ @@ -41,6 +43,6 @@ "testId": "success-message" } ], - "endWaitMs": 10124 + "endWaitMs": 11764 } } diff --git a/apps/web-benchmark/.e2e/cache/e231cfcb187fa226df103f2b97077e50581951814460bfcf67de3ec631840552.json b/apps/web-benchmark/.e2e/cache/0fcdfb6a36534c4f3699d73501494c9322056c4b915e8c6566f26daa1b3b5f2b.json similarity index 66% rename from apps/web-benchmark/.e2e/cache/e231cfcb187fa226df103f2b97077e50581951814460bfcf67de3ec631840552.json rename to apps/web-benchmark/.e2e/cache/0fcdfb6a36534c4f3699d73501494c9322056c4b915e8c6566f26daa1b3b5f2b.json index c05c47bd1..5227c66f6 100644 --- a/apps/web-benchmark/.e2e/cache/e231cfcb187fa226df103f2b97077e50581951814460bfcf67de3ec631840552.json +++ b/apps/web-benchmark/.e2e/cache/0fcdfb6a36534c4f3699d73501494c9322056c4b915e8c6566f26daa1b3b5f2b.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:32.475Z", + "createdAt": "2026-10-04T08:32:21.685Z", "payload": { "actions": [ { @@ -71,9 +71,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20otp-auto-advance", "targetId": "web", - "instructionDigest": "c878aa968fdbc93d377500f8aecb2f3b0ab701c37b7f9757f8d993f0fc87e109" + "instructionDigest": "c878aa968fdbc93d377500f8aecb2f3b0ab701c37b7f9757f8d993f0fc87e109", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Entered the displayed code 493027 into the six segmented inputs. The page now shows “Code verified.”", + "summary": "Entered the displayed code 493027 into the six segmented inputs. The page now displays “Code verified.”", "startPath": "/e/otp-auto-advance", "endPath": "/e/otp-auto-advance", "endAnchors": [ @@ -82,6 +85,41 @@ "testId": "success-message" } ], - "endWaitMs": 10521 + "goneAnchors": [ + { + "text": "Enter code 493027." + }, + { + "role": "textbox", + "testId": "otp-input-0", + "inputPurpose": "one-time-code" + }, + { + "role": "textbox", + "testId": "otp-input-1", + "inputPurpose": "one-time-code" + }, + { + "role": "textbox", + "testId": "otp-input-2", + "inputPurpose": "one-time-code" + }, + { + "role": "textbox", + "testId": "otp-input-3", + "inputPurpose": "one-time-code" + }, + { + "role": "textbox", + "testId": "otp-input-4", + "inputPurpose": "one-time-code" + }, + { + "role": "textbox", + "testId": "otp-input-5", + "inputPurpose": "one-time-code" + } + ], + "endWaitMs": 14371 } } diff --git a/apps/web-benchmark/.e2e/cache/16433aa0d367e6f0dfe2857d4621edd499a5f4343337c8803c316fde3eb9da45.json b/apps/web-benchmark/.e2e/cache/16433aa0d367e6f0dfe2857d4621edd499a5f4343337c8803c316fde3eb9da45.json deleted file mode 100644 index 667af6966..000000000 --- a/apps/web-benchmark/.e2e/cache/16433aa0d367e6f0dfe2857d4621edd499a5f4343337c8803c316fde3eb9da45.json +++ /dev/null @@ -1,154 +0,0 @@ -{ - "schemaVersion": "trace-1", - "createdAt": "2026-09-25T08:02:14.210Z", - "payload": { - "actions": [ - { - "name": "type", - "summary": "type \"release approved\" into textbox \"editor\"", - "target": { - "role": "textbox", - "testId": "editor" - }, - "value": "release approved" - }, - { - "name": "press", - "summary": "press \"Control+Shift+ArrowLeft\" on textbox \"release approved\"", - "target": { - "role": "textbox", - "text": "release approved", - "testId": "editor" - }, - "key": "Control+Shift+ArrowLeft" - }, - { - "name": "press", - "summary": "press \"ArrowRight\" on textbox \"release approved\"", - "target": { - "role": "textbox", - "text": "release approved", - "testId": "editor" - }, - "key": "ArrowRight" - }, - { - "name": "press", - "summary": "press \"Shift+ArrowLeft\" on textbox \"release approved\"", - "target": { - "role": "textbox", - "text": "release approved", - "testId": "editor" - }, - "key": "Shift+ArrowLeft" - }, - { - "name": "press", - "summary": "press \"Shift+ArrowLeft\" on textbox \"release approved\"", - "target": { - "role": "textbox", - "text": "release approved", - "testId": "editor" - }, - "key": "Shift+ArrowLeft" - }, - { - "name": "press", - "summary": "press \"Shift+ArrowLeft\" on textbox \"release approved\"", - "target": { - "role": "textbox", - "text": "release approved", - "testId": "editor" - }, - "key": "Shift+ArrowLeft" - }, - { - "name": "press", - "summary": "press \"Shift+ArrowLeft\" on textbox \"release approved\"", - "target": { - "role": "textbox", - "text": "release approved", - "testId": "editor" - }, - "key": "Shift+ArrowLeft" - }, - { - "name": "press", - "summary": "press \"Shift+ArrowLeft\" on textbox \"release approved\"", - "target": { - "role": "textbox", - "text": "release approved", - "testId": "editor" - }, - "key": "Shift+ArrowLeft" - }, - { - "name": "press", - "summary": "press \"Shift+ArrowLeft\" on textbox \"release approved\"", - "target": { - "role": "textbox", - "text": "release approved", - "testId": "editor" - }, - "key": "Shift+ArrowLeft" - }, - { - "name": "press", - "summary": "press \"Shift+ArrowLeft\" on textbox \"release approved\"", - "target": { - "role": "textbox", - "text": "release approved", - "testId": "editor" - }, - "key": "Shift+ArrowLeft" - }, - { - "name": "press", - "summary": "press \"Shift+ArrowLeft\" on textbox \"release approved\"", - "target": { - "role": "textbox", - "text": "release approved", - "testId": "editor" - }, - "key": "Shift+ArrowLeft" - }, - { - "name": "tap", - "summary": "tap button \"B\"", - "target": { - "role": "button", - "name": "B", - "testId": "bold-button" - } - }, - { - "name": "tap", - "summary": "tap button \"Check document\"", - "target": { - "role": "button", - "name": "Check document", - "testId": "check-button" - } - } - ], - "executor": { - "name": "e2e-default-agent", - "version": "2" - }, - "recordedFor": { - "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20rich-text-editor", - "targetId": "web", - "instructionDigest": "e37ad9ac878deebe5c840d23fded9589f67228d8229a059b855347ab5e49ab96" - }, - "summary": "Entered “release approved,” selected exactly “approved,” applied bold, and checked the document. The page displays “Document approved.”.", - "startPath": "/e/rich-text-editor", - "endPath": "/e/rich-text-editor", - "endAnchors": [ - { - "text": "Document approved", - "testId": "success-message" - } - ], - "endWaitMs": 12292 - } -} diff --git a/apps/web-benchmark/.e2e/cache/8e069e91bdee5edf7f8e6ce5e2880fdb02a4f16e2f0056d3d9a1f40d507ea0f6.json b/apps/web-benchmark/.e2e/cache/1b0794f168443c714d5e4b0485f8e2a59ee9da64f952568313561faba74e666f.json similarity index 73% rename from apps/web-benchmark/.e2e/cache/8e069e91bdee5edf7f8e6ce5e2880fdb02a4f16e2f0056d3d9a1f40d507ea0f6.json rename to apps/web-benchmark/.e2e/cache/1b0794f168443c714d5e4b0485f8e2a59ee9da64f952568313561faba74e666f.json index 52673baf6..35fda1369 100644 --- a/apps/web-benchmark/.e2e/cache/8e069e91bdee5edf7f8e6ce5e2880fdb02a4f16e2f0056d3d9a1f40d507ea0f6.json +++ b/apps/web-benchmark/.e2e/cache/1b0794f168443c714d5e4b0485f8e2a59ee9da64f952568313561faba74e666f.json @@ -1,16 +1,16 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:08.490Z", + "createdAt": "2026-10-04T08:29:16.739Z", "payload": { "actions": [ { - "name": "scroll", - "summary": "scroll down on node \"feed\"", + "name": "scrollUntil", + "summary": "scroll down on node \"feed\" until \"Golden Ticket\" shows (20 screens)", + "text": "Golden Ticket", "direction": "down", "target": { "testId": "feed" }, - "times": 25, "spans": 0.3083 }, { @@ -30,9 +30,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20infinite-scroll", "targetId": "web", - "instructionDigest": "5693e0682199f2201375a207ac10bd69b22c26925bbf8fd87e06a9f6d1abeca9" + "instructionDigest": "5693e0682199f2201375a207ac10bd69b22c26925bbf8fd87e06a9f6d1abeca9", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Scrolled the feed until the Golden Ticket appeared at entry 138, clicked Claim, and verified the success message “Golden Ticket claimed.”", + "summary": "Scrolled the feed until the Golden Ticket appeared and clicked Claim. The page now shows “Golden Ticket claimed.”", "startPath": "/e/infinite-scroll", "endPath": "/e/infinite-scroll", "endAnchors": [ @@ -62,6 +65,6 @@ "text": "Item 44" } ], - "endWaitMs": 10404 + "endWaitMs": 11656 } } diff --git a/apps/web-benchmark/.e2e/cache/c1fb51f3e63a172196dc4f237a90044582c6ede099f5cb0e8b906a83281381a6.json b/apps/web-benchmark/.e2e/cache/3f1a446b8087ce7235406cdc2d010e8a6f2dcbb3240d0dc75d18f4470055b959.json similarity index 78% rename from apps/web-benchmark/.e2e/cache/c1fb51f3e63a172196dc4f237a90044582c6ede099f5cb0e8b906a83281381a6.json rename to apps/web-benchmark/.e2e/cache/3f1a446b8087ce7235406cdc2d010e8a6f2dcbb3240d0dc75d18f4470055b959.json index a28aea152..a476121a9 100644 --- a/apps/web-benchmark/.e2e/cache/c1fb51f3e63a172196dc4f237a90044582c6ede099f5cb0e8b906a83281381a6.json +++ b/apps/web-benchmark/.e2e/cache/3f1a446b8087ce7235406cdc2d010e8a6f2dcbb3240d0dc75d18f4470055b959.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T18:03:17.359Z", + "createdAt": "2026-10-04T08:31:50.082Z", "payload": { "actions": [ { @@ -42,9 +42,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20kanban-board", "targetId": "web", - "instructionDigest": "79f28df979547ae9abe14bba1c341873d1aa9d74dccc07d1e76bf46c279b8cb5" + "instructionDigest": "79f28df979547ae9abe14bba1c341873d1aa9d74dccc07d1e76bf46c279b8cb5", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Moved Fix payment bug to Done and Refactor onboarding to In Progress, then submitted the board. The page confirmed “Board matches the goal.”.", + "summary": "Moved Fix payment bug to Done and Refactor onboarding to In Progress, then submitted. The page confirmed “Board matches the goal.”", "startPath": "/e/kanban-board", "endPath": "/e/kanban-board", "endAnchors": [ @@ -53,6 +56,11 @@ "testId": "success-message" } ], - "endWaitMs": 12168 + "goneAnchors": [ + { + "text": "Empty" + } + ], + "endWaitMs": 11433 } } diff --git a/apps/web-benchmark/.e2e/cache/40c9afad8973912eaf9f2b75c5385d5ea7c9c6ec2a1b873685e9f111f6bb5189.json b/apps/web-benchmark/.e2e/cache/450275e0bc6a1e6c485db03cde0314244a5250f6ee8afab5163db3a2117c1057.json similarity index 53% rename from apps/web-benchmark/.e2e/cache/40c9afad8973912eaf9f2b75c5385d5ea7c9c6ec2a1b873685e9f111f6bb5189.json rename to apps/web-benchmark/.e2e/cache/450275e0bc6a1e6c485db03cde0314244a5250f6ee8afab5163db3a2117c1057.json index c82a2c4b7..b50a64a2d 100644 --- a/apps/web-benchmark/.e2e/cache/40c9afad8973912eaf9f2b75c5385d5ea7c9c6ec2a1b873685e9f111f6bb5189.json +++ b/apps/web-benchmark/.e2e/cache/450275e0bc6a1e6c485db03cde0314244a5250f6ee8afab5163db3a2117c1057.json @@ -1,11 +1,11 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-24T23:04:31.378Z", + "createdAt": "2026-10-04T08:31:06.435Z", "payload": { "actions": [ { "name": "scrollUntil", - "summary": "scroll down on node \"table-viewport\" until \"Row 4322\" shows (432 screens)", + "summary": "scroll down on node \"table-viewport\" until \"Row 4322\" shows (431 screens)", "text": "Row 4322", "direction": "down", "target": { @@ -30,9 +30,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20virtualized-table", "targetId": "web", - "instructionDigest": "3e0bdeec3c893eadd972848c89ef841ec7722fd896ed12998ef945cf20af30ec" + "instructionDigest": "3e0bdeec3c893eadd972848c89ef841ec7722fd896ed12998ef945cf20af30ec", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "The current screen shows “Golden Row claimed” while Row 4322 — Golden Row is rendered in the table, confirming the claim succeeded.", + "summary": "Scrolled the windowed table until Row 4322 — Golden Row was rendered, clicked Claim, and verified the “Golden Row claimed” success message.", "startPath": "/e/virtualized-table", "endPath": "/e/virtualized-table", "endAnchors": [ @@ -41,27 +44,53 @@ "testId": "success-message" }, { - "text": "Row 4316" + "text": "Row 4306" }, { - "text": "Row 4317" + "text": "Row 4307" }, { - "text": "Row 4318" + "text": "Row 4308" }, { - "text": "Row 4319" + "text": "Row 4309" }, { - "text": "Row 4320" + "text": "Row 4310" }, { - "text": "Row 4321" + "text": "Row 4311" }, { - "text": "Row 4322 — Golden Row" + "text": "Row 4312" } ], - "endWaitMs": 27632 + "goneAnchors": [ + { + "text": "Row 1" + }, + { + "text": "Row 2" + }, + { + "text": "Row 3" + }, + { + "text": "Row 4" + }, + { + "text": "Row 5" + }, + { + "text": "Row 6" + }, + { + "text": "Row 7" + }, + { + "text": "Row 8" + } + ], + "endWaitMs": 11397 } } diff --git a/apps/web-benchmark/.e2e/cache/46163014c3c423f814e3fa6c2a8864466b009e647bda83569d8627166664be9f.json b/apps/web-benchmark/.e2e/cache/46163014c3c423f814e3fa6c2a8864466b009e647bda83569d8627166664be9f.json deleted file mode 100644 index 4db6668e1..000000000 --- a/apps/web-benchmark/.e2e/cache/46163014c3c423f814e3fa6c2a8864466b009e647bda83569d8627166664be9f.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:29:45.906Z", - "payload": { - "actions": [ - { - "name": "tool", - "summary": "tool type (run-time value)" - }, - { - "name": "tap", - "summary": "tap button \"Apply\"", - "target": { - "role": "button", - "name": "Apply" - } - } - ], - "executor": { - "name": "e2e-default-agent", - "version": "2" - }, - "recordedFor": { - "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20iframe-form", - "targetId": "web", - "instructionDigest": "5ea16155d5cd3ee4fc67e7511534964bca91116fa0412e4523428844f9f85c15" - }, - "summary": "Read coupon code SAVE-1234 from the first frame, entered it in the checkout form in the other frame, and applied it. The page displays “Coupon applied.”.", - "startPath": "/e/iframe-form", - "endPath": "/e/iframe-form", - "endAnchors": [ - { - "text": "Coupon applied", - "testId": "success-message" - } - ], - "endWaitMs": 11883 - } -} diff --git a/apps/web-benchmark/.e2e/cache/629d339911fd84a9271781ed96bc9ae181f3e982f72e337d53b7dc1c9b6bd762.json b/apps/web-benchmark/.e2e/cache/4a841f63d914134982d6ad47d25a351c3f65b1642d57d569c7fc14f631d5ff7f.json similarity index 66% rename from apps/web-benchmark/.e2e/cache/629d339911fd84a9271781ed96bc9ae181f3e982f72e337d53b7dc1c9b6bd762.json rename to apps/web-benchmark/.e2e/cache/4a841f63d914134982d6ad47d25a351c3f65b1642d57d569c7fc14f631d5ff7f.json index 0bda864f4..59bd56e68 100644 --- a/apps/web-benchmark/.e2e/cache/629d339911fd84a9271781ed96bc9ae181f3e982f72e337d53b7dc1c9b6bd762.json +++ b/apps/web-benchmark/.e2e/cache/4a841f63d914134982d6ad47d25a351c3f65b1642d57d569c7fc14f631d5ff7f.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-24T19:34:54.319Z", + "createdAt": "2026-10-04T08:29:03.565Z", "payload": { "actions": [ { @@ -29,9 +29,12 @@ "recordedFor": { "testId": "tests-agent/control-inventory.e2e.ts::control%20inventory::a%20long%20press%20and%20a%20double%20click%20are%20chosen%20over%20a%20plain%20tap", "targetId": "web", - "instructionDigest": "9fe8b7fcfeb1bd36e8d003d5c7e5feb16042211ec61970b63314a6c7bac345b0" + "instructionDigest": "9fe8b7fcfeb1bd36e8d003d5c7e5feb16042211ec61970b63314a6c7bac345b0", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Long-pressed the Hold me button; its state reads “long-pressed.” Double-clicked the Double-click me button; its state reads “double-clicked.”", + "summary": "Long-pressed Hold me; Hold state reads “long-pressed.” Double-clicked Double-click me; Double-click state reads “double-clicked.”", "startPath": "/e/control-inventory", "endPath": "/e/control-inventory", "endAnchors": [ @@ -46,6 +49,18 @@ "text": "double-clicked" } ], - "endWaitMs": 11922 + "goneAnchors": [ + { + "role": "status", + "name": "Hold state", + "text": "none" + }, + { + "role": "status", + "name": "Double-click state", + "text": "none" + } + ], + "endWaitMs": 11370 } } diff --git a/apps/web-benchmark/.e2e/cache/ee64bc2d2dacd6b5bdd5537e6e0c4b4f3f1bb916d023d5d712627cbce856e9da.json b/apps/web-benchmark/.e2e/cache/53b9612ca2090821af954d177fec1d1172095c54ae418c6169723a6f7a8b77b9.json similarity index 63% rename from apps/web-benchmark/.e2e/cache/ee64bc2d2dacd6b5bdd5537e6e0c4b4f3f1bb916d023d5d712627cbce856e9da.json rename to apps/web-benchmark/.e2e/cache/53b9612ca2090821af954d177fec1d1172095c54ae418c6169723a6f7a8b77b9.json index 2a993ff23..158ebb0a4 100644 --- a/apps/web-benchmark/.e2e/cache/ee64bc2d2dacd6b5bdd5537e6e0c4b4f3f1bb916d023d5d712627cbce856e9da.json +++ b/apps/web-benchmark/.e2e/cache/53b9612ca2090821af954d177fec1d1172095c54ae418c6169723a6f7a8b77b9.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:26.359Z", + "createdAt": "2026-10-04T08:31:27.021Z", "payload": { "actions": [ { @@ -14,7 +14,8 @@ }, { "name": "tool", - "summary": "tool type (run-time value)" + "summary": "tool type (run-time value)", + "derived": "minted-token" }, { "name": "tap", @@ -33,9 +34,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20async-states", "targetId": "web", - "instructionDigest": "3d8730745312e1ce478b253dcad7910775eb76246846228b0ca02a78e514aa88" + "instructionDigest": "3d8730745312e1ce478b253dcad7910775eb76246846228b0ca02a78e514aa88", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Read verification code 4407 from the toast, entered it, and submitted Verify. The page displays “Code verified successfully.”.", + "summary": "Waited for the page to load, sent the code, read 4407 from the toast, entered it, and submitted once Verify was enabled. The page shows “Code verified successfully.”", "startPath": "/e/async-states", "endPath": "/e/async-states", "endAnchors": [ @@ -44,6 +48,11 @@ "testId": "success-message" } ], - "endWaitMs": 12049 + "goneAnchors": [ + { + "testId": "initial-loading" + } + ], + "endWaitMs": 11512 } } diff --git a/apps/web-benchmark/.e2e/cache/38af005a1f14beeb5db725a88da8119011579d5050839d7a468712c1516ac7ad.json b/apps/web-benchmark/.e2e/cache/587d13f7c2fb5d376fa433ee583f2f6f88f6bdfe0e72c25ebc0390fb44e93c83.json similarity index 54% rename from apps/web-benchmark/.e2e/cache/38af005a1f14beeb5db725a88da8119011579d5050839d7a468712c1516ac7ad.json rename to apps/web-benchmark/.e2e/cache/587d13f7c2fb5d376fa433ee583f2f6f88f6bdfe0e72c25ebc0390fb44e93c83.json index d45b8aa50..b36eda2fc 100644 --- a/apps/web-benchmark/.e2e/cache/38af005a1f14beeb5db725a88da8119011579d5050839d7a468712c1516ac7ad.json +++ b/apps/web-benchmark/.e2e/cache/587d13f7c2fb5d376fa433ee583f2f6f88f6bdfe0e72c25ebc0390fb44e93c83.json @@ -1,8 +1,17 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-26T20:48:22.536Z", + "createdAt": "2026-10-04T08:29:22.846Z", "payload": { "actions": [ + { + "name": "tap", + "summary": "tap button \"Attachments\" in \"Files\"", + "target": { + "role": "button", + "name": "Attachments", + "within": "Files" + } + }, { "name": "upload", "summary": "upload \"fixtures/attachment.txt\", \"fixtures/second.txt\" to button \"Attachments\" in \"Files\"", @@ -24,9 +33,12 @@ "recordedFor": { "testId": "tests-agent/control-inventory.e2e.ts::control%20inventory::upload%20attaches%20the%20named%20project%20files", "targetId": "web", - "instructionDigest": "6eb8c4cb34d6623811808727d85db0d55946be5724f0cdc01b3c633db3d6f44d" + "instructionDigest": "6eb8c4cb34d6623811808727d85db0d55946be5724f0cdc01b3c633db3d6f44d", + "paramsDigest": "38fc51716f1b76431a49c2fd70d8a6d11405fa9b9451a1d96cb9e357e70321ce", + "callIndex": 0, + "agent": "default" }, - "summary": "Attached both fixtures/attachment.txt and fixtures/second.txt. The Attachments state now lists attachment.txt and second.txt.", + "summary": "Attached both fixtures/attachment.txt and fixtures/second.txt to the Attachments input. The state now shows “attachment.txt, second.txt” and the control indicates 2 files.", "startPath": "/e/control-inventory", "endPath": "/e/control-inventory", "endAnchors": [ @@ -34,8 +46,24 @@ "role": "status", "name": "Attachments state", "text": "attachment.txt, second.txt" + }, + { + "role": "button", + "name": "Attachments", + "value": "C:\\fakepath\\attachment.txt" + } + ], + "goneAnchors": [ + { + "role": "status", + "name": "Attachments state", + "text": "none" + }, + { + "role": "button", + "name": "Attachments" } ], - "endWaitMs": 11392 + "endWaitMs": 12311 } } diff --git a/apps/web-benchmark/.e2e/cache/61711c2e97b6ce5f9a43edaeae2c13c12f9c408e36b18006f43840ac6beecf8c.json b/apps/web-benchmark/.e2e/cache/61711c2e97b6ce5f9a43edaeae2c13c12f9c408e36b18006f43840ac6beecf8c.json new file mode 100644 index 000000000..5499066ea --- /dev/null +++ b/apps/web-benchmark/.e2e/cache/61711c2e97b6ce5f9a43edaeae2c13c12f9c408e36b18006f43840ac6beecf8c.json @@ -0,0 +1,74 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-10-04T08:29:07.236Z", + "payload": { + "actions": [ + { + "name": "tool", + "summary": "tool type (run-time value)", + "derived": "whole-node" + }, + { + "name": "tap", + "summary": "tap button \"Apply\"", + "target": { + "role": "button", + "name": "Apply" + } + } + ], + "executor": { + "name": "e2e-default-agent", + "version": "2" + }, + "recordedFor": { + "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20iframe-form", + "targetId": "web", + "instructionDigest": "5ea16155d5cd3ee4fc67e7511534964bca91116fa0412e4523428844f9f85c15", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" + }, + "summary": "Read coupon SAVE-1234 from the first frame, entered it in the checkout form, and applied it. The page now shows “Coupon applied.”", + "startPath": "/e/iframe-form", + "endPath": "/e/iframe-form", + "endAnchors": [ + { + "text": "Coupon applied", + "testId": "success-message" + } + ], + "goneAnchors": [ + { + "text": "Read the coupon code from the first frame and apply it in the second." + }, + { + "text": "SAVE-1234" + }, + { + "text": "Apply your coupon at checkout" + }, + { + "role": "textbox", + "name": "Coupon code", + "placeholder": "Coupon code" + }, + { + "role": "button", + "name": "Apply" + }, + { + "role": "iframe", + "name": "coupon-frame" + }, + { + "text": "Your coupon code:" + }, + { + "role": "iframe", + "name": "checkout-frame" + } + ], + "endWaitMs": 11742 + } +} diff --git a/apps/web-benchmark/.e2e/cache/301ca75e59525df56d8582a43689dc7a049b9d1b1642edf8c0b3fa749adfa888.json b/apps/web-benchmark/.e2e/cache/6faed48a7591e228ac3b7f8109a44e3e0fa04d3aef50dbb7156d7591586efbf6.json similarity index 71% rename from apps/web-benchmark/.e2e/cache/301ca75e59525df56d8582a43689dc7a049b9d1b1642edf8c0b3fa749adfa888.json rename to apps/web-benchmark/.e2e/cache/6faed48a7591e228ac3b7f8109a44e3e0fa04d3aef50dbb7156d7591586efbf6.json index f95a9e515..76e5448f7 100644 --- a/apps/web-benchmark/.e2e/cache/301ca75e59525df56d8582a43689dc7a049b9d1b1642edf8c0b3fa749adfa888.json +++ b/apps/web-benchmark/.e2e/cache/6faed48a7591e228ac3b7f8109a44e3e0fa04d3aef50dbb7156d7591586efbf6.json @@ -1,17 +1,8 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:36.184Z", + "createdAt": "2026-10-04T08:33:55.168Z", "payload": { "actions": [ - { - "name": "tap", - "summary": "tap button \"Accept\"", - "target": { - "role": "button", - "name": "Accept", - "testId": "accept-cookies" - } - }, { "name": "tap", "summary": "tap button \"Buy now\" in \"Wireless Headphones\"", @@ -30,9 +21,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20promo-storefront", "targetId": "web", - "instructionDigest": "1bf3aabb828b9b084d447df3a78d1fa700b5b28b8d5847dcc5ffcb9b0ce3bd45" + "instructionDigest": "1bf3aabb828b9b084d447df3a78d1fa700b5b28b8d5847dcc5ffcb9b0ce3bd45", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "The Wireless Headphones purchase is confirmed; the page displays “Order confirmed” for that product.", + "summary": "Bought the Wireless Headphones. The page now shows “Order confirmed.”", "startPath": "/e/promo-storefront", "endPath": "/e/promo-storefront", "endAnchors": [ @@ -58,6 +52,13 @@ "text": "★★★★ Comfortable fit, pairing was instant." } ], - "endWaitMs": 11175 + "goneAnchors": [ + { + "role": "listitem", + "name": "Wireless Headphones $80.00 $40.00 50% OFF Buy now", + "testId": "deal-headphones" + } + ], + "endWaitMs": 11574 } } diff --git a/apps/web-benchmark/.e2e/cache/76191b9464eff355a227e8732cd2d41e51e5c5ecdfc77e19d56355447de218a5.json b/apps/web-benchmark/.e2e/cache/76191b9464eff355a227e8732cd2d41e51e5c5ecdfc77e19d56355447de218a5.json new file mode 100644 index 000000000..3a3b3cefa --- /dev/null +++ b/apps/web-benchmark/.e2e/cache/76191b9464eff355a227e8732cd2d41e51e5c5ecdfc77e19d56355447de218a5.json @@ -0,0 +1,69 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-10-04T08:28:59.224Z", + "payload": { + "actions": [ + { + "name": "tap", + "summary": "tap radio \"Express\" in \"Delivery\"", + "target": { + "role": "radio", + "name": "Express", + "within": "Delivery" + } + } + ], + "executor": { + "name": "e2e-default-agent", + "version": "2" + }, + "recordedFor": { + "testId": "tests-agent/control-inventory.e2e.ts::control%20inventory::check%20picks%20a%20radio%20the%20pick%20replaces%20with%20its%20summary", + "targetId": "web", + "instructionDigest": "df853ab2fc826564b99c9b044a65fd7ee21cdc2b487273cebcd32bcbe3607c66", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" + }, + "summary": "Selected Express delivery; the delivery state now reads “Express” and the page confirms “Express delivery selected.”", + "startPath": "/e/control-inventory", + "endPath": "/e/control-inventory", + "endAnchors": [ + { + "role": "status", + "name": "Delivery state", + "text": "Express" + }, + { + "text": "Express delivery selected" + }, + { + "role": "button", + "name": "Change delivery" + } + ], + "goneAnchors": [ + { + "role": "status", + "name": "Delivery state", + "text": "unset" + }, + { + "role": "radio", + "name": "Standard" + }, + { + "role": "radio", + "name": "Express" + }, + { + "role": "group", + "name": "Delivery" + }, + { + "text": "Standard" + } + ], + "endWaitMs": 11721 + } +} diff --git a/apps/web-benchmark/.e2e/cache/d08d3b277fb7b7831a72e44bbe3d17130dd378a936362f99f6011529ef7bb36b.json b/apps/web-benchmark/.e2e/cache/7f48111a643a86547628175b52e0493a95ab67cc850db1ab21bb6baa69670977.json similarity index 63% rename from apps/web-benchmark/.e2e/cache/d08d3b277fb7b7831a72e44bbe3d17130dd378a936362f99f6011529ef7bb36b.json rename to apps/web-benchmark/.e2e/cache/7f48111a643a86547628175b52e0493a95ab67cc850db1ab21bb6baa69670977.json index 70a80dff5..e9db5ac8c 100644 --- a/apps/web-benchmark/.e2e/cache/d08d3b277fb7b7831a72e44bbe3d17130dd378a936362f99f6011529ef7bb36b.json +++ b/apps/web-benchmark/.e2e/cache/7f48111a643a86547628175b52e0493a95ab67cc850db1ab21bb6baa69670977.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:29:31.881Z", + "createdAt": "2026-10-04T08:28:57.613Z", "payload": { "actions": [ { @@ -43,9 +43,12 @@ "recordedFor": { "testId": "tests-agent/login-form.e2e.ts::act%20signs%20in%20with%20the%20benchmark%20account", "targetId": "web", - "instructionDigest": "c26456300e1d572e5598b0d8d2584171681d80d8a20bef86bdda6e9d2c7f976e" + "instructionDigest": "c26456300e1d572e5598b0d8d2584171681d80d8a20bef86bdda6e9d2c7f976e", + "paramsDigest": "10cf8df51d5b30096b97501956aa95d056885035f7bc67d093d076b7c4caecf0", + "callIndex": 0, + "agent": "default" }, - "summary": "Logged in with tester@tester.army and the benchmark password; the page shows “Logged in successfully” and a Log out button.", + "summary": "Logged in with tester@tester.army and the supplied password. The page shows “Logged in successfully” and a Log out button.", "startPath": "/e/login-form", "endPath": "/e/login-form", "endAnchors": [ @@ -59,6 +62,29 @@ "testId": "logout-button" } ], - "endWaitMs": 10103 + "goneAnchors": [ + { + "text": "Use tester@tester.army / " + }, + { + "role": "textbox", + "name": "Email", + "testId": "email-input", + "placeholder": "Email" + }, + { + "role": "textbox", + "name": "Password", + "testId": "password-input", + "placeholder": "Password", + "inputPurpose": "password" + }, + { + "role": "button", + "name": "Log in", + "testId": "login-button" + } + ], + "endWaitMs": 11548 } } diff --git a/apps/web-benchmark/.e2e/cache/95eebf3ab01eabdd09afe6bf0e31216ab141c487cb267a6bd1c2c941de7c9e88.json b/apps/web-benchmark/.e2e/cache/95eebf3ab01eabdd09afe6bf0e31216ab141c487cb267a6bd1c2c941de7c9e88.json new file mode 100644 index 000000000..57ccdd923 --- /dev/null +++ b/apps/web-benchmark/.e2e/cache/95eebf3ab01eabdd09afe6bf0e31216ab141c487cb267a6bd1c2c941de7c9e88.json @@ -0,0 +1,328 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-10-04T08:33:45.030Z", + "payload": { + "actions": [ + { + "name": "type", + "summary": "type \"release approved\" into textbox \"editor\"", + "target": { + "role": "textbox", + "testId": "editor" + }, + "value": "release approved" + }, + { + "name": "press", + "summary": "press \"Control+Shift+ArrowLeft\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "Control+Shift+ArrowLeft" + }, + { + "name": "press", + "summary": "press \"ArrowRight\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "ArrowRight" + }, + { + "name": "press", + "summary": "press \"Control+Shift+ArrowLeft\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "Control+Shift+ArrowLeft" + }, + { + "name": "tap", + "summary": "tap textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + } + }, + { + "name": "press", + "summary": "press \"Shift+End\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "Shift+End" + }, + { + "name": "press", + "summary": "press \"Shift+ArrowRight\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "tap", + "summary": "tap textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + } + }, + { + "name": "press", + "summary": "press \"Control+a\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "Control+a" + }, + { + "name": "tap", + "summary": "tap textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + } + }, + { + "name": "press", + "summary": "press \"Control+a\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "Control+a" + }, + { + "name": "press", + "summary": "press \"Control+Shift+ArrowLeft\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "Control+Shift+ArrowLeft" + }, + { + "name": "press", + "summary": "press \"Home\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "Home" + }, + { + "name": "press", + "summary": "press \"Shift+ArrowRight\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "press", + "summary": "press \"ArrowRight\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "ArrowRight" + }, + { + "name": "press", + "summary": "press \"Shift+ArrowRight\" on textbox \"release approved\"", + "target": { + "role": "textbox", + "text": "release approved", + "testId": "editor" + }, + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "pressKey", + "summary": "press \"Shift+ArrowRight\" on the focused field", + "key": "Shift+ArrowRight" + }, + { + "name": "tap", + "summary": "tap button \"B\"", + "target": { + "role": "button", + "name": "B", + "testId": "bold-button" + } + }, + { + "name": "tap", + "summary": "tap button \"Check document\"", + "target": { + "role": "button", + "name": "Check document", + "testId": "check-button" + } + } + ], + "executor": { + "name": "e2e-default-agent", + "version": "2" + }, + "recordedFor": { + "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20rich-text-editor", + "targetId": "web", + "instructionDigest": "e37ad9ac878deebe5c840d23fded9589f67228d8229a059b855347ab5e49ab96", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" + }, + "summary": "Entered “release approved,” selected only “approved” and applied bold formatting, then checked the document. The page confirmed “Document approved.”", + "startPath": "/e/rich-text-editor", + "endPath": "/e/rich-text-editor", + "endAnchors": [ + { + "text": "Document approved", + "testId": "success-message" + } + ], + "goneAnchors": [ + { + "text": "Type \"release approved\" in the editor and make the word approved bold, then check the document." + }, + { + "role": "button", + "name": "B", + "testId": "bold-button" + }, + { + "role": "textbox", + "testId": "editor" + }, + { + "role": "button", + "name": "Check document", + "testId": "check-button" + } + ], + "endWaitMs": 11834 + } +} diff --git a/apps/web-benchmark/.e2e/cache/21d3170211e57f3727e39c689d20db194c05a3d75ed7b2df912e496cbbb7470d.json b/apps/web-benchmark/.e2e/cache/95fca6f8a199510dc5c35bcd0ce03c590721dfc03c606147b54d65ef1bcd481e.json similarity index 59% rename from apps/web-benchmark/.e2e/cache/21d3170211e57f3727e39c689d20db194c05a3d75ed7b2df912e496cbbb7470d.json rename to apps/web-benchmark/.e2e/cache/95fca6f8a199510dc5c35bcd0ce03c590721dfc03c606147b54d65ef1bcd481e.json index de7645151..ef9f9eade 100644 --- a/apps/web-benchmark/.e2e/cache/21d3170211e57f3727e39c689d20db194c05a3d75ed7b2df912e496cbbb7470d.json +++ b/apps/web-benchmark/.e2e/cache/95fca6f8a199510dc5c35bcd0ce03c590721dfc03c606147b54d65ef1bcd481e.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T18:02:51.969Z", + "createdAt": "2026-10-04T08:31:38.993Z", "payload": { "actions": [ { @@ -33,9 +33,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20hover-menu", "targetId": "web", - "instructionDigest": "4d2bd092cafaf0bdc40c9442fb592d58283fd7a946e7efb98e1142cffe3c07a3" + "instructionDigest": "4d2bd092cafaf0bdc40c9442fb592d58283fd7a946e7efb98e1142cffe3c07a3", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Hovered over Account, then Billing, and selected Redeem voucher. The page now displays “Voucher redeemed.”.", + "summary": "Hovered Account, then Billing, and selected Redeem voucher. The page confirms “Voucher redeemed.”", "startPath": "/e/hover-menu", "endPath": "/e/hover-menu", "endAnchors": [ @@ -44,6 +47,23 @@ "testId": "success-message" } ], - "endWaitMs": 11764 + "goneAnchors": [ + { + "text": "Using hover only, find Redeem voucher under Account." + }, + { + "text": "Products", + "testId": "menu-products" + }, + { + "text": "Settings", + "testId": "menu-settings" + }, + { + "text": "Account", + "testId": "menu-account" + } + ], + "endWaitMs": 11468 } } diff --git a/apps/web-benchmark/.e2e/cache/dbf1a6c7f66db2213c503537198d1d9e1be4cb7290ed6a73eefc7dfe4d88c03f.json b/apps/web-benchmark/.e2e/cache/9c5a50b0bd4690906e0400a969a666387c904f089842a6e4806bb3c6f9c4328a.json similarity index 68% rename from apps/web-benchmark/.e2e/cache/dbf1a6c7f66db2213c503537198d1d9e1be4cb7290ed6a73eefc7dfe4d88c03f.json rename to apps/web-benchmark/.e2e/cache/9c5a50b0bd4690906e0400a969a666387c904f089842a6e4806bb3c6f9c4328a.json index 4986529ad..3bc799d7c 100644 --- a/apps/web-benchmark/.e2e/cache/dbf1a6c7f66db2213c503537198d1d9e1be4cb7290ed6a73eefc7dfe4d88c03f.json +++ b/apps/web-benchmark/.e2e/cache/9c5a50b0bd4690906e0400a969a666387c904f089842a6e4806bb3c6f9c4328a.json @@ -1,8 +1,15 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:30.518Z", + "createdAt": "2026-10-04T08:32:03.098Z", "payload": { "actions": [ + { + "name": "secondaryTap", + "summary": "secondary-tap node \"report.pdf\"", + "target": { + "text": "report.pdf" + } + }, { "name": "tap", "summary": "tap button \"More actions for report.pdf\"", @@ -45,9 +52,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20context-menu-trap", "targetId": "web", - "instructionDigest": "63530e281eafd2d07743a56500bfed3eae14e1ab7a4c0ad00e838e8b51912917" + "instructionDigest": "63530e281eafd2d07743a56500bfed3eae14e1ab7a4c0ad00e838e8b51912917", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Renamed report.pdf to summary.pdf using the file's More actions menu; the file list now shows summary.pdf and a success message confirms the rename.", + "summary": "Renamed report.pdf to summary.pdf. The file manager now lists summary.pdf and shows “File renamed to summary.pdf.”", "startPath": "/e/context-menu-trap", "endPath": "/e/context-menu-trap", "endAnchors": [ @@ -64,6 +74,16 @@ "testId": "success-message" } ], - "endWaitMs": 10102 + "goneAnchors": [ + { + "text": "report.pdf" + }, + { + "role": "button", + "name": "More actions for report.pdf", + "text": "⋯" + } + ], + "endWaitMs": 11888 } } diff --git a/apps/web-benchmark/.e2e/cache/6e9dadfddcfeb384163165db2579e58102db60f5dd29b524a0705fcbc50a09db.json b/apps/web-benchmark/.e2e/cache/9c9d5f9062162fb6e748f47c1788677bef1b1d9366b1a87a3013842625e659c6.json similarity index 57% rename from apps/web-benchmark/.e2e/cache/6e9dadfddcfeb384163165db2579e58102db60f5dd29b524a0705fcbc50a09db.json rename to apps/web-benchmark/.e2e/cache/9c9d5f9062162fb6e748f47c1788677bef1b1d9366b1a87a3013842625e659c6.json index 089f57263..e32857d5f 100644 --- a/apps/web-benchmark/.e2e/cache/6e9dadfddcfeb384163165db2579e58102db60f5dd29b524a0705fcbc50a09db.json +++ b/apps/web-benchmark/.e2e/cache/9c9d5f9062162fb6e748f47c1788677bef1b1d9366b1a87a3013842625e659c6.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:38.861Z", + "createdAt": "2026-10-04T08:34:05.073Z", "payload": { "actions": [ { @@ -42,9 +42,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20gift-card-purchase", "targetId": "web", - "instructionDigest": "58b82e08250d41e942c183d1c49b7e031fb5754f62c785c3f064c404c6b44851" + "instructionDigest": "58b82e08250d41e942c183d1c49b7e031fb5754f62c785c3f064c404c6b44851", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Selected a $25 gift card, entered friend@example.com, and completed the purchase. The page confirms the gift card was sent and is on its way.", + "summary": "Selected a $25 gift card, entered friend@example.com, and completed the purchase. The page confirms the gift card was sent and says a $25.00 gift card is on its way.", "startPath": "/e/gift-card-purchase", "endPath": "/e/gift-card-purchase", "endAnchors": [ @@ -56,6 +59,47 @@ "text": "A $25.00 gift card is on its way." } ], - "endWaitMs": 10101 + "goneAnchors": [ + { + "text": "Send a digital gift card. Delivered by email within minutes." + }, + { + "role": "button", + "name": "gift_card_amount_option_undefined", + "text": "$10", + "testId": "amount-10" + }, + { + "role": "button", + "name": "gift_card_amount_option_undefined", + "text": "$25", + "testId": "amount-25" + }, + { + "role": "button", + "name": "gift_card_amount_option_undefined", + "text": "$50", + "testId": "amount-50" + }, + { + "role": "textbox", + "name": "{{recipient_email_label}}", + "testId": "recipient-input", + "placeholder": "friend@example.com" + }, + { + "text": "Total" + }, + { + "text": "Total: $NaN" + }, + { + "role": "button", + "name": "btn_purchase_label", + "text": "Buy gift card", + "testId": "buy-button" + } + ], + "endWaitMs": 11710 } } diff --git a/apps/web-benchmark/.e2e/cache/9b6349a07095a300952960a00e4596fc14fdafe4527396654923943598cfc633.json b/apps/web-benchmark/.e2e/cache/9e0a6ad84bfa325d98bcb19080a64e331851f0d8e03ededc3ba6461df080565e.json similarity index 64% rename from apps/web-benchmark/.e2e/cache/9b6349a07095a300952960a00e4596fc14fdafe4527396654923943598cfc633.json rename to apps/web-benchmark/.e2e/cache/9e0a6ad84bfa325d98bcb19080a64e331851f0d8e03ededc3ba6461df080565e.json index ef06e7783..f8d4953cf 100644 --- a/apps/web-benchmark/.e2e/cache/9b6349a07095a300952960a00e4596fc14fdafe4527396654923943598cfc633.json +++ b/apps/web-benchmark/.e2e/cache/9e0a6ad84bfa325d98bcb19080a64e331851f0d8e03ededc3ba6461df080565e.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:28.966Z", + "createdAt": "2026-10-04T08:31:33.402Z", "payload": { "actions": [ { @@ -31,9 +31,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20debounced-search", "targetId": "web", - "instructionDigest": "e5044d8a420a21763ff0ba6ba21266a9cf45e9607dd3496c435089c8d757d4e8" + "instructionDigest": "e5044d8a420a21763ff0ba6ba21266a9cf45e9607dd3496c435089c8d757d4e8", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Searched for “trail mix,” selected exactly “Trail Mix 500 g,” and verified the page shows “Added Trail Mix 500 g to the cart.”", + "summary": "Searched for trail mix and selected exactly Trail Mix 500 g. The page confirms it was added to the cart.", "startPath": "/e/debounced-search", "endPath": "/e/debounced-search", "endAnchors": [ @@ -42,6 +45,17 @@ "testId": "success-message" } ], - "endWaitMs": 10121 + "goneAnchors": [ + { + "text": "Search for trail mix and pick exactly Trail Mix 500 g." + }, + { + "role": "combobox", + "name": "Search snacks", + "testId": "search-input", + "placeholder": "Search snacks" + } + ], + "endWaitMs": 11449 } } diff --git a/apps/web-benchmark/.e2e/cache/9e1dbad15746e685363092d30b7c43e49f4cdba5c9f330a8337e99429457a6ae.json b/apps/web-benchmark/.e2e/cache/9e1dbad15746e685363092d30b7c43e49f4cdba5c9f330a8337e99429457a6ae.json new file mode 100644 index 000000000..32ad60eba --- /dev/null +++ b/apps/web-benchmark/.e2e/cache/9e1dbad15746e685363092d30b7c43e49f4cdba5c9f330a8337e99429457a6ae.json @@ -0,0 +1,79 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-10-04T08:29:03.146Z", + "payload": { + "actions": [ + { + "name": "tap", + "summary": "tap button \"Cancel order\"", + "target": { + "role": "button", + "name": "Cancel order", + "text": "Pay now" + } + }, + { + "name": "tool", + "summary": "tool type (run-time value)", + "derived": "minted-token" + }, + { + "name": "tap", + "summary": "tap button \"Discard draft\"", + "target": { + "role": "button", + "name": "Discard draft", + "text": "Submit payment" + } + } + ], + "executor": { + "name": "e2e-default-agent", + "version": "2" + }, + "recordedFor": { + "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20lying-labels", + "targetId": "web", + "instructionDigest": "75cac33ad209d332c191820f8b966e11e3b86f06d25432d913adf93fe369052d", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" + }, + "summary": "Followed the visible labels and page instructions: entered 42.50 in the field beside “Amount,” left the visibly labeled Reference field empty, and submitted. The page now shows “Payment sent.”", + "startPath": "/e/lying-labels", + "endPath": "/e/lying-labels", + "endAnchors": [ + { + "text": "Payment sent", + "testId": "success-message" + } + ], + "goneAnchors": [ + { + "text": "Click the button that visibly says Pay now." + }, + { + "text": "Do not press this button" + }, + { + "text": "Pay now safely" + }, + { + "role": "button", + "name": "Confirm payment", + "text": "Contact support" + }, + { + "role": "button", + "name": "Cancel order", + "text": "Pay now" + }, + { + "role": "button", + "name": "Pay now", + "text": "Cancel order" + } + ], + "endWaitMs": 11896 + } +} diff --git a/apps/web-benchmark/.e2e/cache/9a37ff39c572841d0303e55b4c7322295aeffa6332a2ed39d13db3aa9b41b21b.json b/apps/web-benchmark/.e2e/cache/a836bf1e72f05dd86bdb9ad4a5460aac4e24c0268326c0754e6827941fbbe8c8.json similarity index 61% rename from apps/web-benchmark/.e2e/cache/9a37ff39c572841d0303e55b4c7322295aeffa6332a2ed39d13db3aa9b41b21b.json rename to apps/web-benchmark/.e2e/cache/a836bf1e72f05dd86bdb9ad4a5460aac4e24c0268326c0754e6827941fbbe8c8.json index 24ba3c26d..42ad0064e 100644 --- a/apps/web-benchmark/.e2e/cache/9a37ff39c572841d0303e55b4c7322295aeffa6332a2ed39d13db3aa9b41b21b.json +++ b/apps/web-benchmark/.e2e/cache/a836bf1e72f05dd86bdb9ad4a5460aac4e24c0268326c0754e6827941fbbe8c8.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-24T19:35:19.528Z", + "createdAt": "2026-10-04T08:29:31.075Z", "payload": { "actions": [ { @@ -18,9 +18,12 @@ "recordedFor": { "testId": "tests-agent/control-inventory.e2e.ts::control%20inventory::scroll_to%20brings%20the%20footnote%20into%20view", "targetId": "web", - "instructionDigest": "60d7d1c1109f3844ed7b5cd3f6f351e1b297af10aa7e4371ed45de087cce0df3" + "instructionDigest": "60d7d1c1109f3844ed7b5cd3f6f351e1b297af10aa7e4371ed45de087cce0df3", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Scrolled the Footnote paragraph into view; the Footnote state now shows “in view.”", + "summary": "Scrolled the page until the Footnote paragraph was in view; the Footnote state now reads “in view”.", "startPath": "/e/control-inventory", "endPath": "/e/control-inventory", "endAnchors": [ @@ -30,6 +33,13 @@ "text": "in view" } ], - "endWaitMs": 12064 + "goneAnchors": [ + { + "role": "status", + "name": "Footnote state", + "text": "out of view" + } + ], + "endWaitMs": 11385 } } diff --git a/apps/web-benchmark/.e2e/cache/b49868122bd279e5544bdfad218fe2a728363d123a4d037133b429edd66980de.json b/apps/web-benchmark/.e2e/cache/b49868122bd279e5544bdfad218fe2a728363d123a4d037133b429edd66980de.json new file mode 100644 index 000000000..9128b77b5 --- /dev/null +++ b/apps/web-benchmark/.e2e/cache/b49868122bd279e5544bdfad218fe2a728363d123a4d037133b429edd66980de.json @@ -0,0 +1,68 @@ +{ + "schemaVersion": "trace-1", + "createdAt": "2026-10-04T08:34:08.113Z", + "payload": { + "actions": [ + { + "name": "type", + "summary": "type \"mug\" into textbox \"Filter products\"", + "target": { + "role": "textbox", + "name": "Filter products", + "testId": "filter-input", + "placeholder": "Filter products" + }, + "value": "mug" + } + ], + "executor": { + "name": "e2e-default-agent", + "version": "2" + }, + "recordedFor": { + "testId": "tests-agent/scenarios.e2e.ts::act%20filters%20the%20list%20and%20the%20URL%20carries%20the%20filter", + "targetId": "web", + "instructionDigest": "c54ac6f470b19c027dd580aa19d24ab48c4f0612b3930178028f20eb76cdd216", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" + }, + "summary": "Entered “mug” in the product filter. The list is filtered to Titanium Mug, and the page URL is /e/filter-deep-link?q=mug.", + "startPath": "/e/filter-deep-link", + "endPath": "/e/filter-deep-link?q=mug", + "endAnchors": [ + { + "role": "textbox", + "name": "Filter products", + "testId": "filter-input", + "placeholder": "Filter products", + "value": "mug" + } + ], + "goneAnchors": [ + { + "role": "textbox", + "name": "Filter products", + "testId": "filter-input", + "placeholder": "Filter products" + }, + { + "role": "listitem", + "name": "Trail Backpack" + }, + { + "role": "listitem", + "name": "Steel Water Bottle" + }, + { + "role": "listitem", + "name": "Merino Hiking Socks" + }, + { + "role": "listitem", + "name": "Camp Lantern" + } + ], + "endWaitMs": 11489 + } +} diff --git a/apps/web-benchmark/.e2e/cache/ffc271f65288ac2b62a497a04105198017a463eaf9410aeabd1e4afe79a79dc5.json b/apps/web-benchmark/.e2e/cache/c8d793f4eb8f1beff83231ec5037fc53f1b4261ccc1523f0faaee6945a05cd9b.json similarity index 53% rename from apps/web-benchmark/.e2e/cache/ffc271f65288ac2b62a497a04105198017a463eaf9410aeabd1e4afe79a79dc5.json rename to apps/web-benchmark/.e2e/cache/c8d793f4eb8f1beff83231ec5037fc53f1b4261ccc1523f0faaee6945a05cd9b.json index 1027832f5..4846cc7ae 100644 --- a/apps/web-benchmark/.e2e/cache/ffc271f65288ac2b62a497a04105198017a463eaf9410aeabd1e4afe79a79dc5.json +++ b/apps/web-benchmark/.e2e/cache/c8d793f4eb8f1beff83231ec5037fc53f1b4261ccc1523f0faaee6945a05cd9b.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T10:41:22.965Z", + "createdAt": "2026-10-04T08:28:56.187Z", "payload": { "actions": [ { @@ -24,9 +24,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20shadow-dom-form", "targetId": "web", - "instructionDigest": "c6884af43cef64f928cde4eaa3be0cbc128893f01611d32236baf514a915c0a6" + "instructionDigest": "c6884af43cef64f928cde4eaa3be0cbc128893f01611d32236baf514a915c0a6", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Entered SHADOW-42 into the nested-shadow-root form and submitted it. The page now displays “Access granted.”.", + "summary": "Entered SHADOW-42 and submitted the form. The page now shows “Access granted.”", "startPath": "/e/shadow-dom-form", "endPath": "/e/shadow-dom-form", "endAnchors": [ @@ -35,6 +38,23 @@ "testId": "success-message" } ], - "endWaitMs": 11941 + "goneAnchors": [ + { + "text": "The form below lives inside a closed shadow root. Enter SHADOW-42 and press Submit." + }, + { + "text": "Access code hint: SHADOW-42" + }, + { + "role": "textbox", + "name": "Access code", + "placeholder": "Access code" + }, + { + "role": "button", + "name": "Submit" + } + ], + "endWaitMs": 11646 } } diff --git a/apps/web-benchmark/.e2e/cache/84586bbbf74c35b93fa6418467148a243d902bd7aa6394c744732cd0cbd1281d.json b/apps/web-benchmark/.e2e/cache/c9ae158cd9a15f3caf0fcdc7abf497ebcf86304886a6800692a03c47c0277fbd.json similarity index 57% rename from apps/web-benchmark/.e2e/cache/84586bbbf74c35b93fa6418467148a243d902bd7aa6394c744732cd0cbd1281d.json rename to apps/web-benchmark/.e2e/cache/c9ae158cd9a15f3caf0fcdc7abf497ebcf86304886a6800692a03c47c0277fbd.json index dfed30cce..e68f03b84 100644 --- a/apps/web-benchmark/.e2e/cache/84586bbbf74c35b93fa6418467148a243d902bd7aa6394c744732cd0cbd1281d.json +++ b/apps/web-benchmark/.e2e/cache/c9ae158cd9a15f3caf0fcdc7abf497ebcf86304886a6800692a03c47c0277fbd.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:34.492Z", + "createdAt": "2026-10-04T08:33:52.251Z", "payload": { "actions": [ { @@ -49,17 +49,61 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20playbook-cleanup", "targetId": "web", - "instructionDigest": "423676dde62142adee61619b5958db96013371bf84519c7e0441aaa32d6f9fce" + "instructionDigest": "423676dde62142adee61619b5958db96013371bf84519c7e0441aaa32d6f9fce", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Deleted the playbook \"Launch Approval Draft,\" confirmed its removal, and searched the exact name. The page shows: No playbooks match \"Launch Approval Draft.\"", + "summary": "Deleted the playbook “Launch Approval Draft” and confirmed the removal. Searching for its exact name shows: “No playbooks match ‘Launch Approval Draft’.”", "startPath": "/e/playbook-cleanup", "endPath": "/e/playbook-cleanup", "endAnchors": [ + { + "role": "searchbox", + "name": "Search playbooks", + "placeholder": "Search", + "value": "Launch Approval Draft" + }, { "text": "No playbooks match \"Launch Approval Draft\".", "testId": "success-message" } ], - "endWaitMs": 10103 + "goneAnchors": [ + { + "role": "searchbox", + "name": "Search playbooks", + "placeholder": "Search" + }, + { + "role": "heading", + "name": "NDA Review" + }, + { + "text": "8 rules" + }, + { + "role": "button", + "name": "More actions for NDA Review", + "text": "⋯" + }, + { + "role": "heading", + "name": "Launch Approval Draft" + }, + { + "text": "4 rules" + }, + { + "role": "button", + "name": "More actions for Launch Approval Draft", + "text": "⋯" + }, + { + "role": "heading", + "name": "Vendor Onboarding" + } + ], + "endWaitMs": 11698 } } diff --git a/apps/web-benchmark/.e2e/cache/62e37c5052c64eac130df6b297136c0caa3067b2872ececd647afa6c1a5514c1.json b/apps/web-benchmark/.e2e/cache/cc0f6ff334fc11d69f3cb984791c64ea01c31b33b33334f2e476049f38606a33.json similarity index 53% rename from apps/web-benchmark/.e2e/cache/62e37c5052c64eac130df6b297136c0caa3067b2872ececd647afa6c1a5514c1.json rename to apps/web-benchmark/.e2e/cache/cc0f6ff334fc11d69f3cb984791c64ea01c31b33b33334f2e476049f38606a33.json index 69a25d872..5c4b632a7 100644 --- a/apps/web-benchmark/.e2e/cache/62e37c5052c64eac130df6b297136c0caa3067b2872ececd647afa6c1a5514c1.json +++ b/apps/web-benchmark/.e2e/cache/cc0f6ff334fc11d69f3cb984791c64ea01c31b33b33334f2e476049f38606a33.json @@ -1,8 +1,17 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-25T18:09:48.642Z", + "createdAt": "2026-10-04T08:29:14.465Z", "payload": { "actions": [ + { + "name": "secondaryTap", + "summary": "secondary-tap listitem \"report.pdf\" in \"Files\"", + "target": { + "role": "listitem", + "name": "report.pdf", + "within": "Files" + } + }, { "name": "secondaryTap", "summary": "secondary-tap node \"report.pdf\"", @@ -21,15 +30,18 @@ } ], "executor": { - "name": "web-benchmark-inventory-recorder", - "version": "1" + "name": "e2e-default-agent", + "version": "2" }, "recordedFor": { "testId": "tests-agent/control-inventory.e2e.ts::control%20inventory::a%20right-click%20opens%20the%20file%20menu%20and%20Rename%20renames%20the%20file", "targetId": "web", - "instructionDigest": "cb3da13c69639f87ac80d3bd6fa2b908c9df0bbbda211433bdc779611d4909fe" + "instructionDigest": "cb3da13c69639f87ac80d3bd6fa2b908c9df0bbbda211433bdc779611d4909fe", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Opened the report.pdf context menu and chose Rename. File state reads \"report.pdf renamed to summary.pdf\", and the file list contains summary.pdf.", + "summary": "Right-clicked report.pdf to open its actions menu, selected Rename, and verified the file list now shows summary.pdf with the status “report.pdf renamed to summary.pdf.”", "startPath": "/e/control-inventory", "endPath": "/e/control-inventory", "endAnchors": [ @@ -46,6 +58,20 @@ "name": "summary.pdf" } ], - "endWaitMs": 10222 + "goneAnchors": [ + { + "role": "status", + "name": "File state", + "text": "no menu open" + }, + { + "text": "report.pdf" + }, + { + "role": "listitem", + "name": "report.pdf" + } + ], + "endWaitMs": 11535 } } diff --git a/apps/web-benchmark/.e2e/cache/91c5fa1cc7fd3ed41e763e1c3a344d4d519a1f8703c012a1a1d46045ded8d4bd.json b/apps/web-benchmark/.e2e/cache/cc89e287153d37442a76efb692f7ae15f79a39c3c44893c8592ece73d46d413b.json similarity index 75% rename from apps/web-benchmark/.e2e/cache/91c5fa1cc7fd3ed41e763e1c3a344d4d519a1f8703c012a1a1d46045ded8d4bd.json rename to apps/web-benchmark/.e2e/cache/cc89e287153d37442a76efb692f7ae15f79a39c3c44893c8592ece73d46d413b.json index 1c787e406..f028db332 100644 --- a/apps/web-benchmark/.e2e/cache/91c5fa1cc7fd3ed41e763e1c3a344d4d519a1f8703c012a1a1d46045ded8d4bd.json +++ b/apps/web-benchmark/.e2e/cache/cc89e287153d37442a76efb692f7ae15f79a39c3c44893c8592ece73d46d413b.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:33.632Z", + "createdAt": "2026-10-04T08:32:31.439Z", "payload": { "actions": [ { @@ -66,9 +66,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20date-picker", "targetId": "web", - "instructionDigest": "dcf88f9a5a9a284823967721c9316e3541f960b36032c247343f678217019dde" + "instructionDigest": "dcf88f9a5a9a284823967721c9316e3541f960b36032c247343f678217019dde", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Navigated the calendar from June to September 2026, selected September 17, 2026, and confirmed the booking. The page displays “Appointment booked for September 17, 2026.”", + "summary": "Navigated the calendar from June to September 2026, selected September 17, and confirmed. The page shows “Appointment booked for September 17, 2026.”", "startPath": "/e/date-picker", "endPath": "/e/date-picker", "endAnchors": [ @@ -77,6 +80,19 @@ "testId": "success-message" } ], - "endWaitMs": 10099 + "goneAnchors": [ + { + "role": "textbox", + "name": "Select date", + "testId": "date-input", + "placeholder": "Select date" + }, + { + "role": "button", + "name": "Confirm booking", + "testId": "confirm-button" + } + ], + "endWaitMs": 11647 } } diff --git a/apps/web-benchmark/.e2e/cache/52c2b9053d6bfb1471ac2d8846b2fd6048f426611cf04f6eda3ee632a702cd78.json b/apps/web-benchmark/.e2e/cache/cc8cc932b468583080dc18ea59fda1bc1a0ad0d0974a0834eefbe08bb6fbfd5c.json similarity index 58% rename from apps/web-benchmark/.e2e/cache/52c2b9053d6bfb1471ac2d8846b2fd6048f426611cf04f6eda3ee632a702cd78.json rename to apps/web-benchmark/.e2e/cache/cc8cc932b468583080dc18ea59fda1bc1a0ad0d0974a0834eefbe08bb6fbfd5c.json index 4f98053e8..4f97138de 100644 --- a/apps/web-benchmark/.e2e/cache/52c2b9053d6bfb1471ac2d8846b2fd6048f426611cf04f6eda3ee632a702cd78.json +++ b/apps/web-benchmark/.e2e/cache/cc8cc932b468583080dc18ea59fda1bc1a0ad0d0974a0834eefbe08bb6fbfd5c.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:31.162Z", + "createdAt": "2026-10-04T08:32:08.612Z", "payload": { "actions": [ { @@ -38,9 +38,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20stale-dom", "targetId": "web", - "instructionDigest": "c7958b2326b3f2b42541d8b8261d04bb3d18d3fbbf439d754e64d72856a74ba4" + "instructionDigest": "c7958b2326b3f2b42541d8b8261d04bb3d18d3fbbf439d754e64d72856a74ba4", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Clicked the Tap me button three times, and the page now shows Progress: 3 / 3 with the success message “Target clicked 3 times.”", + "summary": "Clicked Tap me three times. The page shows Progress: 3 / 3 and “Target clicked 3 times.”", "startPath": "/e/stale-dom", "endPath": "/e/stale-dom", "endAnchors": [ @@ -53,6 +56,37 @@ "testId": "success-message" } ], - "endWaitMs": 10100 + "goneAnchors": [ + { + "text": "Progress: 0 / 3", + "testId": "progress" + }, + { + "text": "Slot 1" + }, + { + "role": "button", + "name": "Decoy", + "testId": "decoy-button" + }, + { + "text": "Slot 2" + }, + { + "text": "Slot 3" + }, + { + "role": "button", + "name": "Tap me", + "testId": "target-button" + }, + { + "text": "Slot 4" + }, + { + "text": "Slot 5" + } + ], + "endWaitMs": 11441 } } diff --git a/apps/web-benchmark/.e2e/cache/d01c00f9543a36210f2840b6eed3d6de7474f88fc2d442611cf7a31280facf66.json b/apps/web-benchmark/.e2e/cache/d01c00f9543a36210f2840b6eed3d6de7474f88fc2d442611cf7a31280facf66.json deleted file mode 100644 index 40e315919..000000000 --- a/apps/web-benchmark/.e2e/cache/d01c00f9543a36210f2840b6eed3d6de7474f88fc2d442611cf7a31280facf66.json +++ /dev/null @@ -1,49 +0,0 @@ -{ - "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:29:40.106Z", - "payload": { - "actions": [ - { - "name": "tap", - "summary": "tap button \"Cancel order\"", - "target": { - "role": "button", - "name": "Cancel order", - "text": "Pay now" - } - }, - { - "name": "tool", - "summary": "tool type (run-time value)" - }, - { - "name": "tap", - "summary": "tap button \"Discard draft\"", - "target": { - "role": "button", - "name": "Discard draft", - "text": "Submit payment" - } - } - ], - "executor": { - "name": "e2e-default-agent", - "version": "2" - }, - "recordedFor": { - "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20lying-labels", - "targetId": "web", - "instructionDigest": "75cac33ad209d332c191820f8b966e11e3b86f06d25432d913adf93fe369052d" - }, - "summary": "Entered 42.50 in the textbox visibly labeled Amount, left the field visibly labeled Reference empty, and submitted via the control visibly labeled Submit payment. The page now shows “Payment sent.”.", - "startPath": "/e/lying-labels", - "endPath": "/e/lying-labels", - "endAnchors": [ - { - "text": "Payment sent", - "testId": "success-message" - } - ], - "endWaitMs": 11664 - } -} diff --git a/apps/web-benchmark/.e2e/cache/e2f557718b56e9e4ecf51224b005d95f429fc2eb7309fada37ece71c98421513.json b/apps/web-benchmark/.e2e/cache/e2f557718b56e9e4ecf51224b005d95f429fc2eb7309fada37ece71c98421513.json deleted file mode 100644 index 949349d13..000000000 --- a/apps/web-benchmark/.e2e/cache/e2f557718b56e9e4ecf51224b005d95f429fc2eb7309fada37ece71c98421513.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:39.253Z", - "payload": { - "actions": [ - { - "name": "type", - "summary": "type \"mug\" into textbox \"Filter products\"", - "target": { - "role": "textbox", - "name": "Filter products", - "testId": "filter-input", - "placeholder": "Filter products" - }, - "value": "mug" - } - ], - "executor": { - "name": "e2e-default-agent", - "version": "2" - }, - "recordedFor": { - "testId": "tests-agent/scenarios.e2e.ts::act%20filters%20the%20list%20and%20the%20URL%20carries%20the%20filter", - "targetId": "web", - "instructionDigest": "c54ac6f470b19c027dd580aa19d24ab48c4f0612b3930178028f20eb76cdd216" - }, - "summary": "Entered “mug” in the product filter. The URL is now /e/filter-deep-link?q=mug and the list contains only “Titanium Mug”.", - "startPath": "/e/filter-deep-link", - "endPath": "/e/filter-deep-link?q=mug", - "endWaitMs": 10100 - } -} diff --git a/apps/web-benchmark/.e2e/cache/e5c98b82c848888ebf302d3e1949feb5cc0140af603317477dafbac69674d6ba.json b/apps/web-benchmark/.e2e/cache/e52795af76bcb5757ab47b6824e016e2906dfe9d0e9b557c1d6197a17a87df8b.json similarity index 80% rename from apps/web-benchmark/.e2e/cache/e5c98b82c848888ebf302d3e1949feb5cc0140af603317477dafbac69674d6ba.json rename to apps/web-benchmark/.e2e/cache/e52795af76bcb5757ab47b6824e016e2906dfe9d0e9b557c1d6197a17a87df8b.json index e59116807..1d04f3b32 100644 --- a/apps/web-benchmark/.e2e/cache/e5c98b82c848888ebf302d3e1949feb5cc0140af603317477dafbac69674d6ba.json +++ b/apps/web-benchmark/.e2e/cache/e52795af76bcb5757ab47b6824e016e2906dfe9d0e9b557c1d6197a17a87df8b.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:29:31.865Z", + "createdAt": "2026-10-04T08:28:57.166Z", "payload": { "actions": [ { @@ -42,9 +42,12 @@ "recordedFor": { "testId": "tests-agent/unique-cache.e2e.ts::act%20adds%20an%20address%20with%20a%20run-unique%20label", "targetId": "web", - "instructionDigest": "7e6f99066c5bc180c1e5188150277917670b918eb0a9646ac967ef798648b455" + "instructionDigest": "7e6f99066c5bc180c1e5188150277917670b918eb0a9646ac967ef798648b455", + "paramsDigest": "49ed432ec36207f5728e4da6da98006ed0d126739d72fdcdb919e032bfd6a511", + "callIndex": 0, + "agent": "default" }, - "summary": "The address was saved successfully. The list now includes “{{param:/label}}” at “7 Pine Way, Bend,” and the screen shows an “Address saved” confirmation.", + "summary": "Added address labelled “Cabin mutk7a4i” with line “7 Pine Way, Bend”; the new address appears in the list and an “Address saved” confirmation is shown.", "startPath": "/e/saved-addresses", "endPath": "/e/saved-addresses", "endAnchors": [ @@ -64,6 +67,6 @@ "text": "7 Pine Way, Bend" } ], - "endWaitMs": 10129 + "endWaitMs": 12289 } } diff --git a/apps/web-benchmark/.e2e/cache/bca42cea37fc800e340c8f4c3d8d0dc51a83b1f30e1ebbe5ae2070755a48c335.json b/apps/web-benchmark/.e2e/cache/ec7a25eba0e1a7efaf68f40784a69d7760d3c584ec6e497fbafb7ee166da771b.json similarity index 70% rename from apps/web-benchmark/.e2e/cache/bca42cea37fc800e340c8f4c3d8d0dc51a83b1f30e1ebbe5ae2070755a48c335.json rename to apps/web-benchmark/.e2e/cache/ec7a25eba0e1a7efaf68f40784a69d7760d3c584ec6e497fbafb7ee166da771b.json index d056521e5..5c2a8771c 100644 --- a/apps/web-benchmark/.e2e/cache/bca42cea37fc800e340c8f4c3d8d0dc51a83b1f30e1ebbe5ae2070755a48c335.json +++ b/apps/web-benchmark/.e2e/cache/ec7a25eba0e1a7efaf68f40784a69d7760d3c584ec6e497fbafb7ee166da771b.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T18:03:08.643Z", + "createdAt": "2026-10-04T08:31:43.811Z", "payload": { "actions": [ { @@ -33,9 +33,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20drag-and-drop", "targetId": "web", - "instructionDigest": "24ea40459eba1c0e7841d174c9d6b29255cc71346083add2a38c39491aff758c" + "instructionDigest": "24ea40459eba1c0e7841d174c9d6b29255cc71346083add2a38c39491aff758c", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Dragged Banana into the dropzone, then Cherry, and the page displayed “Items dropped in the right order.” Apple was left alone.", + "summary": "Dragged Banana and then Cherry into the dropzone, leaving Apple alone. The page displays “Items dropped in the right order.”", "startPath": "/e/drag-and-drop", "endPath": "/e/drag-and-drop", "endAnchors": [ @@ -44,6 +47,11 @@ "testId": "success-message" } ], - "endWaitMs": 12698 + "goneAnchors": [ + { + "text": "Drop items here" + } + ], + "endWaitMs": 11777 } } diff --git a/apps/web-benchmark/.e2e/cache/5889b07715480e781982f1c22be86a9ec12fb0461da4f772d0e8dac3feb1fa92.json b/apps/web-benchmark/.e2e/cache/eddce497b6948544b8bef8d0459a2ec07e77041b3e0362c6419f54c0feb1c0d0.json similarity index 58% rename from apps/web-benchmark/.e2e/cache/5889b07715480e781982f1c22be86a9ec12fb0461da4f772d0e8dac3feb1fa92.json rename to apps/web-benchmark/.e2e/cache/eddce497b6948544b8bef8d0459a2ec07e77041b3e0362c6419f54c0feb1c0d0.json index 6dd9998a4..a080e2ee6 100644 --- a/apps/web-benchmark/.e2e/cache/5889b07715480e781982f1c22be86a9ec12fb0461da4f772d0e8dac3feb1fa92.json +++ b/apps/web-benchmark/.e2e/cache/eddce497b6948544b8bef8d0459a2ec07e77041b3e0362c6419f54c0feb1c0d0.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-24T19:34:48.350Z", + "createdAt": "2026-10-04T08:28:55.544Z", "payload": { "actions": [ { @@ -31,9 +31,12 @@ "recordedFor": { "testId": "tests-agent/control-inventory.e2e.ts::control%20inventory::check%20sets%20the%20checkbox%20and%20the%20radio", "targetId": "web", - "instructionDigest": "31e69bd3b8b58acb08fa60eaea2976e3271b633b840b8ffca9440e21488961fc" + "instructionDigest": "31e69bd3b8b58acb08fa60eaea2976e3271b633b840b8ffca9440e21488961fc", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Agreed to the terms and selected Medium. The Toggles state now shows “terms agreed, size Medium.”", + "summary": "Agreed to the terms and selected Medium. The Toggles state confirms “terms agreed, size Medium.”", "startPath": "/e/control-inventory", "endPath": "/e/control-inventory", "endAnchors": [ @@ -41,8 +44,37 @@ "role": "status", "name": "Toggles state", "text": "terms agreed, size Medium" + }, + { + "role": "checkbox", + "name": "Agree to terms", + "states": [ + "checked" + ] + }, + { + "role": "radio", + "name": "Medium", + "states": [ + "checked" + ] + } + ], + "goneAnchors": [ + { + "role": "status", + "name": "Toggles state", + "text": "terms not agreed, size unset" + }, + { + "role": "checkbox", + "name": "Agree to terms" + }, + { + "role": "radio", + "name": "Medium" } ], - "endWaitMs": 12023 + "endWaitMs": 11356 } } diff --git a/apps/web-benchmark/.e2e/cache/aa933fbea4f49718b9beae66e90c3c5b1a0f807b74991ae506bb0ece2573a914.json b/apps/web-benchmark/.e2e/cache/fbb677dfe2419445bd605212e7afdc5207510f44aed0d470ab387c9e38a69bed.json similarity index 56% rename from apps/web-benchmark/.e2e/cache/aa933fbea4f49718b9beae66e90c3c5b1a0f807b74991ae506bb0ece2573a914.json rename to apps/web-benchmark/.e2e/cache/fbb677dfe2419445bd605212e7afdc5207510f44aed0d470ab387c9e38a69bed.json index c014c075c..1c272ce17 100644 --- a/apps/web-benchmark/.e2e/cache/aa933fbea4f49718b9beae66e90c3c5b1a0f807b74991ae506bb0ece2573a914.json +++ b/apps/web-benchmark/.e2e/cache/fbb677dfe2419445bd605212e7afdc5207510f44aed0d470ab387c9e38a69bed.json @@ -1,6 +1,6 @@ { "schemaVersion": "trace-1", - "createdAt": "2026-09-23T13:30:09.534Z", + "createdAt": "2026-10-04T08:31:11.942Z", "payload": { "actions": [ { @@ -37,9 +37,12 @@ "recordedFor": { "testId": "tests-agent/scenarios.e2e.ts::act%20completes%20overlay-trap", "targetId": "web", - "instructionDigest": "6705af9140f936652f275b785be7e42b157ba15b3fa16eefba2c51d6f3fdcd1b" + "instructionDigest": "6705af9140f936652f275b785be7e42b157ba15b3fa16eefba2c51d6f3fdcd1b", + "paramsDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "callIndex": 0, + "agent": "default" }, - "summary": "Accepted the cookie banner, closed the overlay, and pressed Continue. The page now shows the success message “You made it past the overlay.”", + "summary": "Dismissed the cookie banner, closed the overlay, and pressed Continue. The page shows “You made it past the overlay.”", "startPath": "/e/overlay-trap", "endPath": "/e/overlay-trap", "endAnchors": [ @@ -48,6 +51,28 @@ "testId": "success-message" } ], - "endWaitMs": 10096 + "goneAnchors": [ + { + "text": "Accept cookies, close whatever is blocking clicks, then press Continue." + }, + { + "text": "Blocked clicks: 0", + "testId": "blocked-count" + }, + { + "role": "button", + "name": "Continue", + "testId": "continue-button" + }, + { + "text": "We use cookies to run this benchmark." + }, + { + "role": "button", + "name": "Accept cookies", + "testId": "accept-cookies-button" + } + ], + "endWaitMs": 11329 } } diff --git a/apps/web-benchmark/app/src/Examples/ControlInventory.tsx b/apps/web-benchmark/app/src/Examples/ControlInventory.tsx index 1928faf90..353756bd0 100644 --- a/apps/web-benchmark/app/src/Examples/ControlInventory.tsx +++ b/apps/web-benchmark/app/src/Examples/ControlInventory.tsx @@ -3,6 +3,7 @@ import { type CSSProperties, useEffect, useRef, useState } from "react"; const SIZES = ["Small", "Medium", "Large"] as const; +const DELIVERIES = ["Standard", "Express"] as const; const INITIAL_FILES = ["quarterly-report.pdf", "report.pdf", "notes.txt"]; const TARGET_FILE = "report.pdf"; const RENAMED_FILE = "summary.pdf"; @@ -10,6 +11,7 @@ const LONG_PRESS_MS = 500; const DETAILS_QUERY = "view=details"; type Size = (typeof SIZES)[number]; +type Delivery = (typeof DELIVERIES)[number]; type View = "inventory" | "details"; /** @@ -20,7 +22,7 @@ const viewFromLocation = (): View => (window.location.search.includes(DETAILS_QU /** * The deterministic contract surface: plain controls with every state * exposed, one exercise per agent verb the hard scenarios never reach (a - * checkbox and a radio to set, a long press and a double click to tell from a + * checkbox and a radio to set, a radio the pick replaces with its summary, a long press and a double click to tell from a * tap, a context menu that opens on right-click only, a file input, the * browser history, a paragraph to scroll into view). Not a hard surface: each * control reports its state in a status line, and the success message shows @@ -30,6 +32,7 @@ const viewFromLocation = (): View => (window.location.search.includes(DETAILS_QU export default function ControlInventory() { const [agreed, setAgreed] = useState(false); const [size, setSize] = useState(null); + const [delivery, setDelivery] = useState(null); const [holdState, setHoldState] = useState("none"); const [longPressed, setLongPressed] = useState(false); const [doubleClickState, setDoubleClickState] = useState("none"); @@ -75,6 +78,7 @@ export default function ControlInventory() { const done = agreed && size === "Medium" && + delivery === "Express" && longPressed && doubleClicked && renamed && @@ -120,7 +124,7 @@ export default function ControlInventory() {

) : null}

- Plain controls, one exercise each. Agree to the terms and pick the Medium size; hold the Hold me button, then + Plain controls, one exercise each. Agree to the terms and pick the Medium size; pick Express delivery; hold the Hold me button, then double-click the Double-click me button; rename report.pdf to summary.pdf through its context menu, which opens on right-click only; attach a file; open the details and return with the browser history; bring the footnote at the bottom into view. @@ -131,6 +135,10 @@ export default function ControlInventory() {

{`${agreed ? "terms agreed" : "terms not agreed"}, size ${size ?? "unset"}`}
+
Delivery
+
+ {delivery ?? "unset"} +
Hold
{holdState} @@ -181,6 +189,24 @@ export default function ControlInventory() { ))} + {delivery === null ? ( +
+ Delivery + {DELIVERIES.map((option) => ( + + ))} +
+ ) : ( +
+ {`${delivery} delivery selected`} + +
+ )}
diff --git a/apps/web-benchmark/package.json b/apps/web-benchmark/package.json index 6f11e2930..887ee826c 100644 --- a/apps/web-benchmark/package.json +++ b/apps/web-benchmark/package.json @@ -26,7 +26,6 @@ "@types/react": "19.3.0", "@types/react-dom": "19.3.0", "ai": "7.0.107", - "playwright": "1.63.0", "typescript": "7.0.2" } } diff --git a/apps/web-benchmark/tests-agent/control-inventory.e2e.ts b/apps/web-benchmark/tests-agent/control-inventory.e2e.ts index cf6ee86da..a11c4adeb 100644 --- a/apps/web-benchmark/tests-agent/control-inventory.e2e.ts +++ b/apps/web-benchmark/tests-agent/control-inventory.e2e.ts @@ -1,6 +1,6 @@ /** * The agent verbs the hard scenarios never reach, each on the plain control - * built for it: check, the long press and the double click, a context menu + * built for it: check, on a radio the pick replaces too, the long press and the double click, a context menu * that opens on right-click only, a file input, the browser history, a scroll * into view. One step per verb, so a red test names the verb, and a locator * check pins each outcome the way the deterministic twin reads it. @@ -21,6 +21,11 @@ test.describe('control inventory', () => { await expect(screen.getByLabel('Toggles state')).toHaveText('terms agreed, size Medium'); }); + test('check picks a radio the pick replaces with its summary', TAGS, async ({ agent, screen }) => { + await agent.act('pick Express delivery'); + await expect(screen.getByLabel('Delivery state')).toHaveText('Express'); + }); + test('a long press and a double click are chosen over a plain tap', TAGS, async ({ agent, screen }) => { await agent.act('long-press the Hold me button until its state reads long-pressed, then double-click the Double-click me button'); await expect(screen.getByLabel('Hold state')).toHaveText('long-pressed'); diff --git a/apps/web-benchmark/tests/control-inventory.e2e.ts b/apps/web-benchmark/tests/control-inventory.e2e.ts index af2116e3b..387aeaa28 100644 --- a/apps/web-benchmark/tests/control-inventory.e2e.ts +++ b/apps/web-benchmark/tests/control-inventory.e2e.ts @@ -17,6 +17,12 @@ test.describe('control inventory', () => { await expect(screen.getByLabel('Toggles state')).toHaveText('terms not agreed, size Medium'); }); + test('check picks a radio the pick replaces with its summary', async ({ screen }) => { + await screen.getByRole('radio', 'Express').check(); + await expect(screen.getByLabel('Delivery state')).toHaveText('Express'); + await expect(screen.getByText('Express delivery selected')).toBeVisible(); + }); + test('a long press and a double click are told apart from a tap', async ({ screen }) => { const hold = screen.getByRole('button', 'Hold me'); await hold.tap(); @@ -64,6 +70,7 @@ test.describe('control inventory', () => { test('every exercise done shows the success message', async ({ app, screen }) => { await screen.getByLabel('Agree to terms').check(); await screen.getByRole('radio', 'Medium').check(); + await screen.getByRole('radio', 'Express').check(); await screen.getByRole('button', 'Hold me').longPress({ duration: 700 }); await screen.getByRole('button', 'Double-click me').doubleTap(); await screen.getByText('report.pdf', { exact: true }).secondaryTap(); diff --git a/docs/agents.mdx b/docs/agents.mdx index 9598d3076..0ddaca16a 100644 --- a/docs/agents.mdx +++ b/docs/agents.mdx @@ -109,9 +109,12 @@ agent if the problem affects several tests: 5. Compare models or `providerOptions` by running the same tests with `--agent `. -Named agents can use the [replay cache](/cache). Custom tools and executors -have their own replay constraints, described under [project tools](/tools) -and [custom executors](/executors#replay-cache). +Named agents can use the [replay cache](/cache). Each agent records its +own entries, keyed by its name and its context, so a test that runs as +`buyer` and as `admin` never replays one persona's steps for the other, and +changing `agentContext` records again. Custom tools and executors have their +own replay constraints, described under [project tools](/tools) and +[custom executors](/executors#replay-cache). ## Pick an agent per run diff --git a/docs/assertions.mdx b/docs/assertions.mdx new file mode 100644 index 000000000..2ab2d4dcc --- /dev/null +++ b/docs/assertions.mdx @@ -0,0 +1,94 @@ +--- +title: Assertions +description: Check the screen with agent.assert, wait for a condition, and extract data. +--- + +An assertion checks what is on screen. The agent has three judgment calls: +`assert` checks the screen now, `waitFor` waits until a condition holds, and +`extract` reads structured data. When you know the exact value, use an +`expect` [locator assertion](#locator-assertions) instead, with no model call. + +```ts +await agent.assert('the dashboard shows a trial badge'); +``` + +## `agent.assert` + +`assert` checks a statement against the current screen. A false statement +fails with `ASSERTION_FAILED`. If the screen does not provide enough evidence, +it fails with `ASSERTION_INCONCLUSIVE`. + +The judging model sees the statement and the current screen. It does not see +earlier steps or the acting agent's summaries. + +## `agent.waitFor` + +`assert` checks the screen now. Use `waitFor` when the condition may become true +later. It polls the screen and only calls the model when the observation changes. + +```ts +await agent.waitFor('the export finishes and a download link appears', { + interval: 500, + timeout: 120_000, +}); +``` + +## `agent.extract` + +`extract` reads structured data from the screen. Provide a +[Standard Schema](https://standardschema.dev) validator, such as Zod, to define +the result. + +```ts +import { expect } from 'e2e'; +import { z } from 'zod'; + +const data = await agent.extract('every todo title and how many remain', { + schema: z.object({ titles: z.array(z.string()), remaining: z.number().int() }), +}); +expect(data.titles).toContain('Buy milk'); +``` + + +For model-generated descriptions, check the relevant meaning rather than the +entire sentence. Use a [locator assertion](/locators) when the exact text matters. + + +## Judging pixels + +By default, judgments read a redacted text snapshot of the screen. Add +`vision: true` for visual checks, such as charts or layout. Use +`vision: 'only'` to judge the screenshot alone. + +```ts +await agent.assert('the chart trends upward', { vision: true }); +await agent.assert('the search form is not covered by a banner', { vision: 'only' }); +``` + +Images add input tokens. Screenshots are withheld for the rest of an attempt +after a secret is filled. The [agent reference](/reference/agent#vision) +lists the modes and restrictions. + +## Locator assertions + +`expect` on a locator checks an exact value and calls no model. It retries +until the check passes or the assertion timeout runs out. + +```ts +await expect(screen.getByRole('status')).toHaveText('2 remaining'); +await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible(); +``` + +[Locators](/locators) covers queries and actions. The +[expect reference](/reference/expect#locator-matchers) lists every matcher. + +## Next + + + + Exact actions and checks with `screen` and `expect`. + + + Verdicts, exit codes, and what the model sees. + + diff --git a/docs/authentication.mdx b/docs/authentication.mdx index 13a6ddc94..9a1cabd9e 100644 --- a/docs/authentication.mdx +++ b/docs/authentication.mdx @@ -211,7 +211,9 @@ export default { ``` `E2E_SECRET_` overrides an entry per run, with the same uppercasing rule -as credentials, so a CI job can supply the value without touching the config: +as credentials, so a CI job can supply the value without touching the config. +Two entries that map to one variable, like `stripe-key` and `stripe_key`, fail +the config load with `INVALID_CONFIG`, as two such credentials do: ```bash npm diff --git a/docs/browser.mdx b/docs/browser.mdx index bf4e4a5b6..da6bf58df 100644 --- a/docs/browser.mdx +++ b/docs/browser.mdx @@ -39,7 +39,9 @@ await app.open('/checkout'); await expect(screen.getByText('$42.00')).toBeVisible(); ``` -Each handler must call exactly one of `fulfill`, `continue`, or `abort`. An +Each handler must call exactly one of `fulfill`, `continue`, `fallback`, or +`abort`. The newest route runs first; `continue` sends the request straight to +the network, while `fallback` passes it to the route registered before. An `expect` that fails inside the handler fails the test, at the next step or at the end of the test when nothing follows. To inspect a real response, start waiting before the action that sends it: @@ -114,14 +116,33 @@ const flag = await browser.evaluate((name: string) => window.localStorage.getIte Use `evaluate` for state that `screen` cannot read. Prefer locators and assertions for visible UI so tests depend less on the DOM structure. +## Run a script before the page + +An init script runs in every document before the page's own scripts: mock a +browser API, seed `Math.random`, or set a flag the app reads at boot. For every +test: + +```ts +web({ initScripts: [{ path: 'tests/init.js' }] }) +``` + +For one test, before the page opens: + +```ts +await browser.addInitScript((value) => { Math.random = () => value; }, 0.5); +await app.open('/raffle'); +``` + +See [init scripts](/reference/web#init-scripts). + ## Cookies ```ts await browser.setCookies([{ url: 'http://localhost:3000', name: 'tz', value: 'UTC' }]); ``` -A cookie URL that does not parse, or uses `file:`, `data:`, or -`javascript:`, is `POLICY_DENIED`; a domain cookie is checked as the origin +A cookie URL that does not parse, or uses a scheme other than `http:` or +`https:` (`about:blank` included), is `POLICY_DENIED`; a domain cookie is checked as the origin it would be sent to. If any entry is refused, no cookies are set. To resize the page, see [Testing viewports](/viewports). @@ -144,8 +165,23 @@ export default { } satisfies E2EConfig; ``` -`npx playwright install webkit --with-deps` provisions the extra browser in -CI. `connect` attaches to a remote Chromium over CDP instead of launching +In CI, provision the extra browser with the [install command](/reference/web#installing-browsers): + + +```bash npm +npx @e2e-dev/web install webkit --with-deps +``` + +```bash pnpm +pnpm exec e2e-web install webkit --with-deps +``` + +```bash bun +bunx @e2e-dev/web install webkit --with-deps +``` + + +`connect` attaches to a remote Chromium over CDP instead of launching one; a [browser provider](#hosted-browsers) does the same and also leases and releases the browser. [Testing viewports](/viewports) covers sizes. @@ -237,7 +273,7 @@ The attempt runs in the browser's persistent default context, on the browser's own first tab, so the remote must contain only that context, and the limits of [`connect.reconnectEndpoint`](/reference/web#recovering-a-cdp-transport) -apply: no `headers`, `basicAuth`, or `userAgent` (each is `INVALID_CONFIG`), no +apply: no `headers`, `basicAuth`, `userAgent`, `locale`, or `timezoneId` (each is `INVALID_CONFIG`), no `app.clearState()` (`UNSUPPORTED_CAPABILITY` naming the scope), and no session state capture or restore, so a run whose tests consume a [session](/authentication) on the target fails with `COLLECTION_ERROR` diff --git a/docs/cache.mdx b/docs/cache.mdx index 217da3fcd..03cab32e1 100644 --- a/docs/cache.mdx +++ b/docs/cache.mdx @@ -45,27 +45,60 @@ bunx e2e run --no-cache ## How a replay works -1. Check the starting screen: the same route, whatever the record id, the - query, or the fragment. Retries always run live. +1. Check the starting screen: the same route, whatever the record id or the + fragment. Retries always run live. 2. Find each recorded control and repeat its action. The runner matches controls by their role, name, test id, and surrounding context. It waits for missing controls and stops if a match is ambiguous. A control with no - name is found by its place among the unnamed controls of its kind. -3. Check the final path and controls that appeared during the original step. - Text that changes with the data, a count, a date, an id, is not used. + name is found by its place among the unnamed controls of its kind inside + the same named row, list item, or group; with no such container to tell + it from its twins, the agent takes the step. +3. Check the final route and what the original step changed: the controls + that appeared, with their checked, selected, expanded, or pressed state, + and the ones that went away. At least one of those changes must happen + during the replay, counted from the first screen the replay saw on the + route it ended on, so an outcome already on screen before the actions + that should produce it proves nothing; a step whose last action moved to + another route has that move as its evidence. An alert the replay raised + that the recording never saw stops it too. Text that reads differently on + every run, a date, a time, a duration, an id, is not used, except in an + alert, where it is read as a placeholder. 4. Finish without a model call, or let the agent continue with a record of what ran and why replay stopped. +A route is the origin, the path, and the query. Ids, tokens, timestamps, +and cache busters in either the path or the query are read as an id +placeholder, so `/orders/42?t=1727780000` and `/orders/7?t=1727780999` are +one route. Every other query value is part of it: `?mode=safe` and +`?mode=unsafe`, or `/companies` and `/companies?tab=notes`, are different +screens, and so is a tracking parameter that was not there when the step +was recorded. Two routes that differ in one word, `/products/summer-sneaker` +and `/products/winter-boot`, are different screens too, even when they look +alike. A location on the app's own origin is compared by its path, so a +recording follows the app to a preview deployment that shares its +`app.identity`; a page on any other origin is another route. + +A count that names what it counts, `3 records imported` or `Showing 1 to 9 +of 9 results`, is part of the effect when the step made it appear on the +screen it acted on, and must read the same on replay. A count that was +already on screen and only moved, `41 items` to `42 items`, or one on a +screen the step navigated to, is not used, since leftover data from earlier +runs moves it again. A bare number such as a badge is never used. + Typed values replay as recorded, even when an earlier run left the same value on screen. Secret fills resolve the current value by name and run through authorization again. ## What must match -A cache entry belongs to a specific test, target, instruction, and set of -params. Renaming the test or target, changing the instruction or an ordinary -param, or upgrading the engine's major or minor version causes a miss. -Changing the model does not. +A cache entry belongs to a specific test, target, instruction, set of +params, and agent. Renaming the test or target, changing the instruction or +an ordinary param, running with another configured agent, changing the +agent's `context` or the test's `agentContext`, or upgrading the engine's +major or minor version causes a miss. A test that runs once per agent +(`agent: ['buyer', 'admin']`) records once per agent. Changing the model +does not. The agent context enters the key as a hash of its redacted text, +so a secret in it contributes only its name. A timestamp or a fresh email address would cause a miss on every run. Wrap values like these in `unique()` so the runner substitutes the current value @@ -96,7 +129,9 @@ identity. On devices, replay uses the In `read-write` mode, the runner saves a recording at the end of the attempt when a later verification step passed. A step that took no action -records nothing. +records nothing, and neither does one that changed nothing the replay could +check: no control appeared, went away, or changed state, and the route +stayed the same. | Counts as verification | Does not count | | --- | --- | @@ -145,14 +180,14 @@ such as an `afterEach` assertion. | `retry` | Retries never replay. | Nothing. | | `invalid-entry` | The file is unreadable, over 1 MiB, or from another schema version. | `npx e2e cache clear`. | | `truncated` | The recording hit a limit, see [below](#limits-and-gaps). | Split the step, or pass the secret as a `Secret`. | -| `wrong-context` | The app was on another screen than when the step was recorded: a different route, not a different record, query, or fragment. | Open the same screen before the step. | -| `target-not-found` | A recorded control did not appear within 15 seconds. | The UI changed. The next passing run re-records. | +| `wrong-context` | The app was on another screen than when the step was recorded: a different origin, path, or query, not a different record id or fragment. | Open the same screen before the step. | +| `target-not-found` | A recorded control did not appear within 15 seconds, or it has no name and no named container to tell it from its twins. | The UI changed: the next passing run re-records. For an unnamed control, give it a label or put it in a named row or group. | | `target-ambiguous` | The number of matching controls changed. | Give repeated controls distinct names or use unique test data. | | `gap` | An action cannot be replayed, see [below](#limits-and-gaps). | The agent continues this part live. Put stable values in `params`. | | `action-failed` | The app rejected a replayed action. | Check the app. | | `action-uncertain` | A replayed action failed with an unknown commit state. The agent is told to check before acting again. | Check the app. | | `viewport-changed` | A bare-point tap or hover was recorded on a viewport of another size. | Keep the viewport fixed. | -| `end-mismatch` | Every action ran, but the recorded path or a control that should have appeared did not come back. | Leftover data from an earlier run, a banner that comes and goes, or a regression. | +| `end-mismatch` | Every action ran, but the recorded route, a control that should have appeared or gone away, or a recorded state did not come back; the outcome was already on screen before the replay; or the replay raised an alert the recording never saw. | Leftover data from an earlier run, a banner that comes and goes, or a regression. | A stop before any action ran is a miss. After replay starts, it is a hand-off. Unique test data and cleanup in `afterEach` help avoid stale @@ -176,8 +211,9 @@ the instruction or `params`, or ones the agent composed itself, replay. Each entry is a JSON file under `.e2e/cache/`, named by a hash of its key. It contains actions, target descriptions, typed text, a step summary, and -checks for the final state. Those checks include the path and up to eight -controls that appeared during the step. +checks for the final state. Those checks include the route, up to eight +controls that appeared during the step, alerts and status messages first, +and up to eight that went away. Entries contain no prompts, conversations, screenshots, observed trees, or secret values. A secret fill stores only the secret's name. Other typed @@ -220,12 +256,16 @@ Local runs leave a committed directory clean unless something changed. A replayed step never rewrites its entry, and a live re-recording with the same actions, paths, and anchors is not written: the summary, the measured end wait, and the rule that flagged a typed run-time value differ between two -live runs of the same flow and do not count. A file changes when: +live runs of the same flow and do not count. Upgrading `e2e` to a version +that changes how entries are matched re-keys every entry, so the first run +after it records them again. A file changes when: - a step records for the first time, - the agent finishes a step after a hand-off, re-recording what it saw, - a step fails after replaying, or ends in an `end-mismatch` the agent had to repair, which deletes the entry, +- a step with an entry passes without leaving anything to record, which + deletes the entry, - no verification step confirmed the entry by the end of the attempt, which deletes it. @@ -257,8 +297,14 @@ CI=1 bunx e2e run --strict-cache The reasons that fail are the ones that mean the app or the entry changed: `invalid-entry`, `wrong-context`, `target-not-found`, `target-ambiguous`, `viewport-changed`, `action-failed`, `action-uncertain`, and `end-mismatch`. -A step with no recording (`no-entry`), a retry, a `gap`, and a `truncated` -recording still run live. `REPLAY_STALE` exits 2 and is not retried, since a +A `no-entry` miss fails too when the cache directory holds a recording of the +same step under another key, from before an `e2e` or engine upgrade or a +change to the project id, the app's identity, or the agent's context. The +message names that file; delete it once nothing replays it. A custom +`cache.store` is not checked this way. Entries recorded before this check +existed are covered once a `read-write` run replays or re-records them. A +step never recorded (a new or edited instruction or params), a retry, a +`gap`, and a `truncated` recording still run live. `REPLAY_STALE` exits 2 and is not retried, since a retry never replays and would pass live. The entry stays where it is, even in `read-write` mode, so every strict run fails on it until it is re-recorded: run the test in `read-write` mode without the flag, and with `cache.strict` diff --git a/docs/ci.mdx b/docs/ci.mdx index 6d64efbcc..d9d44987d 100644 --- a/docs/ci.mdx +++ b/docs/ci.mdx @@ -35,7 +35,7 @@ jobs: - run: pnpm install --frozen-lockfile - name: Install browsers - run: npx playwright install chromium --with-deps + run: pnpm exec e2e-web install chromium --with-deps - name: Run e2e run: npx e2e run --reporter list,junit @@ -66,7 +66,9 @@ jobs: ``` The workflow pins actions to commit SHAs and installs dependencies from the -lockfile. `npx e2e` runs the installed version. +lockfile. `npx e2e` runs the installed version. The +[install command](/reference/web#installing-browsers) downloads the browsers +for the Playwright version the engine pins, with their system libraries. Artifacts upload on every run that is not cancelled, so a test that failed and then passed on retry still has its evidence. @@ -139,10 +141,12 @@ implemented yet. `--last-failed` runs only the tests the previous run did not pass, read from `report.json` in the [`output`](/reference/config#output) directory (`.e2e` -by default). The rerun empties `artifacts/` there when it starts, so -upload the first run's artifacts before rerunning in the same job if you want -both. The [CLI reference](/reference/cli#flags) lists what counts as not -passed. Locally it is the loop after a red run: +by default). The rerun keeps the artifacts the first run's report names and +writes its own under `artifacts/rerun-1/`, so one upload after both steps +holds the evidence of each. A test another filter keeps out of the rerun +stays owed: the next `--last-failed` runs it. The +[CLI reference](/reference/cli#flags) lists what counts as not passed. +Locally it is the loop after a red run: ```bash npm @@ -193,7 +197,7 @@ flaky, so you can identify flows that need investigation. `--max-failures ` stops the run once that many tests have failed, so a broken deploy does not spend a model call on every remaining agent step. The tests not started are skipped with cause `failure-limit`, the ones running -end as `interrupted`, and the job exits 1. +end as `interrupted`, counted apart from the failures, and the job exits 1. ## Exit codes diff --git a/docs/debugging.mdx b/docs/debugging.mdx index ec34b1d08..96a7f4b43 100644 --- a/docs/debugging.mdx +++ b/docs/debugging.mdx @@ -63,7 +63,9 @@ URL: `http://127.0.0.1:3000/todos` `Look at` points to the failing call. Assertion failures include expected and observed values; locator failures include the requested locator and -nearby matches. Repeated failure details help you compare attempts. +nearby matches. Repeated failure details help you compare attempts. The +runner spends at most 5 s capturing the screen and screenshot once per failed +attempt; an app that does not answer in time gets no screen section. ## The report @@ -91,7 +93,9 @@ failed result: Artifacts land under `.e2e/artifacts/` (`/artifacts/` when the [`output`](/reference/config#output) is moved), one directory per attempt. -Each run empties it first, so it holds the latest run's evidence: +Each run empties it first, so it holds the latest run's evidence. A +`--last-failed` rerun instead empties it down to the files the report it +reruns names, and writes its own under `rerun-/`: - **Screenshots.** Failed attempts capture one whenever the engine can and pixels are allowed. The built-in `agent.assert` also captures one after @@ -99,12 +103,13 @@ Each run empties it first, so it holds the latest run's evidence: be withheld. Use `app.screenshot('label')` to capture one explicitly. - **Playwright traces** on the attempts the [`trace`](/reference/config#trace) mode records: every attempt locally, the first retry in CI. Open one with - `npx playwright show-trace `; `--trace off` skips the cost for one - run, `--trace` records every attempt of a CI run. + `npx playwright-core@1.63.0 show-trace ` (`pnpm dlx` under pnpm); + `--trace off` skips the cost for one run, `--trace` records every attempt + of a CI run. - **Downloads** saved through `browser.waitForDownload`, recorded with kind - `download`. They are the app's bytes: `redaction: "incomplete"` unless a - secret reached the session (filled, or held by the engine for basic auth) - and the file is text, which the runner rewrites and marks `complete`. + `download`. They are the app's bytes: `redaction: "incomplete"` unless the + session knows a secret value and the file is text, which the runner rewrites and + marks `complete`. - **Videos** and **agent transcripts** when you ask for them; see below. After a secret fill, no screenshots are taken for the rest of the attempt; see diff --git a/docs/decision-models.mdx b/docs/decision-models.mdx new file mode 100644 index 000000000..23b84ef12 --- /dev/null +++ b/docs/decision-models.mdx @@ -0,0 +1,138 @@ +--- +title: Decision models +description: Run bounded actions with a decision model and a small text model instead of a full LLM agent. +--- + +`@e2e-dev/decision` runs `agent.act` and `agent.assert` through a decision +model instead of an LLM. For each action the decision model answers one +request: which operation, and which element for each operation that has +more than one, chosen from what the executor builds out of the latest +semantic tree. It never generates free text: when the choice is `type`, a +small language model writes the field value. Both models are +plain AI SDK models, so any evaluation provider that answers `choice` +questions with probability distributions (Jev) and any language-model +provider works; the runner still authorizes every dispatched action. + +## Install and configure + + +```bash npm +npm install -D @e2e-dev/decision +``` + +```bash pnpm +pnpm add -D @e2e-dev/decision +``` + +```bash bun +bun add -d @e2e-dev/decision +``` + + +The decision model answers `choice` questions with probability +distributions; the text model writes field values. Each installs as its +own package: + + +```bash npm +npm install -D @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider +``` + +```bash pnpm +pnpm add -D @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider +``` + +```bash bun +bun add -d @ai-sdk/typesafe-ai @openrouter/ai-sdk-provider +``` + + +```ts title="e2e.config.ts" +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { decisionExecutor } from '@e2e-dev/decision'; +import { typeSafeAi } from '@ai-sdk/typesafe-ai'; +import { openrouter } from '@openrouter/ai-sdk-provider'; + +export default { + targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }], + agents: { + default: { + executor: decisionExecutor({ + model: typeSafeAi.evaluationModel('jev-latest'), + textModel: openrouter('inception/mercury-2.5'), + }), + }, + }, +} satisfies E2EConfig; +``` + +Tests stay plain natural language, with no params needed: + +```ts title="tests/todos.e2e.ts" +import { test, expect } from 'e2e'; + +test('adds a todo', async ({ app, screen, agent }) => { + await app.open('/todos'); + await agent.act('Add a todo named Buy coffee, and verify it is listed.'); + await expect(screen.getByTestId('todo')).toContainText('Buy coffee'); + await agent.assert('The todo list contains Buy coffee.'); +}); +``` + +## How text gets typed + +The decision model only ever names an element index. When it picks `type`, +the text model writes the value from the goal, the field's label, and the +page text, and the executor types exactly that string. Model output never +becomes a selector, a URL, or a key. When the goal supplies no value, +nothing is typed and the decision model picks another action. Password +fields never reach the text model; secrets stay declared handles that only +`typeSecret` fills. + +The text model is also the agent's judgment tier: it is exposed as the +executor's `model`, so `waitFor` and `extract` use it too. It can come +from the agents entry's `model` instead. Without either, `type` is never +offered. + +## Finishing a step + +When the decision model answers `done` or `failed`, the executor checks +the claim on a fresh screen with one question: is the step's task complete? +It sees the step's non-secret params, the actions taken with the non-secret +values typed, and the page, including the text of named status regions, but +no model reasoning or secret. A confirmed `done` passes the +step and a confirmed `failed` fails it with `ACTION_FAILED`. An +unconfirmed claim goes back to the decision model as history; the second one +ends the step, as `ACTION_FAILED` when the screen contradicts the goal and +as `blocked` (`AUTOMATION_UNSUPPORTED`) otherwise. + +## Gates + +Both gates are off by default: the executor follows the most probable +choice. Set `minProbability` or `minConfidence` to block weaker actions, +reject weaker completion checks, and fail weaker assertions as +inconclusive instead: + +```ts title="Executor setup" +const executor = decisionExecutor({ model, minProbability: 0.9, minConfidence: 0.8 }); +``` + +Probability and confidence are different provider fields: probability is the +selected option's share of the answer distribution, confidence is the +provider's separate statistic. A threshold is a policy choice; measure +accuracy on your own tests before changing it. + +## Limits + +- No vision: screenshots never reach the model; canvas-only controls and + image-based targets are not implemented. +- No `navigate`: in-app navigation is a `tap` on a link. URLs belong in + test code (`browser.goto`, `app.open`). +- No multi-select `listbox` in v1. +- The choice cap is 255 options per question (TypeSafe's limit). Larger + screens list viewport nodes first and count the rest as omitted, so the + model can scroll; the step never blocks just for being large. +- Drag, upload, and hover are not offered in v1. +- `waitFor` and `extract` use the text model through the runner's + built-in judgment tier. diff --git a/docs/docs.json b/docs/docs.json index e86010dd9..30906c169 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -134,7 +134,9 @@ "pages": [ "index", "quickstart", - "writing-tests" + "goals", + "assertions", + "locators" ] }, { @@ -150,7 +152,8 @@ "group": "Mobile", "icon": "mobile-screen", "pages": [ - "mobile" + "mobile", + "physical-devices" ] }, { @@ -165,6 +168,7 @@ "icon": "wand-magic-sparkles", "pages": [ "models", + "decision-models", "subscriptions", "agents", "agent-steps", @@ -340,6 +344,10 @@ { "source": "/reference/app-web", "destination": "/reference/web" + }, + { + "source": "/writing-tests", + "destination": "/goals" } ] } diff --git a/docs/examples/models/e2e.decision.config.ts b/docs/examples/models/e2e.decision.config.ts new file mode 100644 index 000000000..52c869c8b --- /dev/null +++ b/docs/examples/models/e2e.decision.config.ts @@ -0,0 +1,17 @@ +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { decisionExecutor } from '@e2e-dev/decision'; +import { typeSafeAi } from '@ai-sdk/typesafe-ai'; +import { openrouter } from '@openrouter/ai-sdk-provider'; + +export default { + targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }], + agents: { + default: { + executor: decisionExecutor({ + model: typeSafeAi.evaluationModel('jev-latest'), + textModel: openrouter('inception/mercury-2.5'), + }), + }, + }, +} satisfies E2EConfig; diff --git a/docs/examples/models/tests/decision.e2e.ts b/docs/examples/models/tests/decision.e2e.ts new file mode 100644 index 000000000..f769596f4 --- /dev/null +++ b/docs/examples/models/tests/decision.e2e.ts @@ -0,0 +1,8 @@ +import { test, expect } from 'e2e'; + +test('adds a todo', async ({ app, screen, agent }) => { + await app.open('/todos'); + await agent.act('Add a todo named Buy coffee, and verify it is listed.'); + await expect(screen.getByTestId('todo')).toContainText('Buy coffee'); + await agent.assert('The todo list contains Buy coffee.'); +}); diff --git a/docs/executors.mdx b/docs/executors.mdx index 773692323..8ec3fb785 100644 --- a/docs/executors.mdx +++ b/docs/executors.mdx @@ -14,6 +14,8 @@ not enough. There are two options: `ai` package installed. Both receive a deadline, redacted observations, and checked action methods. +For a provider that returns bounded choices instead of generated text, see +the [Clef and Jev decision executor](/decision-models). The runner records their verdicts. Either goes under `executor` in an agents entry, beside the options that still apply: `model`, `judge`, `context`, and the budgets. An entry with `executor` rejects `system` and `tools`; pass @@ -128,7 +130,7 @@ The executor receives this context: | Member | What it is | | --- | --- | -| `ctx.step` | The instruction, params, and `index` in the attempt's timeline | +| `ctx.step` | The instruction, params, and `index` in the attempt's timeline. A registered secret value the test spelled into the instruction or a param string reads ``; a `secrets.get()` handle is a placeholder you fill with `typeSecret` | | `ctx.observe({ tree, pixels })` | Redacted screen text, with an optional node tree and masked screenshot | | `ctx.actions` | Checked and recorded actions: `tap`, `doubleTap`, `longPress`, `secondaryTap`, `hover`, `type`, `typeSecret`, `press`, `select`, `check`, `drag`, `scrollTo`, `scrollUntil`, `upload`, `scroll`, `navigate`, `back`, and the point and keyboard verbs below | | `ctx.target` | The target's `name` and `platform`, and `verbs`, the actions its engine supports. Offer only these to the model. | @@ -215,8 +217,10 @@ default, follows the configured cache mode. A recorded step replays without calling `runStep`. A replay that stops partway calls `runStep` with `ctx.replayedPrefix`: continue from the current screen and do not redo the replayed actions. What a step does through `ctx.actions` is recorded; a -mutating tool call leaves a gap the executor fills on replay. Entries are not -keyed by executor, so an entry one agent recorded replays for another. +mutating tool call leaves a gap the executor fills on replay. Entries are +keyed by the configured agent and its context, so an entry one agent +recorded never replays for another; the executor itself is not part of the +key, so changing an agent's executor keeps its entries. `'off'` never replays or records the executor's steps, so every step reaches `runStep`. Use it for a brain whose own history or memory is the source of diff --git a/docs/github.mdx b/docs/github.mdx index f7570579a..1c35b9d1a 100644 --- a/docs/github.mdx +++ b/docs/github.mdx @@ -63,7 +63,10 @@ Upload artifacts even when a retry passes. Each failure's evidence links open the run's Artifacts section; inside the download, every test has a directory named after it and the agent that ran it (`web/tests_billing.e2e.ts__upgrades/default/attempt-0/`), so keep -`path: .e2e/artifacts` and the download lines up with the report. A project +`path: .e2e/artifacts` and the download lines up with the report. A +`--last-failed` rerun keeps the first run's directories and puts its own +under `rerun-1/`, so a flaky test's failed attempt still links to its own +files. A project below the checkout root, such as `packages/e2e-tests`, gets source links under that path. The checkout root is the nearest directory holding `.git` inside `GITHUB_WORKSPACE`, so `actions/checkout` with a `path` input works too. @@ -143,9 +146,14 @@ passes: The comment covers the whole suite, not only the tests the rerun ran again. A test that failed and then passed shows as flaky, a test that failed twice -stays failed, and the rest keep their first-pass results. The comment -matches the job's status: it stays red when another filter kept a failed -test out of the rerun. +stays failed, one that was interrupted and then passed shows as passed, and +the rest keep their first-pass results. The comment stays red while the +rerun left out something it owed: a failed test, one `--max-failures` stopped +before it ran, or a failed `beforeAll` or `afterAll` whose tests another +filter kept out of the rerun. It shows ⏹️ when the rerun left out a test the +first pass never finished. The rerun's own exit code counts only what it ran. +A failed attempt's evidence links to the first run's files, which the rerun +keeps. ## Matrix jobs diff --git a/docs/writing-tests.mdx b/docs/goals.mdx similarity index 58% rename from docs/writing-tests.mdx rename to docs/goals.mdx index 9d2d2e62a..217b6c868 100644 --- a/docs/writing-tests.mdx +++ b/docs/goals.mdx @@ -1,10 +1,11 @@ --- -title: Writing tests -description: Goals, checks, extraction, and the deterministic APIs. +title: Goals +description: Write test steps as goals with agent.act, and structure the tests around them. --- -Use agent calls for goals and questions about the screen. Use `screen` and -`expect` when you know the exact action or value to check. A test can freely mix both. +Use agent calls for goals and questions about the screen. Use +[locators](/locators) when you know the exact action or value to check. A test +can freely mix both. ```ts title="tests/todos.e2e.ts" import { test, expect } from 'e2e'; @@ -23,7 +24,9 @@ test('todos survive a filter round-trip', async ({ app, agent, screen }) => { ## Goals: `agent.act` Give `act` one goal. The agent chooses and performs the actions, then returns -when the goal passes or throws when it fails. +when the goal passes or throws when it fails. Await every agent and screen +call, and every `expect.poll`; a test that returns while one is still running +fails with `STEP_NOT_AWAITED`. ```ts await agent.act('complete checkout with the test card'); @@ -68,60 +71,9 @@ A step can pass, fail, or be blocked by something such as missing credentials. [How agent steps work](/agent-steps#how-a-step-ends) explains the verdicts and exit codes. -## Checks: `assert`, `waitFor`, `extract` - -`assert` checks a statement against the current screen. A false statement -fails with `ASSERTION_FAILED`. If the screen does not provide enough evidence, -it fails with `ASSERTION_INCONCLUSIVE`. - -The judging model sees the statement and the current screen. It does not see -earlier steps or the acting agent's summaries. - -```ts -await agent.assert('the dashboard shows a trial badge'); -``` - -`assert` checks the screen now. Use `waitFor` when the condition may become true -later. It polls the screen and only calls the model when the observation changes. - -```ts -await agent.waitFor('the export finishes and a download link appears', { - interval: 500, - timeout: 120_000, -}); -``` - -`extract` reads structured data from the screen. Provide a -[Standard Schema](https://standardschema.dev) validator, such as Zod, to define -the result. - -```ts -import { z } from 'zod'; - -const data = await agent.extract('every todo title and how many remain', { - schema: z.object({ titles: z.array(z.string()), remaining: z.number().int() }), -}); -expect(data.titles).toContain('Buy milk'); -``` - - -For model-generated descriptions, check the relevant meaning rather than the -entire sentence. Use a locator assertion when the exact text matters. - - -### Judging pixels - -By default, judgments read a redacted text snapshot of the screen. Add -`vision: true` for visual checks, such as charts or layout. Use -`vision: 'only'` to judge the screenshot alone. - -```ts -await agent.assert('the chart trends upward', { vision: true }); -await agent.assert('the search form is not covered by a banner', { vision: 'only' }); -``` - -`act` requests screenshots itself when it needs them, so it has no `vision` -option. On supported engines, it can tap and type at points in the image: +`act` requests screenshots itself when it needs them, so unlike the +[judgments](/assertions#judging-pixels) it has no `vision` option. On +supported engines, it can tap and type at points in the image: ```ts await agent.act('enter the code 3141 on the drawn keypad and press OK'); @@ -131,44 +83,8 @@ Images add input tokens. Screenshots are withheld for the rest of an attempt after a secret is filled. The [agent reference](/reference/agent#vision) lists the modes and restrictions. -## Deterministic APIs: `screen` and `expect` - -Use locators to interact with known controls and check exact values. -An `expect` after `act` checks the exact result. It is also what lets the -[replay cache](/cache) record the step and keep it. - -```ts -// Semantic queries: role and name first, label second, test ID last. -await screen.getByRole('button', 'Sign in').tap(); -await screen.getByLabel('Email').fill('ada@example.test'); -await screen.getByTestId('todo').first().tap(); - -// Scope into a row, gesture like a user: -const row = screen.getByRole('listitem').filter({ hasText: 'Design review' }); -await row.getByRole('button', 'Archive').tap(); -await screen.swipe({ direction: 'up', momentum: 'fast' }); - -// Coordinates, for what the tree does not list: -await screen.getByRole('image', 'Seat map').tap({ position: { x: 120, y: 48 } }); -await screen.swipe({ from: { x: 300, y: 400 }, to: { x: 40, y: 400 } }); - -// Polling assertions: -await expect(screen.getByRole('status')).toHaveText('2 remaining'); -await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible(); -``` - -Actions and single-node reads or assertions fail when a query matches more -than one node. Narrow it with `.filter()`, `.first()`, or `.nth()`. -Locator assertions retry until they pass or time out. Direct reads such as -`textContent()` do not retry a mismatched value. - -Use the `browser` fixture for browser features such as routes, cookies, dialogs, -frames, and downloads; see [Working with the browser](/browser). The -[screen reference](/reference/screen) lists every query and action. - -Share setup and cleanup through `test.extend`. A fixture can create a test -workspace before the test and delete it afterward, including when the test -fails. See [Your own fixtures](/reference/test#your-own-fixtures). +To check what a goal left on screen, follow it with an +[assertion](/assertions) or a [locator check](/locators). ## Tests without a screen @@ -220,6 +136,10 @@ fixtures or hooks. When a flow must span several tests, mark the group `{ serial: true }`. Members share app state, run in order on one worker, and retry as a whole. +Share setup and cleanup through `test.extend`. A fixture can create a test +workspace before the test and delete it afterward, including when the test +fails. See [Your own fixtures](/reference/test#your-own-fixtures). + ## Run one test @@ -247,10 +167,10 @@ The [CLI reference](/reference/cli#e2e-run) lists every flag. ## Next - - Sessions and secrets. + + Check the screen with `assert`, `waitFor`, and `extract`. - - Several agents in one suite, and running a flow as each of them. + + Exact actions and checks with `screen` and `expect`. diff --git a/docs/index.mdx b/docs/index.mdx index bc532bbe3..0a810e814 100644 --- a/docs/index.mdx +++ b/docs/index.mdx @@ -9,9 +9,9 @@ e2e is a next-generation testing framework for web and mobile apps, built to keep up with the speed we ship software today. You choose how agentic each test is: fully deterministic with -[locators](/writing-tests#deterministic-apis-screen-and-expect) and +[locators](/locators) and [assertions](/reference/expect#locator-matchers), fully driven by an agent -from [goals](/writing-tests#goals-agent-act) in natural language, or a mix of +from [goals](/goals) in natural language, or a mix of both in the same test. One prompt for your coding agent, or set up by hand. - - Write goals, check outcomes, and extract data. + + Write test steps as goals in natural language. Let your coding agent write, run, and debug tests. diff --git a/docs/integrations/kernel.mdx b/docs/integrations/kernel.mdx index 247bdcd1d..5e3107398 100644 --- a/docs/integrations/kernel.mdx +++ b/docs/integrations/kernel.mdx @@ -90,7 +90,7 @@ cookies and storage are not there. that browser's default context, on its own first tab, so a `profile` applies and the live view shows one tab. It costs a browser start per test and carries the limits of a [per-attempt lease](/browser#hosted-browsers): no -`headers`, `basicAuth`, or `userAgent`, no `app.clearState()`, and no +`headers`, `basicAuth`, `userAgent`, `locale`, or `timezoneId`, no `app.clearState()`, and no [sessions](/authentication), which fail the run before any test. ## Recordings diff --git a/docs/locators.mdx b/docs/locators.mdx new file mode 100644 index 000000000..1cdabe914 --- /dev/null +++ b/docs/locators.mdx @@ -0,0 +1,64 @@ +--- +title: Locators +description: Deterministic actions and checks with screen and expect. +--- + +Use locators to interact with known controls and check exact values. Locator +steps call no model. + +```ts title="tests/todos.e2e.ts" +import { test, expect } from 'e2e'; + +test('adds a todo', async ({ app, screen }) => { + await app.open('/todos'); + await screen.getByLabel('New todo').fill('Buy milk'); + await screen.getByRole('button', 'Add').tap(); + await expect(screen.getByTestId('todo')).toHaveCount(1); +}); +``` + +An `expect` after an [`agent.act` goal](/goals) checks the exact result. It is +also what lets the [replay cache](/cache) record the step and keep it. + +## Queries, actions, and assertions + +```ts +// Semantic queries: role and name first, label second, test ID last. +await screen.getByRole('button', 'Sign in').tap(); +await screen.getByLabel('Email').fill('ada@example.test'); +await screen.getByTestId('todo').first().tap(); + +// Scope into a row, gesture like a user: +const row = screen.getByRole('listitem').filter({ hasText: 'Design review' }); +await row.getByRole('button', 'Archive').tap(); +await screen.swipe({ direction: 'up', momentum: 'fast' }); + +// Coordinates, for what the tree does not list: +await screen.getByRole('image', 'Seat map').tap({ position: { x: 120, y: 48 } }); +await screen.swipe({ from: { x: 300, y: 400 }, to: { x: 40, y: 400 } }); + +// Polling assertions: +await expect(screen.getByRole('status')).toHaveText('2 remaining'); +await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible(); +``` + +Actions and single-node reads or assertions fail when a query matches more +than one node. Narrow it with `.filter()`, `.first()`, or `.nth()`. +Locator assertions retry until they pass or time out. Direct reads such as +`textContent()` do not retry a mismatched value. + +Use the `browser` fixture for browser features such as routes, cookies, dialogs, +frames, and downloads; see [Working with the browser](/browser). The +[screen reference](/reference/screen) lists every query and action, and the +[expect reference](/reference/expect#locator-matchers) lists every matcher. + +## Next + + + + Sessions and secrets. + + + Several agents in one suite, and running a flow as each of them. + + diff --git a/docs/migrate/cypress.mdx b/docs/migrate/cypress.mdx index 8c3d0f700..686a21cc4 100644 --- a/docs/migrate/cypress.mdx +++ b/docs/migrate/cypress.mdx @@ -115,8 +115,8 @@ Start with a spec that often breaks and whose fixed `cy.wait` calls can become assertions on the condition the test needs. - - Goals, checks, extraction, and the deterministic APIs. + + Write test steps as goals instead of scripted selectors. Routes, dialogs, downloads, iframes, and page evaluation. diff --git a/docs/migrate/maestro.mdx b/docs/migrate/maestro.mdx index 159ccf120..c293363c7 100644 --- a/docs/migrate/maestro.mdx +++ b/docs/migrate/maestro.mdx @@ -223,7 +223,7 @@ often need updating. Configure your app and devices. - - Goals, checks, extraction, and the deterministic APIs. + + Write test steps as goals instead of scripted selectors. diff --git a/docs/migrate/playwright.mdx b/docs/migrate/playwright.mdx index 1c2beeebe..bc953d776 100644 --- a/docs/migrate/playwright.mdx +++ b/docs/migrate/playwright.mdx @@ -3,8 +3,8 @@ title: From Playwright description: Run e2e alongside Playwright and migrate tests one file at a time. --- -`@e2e-dev/web` uses your installed Playwright when its version satisfies -the engine's peer dependency (`>=1.63.0 <2`). Both runners can stay in one +`@e2e-dev/web` brings its own pinned `playwright-core`, so your +`@playwright/test` version is yours to keep. Both runners can stay in one project. e2e reads `e2e.config.ts` and finds `tests/**/*.e2e.ts` by default, so keep the two file patterns apart. @@ -59,7 +59,8 @@ A Playwright key the runner recognizes (`testDir`, `baseURL`, `webServer`, | `use.trace` | `trace` with the same modes (`'off'`, `'on'`, `'retain-on-failure'`, `'on-first-retry'`, `'on-all-retries'`), at the top of the config, per target, `test('x', { trace }, ...)`, or `--trace [mode]` for one run; the default differs, see [below](#defaults-that-differ) | same | | `use.video` | `video` with the same modes as `trace`, at the top of the config, per target in place of a project, `test('x', { video }, ...)` in place of `test.use`, or `--video [mode]` for one run | same | | `use.headless: false` | `--headed` on the CLI | renamed | -| `use.locale`, `timezoneId`, `colorScheme`, `geolocation`, `permissions` | not yet | missing | +| `use.locale`, `use.timezoneId` | `web({ locale, timezoneId })` | renamed | +| `colorScheme`, `geolocation`, `permissions` | not yet | missing | | `use.userAgent` | `web({ userAgent })` | renamed | | `devices['iPhone 15']`, `deviceScaleFactor`, `isMobile`, `hasTouch` | not yet; `web({ viewport })` sets the size only | missing | | `ignoreHTTPSErrors`, `bypassCSP`, `offline`, `launchOptions`, `proxy` | not yet | missing | @@ -139,7 +140,9 @@ what to do today. | `fill(value)` | the same; also takes a `Secret` | same | | `pressSequentially(text, { delay })` | the same; keystrokes through the platform keyboard, a `Secret` is refused | same | | `press(key)` | the same; modifiers join with `+`, see the [key grammar](/reference/engine#keys) | same | -| `clear()`, `check()`, `uncheck()`, `focus()`, `hover()`, `selectOption()`, `setInputFiles()`, `dragTo()` | the same | same | +| `clear()`, `focus()`, `hover()`, `setInputFiles()`, `dragTo()` | the same | same | +| `check()`, `uncheck()` | the same, except for a control the click removes or navigates away from, such as a radio the app replaces with its selected view: the click landed, so the action passes, even when the app renders an unchecked control in its place. Playwright fails it because it cannot read the state back. Assert on what replaced the control | by design | +| `selectOption('pro')` | `selectOption('Pro')`: a string matches the option's label only, where Playwright matches its value or its label; `selectOption({ value: 'pro' })` for the value | by design | | `selectOption(['a', 'b'])` | not yet; one option per call. An array is `INVALID_ARGUMENT` | missing | | `setInputFiles({ name, buffer })` | not yet; paths only, resolved from the project root | missing | | `scrollIntoViewIfNeeded()` | `scrollIntoView()` | renamed | @@ -149,8 +152,7 @@ what to do today. | `isChecked()` | the same on a checkbox, radio, or `aria-checked` node; `false` on any other node, where Playwright throws | by design | | `isHidden()`, `isDisabled()` | the same | same | | `isEditable()` | not yet | missing | -| `waitFor({ state: 'visible' })`, `'hidden'` | the same | same | -| `waitFor({ state: 'attached' })`, `'detached'` | not yet; `expect(locator).toBeAttached()` and `.not.toBeAttached()` | missing | +| `waitFor({ state: 'visible' })`, `'hidden'`, `'attached'`, `'detached'` | the same | same | | `locator.evaluate(fn)`, `evaluateAll(fn)` | not yet; `browser.evaluate(fn, selector)` with a CSS selector argument | missing | | `locator.screenshot()` | not yet; `app.screenshot(label)` captures the viewport | missing | | `innerHTML()`, `dispatchEvent()`, `selectText()`, `blur()`, `highlight()` | not yet; `browser.evaluate` | missing | @@ -173,14 +175,14 @@ what to do today. | `toHaveText(['a', 'b'])`, `toContainText([...])` on several matches | the same: `toHaveText` needs the count to match, `toContainText` an ordered subsequence | same | | `toBeChecked({ checked: false })`, `toBeEnabled({ enabled })`, `toBeVisible({ visible })`, `toBeAttached({ attached })` | the same | same | | `{ ignoreCase }` on `toHaveText()`, `toContainText()`, `toHaveAccessibleName()`, `toHaveAttribute(name, value)` | the same | same | -| `toBeChecked({ indeterminate })`, `{ useInnerText }`, `toHaveURL(url, { ignoreCase })` | not yet; `INVALID_ARGUMENT`, so the matcher never passes on its default | missing | +| `toBeChecked({ indeterminate })`, `{ useInnerText }` | not yet; `INVALID_ARGUMENT`, so the matcher never passes on its default | missing | | `toHaveAttribute('aria-selected', 'true')`, `'aria-expanded'` | `toBeSelected()`, `toBeExpanded()` | renamed | | a positive matcher on several matches | `LOCATOR_AMBIGUOUS` at once; narrow with `filter`, `first`, or `{ visible: true }` | by design | | `toBeAttached()` | the same; passes on a hidden match too | same | | `toBeEmpty()`, `toBeInViewport()`, `toBeEditable()`, `toHaveRole()`, `toHaveAccessibleDescription()` | not yet | missing | | `expect(locator).toHaveClass(cls)` | `expect(browser).toHaveClass(locator, cls)`; a DOM fact is asserted through `browser`; reads the `class` attribute, so `''` matches `class=""` and not a missing attribute | web only | | `toHaveCSS()`, `toHaveId()`, `toHaveJSProperty()`, `toContainClass()`, `toHaveValues()` | not yet; `browser.evaluate` plus a value matcher | missing | -| `expect(page).toHaveURL()`, `toHaveTitle()` | `expect(browser).toHaveURL()`, `toHaveTitle()` | web only | +| `expect(page).toHaveURL()`, `toHaveTitle()` | `expect(browser).toHaveURL()`, `toHaveTitle()`; `toHaveURL` takes `ignoreCase` | web only | | `toHaveScreenshot()`, `toMatchSnapshot()` | not yet; `agent.assert('...', { vision: true })` is a judgment, not a pixel diff | missing | | `toMatchAriaSnapshot()` | not yet | missing | | `expect(value).toBe()`, `toEqual()`, `toMatch()`, `toBeTruthy()`, `toBeCloseTo()` and the rest | the same, except `toContain`, below, see [value matchers](/reference/expect#value-matchers) | same | @@ -219,9 +221,9 @@ what to do today. | Playwright | e2e | Status | | --- | --- | --- | | `page.route(pattern, handler)` | `browser.route(pattern, handler)`; a string or `RegExp` pattern, no predicate function; the handler decides exactly once | web only | -| `route.fulfill({ json, body, status, headers })` | the same; `json` or `body`, never both | web only | -| `route.fulfill({ path, contentType })`, `route.fetch()` | not yet; read the file in the test and pass `body` | missing | -| `route.continue(overrides)`, `route.abort(code)` | `continue()` and `abort()` take no arguments | missing | +| `route.fulfill({ json, body, path, status, headers, contentType })` | the same; one of `json`, `body`, or `path`, and `path` is relative to the project root | web only | +| `route.continue({ url, method, headers, postData })`, `route.fallback()` | the same; a `url` goes through the navigation rule, and `fallback()` takes no overrides | web only | +| `route.fetch()`, `route.abort(code)`, `route.fallback(overrides)` | not yet; `abort()` takes no error code | missing | | `page.unroute()` | `browser.unroute()` | web only | | `page.waitForResponse(pattern)` | `browser.waitForResponse(pattern)`: a string or `RegExp`, no predicate function; `url`, `status`, `headers`, `json()`, `text()` | web only | | `response.ok()`, `response.body()`, `response.request()` | not yet; `status` and `text()` | missing | @@ -232,7 +234,8 @@ what to do today. | `page.waitForEvent('popup')`, `context.on('page')` | not yet; a window the app opens is not driven | missing | | `page.on('console')`, `page.on('pageerror')` | not yet; assert on what the page shows | missing | | `page.evaluate(fn, arg)` | `browser.evaluate(fn, arg)`; JSON in and out, no handles | web only | -| `page.exposeFunction()`, `addInitScript()`, `addStyleTag()`, `setExtraHTTPHeaders()`, `emulateMedia()`, `clock` | not yet; `browser.evaluate` appends a ` @@ -310,8 +334,8 @@ describe('contenteditable editing hosts', () => { await page.frames()[1]?.waitForLoadState('domcontentloaded'); const { tree } = await capture(); const roles = Object.fromEntries(flatten(tree).filter((node) => node.testId !== undefined).map((node) => [node.testId, node.role])); - // A pixel-only surface stays unlisted: with no role, name, or text it is not a node at all. - expect(roles).toEqual({ frame: 'iframe', host: 'textbox' }); + // A canvas, with no role, name, or text, is not a node at all; an svg has the image role, so it is listed. + expect(roles).toEqual({ svg: 'image', frame: 'iframe', host: 'textbox' }); }); it('drops a name taken from a block placeholder once the editor has content, and keeps one from the host', async () => { diff --git a/packages/web/tests/integration/read-node-selection.test.ts b/packages/web/tests/integration/read-node-selection.test.ts index 3f62a3421..06e300498 100644 --- a/packages/web/tests/integration/read-node-selection.test.ts +++ b/packages/web/tests/integration/read-node-selection.test.ts @@ -1,6 +1,6 @@ /** The real reader reports the text selected inside the focused field, and only there. */ -import { chromium, type Browser, type ElementHandle, type Page } from 'playwright'; +import { chromium, type Browser, type ElementHandle, type Page } from 'playwright-core'; import { afterAll, beforeAll, describe, expect, it } from 'vitest'; import type { SemanticNode } from 'e2e/engine'; import { captureDocument } from '../../src/observation.ts'; diff --git a/packages/web/tests/integration/read-node-states.test.ts b/packages/web/tests/integration/read-node-states.test.ts index 0f65db840..d0a05dd6a 100644 --- a/packages/web/tests/integration/read-node-states.test.ts +++ b/packages/web/tests/integration/read-node-states.test.ts @@ -1,6 +1,6 @@ /** The real reader's `disabled` state is the effective one: a control inherits it from a disabled fieldset and from an `aria-disabled` ancestor, as the browser and Playwright report it. */ -import { chromium, type Browser, type ElementHandle, type Page } from 'playwright'; +import { chromium, type Browser, type ElementHandle, type Page } from 'playwright-core'; import { afterAll, beforeAll, describe, expect, it } from 'vitest'; import type { SemanticNode } from 'e2e/engine'; import { captureDocument } from '../../src/observation.ts'; diff --git a/packages/web/tests/integration/routes.test.ts b/packages/web/tests/integration/routes.test.ts new file mode 100644 index 000000000..f38b5a03e --- /dev/null +++ b/packages/web/tests/integration/routes.test.ts @@ -0,0 +1,280 @@ +/** + * Route decisions against a server that counts what reaches it: `continue` + * goes to the network past every other route and carries the configured + * site headers itself, `fallback` hands the request to the route before it, + * the overrides each decision implements reach the wire, and an option it + * does not implement fails the next step instead of being dropped. Also the + * one call beside routing that takes a URL the same way: a relative cookie + * URL. + */ + +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { createServer, type Server } from 'node:http'; +import type { AddressInfo } from 'node:net'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'; +import type { Page } from 'playwright-core'; +import { TestError, type EngineFixtureContext } from 'e2e/engine'; +import { PlaywrightSurface } from '../../src/surface.ts'; +import { createBrowserFixture, type Browser, type WebRoute } from '../../src/browser.ts'; +import { noSecrets } from '../helpers/secrets.ts'; + +interface Echo { + readonly path: string; + readonly method: string; + readonly headers: Record; + readonly body: string; +} + +/** + * `/echo` answers with what it received; `/api/*` answers `network:`. + * Every request is counted by path. + */ +function startCountingServer(hits: Map): Promise<{ server: Server; port: number }> { + const server = createServer((request, response) => { + const pathname = new URL(request.url ?? '/', 'http://host').pathname; + hits.set(pathname, (hits.get(pathname) ?? 0) + 1); + if (pathname === '/echo') { + let body = ''; + request.on('data', (chunk: Buffer) => { body += chunk.toString(); }); + request.on('end', () => { + response.writeHead(200, { 'content-type': 'application/json' }); + response.end(JSON.stringify({ path: pathname, method: request.method, headers: request.headers, body })); + }); + return; + } + if (pathname.startsWith('/api/')) { + response.writeHead(200, { 'content-type': 'text/plain' }); + response.end(`network:${pathname}`); + return; + } + response.writeHead(200, { 'content-type': 'text/html' }); + response.end('Routes

Routes

'); + }); + return new Promise((resolve) => { + server.listen(0, '127.0.0.1', () => resolve({ server, port: (server.address() as AddressInfo).port })); + }); +} + +describe('browser.route decisions', () => { + const surface = new PlaywrightSurface({ headers: { 'X-Gate': 'open-sesame' } }); + const projectRoot = mkdtempSync(path.join(tmpdir(), 'e2e-routes-')); + const signal = new AbortController().signal; + const hits = new Map(); + let server: Server; + let origin: string; + let otherSite: string; + let page: Page; + let browser: Browser; + + /** The harness's navigation rule: relative to the base, only http(s) and the exact `about:blank` admitted. */ + const resolveUrl = (url: string): string => { + const resolved = new URL(url, `${origin}/`); + if (!['http:', 'https:'].includes(resolved.protocol) && resolved.href !== 'about:blank') { + throw new TestError('POLICY_DENIED', `forbidden URL scheme: ${resolved.protocol}`); + } + return resolved.href; + }; + + beforeAll(async () => { + const started = await startCountingServer(hits); + server = started.server; + origin = `http://127.0.0.1:${started.port}`; + // The same server under another host name: `localhost` is a site of its own. + otherSite = `http://localhost:${started.port}`; + writeFileSync(path.join(projectRoot, 'quote.json'), '{"cents":4200}\n'); + await surface.init({ runId: 'routes', targetName: 'web', projectRoot, app: { site: '127.0.0.1' }, + env: {}, headed: false, workerSlot: 0, signal, log: () => undefined }); + }); + + beforeEach(async () => { + hits.clear(); + await surface.startAttempt({ attemptId: 'routes', artifactsDir: projectRoot, signal, resolveSecret: noSecrets }); + page = await surface.ensurePage(); + await page.goto(`${origin}/`); + browser = createBrowserFixture(surface, { + app: { site: '127.0.0.1', resolveUrl }, + operation: (timeoutMs = 5_000) => ({ signal, timeoutMs, runId: 'routes', attemptId: 'routes', origin: 'test' }), + expectable: (target: object) => target, + fixture: (_name: string, target: object) => target, + } as unknown as EngineFixtureContext); + }); + + afterEach(async () => { + await surface.endAttempt({ signal, timeoutMs: 5_000 }); + }); + + afterAll(async () => { + await surface.dispose({ signal, timeoutMs: 5_000 }); + await new Promise((resolve) => server.close(() => resolve())); + rmSync(projectRoot, { recursive: true, force: true }); + }); + + /** Fetches `url` from the page; a failed fetch reads as `failed`. */ + const fetchText = (url: string): Promise => + page.evaluate((target) => fetch(target).then((reply) => reply.text(), () => 'failed'), url); + + /** Fetches `url` from the page and returns its response headers and body. */ + const fetchReply = (url: string): Promise<{ contentType: string | null; body: string }> => + page.evaluate(async (target) => { + const reply = await fetch(target); + return { contentType: reply.headers.get('content-type'), body: await reply.text() }; + }, url); + + /** Fetches `url` from the page and parses what `/echo` received. */ + const fetchEcho = async (url: string, init?: { method: string; body: string }): Promise => + JSON.parse( + await page.evaluate(([target, options]) => fetch(target!, options).then((reply) => reply.text()), [url, init] as const), + ) as Echo; + + /** Calls a decision the way a JavaScript caller can: any argument, typed or not. */ + const untyped = (decide: (...args: never[]) => Promise, ...args: unknown[]): Promise => + (decide as (...rest: unknown[]) => Promise)(...args); + + /** Waits for the latched route failure the next step reports. */ + const nextStepFailure = (): Promise => browser.url().then(() => undefined, (error: unknown) => error); + + it('answers with the route registered last when two fulfill', async () => { + await browser.route('**/api/**', (route) => route.fulfill({ body: 'earlier' })); + await browser.route('**/api/quote', (route) => route.fulfill({ body: 'later' })); + expect(await fetchText(`${origin}/api/quote`)).toBe('later'); + expect(hits.get('/api/quote')).toBeUndefined(); + }); + + it('continue goes to the network past an earlier route', async () => { + let earlier = 0; + await browser.route('**/api/**', (route) => { + earlier += 1; + return route.fulfill({ body: 'earlier' }); + }); + await browser.route('**/api/quote', (route) => route.continue()); + expect(await fetchText(`${origin}/api/quote`)).toBe('network:/api/quote'); + expect(earlier).toBe(0); + expect(hits.get('/api/quote')).toBe(1); + }); + + it('fallback hands the request to the route before it', async () => { + await browser.route('**/api/**', (route) => route.fulfill({ body: 'earlier' })); + await browser.route('**/api/quote', (route) => route.fallback()); + expect(await fetchText(`${origin}/api/quote`)).toBe('earlier'); + expect(hits.get('/api/quote')).toBeUndefined(); + }); + + it.each([ + ['continue', (route: WebRoute) => route.continue()], + ['fallback', (route: WebRoute) => route.fallback()], + ])('keeps the configured site header through %s', async (_name, decide) => { + await browser.route('**/echo', decide); + const echo = await fetchEcho(`${origin}/echo`); + expect(echo.headers['x-gate']).toBe('open-sesame'); + }); + + it('continue replaces the request headers and adds the site header on top', async () => { + await browser.route('**/echo', (route) => + route.continue({ headers: { ...route.request.headers, 'X-Added': 'yes', 'X-GATE': 'forged' } })); + const echo = await fetchEcho(`${origin}/echo`); + expect(echo.headers['x-added']).toBe('yes'); + expect(echo.headers['x-gate']).toBe('open-sesame'); + }); + + it('continue sends a rewritten url, method, and body without touching the original', async () => { + await browser.route('**/api/quote', (route) => + route.continue({ url: '/echo', method: 'PUT', postData: 'payload' })); + const echo = await fetchEcho(`${origin}/api/quote`, { method: 'POST', body: 'original' }); + expect([echo.path, echo.method, echo.body]).toEqual(['/echo', 'PUT', 'payload']); + expect(echo.headers['x-gate']).toBe('open-sesame'); + expect(hits.get('/api/quote')).toBeUndefined(); + expect(hits.get('/echo')).toBe(1); + }); + + it('continue to another site leaves the site header behind', async () => { + await browser.route('**/api/quote', (route) => route.continue({ url: `${otherSite}/echo` })); + const echo = await fetchEcho(`${origin}/api/quote`); + expect(echo.headers.host).toBe(`localhost:${new URL(origin).port}`); + expect(echo.headers['x-gate']).toBeUndefined(); + expect(hits.get('/api/quote')).toBeUndefined(); + }); + + it.each([ + ['a forbidden scheme', 'file:///etc/hosts', 'POLICY_DENIED', 'forbidden URL scheme: file:'], + ['a wrapped scheme', 'view-source:file:///etc/hosts', 'POLICY_DENIED', 'forbidden URL scheme: view-source:'], + ['about:blank', 'about:blank', 'INVALID_ARGUMENT', "route.continue url must keep the request's http: scheme; got about:"], + ['another scheme', 'https://127.0.0.1/echo', 'INVALID_ARGUMENT', "route.continue url must keep the request's http: scheme; got https:"], + ])('continue to %s aborts the request and fails the next step', async (_name, url, code, message) => { + await browser.route('**/api/quote', (route) => route.continue({ url })); + expect(await fetchText(`${origin}/api/quote`)).toBe('failed'); + expect(hits.get('/api/quote')).toBeUndefined(); + expect(await nextStepFailure()).toMatchObject({ code, message }); + }); + + it('fulfills a project file with its bytes and a type from its extension', async () => { + await browser.route('**/api/quote', (route) => route.fulfill({ path: 'quote.json' })); + expect(await fetchReply(`${origin}/api/quote`)).toEqual({ contentType: 'application/json', body: '{"cents":4200}\n' }); + expect(hits.get('/api/quote')).toBeUndefined(); + }); + + it.each([ + ['a body', { body: 'plain', contentType: 'text/x-quote' }, 'plain'], + ['json', { json: { cents: 1 }, contentType: 'application/vnd.quote+json' }, '{"cents":1}'], + ['a file', { path: 'quote.json', contentType: 'text/plain' }, '{"cents":4200}\n'], + ] as const)('fulfills %s under the given contentType', async (_name, response, body) => { + await browser.route('**/api/quote', (route) => route.fulfill(response)); + const reply = await fetchReply(`${origin}/api/quote`); + expect(reply).toEqual({ contentType: response.contentType, body }); + }); + + it.each([ + ['json', { json: { cents: 1 }, headers: { 'Content-Type': 'application/vnd.quote+json' } }, '{"cents":1}'], + ['a file', { path: 'quote.json', headers: { 'Content-Type': 'text/plain' } }, '{"cents":4200}\n'], + ] as const)('fulfills %s under a content-type header', async (_name, response, body) => { + await browser.route('**/api/quote', (route) => route.fulfill(response)); + const reply = await fetchReply(`${origin}/api/quote`); + expect(reply).toEqual({ contentType: response.headers['Content-Type'], body }); + }); + + it.each([ + ['an unknown fulfill key', (route: WebRoute) => untyped(route.fulfill, { bodyy: 'typo' }), + 'route.fulfill options has no key "bodyy"; it takes status, headers, contentType, json, body, path'], + ['two fulfill sources', (route: WebRoute) => untyped(route.fulfill, { body: 'a', json: { b: 1 } }), + 'route.fulfill takes one of json, body, or path; got json and body'], + ['a missing fulfill file', (route: WebRoute) => route.fulfill({ path: 'missing.json' }), + `route.fulfill path is not a readable file: ${path.join(projectRoot, 'missing.json')}`], + ['a fulfill path through a file', (route: WebRoute) => route.fulfill({ path: 'quote.json/inner' }), + `route.fulfill path is not a readable file: ${path.join(projectRoot, 'quote.json', 'inner')}`], + ['a fulfill status out of range', (route: WebRoute) => route.fulfill({ status: 42 }), + 'route.fulfill status must be an integer from 100 to 599'], + ['fulfill json that is not JSON', (route: WebRoute) => untyped(route.fulfill, { json: { n: 1n } }), + expect.stringContaining('route.fulfill json')], + ['an unknown continue key', (route: WebRoute) => untyped(route.continue, { urll: '/echo' }), + 'route.continue options has no key "urll"; it takes url, method, headers, postData'], + ['non-string continue headers', (route: WebRoute) => untyped(route.continue, { headers: { 'x-n': 1 } }), + 'route.continue header "x-n" must be a string, got number'], + ['a continue header name outside the token grammar', (route: WebRoute) => route.continue({ headers: { 'x a': 'b' } }), + 'route.continue has an invalid header name: "x a"'], + ['a line break in a fulfill header value', (route: WebRoute) => route.fulfill({ headers: { 'x-a': 'b\r\nset-cookie: c=d' } }), + 'route.fulfill header "x-a" must not contain a control character'], + ['an abort error code', (route: WebRoute) => untyped(route.abort, 'failed'), + 'route.abort() takes no arguments'], + ['a fallback override', (route: WebRoute) => untyped(route.fallback, { url: '/echo' }), + 'route.fallback() takes no arguments'], + ])('rejects %s before deciding, and aborts the request', async (_name, decide, message) => { + await browser.route('**/api/quote', decide); + expect(await fetchText(`${origin}/api/quote`)).toBe('failed'); + expect(hits.get('/api/quote')).toBeUndefined(); + expect(await nextStepFailure()).toMatchObject({ code: 'INVALID_ARGUMENT', message }); + }); + + it('takes the decision the moment fulfill is called, awaited or not', async () => { + await browser.route('**/api/quote', (route) => { void route.fulfill({ path: 'quote.json' }); }); + expect(await fetchText(`${origin}/api/quote`)).toBe('{"cents":4200}\n'); + await expect(browser.url()).resolves.toBe(`${origin}/`); + }); + + it('sets a cookie with a relative url on the base URL', async () => { + await browser.setCookies([{ name: 'flavor', value: 'oatmeal', url: '/' }]); + const cookies = await browser.cookies(); + expect(cookies.map(({ name, value, domain, path: cookiePath }) => ({ name, value, domain, path: cookiePath }))) + .toEqual([{ name: 'flavor', value: 'oatmeal', domain: '127.0.0.1', path: '/' }]); + }); +}); diff --git a/packages/web/tests/integration/visibility.test.ts b/packages/web/tests/integration/visibility.test.ts index a9fba4b11..2a372d09d 100644 --- a/packages/web/tests/integration/visibility.test.ts +++ b/packages/web/tests/integration/visibility.test.ts @@ -2,10 +2,13 @@ * The reader's `hidden` state, which `isVisible()`, `toBeVisible()`, and * `toBeHidden()` read, agrees with Playwright's own visibility on the cases * a rect count alone gets wrong: a hidden SVG, a zero-size box, closed - * `
` content, and an empty `display: contents` element. + * `
` content, an empty `display: contents` element, skipped + * `content-visibility` content, and an `aria-hidden` node that still paints. + * The tree walk keeps `aria-hidden` subtrees out while it walks into a + * `visibility: hidden` one a child shows again. */ -import { chromium, type Browser, type Page } from 'playwright'; +import { chromium, type Browser, type Page } from 'playwright-core'; import { afterAll, beforeAll, describe, expect, it } from 'vitest'; import type { SemanticNode } from 'e2e/engine'; import { captureDocument } from '../../src/observation.ts'; @@ -49,6 +52,18 @@ const PAGE = `
gone
bare text
+ + + + +
+ + + `; describe('visibility agrees with Playwright', () => { @@ -72,11 +87,50 @@ describe('visibility agrees with Playwright', () => { ['a display: contents element whose only child is hidden', '#contents-hidden-child', true], ['a display: contents element with a painted child', '#contents-painted', false], ['a display: contents element with bare text', '#contents-text', false], + ['a hidden display: contents element whose child inherits it', '#contents-hidden-element', true], + ['a hidden display: contents element whose child is visible again', '#contents-shown-child', false], + ['an aria-hidden spinner that paints', '#spinner', false], + ['a button under an aria-hidden container', '#under-aria-hidden', false], + ['a content-visibility: hidden element, which keeps its box', '#skipping', false], + ['a button content-visibility: hidden skips', '#skipped', true], + ['a button slotted under a content-visibility: hidden host', '#skipped-slotted', true], + ['a visibility: hidden parent', '#vis-parent', true], + ['a child that sets visibility: visible under a hidden parent', '#vis-child', false], ])('%s', async (_case, selector, hidden) => { expect(await hiddenOf(selector)).toEqual({ reader: hidden, playwright: hidden }); }); }); +describe('visibility where Playwright reads a range box alone', () => { + it('hides text under a display: contents element whose visibility is hidden, which Playwright calls visible', async () => { + await page.setContent(''); + expect(await hiddenOf('#contents-hidden-text')).toEqual({ reader: true, playwright: false }); + }); +}); + +describe('the bounding box', () => { + /** The reader's rect for one element next to Playwright's `boundingBox()`. */ + async function boxOf(selector: string) { + const locator = page.locator(selector); + const raw = await locator.evaluate(readSemanticsFunction, READ_OPTIONS); + return { reader: raw.rect, playwright: await locator.boundingBox() }; + } + + it.each([ + ['display: none', '', null], + ['under a display: none ancestor', '
gone
', null], + ['an empty display: contents element', '
', null], + ['display: contents around text', '
text
', null], + ['visibility: hidden, which keeps its box', '', { width: 30, height: 20 }], + ['a zero-size box', '
', { width: 0, height: 0 }], + ])('%s', async (_case, html, size) => { + await page.setContent(`${html}`); + const { reader, playwright } = await boxOf('#box'); + expect(reader).toEqual(playwright); + expect(reader === null ? null : { width: reader.width, height: reader.height }).toEqual(size); + }); +}); + function flatten(node: SemanticNode, out: SemanticNode[] = []): SemanticNode[] { out.push(node); for (const child of node.children ?? []) flatten(child, out); @@ -158,3 +212,22 @@ describe('an inert subtree', () => { expect(buttons).toEqual(['Outside']); }); }); + +describe('the tree walk through hidden content', () => { + it('walks into a visibility: hidden subtree for a child that shows again, and stops at aria-hidden and skipped content', async () => { + await page.setContent(` +
+ +
Skipped text
+
+ `); + const nodes = await captureTree(); + const buttons = nodes.filter((node) => node.role === 'button'); + expect(buttons.map((node) => [node.name, node.states?.hidden])).toEqual([['Shown child', undefined]]); + const skipping = nodes.find((node) => node.testId === 'skipping'); + expect(skipping?.states?.hidden).toBeUndefined(); + // The container keeps its box and paints none of its text, which innerText reads empty too. + expect(skipping?.text).toBe(''); + expect(await page.getByTestId('skipping').innerText()).toBe(''); + }); +}); diff --git a/packages/web/tests/integration/wait-for-response.test.ts b/packages/web/tests/integration/wait-for-response.test.ts index a2c617729..0b8c12f6e 100644 --- a/packages/web/tests/integration/wait-for-response.test.ts +++ b/packages/web/tests/integration/wait-for-response.test.ts @@ -4,8 +4,8 @@ import type { AddressInfo } from 'node:net'; import { tmpdir } from 'node:os'; import path from 'node:path'; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'; -import type { Page } from 'playwright'; -import type { EngineFixtureContext } from 'e2e/engine'; +import type { Page } from 'playwright-core'; +import { withTimeout, type EngineFixtureContext } from 'e2e/engine'; import { PlaywrightSurface } from '../../src/surface.ts'; import { createBrowserFixture, type Browser } from '../../src/browser.ts'; import { noSecrets } from '../helpers/secrets.ts'; @@ -14,7 +14,8 @@ import { noSecrets } from '../helpers/secrets.ts'; * One origin with a body for every outcome `waitForResponse` reports: a full * JSON body, a genuinely empty body under 200 and 204, a redirect whose body * the browser drops, and a body cut short by a connection reset before the - * declared `Content-Length` was sent. The fragment is flushed before the + * declared `Content-Length` was sent, one whose body follows its headers + * 1200ms later, and one that never finishes. The fragment is flushed before the * reset so the browser has seen the headers and reports the response rather * than an empty reply. */ @@ -41,6 +42,15 @@ function startBodyServer(): Promise<{ server: Server; url: string }> { response.writeHead(302, { location: '/api/full', 'content-type': 'text/plain' }); response.end('moved'); return; + case '/api/slow': + response.writeHead(200, { 'content-type': 'text/plain' }); + response.flushHeaders(); + setTimeout(() => response.end('slow-body'), 1_200); + return; + case '/api/endless': + response.writeHead(200, { 'content-type': 'text/plain' }); + response.write('a first chunk and never the rest'); + return; case '/api/cut': response.writeHead(200, { 'content-type': 'application/json', 'content-length': '1000' }); response.write('{"partial":', () => { @@ -80,7 +90,7 @@ describe('browser.waitForResponse bodies', () => { page = await surface.ensurePage(); await page.goto(`${origin}/`); browser = createBrowserFixture(surface, { - operation: () => ({ signal, timeoutMs: 5_000, runId: 'responses', attemptId: 'responses', origin: 'test' }), + operation: (timeoutMs = 2_000) => ({ signal, timeoutMs, runId: 'responses', attemptId: 'responses', origin: 'test' }), expectable: (target: object) => target, fixture: (_name: string, target: object) => target, } as unknown as EngineFixtureContext); @@ -100,11 +110,13 @@ describe('browser.waitForResponse bodies', () => { * Fires one page-side fetch that reads its body the way an app does and * returns the response `waitForResponse` observed for it. */ - async function observe(pathname: string) { + async function observe(pathname: string, options?: { timeout: number }) { + // The harness's step bound, as the fixture recorder applies it to every `browser` call. + const bound = options?.timeout ?? 2_000; const [response] = await Promise.all([ - browser.waitForResponse(`**${pathname}`), + withTimeout(browser.waitForResponse(`**${pathname}`, options), bound, () => new Error(`step exceeded ${bound}ms`)), page.evaluate( - (url) => fetch(url).then((reply) => reply.text()).catch(() => undefined), + (url) => { void fetch(url).then((reply) => reply.text()).catch(() => undefined); }, `${origin}${pathname}`, ), ]); @@ -139,6 +151,21 @@ describe('browser.waitForResponse bodies', () => { await expect(response.json()).rejects.toMatchObject({ code: 'ACTION_FAILED' }); }); + it('matches on headers inside its timeout and lets text() wait for a slower body', async () => { + const response = await observe('/api/slow', { timeout: 500 }); + expect(response.status).toBe(200); + await expect(response.text()).resolves.toBe('slow-body'); + }); + + it('rejects reading a body that never finishes once the action budget runs out', async () => { + const response = await observe('/api/endless', { timeout: 500 }); + expect(response.status).toBe(200); + await expect(response.text()).rejects.toMatchObject({ + code: 'ACTION_FAILED', + message: 'waitForResponse: response body did not finish within 2000ms', + }); + }); + it('keeps the status of a redirect and rejects reading its body', async () => { const response = await observe('/api/redirect'); expect(response.status).toBe(302); diff --git a/packages/web/tests/types/connection.ts b/packages/web/tests/types/connection.ts index 9877cf6a2..9d0d327e8 100644 --- a/packages/web/tests/types/connection.ts +++ b/packages/web/tests/types/connection.ts @@ -1,6 +1,6 @@ /** Compile-time assertions for endpoint provisioning, transport recovery, and the browser provider seam. */ import { secrets } from 'e2e'; -import type { BrowserLease, BrowserProvider, WebConnectOptions, WebOptions } from '../../src/index.ts'; +import type { BrowserLease, BrowserProvider, WebConnectOptions, WebInitScript, WebOptions } from '../../src/index.ts'; ({ cdpEndpoint: () => 'ws://localhost:9222', reconnectEndpoint: async (signal) => { signal.throwIfAborted(); @@ -55,6 +55,17 @@ import type { BrowserLease, BrowserProvider, WebConnectOptions, WebOptions } fro ({ testIdAttribute: ['data-qa', 'data-test'] }) satisfies WebOptions; // @ts-expect-error a user agent is one string. ({ userAgent: 3 }) satisfies WebOptions; +// The locale and time zone every context runs in. +({ locale: 'de-DE', timezoneId: 'Europe/Berlin' }) satisfies WebOptions; +// @ts-expect-error a locale is one tag; Playwright takes no fallback list. +({ locale: ['de-DE', 'en-US'] }) satisfies WebOptions; +// Init scripts: source, a file, or a function that takes no argument. +({ initScripts: ['window.x = 1', { path: 'shim.js' }, () => undefined] }) satisfies WebOptions; +({ path: 'shim.js' }) satisfies WebInitScript; +// @ts-expect-error a configured script takes no argument; inline the value or use browser.addInitScript. +({ initScripts: [(arg: { a: number }) => arg.a] }) satisfies WebOptions; +// @ts-expect-error Playwright's { content } is a plain string here. +({ initScripts: [{ content: 'window.x = 1' }] }) satisfies WebOptions; // The page screencast's frame size and quality; which attempts record is the config's video. ({ screencast: { size: { width: 1280, height: 720 }, quality: 80 } }) satisfies WebOptions; // @ts-expect-error web({ video }) was renamed web({ screencast }). diff --git a/packages/web/tests/types/init-scripts.ts b/packages/web/tests/types/init-scripts.ts new file mode 100644 index 000000000..447c46025 --- /dev/null +++ b/packages/web/tests/types/init-scripts.ts @@ -0,0 +1,15 @@ +/** Compile-time assertions for `browser.addInitScript`: the scripts it takes, and an argument only beside a function. */ +import type { Browser } from '../../src/index.ts'; + +declare const browser: Browser; +void browser.addInitScript('window.x = 1'); +void browser.addInitScript({ path: 'shim.js' }); +void browser.addInitScript(() => undefined); +void browser.addInitScript((wallet) => wallet.address.length, { address: '0xabc' }); + +// @ts-expect-error an argument goes to a function only. +void browser.addInitScript('window.x = 1', { a: 1 }); +// @ts-expect-error the argument must be JSON-safe. +void browser.addInitScript((arg: Date) => arg, new Date()); +// @ts-expect-error a function that takes an argument needs one. +void browser.addInitScript((arg: { a: number }) => arg.a); diff --git a/packages/web/tests/types/routes.ts b/packages/web/tests/types/routes.ts new file mode 100644 index 000000000..499bda969 --- /dev/null +++ b/packages/web/tests/types/routes.ts @@ -0,0 +1,24 @@ +/** Compile-time assertions for route decisions: the options each one takes, and the ones it refuses at runtime too. */ +import type { RouteContinueOverrides, RouteFulfillResponse, WebRoute } from '../../src/index.ts'; + +({ json: { ok: true }, contentType: 'application/vnd.quote+json' }) satisfies RouteFulfillResponse; +({ path: 'fixtures/quote.json', status: 201 }) satisfies RouteFulfillResponse; +({}) satisfies RouteFulfillResponse; +({ url: '/api/other', method: 'PUT', headers: { 'x-a': 'b' }, postData: '{}' }) satisfies RouteContinueOverrides; + +// @ts-expect-error a response has one source: `path` and `body` together are not one. +({ path: 'quote.json', body: 'x' }) satisfies RouteFulfillResponse; +// @ts-expect-error a response has one source: `path` and `json` together are not one. +({ path: 'quote.json', json: 1 }) satisfies RouteFulfillResponse; +// @ts-expect-error `fulfill({ response })` from Playwright is not implemented. +({ response: {} }) satisfies RouteFulfillResponse; +// @ts-expect-error header values are strings. +({ headers: { 'x-n': 1 } }) satisfies RouteContinueOverrides; + +declare const route: WebRoute; +void route.continue(); +void route.fallback(); +// @ts-expect-error `abort` takes no Playwright error code. +void route.abort('failed'); +// @ts-expect-error `fallback` takes no overrides. +void route.fallback({ url: '/api/other' }); diff --git a/packages/web/tests/unit/actions.test.ts b/packages/web/tests/unit/actions.test.ts index f93cd6857..b0d2125d8 100644 --- a/packages/web/tests/unit/actions.test.ts +++ b/packages/web/tests/unit/actions.test.ts @@ -5,7 +5,7 @@ * support.ts). */ -import type { Locator as PwLocator, Page } from 'playwright'; +import type { Locator as PwLocator, Page } from 'playwright-core'; import { describe, expect, it, vi } from 'vitest'; import { EngineError, type LocatorAction } from 'e2e/engine'; import { TestError } from 'e2e/engine'; @@ -137,8 +137,6 @@ function stubLocator(box = { x: 10, y: 20, width: 200, height: 100 }) { dblclick: vi.fn(async () => undefined), fill: vi.fn(async () => undefined), press: vi.fn(async () => undefined), - check: vi.fn(async () => undefined), - uncheck: vi.fn(async () => undefined), hover: vi.fn(async () => undefined), focus: vi.fn(async () => undefined), scrollIntoViewIfNeeded: vi.fn(async () => undefined), @@ -165,8 +163,6 @@ describe('dispatchLocatorAction', () => { [{ kind: 'fill', value: 'hunter2', sensitive: true }, 'fill', ['hunter2', { timeout: 7 }]], [{ kind: 'clear' }, 'fill', ['', { timeout: 7 }]], [{ kind: 'press', key: 'Enter' }, 'press', ['Enter', { timeout: 7 }]], - [{ kind: 'check' }, 'check', [{ timeout: 7 }]], - [{ kind: 'uncheck' }, 'uncheck', [{ timeout: 7 }]], [{ kind: 'focus' }, 'focus', [{ timeout: 7 }]], [{ kind: 'hover' }, 'hover', [{ timeout: 7 }]], [{ kind: 'scrollIntoView' }, 'scrollIntoViewIfNeeded', [{ timeout: 7 }]], diff --git a/packages/web/tests/unit/attempt-session.test.ts b/packages/web/tests/unit/attempt-session.test.ts index d55ec7bd7..ec8865224 100644 --- a/packages/web/tests/unit/attempt-session.test.ts +++ b/packages/web/tests/unit/attempt-session.test.ts @@ -2,7 +2,7 @@ import { tmpdir } from 'node:os'; import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; import path from 'node:path'; -import type { Browser, Page } from 'playwright'; +import type { Browser, Page } from 'playwright-core'; import { beforeEach, describe, expect, it, vi } from 'vitest'; import { connectCdp } from '../../src/browser-connection.ts'; import { AttemptSession } from '../../src/attempt-session.ts'; diff --git a/packages/web/tests/unit/browser-provider.test.ts b/packages/web/tests/unit/browser-provider.test.ts index 5809e3d22..fe78b7b4c 100644 --- a/packages/web/tests/unit/browser-provider.test.ts +++ b/packages/web/tests/unit/browser-provider.test.ts @@ -8,7 +8,7 @@ import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import path from 'node:path'; -import type { Browser, Page } from 'playwright'; +import type { Browser, Page } from 'playwright-core'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import type { EngineAttemptContext, EngineCleanupContext, EngineFinishInfo, EngineInitInfo, EnginePrepareInfo, OperationContext, ProviderRecordContext, ProviderRecording } from 'e2e/engine'; import { connectCdp } from '../../src/browser-connection.ts'; @@ -185,11 +185,13 @@ describe('web({ browser: provider })', () => { expect(() => web({ browser: provider().impl, connect: { cdpEndpoint: () => 'ws://x' } })).toThrow(/two browser sources/); }); - it('rejects creation-time headers, credentials, and user agent with an attempt-scoped provider, as persistent connect does', () => { + it('rejects creation-time headers, credentials, user agent, locale, and time zone with an attempt-scoped provider, as persistent connect does', () => { const cloud = provider({ scope: 'attempt' }).impl; expect(() => web({ browser: cloud, headers: { 'x-preview': 'synthetic' } })).toThrow(/persistent context/); expect(() => web({ browser: cloud, basicAuth: { username: 'user', password: 'synthetic' } })).toThrow(/persistent context/); expect(() => web({ browser: cloud, userAgent: 'synthetic playwright' })).toThrow(/persistent context/); + expect(() => web({ browser: cloud, locale: 'de-DE' })).toThrow(/persistent context; locale requires/); + expect(() => web({ browser: cloud, timezoneId: 'Europe/Berlin' })).toThrow(/persistent context; timezoneId requires/); expect(() => web({ browser: provider().impl, headers: { 'x-preview': 'synthetic' } })).not.toThrow(); }); diff --git a/packages/web/tests/unit/cdp-connect.test.ts b/packages/web/tests/unit/cdp-connect.test.ts index 641e19080..6b0b639d3 100644 --- a/packages/web/tests/unit/cdp-connect.test.ts +++ b/packages/web/tests/unit/cdp-connect.test.ts @@ -34,11 +34,13 @@ describe('web({ connect })', () => { expect(engine.session?.restart).toBeTypeOf('function'); }); - it('rejects creation-time credentials, headers, and user agent with persistent recovery', () => { + it('rejects creation-time credentials, headers, user agent, locale, and time zone with persistent recovery', () => { const connect = { cdpEndpoint: () => 'ws://localhost:0', reconnectEndpoint: () => 'ws://localhost:0' }; expect(() => web({ connect, headers: { 'x-preview': 'synthetic' } })).toThrow(/persistent context/); expect(() => web({ connect, basicAuth: { username: 'user', password: 'synthetic' } })).toThrow(/persistent context/); expect(() => web({ connect, userAgent: 'synthetic playwright' })).toThrow(/persistent context/); + expect(() => web({ connect, locale: 'de-DE' })).toThrow(/persistent context; locale requires/); + expect(() => web({ connect, timezoneId: 'Europe/Berlin' })).toThrow(/persistent context; timezoneId requires/); }); it('accepts a connect option with the default chromium browser', () => { diff --git a/packages/web/tests/unit/cdp-selectors.test.ts b/packages/web/tests/unit/cdp-selectors.test.ts index 8672e26f6..e6d749c20 100644 --- a/packages/web/tests/unit/cdp-selectors.test.ts +++ b/packages/web/tests/unit/cdp-selectors.test.ts @@ -1,5 +1,5 @@ /** The CDP compatibility boundary must support Playwright's proxy channel and fail closed otherwise. */ -import type { BrowserContext } from 'playwright'; +import type { BrowserContext } from 'playwright-core'; import { expect, it, vi } from 'vitest'; import { registerCdpSelectors } from '../../src/cdp-selectors.ts'; import { SELECTOR_ENGINES } from '../../src/selector-engines.ts'; @@ -11,7 +11,7 @@ it('registers every owned selector engine, and nothing else, through a dynamical const channel = new Proxy({}, { get: (_target, key) => key === 'registerSelectorEngine' ? register : undefined }); const options = budget(); await registerCdpSelectors({ _channel: channel } as unknown as BrowserContext, options); - expect(SELECTOR_ENGINES.map(({ name }) => name)).toEqual(['e2e-closed', 'e2e-roots', 'e2e-label', 'e2e-read']); + expect(SELECTOR_ENGINES.map(({ name }) => name)).toEqual(['e2e-closed', 'e2e-roots', 'e2e-label', 'e2e-read', 'e2e-shown']); expect(register).toHaveBeenCalledTimes(SELECTOR_ENGINES.length); for (const { name, source } of SELECTOR_ENGINES) { expect(register).toHaveBeenCalledWith({ selectorEngine: { diff --git a/packages/web/tests/unit/cli.test.ts b/packages/web/tests/unit/cli.test.ts new file mode 100644 index 000000000..c53953796 --- /dev/null +++ b/packages/web/tests/unit/cli.test.ts @@ -0,0 +1,53 @@ +import { describe, expect, it } from 'vitest'; +import { main } from '../../src/cli.ts'; + +/** Runs `e2e-web` with recorded output and a fake Playwright CLI that exits with `exitCode`. */ +async function run(args: readonly string[], exitCode = 0) { + const calls: (readonly string[])[] = []; + let stdout = ''; + let stderr = ''; + const code = await main(args, { + stdout: (text) => { stdout += text; }, + stderr: (text) => { stderr += text; }, + run: async (forwarded) => { + calls.push(forwarded); + return exitCode; + }, + }); + return { code, calls, stdout, stderr }; +} + +describe('e2e-web', () => { + it('installs chromium when no browser is named', async () => { + expect(await run(['install'])).toMatchObject({ code: 0, calls: [['install', 'chromium']] }); + }); + + it('forwards the named browsers and --with-deps', async () => { + expect((await run(['install', 'webkit', '--with-deps', 'firefox'])).calls).toEqual([['install', '--with-deps', 'webkit', 'firefox']]); + }); + + it("exits with the Playwright CLI's code", async () => { + expect((await run(['install'], 1)).code).toBe(1); + }); + + it.each([[['--help']], [['-h']], [['help']], [['install', '--help']], [['install', 'webkit', '-h']]])( + 'prints usage for %j without running anything', + async (args) => { + const result = await run(args); + expect(result).toMatchObject({ code: 0, calls: [] }); + expect(result.stdout).toContain('Usage: npx @e2e-dev/web install [chromium|firefox|webkit ...] [--with-deps]'); + expect(result.stdout).toContain('pnpm exec e2e-web install'); + }, + ); + + it.each([ + [[], 'missing command'], + [['show-report'], 'unknown command "show-report"'], + [['install', '--force'], 'unknown option "--force"'], + [['install', 'chrome'], 'unknown browser "chrome"; expected one of chromium, firefox, webkit'], + ])('rejects %j with exit code 2', async (args, message) => { + const result = await run(args); + expect(result).toMatchObject({ code: 2, calls: [] }); + expect(result.stderr).toContain(message); + }); +}); diff --git a/packages/web/tests/unit/dialogs.test.ts b/packages/web/tests/unit/dialogs.test.ts index 558f59be7..949dcb775 100644 --- a/packages/web/tests/unit/dialogs.test.ts +++ b/packages/web/tests/unit/dialogs.test.ts @@ -3,7 +3,7 @@ * go, and how a failure on the unawaited dialog path reaches the next step. */ -import type { Dialog as PwDialog } from 'playwright'; +import type { Dialog as PwDialog } from 'playwright-core'; import { TestError } from 'e2e/engine'; import { describe, expect, it, vi } from 'vitest'; import { DialogRouter } from '../../src/dialogs.ts'; diff --git a/packages/web/tests/unit/evaluation.test.ts b/packages/web/tests/unit/evaluation.test.ts index 5f2180927..38a62fe2d 100644 --- a/packages/web/tests/unit/evaluation.test.ts +++ b/packages/web/tests/unit/evaluation.test.ts @@ -1,7 +1,8 @@ /** * The page-side evaluation boundary: the caller's source is inlined into a - * fresh function, so it must run whatever the loader compiled it to. tsx's - * esbuild `keepNames` output references a module-scoped `__name` helper. + * fresh function, so it must run whatever the loader compiled it to. The tsx + * loader of e2e before 0.17 emits esbuild `keepNames` output, which + * references a module-scoped `__name` helper. */ import { describe, expect, it } from 'vitest'; diff --git a/packages/web/tests/unit/locators.test.ts b/packages/web/tests/unit/locators.test.ts index ae82fed1d..4a5494c32 100644 --- a/packages/web/tests/unit/locators.test.ts +++ b/packages/web/tests/unit/locators.test.ts @@ -4,7 +4,7 @@ * are read, and which ones the engine cannot express at all. */ -import type { Locator as PwLocator, Page } from 'playwright'; +import type { Locator as PwLocator, Page } from 'playwright-core'; import { describe, expect, it } from 'vitest'; import type { LocatorExpression, TextPattern } from 'e2e/engine'; import { applyPostSteps, projectExpression as projectWith, type PostStep } from '../../src/locators.ts'; @@ -114,8 +114,7 @@ describe('projectExpression', () => { const projected = projectExpression(page, visibleLabel); const visibleLabelChain = [ expect.stringMatching(/^locator\(e2e-label=/), - 'filter(visible=true)', - 'locator(:scope:not([aria-hidden="true"]))', + 'locator(e2e-shown=)', ]; expect(chainOf(projected.locator)).toEqual(visibleLabelChain); expect(projected.visible).toBe(true); @@ -126,16 +125,15 @@ describe('projectExpression', () => { query: { kind: 'displayValue', value: { kind: 'string', value: 'ada', exact: true }, visible: true }, }; expect(chainOf(projectExpression(page, visibleValue).locator)).toEqual([ - 'locator(e2e-roots=input, textarea, select)', - 'filter(visible=true)', - 'locator(:scope:not([aria-hidden="true"]))', + 'locator(e2e-roots=input:not([type="checkbox" i]):not([type="radio" i]), textarea, select)', + 'locator(e2e-shown=)', ]); const visibleText: LocatorExpression = { kind: 'query', query: { kind: 'text', value: { kind: 'string', value: 'Save', exact: true }, visible: true }, }; expect(chainOf(projectExpression(page, visibleText).locator)).toEqual([ - ROOTS, 'text(Save,exact)', 'filter(visible=true)', 'locator(:scope:not([aria-hidden="true"]))', + ROOTS, 'text(Save,exact)', 'locator(e2e-shown=)', ]); }); @@ -200,7 +198,7 @@ describe('projectExpression', () => { source: { kind: 'index', source: shared, index: 'last' }, index: 'first', }); - expect(chainOf(projected.locator)).toEqual(['locator(e2e-roots=input, textarea, select)']); + expect(chainOf(projected.locator)).toEqual(['locator(e2e-roots=input:not([type="checkbox" i]):not([type="radio" i]), textarea, select)']); expect(projected.displayValue).toEqual({ kind: 'string', value: 'shared', exact: true }); expect(projected.steps).toEqual([ { kind: 'index', index: 'last' }, @@ -215,7 +213,7 @@ describe('projectExpression', () => { index: 0, }); expect(chainOf(projected.locator)).toEqual([ - 'locator(e2e-roots=input, textarea, select)', + 'locator(e2e-roots=input:not([type="checkbox" i]):not([type="radio" i]), textarea, select)', `filter(hasText=x,has=${ROOTS}>role(textbox))`, ]); expect(projected.displayValue).not.toBeNull(); @@ -239,7 +237,7 @@ describe('projectExpression', () => { hasText: { kind: 'string', value: 'x', exact: false }, has: textbox, }); - expect(chainOf(projected.locator)).toEqual(['locator(e2e-roots=input, textarea, select)']); + expect(chainOf(projected.locator)).toEqual(['locator(e2e-roots=input:not([type="checkbox" i]):not([type="radio" i]), textarea, select)']); expect(projected.steps).toHaveLength(2); expect(projected.steps[0]).toEqual({ kind: 'index', index: 'first' }); const filter = projected.steps[1]!; diff --git a/packages/web/tests/unit/observation-capture.test.ts b/packages/web/tests/unit/observation-capture.test.ts index 4c50061ac..416bdbbe2 100644 --- a/packages/web/tests/unit/observation-capture.test.ts +++ b/packages/web/tests/unit/observation-capture.test.ts @@ -1,5 +1,5 @@ import { tmpdir } from 'node:os'; -import type { Browser, Page } from 'playwright'; +import type { Browser, Page } from 'playwright-core'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import type { OperationContext } from 'e2e/engine'; import type { RawObservedNode } from '../../src/read-node.ts'; diff --git a/packages/web/tests/unit/protected-app-options.test.ts b/packages/web/tests/unit/protected-app-options.test.ts index 6bc97bcdf..34f43d7c0 100644 --- a/packages/web/tests/unit/protected-app-options.test.ts +++ b/packages/web/tests/unit/protected-app-options.test.ts @@ -1,5 +1,5 @@ /** - * The `headers`, `basicAuth`, `testIdAttribute`, `userAgent`, and `screencast` options are checked at + * The `headers`, `basicAuth`, `testIdAttribute`, `userAgent`, `locale`, `timezoneId`, `initScripts`, and `screencast` options are checked at * config load, so a header the browser could never send or an attribute no * element could carry fails the run before a browser launches. What the * browser does with valid ones is in tests/integration. @@ -7,7 +7,7 @@ import { describe, expect, it } from 'vitest'; import { secrets } from 'e2e'; -import { web } from '../../src/index.ts'; +import { web, type WebOptions } from '../../src/index.ts'; import { httpCredentials } from '../../src/protected-app.ts'; describe('web({ headers })', () => { @@ -106,6 +106,52 @@ describe('web({ userAgent })', () => { }); }); +describe('web({ locale, timezoneId })', () => { + it('accepts a language tag and an IANA time zone', () => { + expect(() => web({ locale: 'de-DE', timezoneId: 'Europe/Berlin' })).not.toThrow(); + expect(() => web({ locale: 'zh-Hant-TW', timezoneId: 'UTC' })).not.toThrow(); + expect(() => web({ locale: 'de-DE-u-co-phonebk', timezoneId: 'US/Eastern' })).not.toThrow(); + expect(() => web({ timezoneId: 'GMT' })).not.toThrow(); + }); + + it('rejects a value that is no language tag or time zone', () => { + for (const locale of ['', 'not a locale', 'de_DE', 'und', 'x-private', 3]) { + expect(() => web({ locale: locale as string })).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG', message: expect.stringMatching(/BCP 47/) })); + } + for (const timezoneId of ['', 'Mars/Olympus_Mons', 'GMT+25', '+01:00', 'europe/berlin', 'utc', 3]) { + expect(() => web({ timezoneId: timezoneId as string })).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG', message: expect.stringMatching(/IANA time zone/) })); + } + }); + + it('rejects an accept-language header beside locale, which would override it on the app site only', () => { + expect(() => web({ locale: 'de-DE', headers: { 'Accept-Language': 'fr' } })).toThrowError(/conflict/); + expect(() => web({ locale: 'de-DE', headers: { 'x-preview': 'token' } })).not.toThrow(); + }); +}); + +describe('web({ initScripts })', () => { + it('accepts source, a path, and a function', () => { + expect(() => web({ initScripts: [] })).not.toThrow(); + expect(() => web({ initScripts: ['window.x = 1', { path: 'shim.js' }, () => undefined] })).not.toThrow(); + }); + + it('rejects anything else, naming the entry', () => { + const refused = (message: RegExp) => expect.objectContaining({ code: 'INVALID_CONFIG', message: expect.stringMatching(message) }); + const cases: [unknown, RegExp][] = [ + ['window.x = 1', /must be an array of scripts/], + [[3], /\[0\] must be a string of source, a \{ path \}, or a function, got number/], + [['ok', null], /\[1\] must be a string of source.*got null/], + [[{ path: '' }], /\[0\] path must be a non-empty string/], + [[{ content: 'x' }], /\[0\] takes only path, got content/], + // oxlint-disable-next-line no-sparse-arrays -- the hole is the case + [['ok', , 'ok'], /\[1\] must be a string of source.*got undefined/], + ]; + for (const [initScripts, message] of cases) { + expect(() => web({ initScripts: initScripts as NonNullable })).toThrow(refused(message)); + } + }); +}); + describe('web({ screencast })', () => { it('accepts a whole-pixel size and a 0-100 quality, each optional', () => { expect(() => web({ screencast: {} })).not.toThrow(); @@ -150,7 +196,7 @@ describe('web() option keys', () => { refused('web() has unknown key "viewprt"; did you mean "viewport"?'), ); expect(() => web({ launchOptions: {} } as unknown as Parameters[0])).toThrow( - refused('web() has unknown key "launchOptions"; expected one of browser, viewport, screencast, connect, headers, basicAuth, testIdAttribute, userAgent'), + refused('web() has unknown key "launchOptions"; expected one of browser, viewport, screencast, connect, headers, basicAuth, testIdAttribute, userAgent, locale, timezoneId, initScripts'), ); }); diff --git a/packages/web/tests/unit/provider-video.test.ts b/packages/web/tests/unit/provider-video.test.ts index 7729beadd..9c8bd4749 100644 --- a/packages/web/tests/unit/provider-video.test.ts +++ b/packages/web/tests/unit/provider-video.test.ts @@ -3,7 +3,7 @@ import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import path from 'node:path'; -import type { Page } from 'playwright'; +import type { Page } from 'playwright-core'; import { afterEach, describe, expect, it } from 'vitest'; import type { ProviderRecording } from 'e2e/engine'; import { ProviderVideo } from '../../src/provider-video.ts'; diff --git a/packages/web/tests/unit/refs-disposal.test.ts b/packages/web/tests/unit/refs-disposal.test.ts index bf9d523f5..c6adf8f7a 100644 --- a/packages/web/tests/unit/refs-disposal.test.ts +++ b/packages/web/tests/unit/refs-disposal.test.ts @@ -1,6 +1,6 @@ /** Element-backed located refs release their handles when pruned or cleared, like observation refs. */ -import type { ElementHandle } from 'playwright'; +import type { ElementHandle } from 'playwright-core'; import { describe, expect, it, vi } from 'vitest'; import { RefRegistry } from '../../src/refs.ts'; diff --git a/packages/web/tests/unit/semantic-fallback.test.ts b/packages/web/tests/unit/semantic-fallback.test.ts index c16eb9ff5..9251a8156 100644 --- a/packages/web/tests/unit/semantic-fallback.test.ts +++ b/packages/web/tests/unit/semantic-fallback.test.ts @@ -1,6 +1,6 @@ import { tmpdir } from 'node:os'; import { afterEach, describe, expect, it, vi } from 'vitest'; -import type { Browser, ElementHandle, Page } from 'playwright'; +import type { Browser, ElementHandle, Page } from 'playwright-core'; import { EngineError, TestError, type OperationContext } from 'e2e/engine'; import { captureDocument, ROOT_NODE_ID } from '../../src/observation.ts'; import { PlaywrightSurface } from '../../src/surface.ts'; diff --git a/packages/web/tests/unit/surface-lifecycle.test.ts b/packages/web/tests/unit/surface-lifecycle.test.ts index 1ad2f635e..b306d878e 100644 --- a/packages/web/tests/unit/surface-lifecycle.test.ts +++ b/packages/web/tests/unit/surface-lifecycle.test.ts @@ -2,7 +2,7 @@ import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import path from 'node:path'; -import type { Browser, Dialog, Route } from 'playwright'; +import type { Browser, Dialog, Route } from 'playwright-core'; import { secrets } from 'e2e'; import type { EngineFixtureContext, OperationContext } from 'e2e/engine'; import { afterEach, beforeEach, expect, it, vi } from 'vitest'; diff --git a/packages/web/tests/unit/video.test.ts b/packages/web/tests/unit/video.test.ts index 9c5601203..668f57af8 100644 --- a/packages/web/tests/unit/video.test.ts +++ b/packages/web/tests/unit/video.test.ts @@ -8,7 +8,7 @@ import { existsSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import path from 'node:path'; -import type { Page } from 'playwright'; +import type { Page } from 'playwright-core'; import { afterEach, describe, expect, it } from 'vitest'; import { EngineError } from 'e2e/engine'; import { VideoRecorder } from '../../src/video.ts'; diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index e80250465..b9f2d6fc1 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -235,9 +235,6 @@ importers: ai: specifier: 7.0.107 version: 7.0.107(zod@4.6.1) - playwright: - specifier: 1.63.0 - version: 1.63.0 typescript: specifier: 7.0.2 version: 7.0.2 @@ -278,15 +275,18 @@ importers: ai: specifier: 7.0.107 version: 7.0.107(zod@4.6.1) - playwright: - specifier: 1.63.0 - version: 1.63.0 typescript: specifier: 7.0.2 version: 7.0.2 docs: devDependencies: + '@ai-sdk/typesafe-ai': + specifier: 3.0.12 + version: 3.0.12(zod@4.6.1) + '@e2e-dev/decision': + specifier: workspace:* + version: link:../packages/decision '@e2e-dev/mobile': specifier: workspace:* version: link:../packages/mobile @@ -296,12 +296,15 @@ importers: '@e2e-dev/web': specifier: workspace:* version: link:../packages/web + '@openrouter/ai-sdk-provider': + specifier: 3.1.0 + version: 3.1.0(ai@7.0.107(zod@4.6.1))(zod@4.6.1) '@types/node': specifier: 26.6.2 version: 26.6.2 agent-device: - specifier: 0.21.18 - version: 0.21.18(ai@7.0.107(zod@4.6.1)) + specifier: 0.21.20 + version: 0.21.20(ai@7.0.107(zod@4.6.1)) ai: specifier: 7.0.107 version: 7.0.107(zod@4.6.1) @@ -311,12 +314,46 @@ importers: mint: specifier: 4.2.909 version: 4.2.909(@base-ui/react@1.8.0(@types/react@19.3.0)(react-dom@19.3.0(react@19.2.3))(react@19.2.3))(@types/node@26.6.2)(@types/react@19.3.0)(debug@4.4.3(supports-color@8.1.1))(react-dom@19.3.0(react@19.2.3))(supports-color@8.1.1)(typescript@7.0.2) - playwright: - specifier: 1.63.0 - version: 1.63.0 typescript: specifier: 7.0.2 version: 7.0.2 + zod: + specifier: 4.6.1 + version: 4.6.1 + + packages/decision: + dependencies: + '@ai-sdk/provider': + specifier: 4.0.17 + version: 4.0.17 + zod: + specifier: 4.6.1 + version: 4.6.1 + devDependencies: + '@e2e-dev/web': + specifier: workspace:* + version: link:../web + '@types/node': + specifier: 26.6.2 + version: 26.6.2 + ai: + specifier: 7.0.107 + version: 7.0.107(zod@4.6.1) + ajv: + specifier: 8.20.0 + version: 8.20.0 + ajv-formats: + specifier: 3.0.1 + version: 3.0.1(ajv@8.20.0) + e2e: + specifier: workspace:* + version: link:../e2e + typescript: + specifier: 7.0.2 + version: 7.0.2 + vitest: + specifier: 5.0.1 + version: 5.0.1(@types/node@26.6.2)(vite@8.1.5(@types/node@26.6.2)(esbuild@0.28.2)(jiti@1.21.7)(terser@5.51.2)(tsx@4.23.14)(yaml@2.9.0)) packages/e2e: dependencies: @@ -326,28 +363,46 @@ importers: '@clack/prompts': specifier: 1.8.1 version: 1.8.1 + '@jridgewell/remapping': + specifier: 2.3.5 + version: 2.3.5 '@modelcontextprotocol/server': specifier: 2.2.0 version: 2.2.0 + '@oxc-project/runtime': + specifier: 0.152.0 + version: 0.152.0 '@vitest/expect': specifier: 5.0.2 version: 5.0.2 commander: specifier: 15.0.0 version: 15.0.0 + get-tsconfig: + specifier: 4.14.3 + version: 4.14.3 + oxc-parser: + specifier: 0.152.0 + version: 0.152.0 + oxc-transform: + specifier: 0.152.0 + version: 0.152.0 picocolors: specifier: 1.1.1 version: 1.1.1 pngjs: specifier: 7.0.0 version: 7.0.0 - tsx: - specifier: 4.23.14 - version: 4.23.14 zod: specifier: 4.6.1 version: 4.6.1 devDependencies: + '@ai-sdk/anthropic': + specifier: 4.0.58 + version: 4.0.58(zod@4.6.1) + '@ai-sdk/google': + specifier: 4.0.77 + version: 4.0.77(zod@4.6.1) '@ai-sdk/openai': specifier: 4.0.71 version: 4.0.71(zod@4.6.1) @@ -381,9 +436,6 @@ importers: ajv-formats: specifier: 3.0.1 version: 3.0.1(ajv@8.20.0) - playwright: - specifier: 1.63.0 - version: 1.63.0 typescript: specifier: 7.0.2 version: 7.0.2 @@ -448,8 +500,8 @@ importers: packages/mobile: dependencies: agent-device: - specifier: 0.21.18 - version: 0.21.18(ai@7.0.107(zod@4.6.1)) + specifier: 0.21.20 + version: 0.21.20(ai@7.0.107(zod@4.6.1)) pngjs: specifier: 7.0.0 version: 7.0.0 @@ -485,8 +537,8 @@ importers: specifier: 26.6.2 version: 26.6.2 agent-device: - specifier: 0.21.18 - version: 0.21.18(ai@7.0.107(zod@4.6.1)) + specifier: 0.21.20 + version: 0.21.20(ai@7.0.107(zod@4.6.1)) e2e: specifier: workspace:* version: link:../e2e @@ -498,6 +550,10 @@ importers: version: 5.0.1(@types/node@26.6.2)(vite@8.1.5(@types/node@26.6.2)(esbuild@0.28.2)(jiti@1.21.7)(terser@5.51.2)(tsx@4.23.14)(yaml@2.9.0)) packages/web: + dependencies: + playwright-core: + specifier: 1.63.0 + version: 1.63.0 devDependencies: '@types/node': specifier: 26.6.2 @@ -508,9 +564,6 @@ importers: e2e: specifier: workspace:* version: link:../e2e - playwright: - specifier: 1.63.0 - version: 1.63.0 pngjs: specifier: 7.0.0 version: 7.0.0 @@ -523,6 +576,12 @@ importers: packages: + '@ai-sdk/anthropic@4.0.58': + resolution: {integrity: sha512-vhXKBz7jnIhBlOoj2d0joyHicWISuIxDNha65SLrAnagAkVgulLoZBRLZRazstxlO/BtXU+cDERe1opVKte2OQ==} + engines: {node: '>=22'} + peerDependencies: + zod: ^3.25.76 || ^4.1.8 + '@ai-sdk/devtools@1.0.22': resolution: {integrity: sha512-Kdm5HSKj7sdYdAC5bKty6+WWwiAAc5XLFwhwvMIEOm9NVwCpLOD7P8AdaxvW0XbAEVlZyW8FV8Lb+9Y/RB6PhQ==} engines: {node: '>=22'} @@ -534,6 +593,12 @@ packages: peerDependencies: zod: ^3.25.76 || ^4.1.8 + '@ai-sdk/google@4.0.77': + resolution: {integrity: sha512-7s1Gy6PL/KRk12s31g+LMMY5v1D80AQT6Lm+HUGXvCXw5T5WCxzbUuxOrBegXhhLPfIXTRunkYgSIjcRys37Zw==} + engines: {node: '>=22'} + peerDependencies: + zod: ^3.25.76 || ^4.1.8 + '@ai-sdk/openai-compatible@3.0.53': resolution: {integrity: sha512-uOeghHYcF+CHP/dOCAtdfKesuKLKqDFtkq79bJFNSx4YOqpiP734TdJAZDA8KIWH7mdGNQ0a4N9238HG4FN2Rg==} engines: {node: '>=22'} @@ -552,6 +617,12 @@ packages: peerDependencies: zod: ^3.25.76 || ^4.1.8 + '@ai-sdk/provider-utils@5.0.53': + resolution: {integrity: sha512-VVe6UDd0y0/B4TfauuKKzcppqvhTmSluwwUVT4WG2/8h7gFbyFm7PM4UGM+ZXe3/RI6bv+Vl1cw/7aiBoQqdxg==} + engines: {node: '>=22'} + peerDependencies: + zod: ^3.25.76 || ^4.1.8 + '@ai-sdk/provider@4.0.17': resolution: {integrity: sha512-VYMBxIQdcHqbIf1j+YZlI9Ati6LZ4wJe0GGd4z4a5H/KxTggjeOiyaVYTnfF7LHZK5jMQ+rofmzz4QPqf++NUw==} engines: {node: '>=22'} @@ -560,6 +631,16 @@ packages: resolution: {integrity: sha512-+GZJIgz1jk86pwEbb3f1BD2bdoSKyWE4Jg4YUc7NMnMozbWemSKYZcw2F4nMaO5qwsL5A8RmioAKW81YktRpIQ==} engines: {node: '>=22'} + '@ai-sdk/provider@4.0.21': + resolution: {integrity: sha512-UpbC9C1oht8dhfKPbVXSLRZS3DI8uk8n5v2uMjBPNIYGsC2kL045ywq7D8Hh9KgnH9rP5W/GxQdYtWScRouqlA==} + engines: {node: '>=22'} + + '@ai-sdk/typesafe-ai@3.0.12': + resolution: {integrity: sha512-XbwOzWckIDSBPJTFLp2JnmGhdkxeQ1s4fwRGMgo0aubdp66AhnD9/I0CAnC8nESJlns5oGfXICC4m/NTGO6/uA==} + engines: {node: '>=22'} + peerDependencies: + zod: ^3.25.76 || ^4.1.8 + '@ai-sdk/xai@5.0.4': resolution: {integrity: sha512-Cg3xUB0H6OA4fvcX+uamkZXuzxdjhi7HH15glxpE8boZUijFmfz3YPM3Y/HvQkuCb6D/ixCe7lpcwVjidLhw9w==} engines: {node: '>=22'} @@ -2181,9 +2262,260 @@ packages: ai: ^7.0.0 zod: ^3.25.76 || ^4.1.8 + '@oxc-parser/binding-android-arm-eabi@0.152.0': + resolution: {integrity: sha512-393VtirINkae4XHBTkq/yQxhOwEr4V+r0et37P2fp+sYZSdYgZTRDPo5Um8vBh0QjMZ+RmaMy4PP0qwr4cTMwA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [android] + + '@oxc-parser/binding-android-arm64@0.152.0': + resolution: {integrity: sha512-iYFcExATJNZ0rZj0o02THbD1Nc9jAW0nVgIQPee6ejbSASpbZXlIMw8jxtpzaG14JQTWwGSZ6U5dEtRJXE6V3Q==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [android] + + '@oxc-parser/binding-darwin-arm64@0.152.0': + resolution: {integrity: sha512-fnU+DfUzi5/rFy0LzmipYRz06PAUUkOS8kCJQ3Eg6oqlunC1XZXoffFqxmdL51RmcbYtUG3P8DVG1U5fckoa2A==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [darwin] + + '@oxc-parser/binding-darwin-x64@0.152.0': + resolution: {integrity: sha512-Y3U2okryWTs7hDld6UhjJwLNf8/do+x4g4C+1U5SiGwrTLwbO83WJYJA8+l9iFSilWBgnfpd/+U76tfeNxUHzA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [darwin] + + '@oxc-parser/binding-freebsd-x64@0.152.0': + resolution: {integrity: sha512-XkhkRrrPffEw+ZLv/d/5pEni0buxinm/E/N6fLbJCunD1DBbeCTtVP23ESV+4CWCWdSz397asks1ViGslpmWiQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [freebsd] + + '@oxc-parser/binding-linux-arm-gnueabihf@0.152.0': + resolution: {integrity: sha512-MLzsRRbdfmdzo6JEhbC1yKU9ItfvM8tEf18HM32jCiNYZEz6J3bRBRl3cTkKR0vBdrk9ZVlcARXtwHccs+epDA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [linux] + + '@oxc-parser/binding-linux-arm-musleabihf@0.152.0': + resolution: {integrity: sha512-HHerTEJz7Iy10Ji6BGsk4gqG6PU0UzFuBu91jIEOX89FnrI5ECWSEr7BLpkYQhp7U0QFtMnBhNfQKI0hYZK78w==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [linux] + + '@oxc-parser/binding-linux-arm64-gnu@0.152.0': + resolution: {integrity: sha512-9Bb8GyzZ2uSLzRDZ27usaKokr07mxiBe2dDF5b6bd2janmaJTT1gL51UfP7AU30yFiXrJvcRyFKREAgKwZYs0w==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + libc: [glibc] + + '@oxc-parser/binding-linux-arm64-musl@0.152.0': + resolution: {integrity: sha512-om7iDMUroI74A/lfRRWY9/Ev3VWrVsrUZ4tyvyh8raBey/sdcfyAD7tYWc+iykAiotnmftyKTpUKFr1QnaBrAQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + libc: [musl] + + '@oxc-parser/binding-linux-ppc64-gnu@0.152.0': + resolution: {integrity: sha512-Lf1OL2lMTqwDMFC6jnRb30B+WYHKV0XT+bPHcFM6xHtNVwxpobiTzUIvKH6a0LWpvfwyru8Yu6zCGHqqFHH+gg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [ppc64] + os: [linux] + libc: [glibc] + + '@oxc-parser/binding-linux-riscv64-gnu@0.152.0': + resolution: {integrity: sha512-BYWSMQK+YJ9ZILStPcmmiSVq8KEG9Qdq/c4BHPC3rUEq/fEDCVjUIwdj5ZzjgZ9P9is9PKu0tcAXzq1ZXg4urQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [riscv64] + os: [linux] + libc: [glibc] + + '@oxc-parser/binding-linux-riscv64-musl@0.152.0': + resolution: {integrity: sha512-d2LqHNNNqdH185sRK5dPKWZu201JrNu9/CRFuVEuT+ht7n/QaVblEMWT9MALQPrDWYvhG10P+NSKJFquDWZ09w==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [riscv64] + os: [linux] + libc: [musl] + + '@oxc-parser/binding-linux-s390x-gnu@0.152.0': + resolution: {integrity: sha512-8gBeHt5rA/8+pKhHl+zLggfa3L/V/L2at96p7ty6pBE7id6lFO19nFUc15u2PcjC4OYeYK465MKy1fhhbpFIGw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [s390x] + os: [linux] + libc: [glibc] + + '@oxc-parser/binding-linux-x64-gnu@0.152.0': + resolution: {integrity: sha512-ltgrSF6L+yPTVsIdaAyqKq3zTqzj/1VlNbxWZuxAe8flR3sWdyNsKvdXukoO8/ucJbLickhmieRBge+o/e8Ugw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + libc: [glibc] + + '@oxc-parser/binding-linux-x64-musl@0.152.0': + resolution: {integrity: sha512-LcMp6BvrBnewSfdkLHoSqidhoAq4syKOTBV5TYHWd+2VnGkfD/nPRRVIzTBt/Lkbt7pfOFDGtkiTntFS0p6sQQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + libc: [musl] + + '@oxc-parser/binding-openharmony-arm64@0.152.0': + resolution: {integrity: sha512-jbhoMCslVxzRCRNQ8TSHxjYdSKRvk8O6ciLVDkahJvmyHDzBqba/bMQHM6ULKj/rgHNNbSzlIiakdx8SEfkJzA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [openharmony] + + '@oxc-parser/binding-win32-arm64-msvc@0.152.0': + resolution: {integrity: sha512-JZYVlhv36mr/nertnRB7vchJECOBb1Lz6lltalILsB+o9M/is+H6BEDRLQKdqA4JKs6NjJszo46+iM5sfwj6jQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [win32] + + '@oxc-parser/binding-win32-ia32-msvc@0.152.0': + resolution: {integrity: sha512-1X9oganvVsunRH0lH10UU05N+H9atpX6WsOybByUjwnfcd3h/0tYwvuv16XuHglrIUOnxV2TodbLVEwwafd64g==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [ia32] + os: [win32] + + '@oxc-parser/binding-win32-x64-msvc@0.152.0': + resolution: {integrity: sha512-sw0M80/dn9rcpu+Rqqph/C+MjvDDce6MlXmFjsNQtPEymha0eJy3ml6+foSUTqFfpDIDyHZZ4eoJ43IoMe8bJw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [win32] + + '@oxc-project/runtime@0.152.0': + resolution: {integrity: sha512-fsq/7JVJo8YAmMPnt3Z7akMdMoCokRND5jjawa5DxuKJBEaI0Dl3ZqEXFlLabKoZhlApIexZ7aycXFcjcTcWvw==} + engines: {node: ^20.19.0 || >=22.12.0} + '@oxc-project/types@0.139.0': resolution: {integrity: sha512-r9gHphtCs+1M7J0pw6Sn/hh/Wpa/iQrOOkrNAlVLF/gHq+/CJmHIWKKUUhdWjcD6CIa8idarspCsASiXCXvFUw==} + '@oxc-project/types@0.152.0': + resolution: {integrity: sha512-oM/5rLBm2tPkg0iBgkH/FOeR3PCDpY19GTgAZjMFM8h9WI9VW7cLgzp6nwtarYKmovavIQZ+Fe/RKX/8C8O/Rw==} + + '@oxc-transform/binding-android-arm-eabi@0.152.0': + resolution: {integrity: sha512-GBVYZdUggANq7ndUuE9HBi1jO6d2+oud3cevfQkpqrV7JsDNwQU5UCGa8rZCvzjIvyn6ikQuJu+sj4RvLyN6Uw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [android] + + '@oxc-transform/binding-android-arm64@0.152.0': + resolution: {integrity: sha512-pqeCLGUnozzyJE9Ed9js2w+V1lHQ43qPImMfgTh7it+a3LJqUjAt+OesJ6pqkPycRFV6b0pUWZm6EkpM8mk7ag==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [android] + + '@oxc-transform/binding-darwin-arm64@0.152.0': + resolution: {integrity: sha512-0d68wdepj2bjnDijPdvC/3MnrZcs8hoCp19qnGX2tyxnSIPevgqa9RX5i31ex5JMczht/VGp4aKeV1p12dwzQQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [darwin] + + '@oxc-transform/binding-darwin-x64@0.152.0': + resolution: {integrity: sha512-jFTRTyZzYhqSrxkaI9hTwClPTXtdqRyPSXUFGLJmStYS+b6qnhbI1ottivWdAg0HJXytyg8YeqgUJWSu2lU/jA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [darwin] + + '@oxc-transform/binding-freebsd-x64@0.152.0': + resolution: {integrity: sha512-L4Y8IfrcaxjK9kK3ddRAyZPa6nzJHn6fgkVIKoBlP53vmCobPL1mtuIdXQThD0VmV6e4aSAJg2An6qkdjeZC8Q==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [freebsd] + + '@oxc-transform/binding-linux-arm-gnueabihf@0.152.0': + resolution: {integrity: sha512-GrohqmIKEK/YXCzAGP8XGQrPVBdUHPpvqdhvw+VutpWZofxDOq39Zed+Gqlea2mycXo+/UNeq1194eaAYkqKQQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [linux] + + '@oxc-transform/binding-linux-arm-musleabihf@0.152.0': + resolution: {integrity: sha512-wUI+JoZEs4+GntxjK2YOKtQDG822wv+aGYm1OUZ2+NAgi8ghOTUP3RH/GX8Vq0qJ2Yw3OwKLUIZzufXklQ7pCQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [linux] + + '@oxc-transform/binding-linux-arm64-gnu@0.152.0': + resolution: {integrity: sha512-EcY52e0BOfynOA/brCInvQO3ZSukbAEc+BsbpRoRdJ4qV10NK0aS/FheBvkj/V3cFidee6QlrUpqgpN4WKV+Sg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + libc: [glibc] + + '@oxc-transform/binding-linux-arm64-musl@0.152.0': + resolution: {integrity: sha512-L4OGw0UuuyYKAPnUvvW+tvq6nS/tBoolVuPekNJqYnDYfe5Nh1619ImfhP0o0Wz+SG4OQW35ByRWjVf/oK3x0w==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + libc: [musl] + + '@oxc-transform/binding-linux-ppc64-gnu@0.152.0': + resolution: {integrity: sha512-5K9qNjNKlyhBdGkkIhYFonO/H3Yc37miy5cXefTB71aPOv9jv6HQrDuj6AJcBkggbmkeyEJhlFamUbGMsMGBiA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [ppc64] + os: [linux] + libc: [glibc] + + '@oxc-transform/binding-linux-riscv64-gnu@0.152.0': + resolution: {integrity: sha512-WlGvnZhmRItWqjsGd2OBKPan75FbqpF4HPuGL7P6f7Oz1VHm6IatnwCK2MI16QXH7BEs6GRohZHpmgThc0V/uw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [riscv64] + os: [linux] + libc: [glibc] + + '@oxc-transform/binding-linux-riscv64-musl@0.152.0': + resolution: {integrity: sha512-TWZX7QGkU4LF9NS+HBUVWsbQXWDuWt58r+L/31gSJ/iHCJw/3jM2yD408IRVjXYX5jFgmsJbxrUlW3xlVeAL3Q==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [riscv64] + os: [linux] + libc: [musl] + + '@oxc-transform/binding-linux-s390x-gnu@0.152.0': + resolution: {integrity: sha512-VIP7463d0nIAGFv2VXb4mRf20qrpZg0kNHBw8E3fLuHj0bt4HVHDgsqAgeFWOonWwSwsM415TFjNmHhlaStCLQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [s390x] + os: [linux] + libc: [glibc] + + '@oxc-transform/binding-linux-x64-gnu@0.152.0': + resolution: {integrity: sha512-hwnHMVCnGIoS/eMh7T836mJpWim2iCSWGlCS+aZ+SjhEEUvSSUE54ljua7bk25vaj4L84cFnruTCa7HqyaQ2YQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + libc: [glibc] + + '@oxc-transform/binding-linux-x64-musl@0.152.0': + resolution: {integrity: sha512-5NPItMgM9GfIW0SbpOXvEFIijin5VcNEParmBEm9j+RXr8Ko7pZ3a1ydk/Sy+EOvaSTkXMY6TkVyLfczwdudSQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + libc: [musl] + + '@oxc-transform/binding-openharmony-arm64@0.152.0': + resolution: {integrity: sha512-pNzMq4AhBUV8E/cc41SgYTjzW/AWq1DtEKz1l56Mn1TKfidJwP8sGnIw0EXhD6zKFUlPoqkWfvoe5TiDAYeT+w==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [openharmony] + + '@oxc-transform/binding-win32-arm64-msvc@0.152.0': + resolution: {integrity: sha512-PU6cGmVDYo2HHG9ul6vKj189YJPZrcscdWzQakJD6NU4CqvqISbZLUHwz6DVe0v8CbOe2d24zUdipILaRofZ5g==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [win32] + + '@oxc-transform/binding-win32-ia32-msvc@0.152.0': + resolution: {integrity: sha512-Sdcoky580dPgj9Qp+i4zrCV7NpIYZW6bBrcQvOwyDmNlJQ2iYK/V3Q3FGNHLulrxgXrpzqQwMcLs8KRIg0u0yw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [ia32] + os: [win32] + + '@oxc-transform/binding-win32-x64-msvc@0.152.0': + resolution: {integrity: sha512-QnN+/Hsig/vI9zoCNVbbqv0GmZWUGqlPg2ZTEyVw/PvgAXd8pRRfN6CxdKEtJTnaVwJFVYaw/5LR6ZWCVBwDeQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [win32] + '@oxlint/binding-android-arm-eabi@1.83.0': resolution: {integrity: sha512-0yGY24EwsLk5YDe6F+VkmZyRHSwJDALa3nIrPpq7FXmp2lV2d0TzvBCGeZk+wgiULRGr5blhyr4QMp5KCXJUqA==} engines: {node: ^20.19.0 || >=22.12.0} @@ -3001,8 +3333,8 @@ packages: engines: {node: '>=18.18'} hasBin: true - agent-device@0.21.18: - resolution: {integrity: sha512-ptNJ7a4jkAXFLSmZqKJDwxL+YpCEUuZv3oCBY76PwE3b2W+yQqQuG0ej0/bOspIHJRmMGNOkWhtxm3gmRW9hRQ==} + agent-device@0.21.20: + resolution: {integrity: sha512-nD2QNRTbQo3mMEnpOTqgguIreY98lri+VfWBo0Njzc+CLBrISHApUJD2Cs2P8zfzvZCvbGibX1D+LP/B/qeQrQ==} engines: {node: '>=22.12'} hasBin: true peerDependencies: @@ -4351,6 +4683,9 @@ packages: resolution: {integrity: sha512-w9UMqWwJxHNOvoNzSJ2oPF5wvYcvP7jUvYzhp67yEhTi17ZDBBC1z9pTdGuzjD+EFIqLSYRweZjqfiPzQ06Ebg==} engines: {node: '>= 0.4'} + get-tsconfig@4.14.3: + resolution: {integrity: sha512-++QEw4DIY7WGoukz+/+A/8dGYPT9l9yIadnmSgZ8Rjr3YVSVDipQSO9CdnJo9ePqFqUUqh+wk9uIaoiAwsiPkA==} + get-uri@6.0.5: resolution: {integrity: sha512-b1O07XYq8eRuVzBNgJLstU6FYc1tS6wnMtF1I1D9lE8LxZSOGZ7LhxN54yPP6mGw5f2CkXY2BQUL9Fx41qvcIg==} engines: {node: '>= 14'} @@ -5730,6 +6065,14 @@ packages: resolution: {integrity: sha512-19YVAg7T+WTrxggPukVq7DjTv6+PJ867TmhCvBsYwmbFCsZd344rq2Ld1p0wo8f8Qrrhgp82c6FJRqdXWtSEhg==} engines: {node: '>= 0.4'} + oxc-parser@0.152.0: + resolution: {integrity: sha512-ZztsZgaCWmPeKYMdoTT5e0oV/oxD1NeLWNR3kAtr/G707cUPYaaozRlVDHLCGnszHJRszN9mJwmPNpqgwloC0A==} + engines: {node: ^20.19.0 || >=22.12.0} + + oxc-transform@0.152.0: + resolution: {integrity: sha512-9rs8qImolzWswXGlhmye9Yz9Mce9hOe4Pivb7rSEhF/6VKn1Q122BUe9Mr1O9xQAoBX+SrF6xXfdgxKHeQf+uQ==} + engines: {node: ^20.19.0 || >=22.12.0} + oxlint@1.83.0: resolution: {integrity: sha512-cyDzSzaw3uzP0TeCeq3lLRPPoaUxkbB4ZOXj+kn+5r+BX9V+4bNVGk9lxer+WrgcpebH4JxLlJ3KQjveVztOLQ==} engines: {node: ^20.19.0 || >=22.12.0} @@ -5854,11 +6197,6 @@ packages: engines: {node: '>=20'} hasBin: true - playwright@1.63.0: - resolution: {integrity: sha512-+7ziBLidS4NaNCdt57SUDT+wYmmd5fmiQejUic/kb+YsYSCPyOOE9sebzMjNmQrsnNpDJqd4WHvV/8lfKfUDUg==} - engines: {node: '>=20'} - hasBin: true - plist@3.1.1: resolution: {integrity: sha512-ZIfcLJC+7E7FBFnDxm9MPmt7D+DidyQ26lewieO75AdhA2ayMtsJSES0iWzqJQbcVRSrTufQoy0DR94xHue0oA==} engines: {node: '>=10.4.0'} @@ -6258,6 +6596,9 @@ packages: resolution: {integrity: sha512-qYg9KP24dD5qka9J47d0aVky0N+b4fTU89LN9iDnjB5waksiC49rvMB0PrUJQGoTmH50XPiqOvAjDfaijGxYZw==} engines: {node: '>=8'} + resolve-pkg-maps@1.0.0: + resolution: {integrity: sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==} + resolve-workspace-root@2.0.1: resolution: {integrity: sha512-nR23LHAvaI6aHtMg6RWoaHpdR4D881Nydkzi2CixINyg9T00KgaJdJI6Vwty+Ps8WLxZHuxsS0BseWjxSA4C+w==} @@ -7295,6 +7636,12 @@ packages: snapshots: + '@ai-sdk/anthropic@4.0.58(zod@4.6.1)': + dependencies: + '@ai-sdk/provider': 4.0.17 + '@ai-sdk/provider-utils': 5.0.45(zod@4.6.1) + zod: 4.6.1 + '@ai-sdk/devtools@1.0.22': dependencies: '@ai-sdk/provider': 4.0.17 @@ -7308,6 +7655,12 @@ snapshots: '@vercel/oidc': 3.2.0 zod: 4.6.1 + '@ai-sdk/google@4.0.77(zod@4.6.1)': + dependencies: + '@ai-sdk/provider': 4.0.17 + '@ai-sdk/provider-utils': 5.0.45(zod@4.6.1) + zod: 4.6.1 + '@ai-sdk/openai-compatible@3.0.53(zod@4.6.1)': dependencies: '@ai-sdk/provider': 4.0.17 @@ -7329,6 +7682,15 @@ snapshots: undici: 7.29.1 zod: 4.6.1 + '@ai-sdk/provider-utils@5.0.53(zod@4.6.1)': + dependencies: + '@ai-sdk/provider': 4.0.21 + '@standard-schema/spec': 1.1.0 + '@workflow/serde': 4.1.0 + eventsource-parser: 3.1.1 + undici: 7.29.1 + zod: 4.6.1 + '@ai-sdk/provider@4.0.17': dependencies: json-schema: 0.4.0 @@ -7337,6 +7699,16 @@ snapshots: dependencies: json-schema: 0.4.0 + '@ai-sdk/provider@4.0.21': + dependencies: + json-schema: 0.4.0 + + '@ai-sdk/typesafe-ai@3.0.12(zod@4.6.1)': + dependencies: + '@ai-sdk/provider': 4.0.21 + '@ai-sdk/provider-utils': 5.0.53(zod@4.6.1) + zod: 4.6.1 + '@ai-sdk/xai@5.0.4(zod@4.6.1)': dependencies: '@ai-sdk/provider': 4.0.17 @@ -9404,8 +9776,126 @@ snapshots: ai: 7.0.107(zod@4.6.1) zod: 4.6.1 + '@oxc-parser/binding-android-arm-eabi@0.152.0': + optional: true + + '@oxc-parser/binding-android-arm64@0.152.0': + optional: true + + '@oxc-parser/binding-darwin-arm64@0.152.0': + optional: true + + '@oxc-parser/binding-darwin-x64@0.152.0': + optional: true + + '@oxc-parser/binding-freebsd-x64@0.152.0': + optional: true + + '@oxc-parser/binding-linux-arm-gnueabihf@0.152.0': + optional: true + + '@oxc-parser/binding-linux-arm-musleabihf@0.152.0': + optional: true + + '@oxc-parser/binding-linux-arm64-gnu@0.152.0': + optional: true + + '@oxc-parser/binding-linux-arm64-musl@0.152.0': + optional: true + + '@oxc-parser/binding-linux-ppc64-gnu@0.152.0': + optional: true + + '@oxc-parser/binding-linux-riscv64-gnu@0.152.0': + optional: true + + '@oxc-parser/binding-linux-riscv64-musl@0.152.0': + optional: true + + '@oxc-parser/binding-linux-s390x-gnu@0.152.0': + optional: true + + '@oxc-parser/binding-linux-x64-gnu@0.152.0': + optional: true + + '@oxc-parser/binding-linux-x64-musl@0.152.0': + optional: true + + '@oxc-parser/binding-openharmony-arm64@0.152.0': + optional: true + + '@oxc-parser/binding-win32-arm64-msvc@0.152.0': + optional: true + + '@oxc-parser/binding-win32-ia32-msvc@0.152.0': + optional: true + + '@oxc-parser/binding-win32-x64-msvc@0.152.0': + optional: true + + '@oxc-project/runtime@0.152.0': {} + '@oxc-project/types@0.139.0': {} + '@oxc-project/types@0.152.0': {} + + '@oxc-transform/binding-android-arm-eabi@0.152.0': + optional: true + + '@oxc-transform/binding-android-arm64@0.152.0': + optional: true + + '@oxc-transform/binding-darwin-arm64@0.152.0': + optional: true + + '@oxc-transform/binding-darwin-x64@0.152.0': + optional: true + + '@oxc-transform/binding-freebsd-x64@0.152.0': + optional: true + + '@oxc-transform/binding-linux-arm-gnueabihf@0.152.0': + optional: true + + '@oxc-transform/binding-linux-arm-musleabihf@0.152.0': + optional: true + + '@oxc-transform/binding-linux-arm64-gnu@0.152.0': + optional: true + + '@oxc-transform/binding-linux-arm64-musl@0.152.0': + optional: true + + '@oxc-transform/binding-linux-ppc64-gnu@0.152.0': + optional: true + + '@oxc-transform/binding-linux-riscv64-gnu@0.152.0': + optional: true + + '@oxc-transform/binding-linux-riscv64-musl@0.152.0': + optional: true + + '@oxc-transform/binding-linux-s390x-gnu@0.152.0': + optional: true + + '@oxc-transform/binding-linux-x64-gnu@0.152.0': + optional: true + + '@oxc-transform/binding-linux-x64-musl@0.152.0': + optional: true + + '@oxc-transform/binding-openharmony-arm64@0.152.0': + optional: true + + '@oxc-transform/binding-win32-arm64-msvc@0.152.0': + optional: true + + '@oxc-transform/binding-win32-ia32-msvc@0.152.0': + optional: true + + '@oxc-transform/binding-win32-x64-msvc@0.152.0': + optional: true + '@oxlint/binding-android-arm-eabi@1.83.0': optional: true @@ -10210,7 +10700,7 @@ snapshots: agent-cli-detector@0.1.7: {} - agent-device@0.21.18(ai@7.0.107(zod@4.6.1)): + agent-device@0.21.20(ai@7.0.107(zod@4.6.1)): optionalDependencies: ai: 7.0.107(zod@4.6.1) @@ -11217,6 +11707,7 @@ snapshots: '@esbuild/win32-arm64': 0.28.2 '@esbuild/win32-ia32': 0.28.2 '@esbuild/win32-x64': 0.28.2 + optional: true escalade@3.2.0: {} @@ -11779,6 +12270,10 @@ snapshots: es-errors: 1.3.0 get-intrinsic: 1.3.0 + get-tsconfig@4.14.3: + dependencies: + resolve-pkg-maps: 1.0.0 + get-uri@6.0.5(supports-color@8.1.1): dependencies: basic-ftp: 5.3.1 @@ -13828,6 +14323,52 @@ snapshots: object-keys: 1.1.1 safe-push-apply: 1.0.0 + oxc-parser@0.152.0: + dependencies: + '@oxc-project/types': 0.152.0 + optionalDependencies: + '@oxc-parser/binding-android-arm-eabi': 0.152.0 + '@oxc-parser/binding-android-arm64': 0.152.0 + '@oxc-parser/binding-darwin-arm64': 0.152.0 + '@oxc-parser/binding-darwin-x64': 0.152.0 + '@oxc-parser/binding-freebsd-x64': 0.152.0 + '@oxc-parser/binding-linux-arm-gnueabihf': 0.152.0 + '@oxc-parser/binding-linux-arm-musleabihf': 0.152.0 + '@oxc-parser/binding-linux-arm64-gnu': 0.152.0 + '@oxc-parser/binding-linux-arm64-musl': 0.152.0 + '@oxc-parser/binding-linux-ppc64-gnu': 0.152.0 + '@oxc-parser/binding-linux-riscv64-gnu': 0.152.0 + '@oxc-parser/binding-linux-riscv64-musl': 0.152.0 + '@oxc-parser/binding-linux-s390x-gnu': 0.152.0 + '@oxc-parser/binding-linux-x64-gnu': 0.152.0 + '@oxc-parser/binding-linux-x64-musl': 0.152.0 + '@oxc-parser/binding-openharmony-arm64': 0.152.0 + '@oxc-parser/binding-win32-arm64-msvc': 0.152.0 + '@oxc-parser/binding-win32-ia32-msvc': 0.152.0 + '@oxc-parser/binding-win32-x64-msvc': 0.152.0 + + oxc-transform@0.152.0: + optionalDependencies: + '@oxc-transform/binding-android-arm-eabi': 0.152.0 + '@oxc-transform/binding-android-arm64': 0.152.0 + '@oxc-transform/binding-darwin-arm64': 0.152.0 + '@oxc-transform/binding-darwin-x64': 0.152.0 + '@oxc-transform/binding-freebsd-x64': 0.152.0 + '@oxc-transform/binding-linux-arm-gnueabihf': 0.152.0 + '@oxc-transform/binding-linux-arm-musleabihf': 0.152.0 + '@oxc-transform/binding-linux-arm64-gnu': 0.152.0 + '@oxc-transform/binding-linux-arm64-musl': 0.152.0 + '@oxc-transform/binding-linux-ppc64-gnu': 0.152.0 + '@oxc-transform/binding-linux-riscv64-gnu': 0.152.0 + '@oxc-transform/binding-linux-riscv64-musl': 0.152.0 + '@oxc-transform/binding-linux-s390x-gnu': 0.152.0 + '@oxc-transform/binding-linux-x64-gnu': 0.152.0 + '@oxc-transform/binding-linux-x64-musl': 0.152.0 + '@oxc-transform/binding-openharmony-arm64': 0.152.0 + '@oxc-transform/binding-win32-arm64-msvc': 0.152.0 + '@oxc-transform/binding-win32-ia32-msvc': 0.152.0 + '@oxc-transform/binding-win32-x64-msvc': 0.152.0 + oxlint@1.83.0: optionalDependencies: '@oxlint/binding-android-arm-eabi': 1.83.0 @@ -13960,10 +14501,6 @@ snapshots: playwright-core@1.63.0: {} - playwright@1.63.0: - dependencies: - playwright-core: 1.63.0 - plist@3.1.1: dependencies: '@xmldom/xmldom': 0.9.12 @@ -14553,6 +15090,8 @@ snapshots: resolve-from@5.0.0: {} + resolve-pkg-maps@1.0.0: {} + resolve-workspace-root@2.0.1: {} resolve@1.22.12: @@ -15330,6 +15869,7 @@ snapshots: esbuild: 0.28.2 optionalDependencies: fsevents: 2.3.3 + optional: true tunnel-agent@0.6.0: dependencies: diff --git a/scripts/check-docs-examples.ts b/scripts/check-docs-examples.ts index aff292df2..aed7a0aad 100644 --- a/scripts/check-docs-examples.ts +++ b/scripts/check-docs-examples.ts @@ -30,6 +30,8 @@ const EXAMPLES: Record = { 'docs/examples/mobile/testmu.config.ts': 'docs/integrations/testmu.mdx', 'docs/examples/mobile/testmu.e2e.ts': 'docs/integrations/testmu.mdx', 'docs/examples/web/browser-provider.ts': 'docs/browser.mdx', + 'docs/examples/models/e2e.decision.config.ts': 'docs/decision-models.mdx', + 'docs/examples/models/tests/decision.e2e.ts': 'docs/decision-models.mdx', 'docs/examples/skill/e2e.config.ts': 'skills/e2e/SKILL.md', 'docs/examples/skill/e2e.setup.config.ts': 'skills/e2e/references/setup.md', }; diff --git a/scripts/check-error-codes.ts b/scripts/check-error-codes.ts index ae8fcc18b..6914d238e 100644 --- a/scripts/check-error-codes.ts +++ b/scripts/check-error-codes.ts @@ -22,6 +22,7 @@ const SOURCE_ROOTS = [ 'packages/web/src', 'packages/mobile/src', 'packages/github/src', + 'packages/decision/src', ]; const TYPES_FILE = 'packages/e2e/src/types.ts'; const CONTRACT_FILE = 'packages/e2e/src/engine/contract.ts'; diff --git a/scripts/check-install-scripts.ts b/scripts/check-install-scripts.ts new file mode 100644 index 000000000..fd34b59e7 --- /dev/null +++ b/scripts/check-install-scripts.ts @@ -0,0 +1,29 @@ +/** + * Fails when installing a published package would run a dependency's + * install script (`install-scripts.ts` says why that breaks pnpm 11 and + * later for every new user). + * + * Usage: `node scripts/check-install-scripts.ts` after `pnpm install`, part + * of `pnpm check`. Exits 1 listing every package that would run one and the + * dependency chain that brings it in. + */ + +import { dirname } from 'node:path'; +import { findInstallScripts } from './install-scripts.ts'; +import { publicPackages } from './public-packages.ts'; + +const problems: string[] = []; +const checked: string[] = []; +for (const { path, manifest } of publicPackages()) { + checked.push(manifest.name); + for (const { chain, runs } of findInstallScripts(dirname(path))) { + problems.push(`${chain} runs an install script (${runs}); every pnpm 11+ install of ${manifest.name} fails on it`); + } +} + +if (problems.length > 0) { + process.stderr.write(`${problems.join('\n')}\n`); + process.exitCode = 1; +} else { + process.stdout.write(`install scripts: none in the dependency trees of ${checked.toSorted().join(', ')}\n`); +} diff --git a/scripts/install-scripts.test.ts b/scripts/install-scripts.test.ts new file mode 100644 index 000000000..a98fd91a2 --- /dev/null +++ b/scripts/install-scripts.test.ts @@ -0,0 +1,53 @@ +import assert from 'node:assert/strict'; +import { mkdirSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { after, describe, it } from 'node:test'; +import { findInstallScripts } from './install-scripts.ts'; + +const root = mkdtempSync(join(tmpdir(), 'install-scripts-')); +after(() => rmSync(root, { recursive: true, force: true })); + +/** Writes a package at `dir` with `manifest` and any extra files. */ +function pkg(dir: string, manifest: Record, files: Record = {}): string { + const path = join(root, dir); + mkdirSync(path, { recursive: true }); + writeFileSync(join(path, 'package.json'), JSON.stringify({ version: '1.0.0', ...manifest })); + for (const [name, content] of Object.entries(files)) writeFileSync(join(path, name), content); + return path; +} + +describe('findInstallScripts', () => { + it('finds install scripts and native builds anywhere in the tree, by the chain that brings them in', () => { + const app = pkg('app', { + name: 'app', + scripts: { build: 'tsc', prepublishOnly: 'check' }, + dependencies: { loader: '1', 'native-both': '1' }, + optionalDependencies: { 'native-other-os': '1', 'native-both': '1' }, + peerDependencies: { peer: '1', 'optional-peer': '1' }, + peerDependenciesMeta: { 'optional-peer': { optional: true } }, + }); + pkg('app/node_modules/loader', { name: 'loader', dependencies: { bundler: '1', shared: '1' } }); + pkg('store/bundler', { name: 'bundler', version: '0.28.2', scripts: { postinstall: 'node install.js' }, dependencies: { shared: '1' } }); + // pnpm's layout: the real directory elsewhere, a link in node_modules. + mkdirSync(join(root, 'app/node_modules/loader/node_modules'), { recursive: true }); + symlinkSync(join(root, 'store/bundler'), join(root, 'app/node_modules/loader/node_modules/bundler'), 'junction'); + pkg('node_modules/shared', { name: 'shared' }); + pkg('node_modules/peer', { name: 'peer' }, { 'binding.gyp': '{}' }); + + assert.deepEqual(findInstallScripts(app), [ + { chain: 'app > peer@1.0.0', runs: 'node-gyp rebuild (binding.gyp)' }, + { chain: 'app > loader@1.0.0 > bundler@0.28.2', runs: 'postinstall: node install.js' }, + ]); + }); + + it('finds an install script of the package itself', () => { + const own = pkg('own', { name: 'own', scripts: { postinstall: 'node setup.js' } }); + assert.deepEqual(findInstallScripts(own), [{ chain: 'own', runs: 'postinstall: node setup.js' }]); + }); + + it('refuses a tree with a required dependency missing', () => { + const lonely = pkg('lonely', { name: 'lonely', dependencies: { absent: '1' } }); + assert.throws(() => findInstallScripts(lonely), /lonely > absent is not installed/); + }); +}); diff --git a/scripts/install-scripts.ts b/scripts/install-scripts.ts new file mode 100644 index 000000000..2f95e4aad --- /dev/null +++ b/scripts/install-scripts.ts @@ -0,0 +1,104 @@ +/** + * Finds the packages a consumer's install would run a script for, in the + * dependency tree of a package as installed on this machine. + * + * pnpm 11 and later fail an install on any dependency build script the + * project has not decided on (`ERR_PNPM_IGNORED_BUILDS`), so one such + * package anywhere under a published package breaks `pnpm install` for every + * new user. npm runs a package's `preinstall`, `install`, and `postinstall`, + * and `node-gyp rebuild` for a `binding.gyp` with no install script of its + * own; any of them counts. + * + * The walk follows `dependencies`, `optionalDependencies`, and the + * `peerDependencies` an install adds unasked (npm 7 and later, pnpm), the way + * Node.js resolves them from each package's real path. An optional + * dependency this platform skipped (another OS's native binary) is not + * there to read. + */ + +import { existsSync, readFileSync, realpathSync } from 'node:fs'; +import { dirname, join } from 'node:path'; + +const INSTALL_HOOKS = ['preinstall', 'install', 'postinstall'] as const; + +interface Manifest { + name: string; + version?: string; + scripts?: Record; + dependencies?: Record; + optionalDependencies?: Record; + peerDependencies?: Record; + peerDependenciesMeta?: Record; +} + +export interface InstallScript { + /** `root > dependency > ... > package@version`. */ + readonly chain: string; + /** What an install would run: the scripts, or `node-gyp rebuild`. */ + readonly runs: string; +} + +function readManifest(dir: string): Manifest { + return JSON.parse(readFileSync(join(dir, 'package.json'), 'utf8')) as Manifest; +} + +/** The real directory of `name` as Node.js resolves it from `fromDir`, or undefined when it is not installed. */ +function locate(name: string, fromDir: string): string | undefined { + for (let dir = fromDir; ; dir = dirname(dir)) { + const candidate = join(dir, 'node_modules', name); + if (existsSync(join(candidate, 'package.json'))) return realpathSync(candidate); + if (dirname(dir) === dir) return undefined; + } +} + +/** What installing the package in `dir` runs, or undefined when nothing. */ +function installRuns(dir: string, manifest: Manifest): string | undefined { + const hooks = INSTALL_HOOKS.filter((hook) => manifest.scripts?.[hook] !== undefined); + if (hooks.length > 0) return hooks.map((hook) => `${hook}: ${manifest.scripts![hook]}`).join('; '); + return existsSync(join(dir, 'binding.gyp')) ? 'node-gyp rebuild (binding.gyp)' : undefined; +} + +/** + * The dependencies an install of `manifest` brings in, each with whether it + * may be absent. A name in `optionalDependencies` is optional even when + * `dependencies` names it too, as npm reads it. + */ +function installedDependencies(manifest: Manifest): [name: string, optional: boolean][] { + const optional = new Set(Object.keys(manifest.optionalDependencies ?? {})); + const peers = Object.keys(manifest.peerDependencies ?? {}).filter((name) => manifest.peerDependenciesMeta?.[name]?.optional !== true); + const required = [...Object.keys(manifest.dependencies ?? {}), ...peers].filter((name) => !optional.has(name)); + return [ + ...[...new Set(required)].map((name): [string, boolean] => [name, false]), + ...[...optional].map((name): [string, boolean] => [name, true]), + ]; +} + +/** + * The package in `packageDir`, then every package in its installed + * dependency tree, whose install runs a script. Throws when a required + * dependency is not installed: the tree cannot be read. + */ +export function findInstallScripts(packageDir: string): InstallScript[] { + const root = readManifest(packageDir); + const ownRuns = installRuns(packageDir, root); + const found: InstallScript[] = ownRuns === undefined ? [] : [{ chain: root.name, runs: ownRuns }]; + const seen = new Set([realpathSync(packageDir)]); + const queue: { dir: string; manifest: Manifest; chain: string }[] = [{ dir: realpathSync(packageDir), manifest: root, chain: root.name }]; + for (let next = queue.shift(); next !== undefined; next = queue.shift()) { + for (const [name, optional] of installedDependencies(next.manifest)) { + const dir = locate(name, next.dir); + if (dir === undefined) { + if (optional) continue; + throw new Error(`${next.chain} > ${name} is not installed; run pnpm install`); + } + if (seen.has(dir)) continue; + seen.add(dir); + const manifest = readManifest(dir); + const chain = `${next.chain} > ${manifest.name}@${manifest.version ?? '?'}`; + const runs = installRuns(dir, manifest); + if (runs !== undefined) found.push({ chain, runs }); + queue.push({ dir, manifest, chain }); + } + } + return found; +} diff --git a/scripts/install-smoke.ts b/scripts/install-smoke.ts new file mode 100644 index 000000000..f4188b7e8 --- /dev/null +++ b/scripts/install-smoke.ts @@ -0,0 +1,93 @@ +/** + * A first install, the way a new user meets it: packs the built `e2e`, + * `@e2e-dev/web`, and `@e2e-dev/mobile`, installs them with a given pnpm + * into a fresh project outside this repository, installs again from the + * lockfile, and runs a TypeScript test through the installed CLI. + * + * pnpm 11 and later fail an install on any dependency build script the + * project has not decided on (`ERR_PNPM_IGNORED_BUILDS`), and write a + * `pnpm-workspace.yaml` asking for the decision. Either one fails this + * check. `check-install-scripts.ts` names the package that would cause it; + * this proves the install itself. + * + * Usage: `node scripts/install-smoke.ts ` after + * `pnpm run build`, e.g. `node scripts/install-smoke.ts 11`. Needs the npm + * registry for the published dependencies. Exits 1 naming what failed. + */ + +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdirSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { REPO_ROOT } from './public-packages.ts'; + +const pnpmVersion = process.argv[2]; +if (pnpmVersion === undefined) { + process.stderr.write('usage: node scripts/install-smoke.ts \n'); + process.exit(2); +} + +/** The packages a new user installs: the runner and its engines. */ +const PACKAGES = ['e2e', 'web', 'mobile'] as const; + +const root = mkdtempSync(join(tmpdir(), 'e2e-install-smoke-')); +const tarballs = join(root, 'tarballs'); +const project = join(root, 'project'); +mkdirSync(tarballs); +mkdirSync(join(project, 'tests'), { recursive: true }); + +/** Runs `command` in `cwd`; returns its exit code and combined output. */ +function run(cwd: string, command: string, args: readonly string[]): { status: number; output: string } { + // pnpm's release-age gate would block a dependency bump for a day; this check is about build scripts. + const env = { ...process.env, E2E_TELEMETRY_DISABLED: '1', pnpm_config_minimum_release_age: '0' }; + const result = spawnSync(command, args, { cwd, env, encoding: 'utf8', shell: process.platform === 'win32' }); + return { status: result.status ?? 1, output: `${result.stdout}${result.stderr}` }; +} + +const pnpm = (args: readonly string[]) => run(project, 'npx', ['--yes', `pnpm@${pnpmVersion}`, ...args]); + +const problems: string[] = []; +try { + for (const name of PACKAGES) { + const packed = run(join(REPO_ROOT, 'packages', name), 'pnpm', ['pack', '--pack-destination', tarballs]); + if (packed.status !== 0) throw new Error(`pnpm pack failed in packages/${name}:\n${packed.output}`); + } + const files = readdirSync(tarballs); + const tarball = (prefix: string): string => `file:${join(tarballs, files.find((file) => file.startsWith(prefix))!)}`; + writeFileSync( + join(project, 'package.json'), + JSON.stringify({ + name: 'install-smoke', + private: true, + type: 'module', + devDependencies: { e2e: tarball('e2e-0'), '@e2e-dev/web': tarball('e2e-dev-web-'), '@e2e-dev/mobile': tarball('e2e-dev-mobile-') }, + }), + ); + writeFileSync(join(project, 'e2e.config.ts'), "import type { E2EConfig } from 'e2e';\n\nexport default { targets: [{ name: 'local', platform: 'test' }] } satisfies E2EConfig;\n"); + writeFileSync( + join(project, 'tests', 'smoke.e2e.ts'), + "import { expect, test } from 'e2e';\n\nenum Answer { Yes = 'yes' }\n\ntest('TypeScript runs', () => {\n expect(Answer.Yes).toBe('yes');\n});\n", + ); + + for (const [step, args] of [ + ['pnpm install', ['install']], + ['pnpm install --frozen-lockfile', ['install', '--frozen-lockfile']], + ['e2e run', ['exec', 'e2e', 'run', '--workers', '1']], + ] as const) { + const { status, output } = pnpm(args); + process.stdout.write(`$ ${step}: exit ${status}\n`); + if (status !== 0) problems.push(`${step} exited ${status}:\n${output}`); + if (output.includes('ERR_PNPM_IGNORED_BUILDS')) problems.push(`${step} reported ERR_PNPM_IGNORED_BUILDS:\n${output}`); + if (existsSync(join(project, 'pnpm-workspace.yaml'))) problems.push(`${step} wrote a pnpm-workspace.yaml`); + if (problems.length > 0) break; + } +} finally { + rmSync(root, { recursive: true, force: true }); +} + +if (problems.length > 0) { + process.stderr.write(`${problems.join('\n\n')}\n`); + process.exitCode = 1; +} else { + process.stdout.write(`install smoke: pnpm ${pnpmVersion} installs ${PACKAGES.join(', ')} with no build scripts and runs a TypeScript test\n`); +} diff --git a/skills/e2e/SKILL.md b/skills/e2e/SKILL.md index 10aea08b6..0067ff17e 100644 --- a/skills/e2e/SKILL.md +++ b/skills/e2e/SKILL.md @@ -10,8 +10,8 @@ one goal; `agent.assert`, `agent.waitFor`, and `agent.extract` judge the screen. `screen`, `app`, `browser`, and `expect` make exact interactions and checks. The replay cache reruns verified actions and checks their recorded end state without a model call; agent judgments still run live. UI targets use -`@e2e-dev/web` for browsers or `@e2e-dev/mobile` for iOS simulators and -Android emulators. A test that takes only `app` can check an API with `fetch` +`@e2e-dev/web` for browsers or `@e2e-dev/mobile` for iOS simulators, +Android emulators, and connected phones. A test that takes only `app` can check an API with `fetch` and `expect` (topic `writing-tests`). Model sign-in commands are in [setup](references/setup.md#subscriptions-and-api-keys). diff --git a/skills/e2e/references/agent.md b/skills/e2e/references/agent.md index 3d190d459..180e89067 100644 --- a/skills/e2e/references/agent.md +++ b/skills/e2e/references/agent.md @@ -6,8 +6,13 @@ model-call budget, no shared transcript. No agent step, no model calls. ## Configure a model -Put an AI SDK model under `agents.default`. Vercel AI Gateway reads -`AI_GATEWAY_API_KEY` or, without it, a Vercel OIDC token: +Put an AI SDK model under `agents.default`. You can use all AI SDK providers +that support tool calls and the language model specification v2 or later +(https://ai-sdk.dev/providers). Examples: `@ai-sdk/openai`, +`@ai-sdk/anthropic`, `@ai-sdk/google`, `@ai-sdk/amazon-bedrock`, +`@openrouter/ai-sdk-provider`, `ollama-ai-provider-v2`. For a server with +`/v1/chat/completions`, use `@ai-sdk/openai-compatible`. Vercel AI Gateway +reads `AI_GATEWAY_API_KEY` or, without it, a Vercel OIDC token: ```ts import type { E2EConfig } from 'e2e'; @@ -24,6 +29,14 @@ Subscription logins and local models: [setup](setup.md#subscriptions-and-api-keys). Keep `ai@^7` installed with any provider. +Decision models pick actions from choices the executor builds: install +`@e2e-dev/decision` and put `decisionExecutor({ model: +typeSafeAi.evaluationModel('jev-latest'), textModel: openrouter('inception/mercury-2.5') })` +under `executor`. Tests stay plain language with no params; the text model +writes field values. See the shipped `docs/decision-models.mdx` or +[the online guide](https://e2e.tester.army/docs/decision-models) for setup, +gates, and limits. + - Pass a model instance, not a string (`INVALID_CONFIG`). - An agents entry is one plain object of `model`, `judge`, `system`, `context`, `tools`, `maxSteps`, `maxModelCalls`, `judgmentTimeout`, @@ -32,7 +45,8 @@ any provider. - `model` drives `agent.act`. Judgments use `judge` when set, else `model`. - A missing model for the built-in agent raises one run-level `MODEL_UNAVAILABLE` when the first test acquires `agent`, exit 2; auth - failures surface on the first model call as `MODEL_PROVIDER_FAILED`. + failures surface on the first model call as `MODEL_PROVIDER_FAILED`, or as + `CONFIG_LOAD_FAILED` for a provider that reads its key when the config loads. - `context` is what the app calls things, sent to every model call, judges included; `system` is how the acting agent works, read only by the act loop. Both live on the agents entry; `agentContext` on a test or group @@ -50,8 +64,8 @@ Tests use `agents.default` unless selected otherwise: - `{ agent: 'name' }` on an `agent.*` call overrides the test's choice. For signed-in personas, pair an agent with a `session` in a describe block -repeated per persona; the cache records per agent step, so a specialised -agent replays too. +repeated per persona; the cache keys every entry by the agent and its +context, so each persona records and replays its own steps. ## act: one goal @@ -236,14 +250,19 @@ matches. Misses and hand-offs use the model; `agent.assert`, `agent.assert`, `agent.waitFor`), so an unchecked `act` never replays; a plain-value `expect`, `expect.poll`, `agent.extract`, another `act`, or the attempt passing confirms nothing. -- A replay needs the app on the recorded path (unless the recording opens - with a navigation), re-finds each control by role, name, test id, - placeholder, and input purpose, and passes alone only when the recorded - end path and the controls seen during the step are back; otherwise the - agent takes over mid-step. `step.cache.reason` says why: `no-entry`, +- A replay needs the app on the recorded route (origin, path, and query, + ids and tokens aside; unless the recording opens with a navigation), + re-finds each control by role, name, test id, placeholder, input purpose, + and its named row (an unnamed control with twins and no named container + hands off), and passes alone only when the recorded end route is back, + every control that appeared (with its checked or selected state) is + there, every one that went away is gone, at least one of those changed + during the replay, and no new alert showed; otherwise the agent takes + over mid-step. Entries are keyed per agent and per `agentContext`. `step.cache.reason` says why: `no-entry`, `wrong-context`, `target-not-found`, `target-ambiguous`, `end-mismatch`, and so on. -- A step recording no actions creates no entry; one whose `unique()` value +- A step recording no actions, or changing nothing on screen or in the + route, creates no entry; one whose `unique()` value equals, is spelled inside, or is the encoded form of another param's value is not recorded either (`step.cache.notRecorded`: `param-collision`). - `e2e init` gitignores `.e2e/cache/`; remove that line to commit entries diff --git a/skills/e2e/references/debugging.md b/skills/e2e/references/debugging.md index abb9176b3..d3c46295a 100644 --- a/skills/e2e/references/debugging.md +++ b/skills/e2e/references/debugging.md @@ -28,7 +28,7 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] (recorded as `skipped`). 4. Artifacts, under `.e2e/artifacts/`: `failure/screen.txt` and the engine's screenshot per failed attempt; a Playwright trace per - traced attempt (`npx playwright show-trace `); downloads; with + traced attempt (`npx playwright-core@1.63.0 show-trace `; `pnpm dlx` under pnpm); downloads; with `--video` the recording (`video/video.webm` in a local browser, each later page `video/video-part.webm` with its own `startedAt`; `video/video.mp4` on a device; a provider's file or link); with @@ -46,7 +46,7 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] | `COLLECTION_ERROR` | `async` describe body, `test.setup` inside `describe`, an option forbidden in a serial group, registration outside collection | Restructure per `writing-tests` | | `HOOK_FAILED` | `beforeAll` or `afterAll` threw; its scope's tests skip | Fix the hook; the report carries its error | | `UNSUPPORTED_ARTIFACT` | A test's or target's `trace` or `video` on an engine that cannot record | Drop it there, or set it at the config root or CLI (such targets skip with a notice) | -| `BROWSER_INSTALL_FAILED`, `LAUNCH_TIMEOUT` | Browser download failed; engine init or attempt start exceeded `launchTimeout` | Run the quoted `npx playwright install ` (`--with-deps` on bare Linux); raise the root `launchTimeout` (60 s default) | +| `BROWSER_INSTALL_FAILED`, `LAUNCH_TIMEOUT` | Browser download failed; engine init or attempt start exceeded `launchTimeout` | Run the quoted `npx @e2e-dev/web install ` or `pnpm exec e2e-web install ` (`--with-deps` on bare Linux); raise the root `launchTimeout` (60 s default) | | `APP_UNREACHABLE` | `app.command` never answered `readyUrl` within `startupTimeout`; on a device, a message naming the iOS automation runner: the runner failed, not the app | Read the quoted log lines; check the port, `app.url`, `app.command.env`, `app.command.startupTimeout`. Runner: rerun, else `npx agent-device daemon stop` and reboot the simulator | | `APP_ALREADY_RUNNING` | Something already serves `url` when `command` should start | Stop it, or `reuseExisting: true` locally | | `APP_URL_REQUIRED`, `APP_NOT_OPEN` | A navigation on a target without `app.url`; a `screen` call before `app.open()` | Add `app.url` to the target; open the app first | @@ -58,14 +58,14 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] | `MODEL_OUTPUT_INVALID` | An `extract` or `assert` answer failed the schema after one repair round | Simplify the schema or question; pick a stronger model | | `ACTION_FAILED` | Element not actionable (covered, disabled, detached), or an operation timed out | `expect` the condition first; close overlays; check `actionTimeout` | | `TEST_TIMEOUT` | The attempt exceeded `timeout` (120 s) | Split the test, or raise `timeout` | -| `STEP_NOT_AWAITED` | The body returned while a step still ran: a call without `await` | `await` the call the code frame names (every `app`, `agent`, `screen`, `expect` call) | +| `STEP_NOT_AWAITED` | The body returned while a step or `expect.poll` still ran, or an `afterEach` hook or fixture teardown returned while an `expect.poll` still ran: a call without `await` | `await` the call the code frame names (every `app`, `agent`, `screen`, `expect`, `expect.poll` call) | | `MODEL_UNAVAILABLE` | `agents..model` holds no AI SDK instance: reported once under `run.errors` at the first `agent` fixture; the run stops. A model-less custom executor fails only `waitFor` and `extract` this way | Construct one, e.g. `gateway('openai/gpt-6-luna-fast')` from `ai`, and export its key (`AI_GATEWAY_API_KEY`) | | `MODEL_PROVIDER_FAILED` | Network, 5xx, rate limit, no credits, or no response in 120 s, after the transport retries | Check the credential (the key; for keyless `gateway()` the Vercel CLI login and `.vercel/project.json`) and quota; retry (exit 3) | | `REPLAY_STALE` | `--strict-cache` or `cache.strict`, and a committed recording no longer replays; `step.cache.reason` says why | Rerun read-write with the knob the message names off; commit the changed entry under the directory it names (a custom `cache.store` is written by that run) | | `AUTOMATION_UNSUPPORTED` | The step needs an interaction the engine's toolset lacks (drag on a device, say); blocked, exit 1 | Do that step with `screen` actions, or run the test on a target that supports it | | `STEP_TIMEOUT`, `STEP_BUDGET_EXHAUSTED` | Goal too big or ambiguous, or a slow provider | Split the goal, use on-screen wording, add vocabulary via `agents..context` or `agentContext`; raise `maxSteps` or `maxModelCalls` (budget), the config `timeout` (`act`), `judgmentTimeout` (judgments); `--debug` | | `CONTEXT_OVERFLOW` | Screen plus step history did not fit the model's context window (`act` after one shrink-and-retry; a judgment on the first overflow) | Lower `agents..maxObservationBytes`, split the step, or pick a larger-window model | -| `POLICY_DENIED` | A `file:`, `data:`, or `javascript:` URL; a password `Secret` into a non-password sink; reading a secure field, `toHaveValue`, `toHaveText`, `toContainText`, `toHaveAttribute` included; `app.screenshot()` after a secret fill | http(s) only; passwords into password inputs only; assert the outcome, not the value; screenshot before filling secrets | +| `POLICY_DENIED` | A URL whose scheme is not `http:` or `https:` (`file:`, `view-source:`, `data:`); a password `Secret` into a non-password sink; reading a secure field, `toHaveValue`, `toHaveText`, `toContainText`, `toHaveAttribute` included; `app.screenshot()` after a secret fill | http(s) or `about:blank` only; passwords into password inputs only; assert the outcome, not the value; screenshot before filling secrets | | `UNSUPPORTED_CAPABILITY` | A fixture the engine lacks (`browser` on a device), `schema` or `vision` on `act`, an action the surface lacks | Declare `requires: ['browser']`; drop the option or action | | `SESSION_UNAVAILABLE`, `SESSION_CONTRACT` | `session: 'x'` with no setup saving `x`; a setup that skipped a declared name. `SESSION_MISMATCH`, `SESSION_EXPIRED`, `SESSION_INVALID`: the stored session is another run's or app's, expired, or corrupt | Add or fix the `test.setup`; rerun it | | `ONLY_IN_CI` | `test.only` reached CI | Remove it | diff --git a/skills/e2e/references/running.md b/skills/e2e/references/running.md index e7c788d2b..e3b24732f 100644 --- a/skills/e2e/references/running.md +++ b/skills/e2e/references/running.md @@ -11,7 +11,7 @@ npx e2e guide [topic] # print this skill; topics: setup, writing-te # running, explore, debugging, mcp, bug-bash npx e2e cache ls|clear|stats # inspect or empty the replay cache npx e2e login|logout|models [provider] # e2e/oauth subscription logins: openai, - # github-copilot, spacexai + # github-copilot, opencode-console, spacexai npx e2e mcp [--target ] # MCP server for a coding agent (topic mcp) npx e2e feedback -m [opts] # report a problem with e2e itself npx e2e telemetry [disable|enable] # anonymous usage telemetry: status or switch @@ -27,7 +27,7 @@ npx e2e telemetry [disable|enable] # anonymous usage telemetry: status or switch | `--tag ` | Any of the tags, comma-separated or repeated; all of them with `--tag-mode all`. An empty `--target`, `--tag`, or `--agent` value is a usage error, exit 2. | | `--exclude-tag ` | Drop tests carrying any of these tags, however selected. | | `--grep `, `--grep-invert ` | Keep, or drop, tests whose title (describe titles and test title joined by spaces, `checkout pays`; not file or tags) matches a regular expression. Bare pattern or `'/pattern/i'`; repeat for alternatives. | -| `--last-failed` | The tests the previous run (`/report.json`) did not pass, plus every test in a failed `beforeAll` or `afterAll` scope. No report is `NO_LAST_RUN`, exit 2. | +| `--last-failed` | The tests the previous run (`/report.json`) did not pass or never ran (`--max-failures` included), plus every test in a failed `beforeAll` or `afterAll` scope. A test another filter leaves out stays owed (`run.carried` in the report) until a rerun runs it. No report is `NO_LAST_RUN`, exit 2. | | `--shard ` | One contiguous slice of the selected tests (`--shard 2/3`), cut after every other filter; serial groups stay whole, each shard brings its own setup tests. | | `--headed` | Visible browser or simulator when the engine supports it. | | `--agent ` | Run unpinned tests as these `agents.` entries (default `agents.default`), comma-separated or repeated; several names run each such test once per agent. | @@ -87,7 +87,8 @@ its own tools. `summary.md`, `failures/`, `ai-trace.json`, `sessions/`, and `artifacts/` (screenshots, Playwright traces, videos, `--debug` transcripts, downloads). `artifacts/` is cleared once a run's tests start; a run stopping before -leaves the last run's files. The report records every artifact path, a +leaves the last run's files, and a `--last-failed` rerun keeps the files +the report it reruns names and writes its own under `artifacts/rerun-/`. The report records every artifact path, a hosted service's video by URL. - `list` (default): setup steps, one line per file and target, a `Failed @@ -102,11 +103,14 @@ hosted service's video by URL. and `run.results[]`, one per test and target: `titlePath`, `file`, `source`, `tags` (`[]` when none), `agent`, `repeat` (0 unless `--repeat-each`), `selected`, `status`, `attempts[]` of `steps[]`, - `artifacts[]`, `error`. + `artifacts[]`, `error`. `run.summary` counts the selected results: + `passed`, `failed` (timed out included), `interrupted` (stopped by the + run: a signal, `--max-failures`, a run-level error; not a failure), `flaky`, and `skipped` add up + to `selected`; `discovered - selected` were left out. - `junit`: `junit.xml` for CI summaries; `--reporter list,junit` keeps the terminal output. - `markdown` (`--reporter list,markdown`): `summary.md` plus one page per - failed or flaky test under `failures/`. The summary holds counts and + failed or flaky test under `failures/` (an interrupted test gets none). The summary holds counts and spend, a block per failed test (error, facts, failing step, whether every attempt failed alike, last model turns, screen location and closest nodes, the line to look at, evidence paths), the flaky tests folded @@ -142,8 +146,9 @@ interrupted workers keeps the failures' or the error's code. Exit 2 is deterministic, never retry it; only exit 3 is worth a job-level retry. The first SIGINT or SIGTERM interrupts and writes the report if any test -had started; the second forces engine teardown; the third kills the app -process groups and exits 130 at once. +had started; the second forces engine teardown and still writes the +report, plus `junit.xml` and `summary.md` when those reporters are on; the +third kills the app process groups and exits 130 at once. ## Continuous integration @@ -171,10 +176,10 @@ jobs: - uses: pnpm/action-setup@9fd676a19091d4595eefd76e4bd31c97133911f1 # v4.2.0 - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: - node-version: 26 # any Node >= 22.12 + node-version: 26 # Node >= 24.8, or >= 22.22.3 on 22 cache: pnpm - run: pnpm install --frozen-lockfile - - run: npx playwright install chromium --with-deps + - run: pnpm exec e2e-web install chromium --with-deps - run: npx e2e run --reporter list,junit env: AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }} diff --git a/skills/e2e/references/setup.md b/skills/e2e/references/setup.md index 0a1331296..88a4bbbc9 100644 --- a/skills/e2e/references/setup.md +++ b/skills/e2e/references/setup.md @@ -2,17 +2,18 @@ ## Requirements -- Node.js 22.12 or newer. +- Node.js 24.8 or newer, or 22.22.3 or newer on Node.js 22. - ES modules: `.ts` config, tests, helpers, and workspace packages exporting `.ts` source load as ESM regardless of the nearest `package.json` `type` (CommonJS packages need no change); never `require` or `module.exports`. -- Browser tests: `@e2e-dev/web` plus `playwright` (`>=1.63.0 <2`), a peer the - engine does not install: an existing Playwright keeps its version and - browser cache, one out of range fails install as an unmet peer (npm's - `ERESOLVE`): upgrade `playwright` within the range. Missing - browsers download on first boot; in CI run `npx playwright install chromium - --with-deps`. Mobile tests: `@e2e-dev/mobile`, pinning `agent-device` - exactly; the pin moves with each engine release. + Imports follow TypeScript: `./x.js` or `./x` loads `x.ts`, and the nearest + `tsconfig.json` `paths` and `baseUrl` apply. Decorators need + `experimentalDecorators`; CommonJS TypeScript goes in `.cts`. +- Browser tests: `@e2e-dev/web`, pinning `playwright-core` exactly; do not + add `playwright` for it. Missing browsers download on first boot; in CI run + `npx @e2e-dev/web install chromium --with-deps` (pnpm: `pnpm exec e2e-web + install chromium --with-deps`). Mobile tests: `@e2e-dev/mobile`, pinning + `agent-device` exactly; each pin moves with its engine release. ## Scaffold @@ -41,7 +42,7 @@ MCP entries. Without the wizard (`ai`, Vercel AI SDK v7, only for `agent.*` steps): ```bash -npm install --save-dev e2e @e2e-dev/web playwright ai@^7 +npm install --save-dev e2e @e2e-dev/web ai@^7 ``` ## Subscriptions and API keys @@ -53,6 +54,7 @@ key, or a local endpoint. Authenticate: | --- | --- | | ChatGPT Plus or Pro | `npx e2e login openai` | | GitHub Copilot | `npx e2e login github-copilot` (GitHub CLI signed in, or your own `--client-id`) | +| OpenCode Console (OpenCode Zen and OpenCode Go) | `npx e2e login opencode-console` (approve the device code, pick the workspace) | | SuperGrok or X Premium+ | `npx e2e login spacexai` | | Vercel AI Gateway | Set `AI_GATEWAY_API_KEY`, or sign in to the Vercel CLI and `npx vercel link`; without the key `gateway()` uses a Vercel OIDC token | | OpenRouter | Set `OPENROUTER_API_KEY` | @@ -63,6 +65,22 @@ set `model: chatgpt('gpt-6-luna')` from `e2e/oauth/chatgpt`, run `npx e2e login openai`. `npx e2e models` lists the ids each login serves. Use API keys in CI. +Switching to Copilot: install `ai`, `@ai-sdk/openai-compatible`, and +`@ai-sdk/openai`, set `model: copilot('')` from `e2e/oauth/copilot`, run +`npx e2e login github-copilot`. `copilot()` calls a model over chat completions, or +over Copilot's Responses API when the plan serves that model only there, choosing +per model from the plan's listing. `npx e2e models github-copilot` marks the models +it cannot call at all: those served only over an API `copilot()` does not speak, and +those the plan has not enabled. + +Switching to OpenCode Console: install `ai`, `@ai-sdk/openai-compatible`, +`@ai-sdk/openai`, `@ai-sdk/anthropic`, and `@ai-sdk/google`, set +`model: opencodeConsole('')` from `e2e/oauth/opencode-console`, run +`npx e2e login opencode-console`. A bare id is an OpenCode Zen model; a `go/` id +(`go/deepseek-v4.1-flash`) is an OpenCode Go model and needs the workspace's Go +subscription. `npx e2e models opencode-console` lists the ids, tagged Zen or Go. In CI, +set a Console service account key as `OPENCODE_API_KEY`. + ## The config `e2e.config.ts` sits at the project root and default-exports an object literal @@ -168,6 +186,8 @@ start a script that brings them up and serves the app. | `headers` | Sent to the app's site only (Vercel's `x-vercel-protection-bypass`, ngrok's `ngrok-skip-browser-warning`), `agent.act` included; disables the browser HTTP cache and service workers. | | `basicAuth` | `{ username, password }` for a `401` challenge; `password` may be `secrets.get('name')`, resolved per attempt and redacted like any secret, the base64 `Authorization` credential too. | | `userAgent` | The `User-Agent` every attempt sends and `navigator.userAgent` reports. | +| `locale`, `timezoneId` | The language (`'de-DE'`: `navigator.language`, `Intl`, `Accept-Language`) and IANA time zone (`'Europe/Berlin'`) every attempt runs in. | +| `initScripts` | Scripts every document runs before the page's own: source, `{ path }`, or a function with no closures. | | `testIdAttribute` | What `getByTestId` reads; default `data-testid`. | | `screencast` | `{ size?, quality? }` for the engine's own video: frame size (default the viewport's), JPEG quality 0 to 100. | @@ -176,7 +196,7 @@ start a script that brings them up and serves the app. the handle (a reference, not the value) in `app.command.env`, a template literal, or `context`; read those from `process.env`. - `reconnectEndpoint` or an attempt-scoped provider rides one persistent - context without `headers`, `basicAuth`, `userAgent`, `app.clearState()`, or + context without `headers`, `basicAuth`, `userAgent`, `locale`, `timezoneId`, `app.clearState()`, or session state. Recovery never repeats a dispatched operation; exhausting the budget is `OPERATION_TIMEOUT`. Without `reconnectEndpoint` a dropped connection is reacquired at the next attempt. @@ -246,16 +266,20 @@ For an app started elsewhere, point `app.url` at it, literally or via | --- | --- | | `AI_GATEWAY_API_KEY`, `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, ... | Read by provider packages, not the runner. | | `E2E_USER__USERNAME`, `E2E_USER__PASSWORD` | Override `credentials.`; `` is the name uppercased, other characters `_`. | -| `E2E_SECRET_` | Overrides `secrets.`, same rule. | +| `E2E_SECRET_` | Overrides `secrets.`, same rule. Two entries of one namespace mapping to one variable are `INVALID_CONFIG`. | | `CI` | CI defaults; list in topic `running`. | | `E2E_TELEMETRY_DISABLED`, `DO_NOT_TRACK` | Disable anonymous telemetry, as does `e2e telemetry disable`; `E2E_TELEMETRY_DEBUG=1` prints events instead of sending. | ## Mobile targets -`@e2e-dev/mobile` drives iOS simulators and Android emulators through -[agent-device](https://github.com/callstack/agent-device); needs Xcode with a -simulator runtime or the Android SDK with an emulator; run -`npx agent-device doctor` once. +`@e2e-dev/mobile` drives iOS simulators, Android emulators, and connected +phones through [agent-device](https://github.com/callstack/agent-device); +needs Xcode with a simulator runtime or the Android SDK with an emulator (a +phone needs Xcode with Developer Mode and runner signing, or adb with USB +debugging authorized); run `npx agent-device doctor` once. Name a phone in `device` by the name +`npx agent-device devices` lists, not its UDID or serial. On an iPhone, device +settings (permissions, `clearState`, network, location, appearance, +biometrics, keychain) and the clipboard are simulator-only. ```ts import type { E2EConfig } from 'e2e'; @@ -292,7 +316,7 @@ export default { `app.command` (`npx expo start --port 8081`) with `readyUrl: 'http://localhost:8081/status'` starts it. - One worker per device. No `device`: every booted simulator or emulator of - the platform is the pool, up to `workers` (none booted: agent-device boots + the platform, and every connected phone, is the pool, up to `workers` (none booted: agent-device boots one); one `device`: one worker whatever `workers` says; a list (`device: ['iPhone 17', 'iPhone 17 Pro']`): an explicit pool. Devices boot in `prepare`, before the run's clock. Two sessions on one device fight over diff --git a/skills/e2e/references/writing-tests.md b/skills/e2e/references/writing-tests.md index 9d6d11567..9a2b719c2 100644 --- a/skills/e2e/references/writing-tests.md +++ b/skills/e2e/references/writing-tests.md @@ -168,7 +168,9 @@ substring; a `RegExp` matches as written. scoping under another locator. Any other `filter` key (`hasNot`, `hasNotText`) or an empty `filter({})` is `INVALID_LOCATOR`. - `visible: true` drops nodes the page hides (a closed drawer) before the - exactly-one rule. + exactly-one rule. Visibility is what renders, as `toBeVisible()` reads it: + an `aria-hidden` spinner that paints is visible, so `toBeHidden()` waits + for it to go. - On the web, queries reach open shadow roots and closed roots attached with `attachShadow`, not declarative closed roots. `browser.locator(css)`, `frameLocator`, and `filter({ hasText })` stop at a closed root; query the @@ -215,7 +217,7 @@ text, both `[]` at zero. Text is the rendered text, whitespace collapsed: on the web what `innerText` reads (`text-transform` applies, `display: none` drops out, `
` is a space); `toHaveText` reads the same. `isChecked()` is `false`, not an error, on a node with no checked state, so query checkable -controls by role. `waitFor({ state?: 'visible' | 'hidden', timeout? })` waits +controls by role. `waitFor({ state?: 'attached' | 'detached' | 'visible' | 'hidden', timeout? })` waits within `actionTimeout`, else `LOCATOR_NOT_FOUND`. For a value that has to settle use `expect`, not a read. Reading a password field's value or attributes is `POLICY_DENIED`, as is `toHaveAttribute` on one, negated too. @@ -224,13 +226,20 @@ attributes is `POLICY_DENIED`, as is `toHaveAttribute` on one, negated too. `expect(locator)` polls up to `config.assertionTimeout` (5 s) or `{ timeout }`; `.not` inverts and passes once the negation has held 1 s -continuously, so it never returns in under a second. `expect(value, +continuously, or for the whole `timeout` when that is shorter, so it never +returns sooner. `expect(value, message?)` is synchronous. `expect.poll(read, { timeout?, interval?, message? })` re-reads until a value matcher passes (`assertionTimeout` and 100 ms by default, stopping with the -attempt); a throwing read keeps polling, and it is not a report step. +attempt); a throwing read keeps polling, and it is not a report step. A +poll that the body or a hook returns without awaiting is cancelled and fails +that phase with `STEP_NOT_AWAITED`. `expect.soft(x)` keeps a failure instead of throwing; the attempt fails -after the body with every soft failure listed. +after the body with every soft failure listed. If the body calls `test.skip`, +the test stays skipped and the failures remain in `secondaryErrors`. The CLI +shows `Skipped After Failure`, also when a retry skips after an earlier +failure. Set `failOnSkippedFailure: true` in the config to fail the run with +exit code 1 in either case. The default is `false`; clean skips stay green. `expect.any(Class)`, `expect.anything()`, `expect.objectContaining(obj)`, `expect.arrayContaining(arr)`, `expect.stringContaining(s)`, and `expect.stringMatching(s | RegExp)` stand in for values inside `toEqual`, @@ -265,8 +274,8 @@ list or tests a RegExp. A failed matcher is `ASSERTION_FAILED`, exit code 1. Playwright's `{ checked: false }`, `{ enabled: false }`, `{ visible: false }`, and `{ attached: false }` flip their matchers, and `{ ignoreCase: true }` -works on `toHaveText`, `toContainText`, `toHaveAccessibleName`, and -`toHaveAttribute(name, value)`. Any other option, `indeterminate` or +works on `toHaveText`, `toContainText`, `toHaveAccessibleName`, +`toHaveAttribute(name, value)`, and `expect(browser).toHaveURL`. Any other option, `indeterminate` or `useInnerText` included, is `INVALID_ARGUMENT`, in JavaScript too. ## Sign-in sessions @@ -323,7 +332,9 @@ credentials: { A function is read at fill time and redacted only from that fill on. - `E2E_USER__USERNAME` and `E2E_USER__PASSWORD` override either field per run, even over a function; `` is the credential name - uppercased, every character outside `[A-Z0-9]` as `_`. + uppercased, every character outside `[A-Z0-9]` as `_`. Two credentials (or + two secrets) mapping to one variable, `svc-a` and `svc_a`, are + `INVALID_CONFIG`. - `credentials.user('admin').password` is a `Secret` with no plaintext accessor, named `admin.password` to the agent and in reports; `secrets.get()` never returns it (separate namespaces). Only `fill()` and `agent.act` params accept it, stringifying it @@ -334,6 +345,11 @@ credentials: { `E2E_SECRET_STRIPE_KEY`; `secrets.get('stripe-key')` is the same kind of handle, fills any editable input, and is redacted by name everywhere the runner writes. +- A registered value passed as a plain string (`process.env.STRIPE_KEY` in a + title, a URL, a locator, an `agent.act` instruction or param) is redacted + too: reports, step labels, the executor, and the model see + `` (and so does the test id, for a value in a title), + so the agent cannot type it. Pass the handle. ## The browser fixture (browser only) @@ -355,14 +371,20 @@ the app opens itself (`target="_blank"`, `window.open`) is not followed: - `evaluate(fn | source, arg?)`: runs a function or source string in the page, JSON in and out, no closures; a throw in the page is `EVALUATE_FAILED`. -- `route(pattern, handler)`, `unroute(pattern)`: intercept requests; - `route.request` has `url`, `method`, `headers`, `postData`. The handler - calls exactly one of `fulfill({ status?, headers?, json | body })`, - `continue()`, or `abort()`; none or two fails the next step with - `ACTION_FAILED`. -- `waitForResponse(pattern, { timeout? })`: resolves with - `{ url, status, headers, json(), text() }`; `text()` and `json()` reject - with `ACTION_FAILED` when the body could not be read. +- `addInitScript(source | { path } | fn)`, `addInitScript(fn, arg)`: runs + before the page's own scripts; call it before `app.open`. `arg` is JSON. +- `route(pattern, handler)`, `unroute(pattern)`: intercept requests, newest + route first; `route.request` has `url`, `method`, `headers`, `postData`. + The handler calls exactly one of + `fulfill({ status?, headers?, contentType?, json | body | path })`, + `continue({ url?, method?, headers?, postData? })` (straight to the + network), `fallback()` (the route registered before it), or `abort()`; + none or two fails the next step with `ACTION_FAILED`, an unsupported + option with `INVALID_ARGUMENT`. +- `waitForResponse(pattern, { timeout? })`: resolves once the headers + arrive, with `{ url, status, headers, json(), text() }`; `text()` and + `json()` wait for the body (up to the action timeout) and reject with + `ACTION_FAILED` when it could not be read. - `cookies()`, `setCookies([...])`: a target is an http(s) URL or a domain. - `setViewport({ width, height })`: resize. - `onDialog('accept' | 'dismiss' | handler)`: awaited; resolves to an async @@ -391,5 +413,5 @@ A `route`, `unroute`, or `waitForResponse` pattern is a glob string or plus value matchers, in a `test.extend` fixture that reads `browser.cookies()` when the API needs the session. - No sleeps or polling loops; a matcher with a longer `timeout` instead. -- `await` every step call, else `STEP_NOT_AWAITED` at the line of the call. +- `await` every step call and `expect.poll`, else `STEP_NOT_AWAITED` at the line of the call. - Assert the fact a model produced with `toContain`, not its exact sentence.