Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
62bfe0a
fix(github): long marker fields keep a digest so distinct keys never …
okwasniewski Oct 2, 2026
dec903b
fix(e2e): redact plain-string secret values in titles, labels, and ag…
okwasniewski Oct 2, 2026
7fd610b
fix(web): list an inline svg as an image, named by its title, as Play…
okwasniewski Oct 2, 2026
a0ee3e9
chore: version packages (#761)
github-actions[bot] Oct 2, 2026
9d61e21
docs(mobile): document physical devices (#768)
okwasniewski Oct 2, 2026
f7c0756
fix(expect): waitFor attached/detached, absent-frame absence, toHaveP…
okwasniewski Oct 2, 2026
7ab80bc
fix(report)!: count interrupted tests apart from failures (#713)
okwasniewski Oct 3, 2026
79931d0
fix(cache): treat short prefixed record ids as minted segments (#777)
DeryFerd Oct 3, 2026
37497dc
docs: split Writing tests into Goals, Assertions, and Locators pages …
kacperkapusciak Oct 3, 2026
eb7830d
docs(models): add an AI SDK provider picker to the Models page (#781)
kacperkapusciak Oct 3, 2026
bfa58c7
fix(runner): preserve failures when tests skip (#741)
Marve10s Oct 3, 2026
a3da00d
fix(mobile): never pin an install's file path as the app app.open() l…
okwasniewski Oct 3, 2026
94ddbfe
fix(config): refuse secrets or credentials sharing an override env va…
okwasniewski Oct 3, 2026
f0f9c8d
fix(security): allow only http(s) and about:blank on explicit navigat…
okwasniewski Oct 3, 2026
7f9ee10
fix(web): read visibility as rendered, null box, checkable values, em…
okwasniewski Oct 3, 2026
6639bfa
fix(oauth): reach Copilot models served only over the Responses API (…
fecolinhares Oct 3, 2026
cdc3a53
fix(web): continue skips earlier routes, decisions take or reject Pla…
okwasniewski Oct 3, 2026
cc54c6b
feat(agent): identify e2e on every model request (#784)
okwasniewski Oct 3, 2026
9844c30
fix(web): quotes in a string role name no longer break chained locato…
okwasniewski Oct 3, 2026
9b93575
fix(expect): unawaited poll fails its own phase, slow first read coun…
okwasniewski Oct 3, 2026
4ce01b2
fix(e2e): keep secrets out of observed node fields and collapsed cuts…
okwasniewski Oct 3, 2026
3a9667c
fix(run): keep --last-failed reruns honest: failure-limit skips, carr…
okwasniewski Oct 3, 2026
3bf295e
fix(cache): stop replays self-finalizing when the effect did not happ…
okwasniewski Oct 3, 2026
b3e1fd1
fix(mobile): update agent-device for iOS toolbar occlusion (#791)
DimaMishchenko Oct 3, 2026
f88233f
feat(web): locale and timezoneId options set the language and time zo…
okwasniewski Oct 3, 2026
4db7d6d
fix(mobile): stop openLink echoing a malformed link's input (#793)
DeryFerd Oct 3, 2026
baa32c6
fix(web): a check whose control the click replaces passes and replays…
okwasniewski Oct 3, 2026
c66176b
feat(oauth): sign in with OpenCode Console for Zen and Go models (#794)
neriousy Oct 3, 2026
6b1fe5f
test(web-benchmark): re-record agent recordings for the conservative/…
okwasniewski Oct 3, 2026
02ad48a
feat(web)!: pin playwright-core and add a browser install command (#798)
okwasniewski Oct 3, 2026
82c2bf5
refactor(mobile): read agent-device's published snapshot, screenshot,…
thymikee Oct 3, 2026
3b026fa
feat(web): init scripts run before the page's own, from web({ initScr…
okwasniewski Oct 3, 2026
96ff11b
fix(cache): record a vanished node only when no end node matches its …
okwasniewski Oct 3, 2026
ae3dc64
fix(mobile): recognize stale-ref refusals by agent-device's typed rea…
thymikee Oct 3, 2026
b5dc0f8
fix(cache): --strict-cache fails a step whose key changed under its r…
okwasniewski Oct 3, 2026
09733be
chore: version packages (#774)
github-actions[bot] Oct 4, 2026
0d1ed15
fix(e2e)!: load TypeScript with e2e's own oxc loader, so pnpm 11+ ins…
okwasniewski Oct 4, 2026
d1f2089
feat(decision): act and assert through an AI SDK decision model (#775)
guhcostan Oct 4, 2026
8657c8b
test(mobile): fake clock in engine tests, drop the simulator suite CI…
okwasniewski Oct 4, 2026
adcfc2b
test(agent): fake time and trimmed duplicates in agent unit tests (#807)
okwasniewski Oct 4, 2026
1731e0d
test(e2e): list reporter goldens, fake time in expect and locator tes…
okwasniewski Oct 4, 2026
a561f06
perf(agent): stop re-waiting for a lost list on every repeat of a rep…
okwasniewski Oct 4, 2026
aeaca4f
fix(run): bound failure evidence to one 5 s capture per attempt (#819)
okwasniewski Oct 4, 2026
78e135a
fix(secrets): redact plain-string secret values from traces and text …
okwasniewski Oct 4, 2026
87e75ae
fix(run): serial group failures reach the exit code and the reporters…
okwasniewski Oct 4, 2026
f7f60ba
fix(e2e): rewrite downloads with the full ledger pass, the way traces…
DeryFerd Oct 4, 2026
f99e80e
test(web-benchmark): re-record agent recordings for @e2e-dev/web 0.12…
okwasniewski Oct 4, 2026
4bfc1aa
Update AGENTS.md (#839)
NirmalyaLenka Oct 4, 2026
7baf454
feat(docs): shimmer the GitHub stars counter every two minutes (#849)
kacperkapusciak Oct 5, 2026
8773058
Merge tester-army/e2e main into LambdaTest/e2e main
amankansal-lt Oct 5, 2026
4a6def6
chore(testmu): follow upstream's agent-device 0.21.20 pin and Node floor
amankansal-lt Oct 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
7 changes: 0 additions & 7 deletions .changeset/atomic-pinned-locate.md

This file was deleted.

5 changes: 0 additions & 5 deletions .changeset/codex-client-version.md

This file was deleted.

5 changes: 0 additions & 5 deletions .changeset/copilot-unusable-models.md

This file was deleted.

11 changes: 11 additions & 0 deletions .changeset/decision-executor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
"@e2e-dev/decision": minor
---

Add `@e2e-dev/decision`, an executor that runs `agent.act` and `agent.assert`
through a decision model instead of an LLM. It takes any AI SDK evaluation
model with choice distributions and decides through `experimental_evaluate`,
one call per action with the operation and target questions fanned out; a
small language model writes field values when the choice is `type`. Tests
stay plain natural language with no params. Probability and confidence
gates are off by default.
5 changes: 5 additions & 0 deletions .changeset/download-full-ledger-pass.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

A text download is rewritten with the full ledger pass — fragments and base64-encoded forms of a secret included — the way trace text entries already are. The download path only matched whole values while still labeling the file `redaction: "complete"`, so a cut or base64 copy of a secret could sit in a download the report vouched for as clean.
5 changes: 5 additions & 0 deletions .changeset/failure-evidence-budget.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'e2e': patch
---

A failed attempt on an app that stops answering no longer waits 5 s for failure evidence at every capture point, or forever when the engine ignores cancellation. The runner captures the screen and screenshot once per attempt, stops waiting after 5 s, and keeps the original error.
7 changes: 0 additions & 7 deletions .changeset/matcher-options-honored-or-rejected.md

This file was deleted.

5 changes: 0 additions & 5 deletions .changeset/mcp-without-ai.md

This file was deleted.

12 changes: 12 additions & 0 deletions .changeset/oxc-typescript-loader.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
'e2e': minor
'@e2e-dev/web': minor
'@e2e-dev/mobile': minor
'@e2e-dev/kernel': minor
'@e2e-dev/eas': minor
'@e2e-dev/github': minor
---

**Breaking: e2e and every `@e2e-dev` package now need Node.js 22.22.3 or newer on Node.js 22, or 24.8.0 or newer** (`engines.node` is `^22.22.3 || >=24.8.0`). Node.js 22.12 through 22.22.2, 23, and 24.0 through 24.7 are no longer supported, and the CLI refuses to start on them, naming the versions to upgrade to. Those releases fail when a CommonJS file a test imports requires TypeScript through the new loader's `module.registerHooks` (nodejs/node#59679 fixed it in 22.22.3 and 24.8.0).

e2e now loads TypeScript with its own loader, built on oxc, instead of tsx. tsx brought in esbuild, whose postinstall script made every `pnpm install` on pnpm 11 and later fail with `ERR_PNPM_IGNORED_BUILDS` until the project approved the build; nothing e2e installs runs an install script now. Config and tests load as before: ESM whatever `package.json` says, `./x.js` and `./x` resolve to `x.ts`, directory index imports, tsconfig `paths` and `baseUrl`, workspace packages exporting `.ts` source, JSX in `.tsx` and `.jsx`, enums, namespaces, `experimentalDecorators`, and source-mapped stack traces. JSON imported without `with { type: 'json' }` still loads. A test file whose name holds `%`, `#`, or a space now collects with its source location instead of failing with "URI malformed" or losing the location. Each file reads its nearest `tsconfig.json` rather than the one in the working directory. What changes: `import` statements in a `.cts` file are no longer compiled into `require` calls (write `import x = require()`); standard decorators without `experimentalDecorators`, auto-accessors, and ES module `import` and value `export` declarations in `.cts` are refused with their line (`import type`, `export type`, `import x = require()`, and `export =` work); the CLI refuses the Bun (`bun --bun`) and Deno runtimes (install with any package manager, run on Node.js); a type imported without `import type` that stays in the output (a decorated member's type under `emitDecoratorMetadata`) fails to load, with an error naming the import; a `.ts` file a CommonJS file requires runs as an ES module like every other `.ts`, so one written with `module.exports` needs the `.cts` extension; and an import written in a JavaScript file resolves as Node.js resolves it, without TypeScript's extension rules or tsconfig `paths`. TypeScript inside an installed package compiles without the project's `tsconfig.json`.
5 changes: 5 additions & 0 deletions .changeset/plain-string-secret-trace.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'e2e': patch
---

A secret value a test passes as a plain string (spelled into an `app.open` URL, typed with `fill`) no longer stays in the Playwright trace. Traces and text downloads are now rewritten whenever the session knows a secret value (every static one, and a provider's once resolved), not only after a secret fill or an engine-held secret, so they are labeled `redaction: "complete"` instead of `not-required` or `incomplete`. Screencast frames are still dropped only after a secret fill. Like any secret trace, a rewritten one masks unrelated text that shares 8 characters with a value.
5 changes: 5 additions & 0 deletions .changeset/replay-lost-list-once.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

Replaying a repeated scroll on a list that filled the screen, when the list can no longer be found, waits for it once and then scrolls the screen for every remaining repeat. It used to wait out the full relocation backoff (about 14 seconds) before each repeat, so a recorded 4x scroll took about a minute to replay.
5 changes: 5 additions & 0 deletions .changeset/serial-group-verdicts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

A serial group whose session fails to launch, or whose file fails to load, now fails the run. Before, every member was skipped and the run passed with exit code 0. The first member now fails with the launch error and the rest skip with cause `serial-predecessor-failed`, so the `list` reporter, `summary.md`, and `junit.xml` show the failure. A retry that fails to launch keeps the members' verdicts from the attempt before it, so a member that failed there still fails. An infrastructure error in a serial group, at launch or in a member, now exits 3 and sets the run status to `error`, as it does for any other test, instead of exiting 1. When a worker crashes during a serial group, the member that was running fails with `WORKER_CRASH` instead of being skipped as if it never started.
10 changes: 0 additions & 10 deletions .changeset/slim-install.md

This file was deleted.

5 changes: 0 additions & 5 deletions .changeset/telemetry-mcp-explore-config.md

This file was deleted.

5 changes: 0 additions & 5 deletions .changeset/web-matcher-options-rejected.md

This file was deleted.

2 changes: 1 addition & 1 deletion .claude/skills/verify/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ repeat.

```bash
pnpm install --frozen-lockfile
pnpm --filter @e2e-dev/web exec playwright install chromium
pnpm --filter @e2e-dev/web exec playwright-core install chromium
pnpm build # every consumer below runs dist; rebuild after each change
```

Expand Down
9 changes: 7 additions & 2 deletions .fallowrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,12 @@
"apps/mobile-benchmark/index.ts",
"packages/e2e/tests/types/sdk-types.ts",
"packages/web/tests/types/connection.ts",
"packages/web/tests/types/init-scripts.ts",
"packages/web/tests/types/routes.ts",
"packages/mobile/tests/types/options.ts",
"packages/e2e/tests/live/*.ts"
"packages/decision/tests/types/options.ts",
"packages/e2e/tests/live/*.ts",
"packages/e2e/src/config/register.ts"
],
"ignorePatterns": [
"docs/**",
Expand All @@ -30,7 +34,8 @@
"@e2e-dev/web",
"@openrouter/ai-sdk-provider",
"@expo/fingerprint",
"@expo/repack-app"
"@expo/repack-app",
"@oxc-project/runtime"
],
"rules": {
"unused-class-member": "warn",
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/agent.yml
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ jobs:
fi
- run: pnpm install --frozen-lockfile
- name: Install browsers
run: pnpm --filter @e2e-dev/web exec playwright install chromium --with-deps
run: pnpm --filter @e2e-dev/web exec playwright-core install chromium --with-deps
- name: Build
run: pnpm run build
- name: Run ${{ matrix.script }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ jobs:
cache: pnpm
- run: pnpm install --frozen-lockfile
- name: Install browsers
run: pnpm --filter @e2e-dev/web exec playwright install chromium --with-deps
run: pnpm --filter @e2e-dev/web exec playwright-core install chromium --with-deps
- name: Build
run: pnpm run build
- name: Run test
Expand Down Expand Up @@ -132,7 +132,7 @@ jobs:
fi
- run: pnpm install --frozen-lockfile
- name: Install browsers
run: pnpm --filter @e2e-dev/web exec playwright install chromium --with-deps
run: pnpm --filter @e2e-dev/web exec playwright-core install chromium --with-deps
- name: Build
run: pnpm run build
- name: Run test:agent
Expand Down
33 changes: 30 additions & 3 deletions .github/workflows/spec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,14 @@ jobs:
run: pnpm docs:check-errors
- name: Peer ranges on the runner
run: pnpm check:peer-ranges
- name: No install scripts in published dependency trees
run: pnpm check:install-scripts
- name: Agent skill scripts
run: |
pnpm exec tsc -p .claude/skills/babysit/scripts
node --test .claude/skills/babysit/scripts/pr-status.test.ts
- name: Install browsers
run: pnpm --filter @e2e-dev/web exec playwright install chromium --with-deps
run: node packages/web/dist/bin.js install chromium --with-deps
- name: Testbed
run: pnpm --filter @e2e-dev/testbed test

Expand All @@ -53,16 +55,41 @@ jobs:
strategy:
fail-fast: false
matrix:
node: [22, 24, 26]
# The newest release of each line, and the floors in engines.node.
node: [22, 24, 26, 22.22.3, 24.8.0]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: pnpm/action-setup@9fd676a19091d4595eefd76e4bd31c97133911f1 # v4.2.0
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: ${{ matrix.node }}
# A major alone resolves to the runner's cached release, which can
# sit below the patch floor in engines.node (22.22.3).
check-latest: true
cache: pnpm
- run: pnpm install --frozen-lockfile
- name: Install browsers
run: pnpm --filter @e2e-dev/web exec playwright install chromium --with-deps
run: pnpm --filter @e2e-dev/web exec playwright-core install chromium --with-deps
- name: Unit and integration tests
run: pnpm test

# A new user's first install of the packed packages, which pnpm 11 and
# later fail on any dependency build script (scripts/install-smoke.ts).
install:
runs-on: blacksmith-4vcpu-ubuntu-2404
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
pnpm: [11, 12]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: pnpm/action-setup@9fd676a19091d4595eefd76e4bd31c97133911f1 # v4.2.0
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 26
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm run build
- name: Fresh install with pnpm ${{ matrix.pnpm }}
run: node scripts/install-smoke.ts ${{ matrix.pnpm }}
Loading