Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
5b6a731
test(e2e): integration suites on the fake engine, cut to what only in…
okwasniewski Oct 5, 2026
e488bde
chore(deps): update fast-uri to address security advisories (#846)
nguyenvanhieu1710 Oct 5, 2026
c681005
test(web): wait for a painted frame before closing a video segment (#…
fecolinhares Oct 5, 2026
9995932
fix(run): charge stray rejections in in-process runs like a worker pr…
okwasniewski Oct 5, 2026
52228f4
docs(readme): GitHub Trending repository of the day badge (#852)
szymonrybczak Oct 5, 2026
2e26ec6
fix(cli): forward flags in npm exec hints (#847)
agammann Oct 5, 2026
dc2afca
feat(mcp)!: e2e mcp is headless by default, and open_session takes he…
KrzysztofMoch Oct 5, 2026
8bc7d85
fix(cache): replay a recorded step that types an empty value (#789) (…
Ray0907 Oct 5, 2026
260ef95
docs(examples): add standalone Vite project (#854)
kacperkapusciak Oct 5, 2026
1cf4c7c
docs(examples): add standalone Next.js project (#855)
kacperkapusciak Oct 5, 2026
0edbd5f
docs(examples): add standalone Expo project (#856)
kacperkapusciak Oct 5, 2026
0277c92
docs(examples): add standalone SwiftUI project (#857)
kacperkapusciak Oct 5, 2026
7149587
docs: link standalone examples from technology cards (#853)
kacperkapusciak Oct 5, 2026
432e9bf
fix(web): re-resolve a locator that turns ambiguous before its action…
okwasniewski Oct 5, 2026
60bf04d
fix(web): evaluate a browser.evaluate string as an expression (#860)
okwasniewski Oct 5, 2026
f2f9a53
fix(web): provision the chromium build a run launches, and collect no…
fecolinhares Oct 5, 2026
5603af1
fix(run): redact run-level errors with the run's secrets (#836)
DeryFerd Oct 5, 2026
8b552c2
fix(e2e): keep runner errors from a config-loaded engine classified (…
okwasniewski Oct 5, 2026
357f844
docs: drop the redundant Example projects nav group (#861)
kacperkapusciak Oct 5, 2026
970e846
docs: add the provider dropdown to the quickstart's API key tab (#863)
kacperkapusciak Oct 5, 2026
366766d
fix(agent): a forced complete_step turn answered with another tool as…
okwasniewski Oct 5, 2026
c9445ce
test(cache): replay backoff on a fake clock, key golden, session refu…
okwasniewski Oct 5, 2026
fe2c345
test(e2e): fake time and trimmed duplicates in runner, config, and CL…
okwasniewski Oct 5, 2026
a13e119
test(web): shared browser in lifecycle, trimmed tables, route and ifr…
okwasniewski Oct 5, 2026
79497a7
fix(mcp): redact secrets from open_session, close_session, and server…
okwasniewski Oct 5, 2026
ac1c610
fix(mcp): redact secrets from what user code prints in the e2e mcp pr…
okwasniewski Oct 5, 2026
6f528be
fix(agent): fail tainted fallback-only steps with POLICY_DENIED (#827)
okwasniewski Oct 5, 2026
df8d421
docs(migrate): add copyable agent prompts to the migration guides (#865)
kacperkapusciak Oct 5, 2026
338378e
fix(run): a worker crash during a retry keeps the attempts before it …
okwasniewski Oct 5, 2026
c31cf58
docs: shimmer the navbar star counter every 45 seconds (#867)
kacperkapusciak Oct 5, 2026
c83b86f
fix(web): fail operations on a hung page at their timeout, not the te…
okwasniewski Oct 5, 2026
01b88d3
fix(run): redact secrets from what the config, collected test files, …
okwasniewski Oct 5, 2026
b33398f
feat(locator): duration option for path screen.swipe (#831) (#840)
Mohit20057 Oct 5, 2026
37806d7
fix(run): kill a worker that blocks its event loop past the test time…
okwasniewski Oct 5, 2026
a3f0f9f
docs: collapse long code blocks outside the reference behind Show mor…
kacperkapusciak Oct 5, 2026
286c391
chore(deps): bump @ai-sdk/provider from 4.0.17 to 4.0.21 (#874)
dependabot[bot] Oct 5, 2026
71b821b
chore(deps): bump @react-navigation/native from 7.4.1 to 7.5.0 (#877)
dependabot[bot] Oct 5, 2026
cedf11d
chore(deps): bump next from 16.3.6 to 16.3.8 (#878)
dependabot[bot] Oct 5, 2026
a0fab56
chore(deps): bump @vitest/expect from 5.0.2 to 5.0.3 (#876)
dependabot[bot] Oct 5, 2026
0de2a6c
chore(deps): bump @modelcontextprotocol/server from 2.2.0 to 2.3.0 (#…
dependabot[bot] Oct 5, 2026
74b5bc4
chore(deps): bump @react-navigation/native-stack from 7.19.2 to 7.20.…
dependabot[bot] Oct 5, 2026
63482bd
docs: reorganize the sidebar into platforms and a deep dive (#870)
kacperkapusciak Oct 5, 2026
b4da513
chore(deps): bump @react-navigation/bottom-tabs from 7.19.2 to 7.20.0…
dependabot[bot] Oct 5, 2026
371706e
docs(agents): list the mobile, decision, and github packages in Layou…
DeryFerd Oct 5, 2026
fd3a0c7
ci(mobile): run the mobile benchmark when the github reporter changes…
DeryFerd Oct 5, 2026
5e032d9
fix(telemetry): count coding agents as machines, no project id outsid…
okwasniewski Oct 6, 2026
24c101e
Merge tester-army/e2e main into LambdaTest/e2e main
amankansal-lt Oct 6, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 5 additions & 0 deletions .changeset/browser-install-verdict.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@e2e-dev/web": patch
---

A headless run no longer downloads the full Chromium when the headless shell is installed, and the install no longer removes other browsers in the cache. Set `PLAYWRIGHT_SKIP_BROWSER_GC=0` to restore cleanup.
5 changes: 5 additions & 0 deletions .changeset/cache-empty-typed-value.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

A recorded step that types an empty value, which clears a field, replays instead of failing with `REPLAY_STALE` (`invalid-entry`). The cache wrote the entry but its reader rejected the empty `type` or `typeText` value, so such a step could never replay from a recording.
5 changes: 5 additions & 0 deletions .changeset/engine-prepare-headed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

`EnginePrepareInfo.headed` is now required, alongside the `headed` an engine already receives on `init`. A host that builds a `prepare` call now passes the run's mode, so an engine provisions for the browser that run launches.
5 changes: 5 additions & 0 deletions .changeset/error-default-redactor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

Run-level errors in reports (hook, collection, provisioning, and worker failures) are now redacted with the config's secrets.
5 changes: 5 additions & 0 deletions .changeset/evaluate-string-expression.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@e2e-dev/web': patch
---

`browser.evaluate` evaluates a string as an expression: `browser.evaluate('document.title')` and `browser.evaluate('(() => document.title)()')` return the title instead of failing with `EVALUATE_FAILED: ... is not a function`. A string that evaluates to a function, such as `'() => document.title'`, is still called with the argument.
5 changes: 5 additions & 0 deletions .changeset/forced-conclusion-wrong-tool.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'e2e': patch
---

An agent step whose model answers a forced `complete_step` turn with another tool no longer fails as `MODEL_PROVIDER_FAILED`. The call goes back to the model as an error and the turn is asked again within the step's turn budget; a model that never concludes fails the step with `STEP_NO_CONCLUSION`.
5 changes: 5 additions & 0 deletions .changeset/foreign-engine-errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

An error an engine throws with a runner code, such as `INVALID_LOCATOR` for an unsupported selector, keeps its code and category when the engine is loaded from `e2e.config.ts`. It was reported as an infrastructure `ENGINE_FAILURE`.
5 changes: 5 additions & 0 deletions .changeset/in-process-stray-rejections.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

A run that executes in the host process, such as `e2e explore`, no longer crashes that process when a test leaves a promise rejection or an exception uncaught. It handles them as a worker process does: a rejection fails the test running when it surfaces, or is recorded as `UNHANDLED_REJECTION` between tests. An uncaught exception, or a rejection that surfaces with no test running before any test has finished, is a run error that fails the test on its way with `WORKER_CRASH` and skips the rest of its file. The handlers are removed when the run ends.
5 changes: 5 additions & 0 deletions .changeset/mcp-error-redaction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'e2e': patch
---

`e2e mcp` now redacts secret values from everything `open_session` and `close_session` return and from its log lines. Before, an engine or app error that carried a configured secret, in a failed open or in a `close_session` "Cleanup:" line, reached the client and stderr verbatim.
5 changes: 5 additions & 0 deletions .changeset/mcp-headless-default.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": minor
---

`e2e mcp` sessions are now headless by default, the same as `e2e run` and `e2e explore`, and the agent decides per session: `open_session {headed: true}` shows the browser or simulator, `headed: false` hides it. `--headed` makes headed the default for sessions that do not say. Before, sessions were headed outside CI and only the server's `--headless` could hide them; `--headless` is still accepted, since it now asks for the default, but it is no longer documented.
5 changes: 5 additions & 0 deletions .changeset/mcp-user-output-redaction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'e2e': patch
---

`e2e mcp` now redacts secret values from what your code prints through `console`, `process.stdout`, or `process.stderr`, such as a `console.log` in a project tool or in the config's top-level code. Before, it reached stderr verbatim. Output printed while a config loads appears once the config has loaded; if the load fails, that output is withheld, since the config's secrets are unknown. An uncaught error or unhandled rejection is logged redacted and the server shuts down its sessions and exits with code 1, instead of crashing with the raw error.
5 changes: 5 additions & 0 deletions .changeset/npm-exec-cli-hint.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

Include npm's argument separator in CLI hints so flags such as `--headed` reach the installed `e2e` binary.
5 changes: 5 additions & 0 deletions .changeset/retry-crash-attempts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

A worker that crashes during a retry no longer drops the attempts before it. Before, a test whose first attempt failed and whose retry crashed the worker showed a single attempt 0 with `WORKER_CRASH`, and the first attempt's error was lost. The report now keeps every finished attempt and adds the crash as the next one. A crash after the last attempt (in an `afterAll`) is recorded on that attempt instead of as a retry. For a serial group, the finished group attempts and the members that finished in the crashed attempt stay in the report, the member that was running fails with `WORKER_CRASH`, and a crash after a member finished fails the group attempt without blaming that member. A forced interrupt keeps the attempts that finished and the verdict they reach, `flaky` included.
5 changes: 5 additions & 0 deletions .changeset/run-user-output-redaction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'e2e': patch
---

`e2e run` redacts secret values from what the config's top-level code, a test file's top level, and reporters print, as it already did for tests, including what a timer or a reporter the run stopped waiting for prints after the run. Output printed while the config loads appears once it has loaded, and is withheld if the load fails. With the `list` reporter, output printed during the run prints under `stdout | runner` or `stderr | runner`, on stdout like a test's output, so the live view no longer paints over it.
5 changes: 5 additions & 0 deletions .changeset/strict-mode-reresolve.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@e2e-dev/web': patch
---

An action whose locator gains a second match just before it runs, such as a screen transition that briefly shows the old and new page together, no longer fails with `ENGINE_FAILURE` and a Playwright strict mode violation. The locator resolves again: two matches still fail with `LOCATOR_AMBIGUOUS`, and a duplicate that went away lets the action run. A drag whose drop target gains a second match after the press fails with `ACTION_FAILED` and is not repeated.
7 changes: 7 additions & 0 deletions .changeset/swipe-path-duration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"e2e": minor
"@e2e-dev/mobile": minor
"@e2e-dev/web": minor
---

`screen.swipe({ from, to, duration })` adds an optional `duration` for path swipes, controlling how long the gesture takes in milliseconds.
5 changes: 5 additions & 0 deletions .changeset/tainted-fallback-policy-denied.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

After a secret fill, an agent step whose screen can only be read from a fallback screenshot (`act`, or a judgment with `vision: true`) now fails with `POLICY_DENIED` naming the secret fill, instead of the engine's timeout.
5 changes: 5 additions & 0 deletions .changeset/telemetry-identity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

Telemetry no longer sends a project id outside git or in a shallow clone, and a coding agent that sets `CI` outside a CI vendor counts as the machine it runs on. EAS Build, Bitrise, Codemagic, and Xcode Cloud are recognized as CI vendors.
6 changes: 6 additions & 0 deletions .changeset/web-hung-page-timeouts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'@e2e-dev/web': patch
'e2e': patch
---

A page whose script never yields no longer holds a web test until its test timeout: a tap, a locate, or a navigation on it now fails with `ACTION_FAILED` at its action or navigation timeout, also while a trace records. A tap or keystroke cut off that way may have reached the page and is never repeated by a replay. `expect(browser)` matchers on such a page end at their own timeout instead of the action timeout.
5 changes: 5 additions & 0 deletions .changeset/worker-hang-watchdog.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'e2e': patch
---

A test that blocks its worker's event loop (`while (true) {}`) no longer hangs the run until Ctrl-C. When the worker has not responded `cleanupTimeout` (at least 5 s) after the test timeout, the runner kills it, reports the test `timed-out` with `TEST_TIMEOUT`, and runs the rest of the file on a new worker. A worker with a debugger attached when the test starts (an editor's auto-attach, or `NODE_OPTIONS=--inspect=0`) is never killed this way. Runs with an inline config or in-memory tests run in the runner's own process and still hang.
1 change: 1 addition & 0 deletions .fallowrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
],
"ignorePatterns": [
"docs/**",
"examples/**",
"**/next-env.d.ts",
"**/.e2e/**",
"packages/e2e/tests/tmp-projects/**"
Expand Down
1 change: 1 addition & 0 deletions .github/filters.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ web:
mobile:
- *runner
- 'packages/mobile/**'
- 'packages/github/**'
- 'apps/mobile-benchmark/**'
- '.github/workflows/mobile.yml'
- '.github/actions/expo-app/**'
4 changes: 3 additions & 1 deletion .oxlintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@
"packages/*/tests/fixtures/**",
// Vendored conformance data, kept byte for byte.
"packages/*/tests/conformance/*/**",
"packages/*/tests/tmp-projects/**"
"packages/*/tests/tmp-projects/**",
// Standalone user-facing projects with their own installs; nothing here builds them.
"examples/**"
],
"rules": {
// Sequential execution is the contract of a test runner, not a perf bug.
Expand Down
32 changes: 30 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,14 @@ suites that consume the built packages the way a user would.
`e2e/engine` only: the semantics every engine must reproduce
(error taxonomy, text and URL matching, assertion polling, JSON-value rules)
are exported there, and there is no `e2e/internal` subpath.
- `packages/mobile` — the published `@e2e-dev/mobile` package: the
iOS/Android engine on agent-device, built with the same public
`defineEngine`, contributing the `device` fixture. Like the web engine it
depends on `e2e` (peer), never the reverse; the engine implementation
imports from `e2e/engine` only, and a target names it as `engine: mobile()`.
The `@e2e-dev/mobile/tools` subpath holds the agent-side `open_app`, `swipe`,
and `alert` tools; focused-field typing uses the engine's keyboard grammar.
The main entry never loads the AI SDK.
- `packages/kernel` - the published `@e2e-dev/kernel` package: Kernel hosted
browsers for the web engine. An official integration with a hosted service
is one package per service, named after it (`@e2e-dev/<service>`), with the
Expand All @@ -77,7 +85,19 @@ suites that consume the built packages the way a user would.
hosted iOS simulators and Android emulators for the mobile engine
(`DeviceProvider`). Expo publishes no SDK for the sessions API, so it calls
Expo's GraphQL API with `fetch`, and `@e2e-dev/mobile` is its only peer.
- `packages/testmu` - the published `@e2e-dev/testmu` package: TestMu AI
- `packages/decision` — the published `@e2e-dev/decision` package: a
`StepExecutor` (`decisionExecutor()`) that drives `agent.act` and
`agent.assert` through an AI SDK *evaluation* model answering `choice`
questions with probability distributions, plus an optional small language
model that writes field values when the decision model picks `type`.
`minProbability` and `minConfidence` gate a chosen operation, target,
secret, or assertion verdict.
- `packages/github` — the published `@e2e-dev/github` package: the reporter
that posts the run as one pull request comment from GitHub Actions and
keeps it current on reruns, writing the same text to the job summary. It
reads the token, event, and repository when the run finishes, never at
config load, and renders the page with `renderMarkdownReport` from `e2e`.
- `packages/testmu` — the published `@e2e-dev/testmu` package: TestMu AI
(formerly LambdaTest) hosted Android emulators, iOS simulators, and real
devices for the mobile engine (`DeviceProvider`). agent-device's `testmu`
cloud provider is the vendor client: the package allocates agent-device
Expand Down Expand Up @@ -120,6 +140,14 @@ suites that consume the built packages the way a user would.
"Committed recordings" under Gotchas. Scenario files are copies: keep
diffs against the source minimal, and name no company a scenario was
distilled from.
- `examples/` — standalone user-facing projects, one per technology
(`with-vite`, `with-next`, `with-expo`, `with-swiftui`), each the same
one-screen greeter demo with deterministic and agent tests. They install
the published packages from npm, sit outside the pnpm workspace, commit no
lockfile, and run in no CI; oxlint and fallow ignore them. A change runs
the example's suite by hand and updates the "Last checked" line in its
README. A SwiftUI example keeps its tests in an `e2e/` folder beside the
native project, as a user would.
- `docs/` (the Mintlify docs site; pages are the `.mdx` files under `docs/`,
navigation, theme, and redirects in `docs/docs.json`, extra CSS in
`docs/style.css`; `docs/examples/` is typechecked and shown verbatim on
Expand Down Expand Up @@ -218,7 +246,7 @@ thread handled, and labeled `Ready for Human Review`. "It compiles" and

## Testing quirks

- Vitest 4, `pool: 'forks'`, two projects. `integration` is capped at
- Vitest 5, `pool: 'forks'`, two projects. `integration` is capped at
`maxWorkers: 3` and runs in a later group — do not raise it; CPU starvation
produces timeouts indistinguishable from real failures.
- Integration tests write throwaway projects into
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@
<a href="https://tester.army/discord"><img alt="Join the community on Discord" src="https://img.shields.io/badge/Join%20the%20community-5865F2.svg?style=for-the-badge&logo=discord&logoColor=white&labelColor=000000" /></a>
</p>

<p align="center">
<a href="https://www.star-history.com/tester-army/e2e"><picture><source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/badge?repo=tester-army/e2e&type=trending&theme=dark" /><img alt="GitHub Trending Repository of the Day" src="https://api.star-history.com/badge?repo=tester-army/e2e&type=trending" /></picture></a>
</p>

# e2e

[e2e](https://tester.army/e2e?utm_source=e2e&utm_medium=github&utm_campaign=readme_intro) is an end-to-end testing framework for web and mobile apps. Describe a goal in natural language and an agent drives the app to reach it. Check the result with locators and assertions in the same test.
Expand Down Expand Up @@ -40,6 +44,11 @@ npx e2e init
config and an example test. The
[quickstart](https://e2e.tester.army/docs/quickstart) covers the rest.

To see a finished setup in your stack, open
[`examples/`](https://github.com/tester-army/e2e/tree/main/examples):
Vite, Next.js, Expo, and SwiftUI, each a standalone project with a passing
suite.

## Packages

| Package | What it does |
Expand Down
6 changes: 3 additions & 3 deletions apps/mobile-benchmark/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@
"capture:snapshots": "node node_modules/e2e/dist/cli/bin.js run --config e2e.capture.config.ts"
},
"dependencies": {
"@react-navigation/bottom-tabs": "7.19.2",
"@react-navigation/native": "7.4.1",
"@react-navigation/native-stack": "7.19.2",
"@react-navigation/bottom-tabs": "7.20.0",
"@react-navigation/native": "7.5.0",
"@react-navigation/native-stack": "7.20.0",
"@stripe/stripe-react-native": "0.64.0",
"expo": "57.0.22",
"expo-audio": "57.0.5",
Expand Down
2 changes: 1 addition & 1 deletion apps/web-benchmark/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"dependencies": {
"e2e": "workspace:*",
"@e2e-dev/web": "workspace:*",
"next": "16.3.6",
"next": "16.3.8",
"react": "19.3.0",
"react-dom": "19.3.0"
},
Expand Down
1 change: 1 addition & 0 deletions docs/agent-steps.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
title: How agent steps work
description: What the model sees, what it may do, how a step ends, and what it costs.
icon: "list-check"
---

An agent step is one `agent.act`, `agent.assert`, `agent.waitFor`, or
Expand Down
1 change: 1 addition & 0 deletions docs/agents.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
title: Agents and personas
description: Configure several agents in one suite, pin tests to them, and run one flow as each persona.
icon: "users"
---

Define named agents in `agents` to give tests different models, instructions,
Expand Down
3 changes: 2 additions & 1 deletion docs/api-testing.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
title: Testing APIs
title: API
description: Check your API with fetch and value matchers, in the same files and run as UI tests.
icon: "server"
---

API tests run in the same files and the same run as UI tests. A test that
Expand Down
94 changes: 0 additions & 94 deletions docs/assertions.mdx

This file was deleted.

Loading