diff --git a/.changeset/ios-webdriver-element-types.md b/.changeset/ios-webdriver-element-types.md new file mode 100644 index 000000000..18e659f7b --- /dev/null +++ b/.changeset/ios-webdriver-element-types.md @@ -0,0 +1,5 @@ +--- +'@e2e-dev/mobile': patch +--- + +iOS devices driven through agent-device's WebDriver runtime (hosted Appium clouds such as TestMu AI, BrowserStack, and AWS Device Farm) resolve roles like a local simulator does. That runtime reports XCUITest class names (`XCUIElementTypeButton`, `XCUIElementTypeStaticText`) where the native runner reports `Button` and `StaticText`, and the engine read them as unknown types, so `getByRole('button', 'CPU Load')` matched nothing and `observe` listed the node as `xcuielement-type-button`. The `XCUIElementType` prefix is now dropped before the type is mapped, so a button is a `button`, a tab bar's buttons are `tab`s, and the navigation bar titles the screen. diff --git a/.changeset/testmu-browsers.md b/.changeset/testmu-browsers.md new file mode 100644 index 000000000..9d02bc331 --- /dev/null +++ b/.changeset/testmu-browsers.md @@ -0,0 +1,5 @@ +--- +"@e2e-dev/testmu": minor +--- + +`@e2e-dev/testmu/web`: `web({ browser: testmuBrowsers() })` runs a web target in TestMu AI (formerly LambdaTest) hosted Chrome or Edge on Windows and macOS, one session per worker slot, or one per attempt with `scope: 'attempt'`. A session starts when its CDP websocket opens and ends when the engine closes it, so the provider only builds the CDP URL (`hub` `cdp.lambdatest.com`) from `browserName`, `browserVersion`, `platform`, `geoLocation`, `timezone`, and further `LT:Options` `capabilities`, with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment, which never appear in a log line. `route` is `/puppeteer` by default, which serves raw CDP on every machine; `/playwright-cdp` labels sessions as Playwright but serves raw CDP only on machines with TestMu AI's newest backend, which is still rolling out. Sessions are named as `testmu()`'s are: `build` defaults to the run id, so a run's device and browser sessions share one build, `sessionName` to `e2e---`, and `project` (default `e2e`) is sent as `LT:Options.project`. `idleTimeout` defaults to 600 seconds. Unknown options, a `scope` or `route` outside their values, a `hub` with a scheme or path, a non-Chromium `browserName`, an empty or non-string string option, `null` for any option, and `capabilities` that set what the provider sets (`user`, `accessKey`, `build`, `name`, `project`, `platform`, `geoLocation`, `timezone`, `browserName`, `browserVersion`, or a nested `LT:Options`) fail with `INVALID_CONFIG` at config load. The package root stays device-only, and `@e2e-dev/mobile`, `@e2e-dev/web`, and `agent-device` are optional peers, so a project installs only the side it uses. diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md new file mode 100644 index 000000000..73dbb3afd --- /dev/null +++ b/.changeset/testmu-devices.md @@ -0,0 +1,5 @@ +--- +"@e2e-dev/testmu": minor +--- + +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`), in a directory per run that it marks as its own and that later runs remove once nothing in it has changed for 24 hours; allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), allocates each lease with agent-device's longest inactivity window and heartbeats it every 2 minutes while the run holds it, so a session that takes up to 10 minutes to start keeps its lease, links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session, starting at the session's start time), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/AGENTS.md b/AGENTS.md index e11501f9b..19c961701 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -97,6 +97,19 @@ suites that consume the built packages the way a user would. keeps it current on reruns, writing the same text to the job summary. It reads the token, event, and repository when the run finishes, never at config load, and renders the page with `renderMarkdownReport` from `e2e`. +- `packages/testmu` — the published `@e2e-dev/testmu` package: TestMu AI + (formerly LambdaTest) hosted Android emulators, iOS simulators, and real + devices for the mobile engine (`DeviceProvider`). agent-device's `testmu` + cloud provider is the vendor client: the package allocates agent-device + leases from a daemon it starts per run, so `agent-device` (which must be + the same copy `@e2e-dev/mobile` uses) and `@e2e-dev/mobile` are its peers. + `@e2e-dev/testmu/web` exports `testmuBrowsers()`, TestMu AI hosted Chrome + and Edge for the web engine (`BrowserProvider`): a session starts when its + CDP websocket opens and ends when it closes, so it only builds the URL and + calls no API. The root stays device-only, so neither side's types or + imports reach the other. Every + peer (`@e2e-dev/mobile`, `@e2e-dev/web`, `agent-device`) is optional, so a + project installs only the side it uses. - `apps/testbed` (`@e2e-dev/testbed`, private) — dogfood project that consumes the **built** packages like a real user would: the playground app where every runner feature (sessions, routes, downloads, frames, uploads, diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 23374bcce..07abd007d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -255,5 +255,5 @@ build against, and what a third-party engine builds against too. A change to that contract bumps all three packages together in one release, with a changeset for each, so an engine and a runner from the same release always match. Engines declare a peer range on `e2e` that points one way only -(engine to runner, `>=x <1`), and each integration (`@e2e-dev/kernel`, `@e2e-dev/eas`) does +(engine to runner, `>=x <1`), and each integration (`@e2e-dev/kernel`, `@e2e-dev/eas`, `@e2e-dev/testmu`) does the same on the engines it plugs into; do not make it mutual or narrow it. diff --git a/README.md b/README.md index 2a1a902d7..5ccbb722b 100644 --- a/README.md +++ b/README.md @@ -60,6 +60,7 @@ suite. | [`@e2e-dev/kernel`](https://www.npmjs.com/package/@e2e-dev/kernel) | Kernel hosted browsers for the web engine. | | [`@e2e-dev/eas`](https://www.npmjs.com/package/@e2e-dev/eas) | EAS Simulators hosted iOS simulators and Android emulators for the mobile engine. | | [`@e2e-dev/decision`](https://e2e.tester.army/docs/decision-models) | Decision-model executors for bounded semantic actions and assertions. | +| [`@e2e-dev/testmu`](https://www.npmjs.com/package/@e2e-dev/testmu) | TestMu AI (formerly LambdaTest) hosted Android emulators, iOS simulators, and real devices for the mobile engine, and hosted Chrome and Edge for the web engine (`@e2e-dev/testmu/web`). | ## Documentation diff --git a/docs/browser.mdx b/docs/browser.mdx index 36914ab75..ec77a493c 100644 --- a/docs/browser.mdx +++ b/docs/browser.mdx @@ -195,8 +195,9 @@ lacks: it asks for a browser when one is needed and gives it back when its scope ends, on every exit path, so a session billed by the minute stops when the run no longer uses it. The engine knows no vendor: for [Kernel](/integrations/kernel), `kernel()` from `@e2e-dev/kernel` is the -provider, and for any other service a provider is a few dozen lines in your -project. Add `viewport: null` when +provider, for [TestMu AI](/integrations/testmu-browsers), +`testmuBrowsers()` from `@e2e-dev/testmu/web` is the provider, and for any other +service a provider is a few dozen lines in your project. Add `viewport: null` when the service shows the window in a live view: the page then [fills the window](/web#fill-the-window) instead of a fixed 1280 by 720 area of it. diff --git a/docs/docs.json b/docs/docs.json index f22c5443a..7343293ff 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -168,7 +168,9 @@ "pages": [ "integrations/index", "integrations/kernel", - "integrations/eas" + "integrations/eas", + "integrations/testmu", + "integrations/testmu-browsers" ] }, { diff --git a/docs/examples/mobile/testmu.config.ts b/docs/examples/mobile/testmu.config.ts new file mode 100644 index 000000000..cad6e264f --- /dev/null +++ b/docs/examples/mobile/testmu.config.ts @@ -0,0 +1,36 @@ +import type { E2EConfig } from 'e2e'; +import { mobile } from '@e2e-dev/mobile'; +import { testmu } from '@e2e-dev/testmu'; + +const apk = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_android.apk'; +const ipa = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_ios.ipa'; + +export default { + targets: [ + { + name: 'android-emulator', + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: apk }), + }), + app: { bundleId: 'com.lambdatest.proverbial' }, + }, + { + name: 'ios-real', + engine: mobile({ + platform: 'ios', + device: testmu({ device: 'iPhone 16', osVersion: '18', app: ipa, deviceType: 'real' }), + }), + app: { bundleId: 'proverbial' }, + }, + { + name: 'android-real', + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Pixel 6', osVersion: '14', app: apk, deviceType: 'real' }), + }), + app: { bundleId: 'com.lambdatest.proverbial' }, + }, + ], + workers: 1, +} satisfies E2EConfig; diff --git a/docs/examples/mobile/testmu.e2e.ts b/docs/examples/mobile/testmu.e2e.ts new file mode 100644 index 000000000..976edf5ca --- /dev/null +++ b/docs/examples/mobile/testmu.e2e.ts @@ -0,0 +1,9 @@ +import { test } from '@e2e-dev/mobile'; +import { expect } from 'e2e'; + +test('Proverbial home screen', async ({ app, screen }) => { + await app.open(); + await expect(screen.getByText('Proverbial')).toBeVisible(); + // Android labels the button GEOLOCATION, iOS GeoLocation. + await expect(screen.getByRole('button', /^geolocation$/i)).toBeVisible(); +}); diff --git a/docs/examples/web/testmu-browsers.config.ts b/docs/examples/web/testmu-browsers.config.ts new file mode 100644 index 000000000..535f23796 --- /dev/null +++ b/docs/examples/web/testmu-browsers.config.ts @@ -0,0 +1,13 @@ +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { testmuBrowsers } from '@e2e-dev/testmu/web'; + +export default { + targets: [ + { + name: 'testmu-web', + engine: web({ browser: testmuBrowsers(), viewport: null }), + app: { url: 'https://staging.example.com' }, + }, + ], +} satisfies E2EConfig; diff --git a/docs/integrations/index.mdx b/docs/integrations/index.mdx index 6358251db..85f8ff050 100644 --- a/docs/integrations/index.mdx +++ b/docs/integrations/index.mdx @@ -17,6 +17,12 @@ integration decides where it comes from. `@e2e-dev/eas`: hosted iOS simulators and Android emulators for mobile targets. + + `@e2e-dev/testmu`: hosted Android emulators, iOS simulators, and real devices for mobile targets. + + + `@e2e-dev/testmu`: hosted Chrome and Edge on Windows and macOS for web targets, one session per worker or per test. + A service without an integration is one small provider in your project: diff --git a/docs/integrations/testmu-browsers.mdx b/docs/integrations/testmu-browsers.mdx new file mode 100644 index 000000000..697a5ee65 --- /dev/null +++ b/docs/integrations/testmu-browsers.mdx @@ -0,0 +1,171 @@ +--- +title: TestMu AI browsers +sidebarTitle: TestMu AI browsers +description: Run web targets in TestMu AI's hosted Chrome and Edge with the testmuBrowsers() browser provider from @e2e-dev/testmu/web. +--- + +`testmuBrowsers()` from `@e2e-dev/testmu/web` is a +[browser provider](/browser#hosted-browsers) for +[TestMu AI](https://www.lambdatest.com) (formerly LambdaTest). By default each worker slot gets its +own TestMu AI session for the run; `scope: 'attempt'` gives every test +attempt its own session instead. Tests and CI stay the same; the config +changes only in the target's `browser`. + +## Setup + + +```bash npm +npm install --save-dev @e2e-dev/testmu +``` + +```bash pnpm +pnpm add -D @e2e-dev/testmu +``` + +```bash bun +bun add -d @e2e-dev/testmu +``` + + +```ts title="e2e.config.ts" +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { testmuBrowsers } from '@e2e-dev/testmu/web'; + +export default { + targets: [ + { + name: 'testmu-web', + engine: web({ browser: testmuBrowsers(), viewport: null }), + app: { url: 'https://staging.example.com' }, + }, + ], +} satisfies E2EConfig; +``` + +The browser provider lives at `@e2e-dev/testmu/web`, so a web-only project +needs neither `@e2e-dev/mobile` nor agent-device. The package root exports +the [device provider](/integrations/testmu), which uses both. + +Set `LT_USERNAME` and `LT_ACCESS_KEY` in the environment `e2e run` starts in, +such as CI secrets. To get them, [sign up for TestMu AI](https://accounts.lambdatest.com/register) +or log in to your account, then copy your username and access key from +**Account Settings → Password & Security → Username and Access Key** +([accounts.lambdatest.com/security/username-accesskey](https://accounts.lambdatest.com/security/username-accesskey)). + +Each worker gets its own session, so `workers` sets how +many run at once. Keep it within your TestMu AI plan's parallel sessions: +a session beyond it waits in TestMu AI's queue, and the web engine gives a +session 60 seconds to accept the CDP connection, so one that waits longer +fails its attempt. See [Limits](#limits). + +## Options + +| Option | Default | Meaning | +| ------ | ------- | ------- | +| `scope` | `'worker'` | `'worker'`: one session per worker slot for the run. `'attempt'`: a fresh session per test attempt, retries included. See [Scopes](#scopes). | +| `browserName` | `'Chrome'` | `'Chrome'` or `'MicrosoftEdge'`. The web engine attaches over CDP, so the browser must be Chromium-based; any other value fails the config. | +| `browserVersion` | `'latest'` | Any version TestMu AI offers for the platform, such as `'140'`. | +| `platform` | `'Windows 11'` | A TestMu AI desktop platform, such as `'Windows 10'` or `'macOS Sequoia'`. | +| `project` | `'e2e'` | Sent as `LT:Options.project`, as [`testmu()`](/integrations/testmu) sends its own. | +| `build` | the run id | The dashboard build the sessions are grouped under. The same default as `testmu()`'s, so a run's device and browser sessions share one build. | +| `sessionName` | `e2e---` | Every session's dashboard name. A given name gets `-` appended when the target has more than one worker slot, and `-` in `attempt` scope, so each session has its own. | +| `geoLocation` | none | Country the browser's IP geolocates to, as a code TestMu AI takes: `'US'`, `'FR'`. | +| `timezone` | none | The machine's time zone, as TestMu AI takes it: `'UTC+05:30'`. | +| `route` | `'/puppeteer'` | The hub route that serves the session's CDP endpoint. See [Routes](#routes). | +| `hub` | `'cdp.lambdatest.com'` | The hub host, without a scheme or path. | +| `capabilities` | none | Further `LT:Options` capabilities, such as `video`, `network`, `console`, `idleTimeout`, `resolution`, or `tunnel` and `tunnelName` for [TestMu AI Tunnel](https://www.lambdatest.com/support/docs/testing-locally-hosted-pages/). `idleTimeout` is 600 seconds when absent. `user`, `accessKey`, `build`, `name`, `project`, `platform`, `geoLocation`, `timezone`, `browserName`, `browserVersion`, and a nested `LT:Options` fail the config here: the provider sets them from the environment and the options above. | + +```ts +testmuBrowsers({ + scope: 'attempt', + platform: 'Windows 10', + build: `checkout ${process.env.GITHUB_SHA}`, + capabilities: { video: true, network: true, idleTimeout: 300 }, +}) +``` + +A misspelled option, a `route` other than the two below, or a `hub` with a +scheme or path fails with `INVALID_CONFIG` when the config loads. + +## Scopes + +`scope: 'worker'` keeps one session per worker slot for the whole run and +gives every attempt a new browser context in it, as a local run does: a test +never sees another test's cookies, storage, or permissions. TestMu AI shows +it as one session, named after the run, the target, and the slot, that covers every +test the worker ran. + +`scope: 'attempt'` starts a session per attempt, so each test is its own +TestMu AI session, named after the run, the target, and the attempt. It costs a +session start per test and carries the limits of a +[per-attempt lease](/browser#hosted-browsers): no `headers`, `basicAuth`, or +`userAgent`, no `app.clearState()`, and no [sessions](/authentication). A +dropped connection cannot be resumed. The engine reconnects to the same CDP +URL, which starts a new TestMu AI session under the same name; the engine +sees a different browser, closes it, and fails the attempt with +`CDP recovery failed`. That second session shows in the dashboard and counts +against your plan while it lasts. + +## Routes + +A TestMu AI session starts when its CDP websocket opens, on one of the +hub's routes: + +- `'/puppeteer'` (default) serves a raw CDP endpoint on every TestMu AI + machine. The dashboard labels these sessions Puppeteer. +- `'/playwright-cdp'` labels sessions as Playwright, but serves raw CDP only on + machines that run TestMu AI's newest backend, which is still rolling out. On + a machine without it the session fails to connect, so keep the default + unless you need the Playwright label. + +## Limits + +- **60-second connect.** The web engine waits 60 seconds for the browser to + accept the CDP connection, and `launchTimeout` does not change that. A + session still in TestMu AI's queue, or one that is slow to provision, fails + its attempt with `connectOverCDP: Timeout 60000ms exceeded` + (`LAUNCH_TIMEOUT`). Chrome on Windows typically connects in about 10 to 30 + seconds; Edge sessions can take longer than the limit. +- **About 24 to 30 workers per target in `worker` scope.** The engine hands + every worker's CDP URL to the workers in one environment variable capped at + 16 KB, and each URL carries its capabilities. Past about 30 workers with the + defaults, or 24 with several `capabilities`, the run fails at start with + `ENGINE_FAILURE: the browser leases take ... bytes`. `scope: 'attempt'` leases in each worker and + has no such ceiling. + +## Recordings + +TestMu AI records the session's screen itself when you pass +`capabilities: { video: true }`; watch it in the TestMu AI dashboard. The +provider does not hand that recording to e2e, so an attempt that records +[video](/reference/config#video) gets the web engine's screencast of the +page, as a local run does. + +## Downloads + +TestMu AI's browsers save downloads on their own machines, and the provider +has no way to read them back, so `browser.waitForDownload` fails with +`ENGINE_FAILURE` naming the provider. Assert on the download link or the +request instead. + +## What the provider does + +- Builds the session's CDP URL with the capabilities above and returns it as + the lease. It calls no TestMu AI API: opening the websocket starts the + session and the engine closing it ends the session, so `release` has + nothing to call. +- Names every session as `testmu()` does, `e2e---` unless `sessionName` is given, in the `build`, sends `project` + as `LT:Options.project`, and + logs `TestMu AI session "" in build ""` to the reporter as + each lease is made. +- Reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and puts + them only in the CDP URL: they never appear in a log line. A missing or + blank one fails the lease with an error naming it, such as + `LT_ACCESS_KEY is not set`; in `worker` scope that ends the run before any + test. +- Sets `idleTimeout` to 600 seconds unless `capabilities` sets one: the hub + otherwise ends a session after 300 seconds without browser traffic, which a + slow step between two browser calls can exceed. The same timeout ends a + session a dead worker held. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx new file mode 100644 index 000000000..0ea55afbd --- /dev/null +++ b/docs/integrations/testmu.mdx @@ -0,0 +1,330 @@ +--- +title: TestMu AI +sidebarTitle: TestMu AI +description: Run mobile targets on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices with the testmu() device provider from @e2e-dev/testmu. +--- + +`testmu()` from `@e2e-dev/testmu` is a +[device provider](/mobile#hosted-devices) for +[TestMu AI](https://www.lambdatest.com) (formerly LambdaTest). The run gets +hosted Android emulators, iOS simulators, or real devices, leased when it +starts and released when it ends. agent-device reaches them through its +`testmu` cloud provider, which drives each session over TestMu AI's Appium +hub. Tests, config, and CI stay the same. + + +No Mac, Xcode, or Android SDK on the machine that runs e2e. + + +## Setup + +The provider runs on agent-device's `testmu` provider, so install +`agent-device` beside the package, at a release that includes it: + + +```bash npm +npm install --save-dev @e2e-dev/testmu agent-device@ +``` + +```bash pnpm +pnpm add -D @e2e-dev/testmu agent-device@ +``` + +```bash bun +bun add -d @e2e-dev/testmu agent-device@ +``` + + +`` is an agent-device release that includes the `testmu` provider. +No published release does yet: until one does, install agent-device from a +build that includes it. The package's `agent-device` peer range excludes +only the releases known to lack the provider, so it cannot tell you whether +a newer one has it. + +`@e2e-dev/mobile` pins its own agent-device. The provider starts the daemon +from your copy and the engine's workers send commands through theirs, so +both must be the same agent-device. Override the engine's pin: + + +```json npm +{ + "overrides": { + "agent-device": "$agent-device" + } +} +``` + +```yaml pnpm +# pnpm-workspace.yaml +overrides: + agent-device: $agent-device +``` + +```json bun +{ + "overrides": { + "agent-device": "" + } +} +``` + + +For npm and bun the `overrides` field goes in `package.json`. + +```ts title="e2e.config.ts" +import type { E2EConfig } from 'e2e'; +import { mobile } from '@e2e-dev/mobile'; +import { testmu } from '@e2e-dev/testmu'; + +const apk = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_android.apk'; +const ipa = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_ios.ipa'; + +export default { + targets: [ + { + name: 'android-emulator', + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: apk }), + }), + app: { bundleId: 'com.lambdatest.proverbial' }, + }, + { + name: 'ios-real', + engine: mobile({ + platform: 'ios', + device: testmu({ device: 'iPhone 16', osVersion: '18', app: ipa, deviceType: 'real' }), + }), + app: { bundleId: 'proverbial' }, + }, + { + name: 'android-real', + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Pixel 6', osVersion: '14', app: apk, deviceType: 'real' }), + }), + app: { bundleId: 'com.lambdatest.proverbial' }, + }, + ], + workers: 1, +} satisfies E2EConfig; +``` + +Every target runs TestMu AI's Proverbial sample app, from its public +Android and iOS builds, so the test below passes on each one. + +`device` and `osVersion` must match TestMu AI's catalog exactly, as the +[capabilities generator](https://www.lambdatest.com/capabilities-generator) +lists them: a virtual iOS device takes a version like `18.0`, a real iOS +device one like `18`, and Android one like `14`. TestMu AI checks them when +the session starts and refuses a name or version it does not list. + +`app` is required: the provider hands the build to TestMu AI, which +uploads a local one and installs it when the session starts. Leave the +target's `app.appPath` out; the provider refuses it on purpose, so a target +that names one fails its lease instead of installing the build twice. +`device.installApp()` with a path still installs a build during a test. +Keep `app.bundleId`: `app.open()` launches the installed bundle id or +package, not the upload name. + +## Authenticate + +Set `LT_USERNAME` and `LT_ACCESS_KEY` to your TestMu AI username and access +key in the environment `e2e run` starts in, such as CI secrets. Without +either, the lease fails before anything starts, naming the missing one. +To get them, [sign up for TestMu AI](https://accounts.lambdatest.com/register) +or log in to your account, then copy your username and access key from +**Account Settings → Password & Security → Username and Access Key** +([accounts.lambdatest.com/security/username-accesskey](https://accounts.lambdatest.com/security/username-accesskey)). + +agent-device's `testmu` runtime reads the credentials from the environment +of the daemon that drives the sessions. The provider starts that daemon from +the runner process, and agent-device starts a daemon with the process's own +environment and takes no other, so while the provider calls the daemon (to +allocate, heartbeat, or release a lease), it sets `LT_USERNAME` and +`LT_ACCESS_KEY` from the run's environment in the runner's `process.env`, +where a daemon the call starts reads them. Once no such call is running, it +puts back the values `process.env` had before, unless the host changed them +in the meantime, so the credentials do not stay there for other child +processes of a long-lived host. In a host that runs several runs with +different credentials at once, their daemon calls take turns, and one run's +slow allocation can delay another run's heartbeats; run one TestMu AI +account per process where you can. + +## Install the app + +`app` names the build TestMu AI installs on every session: + +- **An app id**: an `lt://` reference to a build already in TestMu AI's app + storage. +- **A URL**: an `https` link to the build. +- **A local path**: a build on the machine that runs e2e, resolved against + the directory of `e2e.config.ts` and uploaded when the session starts. + +The build must suit the device: + +| Device | Build | +| --- | --- | +| Android emulator or real device | `.apk` or `.aab` | +| iOS simulator | a zipped `.app` bundle, not an `.ipa` | +| Real iOS device | a signed `.ipa` | + +An iOS simulator target names a zipped simulator build of the app: + +```ts +device: testmu({ device: 'iPhone 16', osVersion: '18.0', app: './build/MyApp.zip' }), +``` + +## Real devices + +`deviceType: 'real'` leases a real device instead of an emulator or +simulator. Real and virtual devices have separate catalogs, so check +`device` and `osVersion` against the real-device one, and give a real iOS +device a signed `.ipa`. Real devices need an agent-device release whose +`testmu` provider supports them. + +## Options + +| Option | Default | Meaning | +| ------------- | --------------- | ------- | +| `device` | required | Device name exactly as TestMu AI's catalog lists it (`'Galaxy S22 Ultra 5G'`, `'iPhone 16'`). | +| `osVersion` | required | OS version exactly as the catalog lists it for that device: `'14'` on Android, `'18.0'` on a virtual iOS device, `'18'` on a real one. | +| `app` | required | The build TestMu AI installs: an `lt://` app id, an `https` URL, or a local path. See [Install the app](#install-the-app). | +| `deviceType` | `'virtual'` | `'virtual'` for an emulator or simulator, `'real'` for a real device. | +| `project` | `'e2e'` | Dashboard project the sessions are grouped under. | +| `build` | the run id | Dashboard build the sessions are grouped under. | +| `sessionName` | `e2e---` | Name of each session on the dashboard. With more than one worker slot, `-` is appended, so every slot's session has its own name; slots count from 1. Give targets that share a `build` different names. | +| `stateDir` | `'.e2e/testmu'` | Directory for the agent-device daemon each run starts, relative to the directory of `e2e.config.ts`. Keep it out of version control. See [What the provider does](#what-the-provider-does) for how long a run's directory is kept. | +| `orientation` | the device's | `'portrait'` or `'landscape'`: the orientation the device starts in. | +| `geoLocation` | TestMu AI's | Country the device's IP address geolocates to, as a country code TestMu AI takes (`'US'`, `'FR'`). | +| `timezone` | the device's | The device's time zone, as TestMu AI takes it (`'UTC+05:30'`). | +| `language` | the device's | The device's language, as a language code (`'fr'`). | +| `locale` | the device's | The device's locale (`'fr_FR'`). | +| `appiumVersion` | TestMu AI's default | Appium version TestMu AI starts for the session (`'2.16.2'`). | + +The device features, `orientation` through `appiumVersion`, are set when the +session starts and hold for the whole session; a test cannot change them. + +An option not in this table, a missing or empty `device`, `osVersion`, or +`app`, another `deviceType` or `orientation`, or any other option given as +something other than a non-empty string fails the config load with +`INVALID_CONFIG`. + +## Write a test + +Tests are the same as on a local device: + +```ts title="tests/home.e2e.ts" +import { test } from '@e2e-dev/mobile'; +import { expect } from 'e2e'; + +test('Proverbial home screen', async ({ app, screen }) => { + await app.open(); + await expect(screen.getByText('Proverbial')).toBeVisible(); + // Android labels the button GEOLOCATION, iOS GeoLocation. + await expect(screen.getByRole('button', /^geolocation$/i)).toBeVisible(); +}); +``` + +## Sessions + +Each target leases up to `workers` devices (`--workers` overrides it), one +per worker slot, before the run's clock starts. Allocating a lease starts +its TestMu AI session, which takes about 30 to 75 seconds, app upload +included, so every slot is billed from the moment it is leased until the run +ends, even when no test runs on it. TestMu AI runs as many sessions at once +as the plan allows, so keep `workers` times the number of TestMu AI targets +within it. + +The provider logs each lease as it is granted: + +```text +ℹ leasing 1 android device(s) from testmu +ℹ testmu (1 of 1): lease 4f0c…: Galaxy S22 Ultra 5G, android 14 (virtual); session e2e-0199…-android-emulator-1 started +ℹ testmu: leased 4f0c… +``` + +The video, device logs, and network logs of every session are on the +TestMu AI dashboard, under the project and build the options name. + +## Recordings + +An attempt that records [video](/reference/config#video) links TestMu AI's +recording of its session in the report, instead of recording through +agent-device, which cannot record over TestMu AI's Appium hub: + +```ts title="e2e.config.ts" +export default { + targets: [ + { + name: 'android-emulator', + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: apk }), + }), + app: { bundleId: 'com.lambdatest.proverbial' }, + video: 'retain-on-failure', + }, + ], +} satisfies E2EConfig; +``` + +TestMu AI records the whole session, from the lease to the end of the run, +so the link is the same video for every attempt on that worker slot, and +the video starts when the session did. When the attempt starts recording, +the provider finds the session through TestMu AI's sessions API by its +build and name, among the sessions of the `LT_USERNAME` user, taking the +newest that matches, and takes the session's start time as the video's, so the +report's timeline offsets land on the attempt within it; if TestMu AI +reports no start time, the time the attempt started recording is used +instead. When the attempt stops recording, the link is the video URL the +API returns for the session. The runner never downloads the video. The +provider reads the API at `TESTMU_API_ENDPOINT` when that is set, as +agent-device does. + +The default `build` (the run id) and `sessionName` (with the run id in it) +give every session its own name. A fixed `build` and a fixed `sessionName` +shared by runs that overlap name several sessions the same, so an attempt +can link another run's video; give such runs their own `build`, or leave +`sessionName` to its default. + +## What the provider does + +- Starts one agent-device daemon per run under `stateDir/` and + allocates a `testmu` lease from it for each worker slot. The allocation + starts the TestMu AI session, which installs `app`. +- Keeps each run's directory, with its daemon's state and logs, after the + run: the daemon exits on its own after 5 idle minutes, and its logs help + explain a failure. The provider marks each run's directory as its own + with a `.e2e-testmu-run` file and touches it on every heartbeat. Before + its first lease, a run removes the marked directories of earlier runs + under `stateDir` in which nothing has changed for more than 24 hours, + never its own. A directory without the marker is never removed, so a + `stateDir` shared with other tools is safe, and a failure to remove one + does not fail the lease. +- Keeps each lease alive while the run holds it. The provider allocates + with agent-device's longest lease window, 10 minutes, and heartbeats the + lease every 2 minutes once it is granted, so a session that takes up to + 10 minutes to start keeps its lease on any agent-device version. A failed heartbeat is logged + once and does not fail the run. +- Hands the worker the lease scope and the device selectors as agent-device + client configuration, so every command lands on the lease's session. +- Releases each lease when the run ends, which ends its session. A lease + is released once however often the engine asks, and a failed release can + be tried again. +- Releases a lease it was granted after the run was interrupted, or could + not hand to the run, before failing the lease. + +## Limits + +Sessions run over TestMu AI's Appium hub through agent-device's WebDriver +runtime, so what agent-device does not support there is not available to +tests: + +- `device` settings: permissions, location, network, and appearance. +- System alerts. +- Recording through agent-device. An attempt's video links TestMu AI's + recording of the session instead; see [Recordings](#recordings). +- Device logs through agent-device; read them on the dashboard. +- Port reverse. To reach a server on your machine or network, use TestMu + AI Tunnel. diff --git a/docs/mobile.mdx b/docs/mobile.mdx index 76efae70a..858edc01c 100644 --- a/docs/mobile.mdx +++ b/docs/mobile.mdx @@ -263,8 +263,10 @@ ends. The engine drives each lease through the agent-device daemon the lease names instead of the local one, so any service that runs an agent-device daemon next to a simulator, an emulator, or a physical device works. Nothing in the framework knows a vendor: for [EAS Simulators](/integrations/eas), -`easSimulators()` from `@e2e-dev/eas` is the provider, and for -any other service a provider is a few dozen lines in your project. +`easSimulators()` from `@e2e-dev/eas` is the provider, for +[TestMu AI](/integrations/testmu), `testmu()` from `@e2e-dev/testmu` is the +provider, and for any other service a provider is a few dozen lines in your +project. ```ts export interface DeviceProvider { @@ -419,7 +421,8 @@ on `app.bundleId`, so traces recorded locally replay on hosted devices. A device cloud agent-device speaks to itself needs no session API of your own: the provider allocates a lease through a daemon it starts for the run and -hands the worker the lease scope as `client`. +hands the worker the lease scope as `client`. `testmu()` from +`@e2e-dev/testmu` works this way for [TestMu AI](/integrations/testmu). ```ts title="device-cloud-provider.ts" expandable import { createAgentDeviceClient } from 'agent-device'; diff --git a/docs/package.json b/docs/package.json index 0b29589de..c161f8ed1 100644 --- a/docs/package.json +++ b/docs/package.json @@ -12,6 +12,7 @@ "@ai-sdk/typesafe-ai": "3.0.12", "@e2e-dev/decision": "workspace:*", "@e2e-dev/mobile": "workspace:*", + "@e2e-dev/testmu": "workspace:*", "@e2e-dev/web": "workspace:*", "@openrouter/ai-sdk-provider": "3.1.0", "@types/node": "26.6.2", diff --git a/docs/reference/environment.mdx b/docs/reference/environment.mdx index ebc3c7707..004dfcc4a 100644 --- a/docs/reference/environment.mdx +++ b/docs/reference/environment.mdx @@ -111,6 +111,7 @@ What is sent and how to read it is on the [Telemetry](/telemetry) page. | `KERNEL_API_KEY` | value | The API key `kernel()` from `@e2e-dev/kernel` creates browsers with; unset, the browser lease fails. See [Kernel](/integrations/kernel) | | `EXPO_TOKEN` | value | The Expo access token `easSimulators()` from `@e2e-dev/eas` starts and stops simulator sessions with; unset, it uses the eas-cli login in `.expo/state.json` under `HOME` (`USERPROFILE` on Windows), and with neither the device lease fails. See [EAS Simulators](/integrations/eas) | | `HOME`, `USERPROFILE` | value | The home `easSimulators()` finds the eas-cli login under when `EXPO_TOKEN` is unset: `USERPROFILE` on Windows, `HOME` elsewhere, the OS user's home when unset | +| `LT_USERNAME`, `LT_ACCESS_KEY` | value | The TestMu AI username and access key `testmu()` from `@e2e-dev/testmu` leases devices with; it sets them in the runner process's environment only while it calls the agent-device daemon it starts, which reads them from there, and with either unset the device lease fails. `testmuBrowsers()` puts them only in the CDP URL of each browser session, and with either unset the browser lease fails. Both come from Account Settings → Password & Security → Username and Access Key on your TestMu AI account. See [TestMu AI](/integrations/testmu) and [TestMu AI browsers](/integrations/testmu-browsers) | | `E2E_AGENT_DEVICE_POOL__` | value | Written by `@e2e-dev/mobile` in `prepare` and read by each worker to find its booted device. Not meant to be set by hand | ## What app processes inherit diff --git a/docs/security.mdx b/docs/security.mdx index 9cb75f3c0..34c439fee 100644 --- a/docs/security.mdx +++ b/docs/security.mdx @@ -296,6 +296,9 @@ Built-in features may connect to: - the GitHub API, only from the opt-in `@e2e-dev/github` reporter - the Kernel API and its hosted browsers, only from [`kernel()`](/integrations/kernel) +- TestMu AI's CDP hub and its hosted browsers, only from + [`testmuBrowsers()`](/integrations/testmu-browsers). The username and + access key travel only in the CDP URL, never in a log line - the Expo API and the simulators' agent-device daemons, only from [`easSimulators()`](/integrations/eas). It authenticates with `EXPO_TOKEN`, else with the eas-cli login in `~/.expo/state.json`: on a CI runner, set diff --git a/package.json b/package.json index a53a86afa..ebd81d417 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,7 @@ ], "type": "module", "scripts": { - "build": "pnpm --filter e2e run build && pnpm --filter @e2e-dev/web run build && pnpm --filter @e2e-dev/mobile run build && pnpm --filter @e2e-dev/github run build && pnpm --filter @e2e-dev/kernel run build && pnpm --filter @e2e-dev/eas run build && pnpm --filter @e2e-dev/decision run build", + "build": "pnpm --filter e2e run build && pnpm --filter @e2e-dev/web run build && pnpm --filter @e2e-dev/mobile run build && pnpm --filter @e2e-dev/github run build && pnpm --filter @e2e-dev/kernel run build && pnpm --filter @e2e-dev/eas run build && pnpm --filter @e2e-dev/decision run build && pnpm --filter @e2e-dev/testmu run build", "check": "pnpm run lint && pnpm run check:dead-code && pnpm run typecheck && pnpm run docs:check-errors && pnpm run check:peer-ranges && pnpm run check:install-scripts && pnpm run docs:check", "check:dead-code": "fallow dead-code", "check:peer-ranges": "node scripts/check-peer-ranges.ts", @@ -25,7 +25,7 @@ "canary": "node scripts/canary-changeset.ts && changeset version --snapshot canary && pnpm run build", "canary:publish": "changeset publish --tag canary --no-git-tag && node scripts/restore-peer-ranges.ts", "typecheck": "pnpm run build && pnpm -r run typecheck && tsc -p scripts", - "test": "pnpm run build && pnpm --filter e2e run test && pnpm --filter @e2e-dev/web run test && pnpm --filter @e2e-dev/mobile run test && pnpm --filter @e2e-dev/github run test && pnpm --filter @e2e-dev/kernel run test && pnpm --filter @e2e-dev/eas run test && pnpm --filter @e2e-dev/decision run test && pnpm run test:scripts", + "test": "pnpm run build && pnpm --filter e2e run test && pnpm --filter @e2e-dev/web run test && pnpm --filter @e2e-dev/mobile run test && pnpm --filter @e2e-dev/github run test && pnpm --filter @e2e-dev/kernel run test && pnpm --filter @e2e-dev/eas run test && pnpm --filter @e2e-dev/decision run test && pnpm --filter @e2e-dev/testmu run test && pnpm run test:scripts", "test:testbed": "pnpm run build && pnpm --filter @e2e-dev/testbed test", "test:web-benchmark": "pnpm run build && pnpm --filter @e2e-dev/web-benchmark test", "test:mobile-benchmark": "pnpm run build && pnpm --filter @e2e-dev/mobile-benchmark test", diff --git a/packages/mobile/src/nodes.ts b/packages/mobile/src/nodes.ts index f66b7d651..55a88e9ce 100644 --- a/packages/mobile/src/nodes.ts +++ b/packages/mobile/src/nodes.ts @@ -232,9 +232,10 @@ const ANDROID_TITLE_IDS = [':id/collapsing_toolbar', ':id/action_bar', ':id/tool /** * Platform element type of one raw node as a kebab-case token: XCTest sends - * `NavigationBar` and `StaticText`, Android sends `android.widget.TextView`; - * both read as one vocabulary here, the Android package prefix dropped. - * `role` is the fallback some platforms send instead. + * `NavigationBar` and `StaticText` (`XCUIElementTypeNavigationBar` over + * WebDriver), Android sends `android.widget.TextView`; all read as one + * vocabulary here, the class and package prefixes dropped. `role` is the + * fallback some platforms send instead. */ function kindOf(raw: RawNode): string { return normalizeKind(raw.type ?? raw.role ?? ''); @@ -245,9 +246,9 @@ function isAndroidClass(type: string | undefined): boolean { return type !== undefined && type.includes('.'); } -/** One element-type spelling for `NavigationBar`, `navigation-bar`, and `android.widget.NavigationBar` alike. */ +/** One element-type spelling for `NavigationBar`, `XCUIElementTypeNavigationBar`, `navigation-bar`, and `android.widget.NavigationBar` alike. */ export function normalizeKind(type: string): string { - const simple = type.slice(type.lastIndexOf('.') + 1); + const simple = type.slice(type.lastIndexOf('.') + 1).replace(/^XCUIElementType(?=[A-Z])/, ''); return simple .replaceAll(/([a-z0-9])([A-Z])/g, '$1-$2') .replaceAll(/[\s_]+/g, '-') diff --git a/packages/mobile/tests/unit/nodes.test.ts b/packages/mobile/tests/unit/nodes.test.ts index 64a10cebc..fd1b01276 100644 --- a/packages/mobile/tests/unit/nodes.test.ts +++ b/packages/mobile/tests/unit/nodes.test.ts @@ -193,6 +193,36 @@ describe('snapshot projection', () => { expect(screenTitle(projected)).toBe('Settings'); }); + it('reads the XCUIElementType class names a WebDriver session reports as the native XCTest types', () => { + const native: RawNode[] = [ + { ref: 'e1', index: 0, depth: 0, type: 'Application', label: 'Profiler', rect: { x: 0, y: 0, width: 390, height: 844 } }, + { ref: 'e2', index: 1, parentIndex: 0, depth: 1, type: 'NavigationBar', identifier: 'Profiler' }, + { ref: 'e3', index: 2, parentIndex: 1, depth: 2, type: 'StaticText', label: 'Profiler' }, + { ref: 'e4', index: 3, parentIndex: 0, depth: 1, type: 'Button', label: 'CPU Load' }, + { ref: 'e5', index: 4, parentIndex: 0, depth: 1, type: 'TextView', label: 'Notes', value: 'idle' }, + { ref: 'e6', index: 5, parentIndex: 0, depth: 1, type: 'TabBar' }, + { ref: 'e7', index: 6, parentIndex: 5, depth: 2, type: 'Button', label: 'Home', selected: true }, + { ref: 'e8', index: 7, parentIndex: 5, depth: 2, type: 'Button', label: 'Settings' }, + ]; + const webDriver = native.map((node) => ({ ...node, type: `XCUIElementType${node.type}` })); + const kindsAndRoles = (nodes: readonly RawNode[]) => project(nodes).index.map((entry) => [entry.kind, entry.node.role]); + expect(kindsAndRoles(webDriver)).toEqual(kindsAndRoles(native)); + const projected = project(webDriver); + expect(projected.index.map((entry) => entry.kind)).toEqual([ + 'application', + 'navigation-bar', + 'static-text', + 'button', + 'text-view', + 'tab-bar', + 'button', + 'button', + ]); + expect(projected.index.map((entry) => entry.node.role)).toEqual(['application', 'navigation', 'text', 'button', 'textbox', 'tablist', 'tab', 'tab']); + expect(projected.viewport).toEqual({ width: 390, height: 844 }); + expect(screenTitle(projected)).toBe('Profiler'); + }); + it('maps iOS composite widgets onto the vocabulary roles a browser reports for them', () => { const projected = project([ { ref: 'e1', index: 0, depth: 0, type: 'Application', label: 'Shop' }, diff --git a/packages/testmu/LICENSE b/packages/testmu/LICENSE new file mode 100644 index 000000000..d64569567 --- /dev/null +++ b/packages/testmu/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/packages/testmu/NOTICE b/packages/testmu/NOTICE new file mode 100644 index 000000000..a5f5c8473 --- /dev/null +++ b/packages/testmu/NOTICE @@ -0,0 +1,5 @@ +@e2e-dev/testmu +Copyright 2026 TesterArmy, Inc. + +This product includes software developed at TesterArmy, Inc. +(https://tester.army). diff --git a/packages/testmu/README.md b/packages/testmu/README.md new file mode 100644 index 000000000..47d96e820 --- /dev/null +++ b/packages/testmu/README.md @@ -0,0 +1,120 @@ +# @e2e-dev/testmu + +[TestMu AI](https://www.lambdatest.com) (formerly LambdaTest) for [`e2e`](https://www.npmjs.com/package/e2e): +`mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on +TestMu AI's hosted Android emulators, iOS simulators, and real devices. + +## Install + +```bash +npm install --save-dev @e2e-dev/testmu agent-device@ +``` + +The provider drives TestMu AI through agent-device's `testmu` cloud +provider, so the project needs an agent-device that includes it, and +`@e2e-dev/mobile` must use that same agent-device: override its pin (npm and +bun `overrides`, pnpm `overrides` in `pnpm-workspace.yaml`). No published +agent-device release includes the `testmu` provider yet; `` is the +first one that does. The peer range only excludes the releases known to lack +it. + +## Usage + +```ts title="e2e.config.ts" +import type { E2EConfig } from 'e2e'; +import { mobile } from '@e2e-dev/mobile'; +import { testmu } from '@e2e-dev/testmu'; + +export default { + targets: [ + { + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: 'https://example.com/app.apk' }), + }), + app: { bundleId: 'com.example.app' }, + }, + ], + workers: 2, +} satisfies E2EConfig; +``` + +It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's +environment. While it calls the agent-device daemon it starts for the run, +it sets both in the runner's `process.env`, where that daemon reads them, +and puts back the previous values afterwards. Each worker slot leases one +device from that daemon. Allocating the lease starts the TestMu AI +session, which installs `app`, so every slot is billed from the moment it is +leased, even when no test runs on it. The lease is released when the run +ends, which ends the session. + +- `device` and `osVersion` must match TestMu AI's catalog exactly: `18.0` + for a virtual iOS device, `18` for a real one, `14` on Android. +- `app` is required: an `lt://` app id, an `https` URL, or a local path + resolved against the project root, which TestMu AI installs when the + session starts. Keep the target's `app.bundleId`. The provider refuses + `app.appPath` on purpose, since it hands TestMu AI the build itself; + `device.installApp()` with a path still works during a test. +- `deviceType: 'real'` picks a real device (default `'virtual'`). +- `project` (default `e2e`), `build` (default the run id), and `sessionName` + (default `e2e---`) label the sessions on the + dashboard. A given `sessionName` gets `-` appended when the target + has more than one worker slot, so each slot's session has its own name. +- `orientation` (`'portrait'` or `'landscape'`), `geoLocation`, `timezone`, + `language`, `locale`, and `appiumVersion` set up the device when the + session starts. +- `stateDir` (default `.e2e/testmu`) holds each run's daemon, in a + directory per run that is kept after the run for its logs. A run removes + directories earlier runs marked as the provider's once nothing in them + has changed for 24 hours; it never removes a directory it did not mark. + +Sessions run over TestMu AI's Appium hub, so agent-device's device settings, +system alerts, recording, device logs, and port reverse are not available +there. An attempt that records video links TestMu AI's recording of the +whole session instead, found by the session's build and name and starting +at the session's start time. Runs that overlap and share a fixed `build` +and `sessionName` can link each other's videos; the defaults never do. + +## Browsers + +`web({ browser: testmuBrowsers() })` runs a web target in TestMu AI's hosted +Chrome or Edge on Windows and macOS, one session per worker slot, or one per +attempt with `scope: 'attempt'`. It is exported from `@e2e-dev/testmu/web`, +which needs only `@e2e-dev/web`; the package root exports the device +provider. + +```ts title="e2e.config.ts" +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { testmuBrowsers } from '@e2e-dev/testmu/web'; + +export default { + targets: [ + { + name: 'testmu-web', + engine: web({ browser: testmuBrowsers({ platform: 'Windows 11' }), viewport: null }), + app: { url: 'https://staging.example.com' }, + }, + ], +} satisfies E2EConfig; +``` + +A session starts when its CDP websocket opens and ends when the engine +closes it, so the provider only builds the CDP URL from the options and +`LT_USERNAME`/`LT_ACCESS_KEY`, and calls no API. + +## Credentials + +Both providers read `LT_USERNAME` and `LT_ACCESS_KEY` from the run's +environment. To get them, [sign up for TestMu AI](https://accounts.lambdatest.com/register) +or log in to your account, then copy your username and access key from +**Account Settings → Password & Security → Username and Access Key** +([accounts.lambdatest.com/security/username-accesskey](https://accounts.lambdatest.com/security/username-accesskey)). + +Full documentation lives at [e2e.tester.army/docs/integrations/testmu](https://e2e.tester.army/docs/integrations/testmu) +(devices) and [e2e.tester.army/docs/integrations/testmu-browsers](https://e2e.tester.army/docs/integrations/testmu-browsers) +(browsers). + +## License + +Apache-2.0 diff --git a/packages/testmu/package.json b/packages/testmu/package.json new file mode 100644 index 000000000..debb63e89 --- /dev/null +++ b/packages/testmu/package.json @@ -0,0 +1,93 @@ +{ + "name": "@e2e-dev/testmu", + "version": "0.0.0", + "description": "TestMu AI (formerly LambdaTest) for e2e: run mobile targets on hosted Android emulators, iOS simulators, and real devices, and web targets on hosted Chrome and Edge", + "keywords": [ + "e2e", + "end-to-end", + "testing", + "mobile-testing", + "testmu", + "lambdatest", + "browser-testing", + "chrome", + "cdp", + "device-cloud", + "real-devices", + "ios-simulator", + "android-emulator", + "agent-device", + "ci" + ], + "license": "Apache-2.0", + "contributors": [ + "Oskar Kwaśniewski ", + "Szymon Rybczak " + ], + "repository": { + "type": "git", + "url": "git+https://github.com/tester-army/e2e.git", + "directory": "packages/testmu" + }, + "bugs": { + "url": "https://github.com/tester-army/e2e/issues" + }, + "homepage": "https://github.com/tester-army/e2e/tree/main/packages/testmu#readme", + "publishConfig": { + "registry": "https://registry.npmjs.org/", + "access": "public", + "tag": "latest" + }, + "type": "module", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "default": "./dist/index.js" + }, + "./web": { + "types": "./dist/web.d.ts", + "default": "./dist/web.js" + } + }, + "files": [ + "dist", + "NOTICE" + ], + "scripts": { + "build": "node -e \"require('node:fs').rmSync('dist',{recursive:true,force:true})\" && tsc --project tsconfig.build.json && node ../../scripts/stamp-dist.ts", + "prepublishOnly": "node ../../scripts/check-dist.ts", + "typecheck": "tsc --noEmit", + "test": "vitest run", + "test:watch": "vitest", + "test:unit": "vitest run tests/unit" + }, + "peerDependencies": { + "@e2e-dev/mobile": ">=0.9.0 <1", + "@e2e-dev/web": ">=0.11.0 <1", + "agent-device": ">0.21.20 <1", + "e2e": ">=0.15.0 <1" + }, + "peerDependenciesMeta": { + "@e2e-dev/mobile": { + "optional": true + }, + "@e2e-dev/web": { + "optional": true + }, + "agent-device": { + "optional": true + } + }, + "devDependencies": { + "@e2e-dev/mobile": "workspace:*", + "@e2e-dev/web": "workspace:*", + "@types/node": "26.6.2", + "agent-device": "0.21.20", + "e2e": "workspace:*", + "typescript": "7.0.2", + "vitest": "5.0.1" + }, + "engines": { + "node": "^22.22.3 || >=24.8.0" + } +} diff --git a/packages/testmu/src/browsers.ts b/packages/testmu/src/browsers.ts new file mode 100644 index 000000000..d8d8bcf69 --- /dev/null +++ b/packages/testmu/src/browsers.ts @@ -0,0 +1,189 @@ +/** TestMu AI's hosted Chrome and Edge as a `BrowserProvider` for the web engine. */ + +import type { BrowserLease, BrowserProvider, BrowserProviderScope, BrowserRequest } from '@e2e-dev/web'; +import { ConfigurationError, rejectUnknownKeys } from 'e2e/engine'; +import { testmuCredentials } from './credentials.ts'; + +/** + * The hub route that serves a raw CDP endpoint. `/puppeteer` serves it on + * every machine. `/playwright-cdp` labels sessions as Playwright, but serves + * raw CDP only on machines that run TestMu AI's newest backend, which is + * still rolling out; elsewhere the session fails to connect. + */ +export type TestmuBrowsersRoute = '/puppeteer' | '/playwright-cdp'; + +/** The browsers the web engine can attach to over CDP: Chromium ones. */ +export type TestmuBrowserName = 'Chrome' | 'MicrosoftEdge'; + +export interface TestmuBrowsersOptions { + /** + * `worker` (default): one TestMu AI session per worker slot for the run. + * `attempt`: a fresh session per test attempt, so each test is its own + * TestMu AI session; rules out `headers`, `basicAuth`, and `userAgent`. + */ + readonly scope?: BrowserProviderScope | undefined; + /** CDP route on the hub; `/puppeteer` when absent. `/playwright-cdp` is still rolling out (see `TestmuBrowsersRoute`). */ + readonly route?: TestmuBrowsersRoute | undefined; + /** The hub host, without a scheme or path; `cdp.lambdatest.com` when absent. */ + readonly hub?: string | undefined; + /** `Chrome` when absent. */ + readonly browserName?: TestmuBrowserName | undefined; + /** `latest` when absent. */ + readonly browserVersion?: string | undefined; + /** TestMu AI platform name, such as `Windows 11` (default) or `macOS Sequoia`. */ + readonly platform?: string | undefined; + /** Sent as `LT:Options.project`, as `testmu()` sends its own. Defaults to `e2e`. */ + readonly project?: string | undefined; + /** Dashboard build the sessions are grouped under. Defaults to the run id, as `testmu()`'s, so a run's device and browser sessions share one build. */ + readonly build?: string | undefined; + /** + * Name of every session on the dashboard, with `-` (worker scope, + * more than one slot) or `-` (attempt scope) appended so each + * session has its own. Defaults to `e2e---`. Slots count from 1. + */ + readonly sessionName?: string | undefined; + /** Country the browser's IP geolocates to, as a code TestMu AI takes: `US`, `FR`. */ + readonly geoLocation?: string | undefined; + /** The machine's time zone, as TestMu AI takes it: `UTC+05:30`. */ + readonly timezone?: string | undefined; + /** + * Further `LT:Options` capabilities (`video`, `network`, `console`, + * `idleTimeout`, `tunnel`, ...). `idleTimeout` is 600 seconds when absent. + * `user`, `accessKey`, `build`, `name`, `project`, `platform`, + * `geoLocation`, `timezone`, `browserName`, `browserVersion`, and a nested + * `LT:Options` are not accepted here: the provider sets them, from the + * environment and the options above. + */ + readonly capabilities?: Readonly> | undefined; +} + +const OPTION_KEYS: readonly string[] = Object.keys({ + scope: true, + route: true, + hub: true, + browserName: true, + browserVersion: true, + platform: true, + project: true, + build: true, + sessionName: true, + geoLocation: true, + timezone: true, + capabilities: true, +} satisfies Record); + +const SCOPES: readonly string[] = ['worker', 'attempt'] satisfies BrowserProviderScope[]; +const ROUTES: readonly string[] = ['/puppeteer', '/playwright-cdp'] satisfies TestmuBrowsersRoute[]; +const BROWSERS: readonly string[] = ['Chrome', 'MicrosoftEdge'] satisfies TestmuBrowserName[]; +const PROVIDER_CAPABILITIES = ['user', 'accessKey', 'build', 'name', 'project', 'platform', 'geoLocation', 'timezone', 'browserName', 'browserVersion', 'LT:Options']; + +/** The options that are strings when given, checked at config load as `testmu()` checks its own. */ +const OPTIONAL_STRING_KEYS = ['browserVersion', 'platform', 'project', 'build', 'sessionName', 'geoLocation', 'timezone'] as const; + +/** The `LT:Options.project` sent when `project` is absent, as `testmu()`'s default. */ +const DEFAULT_PROJECT = 'e2e'; +const HOST = /^[A-Za-z0-9.-]+(:\d+)?$/; + +const DEFAULT_HUB = 'cdp.lambdatest.com'; +const DEFAULT_ROUTE: TestmuBrowsersRoute = '/puppeteer'; +// The hub ends a CDP session after 300 s without client traffic, shorter than a slow test's gaps. +const DEFAULT_IDLE_TIMEOUT_SECONDS = 600; + +/** + * TestMu AI browsers for `web({ browser: testmuBrowsers() })`. A TestMu AI + * session is its websocket: connecting to the CDP URL starts it and closing + * the connection ends it, so `acquire` only builds the URL and `release` has + * nothing to call. Every session is named after the target and the slot or + * attempt, inside one build per run. `LT_USERNAME` and `LT_ACCESS_KEY` come + * from the run's environment and never appear in a log line. + */ +export function testmuBrowsers(options: TestmuBrowsersOptions = {}): BrowserProvider { + if (!isRecord(options)) { + throw new ConfigurationError('INVALID_CONFIG', 'testmuBrowsers() options must be an object'); + } + rejectUnknownKeys('testmuBrowsers()', options, OPTION_KEYS); + // Only an absent value takes the default: `null` from a JavaScript config is refused like any other. + const { scope } = options; + const route = options.route === undefined ? DEFAULT_ROUTE : options.route; + const hub = options.hub === undefined ? DEFAULT_HUB : options.hub; + const browserName = options.browserName === undefined ? 'Chrome' : options.browserName; + if (scope !== undefined && !SCOPES.includes(scope)) { + throw new ConfigurationError('INVALID_CONFIG', `testmuBrowsers: \`scope\` must be one of ${SCOPES.join(', ')}, got ${JSON.stringify(scope)}`); + } + if (!ROUTES.includes(route)) { + throw new ConfigurationError('INVALID_CONFIG', `testmuBrowsers: \`route\` must be one of ${ROUTES.join(', ')}, got "${route}"`); + } + if (typeof hub !== 'string' || !HOST.test(hub)) { + throw new ConfigurationError('INVALID_CONFIG', `testmuBrowsers: \`hub\` must be a host such as "${DEFAULT_HUB}", without a scheme or path, got "${hub}"`); + } + if (!BROWSERS.includes(browserName)) { + throw new ConfigurationError('INVALID_CONFIG', `testmuBrowsers: \`browserName\` must be one of ${BROWSERS.join(', ')}, got "${browserName}"`); + } + for (const key of OPTIONAL_STRING_KEYS) { + const value: unknown = options[key]; + if (value !== undefined && (typeof value !== 'string' || value.trim() === '')) { + throw new ConfigurationError('INVALID_CONFIG', `testmuBrowsers: \`${key}\` must be a non-empty string`); + } + } + if (options.capabilities !== undefined && !isRecord(options.capabilities)) { + throw new ConfigurationError('INVALID_CONFIG', 'testmuBrowsers: `capabilities` must be an object of `LT:Options` fields'); + } + const reserved = PROVIDER_CAPABILITIES.filter((key) => options.capabilities !== undefined && key in options.capabilities); + if (reserved.length > 0) { + throw new ConfigurationError( + 'INVALID_CONFIG', + `testmuBrowsers: \`capabilities\` cannot set ${reserved.map((key) => `\`${key}\``).join(', ')}; use the options of the same name, and LT_USERNAME/LT_ACCESS_KEY for credentials`, + ); + } + return { + name: 'testmu-browsers', + ...(scope === undefined ? {} : { scope }), + async acquire(request: BrowserRequest): Promise { + const { username, accessKey } = testmuCredentials(request.env); + const label = request.attemptId ?? `slot ${request.slot + 1} of ${request.slots}`; + const build = options.build ?? request.runId; + const name = sessionNameFor(options.sessionName, request); + const capabilities = { + browserName, + browserVersion: options.browserVersion ?? 'latest', + 'LT:Options': { + idleTimeout: DEFAULT_IDLE_TIMEOUT_SECONDS, + ...options.capabilities, + platform: options.platform ?? 'Windows 11', + project: options.project ?? DEFAULT_PROJECT, + build, + name, + ...(options.geoLocation === undefined ? {} : { geoLocation: options.geoLocation }), + ...(options.timezone === undefined ? {} : { timezone: options.timezone }), + user: username, + accessKey, + }, + }; + request.log(`TestMu AI session "${name}" in build "${build}"`); + return { + // Worker-scope leases share one bounded environment variable, so the id stays short. + id: `${request.targetName}:${label}`, + cdpEndpoint: `wss://${hub}${route}?capabilities=${encodeURIComponent(JSON.stringify(capabilities))}`, + }; + }, + async release(): Promise { + // Closing the CDP connection, which the engine does, ends the TestMu AI session. + }, + }; +} + +/** + * The dashboard name of a session, unique among the run's sessions of the + * target, in `testmu()`'s format: `e2e---` per worker + * slot, or `-` in attempt scope. + */ +function sessionNameFor(sessionName: string | undefined, { runId, targetName, slot, slots, attemptId }: BrowserRequest): string { + const suffix = attemptId ?? String(slot + 1); + if (sessionName === undefined) return `e2e-${runId}-${targetName}-${suffix}`; + return attemptId !== undefined || slots > 1 ? `${sessionName}-${suffix}` : sessionName; +} + +/** True for a plain object. Config runs as JavaScript, so the types alone are no guard. */ +function isRecord(value: unknown): value is object { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} diff --git a/packages/testmu/src/credentials.ts b/packages/testmu/src/credentials.ts new file mode 100644 index 000000000..39b756b9c --- /dev/null +++ b/packages/testmu/src/credentials.ts @@ -0,0 +1,109 @@ +/** + * The TestMu AI credentials a run authenticates with: `LT_USERNAME` and + * `LT_ACCESS_KEY` from the run's environment. agent-device's `testmu` + * runtime reads them from the environment of the daemon that drives the + * sessions, not from a request. + */ + +const LT_USERNAME = 'LT_USERNAME'; +const LT_ACCESS_KEY = 'LT_ACCESS_KEY'; + +export interface TestmuCredentials { + readonly username: string; + readonly accessKey: string; +} + +/** `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment; throws naming each one that is unset or blank. */ +export function testmuCredentials(env: Readonly>): TestmuCredentials { + const username = envValue(env, LT_USERNAME); + const accessKey = envValue(env, LT_ACCESS_KEY); + if (username === undefined || accessKey === undefined) { + const missing = [username === undefined ? LT_USERNAME : undefined, accessKey === undefined ? LT_ACCESS_KEY : undefined].filter((name) => name !== undefined); + throw new Error( + `${missing.join(' and ')} ${missing.length === 1 ? 'is' : 'are'} not set; set ${LT_USERNAME} and ${LT_ACCESS_KEY} to your TestMu AI username and access key in the environment \`e2e run\` starts in`, + ); + } + return { username, accessKey }; +} + +/** The run's credentials when both are set, else `undefined`, for a call that can do without them. */ +export function optionalTestmuCredentials(env: Readonly>): TestmuCredentials | undefined { + const username = envValue(env, LT_USERNAME); + const accessKey = envValue(env, LT_ACCESS_KEY); + return username === undefined || accessKey === undefined ? undefined : { username, accessKey }; +} + +/** Daemon calls in flight under one set of credentials, and the values those replaced in `process.env`. */ +interface CredentialScope { + readonly credentials: TestmuCredentials; + calls: number; + readonly replaced: { readonly username: string | undefined; readonly accessKey: string | undefined }; + /** Settles once the last call of the scope has finished. */ + readonly closed: Promise; + readonly close: () => void; +} + +let scope: CredentialScope | undefined; + +/** + * Runs a call to the run's daemon with the run's credentials in this + * process's environment, which is where a daemon the call starts gets them: + * agent-device's client starts its local daemon with `process.env` and takes + * no environment of its own, while a run's environment can differ from + * `process.env` (a host that passes `env`). Calls with the same credentials + * overlap; a call with other credentials waits until they have finished, so + * a daemon never starts under another run's user. Once no call is in flight + * the previous values are put back, so the credentials do not linger for + * other child processes of a long-lived host. Every worker of the run + * already starts with these values. + */ +export async function withDaemonCredentials(credentials: TestmuCredentials | undefined, call: () => Promise): Promise { + if (credentials === undefined) return call(); + const current = await enterScope(credentials); + try { + return await call(); + } finally { + leaveScope(current); + } +} + +async function enterScope(credentials: TestmuCredentials): Promise { + // `scope` changes while this waits: another call may open one first. + for (let open = scope; open !== undefined && !sameCredentials(open.credentials, credentials); open = scope) await open.closed; + if (scope === undefined) { + let close!: () => void; + const closed = new Promise((resolve) => (close = resolve)); + scope = { credentials, calls: 0, replaced: { username: process.env[LT_USERNAME], accessKey: process.env[LT_ACCESS_KEY] }, closed, close }; + } + // Set on every call, not only the first: the host may have changed them since. + process.env[LT_USERNAME] = credentials.username; + process.env[LT_ACCESS_KEY] = credentials.accessKey; + scope.calls += 1; + return scope; +} + +function sameCredentials(a: TestmuCredentials, b: TestmuCredentials): boolean { + return a.username === b.username && a.accessKey === b.accessKey; +} + +function leaveScope(current: CredentialScope): void { + current.calls -= 1; + if (current.calls > 0) return; + scope = undefined; + restore(LT_USERNAME, current.credentials.username, current.replaced.username); + restore(LT_ACCESS_KEY, current.credentials.accessKey, current.replaced.accessKey); + current.close(); +} + +/** Puts back `previous`, unless the host changed the value from the one the scope set: that one is the host's. */ +function restore(name: string, set: string, previous: string | undefined): void { + if (process.env[name] !== set) return; + if (previous === undefined) delete process.env[name]; + else process.env[name] = previous; +} + +/** A non-empty variable from the run's environment, trimmed, or `undefined`. */ +function envValue(env: Readonly>, name: string): string | undefined { + const value = env[name]?.trim(); + return value === undefined || value === '' ? undefined : value; +} diff --git a/packages/testmu/src/index.ts b/packages/testmu/src/index.ts new file mode 100644 index 000000000..cdce6162a --- /dev/null +++ b/packages/testmu/src/index.ts @@ -0,0 +1,10 @@ +/** + * `@e2e-dev/testmu` public surface: `testmu()`, a device provider that leases + * TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real + * devices for `@e2e-dev/mobile` through agent-device's `testmu` provider. + * `testmuBrowsers()`, for TestMu AI's hosted Chrome and Edge on `@e2e-dev/web`, + * is exported from `@e2e-dev/testmu/web`. + */ + +export { testmu } from './provider.ts'; +export type { TestmuOptions } from './provider.ts'; diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts new file mode 100644 index 000000000..0923c97de --- /dev/null +++ b/packages/testmu/src/provider.ts @@ -0,0 +1,401 @@ +/** + * TestMu AI's hosted Android emulators, iOS simulators, and real devices as a + * `DeviceProvider` for the mobile engine, through agent-device's `testmu` + * cloud provider: the provider allocates agent-device leases, and the daemon + * holding them runs each session over TestMu AI's Appium hub. + */ + +import { lstat, mkdir, readdir, rm, utimes, writeFile } from 'node:fs/promises'; +import { isAbsolute, join, resolve } from 'node:path'; +import type { DeviceLease, DeviceProvider, DeviceReleaseContext, DeviceRequest } from '@e2e-dev/mobile'; +import { createAgentDeviceClient } from 'agent-device'; +import { ConfigurationError, rejectUnknownKeys, type ProviderRecordContext, type ProviderRecording } from 'e2e/engine'; +import { optionalTestmuCredentials, testmuCredentials, withDaemonCredentials, type TestmuCredentials } from './credentials.ts'; +import { findSession, sessionVideoUrl, testmuApiEndpoint, type SessionRef } from './sessions.ts'; + +/** agent-device's name for TestMu AI, as the lease provider and the tenant. */ +const PROVIDER = 'testmu'; + +/** Where each run's daemon keeps its state, under the project root. */ +const DEFAULT_STATE_DIR = '.e2e/testmu'; + +/** How long an earlier run's directory under `stateDir` is kept: its daemon exits after 5 idle minutes, and its logs help with a failure until then. */ +const RUN_STATE_TTL_MS = 24 * 60 * 60_000; + +/** A run id, which names each run's directory under `stateDir`; nothing else there is pruned. */ +const RUN_DIR_NAME = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + +/** + * The file that marks a run's directory as the provider's: only a marked + * directory is ever pruned, since `stateDir` may be shared with other tools. + * Heartbeats keep it fresh while a run holds leases. + */ +const RUN_MARKER = '.e2e-testmu-run'; + +/** The dashboard project sessions are grouped under when `project` is absent. */ +const DEFAULT_PROJECT = 'e2e'; + +/** + * The inactivity window each lease asks for, agent-device's longest, so a + * session that takes up to 10 minutes to start keeps its lease whether + * agent-device starts the window when the allocation begins or when it + * completes; the heartbeat keeps it alive after that. + */ +const LEASE_TTL_MS = 10 * 60_000; + +/** How often the runner heartbeats each lease it holds, well inside `LEASE_TTL_MS`. */ +const HEARTBEAT_INTERVAL_MS = 2 * 60_000; + +const DEVICE_TYPES: ReadonlySet = new Set(['virtual', 'real']); + +const ORIENTATIONS: ReadonlySet = new Set(['portrait', 'landscape']); + +/** Each device-feature option and the agent-device lease key it is allocated under. */ +const DEVICE_FEATURES = { + orientation: 'providerDeviceOrientation', + geoLocation: 'providerGeoLocation', + timezone: 'providerTimezone', + language: 'providerLanguage', + locale: 'providerLocale', + appiumVersion: 'providerAppiumVersion', +} as const satisfies Partial>; + +/** The options that are strings when given, checked at config load. */ +const OPTIONAL_STRING_KEYS = ['project', 'build', 'sessionName', 'stateDir', ...(Object.keys(DEVICE_FEATURES) as (keyof typeof DEVICE_FEATURES)[])] as const; + +/** Every option `testmu()` takes, kept equal to `TestmuOptions` by the compiler. */ +const OPTION_KEYS: readonly string[] = Object.keys({ + device: true, + osVersion: true, + app: true, + deviceType: true, + project: true, + build: true, + sessionName: true, + stateDir: true, + orientation: true, + geoLocation: true, + timezone: true, + language: true, + locale: true, + appiumVersion: true, +} satisfies Record); + +/** What `testmu()` takes: the device, its OS version, and the app TestMu AI installs on it. */ +export interface TestmuOptions { + /** Device name exactly as TestMu AI's catalog lists it: `Galaxy S22 Ultra 5G`, `iPhone 16`. */ + readonly device: string; + /** OS version exactly as the catalog lists it for that device: `14` on Android, `18.0` on a virtual iOS device, `18` on a real one. */ + readonly osVersion: string; + /** + * The build TestMu AI installs on every session: an `lt://` app id, an + * `https` URL, or a local path, resolved against the project root and + * uploaded when the lease is allocated. + */ + readonly app: string; + /** `'virtual'` (default) for an emulator or simulator, `'real'` for a real device. */ + readonly deviceType?: 'virtual' | 'real' | undefined; + /** Dashboard project the sessions are grouped under. Defaults to `e2e`. */ + readonly project?: string | undefined; + /** Dashboard build the sessions are grouped under. Defaults to the run id. */ + readonly build?: string | undefined; + /** + * Name of every session on the dashboard, with `-` appended when the + * target leases more than one device, so each slot's session has its own. + * Defaults to `e2e---`. Slots count from 1. + */ + readonly sessionName?: string | undefined; + /** + * Directory for the agent-device daemon each run starts, relative to the + * project root. Defaults to `.e2e/testmu`. Each run keeps its own + * directory in it, marked as the provider's, and the first lease of a run + * removes earlier runs' marked directories unchanged for more than a day. + */ + readonly stateDir?: string | undefined; + /** Orientation the device starts in; absent, the device's default. */ + readonly orientation?: 'portrait' | 'landscape' | undefined; + /** Country the device's IP geolocates to, as a code TestMu AI takes: `US`, `FR`. */ + readonly geoLocation?: string | undefined; + /** The device's time zone, as TestMu AI takes it: `UTC+05:30`. */ + readonly timezone?: string | undefined; + /** The device's language, as a language code: `fr`. */ + readonly language?: string | undefined; + /** The device's locale: `fr_FR`. */ + readonly locale?: string | undefined; + /** Appium version TestMu AI starts for the session; absent, its default for the device. */ + readonly appiumVersion?: string | undefined; +} + +type LeaseBackend = 'ios-instance' | 'android-instance'; + +/** The lease scope agent-device resolves a leased device by, as `leases.allocate` granted it. */ +interface LeaseScope { + readonly tenant: string; + readonly runId: string; + readonly leaseId: string; + readonly leaseBackend: LeaseBackend; + readonly leaseProvider: string; +} + +/** A lease's daemon and scope: what releasing it needs. */ +interface LeaseHandle { + readonly stateDir: string; + readonly scope: LeaseScope; + /** What a daemon the call starts authenticates with, when the caller has them. */ + readonly credentials: TestmuCredentials | undefined; +} + +/** + * TestMu AI devices for `mobile({ device: testmu({ device, osVersion, app }) })`: + * one hosted device per worker slot, leased when the run starts and released + * when it ends. Each lease comes from an agent-device daemon the provider + * starts for the run under `stateDir`; allocating it starts the TestMu AI + * session, which installs `app`, so every slot is billed from the moment it + * is leased, whether or not a test runs on it, and releasing the lease ends it. + * An attempt that records video links TestMu AI's recording of the whole + * session, found by the lease's build and session name, and starting when + * the session did. It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's + * environment, and needs an agent-device with the `testmu` provider. + */ +export function testmu(options: TestmuOptions): DeviceProvider { + rejectUnknownKeys('testmu()', options, OPTION_KEYS); + for (const key of ['device', 'osVersion', 'app'] as const) { + const value: unknown = options[key]; + if (typeof value !== 'string' || value.trim() === '') { + throw new ConfigurationError('INVALID_CONFIG', `testmu: \`${key}\` is required, as a non-empty string`); + } + } + // Only an absent value takes the default: `null` from a JavaScript config is refused like any other. + const deviceType = options.deviceType === undefined ? 'virtual' : options.deviceType; + if (!DEVICE_TYPES.has(deviceType)) { + throw new ConfigurationError('INVALID_CONFIG', `testmu: \`deviceType\` must be 'virtual' or 'real', not ${JSON.stringify(deviceType)}`); + } + for (const key of OPTIONAL_STRING_KEYS) { + const value: unknown = options[key]; + if (value !== undefined && (typeof value !== 'string' || value.trim() === '')) { + throw new ConfigurationError('INVALID_CONFIG', `testmu: \`${key}\` must be a non-empty string`); + } + } + if (options.orientation !== undefined && !ORIENTATIONS.has(options.orientation)) { + throw new ConfigurationError('INVALID_CONFIG', `testmu: \`orientation\` must be 'portrait' or 'landscape', not ${JSON.stringify(options.orientation)}`); + } + const deviceFeatures: Record = {}; + for (const [key, leaseKey] of Object.entries(DEVICE_FEATURES)) { + const value = options[key as keyof typeof DEVICE_FEATURES]; + if (value !== undefined) deviceFeatures[leaseKey] = value; + } + const { device, osVersion, app, project, build, sessionName } = options; + /** One release per lease, shared by every caller: the engine's, and `acquire`'s own after a failure. */ + const releases = new Map>(); + /** Stops each held lease's heartbeat, by lease id. */ + const heartbeats = new Map void>(); + let pruning: Promise | undefined; + const release = (id: string, handle: LeaseHandle): Promise => { + heartbeats.get(id)?.(); + heartbeats.delete(id); + let pending = releases.get(id); + if (pending === undefined) { + pending = releaseLease(handle); + releases.set(id, pending); + // A failed release may be tried again. + pending.catch(() => releases.delete(id)); + } + return pending; + }; + return { + name: PROVIDER, + async acquire(request: DeviceRequest): Promise { + if (request.appPath !== undefined) throw new Error("TestMu AI installs the app from `app`; leave the target's `app.appPath` out"); + const credentials = testmuCredentials(request.env); + const baseDir = resolve(request.projectRoot, options.stateDir ?? DEFAULT_STATE_DIR); + const stateDir = join(baseDir, request.runId); + pruning ??= pruneEarlierRuns(baseDir, request.runId); + await pruning; + await markRun(stateDir); + if (request.signal.aborted) throw new Error('cancelled before a lease was allocated'); + const leaseBackend: LeaseBackend = request.platform === 'ios' ? 'ios-instance' : 'android-instance'; + const selectors = { + platform: request.platform, + target: 'mobile' as const, + device, + providerOsVersion: osVersion, + providerApp: appSource(app, request.projectRoot), + providerDeviceType: deviceType, + providerProject: project ?? DEFAULT_PROJECT, + providerBuild: build ?? request.runId, + providerSessionName: slotSessionName(sessionName, request), + ...deviceFeatures, + }; + // Not cancellable: the daemon may grant the lease after an interrupt, and only a lease this returns or releases is ever released. + const granted = await withDaemonCredentials(credentials, () => + createAgentDeviceClient({ stateDir, session: `lease-${request.slot}` }).leases.allocate({ + tenant: PROVIDER, + runId: request.runId, + leaseBackend, + leaseProvider: PROVIDER, + ttlMs: LEASE_TTL_MS, + ...selectors, + }), + ); + const scope: LeaseScope = { tenant: granted.tenantId, runId: granted.runId, leaseId: granted.leaseId, leaseBackend, leaseProvider: PROVIDER }; + heartbeats.set(scope.leaseId, keepAlive({ stateDir, scope, credentials }, request.log)); + try { + if (request.signal.aborted) throw new Error('cancelled'); + request.log(`lease ${scope.leaseId}: ${device}, ${request.platform} ${osVersion} (${deviceType}); session ${selectors.providerSessionName} started`); + // The worker's client is created with these fields: the scope picks the lease, and the selectors match the session it holds. + return { id: scope.leaseId, client: { stateDir, ...scope, ...selectors } }; + } catch (cause) { + // The engine releases only leases `acquire` returned. + const outcome = await release(scope.leaseId, { stateDir, scope, credentials }).then( + () => 'released it', + (releaseCause: unknown) => `releasing it failed (${messageOf(releaseCause)})`, + ); + throw new Error(`lease ${scope.leaseId} was not handed to the run: ${messageOf(cause)}; ${outcome}`, { cause }); + } + }, + async release(lease: DeviceLease, context: DeviceReleaseContext): Promise { + const handle = leaseHandle(lease, optionalTestmuCredentials(context.env)); + if (handle === undefined) throw new Error(`lease ${lease.id} carries no agent-device lease scope to release`); + await release(lease.id, handle); + }, + // Runs in the worker, from the lease alone. TestMu AI records the whole session, so its video starts when the session did. + async record(lease: DeviceLease, context: ProviderRecordContext): Promise { + const calledAt = new Date().toISOString(); + const session = sessionRef(lease); + if (session === undefined) throw new Error(`lease ${lease.id} carries no TestMu AI build and session name to find its recording by`); + const credentials = testmuCredentials(context.env); + const endpoint = testmuApiEndpoint(context.env); + const found = await findSession(endpoint, credentials, session, context.signal); + return { + // Without a usable start time from TestMu AI, the moment recording was asked for is the closest known bound. + startedAt: found.startedAt ?? calledAt, + stop: async ({ signal }) => ({ url: await sessionVideoUrl(endpoint, credentials, found.id, session, signal), mediaType: 'video/mp4' }), + }; + }, + }; +} + +/** Marks a run's directory as the provider's, creating it. Best effort: an unmarked directory is only never pruned. */ +async function markRun(stateDir: string): Promise { + try { + await mkdir(stateDir, { recursive: true }); + await writeFile(join(stateDir, RUN_MARKER), ''); + } catch { + // The daemon reports a state directory it cannot use. + } +} + +/** + * Removes the directories earlier runs left under `baseDir` once nothing in + * them has changed for a day: only directories named after a run id and + * holding the provider's marker, never the current run's. A run still going + * in another process stays, since its heartbeats touch the marker and its + * daemon writes its log. Best effort: a failure leaves them. + */ +async function pruneEarlierRuns(baseDir: string, runId: string): Promise { + const cutoff = Date.now() - RUN_STATE_TTL_MS; + let entries: string[]; + try { + entries = await readdir(baseDir); + } catch { + return; + } + await Promise.all( + entries + .filter((name) => name !== runId && RUN_DIR_NAME.test(name)) + .map(async (name) => { + const dir = join(baseDir, name); + try { + if (!(await lstat(dir)).isDirectory() || !(await lstat(join(dir, RUN_MARKER))).isFile()) return; + if ((await newestChange(dir)) < cutoff) await rm(dir, { recursive: true, force: true }); + } catch { + // Not marked, gone already, or not ours to remove. + } + }), + ); +} + +/** The latest modification time of a directory and of each entry directly in it. */ +async function newestChange(dir: string): Promise { + const times = await Promise.all([dir, ...(await readdir(dir)).map((name) => join(dir, name))].map(async (path) => (await lstat(path)).mtimeMs)); + return Math.max(...times); +} + +/** Releases a lease through the daemon that granted it, which ends its TestMu AI session. A lease the daemon no longer knows counts as released. */ +async function releaseLease({ stateDir, scope, credentials }: LeaseHandle): Promise { + await withDaemonCredentials(credentials, () => createAgentDeviceClient({ stateDir, session: 'release' }).leases.release(scope)); +} + +/** + * Heartbeats a lease until the returned function is called, touching the + * run's marker so pruning never takes a run that is still going. A command + * still running does not keep its lease alive, so without this a lease can + * lapse while the run holds it. A failed heartbeat is logged once and the next one + * tried; it never fails the run. + */ +function keepAlive({ stateDir, scope, credentials }: LeaseHandle, log: (line: string) => void): () => void { + const client = createAgentDeviceClient({ stateDir, session: 'heartbeat' }); + let warned = false; + const marker = join(stateDir, RUN_MARKER); + const timer = setInterval(() => { + const now = new Date(); + utimes(marker, now, now).catch(() => undefined); + withDaemonCredentials(credentials, () => client.leases.heartbeat({ ...scope, ttlMs: LEASE_TTL_MS })).catch((cause: unknown) => { + if (warned) return; + warned = true; + try { + log(`lease ${scope.leaseId}: heartbeat failed (${messageOf(cause)}); agent-device ends the lease after ${LEASE_TTL_MS / 60_000} minutes without one`); + } catch { + // The run's log may be closed by now. + } + }); + }, HEARTBEAT_INTERVAL_MS); + timer.unref(); + return () => clearInterval(timer); +} + +/** + * The dashboard name of a slot's session, unique among the run's slots of the + * target so `record` can find the session by it within its build. + */ +function slotSessionName(sessionName: string | undefined, { runId, targetName, slot, slots }: DeviceRequest): string { + if (sessionName === undefined) return `e2e-${runId}-${targetName}-${slot + 1}`; + return slots > 1 ? `${sessionName}-${slot + 1}` : sessionName; +} + +/** `app` as the daemon reads it: an `lt://` id or URL as written, a local path resolved against the project root, never the daemon's working directory. */ +function appSource(app: string, projectRoot: string): string { + if (isAbsolute(app) || /^[a-z][a-z0-9+.-]*:\/\//i.test(app)) return app; + return resolve(projectRoot, app); +} + +/** The daemon and scope `acquire` put on a lease's `client`, when they are there. */ +function leaseHandle(lease: DeviceLease, credentials: TestmuCredentials | undefined): LeaseHandle | undefined { + const client = lease.client as Record | undefined; + if (client === undefined) return undefined; + const { stateDir, tenant, runId, leaseId, leaseBackend, leaseProvider } = client; + if ( + typeof stateDir !== 'string' || + typeof tenant !== 'string' || + typeof runId !== 'string' || + typeof leaseId !== 'string' || + (leaseBackend !== 'ios-instance' && leaseBackend !== 'android-instance') || + typeof leaseProvider !== 'string' + ) { + return undefined; + } + return { stateDir, scope: { tenant, runId, leaseId, leaseBackend, leaseProvider }, credentials }; +} + +/** The build and session name `acquire` put on a lease's `client`, when they are there. */ +function sessionRef(lease: DeviceLease): SessionRef | undefined { + const client = lease.client as Record | undefined; + const build = client?.['providerBuild']; + const sessionName = client?.['providerSessionName']; + if (typeof build !== 'string' || build === '' || typeof sessionName !== 'string' || sessionName === '') return undefined; + return { build, sessionName }; +} + +function messageOf(cause: unknown): string { + return cause instanceof Error ? cause.message : String(cause); +} diff --git a/packages/testmu/src/sessions.ts b/packages/testmu/src/sessions.ts new file mode 100644 index 000000000..bbadd53f6 --- /dev/null +++ b/packages/testmu/src/sessions.ts @@ -0,0 +1,146 @@ +/** + * TestMu AI's mobile automation sessions API over `fetch`, the one + * agent-device's `testmu` provider reads session artifacts from: finds a + * session by its build and name, with its start time, and reads the URL of + * its video. + */ + +import { ConfigurationError } from 'e2e/engine'; +import type { TestmuCredentials } from './credentials.ts'; + +/** The API's base, as agent-device's `TESTMU_API_ENDPOINT` sets it. */ +const DEFAULT_API_ENDPOINT = 'https://mobile-api.lambdatest.com/mobile-automation/api/v1'; + +const REQUEST_TIMEOUT_MS = 15_000; + +/** Sessions the list asks for at once. */ +const PAGE_SIZE = 50; + +/** Pages the lookup reads before it gives up on a build: the list is newest first, so the session is near the top. */ +const MAX_PAGES = 10; + +/** A session the recording covers, as the lease named it. */ +export interface SessionRef { + readonly build: string; + readonly sessionName: string; +} + +/** + * The sessions API at `TESTMU_API_ENDPOINT` from the run's environment, else + * TestMu AI's own. Throws, without repeating the value, when the override is + * not an http(s) URL or carries a username or password, which a failed + * request would otherwise put in its error message. + */ +export function testmuApiEndpoint(env: Readonly>): string { + const override = env['TESTMU_API_ENDPOINT']?.trim(); + if (override === undefined || override === '') return DEFAULT_API_ENDPOINT; + if (!URL.canParse(override) || !/^https?:$/.test(new URL(override).protocol)) throw new ConfigurationError('INVALID_CONFIG', 'TESTMU_API_ENDPOINT is not an http(s) URL'); + const url = new URL(override); + if (url.username !== '' || url.password !== '') { + throw new ConfigurationError('INVALID_CONFIG', 'TESTMU_API_ENDPOINT must not carry a username or password; set LT_USERNAME and LT_ACCESS_KEY instead'); + } + return override.replace(/\/+$/, ''); +} + +/** A session the list found: its `test_id`, and when it started, if the list says. */ +export interface FoundSession { + readonly id: string; + /** ISO timestamp in UTC; `undefined` when the row has no start time or one that does not parse. */ + readonly startedAt: string | undefined; +} + +/** + * The newest session named `sessionName` in `build` that the credentials' + * user started, reading the list a page at a time. Every request is bounded by `REQUEST_TIMEOUT_MS` and `signal`, + * and errors name the build and the session, never the credentials or a URL. + */ +export async function findSession(endpoint: string, credentials: TestmuCredentials, { build, sessionName }: SessionRef, signal: AbortSignal): Promise { + const auth = basicAuth(credentials); + const what = `TestMu AI session lookup for build ${JSON.stringify(build)}`; + for (let page = 0; page < MAX_PAGES; page += 1) { + const url = new URL(`${endpoint}/sessions`); + url.searchParams.set('build', build); + // The build filter spans the whole organization; a build name another user's run shares would find their session. + url.searchParams.set('username', credentials.username); + url.searchParams.set('limit', String(PAGE_SIZE)); + url.searchParams.set('offset', String(page * PAGE_SIZE)); + const body = await getJson(url, auth, signal, what); + // An empty page comes back as `data: null`. + const rows = body['data'] ?? []; + if (!Array.isArray(rows)) throw new Error(`${what} failed: no session list in the response`); + for (const row of rows) { + const { name, test_id: id, start_timestamp: start } = asRecord(row) ?? {}; + if (name === sessionName && typeof id === 'string' && id !== '') return { id, startedAt: utcTimestamp(start) }; + } + if (rows.length < PAGE_SIZE) break; + } + throw new Error(`TestMu AI has no session named ${JSON.stringify(sessionName)} in build ${JSON.stringify(build)}`); +} + +/** The `video_url` of session `id`'s details, bounded and named as `findSession` is. */ +export async function sessionVideoUrl(endpoint: string, credentials: TestmuCredentials, id: string, { build, sessionName }: SessionRef, signal: AbortSignal): Promise { + const what = `TestMu AI session ${id} (${JSON.stringify(sessionName)}, build ${JSON.stringify(build)})`; + const body = await getJson(new URL(`${endpoint}/sessions/${encodeURIComponent(id)}`), basicAuth(credentials), signal, `${what} details`); + const details = asRecord(body['data']); + const videoUrl = details?.['video_url']; + if (typeof videoUrl !== 'string' || !isHttpUrl(videoUrl)) throw new Error(`${what} reports no video URL`); + return videoUrl; +} + +const TIMESTAMP = /^(\d{4}-\d{2}-\d{2})[T ](\d{2}:\d{2}:\d{2}(?:\.\d+)?)(Z|[+-]\d{2}:\d{2})?$/; + +/** + * A `start_timestamp` as an ISO timestamp in UTC. The API formats the + * database time as RFC 3339 with its zone; a time without one is read as UTC, + * the zone the API's database driver reads it in. + */ +function utcTimestamp(value: unknown): string | undefined { + if (typeof value !== 'string') return undefined; + const match = TIMESTAMP.exec(value.trim()); + if (match === null) return undefined; + const [, date, time, zone] = match; + const ms = Date.parse(`${date}T${time}${zone ?? 'Z'}`); + return Number.isNaN(ms) ? undefined : new Date(ms).toISOString(); +} + +function basicAuth({ username, accessKey }: TestmuCredentials): string { + return `Basic ${Buffer.from(`${username}:${accessKey}`).toString('base64')}`; +} + +/** One authenticated GET answered with a JSON object; anything else throws, as `what`, with the HTTP status and the API's message. */ +async function getJson(url: URL, auth: string, signal: AbortSignal, what: string): Promise> { + const timeout = new AbortController(); + const timer = setTimeout(() => timeout.abort(), REQUEST_TIMEOUT_MS); + let response: Response; + let text: string; + try { + response = await fetch(url, { headers: { Authorization: auth, Accept: 'application/json' }, signal: AbortSignal.any([signal, timeout.signal]) }); + text = await response.text(); + } catch (cause) { + if (signal.aborted) throw new Error(`${what} cancelled`, { cause }); + if (timeout.signal.aborted) throw new Error(`${what} got no answer within ${REQUEST_TIMEOUT_MS / 1000} s`, { cause }); + throw new Error(`${what} failed: ${cause instanceof Error ? cause.message : String(cause)}`, { cause }); + } finally { + clearTimeout(timer); + } + let body: Record | undefined; + try { + body = asRecord(JSON.parse(text)); + } catch { + body = undefined; + } + if (body === undefined) throw new Error(`${what} failed: HTTP ${response.status}, not JSON`); + if (!response.ok) { + const message = body['message']; + throw new Error(`${what} failed: HTTP ${response.status}${typeof message === 'string' && message !== '' ? ` (${message.slice(0, 200)})` : ''}`); + } + return body; +} + +function asRecord(value: unknown): Record | undefined { + return typeof value === 'object' && value !== null && !Array.isArray(value) ? (value as Record) : undefined; +} + +function isHttpUrl(value: string): boolean { + return URL.canParse(value) && /^https?:$/.test(new URL(value).protocol); +} diff --git a/packages/testmu/src/web.ts b/packages/testmu/src/web.ts new file mode 100644 index 000000000..07ae2ccca --- /dev/null +++ b/packages/testmu/src/web.ts @@ -0,0 +1,8 @@ +/** + * `@e2e-dev/testmu/web`: `testmuBrowsers()`, for web targets. It loads neither + * `@e2e-dev/mobile` nor agent-device, and the package root, which exports the + * device provider, does not reference `@e2e-dev/web`. + */ + +export { testmuBrowsers } from './browsers.ts'; +export type { TestmuBrowserName, TestmuBrowsersOptions, TestmuBrowsersRoute } from './browsers.ts'; diff --git a/packages/testmu/tests/unit/browsers.test.ts b/packages/testmu/tests/unit/browsers.test.ts new file mode 100644 index 000000000..b6a3e52ef --- /dev/null +++ b/packages/testmu/tests/unit/browsers.test.ts @@ -0,0 +1,212 @@ +/** + * `testmuBrowsers()` builds the CDP URL a TestMu AI session starts on: the + * capabilities it encodes, credentials read from the run's environment + * only, session and build names, the route and hub options, scope, option + * validation, log lines that never carry the access key, and a release that + * calls nothing. + */ + +import type { BrowserReleaseContext, BrowserRequest } from '@e2e-dev/web'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { testmuBrowsers, type TestmuBrowsersOptions } from '../../src/web.ts'; + +const ACCESS_KEY = 'LT_secret-access-key'; + +interface Capabilities { + browserName: string; + browserVersion: string; + 'LT:Options': Record; +} + +function request(overrides: Partial = {}): BrowserRequest & { lines: string[] } { + const lines: string[] = []; + return { + runId: 'run-1', + targetName: 'chromium', + slot: 0, + slots: 2, + env: { LT_USERNAME: 'alice', LT_ACCESS_KEY: ACCESS_KEY }, + signal: new AbortController().signal, + log: (line: string) => lines.push(line), + lines, + ...overrides, + }; +} + +function decode(cdpEndpoint: string): { url: URL; capabilities: Capabilities } { + const url = new URL(cdpEndpoint); + return { url, capabilities: JSON.parse(url.searchParams.get('capabilities') ?? '{}') as Capabilities }; +} + +beforeEach(() => { + vi.stubGlobal('fetch', () => { + throw new Error('testmuBrowsers() must not call the network'); + }); +}); + +afterEach(() => { + vi.unstubAllGlobals(); + vi.unstubAllEnvs(); +}); + +describe('testmuBrowsers()', () => { + it('builds a /puppeteer CDP URL on cdp.lambdatest.com with the run and target in the names', async () => { + const lease = await testmuBrowsers().acquire(request()); + const { url, capabilities } = decode(lease.cdpEndpoint); + + expect(url.protocol).toBe('wss:'); + expect(url.host).toBe('cdp.lambdatest.com'); + expect(url.pathname).toBe('/puppeteer'); + expect(capabilities).toEqual({ + browserName: 'Chrome', + browserVersion: 'latest', + 'LT:Options': { + idleTimeout: 600, + platform: 'Windows 11', + project: 'e2e', + build: 'run-1', + name: 'e2e-run-1-chromium-1', + user: 'alice', + accessKey: ACCESS_KEY, + }, + }); + expect(lease.id).toBe('chromium:slot 1 of 2'); + }); + + it('names a per-attempt session after the attempt', async () => { + const provider = testmuBrowsers({ scope: 'attempt' }); + const lease = await provider.acquire(request({ attemptId: 'attempt-7' })); + + expect(provider.scope).toBe('attempt'); + expect(decode(lease.cdpEndpoint).capabilities['LT:Options']['name']).toBe('e2e-run-1-chromium-attempt-7'); + expect(lease.id).toBe('chromium:attempt-7'); + }); + + it('leaves scope to the engine default when none is given', () => { + expect('scope' in testmuBrowsers()).toBe(false); + }); + + it('honours the route, hub, browser, platform, build, and extra capabilities', async () => { + const lease = await testmuBrowsers({ + route: '/playwright-cdp', + hub: 'cdp.eu.example.test', + browserName: 'MicrosoftEdge', + browserVersion: '140', + platform: 'macOS Sequoia', + build: 'nightly', + project: 'checkout', + geoLocation: 'FR', + timezone: 'UTC+01:00', + capabilities: { video: true, idleTimeout: 300 }, + }).acquire(request()); + const { url, capabilities } = decode(lease.cdpEndpoint); + + expect(url.host).toBe('cdp.eu.example.test'); + expect(url.pathname).toBe('/playwright-cdp'); + expect(capabilities.browserName).toBe('MicrosoftEdge'); + expect(capabilities.browserVersion).toBe('140'); + expect(capabilities['LT:Options']).toMatchObject({ + platform: 'macOS Sequoia', + project: 'checkout', + build: 'nightly', + geoLocation: 'FR', + timezone: 'UTC+01:00', + video: true, + idleTimeout: 300, + }); + }); + + it('names sessions like testmu() does: a given sessionName gets the slot or attempt appended when needed', async () => { + const nameOf = async (options: TestmuBrowsersOptions, overrides: Partial = {}) => + decode((await testmuBrowsers(options).acquire(request(overrides))).cdpEndpoint).capabilities['LT:Options']['name']; + + expect(await nameOf({ sessionName: 'smoke' }, { slots: 1 })).toBe('smoke'); + expect(await nameOf({ sessionName: 'smoke' }, { slot: 1, slots: 2 })).toBe('smoke-2'); + expect(await nameOf({ sessionName: 'smoke', scope: 'attempt' }, { attemptId: 'attempt-3' })).toBe('smoke-attempt-3'); + expect(await nameOf({}, { slot: 1, slots: 2 })).toBe('e2e-run-1-chromium-2'); + }); + + it('takes the credentials from the run environment, not process.env', async () => { + vi.stubEnv('LT_USERNAME', 'from-process-env'); + vi.stubEnv('LT_ACCESS_KEY', 'from-process-env'); + const options = decode((await testmuBrowsers().acquire(request())).cdpEndpoint).capabilities['LT:Options']; + + expect(options['user']).toBe('alice'); + expect(options['accessKey']).toBe(ACCESS_KEY); + }); + + it('fails the lease when a credential is missing or blank', async () => { + await expect(testmuBrowsers().acquire(request({ env: { LT_USERNAME: 'alice' } }))).rejects.toThrow('LT_ACCESS_KEY is not set'); + await expect(testmuBrowsers().acquire(request({ env: { LT_USERNAME: ' ', LT_ACCESS_KEY: ACCESS_KEY } }))).rejects.toThrow('LT_USERNAME is not set'); + }); + + it.each([ + [{ platfrom: 'macOS Sequoia' }, 'testmuBrowsers() has unknown key "platfrom"; did you mean "platform"?'], + [{ route: 'playwright-cdp' }, '`route` must be one of /puppeteer, /playwright-cdp'], + [{ hub: 'https://cdp.lambdatest.com' }, '`hub` must be a host'], + [{ hub: 'cdp.lambdatest.com/puppeteer' }, '`hub` must be a host'], + [{ browserName: 'Firefox' }, '`browserName` must be one of Chrome, MicrosoftEdge'], + [{ capabilities: { user: 'mallory', accessKey: 'spoofed' } }, '`capabilities` cannot set `user`, `accessKey`'], + [{ capabilities: { build: 'nightly', platform: 'Windows 10' } }, '`capabilities` cannot set `build`, `platform`'], + [{ capabilities: { browserName: 'Firefox', browserVersion: '120' } }, '`capabilities` cannot set `browserName`, `browserVersion`'], + [{ capabilities: { 'LT:Options': { video: true } } }, '`capabilities` cannot set `LT:Options`'], + [{ hub: null }, '`hub` must be a host'], + [{ route: null }, '`route` must be one of'], + [{ browserName: null }, '`browserName` must be one of'], + [{ scope: 'session' }, '`scope` must be one of worker, attempt'], + [{ scope: null }, '`scope` must be one of worker, attempt'], + [{ build: '' }, '`build` must be a non-empty string'], + [{ project: 1 }, '`project` must be a non-empty string'], + [{ sessionName: null }, '`sessionName` must be a non-empty string'], + [{ browserVersion: ' ' }, '`browserVersion` must be a non-empty string'], + [{ capabilities: { project: 'x', geoLocation: 'US' } }, '`capabilities` cannot set `project`, `geoLocation`'], + [null, 'testmuBrowsers() options must be an object'], + [[], 'testmuBrowsers() options must be an object'], + [{ capabilities: null }, '`capabilities` must be an object'], + [{ capabilities: ['video'] }, '`capabilities` must be an object'], + [{ capabilities: 'video' }, '`capabilities` must be an object'], + ])('rejects %j when the provider is created', (options, message) => { + expect(() => testmuBrowsers(options as unknown as TestmuBrowsersOptions)).toThrow( + expect.objectContaining({ code: 'INVALID_CONFIG', message: expect.stringContaining(message) }), + ); + }); + + it('keeps 24 worker-scope leases inside the engine\'s 16 KB hand-off', async () => { + const provider = testmuBrowsers({ capabilities: { video: true, network: true, console: true, tunnel: true, tunnelName: 'ci-tunnel' } }); + const runId = '01a0fcab-e8da-797a-8b35-2168b81385c0'; + const env = { LT_USERNAME: 'some.user.name', LT_ACCESS_KEY: `LT_${'x'.repeat(46)}` }; + const leases = await Promise.all( + Array.from({ length: 24 }, (_, slot) => provider.acquire(request({ runId, targetName: 'checkout-web', slot, slots: 24, env }))), + ); + + expect(Buffer.byteLength(JSON.stringify({ slots: 24, leases }))).toBeLessThan(16 * 1024); + }); + + it('logs the session and build names but never the access key or the URL', async () => { + const req = request(); + await testmuBrowsers().acquire(req); + + expect(req.lines).toEqual(['TestMu AI session "e2e-run-1-chromium-1" in build "run-1"']); + expect(req.lines.join('\n')).not.toContain(ACCESS_KEY); + expect(req.lines.join('\n')).not.toContain('wss://'); + }); + + it('is exported from @e2e-dev/testmu/web only: the package root stays device-only', async () => { + const root: Record = await import('../../src/index.ts'); + expect(Object.keys(root)).toEqual(['testmu']); + }); + + it('releases without calling anything', async () => { + const provider = testmuBrowsers(); + const lease = await provider.acquire(request()); + const context: BrowserReleaseContext = { + runId: 'run-1', + targetName: 'chromium', + env: {}, + signal: new AbortController().signal, + log: () => {}, + }; + + await expect(provider.release(lease, context)).resolves.toBeUndefined(); + }); +}); diff --git a/packages/testmu/tests/unit/prune.test.ts b/packages/testmu/tests/unit/prune.test.ts new file mode 100644 index 000000000..e91cbeebd --- /dev/null +++ b/packages/testmu/tests/unit/prune.test.ts @@ -0,0 +1,137 @@ +/** + * `testmu()` pruning the state directories earlier runs left under + * `stateDir`: once per provider, only directories it marked as its own whose + * newest file is older than a day, never the current run's, and never + * failing the lease; and the marker it writes and keeps fresh. + */ + +import { existsSync, mkdirSync, mkdtempSync, readdirSync, rmSync, statSync, utimesSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import type { DeviceRequest } from '@e2e-dev/mobile'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { testmu, type TestmuOptions } from '../../src/index.ts'; + +vi.mock('agent-device', () => ({ + createAgentDeviceClient: () => ({ + leases: { + allocate: async (options: Record) => ({ leaseId: 'lease-1', tenantId: options['tenant'], runId: options['runId'] }), + heartbeat: async () => ({}), + release: async () => ({ released: true }), + }, + }), +})); + +const options: TestmuOptions = { device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: 'lt://APP1', stateDir: 'state' }; +const CURRENT_RUN = '01a0fc76-002f-736c-991a-fa4778d0543e'; +const OLD_RUN = '01a0e000-0000-7000-8000-000000000001'; +const RECENT_RUN = '01a0e000-0000-7000-8000-000000000002'; +/** A file named like a run: only directories are pruned. */ +const RUN_FILE = '01a0e000-0000-7000-8000-000000000003'; +/** A directory named like a run that another tool keeps under the same directory. */ +const FOREIGN_RUN = '01a0e000-0000-7000-8000-000000000004'; +/** A marked run, still going in another process, whose daemon wrote its log recently. */ +const LONG_RUN = '01a0e000-0000-7000-8000-000000000005'; +const MARKER = '.e2e-testmu-run'; +const HOUR = 60 * 60_000; +const MINUTE = 60_000; + +let root: string; +let base: string; + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'testmu-prune-')); + base = join(root, 'state'); + mkdirSync(base); +}); + +afterEach(() => { + rmSync(root, { recursive: true, force: true }); +}); + +/** Sets a path's modification time `ageMs` ago. */ +function age(path: string, ageMs: number): void { + const time = new Date(Date.now() - ageMs); + utimesSync(path, time, time); +} + +/** A run directory under the state directory with a daemon log, marked as the provider's unless `marked` is false, every entry `ageMs` old. */ +function runDir(name: string, ageMs: number, { marked = true } = {}): void { + const dir = join(base, name); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, 'daemon.log'), 'log'); + age(join(dir, 'daemon.log'), ageMs); + if (marked) { + writeFileSync(join(dir, MARKER), ''); + age(join(dir, MARKER), ageMs); + } + age(dir, ageMs); +} + +function request(overrides: Partial = {}): DeviceRequest { + return { + platform: 'android', + runId: CURRENT_RUN, + targetName: 'android', + slot: 0, + slots: 1, + projectRoot: root, + agentDeviceVersion: '0.21.18', + env: { LT_USERNAME: 'ada', LT_ACCESS_KEY: 'lt-key' }, + signal: new AbortController().signal, + log: () => undefined, + ...overrides, + }; +} + +describe('testmu() state directory pruning', () => { + it("removes earlier runs' marked directories older than a day, and keeps the current run's, recent ones, and anything it did not mark", async () => { + runDir(OLD_RUN, 25 * HOUR); + runDir(RECENT_RUN, 23 * HOUR); + runDir(CURRENT_RUN, 48 * HOUR); + runDir(FOREIGN_RUN, 48 * HOUR, { marked: false }); + runDir('notes', 48 * HOUR); + writeFileSync(join(base, RUN_FILE), 'a file'); + age(join(base, RUN_FILE), 48 * HOUR); + await testmu(options).acquire(request()); + expect(readdirSync(base).toSorted()).toEqual([CURRENT_RUN, FOREIGN_RUN, RECENT_RUN, RUN_FILE, 'notes'].toSorted()); + }); + + it('keeps a marked run whose daemon wrote a file within the day, however old its marker and directory', async () => { + runDir(LONG_RUN, 30 * HOUR); + age(join(base, LONG_RUN, 'daemon.log'), HOUR); + age(join(base, LONG_RUN), 30 * HOUR); + await testmu(options).acquire(request()); + expect(existsSync(join(base, LONG_RUN))).toBe(true); + }); + + it("marks the run's directory when it leases, and touches the marker on every heartbeat", async () => { + vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] }); + try { + const provider = testmu(options); + const lease = await provider.acquire(request()); + const marker = join(base, CURRENT_RUN, MARKER); + expect(statSync(marker).isFile()).toBe(true); + age(marker, 30 * HOUR); + await vi.advanceTimersByTimeAsync(2 * MINUTE); + await vi.waitFor(() => expect(Date.now() - statSync(marker).mtimeMs).toBeLessThan(HOUR)); + await provider.release(lease, { runId: CURRENT_RUN, targetName: 'android', env: {}, signal: new AbortController().signal, log: () => undefined }); + } finally { + vi.useRealTimers(); + } + }); + + it('prunes once per provider, on its first acquire', async () => { + const provider = testmu(options); + await provider.acquire(request()); + runDir(OLD_RUN, 25 * HOUR); + await provider.acquire(request({ slot: 1, slots: 2 })); + expect(existsSync(join(base, OLD_RUN))).toBe(true); + }); + + it('leases a device when there is nothing to prune or pruning fails', async () => { + await expect(testmu({ ...options, stateDir: 'missing' }).acquire(request())).resolves.toMatchObject({ id: 'lease-1' }); + writeFileSync(join(root, 'plain-file'), 'not a directory'); + await expect(testmu({ ...options, stateDir: 'plain-file' }).acquire(request())).resolves.toMatchObject({ id: 'lease-1' }); + }); +}); diff --git a/packages/testmu/tests/unit/recording.test.ts b/packages/testmu/tests/unit/recording.test.ts new file mode 100644 index 000000000..fb96b1ff3 --- /dev/null +++ b/packages/testmu/tests/unit/recording.test.ts @@ -0,0 +1,223 @@ +/** + * `testmu().record()` against a fake TestMu AI sessions API behind a stubbed + * `fetch`: the session it finds by build and name, the start time it takes + * from it, the video it links, the endpoint override, and the failures it + * names without leaking credentials or signed URLs. + */ + +import type { DeviceLease } from '@e2e-dev/mobile'; +import type { ProviderRecordContext } from 'e2e/engine'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { testmu, type TestmuOptions } from '../../src/index.ts'; + +vi.mock('agent-device', () => ({ + createAgentDeviceClient: () => { + throw new Error('recording starts no daemon'); + }, +})); + +const API = 'https://mobile-api.lambdatest.com/mobile-automation/api/v1'; +const VIDEO = 'https://videos.example.com/orgId-1/T2/video/video.mp4?X-Amz-Signature=secret-signature'; +const options: TestmuOptions = { device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: 'lt://APP1' }; +const env = { LT_USERNAME: 'ada', LT_ACCESS_KEY: 'lt-key' }; +const lease: DeviceLease = { + id: 'lease-1', + client: { stateDir: '/work/.e2e/testmu/run-1', tenant: 'testmu', runId: 'run-1', leaseId: 'lease-1', providerBuild: 'run-1', providerSessionName: 'e2e-run-1-android-2' }, +}; + +interface Call { + readonly url: URL; + readonly authorization: string | undefined; +} + +const api = { + calls: [] as Call[], + /** Every session the build holds, newest first, as the list returns them. */ + sessions: [] as Record[], + /** Each session's details by `test_id`. */ + details: {} as Record>, + /** A response the next request gets instead of the fake's own. */ + override: undefined as (() => Response | Promise) | undefined, +}; + +beforeEach(() => { + Object.assign(api, { + calls: [], + sessions: [ + { test_id: 'T4', name: 'e2e-run-1-android-2', build_name: 'run-1', username: 'grace', start_timestamp: '2026-10-02T09:59:00Z' }, + { test_id: 'T3', name: 'e2e-run-1-android-1', build_name: 'run-1', username: 'ada', start_timestamp: '2026-10-02T09:58:05Z' }, + { test_id: 'T2', name: 'e2e-run-1-android-2', build_name: 'run-1', username: 'ada', start_timestamp: '2026-10-02T09:58:07Z' }, + { test_id: 'T1', name: 'e2e-run-1-android-2', build_name: 'run-1', username: 'ada', start_timestamp: '2026-10-01T08:00:00Z' }, + ], + details: { T2: { test_id: 'T2', name: 'e2e-run-1-android-2', video_url: VIDEO }, T1: { test_id: 'T1', video_url: 'https://videos.example.com/old.mp4' } }, + override: undefined, + }); + vi.stubGlobal('fetch', async (input: string | URL, init: RequestInit = {}) => { + const url = new URL(input); + api.calls.push({ url, authorization: (init.headers as Record | undefined)?.['Authorization'] }); + init.signal?.throwIfAborted(); + const override = api.override; + if (override !== undefined) { + api.override = undefined; + return override(); + } + const base = new URL(API).pathname; + if (url.pathname === `${base}/sessions`) { + const build = url.searchParams.get('build'); + const username = url.searchParams.get('username'); + const limit = Number(url.searchParams.get('limit') ?? '10'); + const offset = Number(url.searchParams.get('offset') ?? '0'); + const rows = api.sessions.filter((row) => row['build_name'] === build && (username === null || row['username'] === username)).slice(offset, offset + limit); + // An empty page is Go's nil slice: `data: null`. + return Response.json({ status: 'success', data: rows.length === 0 ? null : rows, message: 'Retrieve session list was successful', Meta: { result_set: { count: rows.length } } }); + } + const id = decodeURIComponent(url.pathname.slice(`${base}/sessions/`.length)); + const details = api.details[id]; + if (details === undefined) return Response.json({ status: 'fail', message: 'Not found' }, { status: 404 }); + return Response.json({ status: 'success', data: details, message: 'Retrieve session was successful' }); + }); +}); + +afterEach(() => { + vi.unstubAllGlobals(); + vi.useRealTimers(); +}); + +function context(overrides: Partial = {}): ProviderRecordContext { + return { runId: 'run-1', targetName: 'android', attemptId: 'attempt-1', env, signal: new AbortController().signal, ...overrides }; +} + +const stopContext = (signal = new AbortController().signal) => ({ dir: '/work/.e2e/artifacts/attempt-1/video', signal }); + +/** The error `promise` rejects with; fails the test when it resolves. */ +async function failure(promise: Promise): Promise { + return promise.then( + () => { + throw new Error('expected a failure'); + }, + (cause: unknown) => cause as Error, + ); +} + +async function record(recordLease: DeviceLease = lease, recordContext: ProviderRecordContext = context()) { + const provider = testmu(options); + if (provider.record === undefined) throw new Error('testmu() records nothing'); + return provider.record(recordLease, recordContext); +} + +describe('testmu().record()', () => { + it("starts at the newest session's start time, found by the slot's name in the lease's build among the user's own, and links its video when it stops", async () => { + const recording = await record(); + expect(recording.startedAt).toBe('2026-10-02T09:58:07.000Z'); + expect(api.calls.map((call) => call.url.href)).toEqual([`${API}/sessions?build=run-1&username=ada&limit=50&offset=0`]); + await expect(recording.stop(stopContext())).resolves.toEqual({ url: VIDEO, mediaType: 'video/mp4' }); + expect(api.calls.map((call) => call.url.href)).toEqual([`${API}/sessions?build=run-1&username=ada&limit=50&offset=0`, `${API}/sessions/T2`]); + expect(api.calls.map((call) => call.authorization)).toEqual([`Basic ${Buffer.from('ada:lt-key').toString('base64')}`, `Basic ${Buffer.from('ada:lt-key').toString('base64')}`]); + }); + + it.each([ + ['2026-10-02T09:58:07.25+05:30', '2026-10-02T04:28:07.250Z'], + ['2026-10-02 09:58:07', '2026-10-02T09:58:07.000Z'], + ])('reads the start time %s as %s, a time without a zone as UTC', async (start, iso) => { + api.sessions[2]!['start_timestamp'] = start; + expect((await record()).startedAt).toBe(iso); + }); + + it.each([undefined, null, '', 'yesterday', '2026-13-45T99:00:00Z'])('falls back to the moment it is called without a usable start time (%j)', async (start) => { + vi.useFakeTimers({ toFake: ['Date'] }); + vi.setSystemTime(new Date('2026-10-02T10:00:00.000Z')); + if (start === undefined) delete api.sessions[2]!['start_timestamp']; + else api.sessions[2]!['start_timestamp'] = start; + expect((await record()).startedAt).toBe('2026-10-02T10:00:00.000Z'); + }); + + it('reads the lease after a round trip through JSON, as the worker gets it', async () => { + const recording = await record(JSON.parse(JSON.stringify(lease)) as DeviceLease); + await expect(recording.stop(stopContext())).resolves.toMatchObject({ url: VIDEO }); + }); + + it('pages through a build holding more sessions than one page', async () => { + api.sessions = [...Array.from({ length: 50 }, (_, index) => ({ test_id: `X${index}`, name: `other-${index}`, build_name: 'run-1', username: 'ada' })), ...api.sessions]; + await expect((await record()).stop(stopContext())).resolves.toMatchObject({ url: VIDEO }); + expect(api.calls.map((call) => call.url.searchParams.get('offset'))).toEqual(['0', '50', null]); + }); + + it('asks the API TESTMU_API_ENDPOINT names', async () => { + const recording = await record(lease, context({ env: { ...env, TESTMU_API_ENDPOINT: 'https://stage-mobile-api.lambdatest.com/mobile-automation/api/v1/' } })); + await expect(recording.stop(stopContext())).resolves.toMatchObject({ url: VIDEO }); + expect(api.calls.map((call) => call.url.href)).toEqual([ + 'https://stage-mobile-api.lambdatest.com/mobile-automation/api/v1/sessions?build=run-1&username=ada&limit=50&offset=0', + 'https://stage-mobile-api.lambdatest.com/mobile-automation/api/v1/sessions/T2', + ]); + }); + + it('fails to start, naming the build and the session, when the build has no session by that name', async () => { + api.sessions = api.sessions.filter((row) => row['name'] !== 'e2e-run-1-android-2'); + await expect(record()).rejects.toThrow('TestMu AI has no session named "e2e-run-1-android-2" in build "run-1"'); + }); + + it('names the session when its details carry no video URL', async () => { + api.details['T2'] = { test_id: 'T2', video_url: 'not a url' }; + await expect((await record()).stop(stopContext())).rejects.toThrow('TestMu AI session T2 ("e2e-run-1-android-2", build "run-1") reports no video URL'); + delete api.details['T2']!['video_url']; + await expect((await record()).stop(stopContext())).rejects.toThrow('reports no video URL'); + }); + + it('reports an HTTP failure with its status and message, never the credentials or the request URL', async () => { + api.override = () => Response.json({ status: 'fail', message: 'Unauthorized' }, { status: 401 }); + const error = await failure(record()); + expect(error.message).toBe('TestMu AI session lookup for build "run-1" failed: HTTP 401 (Unauthorized)'); + expect(JSON.stringify(error, Object.getOwnPropertyNames(error))).not.toContain('lt-key'); + }); + + it('names the session when its details request fails', async () => { + api.details = {}; + await expect((await record()).stop(stopContext())).rejects.toThrow('TestMu AI session T2 ("e2e-run-1-android-2", build "run-1") details failed: HTTP 404 (Not found)'); + }); + + it('reports a response that is not JSON, or not a session list', async () => { + api.override = () => new Response('Bad gateway', { status: 502 }); + await expect(record()).rejects.toThrow('TestMu AI session lookup for build "run-1" failed: HTTP 502, not JSON'); + api.override = () => Response.json({ status: 'success', data: { sessions: [] } }); + await expect(record()).rejects.toThrow('TestMu AI session lookup for build "run-1" failed: no session list in the response'); + }); + + it('gives up on a request TestMu AI does not answer within 15 seconds', async () => { + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }); + vi.stubGlobal('fetch', (input: string | URL, init: RequestInit = {}) => { + api.calls.push({ url: new URL(input), authorization: undefined }); + return new Promise((_, reject) => init.signal?.addEventListener('abort', () => reject(init.signal?.reason))); + }); + const started = failure(record()); + await vi.advanceTimersByTimeAsync(15_000); + expect((await started).message).toBe('TestMu AI session lookup for build "run-1" got no answer within 15 s'); + }); + + it('stops asking once the attempt or the stop is cancelled', async () => { + const controller = new AbortController(); + controller.abort(); + await expect(record(lease, context({ signal: controller.signal }))).rejects.toThrow('TestMu AI session lookup for build "run-1" cancelled'); + await expect((await record()).stop(stopContext(controller.signal))).rejects.toThrow('TestMu AI session T2 ("e2e-run-1-android-2", build "run-1") details cancelled'); + }); + + it('refuses to start for a lease without a build and a session name', async () => { + await expect(record({ id: 'other', client: { stateDir: '/tmp/x', providerBuild: 'run-1' } })).rejects.toThrow( + 'lease other carries no TestMu AI build and session name to find its recording by', + ); + }); + + it('refuses to start without credentials or with an endpoint that is not a URL', async () => { + await expect(record(lease, context({ env: {} }))).rejects.toThrow('LT_USERNAME and LT_ACCESS_KEY are not set'); + await expect(record(lease, context({ env: { ...env, TESTMU_API_ENDPOINT: 'mobile-api' } }))).rejects.toThrow('TESTMU_API_ENDPOINT is not an http(s) URL'); + expect(api.calls).toEqual([]); + }); + + it('refuses an endpoint carrying credentials, without repeating them', async () => { + for (const endpoint of ['https://ada:lt-key@mobile-api.lambdatest.com/mobile-automation/api/v1', 'https://lt-key@mobile-api.lambdatest.com']) { + const error = await failure(record(lease, context({ env: { ...env, TESTMU_API_ENDPOINT: endpoint } }))); + expect(error).toMatchObject({ code: 'INVALID_CONFIG', message: 'TESTMU_API_ENDPOINT must not carry a username or password; set LT_USERNAME and LT_ACCESS_KEY instead' }); + expect(JSON.stringify(error, Object.getOwnPropertyNames(error))).not.toContain('lt-key'); + } + expect(api.calls).toEqual([]); + }); +}); diff --git a/packages/testmu/tests/unit/testmu.test.ts b/packages/testmu/tests/unit/testmu.test.ts new file mode 100644 index 000000000..008f82af8 --- /dev/null +++ b/packages/testmu/tests/unit/testmu.test.ts @@ -0,0 +1,513 @@ +/** + * `testmu()` against a stubbed agent-device client: the options it refuses, + * the lease it allocates and hands the worker, the credentials it reads and + * shares with the daemon, the heartbeat that keeps it alive, and the release + * on every exit path. `recording.test.ts` covers `record`. + */ + +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import type { DeviceLease, DeviceReleaseContext, DeviceRequest } from '@e2e-dev/mobile'; +import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { testmu, type TestmuOptions } from '../../src/index.ts'; + +interface ClientCall { + readonly config: Record; + readonly operation: 'allocate' | 'heartbeat' | 'release'; + readonly options: Record; +} + +const daemon = { + calls: [] as ClientCall[], + /** Runs inside `allocate`, before it answers. */ + onAllocate: undefined as (() => void) | undefined, + /** What an `allocate` from a client of that session waits for, once, before it runs `onAllocate`. */ + allocateGates: {} as Record>, + /** Runs inside `heartbeat` and `release`, before they answer. */ + onHeartbeat: undefined as (() => void) | undefined, + onRelease: undefined as (() => void) | undefined, + allocateError: undefined as Error | undefined, + /** Errors `release` answers with, in order, before it succeeds. */ + releaseErrors: [] as Error[], + /** Errors `heartbeat` answers with, in order, before it succeeds. */ + heartbeatErrors: [] as Error[], +}; + +vi.mock('agent-device', () => ({ + createAgentDeviceClient: (config: Record) => ({ + leases: { + allocate: async (options: Record) => { + daemon.calls.push({ config, operation: 'allocate', options }); + const gate = daemon.allocateGates[String(config['session'])]; + delete daemon.allocateGates[String(config['session'])]; + if (gate !== undefined) await gate; + daemon.onAllocate?.(); + if (daemon.allocateError !== undefined) throw daemon.allocateError; + return { leaseId: `lease-${daemon.calls.length}`, tenantId: options['tenant'], runId: options['runId'], backend: options['leaseBackend'], leaseProvider: options['leaseProvider'] }; + }, + heartbeat: async (options: Record) => { + daemon.calls.push({ config, operation: 'heartbeat', options }); + daemon.onHeartbeat?.(); + const error = daemon.heartbeatErrors.shift(); + if (error !== undefined) throw error; + return { leaseId: options['leaseId'], tenantId: options['tenant'], runId: options['runId'], backend: options['leaseBackend'] }; + }, + release: async (options: Record) => { + daemon.calls.push({ config, operation: 'release', options }); + daemon.onRelease?.(); + const error = daemon.releaseErrors.shift(); + if (error !== undefined) throw error; + return { released: true }; + }, + }, + }), +})); + +/** A real directory: the provider writes each run's marker under the project root. */ +const ROOT = mkdtempSync(join(tmpdir(), 'testmu-unit-')); + +afterAll(() => { + rmSync(ROOT, { recursive: true, force: true }); +}); +const env = { LT_USERNAME: 'ada', LT_ACCESS_KEY: 'lt-key' }; +const options: TestmuOptions = { device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: 'https://example.com/app.apk' }; +const saved = { LT_USERNAME: process.env['LT_USERNAME'], LT_ACCESS_KEY: process.env['LT_ACCESS_KEY'] }; + +beforeEach(() => { + Object.assign(daemon, { + calls: [], + onAllocate: undefined, + allocateGates: {}, + onHeartbeat: undefined, + onRelease: undefined, + allocateError: undefined, + releaseErrors: [], + heartbeatErrors: [], + }); +}); + +afterEach(() => { + vi.useRealTimers(); + for (const [name, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + } +}); + +function request(overrides: Partial = {}): DeviceRequest & { lines: string[] } { + const lines: string[] = []; + return { + platform: 'android', + runId: 'run-1', + targetName: 'android', + slot: 0, + slots: 2, + app: 'com.example.app', + agentDeviceVersion: '0.21.18', + projectRoot: ROOT, + env, + signal: new AbortController().signal, + log: (line) => lines.push(line), + lines, + ...overrides, + }; +} + +const context: DeviceReleaseContext = { runId: 'run-1', targetName: 'android', env, signal: new AbortController().signal, log: () => undefined }; + +const operations = () => daemon.calls.map((call) => call.operation); + +const runnerCredentials = () => [process.env['LT_USERNAME'], process.env['LT_ACCESS_KEY']]; + +const MINUTE = 60_000; + +describe('testmu()', () => { + it("is a device provider named testmu that records through TestMu AI's own session video", () => { + const provider = testmu(options); + expect(provider.name).toBe('testmu'); + expect(provider.record).toBeTypeOf('function'); + }); + + it('allocates an Android lease from a daemon under the project root, with the device selectors and dashboard labels', async () => { + await testmu(options).acquire(request({ slot: 1 })); + expect(daemon.calls).toEqual([ + { + config: { stateDir: join(ROOT, '.e2e', 'testmu', 'run-1'), session: 'lease-1' }, + operation: 'allocate', + options: { + tenant: 'testmu', + runId: 'run-1', + leaseBackend: 'android-instance', + leaseProvider: 'testmu', + platform: 'android', + target: 'mobile', + device: 'Galaxy S22 Ultra 5G', + providerOsVersion: '14', + providerApp: 'https://example.com/app.apk', + providerDeviceType: 'virtual', + providerProject: 'e2e', + providerBuild: 'run-1', + providerSessionName: 'e2e-run-1-android-2', + ttlMs: 10 * MINUTE, + }, + }, + ]); + }); + + it('allocates an iOS lease on a real device with its own project, build, session name, and state directory', async () => { + await testmu({ device: 'iPhone 16', osVersion: '18', app: 'lt://APP123', deviceType: 'real', project: 'shop', build: 'nightly', sessionName: 'checkout', stateDir: 'tmp/devices' }).acquire( + request({ platform: 'ios' }), + ); + expect(daemon.calls[0]?.config['stateDir']).toBe(join(ROOT, 'tmp', 'devices', 'run-1')); + expect(daemon.calls[0]?.options).toMatchObject({ + leaseBackend: 'ios-instance', + platform: 'ios', + device: 'iPhone 16', + providerOsVersion: '18', + providerApp: 'lt://APP123', + providerDeviceType: 'real', + providerProject: 'shop', + providerBuild: 'nightly', + providerSessionName: 'checkout-1', + }); + }); + + it("names each slot's session after the run, the target, and the slot, and keeps a given name unique per slot", async () => { + const provider = testmu(options); + await provider.acquire(request({ runId: 'run-7', targetName: 'pixel', slot: 0, slots: 3 })); + await provider.acquire(request({ runId: 'run-7', targetName: 'pixel', slot: 2, slots: 3 })); + await testmu({ ...options, sessionName: 'checkout' }).acquire(request({ slot: 2, slots: 3 })); + await testmu({ ...options, sessionName: 'checkout' }).acquire(request({ slot: 0, slots: 1 })); + expect(daemon.calls.map((call) => call.options['providerSessionName'])).toEqual(['e2e-run-7-pixel-1', 'e2e-run-7-pixel-3', 'checkout-3', 'checkout']); + }); + + it('passes the device features to the allocation and the worker under agent-device\'s keys', async () => { + const lease = await testmu({ ...options, orientation: 'landscape', geoLocation: 'US', timezone: 'UTC+05:30', language: 'fr', locale: 'fr_FR', appiumVersion: '2.16.2' }).acquire(request()); + const features = { + providerDeviceOrientation: 'landscape', + providerGeoLocation: 'US', + providerTimezone: 'UTC+05:30', + providerLanguage: 'fr', + providerLocale: 'fr_FR', + providerAppiumVersion: '2.16.2', + }; + expect(daemon.calls[0]?.options).toMatchObject(features); + expect(lease.client).toMatchObject(features); + }); + + it('resolves a local build against the project root, never the working directory', async () => { + await testmu({ ...options, app: 'build/app.apk' }).acquire(request()); + expect(daemon.calls[0]?.options['providerApp']).toBe(join(ROOT, 'build', 'app.apk')); + await testmu({ ...options, app: join('/', 'builds', 'app.apk') }).acquire(request()); + expect(daemon.calls[1]?.options['providerApp']).toBe(join('/', 'builds', 'app.apk')); + }); + + it('hands the worker the lease scope and the selectors as JSON client configuration', async () => { + const req = request(); + const lease = await testmu(options).acquire(req); + expect(lease).toEqual({ + id: 'lease-1', + client: { + stateDir: join(ROOT, '.e2e', 'testmu', 'run-1'), + tenant: 'testmu', + runId: 'run-1', + leaseId: 'lease-1', + leaseBackend: 'android-instance', + leaseProvider: 'testmu', + platform: 'android', + target: 'mobile', + device: 'Galaxy S22 Ultra 5G', + providerOsVersion: '14', + providerApp: 'https://example.com/app.apk', + providerDeviceType: 'virtual', + providerProject: 'e2e', + providerBuild: 'run-1', + providerSessionName: 'e2e-run-1-android-1', + }, + }); + const json = JSON.stringify(lease); + expect(JSON.parse(json)).toEqual(lease); + expect(Buffer.byteLength(json)).toBeLessThan(1024); + expect(json).not.toContain('lt-key'); + expect(req.lines).toEqual(['lease lease-1: Galaxy S22 Ultra 5G, android 14 (virtual); session e2e-run-1-android-1 started']); + }); + + it("shares the run's credentials with a daemon the allocation starts, and puts back the process's own after it", async () => { + delete process.env['LT_USERNAME']; + process.env['LT_ACCESS_KEY'] = 'stale'; + daemon.onAllocate = () => expect(runnerCredentials()).toEqual(['ada', 'lt-key']); + await testmu(options).acquire(request({ env: { LT_USERNAME: ' ada ', LT_ACCESS_KEY: 'lt-key' } })); + expect(operations()).toEqual(['allocate']); + expect(runnerCredentials()).toEqual([undefined, 'stale']); + }); + + it('keeps the credentials in place until every allocation running at once has finished', async () => { + delete process.env['LT_USERNAME']; + delete process.env['LT_ACCESS_KEY']; + let open!: () => void; + daemon.allocateGates = { 'lease-1': new Promise((resolve) => (open = resolve)) }; + const seen: unknown[] = []; + daemon.onAllocate = () => seen.push(runnerCredentials()); + const provider = testmu(options); + const first = provider.acquire(request({ slot: 0 })); + const second = provider.acquire(request({ slot: 1 })); + await vi.waitFor(() => expect(operations()).toEqual(['allocate', 'allocate'])); + await first; + expect(runnerCredentials()).toEqual(['ada', 'lt-key']); + open(); + await second; + expect(seen).toEqual([ + ['ada', 'lt-key'], + ['ada', 'lt-key'], + ]); + expect(runnerCredentials()).toEqual([undefined, undefined]); + }); + + it('makes a daemon call with other credentials wait until the calls with the first have finished', async () => { + delete process.env['LT_USERNAME']; + delete process.env['LT_ACCESS_KEY']; + let open!: () => void; + daemon.allocateGates = { 'lease-0': new Promise((resolve) => (open = resolve)) }; + const seen: unknown[] = []; + daemon.onAllocate = () => seen.push(runnerCredentials()); + const alice = testmu(options).acquire(request({ env: { LT_USERNAME: 'alice', LT_ACCESS_KEY: 'alice-key' } })); + await vi.waitFor(() => expect(operations()).toEqual(['allocate'])); + const bob = testmu(options).acquire(request({ slot: 1, env: { LT_USERNAME: 'bob', LT_ACCESS_KEY: 'bob-key' } })); + await new Promise((resolve) => setTimeout(resolve, 20)); + expect(operations()).toEqual(['allocate']); + expect(runnerCredentials()).toEqual(['alice', 'alice-key']); + open(); + await Promise.all([alice, bob]); + expect(seen).toEqual([ + ['alice', 'alice-key'], + ['bob', 'bob-key'], + ]); + expect(runnerCredentials()).toEqual([undefined, undefined]); + }); + + it('leaves a value the host changed during a daemon call', async () => { + process.env['LT_USERNAME'] = 'host'; + process.env['LT_ACCESS_KEY'] = 'host-key'; + daemon.onAllocate = () => { + process.env['LT_USERNAME'] = 'changed-by-host'; + }; + await testmu(options).acquire(request()); + expect(runnerCredentials()).toEqual(['changed-by-host', 'host-key']); + }); + + it('shares the credentials with a daemon a heartbeat or a release starts', async () => { + vi.useFakeTimers(); + process.env['LT_USERNAME'] = 'host'; + delete process.env['LT_ACCESS_KEY']; + const seen: unknown[] = []; + daemon.onHeartbeat = () => seen.push(['heartbeat', ...runnerCredentials()]); + daemon.onRelease = () => seen.push(['release', ...runnerCredentials()]); + const provider = testmu(options); + const lease = await provider.acquire(request()); + await vi.advanceTimersByTimeAsync(2 * MINUTE); + expect(runnerCredentials()).toEqual(['host', undefined]); + await provider.release(lease, { ...context, env: { LT_USERNAME: 'grace', LT_ACCESS_KEY: 'other-key' } }); + expect(seen).toEqual([ + ['heartbeat', 'ada', 'lt-key'], + ['release', 'grace', 'other-key'], + ]); + expect(runnerCredentials()).toEqual(['host', undefined]); + }); + + it('releases a lease without credentials in the release environment, through the daemon already running', async () => { + const provider = testmu(options); + const lease = await provider.acquire(request()); + await provider.release(lease, { ...context, env: {} }); + expect(operations()).toEqual(['allocate', 'release']); + }); + + it.each([ + [{}, 'LT_USERNAME and LT_ACCESS_KEY are not set'], + [{ LT_USERNAME: 'ada' }, 'LT_ACCESS_KEY is not set'], + [{ LT_USERNAME: ' ', LT_ACCESS_KEY: 'lt-key' }, 'LT_USERNAME is not set'], + ])('fails before any daemon starts without credentials in the run\'s environment (%j)', async (runEnv, message) => { + process.env['LT_USERNAME'] = 'from-the-runner'; + process.env['LT_ACCESS_KEY'] = 'from-the-runner'; + await expect(testmu(options).acquire(request({ env: runEnv }))).rejects.toThrow( + `${message}; set LT_USERNAME and LT_ACCESS_KEY to your TestMu AI username and access key in the environment \`e2e run\` starts in`, + ); + expect(daemon.calls).toEqual([]); + }); + + it("refuses the target's app.appPath, since TestMu AI installs `app`", async () => { + await expect(testmu(options).acquire(request({ appPath: join(ROOT, 'build', 'app.apk') }))).rejects.toThrow( + "TestMu AI installs the app from `app`; leave the target's `app.appPath` out", + ); + expect(daemon.calls).toEqual([]); + }); + + it('allocates nothing once the run is interrupted', async () => { + const controller = new AbortController(); + controller.abort(); + await expect(testmu(options).acquire(request({ signal: controller.signal }))).rejects.toThrow('cancelled before a lease was allocated'); + expect(daemon.calls).toEqual([]); + }); + + it('releases a lease granted after an interrupt instead of handing it over', async () => { + const controller = new AbortController(); + daemon.onAllocate = () => controller.abort(); + const req = request({ signal: controller.signal }); + await expect(testmu(options).acquire(req)).rejects.toThrow('lease lease-1 was not handed to the run: cancelled; released it'); + expect(daemon.calls[1]).toEqual({ + config: { stateDir: join(ROOT, '.e2e', 'testmu', 'run-1'), session: 'release' }, + operation: 'release', + options: { tenant: 'testmu', runId: 'run-1', leaseId: 'lease-1', leaseBackend: 'android-instance', leaseProvider: 'testmu' }, + }); + expect(req.lines).toEqual([]); + }); + + it('releases a granted lease when handing it over fails, and says when that release failed too', async () => { + daemon.releaseErrors = [new Error('daemon gone')]; + const req = request({ + log: () => { + throw new Error('reporter closed'); + }, + }); + await expect(testmu(options).acquire(req)).rejects.toThrow('lease lease-1 was not handed to the run: reporter closed; releasing it failed (daemon gone)'); + expect(operations()).toEqual(['allocate', 'release']); + }); + + it('passes an allocation failure through with nothing to release', async () => { + daemon.allocateError = new Error('unknown lease provider testmu'); + await expect(testmu(options).acquire(request())).rejects.toThrow('unknown lease provider testmu'); + expect(operations()).toEqual(['allocate']); + }); + + it('releases a lease through the daemon that granted it, once however often it is asked', async () => { + const provider = testmu(options); + const lease = await provider.acquire(request()); + await Promise.all([provider.release(lease, context), provider.release(lease, context)]); + await provider.release(lease, context); + expect(daemon.calls.slice(1)).toEqual([ + { + config: { stateDir: join(ROOT, '.e2e', 'testmu', 'run-1'), session: 'release' }, + operation: 'release', + options: { tenant: 'testmu', runId: 'run-1', leaseId: 'lease-1', leaseBackend: 'android-instance', leaseProvider: 'testmu' }, + }, + ]); + }); + + it('releases from the lease alone, after a round trip through JSON', async () => { + const lease = JSON.parse(JSON.stringify(await testmu(options).acquire(request({ platform: 'ios' })))) as DeviceLease; + await testmu(options).release(lease, context); + expect(daemon.calls[1]?.options).toEqual({ tenant: 'testmu', runId: 'run-1', leaseId: 'lease-1', leaseBackend: 'ios-instance', leaseProvider: 'testmu' }); + }); + + it('tries a release again after one failed', async () => { + daemon.releaseErrors = [new Error('daemon busy')]; + const provider = testmu(options); + const lease = await provider.acquire(request()); + await expect(provider.release(lease, context)).rejects.toThrow('daemon busy'); + await provider.release(lease, context); + expect(operations()).toEqual(['allocate', 'release', 'release']); + }); + + it('refuses to release a lease without an agent-device scope', async () => { + await expect(testmu(options).release({ id: 'other', client: { stateDir: '/tmp/x' } }, context)).rejects.toThrow('lease other carries no agent-device lease scope to release'); + expect(daemon.calls).toEqual([]); + }); + + it('heartbeats each lease it holds every two minutes, asking for the longest lease, until it is released', async () => { + vi.useFakeTimers(); + const provider = testmu(options); + const lease = await provider.acquire(request()); + await vi.advanceTimersByTimeAsync(2 * MINUTE - 1); + expect(operations()).toEqual(['allocate']); + await vi.advanceTimersByTimeAsync(1); + expect(daemon.calls[1]).toEqual({ + config: { stateDir: join(ROOT, '.e2e', 'testmu', 'run-1'), session: 'heartbeat' }, + operation: 'heartbeat', + options: { tenant: 'testmu', runId: 'run-1', leaseId: 'lease-1', leaseBackend: 'android-instance', leaseProvider: 'testmu', ttlMs: 10 * MINUTE }, + }); + await vi.advanceTimersByTimeAsync(2 * MINUTE); + expect(operations()).toEqual(['allocate', 'heartbeat', 'heartbeat']); + await provider.release(lease, context); + await vi.advanceTimersByTimeAsync(10 * MINUTE); + expect(operations()).toEqual(['allocate', 'heartbeat', 'heartbeat', 'release']); + }); + + it('stops heartbeating a lease it released because handing it over failed', async () => { + vi.useFakeTimers(); + const controller = new AbortController(); + daemon.onAllocate = () => controller.abort(); + await expect(testmu(options).acquire(request({ signal: controller.signal }))).rejects.toThrow('was not handed to the run'); + await vi.advanceTimersByTimeAsync(10 * MINUTE); + expect(operations()).toEqual(['allocate', 'release']); + }); + + it('heartbeats nothing when the allocation failed', async () => { + vi.useFakeTimers(); + daemon.allocateError = new Error('no capacity'); + await expect(testmu(options).acquire(request())).rejects.toThrow('no capacity'); + await vi.advanceTimersByTimeAsync(10 * MINUTE); + expect(operations()).toEqual(['allocate']); + }); + + it('logs the first failed heartbeat and keeps beating, without failing the run', async () => { + vi.useFakeTimers(); + daemon.heartbeatErrors = [new Error('Lease is not active'), new Error('daemon gone')]; + const req = request(); + await testmu(options).acquire(req); + await vi.advanceTimersByTimeAsync(6 * MINUTE); + expect(operations()).toEqual(['allocate', 'heartbeat', 'heartbeat', 'heartbeat']); + expect(req.lines.slice(1)).toEqual(['lease lease-1: heartbeat failed (Lease is not active); agent-device ends the lease after 10 minutes without one']); + }); + + it('survives a failed heartbeat when the run\'s log is closed', async () => { + vi.useFakeTimers(); + daemon.heartbeatErrors = [new Error('daemon gone')]; + let closed = false; + await testmu(options).acquire( + request({ + log: () => { + if (closed) throw new Error('reporter closed'); + }, + }), + ); + closed = true; + await vi.advanceTimersByTimeAsync(4 * MINUTE); + expect(operations()).toEqual(['allocate', 'heartbeat', 'heartbeat']); + }); + + it('rejects an option it does not take with INVALID_CONFIG, naming the nearest one', () => { + expect(() => testmu({ ...options, osVerison: '14' } as unknown as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG', message: expect.stringContaining('osVersion') })); + }); + + it.each(['device', 'osVersion', 'app'] as const)('requires `%s` with INVALID_CONFIG', (key) => { + expect(() => testmu({ ...options, [key]: ' ' })).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG', message: `testmu: \`${key}\` is required, as a non-empty string` })); + const { [key]: _, ...rest } = options; + expect(() => testmu(rest as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG' })); + }); + + it('refuses an orientation other than portrait or landscape', () => { + expect(() => testmu({ ...options, orientation: 'PORTRAIT' as 'portrait' })).toThrow( + expect.objectContaining({ code: 'INVALID_CONFIG', message: 'testmu: `orientation` must be \'portrait\' or \'landscape\', not "PORTRAIT"' }), + ); + }); + + it.each(['orientation', 'geoLocation', 'timezone', 'language', 'locale', 'appiumVersion'] as const)('refuses an empty or non-string `%s` with INVALID_CONFIG', (key) => { + expect(() => testmu({ ...options, [key]: ' ' } as unknown as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG', message: `testmu: \`${key}\` must be a non-empty string` })); + expect(() => testmu({ ...options, [key]: 2 } as unknown as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG' })); + expect(() => testmu({ ...options, [key]: null } as unknown as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG' })); + }); + + it.each(['emulator', null])('refuses a device type other than virtual or real (%j)', (deviceType) => { + expect(() => testmu({ ...options, deviceType } as unknown as TestmuOptions)).toThrow( + expect.objectContaining({ code: 'INVALID_CONFIG', message: `testmu: \`deviceType\` must be 'virtual' or 'real', not ${JSON.stringify(deviceType)}` }), + ); + }); + + it.each(['project', 'build', 'sessionName', 'stateDir'] as const)('refuses an empty or non-string `%s` with INVALID_CONFIG', (key) => { + for (const value of [' ', 1, null]) { + expect(() => testmu({ ...options, [key]: value } as unknown as TestmuOptions)).toThrow( + expect.objectContaining({ code: 'INVALID_CONFIG', message: `testmu: \`${key}\` must be a non-empty string` }), + ); + } + }); +}); diff --git a/packages/testmu/tsconfig.build.json b/packages/testmu/tsconfig.build.json new file mode 100644 index 000000000..65d5a8150 --- /dev/null +++ b/packages/testmu/tsconfig.build.json @@ -0,0 +1,11 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "dist", + "noEmit": false, + "declaration": true + }, + "include": ["src/**/*.ts"], + "exclude": ["dist", "node_modules"] +} diff --git a/packages/testmu/tsconfig.json b/packages/testmu/tsconfig.json new file mode 100644 index 000000000..635a4f631 --- /dev/null +++ b/packages/testmu/tsconfig.json @@ -0,0 +1,9 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "rootDir": ".", + "noEmit": true + }, + "include": ["src/**/*.ts", "tests/**/*.ts"], + "exclude": ["dist", "node_modules"] +} diff --git a/packages/testmu/vitest.config.ts b/packages/testmu/vitest.config.ts new file mode 100644 index 000000000..c7516095d --- /dev/null +++ b/packages/testmu/vitest.config.ts @@ -0,0 +1,10 @@ +import { defineConfig } from 'vitest/config'; + +/** Pure unit tests: agent-device's client is a stub, no daemon and no network. */ +export default defineConfig({ + test: { + include: ['tests/unit/**/*.test.ts'], + testTimeout: 30_000, + pool: 'forks', + }, +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index a04145870..e45f52d22 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -290,6 +290,9 @@ importers: '@e2e-dev/mobile': specifier: workspace:* version: link:../packages/mobile + '@e2e-dev/testmu': + specifier: workspace:* + version: link:../packages/testmu '@e2e-dev/web': specifier: workspace:* version: link:../packages/web @@ -525,6 +528,30 @@ importers: specifier: 5.0.1 version: 5.0.1(@types/node@26.6.2)(vite@8.1.5(@types/node@26.6.2)(esbuild@0.28.2)(jiti@1.21.7)(terser@5.51.2)(tsx@4.23.13)(yaml@2.9.0)) + packages/testmu: + devDependencies: + '@e2e-dev/mobile': + specifier: workspace:* + version: link:../mobile + '@e2e-dev/web': + specifier: workspace:* + version: link:../web + '@types/node': + specifier: 26.6.2 + version: 26.6.2 + agent-device: + specifier: 0.21.20 + version: 0.21.20(ai@7.0.107(zod@4.6.1)) + e2e: + specifier: workspace:* + version: link:../e2e + typescript: + specifier: 7.0.2 + version: 7.0.2 + vitest: + specifier: 5.0.1 + version: 5.0.1(@types/node@26.6.2)(vite@8.1.5(@types/node@26.6.2)(esbuild@0.28.2)(jiti@1.21.7)(terser@5.51.2)(tsx@4.23.14)(yaml@2.9.0)) + packages/web: dependencies: playwright-core: diff --git a/scripts/check-docs-examples.ts b/scripts/check-docs-examples.ts index 7edbc0c2b..a9e1ef386 100644 --- a/scripts/check-docs-examples.ts +++ b/scripts/check-docs-examples.ts @@ -27,7 +27,10 @@ const EXAMPLES: Record = { 'docs/examples/quickstart/e2e.command.config.ts': 'docs/web.mdx', 'docs/examples/mobile/device-provider.ts': 'docs/mobile.mdx', 'docs/examples/mobile/device-cloud-provider.ts': 'docs/mobile.mdx', + 'docs/examples/mobile/testmu.config.ts': 'docs/integrations/testmu.mdx', + 'docs/examples/mobile/testmu.e2e.ts': 'docs/integrations/testmu.mdx', 'docs/examples/web/browser-provider.ts': 'docs/browser.mdx', + 'docs/examples/web/testmu-browsers.config.ts': 'docs/integrations/testmu-browsers.mdx', 'docs/examples/models/e2e.decision.config.ts': 'docs/decision-models.mdx', 'docs/examples/models/tests/decision.e2e.ts': 'docs/decision-models.mdx', 'docs/examples/skill/e2e.config.ts': 'skills/e2e/SKILL.md', diff --git a/skills/e2e/references/setup.md b/skills/e2e/references/setup.md index 25d77b15b..aa161b7f5 100644 --- a/skills/e2e/references/setup.md +++ b/skills/e2e/references/setup.md @@ -180,7 +180,7 @@ start a script that brings them up and serves the app. | Option | Meaning | | --- | --- | -| `browser` | `'chromium'` (default), `'firefox'`, `'webkit'`, or a `BrowserProvider` leasing hosted browsers over CDP (`kernel()` from `@e2e-dev/kernel`, or your own), which implies chromium and excludes `connect`. Scope `'worker'` (default): one browser per worker slot from `prepare` to `finish`; `'attempt'`: one per attempt, with `reconnectEndpoint`'s limits. | +| `browser` | `'chromium'` (default), `'firefox'`, `'webkit'`, or a `BrowserProvider` leasing hosted browsers over CDP (`kernel()` from `@e2e-dev/kernel`, `testmuBrowsers()` from `@e2e-dev/testmu/web`, or your own), which implies chromium and excludes `connect`. Scope `'worker'` (default): one browser per worker slot from `prepare` to `finish`; `'attempt'`: one per attempt, with `reconnectEndpoint`'s limits. | | `viewport` | `{ width, height }`, default 1280x720; `null` follows the browser window (hosted live view, headed run). On a headed hosted browser (Kernel) use `null` and size the service's screen; a fixed size gives a smaller, unmaximized window. | | `connect` | `{ cdpEndpoint }` attaches to a remote Chromium over CDP; both it and `reconnectEndpoint` are resolvers `(signal) => url`, not strings. With `reconnectEndpoint` it rides one persistent default context and reconnects only to the original browser and page. | | `headers` | Sent to the app's site only (Vercel's `x-vercel-protection-bypass`, ngrok's `ngrok-skip-browser-warning`), `agent.act` included; disables the browser HTTP cache and service workers. | @@ -329,6 +329,13 @@ export default { the app (omit `app.appPath`). A run must fit one session: `maxDurationMinutes`, absent, is the account's cap (40 on a standard plan). `videoTouches: false` on the engine for video there. + `testmu({ device, osVersion, app })` from `@e2e-dev/testmu` leases TestMu + AI (formerly LambdaTest) emulators, simulators, or real devices + (`deviceType: 'real'`); `device` and `osVersion` match its catalog + exactly, TestMu AI installs `app` (omit `app.appPath`), it reads + `LT_USERNAME` and `LT_ACCESS_KEY`, and needs an agent-device with the + `testmu` provider shared with `@e2e-dev/mobile` (override its pin). + An attempt's video links TestMu AI's recording of the whole session. - Only a control that appeared or moved with the previous action waits out `transition` (default 500 ms); agent actions settle `settle` ms (default 150) before the next observation, `settle: false` skips it.