Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/ios-webdriver-element-types.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@e2e-dev/mobile': patch
---

iOS devices driven through agent-device's WebDriver runtime (hosted Appium clouds such as TestMu AI, BrowserStack, and AWS Device Farm) resolve roles like a local simulator does. That runtime reports XCUITest class names (`XCUIElementTypeButton`, `XCUIElementTypeStaticText`) where the native runner reports `Button` and `StaticText`, and the engine read them as unknown types, so `getByRole('button', 'CPU Load')` matched nothing and `observe` listed the node as `xcuielement-type-button`. The `XCUIElementType` prefix is now dropped before the type is mapped, so a button is a `button`, a tab bar's buttons are `tab`s, and the navigation bar titles the screen.
5 changes: 5 additions & 0 deletions .changeset/testmu-devices.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@e2e-dev/testmu": minor
---

`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); the worker's first command starts the TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id), heartbeats each lease while the run holds it so a session slow to start keeps its lease, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers.
6 changes: 6 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,12 @@ suites that consume the built packages the way a user would.
hosted iOS simulators and Android emulators for the mobile engine
(`DeviceProvider`). Expo publishes no SDK for the sessions API, so it calls
Expo's GraphQL API with `fetch`, and `@e2e-dev/mobile` is its only peer.
- `packages/testmu` - the published `@e2e-dev/testmu` package: TestMu AI
(formerly LambdaTest) hosted Android emulators, iOS simulators, and real
devices for the mobile engine (`DeviceProvider`). agent-device's `testmu`
cloud provider is the vendor client: the package allocates agent-device
leases from a daemon it starts per run, so `agent-device` (which must be
the same copy `@e2e-dev/mobile` uses) and `@e2e-dev/mobile` are its peers.
- `apps/testbed` (`@e2e-dev/testbed`, private) — dogfood project that
consumes the **built** packages like a real user would: the playground app
where every runner feature (sessions, routes, downloads, frames, uploads,
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,5 +254,5 @@ build against, and what a third-party engine builds against too. A change to
that contract bumps all three packages together in one release, with a
changeset for each, so an engine and a runner from the same release always
match. Engines declare a peer range on `e2e` that points one way only
(engine to runner, `>=x <1`), and each integration (`@e2e-dev/kernel`, `@e2e-dev/eas`) does
(engine to runner, `>=x <1`), and each integration (`@e2e-dev/kernel`, `@e2e-dev/eas`, `@e2e-dev/testmu`) does
the same on the engines it plugs into; do not make it mutual or narrow it.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ config and an example test. The
| [`@e2e-dev/github`](https://www.npmjs.com/package/@e2e-dev/github) | Reporter that posts results as a pull request comment. |
| [`@e2e-dev/kernel`](https://www.npmjs.com/package/@e2e-dev/kernel) | Kernel hosted browsers for the web engine. |
| [`@e2e-dev/eas`](https://www.npmjs.com/package/@e2e-dev/eas) | EAS Simulators hosted iOS simulators and Android emulators for the mobile engine. |
| [`@e2e-dev/testmu`](https://www.npmjs.com/package/@e2e-dev/testmu) | TestMu AI (formerly LambdaTest) hosted Android emulators, iOS simulators, and real devices for the mobile engine. |

## Documentation

Expand Down
3 changes: 2 additions & 1 deletion docs/docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,8 @@
"pages": [
"integrations/index",
"integrations/kernel",
"integrations/eas"
"integrations/eas",
"integrations/testmu"
]
},
{
Expand Down
35 changes: 35 additions & 0 deletions docs/examples/mobile/testmu.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
import type { E2EConfig } from 'e2e';
import { mobile } from '@e2e-dev/mobile';
import { testmu } from '@e2e-dev/testmu';

const apk = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_android.apk';

export default {
targets: [
{
name: 'android-emulator',
engine: mobile({
platform: 'android',
device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: apk }),
}),
app: { bundleId: 'com.lambdatest.proverbial' },
},
{
name: 'ios-simulator',
engine: mobile({
platform: 'ios',
device: testmu({ device: 'iPhone 16', osVersion: '18.0', app: './build/MyApp.zip' }),
}),
app: { bundleId: 'com.example.app' },
},
{
name: 'android-real',
engine: mobile({
platform: 'android',
device: testmu({ device: 'Pixel 6', osVersion: '14', app: apk, deviceType: 'real' }),
}),
app: { bundleId: 'com.lambdatest.proverbial' },
},
],
workers: 1,
} satisfies E2EConfig;
8 changes: 8 additions & 0 deletions docs/examples/mobile/testmu.e2e.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import { test } from '@e2e-dev/mobile';
import { expect } from 'e2e';

test('Proverbial home screen', async ({ app, screen }) => {
await app.open();
await expect(screen.getByText('Proverbial')).toBeVisible();
await expect(screen.getByRole('button', 'GEOLOCATION')).toBeVisible();
});
3 changes: 3 additions & 0 deletions docs/integrations/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ integration decides where it comes from.
<Card title="EAS Simulators" icon="/images/integrations/expo.svg" href="/integrations/eas">
`@e2e-dev/eas`: hosted iOS simulators and Android emulators for mobile targets.
</Card>
<Card title="TestMu AI" icon="mobile-screen" href="/integrations/testmu">
`@e2e-dev/testmu`: hosted Android emulators, iOS simulators, and real devices for mobile targets.
</Card>
</CardGroup>

A service without an integration is one small provider in your project:
Expand Down
240 changes: 240 additions & 0 deletions docs/integrations/testmu.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,240 @@
---
title: TestMu AI
sidebarTitle: TestMu AI
description: Run mobile targets on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices with the testmu() device provider from @e2e-dev/testmu.
---

`testmu()` from `@e2e-dev/testmu` is a
[device provider](/mobile#hosted-devices) for
[TestMu AI](https://www.lambdatest.com) (formerly LambdaTest). The run gets
hosted Android emulators, iOS simulators, or real devices, leased when it
starts and released when it ends. agent-device reaches them through its
`testmu` cloud provider, which drives each session over TestMu AI's Appium
hub. Tests, config, and CI stay the same.

<Tip>
No Mac, Xcode, or Android SDK on the machine that runs e2e.
</Tip>

## Setup

The provider runs on agent-device's `testmu` provider, so install
`agent-device` beside the package, at a release that includes it:

<CodeGroup>
```bash npm
npm install --save-dev @e2e-dev/testmu agent-device@<version>
```

```bash pnpm
pnpm add -D @e2e-dev/testmu agent-device@<version>
```

```bash bun
bun add -d @e2e-dev/testmu agent-device@<version>
```
</CodeGroup>

`<version>` is an agent-device release that includes the `testmu` provider.
No published release does yet: until one does, install agent-device from a
build that includes it. The package's `agent-device` peer range excludes
only the releases known to lack the provider, so it cannot tell you whether
a newer one has it.

`@e2e-dev/mobile` pins its own agent-device. The provider starts the daemon
from your copy and the engine's workers send commands through theirs, so
both must be the same agent-device. Override the engine's pin:

<CodeGroup>
```json npm
{
"overrides": {
"agent-device": "$agent-device"
}
}
```

```yaml pnpm
# pnpm-workspace.yaml
overrides:
agent-device: $agent-device
```

```json bun
{
"overrides": {
"agent-device": "<version>"
}
}
```
</CodeGroup>

For npm and bun the `overrides` field goes in `package.json`.

```ts title="e2e.config.ts"
import type { E2EConfig } from 'e2e';
import { mobile } from '@e2e-dev/mobile';
import { testmu } from '@e2e-dev/testmu';

const apk = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_android.apk';

export default {
targets: [
{
name: 'android-emulator',
engine: mobile({
platform: 'android',
device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: apk }),
}),
app: { bundleId: 'com.lambdatest.proverbial' },
},
{
name: 'ios-simulator',
engine: mobile({
platform: 'ios',
device: testmu({ device: 'iPhone 16', osVersion: '18.0', app: './build/MyApp.zip' }),
}),
app: { bundleId: 'com.example.app' },
},
{
name: 'android-real',
engine: mobile({
platform: 'android',
device: testmu({ device: 'Pixel 6', osVersion: '14', app: apk, deviceType: 'real' }),
}),
app: { bundleId: 'com.lambdatest.proverbial' },
},
],
workers: 1,
} satisfies E2EConfig;
```

`device` and `osVersion` must match TestMu AI's catalog exactly, as the
[capabilities generator](https://www.lambdatest.com/capabilities-generator)
lists them: a virtual iOS device takes a version like `18.0`, a real iOS
device one like `18`, and Android one like `14`. TestMu AI checks them when
the session starts and refuses a name or version it does not list.

Keep `app.bundleId`: TestMu AI installs the app when the session starts,
and `app.open()` launches the installed bundle id or package, not the upload
name. Leave `app.appPath` out; a target that names one fails its lease.

## Authenticate

Set `LT_USERNAME` and `LT_ACCESS_KEY` to your TestMu AI username and access
key in the environment `e2e run` starts in, such as CI secrets. Without
either, the lease fails before anything starts, naming the missing one.

agent-device's `testmu` runtime reads the credentials from the environment
of the daemon that drives the sessions. The provider starts that daemon from
the runner process, and agent-device starts a daemon with the process's own
environment, so the provider copies both values from the run's environment
into it first.

## Install the app

`app` names the build TestMu AI installs on every session:

- **An app id**: an `lt://` reference to a build already in TestMu AI's app
storage.
- **A URL**: an `https` link to the build.
- **A local path**: a build on the machine that runs e2e, resolved against
the directory of `e2e.config.ts` and uploaded when the session starts.

The build must suit the device:

| Device | Build |
| --- | --- |
| Android emulator or real device | `.apk` or `.aab` |
| iOS simulator | a zipped `.app` bundle, not an `.ipa` |
| Real iOS device | a signed `.ipa` |

## Real devices

`deviceType: 'real'` leases a real device instead of an emulator or
simulator. Real and virtual devices have separate catalogs, so check
`device` and `osVersion` against the real-device one, and give a real iOS
device a signed `.ipa`. Real devices need an agent-device release whose
`testmu` provider supports them.

## Options

| Option | Default | Meaning |
| ------------- | --------------- | ------- |
| `device` | required | Device name exactly as TestMu AI's catalog lists it (`'Galaxy S22 Ultra 5G'`, `'iPhone 16'`). |
| `osVersion` | required | OS version exactly as the catalog lists it for that device: `'14'` on Android, `'18.0'` on a virtual iOS device, `'18'` on a real one. |
| `app` | required | The build TestMu AI installs: an `lt://` app id, an `https` URL, or a local path. See [Install the app](#install-the-app). |
| `deviceType` | `'virtual'` | `'virtual'` for an emulator or simulator, `'real'` for a real device. |
| `project` | `'e2e'` | Dashboard project the sessions are grouped under. |
| `build` | the run id | Dashboard build the sessions are grouped under. |
| `sessionName` | TestMu AI's | Name of every session on the dashboard. |
| `stateDir` | `'.e2e/testmu'` | Directory for the agent-device daemon each run starts, relative to the directory of `e2e.config.ts`. Keep it out of version control. |

An option not in this table, a missing or empty `device`, `osVersion`, or
`app`, or another `deviceType` fails the config load with `INVALID_CONFIG`.

## Write a test

Tests are the same as on a local device:

```ts title="tests/home.e2e.ts"
import { test } from '@e2e-dev/mobile';
import { expect } from 'e2e';

test('Proverbial home screen', async ({ app, screen }) => {
await app.open();
await expect(screen.getByText('Proverbial')).toBeVisible();
await expect(screen.getByRole('button', 'GEOLOCATION')).toBeVisible();
});
```

## Sessions

Each target leases up to `workers` devices (`--workers` overrides it), one
per worker slot, before the run's clock starts. A session takes about 30 to
75 seconds to start, allocation and app upload included. TestMu AI
runs as many sessions at once as the plan allows, so keep `workers` times
the number of TestMu AI targets within it.

The provider logs each lease as it is granted:

```text
ℹ leasing 1 android device(s) from testmu
ℹ testmu (1 of 1): lease 4f0c…: Galaxy S22 Ultra 5G, android 14 (virtual); the session starts on the first command
ℹ testmu: leased 4f0c…
```

The video, device logs, and network logs of every session are on the
TestMu AI dashboard, under the project and build the options name.

## What the provider does

- Starts one agent-device daemon per run under `stateDir/<run id>` and
allocates a `testmu` lease from it for each worker slot.
- Keeps each lease alive while the run holds it. A lease asks for
agent-device's longest inactivity window, 10 minutes, and the provider
heartbeats it every 2 minutes, so a session that takes long to start keeps
its lease. A failed heartbeat is logged once and does not fail the run.
- Hands the worker the lease scope and the device selectors as agent-device
client configuration. The worker's first command starts the TestMu AI
session from them, which installs `app`.
- Releases each lease when the run ends, which ends its session. A lease
is released once however often the engine asks, and a failed release can
be tried again.
- Releases a lease it was granted after the run was interrupted, or could
not hand to the run, before failing the lease.

## Limits

Sessions run over TestMu AI's Appium hub through agent-device's WebDriver
runtime, so what agent-device does not support there is not available to
tests:

- `device` settings: permissions, location, network, and appearance.
- System alerts.
- Recording through agent-device. Leave [`video`](/reference/config#video)
off for these targets. TestMu AI records every session itself, and the
video and device logs are on its dashboard.
- Device logs through agent-device; read them on the dashboard.
- Port reverse. To reach a server on your machine or network, use TestMu
AI Tunnel.
6 changes: 4 additions & 2 deletions docs/mobile.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,8 @@ ends. The engine drives each lease through the agent-device daemon the lease
names instead of the local one, so any service that runs an agent-device
daemon next to a simulator, an emulator, or a physical device works. Nothing
in the framework knows a vendor: for [EAS Simulators](/integrations/eas),
`easSimulators()` from `@e2e-dev/eas` is the provider, and for
`easSimulators()` from `@e2e-dev/eas` is the provider, for
[TestMu AI](/integrations/testmu) `testmu()` from `@e2e-dev/testmu`, and for
any other service a provider is a few dozen lines in your project.

```ts
Expand Down Expand Up @@ -302,7 +303,8 @@ on `app.bundleId`, so traces recorded locally replay on hosted devices.

A device cloud agent-device speaks to itself needs no session API of your
own: the provider allocates a lease through a daemon it starts for the run and
hands the worker the lease scope as `client`.
hands the worker the lease scope as `client`. `testmu()` from
`@e2e-dev/testmu` works this way for [TestMu AI](/integrations/testmu).

```ts title="device-cloud-provider.ts"
import { createAgentDeviceClient } from 'agent-device';
Expand Down
1 change: 1 addition & 0 deletions docs/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
},
"devDependencies": {
"@e2e-dev/mobile": "workspace:*",
"@e2e-dev/testmu": "workspace:*",
"e2e": "workspace:*",
"@e2e-dev/web": "workspace:*",
"@types/node": "26.6.2",
Expand Down
1 change: 1 addition & 0 deletions docs/reference/environment.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ What is sent and how to read it is on the [Telemetry](/telemetry) page.
| `KERNEL_API_KEY` | value | The API key `kernel()` from `@e2e-dev/kernel` creates browsers with; unset, the browser lease fails. See [Kernel](/integrations/kernel) |
| `EXPO_TOKEN` | value | The Expo access token `easSimulators()` from `@e2e-dev/eas` starts and stops simulator sessions with; unset, it uses the eas-cli login in `.expo/state.json` under `HOME` (`USERPROFILE` on Windows), and with neither the device lease fails. See [EAS Simulators](/integrations/eas) |
| `HOME`, `USERPROFILE` | value | The home `easSimulators()` finds the eas-cli login under when `EXPO_TOKEN` is unset: `USERPROFILE` on Windows, `HOME` elsewhere, the OS user's home when unset |
| `LT_USERNAME`, `LT_ACCESS_KEY` | value | The TestMu AI username and access key `testmu()` from `@e2e-dev/testmu` leases devices with; it copies them into the runner process's environment for the agent-device daemon it starts, and with either unset the device lease fails. See [TestMu AI](/integrations/testmu) |
| `E2E_AGENT_DEVICE_POOL_<TARGET>_<digest>` | value | Written by `@e2e-dev/mobile` in `prepare` and read by each worker to find its booted device. Not meant to be set by hand |

## What app processes inherit
Expand Down
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
],
"type": "module",
"scripts": {
"build": "pnpm --filter e2e run build && pnpm --filter @e2e-dev/web run build && pnpm --filter @e2e-dev/mobile run build && pnpm --filter @e2e-dev/github run build && pnpm --filter @e2e-dev/kernel run build && pnpm --filter @e2e-dev/eas run build",
"build": "pnpm --filter e2e run build && pnpm --filter @e2e-dev/web run build && pnpm --filter @e2e-dev/mobile run build && pnpm --filter @e2e-dev/github run build && pnpm --filter @e2e-dev/kernel run build && pnpm --filter @e2e-dev/eas run build && pnpm --filter @e2e-dev/testmu run build",
"check": "pnpm run lint && pnpm run check:dead-code && pnpm run typecheck && pnpm run docs:check-errors && pnpm run check:peer-ranges && pnpm run docs:check",
"check:dead-code": "fallow dead-code",
"check:peer-ranges": "node scripts/check-peer-ranges.ts",
Expand All @@ -24,7 +24,7 @@
"canary": "node scripts/canary-changeset.ts && changeset version --snapshot canary && pnpm run build",
"canary:publish": "changeset publish --tag canary --no-git-tag && node scripts/restore-peer-ranges.ts",
"typecheck": "pnpm run build && pnpm -r run typecheck && tsc -p scripts",
"test": "pnpm run build && pnpm --filter e2e run test && pnpm --filter @e2e-dev/web run test && pnpm --filter @e2e-dev/mobile run test && pnpm --filter @e2e-dev/github run test && pnpm --filter @e2e-dev/kernel run test && pnpm --filter @e2e-dev/eas run test && pnpm run test:scripts",
"test": "pnpm run build && pnpm --filter e2e run test && pnpm --filter @e2e-dev/web run test && pnpm --filter @e2e-dev/mobile run test && pnpm --filter @e2e-dev/github run test && pnpm --filter @e2e-dev/kernel run test && pnpm --filter @e2e-dev/eas run test && pnpm --filter @e2e-dev/testmu run test && pnpm run test:scripts",
"test:testbed": "pnpm run build && pnpm --filter @e2e-dev/testbed test",
"test:web-benchmark": "pnpm run build && pnpm --filter @e2e-dev/web-benchmark test",
"test:mobile-benchmark": "pnpm run build && pnpm --filter @e2e-dev/mobile-benchmark test",
Expand Down
Loading