Skip to content

feat(testmu): add @e2e-dev/testmu device provider - #772

Open
amankansal-lt wants to merge 29 commits into
tester-army:mainfrom
LambdaTest:main
Open

amankansal-lt wants to merge 29 commits into
tester-army:mainfrom
LambdaTest:main

Conversation

@amankansal-lt

@amankansal-lt amankansal-lt commented Oct 2, 2026 •

Copy link
Copy Markdown

What and why

  • Adds @e2e-dev/testmu, a DeviceProvider that runs mobile targets on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators and real devices, through agent-device's testmu cloud provider (feat(provider-webdriver): add TestMu AI device cloud provider callstack/agent-device#3118).

    import { mobile } from '@e2e-dev/mobile';
    import { testmu } from '@e2e-dev/testmu';
    
    engine: mobile({
      platform: 'android',
      device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: 'lt://APP-id' }),
    }),
  • Adds testmuBrowsers(), exported from @e2e-dev/testmu/web: a BrowserProvider for @e2e-dev/web running TestMu AI hosted Chrome and Edge on Windows and macOS. This consolidates the browser side from feat(testmuai): @e2e-dev/testmuai, TestMu AI hosted browsers for the web engine #765, as asked when that PR closed. A session starts when its CDP websocket opens and ends when it closes, so the provider only builds the CDP URL; there's no API call and no SDK. The package root stays device-only, and /web never loads agent-device or @e2e-dev/mobile. All three peers are optional, so a project installs only the side it uses.

    import { web } from '@e2e-dev/web';
    import { testmuBrowsers } from '@e2e-dev/testmu/web';
    
    engine: web({ browser: testmuBrowsers({ platform: 'Windows 11' }), viewport: null }),
  • Leases: each worker slot allocates an agent-device lease. The worker's first command starts the session and installs the app; release ends it. Release is idempotent and also runs when a lease arrives after an interrupt.

  • Credentials: LT_USERNAME and LT_ACCESS_KEY come from the run's environment. If one is missing, the lease fails before anything starts.

  • Options: deviceType: 'real' selects real devices. project, build, sessionName and stateDir are optional. Unknown options fail at config load.

  • Lease keep-alive: agent-device leases expire after 60 s idle, and an iOS simulator's first command (upload, allocation, boot) takes about 70 s. The provider allocates with the daemon's maximum window (10 min) and heartbeats every 2 minutes until release.

  • fix(mobile): XCUIElementType-prefixed iOS element types. agent-device's WebDriver runtime, used by every hosted Appium cloud including BrowserStack and AWS Device Farm, reports XCUIElementTypeButton where the native runner reports Button. Before this, getByRole('button', …) matched nothing on those iOS devices. Patch changeset for @e2e-dev/mobile.

  • Requirement (devices only): the testmu provider for agent-device, which is now the managed plugin @agent-device/testmu in feat(provider-webdriver): add TestMu AI device cloud provider callstack/agent-device#3118. agent-device is a peer (>0.21.20 <1), so the provider and the engine share one copy. Until a release ships, the docs page keeps the version placeholder. Browsers need nothing beyond @e2e-dev/web.

Since opening

  • Video: the provider implements DeviceProvider.record(). An attempt that records video links TestMu AI's recording of the slot's session, which is looked up by build, session name and user. Its startedAt is the session's start time. Without this, video: 'on' and retain-on-failure failed, because agent-device's WebDriver runtime cannot record.
  • Sessions: each slot's session is named e2e-<run id>-<target>-<slot> by default. The docs now say the session starts when the lease is allocated, which is also when billing starts.
  • New options: orientation, geoLocation, timezone, language, locale, appiumVersion.
  • Hardening:
    • Credentials sit in process.env only while the provider calls the daemon.
    • Every option is validated at config load.
    • An API endpoint that carries credentials is refused.
    • Run state is pruned after 24 h, but only in directories the provider marked as its own.
  • Browsers: testmuBrowsers() (co-authored by @SahilSawLT).
    • It follows testmu()'s conventions: project, build and sessionName defaults, top-level geoLocation and timezone, strict validation, and capabilities that can't override what the provider sets.
    • Scopes are worker (one session per slot) or attempt.
    • Routes are /puppeteer (the default) or /playwright-cdp, which is still rolling out on TestMu AI's side.
  • Upstream sync: merged main twice (the README, docs package.json, root package.json and AGENTS.md lists). @e2e-dev/testmu follows @e2e-dev/mobile's agent-device 0.21.20 pin and its Node engines.
  • Live on TestMu AI production at this head: Android emulator with video: 'on' (the attempt links the session's video.mp4), iOS simulator and a real iPhone all passed.

Verified

Ran it locally: yes

  • pnpm check passes: lint, dead code, typecheck, error codes, peer ranges, install scripts, docs and broken links.
  • Unit tests: @e2e-dev/testmu 108 passed, @e2e-dev/mobile 232 passed, @e2e-dev/web 504 passed (1 skipped).
  • Live against TestMu AI production with the packed packages from this branch. Each test opens the app and checks it with getByRole / getByText; every device was released afterwards.
Target Device App Engine ready Test
Android emulator Galaxy S22 Ultra 5G, 14 .apk by URL 54 s passed
iOS simulator iPhone 16, 18.0 local zipped .app 71 s passed
Android real device Pixel 6, 14 .apk by URL 27 s passed
iOS real device iPhone 16, 18 signed .ipa 29 s passed

Browsers on TestMu AI production, with a browser-only install of the packed package (e2e 0.17.0 and @e2e-dev/web 0.12.0). npm pulled neither agent-device nor @e2e-dev/mobile:

Route Platform Result
/puppeteer (default) Windows 11, Chrome passed, recorded under build <run id>
/playwright-cdp macOS Sequoia, Chrome, on a machine with the new backend passed

The platform issues we've seen are documented under Limits and reported to TestMu AI. They reproduce with plain Puppeteer and Playwright, so they aren't provider bugs:

  • Edge over CDP gets a hub 500 on Windows 11.
  • geoLocation DE and FR exit in neighbouring countries.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md). n/a, not breaking.
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures. n/a
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile. n/a

🤖 Generated with Claude Code

amankansal-lt and others added 5 commits October 2, 2026 19:55
Add @e2e-dev/testmu, a DeviceProvider for the mobile engine that runs
targets on TestMu AI (formerly LambdaTest) Android emulators, iOS
simulators, and real devices through agent-device's testmu cloud
provider:

  mobile({ device: testmu({ device, osVersion, app }) })

Each worker slot allocates an agent-device lease from a daemon the
provider starts for the run under stateDir (default .e2e/testmu). The
lease's client configuration carries the lease scope and the device
selectors, so the worker's first command starts the TestMu AI session
and installs the app; releasing the lease ends it.

- Credentials come from LT_USERNAME and LT_ACCESS_KEY in the run's
  environment; a missing one fails the lease before anything starts.
  agent-device's client spawns its daemon with process.env and takes no
  environment of its own, so the provider copies the run's values there
  before the first allocation.
- A lease granted after an interrupt, or one that cannot be handed to
  the run, is released before acquire throws. Release is idempotent and
  works from the lease alone, after a round trip through JSON.
- A relative local app path resolves against the project root, not the
  daemon's working directory. The target's app.appPath is refused, since
  TestMu AI installs the app itself.
- Unknown options, missing device/osVersion/app, and a deviceType other
  than virtual or real fail with INVALID_CONFIG.

agent-device is a peer (>0.21.18 <1): the provider and the engine's
workers must share one agent-device, and no published release includes
the testmu provider yet, so the range excludes only the releases known
to lack it. No record hook: TestMu AI records whole sessions, not
attempts, and its video is only available once the session ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a TestMu AI (formerly LambdaTest) integration page for the testmu()
device provider from @e2e-dev/testmu, in the shape of the EAS Simulators
page: setup, authentication, installing the app, real devices, an
options table, sessions, what the provider does, and limits.

The page covers the requirement for an agent-device that includes the
testmu provider and the override that makes @e2e-dev/mobile use the same
copy, the catalog-exact device names and OS versions (18.0 for a virtual
iOS device, 18 for a real one, 14 on Android), the app formats per
device, and the limits of agent-device's WebDriver runtime.

The config and test examples are shown verbatim and typecheck against
the built package. The integrations overview, the hosted devices section
of the mobile page, the environment reference (LT_USERNAME,
LT_ACCESS_KEY), and the agent skill's setup reference name the package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-device ends a lease after 60 seconds without activity by default,
and a command still running does not count as activity. The worker's first
command uploads the app, allocates the hosted session, and boots it; on an
iOS simulator that takes about 70 seconds, so the lease lapsed mid-start
and the engine's prepare failed with "boot failed: Lease is not active"
(LEASE_NOT_FOUND).

Each lease now asks for agent-device's longest inactivity window (10
minutes), and the provider heartbeats every lease it holds every 2 minutes
from the runner process until the lease is released, including the
release acquire does itself when it cannot hand a lease to the run. The
timer is unref'd so it never keeps the runner alive. A failed heartbeat is
logged once per lease and the next one is still tried; it never fails the
run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-device's native iOS runner reports element types as `Button`,
`StaticText`, `NavigationBar`. Its WebDriver runtime, which hosted Appium
clouds (TestMu AI, BrowserStack, AWS Device Farm) go through, reports the
XCUITest class names instead: `XCUIElementTypeButton`,
`XCUIElementTypeStaticText`. normalizeKind kebab-cased those to
`xcuielement-type-button`, which maps to no role, so on a real iPhone
`getByRole('button', 'CPU Load')` matched nothing and observe listed the
node by its raw class name.

normalizeKind now drops a leading `XCUIElementType` before kebab-casing,
so both spellings land on one vocabulary: buttons, tab bar items as tabs,
the navigation bar as the screen title. Only that exact prefix followed by
a type name is stripped; Android class names are untouched, and the golden
device trees project unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(testmu): add @e2e-dev/testmu device provider

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 30 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread docs/package.json
Comment thread docs/examples/mobile/testmu.e2e.ts
Comment thread packages/testmu/src/credentials.ts Outdated
Comment thread packages/testmu/src/provider.ts
Comment thread packages/testmu/src/provider.ts Outdated
Comment thread docs/mobile.mdx Outdated
amankansal-lt and others added 19 commits October 2, 2026 22:06
agent-device's `leases.allocate` creates the TestMu AI session before it
returns, so the session starts at acquire, not on the worker's first
command, and every slot is billed from the moment it is leased. The lease
log line, the provider's comments, the README, the docs page, and the
changeset now say so, and explain the long lease window and heartbeat by
agent-device starting a lease's window before a slow allocation returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without `sessionName`, each slot's session is now named
`e2e-<run id>-<target>-<slot>` instead of whatever TestMu AI picks, and a
given `sessionName` gets `-<slot>` appended when the target leases more than
one device. Every slot's session then has a name of its own within its
build, which is what finding the session again by name relies on. The lease
log line names the session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`orientation`, `geoLocation`, `timezone`, `language`, `locale`, and
`appiumVersion` set up the device when its session starts. Each one is
allocated under agent-device's matching lease key
(`providerDeviceOrientation`, `providerGeoLocation`, `providerTimezone`,
`providerLanguage`, `providerLocale`, `providerAppiumVersion`), which its
`testmu` provider turns into TestMu AI capabilities. An empty value, or an
orientation other than portrait or landscape, fails the config load with
INVALID_CONFIG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-device's cloud WebDriver runtime cannot record, so a run with
`video: 'on'` or `'retain-on-failure'` on a TestMu AI target failed when
the attempt started its video. The provider now implements `record`: the
attempt's video links TestMu AI's own recording of the slot's session.

`record` runs in the worker from the lease alone. It reads the build and
session name the lease carries and the credentials from the run's
environment, and starts at the moment it is called. `stop` finds the
session through TestMu AI's sessions API (the newest session with that name
in the build's list, paged) and returns the `video_url` of its details as
`video/mp4`. The API defaults to
https://mobile-api.lambdatest.com/mobile-automation/api/v1 and follows
`TESTMU_API_ENDPOINT`, as agent-device does. Every request is bounded by a
15-second timeout and the stop's signal, and errors name the build and the
session without credentials or URLs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each run leaves `<stateDir>/<run id>` behind with its daemon's state and
logs, and nothing removed it. The directory is still kept at release, since
the daemon exits by itself after 5 idle minutes and its logs help with a
failure. Instead, the provider's first acquire removes earlier runs'
directories under `stateDir` last modified more than 24 hours ago. It never
touches the current run's directory or anything not named like a run id,
and a failure leaves the directory and does not fail the lease.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
State that `app` is required and that `app.appPath` is refused on purpose,
since the provider hands TestMu AI the build itself, while
`device.installApp()` with a path still works during a test. State that the
provider copies `LT_USERNAME` and `LT_ACCESS_KEY` into the runner's
`process.env`, where the agent-device daemon it starts reads them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A provider recording's `startedAt` is the time of the video's first frame,
and TestMu AI's video covers the whole session, so the time `record` was
called put every timeline offset off by however long the session had run.
`record` now finds the session in TestMu AI's sessions list by build and
name, as `stop` did, and takes the row's `start_timestamp` (RFC 3339; a
time without a zone is read as UTC) as `startedAt`. `stop` fetches the
`video_url` of the session `record` found. Without a usable start time,
`startedAt` falls back to when `record` was called.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The TestMu AI example config ran its test, which checks the Proverbial
sample app, on an iOS simulator target configured for a different app. Every
target now runs Proverbial: the iOS target is a real iPhone with TestMu AI's
public Proverbial `.ipa`, and the test matches the geolocation button's
label on both platforms. An iOS simulator target is shown separately in the
build section. The hosted-devices paragraph in the mobile guide names the
TestMu AI provider with the same structure as the EAS one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-device's client starts its local daemon with `process.env` and takes
no environment of its own, so the provider copied `LT_USERNAME` and
`LT_ACCESS_KEY` into `process.env` and left them there, where a long-lived
host would pass them to unrelated child processes. The provider now sets
them only around its daemon calls (allocate, heartbeat, and release, any of
which may start the daemon) and puts back the previous values once no such
call is in flight. A release without credentials in its environment still
goes through the daemon that is already running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`project`, `build`, `sessionName`, and `stateDir` were passed through
unchecked, so a JavaScript config with `project: 1` or `stateDir: 1`
reached the allocation or failed inside `acquire`. They are now checked
with the device-feature options when `testmu()` is called, and any of them
that is not a non-empty string fails with INVALID_CONFIG. `deviceType`
defaults only when it is absent, so `null` is refused as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The provider's comments, the docs page, and the changeset explained the
long lease window by agent-device starting a lease's window before a slow
allocation returns, which stops being true once agent-device starts it when
the allocation completes. They now say what holds on any agent-device
version: the provider allocates with agent-device's longest lease window and
heartbeats every 2 minutes, so a lease stays alive however long its session
takes to start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A `TESTMU_API_ENDPOINT` with a username or password in it would reach a
failed request's error message. `record` now refuses such an endpoint with
INVALID_CONFIG and a message that does not repeat the value, and reports an
endpoint that is not an http(s) URL as INVALID_CONFIG too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestMu AI's session list filters by build name across the whole
organization, so a build name another user's run shares could find their
session. The lookup now also passes `username`, which the API matches
against the session's owner, set to `LT_USERNAME`. The docs say that a fixed
`build` and `sessionName` shared by overlapping runs is ambiguous, and that
the default names avoid it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two runs in one host process with different credentials could overlap
their daemon calls, and a daemon one of them started would then read the
other's credentials from `process.env`. Daemon calls with the same
credentials still overlap; a call with other credentials now waits until
those have finished. When the last call finishes, a value is put back only
if `process.env` still holds the one the provider set, so a change the host
made in the meantime stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With `stateDir` set to a shared directory such as the system temp directory
or the project root, pruning removed any directory named like a run id that
was more than a day old, whoever made it. The provider now writes a
`.e2e-testmu-run` marker into each run's directory when it leases and
touches it on every heartbeat, and pruning removes only marked directories
in which neither the marker nor any other top-level entry has changed for
24 hours, so a run still going in another process is kept. The unit tests
now use a temporary project root, since acquiring writes the marker there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rocess

A lease's window covers a session that takes up to 10 minutes to start;
the heartbeat only takes over once the lease is granted. In a host that
runs several TestMu AI accounts at once, daemon calls take turns, so one
run's slow allocation can delay another's heartbeats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(testmu): record video, name sessions, and harden the provider
Brings in 49 upstream commits (through 7baf454) so tester-army#772
merges cleanly. Conflicts resolved by keeping upstream and re-adding the
testmu entries: the README package row, the docs workspace devDependency,
and the build and test scripts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@e2e-dev/mobile now pins agent-device 0.21.20, so testmu develops against
the same copy, and its peer floor moves past 0.21.20, which still has no
testmu provider. engines.node matches every other @e2e-dev package after
the oxc loader change. This also drops the lockfile's dangling
agent-device 0.21.18 reference left by the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@amankansal-lt

Copy link
Copy Markdown
Author

I've synced this with main (49 commits); the head is now 3cc2ef4d6 and there are no conflicts.

  • Conflicts were only in README.md, docs/package.json and package.json. In each, main's version is kept and only the testmu additions are re-added.
  • @e2e-dev/testmu now follows main: agent-device devDependency 0.21.20 (matching @e2e-dev/mobile's pin), peer >0.21.20 <1, and Node engines ^22.22.3 || >=24.8.0. The lockfile is regenerated.
  • No testmu source changes were needed.
  • Locally on Node 22.22.3, lint, check:dead-code, typecheck, docs:check, check:peer-ranges, check:install-scripts and test all pass (testmu 73, mobile 232, e2e 3160).

Two upstream items still affect this:

  1. The testmu provider is moving to the @agent-device/testmu plugin (feat(provider-webdriver): add TestMu AI device cloud provider callstack/agent-device#3118). Once it's published, this package will document agent-device plugins add @agent-device/testmu and set its agent-device floor to match. The docs keep the version placeholder until then.
  2. When feat(daemon): keep a caller-owned lease through session close with retainOnClose callstack/agent-device#3208 (retainOnClose) ships, @e2e-dev/testmu will opt in. e2e closes the session when it retires a worker, which would otherwise end the TestMu AI session mid-run.

SahilSawLT and others added 3 commits October 6, 2026 13:58
…owsers()

@e2e-dev/testmu/web exports testmuBrowsers(), a BrowserProvider for TestMu AI
(formerly LambdaTest) hosted Chrome and Edge on Windows and macOS. A session
starts when its CDP websocket opens and ends when the engine closes it, so the
provider only builds the CDP URL from the options and LT_USERNAME /
LT_ACCESS_KEY (via the package's testmuCredentials()); it calls no API.

- One session per worker slot, or per attempt with scope: 'attempt'.
- route /puppeteer (default, raw CDP on every machine) or /playwright-cdp
  (Playwright label; raw CDP only where TestMu AI's newest backend has rolled
  out). hub cdp.lambdatest.com.
- Sessions are named as testmu()'s: build (default the run id, so a run's
  device and browser sessions share one build) and sessionName (default
  e2e-<run id>-<target>-<slot or attempt>); project (default e2e) is sent as
  LT:Options.project. geoLocation and timezone are options, as on testmu().
- idleTimeout defaults to 600 s. Unknown options, out-of-range values, empty
  or null strings, and capabilities that set what the provider sets
  (including browserName, browserVersion and a nested LT:Options) fail with
  INVALID_CONFIG at config load.
- The package root stays device-only; the browser provider lives at
  @e2e-dev/testmu/web, and @e2e-dev/mobile, @e2e-dev/web and agent-device
  are optional peers, so neither side's imports or types reach the other.
- Docs: integrations/testmu-browsers page with a checked example
  (docs/examples/web/testmu-browsers.config.ts), nav, card, env, security,
  browser.mdx, README, AGENTS, skill row; credentials sourcing on both pages.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KoNxaN1YAAP9PwmHtMm1WX
feat(testmu): hosted Chrome and Edge for the web engine with testmuBrowsers()
Keeps upstream's decision and github entries in AGENTS.md alongside the
testmu entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@amankansal-lt

Copy link
Copy Markdown
Author

Thanks for starting CI. The single failure, test (22.22.3) (job), is a timing assertion in packages/e2e/tests/integration/web-platform.test.ts:485: fails a download that never started with the wait and the trigger time. It got NaN where it expects the trigger to have slept at least 190 ms. This PR doesn't touch packages/web or packages/e2e, and the same test passed on Node 22, 24, 24.8.0 and 26 in this run (3036 other tests passed). It looks like a timing flake on that runner. Could you re-run that job?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants