Skip to content

feat(testmu): add @e2e-dev/testmu device provider - #771

Closed
amankansal-lt wants to merge 4 commits into
tester-army:mainfrom
amankansal-lt:feat/testmu-ai-devices
Closed

amankansal-lt wants to merge 4 commits into
tester-army:mainfrom
amankansal-lt:feat/testmu-ai-devices

Conversation

@amankansal-lt

@amankansal-lt amankansal-lt commented Oct 2, 2026 •

Copy link
Copy Markdown

What and why

  • Adds @e2e-dev/testmu, a DeviceProvider that runs mobile targets on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators and real devices, through agent-device's testmu cloud provider (feat(provider-webdriver): add TestMu AI device cloud provider callstack/agent-device#3117).

    import { mobile } from '@e2e-dev/mobile';
    import { testmu } from '@e2e-dev/testmu';
    
    engine: mobile({
      platform: 'android',
      device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: 'lt://APP-id' }),
    }),
  • Leases: each worker slot allocates an agent-device lease. The worker's first command starts the session and installs the app; release ends it. Release is idempotent and also runs when a lease arrives after an interrupt.

  • Credentials: LT_USERNAME and LT_ACCESS_KEY come from the run's environment. If one is missing, the lease fails before anything starts.

  • Options: deviceType: 'real' selects real devices. project, build, sessionName and stateDir are optional. Unknown options fail at config load.

  • Lease keep-alive: agent-device leases expire after 60 s idle, and an iOS simulator's first command (upload, allocation, boot) takes about 70 s. The provider allocates with the daemon's maximum window (10 min) and heartbeats every 2 minutes until release.

  • fix(mobile): XCUIElementType-prefixed iOS element types. agent-device's WebDriver runtime, used by every hosted Appium cloud including BrowserStack and AWS Device Farm, reports XCUIElementTypeButton where the native runner reports Button. Before this, getByRole('button', …) matched nothing on those iOS devices. Patch changeset for @e2e-dev/mobile.

  • Requirement: an agent-device release with the testmu provider (feat(provider-webdriver): add TestMu AI device cloud provider callstack/agent-device#3117). agent-device is a peer (>0.21.18 <1) so the provider and the engine share one copy. Until @e2e-dev/mobile pins such a release, the docs page shows a package-manager override, with the version left as a placeholder to fill on release.

Verified

Ran it locally: yes

  • @e2e-dev/testmu unit tests: 28 passed; @e2e-dev/mobile unit tests: 237 passed
  • pnpm lint, pnpm -r run typecheck, pnpm docs:check, check-dist, check:peer-ranges: pass
  • Live against TestMu AI production with the packed packages from this branch. Each test opens the app and checks it with getByRole / getByText; every device was released afterwards.
Target Device App Engine ready Test
Android emulator Galaxy S22 Ultra 5G, 14 .apk by URL 54 s passed
iOS simulator iPhone 16, 18.0 local zipped .app 71 s passed
Android real device Pixel 6, 14 .apk by URL 27 s passed
iOS real device iPhone 16, 18 signed .ipa 29 s passed

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md). n/a, not breaking.
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures. n/a
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile. n/a

🤖 Generated with Claude Code

Review in cubic

amankansal-lt and others added 4 commits October 2, 2026 19:55
Add @e2e-dev/testmu, a DeviceProvider for the mobile engine that runs
targets on TestMu AI (formerly LambdaTest) Android emulators, iOS
simulators, and real devices through agent-device's testmu cloud
provider:

  mobile({ device: testmu({ device, osVersion, app }) })

Each worker slot allocates an agent-device lease from a daemon the
provider starts for the run under stateDir (default .e2e/testmu). The
lease's client configuration carries the lease scope and the device
selectors, so the worker's first command starts the TestMu AI session
and installs the app; releasing the lease ends it.

- Credentials come from LT_USERNAME and LT_ACCESS_KEY in the run's
  environment; a missing one fails the lease before anything starts.
  agent-device's client spawns its daemon with process.env and takes no
  environment of its own, so the provider copies the run's values there
  before the first allocation.
- A lease granted after an interrupt, or one that cannot be handed to
  the run, is released before acquire throws. Release is idempotent and
  works from the lease alone, after a round trip through JSON.
- A relative local app path resolves against the project root, not the
  daemon's working directory. The target's app.appPath is refused, since
  TestMu AI installs the app itself.
- Unknown options, missing device/osVersion/app, and a deviceType other
  than virtual or real fail with INVALID_CONFIG.

agent-device is a peer (>0.21.18 <1): the provider and the engine's
workers must share one agent-device, and no published release includes
the testmu provider yet, so the range excludes only the releases known
to lack it. No record hook: TestMu AI records whole sessions, not
attempts, and its video is only available once the session ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a TestMu AI (formerly LambdaTest) integration page for the testmu()
device provider from @e2e-dev/testmu, in the shape of the EAS Simulators
page: setup, authentication, installing the app, real devices, an
options table, sessions, what the provider does, and limits.

The page covers the requirement for an agent-device that includes the
testmu provider and the override that makes @e2e-dev/mobile use the same
copy, the catalog-exact device names and OS versions (18.0 for a virtual
iOS device, 18 for a real one, 14 on Android), the app formats per
device, and the limits of agent-device's WebDriver runtime.

The config and test examples are shown verbatim and typecheck against
the built package. The integrations overview, the hosted devices section
of the mobile page, the environment reference (LT_USERNAME,
LT_ACCESS_KEY), and the agent skill's setup reference name the package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-device ends a lease after 60 seconds without activity by default,
and a command still running does not count as activity. The worker's first
command uploads the app, allocates the hosted session, and boots it; on an
iOS simulator that takes about 70 seconds, so the lease lapsed mid-start
and the engine's prepare failed with "boot failed: Lease is not active"
(LEASE_NOT_FOUND).

Each lease now asks for agent-device's longest inactivity window (10
minutes), and the provider heartbeats every lease it holds every 2 minutes
from the runner process until the lease is released, including the
release acquire does itself when it cannot hand a lease to the run. The
timer is unref'd so it never keeps the runner alive. A failed heartbeat is
logged once per lease and the next one is still tried; it never fails the
run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-device's native iOS runner reports element types as `Button`,
`StaticText`, `NavigationBar`. Its WebDriver runtime, which hosted Appium
clouds (TestMu AI, BrowserStack, AWS Device Farm) go through, reports the
XCUITest class names instead: `XCUIElementTypeButton`,
`XCUIElementTypeStaticText`. normalizeKind kebab-cased those to
`xcuielement-type-button`, which maps to no role, so on a real iPhone
`getByRole('button', 'CPU Load')` matched nothing and observe listed the
node by its raw class name.

normalizeKind now drops a leading `XCUIElementType` before kebab-casing,
so both spellings land on one vocabulary: buttons, tab bar items as tabs,
the navigation bar as the screen title. Only that exact prefix followed by
a type name is stripped; Android class names are untouched, and the golden
device trees project unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@amankansal-lt

Copy link
Copy Markdown
Author

Superseded by #772, opened from the LambdaTest fork with the same commits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant