feat(testmu): add @e2e-dev/testmu device provider - #772
amankansal-lt wants to merge 29 commits into
Conversation
Add @e2e-dev/testmu, a DeviceProvider for the mobile engine that runs
targets on TestMu AI (formerly LambdaTest) Android emulators, iOS
simulators, and real devices through agent-device's testmu cloud
provider:
mobile({ device: testmu({ device, osVersion, app }) })
Each worker slot allocates an agent-device lease from a daemon the
provider starts for the run under stateDir (default .e2e/testmu). The
lease's client configuration carries the lease scope and the device
selectors, so the worker's first command starts the TestMu AI session
and installs the app; releasing the lease ends it.
- Credentials come from LT_USERNAME and LT_ACCESS_KEY in the run's
environment; a missing one fails the lease before anything starts.
agent-device's client spawns its daemon with process.env and takes no
environment of its own, so the provider copies the run's values there
before the first allocation.
- A lease granted after an interrupt, or one that cannot be handed to
the run, is released before acquire throws. Release is idempotent and
works from the lease alone, after a round trip through JSON.
- A relative local app path resolves against the project root, not the
daemon's working directory. The target's app.appPath is refused, since
TestMu AI installs the app itself.
- Unknown options, missing device/osVersion/app, and a deviceType other
than virtual or real fail with INVALID_CONFIG.
agent-device is a peer (>0.21.18 <1): the provider and the engine's
workers must share one agent-device, and no published release includes
the testmu provider yet, so the range excludes only the releases known
to lack it. No record hook: TestMu AI records whole sessions, not
attempts, and its video is only available once the session ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a TestMu AI (formerly LambdaTest) integration page for the testmu() device provider from @e2e-dev/testmu, in the shape of the EAS Simulators page: setup, authentication, installing the app, real devices, an options table, sessions, what the provider does, and limits. The page covers the requirement for an agent-device that includes the testmu provider and the override that makes @e2e-dev/mobile use the same copy, the catalog-exact device names and OS versions (18.0 for a virtual iOS device, 18 for a real one, 14 on Android), the app formats per device, and the limits of agent-device's WebDriver runtime. The config and test examples are shown verbatim and typecheck against the built package. The integrations overview, the hosted devices section of the mobile page, the environment reference (LT_USERNAME, LT_ACCESS_KEY), and the agent skill's setup reference name the package. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-device ends a lease after 60 seconds without activity by default, and a command still running does not count as activity. The worker's first command uploads the app, allocates the hosted session, and boots it; on an iOS simulator that takes about 70 seconds, so the lease lapsed mid-start and the engine's prepare failed with "boot failed: Lease is not active" (LEASE_NOT_FOUND). Each lease now asks for agent-device's longest inactivity window (10 minutes), and the provider heartbeats every lease it holds every 2 minutes from the runner process until the lease is released, including the release acquire does itself when it cannot hand a lease to the run. The timer is unref'd so it never keeps the runner alive. A failed heartbeat is logged once per lease and the next one is still tried; it never fails the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-device's native iOS runner reports element types as `Button`,
`StaticText`, `NavigationBar`. Its WebDriver runtime, which hosted Appium
clouds (TestMu AI, BrowserStack, AWS Device Farm) go through, reports the
XCUITest class names instead: `XCUIElementTypeButton`,
`XCUIElementTypeStaticText`. normalizeKind kebab-cased those to
`xcuielement-type-button`, which maps to no role, so on a real iPhone
`getByRole('button', 'CPU Load')` matched nothing and observe listed the
node by its raw class name.
normalizeKind now drops a leading `XCUIElementType` before kebab-casing,
so both spellings land on one vocabulary: buttons, tab bar items as tabs,
the navigation bar as the screen title. Only that exact prefix followed by
a type name is stripped; Android class names are untouched, and the golden
device trees project unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(testmu): add @e2e-dev/testmu device provider
There was a problem hiding this comment.
All reported issues were addressed across 30 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
agent-device's `leases.allocate` creates the TestMu AI session before it returns, so the session starts at acquire, not on the worker's first command, and every slot is billed from the moment it is leased. The lease log line, the provider's comments, the README, the docs page, and the changeset now say so, and explain the long lease window and heartbeat by agent-device starting a lease's window before a slow allocation returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without `sessionName`, each slot's session is now named `e2e-<run id>-<target>-<slot>` instead of whatever TestMu AI picks, and a given `sessionName` gets `-<slot>` appended when the target leases more than one device. Every slot's session then has a name of its own within its build, which is what finding the session again by name relies on. The lease log line names the session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` set up the device when its session starts. Each one is allocated under agent-device's matching lease key (`providerDeviceOrientation`, `providerGeoLocation`, `providerTimezone`, `providerLanguage`, `providerLocale`, `providerAppiumVersion`), which its `testmu` provider turns into TestMu AI capabilities. An empty value, or an orientation other than portrait or landscape, fails the config load with INVALID_CONFIG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-device's cloud WebDriver runtime cannot record, so a run with `video: 'on'` or `'retain-on-failure'` on a TestMu AI target failed when the attempt started its video. The provider now implements `record`: the attempt's video links TestMu AI's own recording of the slot's session. `record` runs in the worker from the lease alone. It reads the build and session name the lease carries and the credentials from the run's environment, and starts at the moment it is called. `stop` finds the session through TestMu AI's sessions API (the newest session with that name in the build's list, paged) and returns the `video_url` of its details as `video/mp4`. The API defaults to https://mobile-api.lambdatest.com/mobile-automation/api/v1 and follows `TESTMU_API_ENDPOINT`, as agent-device does. Every request is bounded by a 15-second timeout and the stop's signal, and errors name the build and the session without credentials or URLs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each run leaves `<stateDir>/<run id>` behind with its daemon's state and logs, and nothing removed it. The directory is still kept at release, since the daemon exits by itself after 5 idle minutes and its logs help with a failure. Instead, the provider's first acquire removes earlier runs' directories under `stateDir` last modified more than 24 hours ago. It never touches the current run's directory or anything not named like a run id, and a failure leaves the directory and does not fail the lease. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
State that `app` is required and that `app.appPath` is refused on purpose, since the provider hands TestMu AI the build itself, while `device.installApp()` with a path still works during a test. State that the provider copies `LT_USERNAME` and `LT_ACCESS_KEY` into the runner's `process.env`, where the agent-device daemon it starts reads them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A provider recording's `startedAt` is the time of the video's first frame, and TestMu AI's video covers the whole session, so the time `record` was called put every timeline offset off by however long the session had run. `record` now finds the session in TestMu AI's sessions list by build and name, as `stop` did, and takes the row's `start_timestamp` (RFC 3339; a time without a zone is read as UTC) as `startedAt`. `stop` fetches the `video_url` of the session `record` found. Without a usable start time, `startedAt` falls back to when `record` was called. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The TestMu AI example config ran its test, which checks the Proverbial sample app, on an iOS simulator target configured for a different app. Every target now runs Proverbial: the iOS target is a real iPhone with TestMu AI's public Proverbial `.ipa`, and the test matches the geolocation button's label on both platforms. An iOS simulator target is shown separately in the build section. The hosted-devices paragraph in the mobile guide names the TestMu AI provider with the same structure as the EAS one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent-device's client starts its local daemon with `process.env` and takes no environment of its own, so the provider copied `LT_USERNAME` and `LT_ACCESS_KEY` into `process.env` and left them there, where a long-lived host would pass them to unrelated child processes. The provider now sets them only around its daemon calls (allocate, heartbeat, and release, any of which may start the daemon) and puts back the previous values once no such call is in flight. A release without credentials in its environment still goes through the daemon that is already running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`project`, `build`, `sessionName`, and `stateDir` were passed through unchecked, so a JavaScript config with `project: 1` or `stateDir: 1` reached the allocation or failed inside `acquire`. They are now checked with the device-feature options when `testmu()` is called, and any of them that is not a non-empty string fails with INVALID_CONFIG. `deviceType` defaults only when it is absent, so `null` is refused as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The provider's comments, the docs page, and the changeset explained the long lease window by agent-device starting a lease's window before a slow allocation returns, which stops being true once agent-device starts it when the allocation completes. They now say what holds on any agent-device version: the provider allocates with agent-device's longest lease window and heartbeats every 2 minutes, so a lease stays alive however long its session takes to start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A `TESTMU_API_ENDPOINT` with a username or password in it would reach a failed request's error message. `record` now refuses such an endpoint with INVALID_CONFIG and a message that does not repeat the value, and reports an endpoint that is not an http(s) URL as INVALID_CONFIG too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TestMu AI's session list filters by build name across the whole organization, so a build name another user's run shares could find their session. The lookup now also passes `username`, which the API matches against the session's owner, set to `LT_USERNAME`. The docs say that a fixed `build` and `sessionName` shared by overlapping runs is ambiguous, and that the default names avoid it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two runs in one host process with different credentials could overlap their daemon calls, and a daemon one of them started would then read the other's credentials from `process.env`. Daemon calls with the same credentials still overlap; a call with other credentials now waits until those have finished. When the last call finishes, a value is put back only if `process.env` still holds the one the provider set, so a change the host made in the meantime stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With `stateDir` set to a shared directory such as the system temp directory or the project root, pruning removed any directory named like a run id that was more than a day old, whoever made it. The provider now writes a `.e2e-testmu-run` marker into each run's directory when it leases and touches it on every heartbeat, and pruning removes only marked directories in which neither the marker nor any other top-level entry has changed for 24 hours, so a run still going in another process is kept. The unit tests now use a temporary project root, since acquiring writes the marker there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rocess A lease's window covers a session that takes up to 10 minutes to start; the heartbeat only takes over once the lease is granted. In a host that runs several TestMu AI accounts at once, daemon calls take turns, so one run's slow allocation can delay another's heartbeats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(testmu): record video, name sessions, and harden the provider
Brings in 49 upstream commits (through 7baf454) so tester-army#772 merges cleanly. Conflicts resolved by keeping upstream and re-adding the testmu entries: the README package row, the docs workspace devDependency, and the build and test scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@e2e-dev/mobile now pins agent-device 0.21.20, so testmu develops against the same copy, and its peer floor moves past 0.21.20, which still has no testmu provider. engines.node matches every other @e2e-dev package after the oxc loader change. This also drops the lockfile's dangling agent-device 0.21.18 reference left by the merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chore: sync with tester-army/e2e main
|
I've synced this with
Two upstream items still affect this:
|
…owsers() @e2e-dev/testmu/web exports testmuBrowsers(), a BrowserProvider for TestMu AI (formerly LambdaTest) hosted Chrome and Edge on Windows and macOS. A session starts when its CDP websocket opens and ends when the engine closes it, so the provider only builds the CDP URL from the options and LT_USERNAME / LT_ACCESS_KEY (via the package's testmuCredentials()); it calls no API. - One session per worker slot, or per attempt with scope: 'attempt'. - route /puppeteer (default, raw CDP on every machine) or /playwright-cdp (Playwright label; raw CDP only where TestMu AI's newest backend has rolled out). hub cdp.lambdatest.com. - Sessions are named as testmu()'s: build (default the run id, so a run's device and browser sessions share one build) and sessionName (default e2e-<run id>-<target>-<slot or attempt>); project (default e2e) is sent as LT:Options.project. geoLocation and timezone are options, as on testmu(). - idleTimeout defaults to 600 s. Unknown options, out-of-range values, empty or null strings, and capabilities that set what the provider sets (including browserName, browserVersion and a nested LT:Options) fail with INVALID_CONFIG at config load. - The package root stays device-only; the browser provider lives at @e2e-dev/testmu/web, and @e2e-dev/mobile, @e2e-dev/web and agent-device are optional peers, so neither side's imports or types reach the other. - Docs: integrations/testmu-browsers page with a checked example (docs/examples/web/testmu-browsers.config.ts), nav, card, env, security, browser.mdx, README, AGENTS, skill row; credentials sourcing on both pages. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoNxaN1YAAP9PwmHtMm1WX
feat(testmu): hosted Chrome and Edge for the web engine with testmuBrowsers()
Keeps upstream's decision and github entries in AGENTS.md alongside the testmu entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chore: sync with tester-army/e2e main
|
Thanks for starting CI. The single failure, |
What and why
Adds
@e2e-dev/testmu, aDeviceProviderthat runs mobile targets on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators and real devices, through agent-device'stestmucloud provider (feat(provider-webdriver): add TestMu AI device cloud provider callstack/agent-device#3118).Adds
testmuBrowsers(), exported from@e2e-dev/testmu/web: aBrowserProviderfor@e2e-dev/webrunning TestMu AI hosted Chrome and Edge on Windows and macOS. This consolidates the browser side from feat(testmuai): @e2e-dev/testmuai, TestMu AI hosted browsers for the web engine #765, as asked when that PR closed. A session starts when its CDP websocket opens and ends when it closes, so the provider only builds the CDP URL; there's no API call and no SDK. The package root stays device-only, and/webnever loads agent-device or@e2e-dev/mobile. All three peers are optional, so a project installs only the side it uses.Leases: each worker slot allocates an agent-device lease. The worker's first command starts the session and installs the app; release ends it. Release is idempotent and also runs when a lease arrives after an interrupt.
Credentials:
LT_USERNAMEandLT_ACCESS_KEYcome from the run's environment. If one is missing, the lease fails before anything starts.Options:
deviceType: 'real'selects real devices.project,build,sessionNameandstateDirare optional. Unknown options fail at config load.Lease keep-alive: agent-device leases expire after 60 s idle, and an iOS simulator's first command (upload, allocation, boot) takes about 70 s. The provider allocates with the daemon's maximum window (10 min) and heartbeats every 2 minutes until release.
fix(mobile):XCUIElementType-prefixed iOS element types. agent-device's WebDriver runtime, used by every hosted Appium cloud including BrowserStack and AWS Device Farm, reportsXCUIElementTypeButtonwhere the native runner reportsButton. Before this,getByRole('button', …)matched nothing on those iOS devices. Patch changeset for@e2e-dev/mobile.Requirement (devices only): the
testmuprovider for agent-device, which is now the managed plugin@agent-device/testmuin feat(provider-webdriver): add TestMu AI device cloud provider callstack/agent-device#3118. agent-device is a peer (>0.21.20 <1), so the provider and the engine share one copy. Until a release ships, the docs page keeps the version placeholder. Browsers need nothing beyond@e2e-dev/web.Since opening
DeviceProvider.record(). An attempt that records video links TestMu AI's recording of the slot's session, which is looked up by build, session name and user. ItsstartedAtis the session's start time. Without this,video: 'on'andretain-on-failurefailed, because agent-device's WebDriver runtime cannot record.e2e-<run id>-<target>-<slot>by default. The docs now say the session starts when the lease is allocated, which is also when billing starts.orientation,geoLocation,timezone,language,locale,appiumVersion.process.envonly while the provider calls the daemon.testmuBrowsers()(co-authored by @SahilSawLT).testmu()'s conventions:project,buildandsessionNamedefaults, top-levelgeoLocationandtimezone, strict validation, andcapabilitiesthat can't override what the provider sets.worker(one session per slot) orattempt./puppeteer(the default) or/playwright-cdp, which is still rolling out on TestMu AI's side.maintwice (the README, docspackage.json, rootpackage.jsonandAGENTS.mdlists).@e2e-dev/testmufollows@e2e-dev/mobile's agent-device 0.21.20 pin and its Node engines.video: 'on'(the attempt links the session'svideo.mp4), iOS simulator and a real iPhone all passed.Verified
Ran it locally: yes
pnpm checkpasses: lint, dead code, typecheck, error codes, peer ranges, install scripts, docs and broken links.@e2e-dev/testmu108 passed,@e2e-dev/mobile232 passed,@e2e-dev/web504 passed (1 skipped).getByRole/getByText; every device was released afterwards..apkby URL.app.apkby URL.ipaBrowsers on TestMu AI production, with a browser-only install of the packed package (
e2e0.17.0 and@e2e-dev/web0.12.0). npm pulled neither agent-device nor@e2e-dev/mobile:/puppeteer(default)<run id>/playwright-cdpThe platform issues we've seen are documented under Limits and reported to TestMu AI. They reproduce with plain Puppeteer and Playwright, so they aren't provider bugs:
500on Windows 11.geoLocationDEandFRexit in neighbouring countries.Checklist
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.!, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md). n/a, not breaking.docs/**/*.mdxpage for the behavior, andskills/e2e/if the skill describes it..d.tsreviewed,tests/types/sdk-types.tsupdated; wire change edits the schema and both fixtures. n/ae2e/engine): changesets fore2e,@e2e-dev/web, and@e2e-dev/mobile. n/a🤖 Generated with Claude Code