From 125aa03612de12459c68a0d41ceafc1a38ce71a3 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 19:55:37 +0530 Subject: [PATCH 01/22] feat(testmu): add TestMu AI device provider package Add @e2e-dev/testmu, a DeviceProvider for the mobile engine that runs targets on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices through agent-device's testmu cloud provider: mobile({ device: testmu({ device, osVersion, app }) }) Each worker slot allocates an agent-device lease from a daemon the provider starts for the run under stateDir (default .e2e/testmu). The lease's client configuration carries the lease scope and the device selectors, so the worker's first command starts the TestMu AI session and installs the app; releasing the lease ends it. - Credentials come from LT_USERNAME and LT_ACCESS_KEY in the run's environment; a missing one fails the lease before anything starts. agent-device's client spawns its daemon with process.env and takes no environment of its own, so the provider copies the run's values there before the first allocation. - A lease granted after an interrupt, or one that cannot be handed to the run, is released before acquire throws. Release is idempotent and works from the lease alone, after a round trip through JSON. - A relative local app path resolves against the project root, not the daemon's working directory. The target's app.appPath is refused, since TestMu AI installs the app itself. - Unknown options, missing device/osVersion/app, and a deviceType other than virtual or real fail with INVALID_CONFIG. agent-device is a peer (>0.21.18 <1): the provider and the engine's workers must share one agent-device, and no published release includes the testmu provider yet, so the range excludes only the releases known to lack it. No record hook: TestMu AI records whole sessions, not attempts, and its video is only available once the session ends. Co-Authored-By: Claude Opus 5.5 --- .changeset/testmu-devices.md | 5 + AGENTS.md | 6 + CONTRIBUTING.md | 2 +- README.md | 1 + package.json | 4 +- packages/testmu/LICENSE | 202 +++++++++++++++ packages/testmu/NOTICE | 5 + packages/testmu/README.md | 66 +++++ packages/testmu/package.json | 73 ++++++ packages/testmu/src/credentials.ts | 45 ++++ packages/testmu/src/index.ts | 8 + packages/testmu/src/provider.ts | 193 +++++++++++++++ packages/testmu/tests/unit/testmu.test.ts | 284 ++++++++++++++++++++++ packages/testmu/tsconfig.build.json | 11 + packages/testmu/tsconfig.json | 9 + packages/testmu/vitest.config.ts | 10 + pnpm-lock.yaml | 24 ++ 17 files changed, 945 insertions(+), 3 deletions(-) create mode 100644 .changeset/testmu-devices.md create mode 100644 packages/testmu/LICENSE create mode 100644 packages/testmu/NOTICE create mode 100644 packages/testmu/README.md create mode 100644 packages/testmu/package.json create mode 100644 packages/testmu/src/credentials.ts create mode 100644 packages/testmu/src/index.ts create mode 100644 packages/testmu/src/provider.ts create mode 100644 packages/testmu/tests/unit/testmu.test.ts create mode 100644 packages/testmu/tsconfig.build.json create mode 100644 packages/testmu/tsconfig.json create mode 100644 packages/testmu/vitest.config.ts diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md new file mode 100644 index 000000000..c19d878ec --- /dev/null +++ b/.changeset/testmu-devices.md @@ -0,0 +1,5 @@ +--- +"@e2e-dev/testmu": minor +--- + +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); the worker's first command starts the TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id), releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/AGENTS.md b/AGENTS.md index 6340a3daf..c12d1fee0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,6 +76,12 @@ suites that consume the built packages the way a user would. hosted iOS simulators and Android emulators for the mobile engine (`DeviceProvider`). Expo publishes no SDK for the sessions API, so it calls Expo's GraphQL API with `fetch`, and `@e2e-dev/mobile` is its only peer. +- `packages/testmu` - the published `@e2e-dev/testmu` package: TestMu AI + (formerly LambdaTest) hosted Android emulators, iOS simulators, and real + devices for the mobile engine (`DeviceProvider`). agent-device's `testmu` + cloud provider is the vendor client: the package allocates agent-device + leases from a daemon it starts per run, so `agent-device` (which must be + the same copy `@e2e-dev/mobile` uses) and `@e2e-dev/mobile` are its peers. - `apps/testbed` (`@e2e-dev/testbed`, private) — dogfood project that consumes the **built** packages like a real user would: the playground app where every runner feature (sessions, routes, downloads, frames, uploads, diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cc2969edf..93827362c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -254,5 +254,5 @@ build against, and what a third-party engine builds against too. A change to that contract bumps all three packages together in one release, with a changeset for each, so an engine and a runner from the same release always match. Engines declare a peer range on `e2e` that points one way only -(engine to runner, `>=x <1`), and each integration (`@e2e-dev/kernel`, `@e2e-dev/eas`) does +(engine to runner, `>=x <1`), and each integration (`@e2e-dev/kernel`, `@e2e-dev/eas`, `@e2e-dev/testmu`) does the same on the engines it plugs into; do not make it mutual or narrow it. diff --git a/README.md b/README.md index 9ebbc17c1..0bcb1f5a2 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,7 @@ config and an example test. The | [`@e2e-dev/github`](https://www.npmjs.com/package/@e2e-dev/github) | Reporter that posts results as a pull request comment. | | [`@e2e-dev/kernel`](https://www.npmjs.com/package/@e2e-dev/kernel) | Kernel hosted browsers for the web engine. | | [`@e2e-dev/eas`](https://www.npmjs.com/package/@e2e-dev/eas) | EAS Simulators hosted iOS simulators and Android emulators for the mobile engine. | +| [`@e2e-dev/testmu`](https://www.npmjs.com/package/@e2e-dev/testmu) | TestMu AI (formerly LambdaTest) hosted Android emulators, iOS simulators, and real devices for the mobile engine. | ## Documentation diff --git a/package.json b/package.json index 4d515218b..3fe17722d 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,7 @@ ], "type": "module", "scripts": { - "build": "pnpm --filter e2e run build && pnpm --filter @e2e-dev/web run build && pnpm --filter @e2e-dev/mobile run build && pnpm --filter @e2e-dev/github run build && pnpm --filter @e2e-dev/kernel run build && pnpm --filter @e2e-dev/eas run build", + "build": "pnpm --filter e2e run build && pnpm --filter @e2e-dev/web run build && pnpm --filter @e2e-dev/mobile run build && pnpm --filter @e2e-dev/github run build && pnpm --filter @e2e-dev/kernel run build && pnpm --filter @e2e-dev/eas run build && pnpm --filter @e2e-dev/testmu run build", "check": "pnpm run lint && pnpm run check:dead-code && pnpm run typecheck && pnpm run docs:check-errors && pnpm run check:peer-ranges && pnpm run docs:check", "check:dead-code": "fallow dead-code", "check:peer-ranges": "node scripts/check-peer-ranges.ts", @@ -24,7 +24,7 @@ "canary": "node scripts/canary-changeset.ts && changeset version --snapshot canary && pnpm run build", "canary:publish": "changeset publish --tag canary --no-git-tag && node scripts/restore-peer-ranges.ts", "typecheck": "pnpm run build && pnpm -r run typecheck && tsc -p scripts", - "test": "pnpm run build && pnpm --filter e2e run test && pnpm --filter @e2e-dev/web run test && pnpm --filter @e2e-dev/mobile run test && pnpm --filter @e2e-dev/github run test && pnpm --filter @e2e-dev/kernel run test && pnpm --filter @e2e-dev/eas run test && pnpm run test:scripts", + "test": "pnpm run build && pnpm --filter e2e run test && pnpm --filter @e2e-dev/web run test && pnpm --filter @e2e-dev/mobile run test && pnpm --filter @e2e-dev/github run test && pnpm --filter @e2e-dev/kernel run test && pnpm --filter @e2e-dev/eas run test && pnpm --filter @e2e-dev/testmu run test && pnpm run test:scripts", "test:testbed": "pnpm run build && pnpm --filter @e2e-dev/testbed test", "test:web-benchmark": "pnpm run build && pnpm --filter @e2e-dev/web-benchmark test", "test:mobile-benchmark": "pnpm run build && pnpm --filter @e2e-dev/mobile-benchmark test", diff --git a/packages/testmu/LICENSE b/packages/testmu/LICENSE new file mode 100644 index 000000000..d64569567 --- /dev/null +++ b/packages/testmu/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/packages/testmu/NOTICE b/packages/testmu/NOTICE new file mode 100644 index 000000000..a5f5c8473 --- /dev/null +++ b/packages/testmu/NOTICE @@ -0,0 +1,5 @@ +@e2e-dev/testmu +Copyright 2026 TesterArmy, Inc. + +This product includes software developed at TesterArmy, Inc. +(https://tester.army). diff --git a/packages/testmu/README.md b/packages/testmu/README.md new file mode 100644 index 000000000..09dea7f87 --- /dev/null +++ b/packages/testmu/README.md @@ -0,0 +1,66 @@ +# @e2e-dev/testmu + +[TestMu AI](https://www.lambdatest.com) (formerly LambdaTest) for [`e2e`](https://www.npmjs.com/package/e2e): +`mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on +TestMu AI's hosted Android emulators, iOS simulators, and real devices. + +## Install + +```bash +npm install --save-dev @e2e-dev/testmu agent-device@ +``` + +The provider drives TestMu AI through agent-device's `testmu` cloud +provider, so the project needs an agent-device that includes it, and +`@e2e-dev/mobile` must use that same agent-device: override its pin (npm and +bun `overrides`, pnpm `overrides` in `pnpm-workspace.yaml`). No published +agent-device release includes the `testmu` provider yet; `` is the +first one that does. The peer range only excludes the releases known to lack +it. + +## Usage + +```ts title="e2e.config.ts" +import type { E2EConfig } from 'e2e'; +import { mobile } from '@e2e-dev/mobile'; +import { testmu } from '@e2e-dev/testmu'; + +export default { + targets: [ + { + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: 'https://example.com/app.apk' }), + }), + app: { bundleId: 'com.example.app' }, + }, + ], + workers: 2, +} satisfies E2EConfig; +``` + +It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's +environment. Each worker slot leases one device from an agent-device daemon +the provider starts for the run; the worker's first command starts the +TestMu AI session, which installs `app`, and the lease is released when the +run ends, which ends the session. + +- `device` and `osVersion` must match TestMu AI's catalog exactly: `18.0` + for a virtual iOS device, `18` for a real one, `14` on Android. +- `app` is an `lt://` app id, an `https` URL, or a local path resolved + against the project root. Keep the target's `app.bundleId` and leave + `app.appPath` out. +- `deviceType: 'real'` picks a real device (default `'virtual'`). +- `project` (default `e2e`), `build` (default the run id), and `sessionName` + label the sessions on the dashboard. +- `stateDir` (default `.e2e/testmu`) holds each run's daemon. + +Sessions run over TestMu AI's Appium hub, so agent-device's device settings, +system alerts, recording, device logs, and port reverse are not available +there. TestMu AI records every session itself. + +Full documentation lives at [e2e.tester.army/docs/integrations/testmu](https://e2e.tester.army/docs/integrations/testmu). + +## License + +Apache-2.0 diff --git a/packages/testmu/package.json b/packages/testmu/package.json new file mode 100644 index 000000000..f4fc14f91 --- /dev/null +++ b/packages/testmu/package.json @@ -0,0 +1,73 @@ +{ + "name": "@e2e-dev/testmu", + "version": "0.0.0", + "description": "TestMu AI (formerly LambdaTest) for e2e: run mobile targets on hosted Android emulators, iOS simulators, and real devices", + "keywords": [ + "e2e", + "end-to-end", + "testing", + "mobile-testing", + "testmu", + "lambdatest", + "device-cloud", + "real-devices", + "ios-simulator", + "android-emulator", + "agent-device", + "ci" + ], + "license": "Apache-2.0", + "contributors": [ + "Oskar Kwaśniewski ", + "Szymon Rybczak " + ], + "repository": { + "type": "git", + "url": "git+https://github.com/tester-army/e2e.git", + "directory": "packages/testmu" + }, + "bugs": { + "url": "https://github.com/tester-army/e2e/issues" + }, + "homepage": "https://github.com/tester-army/e2e/tree/main/packages/testmu#readme", + "publishConfig": { + "registry": "https://registry.npmjs.org/", + "access": "public", + "tag": "latest" + }, + "type": "module", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "default": "./dist/index.js" + } + }, + "files": [ + "dist", + "NOTICE" + ], + "scripts": { + "build": "node -e \"require('node:fs').rmSync('dist',{recursive:true,force:true})\" && tsc --project tsconfig.build.json && node ../../scripts/stamp-dist.ts", + "prepublishOnly": "node ../../scripts/check-dist.ts", + "typecheck": "tsc --noEmit", + "test": "vitest run", + "test:watch": "vitest", + "test:unit": "vitest run tests/unit" + }, + "peerDependencies": { + "@e2e-dev/mobile": ">=0.9.0 <1", + "agent-device": ">0.21.18 <1", + "e2e": ">=0.15.0 <1" + }, + "devDependencies": { + "@e2e-dev/mobile": "workspace:*", + "@types/node": "26.6.2", + "agent-device": "0.21.18", + "e2e": "workspace:*", + "typescript": "7.0.2", + "vitest": "5.0.1" + }, + "engines": { + "node": ">=22.12.0" + } +} diff --git a/packages/testmu/src/credentials.ts b/packages/testmu/src/credentials.ts new file mode 100644 index 000000000..faa6b987e --- /dev/null +++ b/packages/testmu/src/credentials.ts @@ -0,0 +1,45 @@ +/** + * The TestMu AI credentials a run authenticates with: `LT_USERNAME` and + * `LT_ACCESS_KEY` from the run's environment. agent-device's `testmu` + * runtime reads them from the environment of the daemon that drives the + * sessions, not from a request. + */ + +const LT_USERNAME = 'LT_USERNAME'; +const LT_ACCESS_KEY = 'LT_ACCESS_KEY'; + +interface TestmuCredentials { + readonly username: string; + readonly accessKey: string; +} + +/** `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment; throws naming each one that is unset or blank. */ +export function testmuCredentials(env: Readonly>): TestmuCredentials { + const username = envValue(env, LT_USERNAME); + const accessKey = envValue(env, LT_ACCESS_KEY); + if (username === undefined || accessKey === undefined) { + const missing = [username === undefined ? LT_USERNAME : undefined, accessKey === undefined ? LT_ACCESS_KEY : undefined].filter((name) => name !== undefined); + throw new Error( + `${missing.join(' and ')} ${missing.length === 1 ? 'is' : 'are'} not set; set ${LT_USERNAME} and ${LT_ACCESS_KEY} to your TestMu AI username and access key in the environment \`e2e run\` starts in`, + ); + } + return { username, accessKey }; +} + +/** + * Puts the run's credentials in this process's environment, which is where + * the daemon gets them: agent-device's client starts its local daemon with + * `process.env` and takes no environment of its own, while a run's + * environment can differ from `process.env` (a host that passes `env`). + * Every worker of the run already starts with these values. + */ +export function shareWithDaemon(credentials: TestmuCredentials): void { + if (process.env[LT_USERNAME] !== credentials.username) process.env[LT_USERNAME] = credentials.username; + if (process.env[LT_ACCESS_KEY] !== credentials.accessKey) process.env[LT_ACCESS_KEY] = credentials.accessKey; +} + +/** A non-empty variable from the run's environment, trimmed, or `undefined`. */ +function envValue(env: Readonly>, name: string): string | undefined { + const value = env[name]?.trim(); + return value === undefined || value === '' ? undefined : value; +} diff --git a/packages/testmu/src/index.ts b/packages/testmu/src/index.ts new file mode 100644 index 000000000..65b060dfa --- /dev/null +++ b/packages/testmu/src/index.ts @@ -0,0 +1,8 @@ +/** + * `@e2e-dev/testmu` public surface: `testmu()`, a device provider that leases + * TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real + * devices for `@e2e-dev/mobile` through agent-device's `testmu` provider. + */ + +export { testmu } from './provider.ts'; +export type { TestmuOptions } from './provider.ts'; diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts new file mode 100644 index 000000000..0198f890e --- /dev/null +++ b/packages/testmu/src/provider.ts @@ -0,0 +1,193 @@ +/** + * TestMu AI's hosted Android emulators, iOS simulators, and real devices as a + * `DeviceProvider` for the mobile engine, through agent-device's `testmu` + * cloud provider: the provider allocates agent-device leases, and the daemon + * holding them runs each session over TestMu AI's Appium hub. + */ + +import { isAbsolute, resolve } from 'node:path'; +import type { DeviceLease, DeviceProvider, DeviceRequest } from '@e2e-dev/mobile'; +import { createAgentDeviceClient } from 'agent-device'; +import { ConfigurationError, rejectUnknownKeys } from 'e2e/engine'; +import { shareWithDaemon, testmuCredentials } from './credentials.ts'; + +/** agent-device's name for TestMu AI, as the lease provider and the tenant. */ +const PROVIDER = 'testmu'; + +/** Where each run's daemon keeps its state, under the project root. */ +const DEFAULT_STATE_DIR = '.e2e/testmu'; + +/** The dashboard project sessions are grouped under when `project` is absent. */ +const DEFAULT_PROJECT = 'e2e'; + +const DEVICE_TYPES: ReadonlySet = new Set(['virtual', 'real']); + +/** Every option `testmu()` takes, kept equal to `TestmuOptions` by the compiler. */ +const OPTION_KEYS: readonly string[] = Object.keys({ + device: true, + osVersion: true, + app: true, + deviceType: true, + project: true, + build: true, + sessionName: true, + stateDir: true, +} satisfies Record); + +/** What `testmu()` takes: the device, its OS version, and the app TestMu AI installs on it. */ +export interface TestmuOptions { + /** Device name exactly as TestMu AI's catalog lists it: `Galaxy S22 Ultra 5G`, `iPhone 16`. */ + readonly device: string; + /** OS version exactly as the catalog lists it for that device: `14` on Android, `18.0` on a virtual iOS device, `18` on a real one. */ + readonly osVersion: string; + /** + * The build TestMu AI installs on every session: an `lt://` app id, an + * `https` URL, or a local path, resolved against the project root and + * uploaded when the session starts. + */ + readonly app: string; + /** `'virtual'` (default) for an emulator or simulator, `'real'` for a real device. */ + readonly deviceType?: 'virtual' | 'real' | undefined; + /** Dashboard project the sessions are grouped under. Defaults to `e2e`. */ + readonly project?: string | undefined; + /** Dashboard build the sessions are grouped under. Defaults to the run id. */ + readonly build?: string | undefined; + /** Name of every session on the dashboard; absent, TestMu AI names it. */ + readonly sessionName?: string | undefined; + /** Directory for the agent-device daemon each run starts, relative to the project root. Defaults to `.e2e/testmu`. */ + readonly stateDir?: string | undefined; +} + +type LeaseBackend = 'ios-instance' | 'android-instance'; + +/** The lease scope agent-device resolves a leased device by, as `leases.allocate` granted it. */ +interface LeaseScope { + readonly tenant: string; + readonly runId: string; + readonly leaseId: string; + readonly leaseBackend: LeaseBackend; + readonly leaseProvider: string; +} + +/** A lease's daemon and scope: what releasing it needs. */ +interface LeaseHandle { + readonly stateDir: string; + readonly scope: LeaseScope; +} + +/** + * TestMu AI devices for `mobile({ device: testmu({ device, osVersion, app }) })`: + * one hosted device per worker slot, leased when the run starts and released + * when it ends. Each lease comes from an agent-device daemon the provider + * starts for the run under `stateDir`; the worker's first command starts the + * TestMu AI session, which installs `app`, and releasing the lease ends it. + * It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's + * environment, and needs an agent-device with the `testmu` provider. + */ +export function testmu(options: TestmuOptions): DeviceProvider { + rejectUnknownKeys('testmu()', options, OPTION_KEYS); + for (const key of ['device', 'osVersion', 'app'] as const) { + const value: unknown = options[key]; + if (typeof value !== 'string' || value.trim() === '') { + throw new ConfigurationError('INVALID_CONFIG', `testmu: \`${key}\` is required, as a non-empty string`); + } + } + const deviceType = options.deviceType ?? 'virtual'; + if (!DEVICE_TYPES.has(deviceType)) { + throw new ConfigurationError('INVALID_CONFIG', `testmu: \`deviceType\` must be 'virtual' or 'real', not ${JSON.stringify(deviceType)}`); + } + const { device, osVersion, app, project, build, sessionName } = options; + /** One release per lease, shared by every caller: the engine's, and `acquire`'s own after a failure. */ + const releases = new Map>(); + const release = (id: string, handle: LeaseHandle): Promise => { + let pending = releases.get(id); + if (pending === undefined) { + pending = releaseLease(handle); + releases.set(id, pending); + // A failed release may be tried again. + pending.catch(() => releases.delete(id)); + } + return pending; + }; + return { + name: PROVIDER, + async acquire(request: DeviceRequest): Promise { + if (request.appPath !== undefined) throw new Error("TestMu AI installs the app from `app`; leave the target's `app.appPath` out"); + shareWithDaemon(testmuCredentials(request.env)); + if (request.signal.aborted) throw new Error('cancelled before a lease was allocated'); + const stateDir = resolve(request.projectRoot, options.stateDir ?? DEFAULT_STATE_DIR, request.runId); + const leaseBackend: LeaseBackend = request.platform === 'ios' ? 'ios-instance' : 'android-instance'; + const selectors = { + platform: request.platform, + target: 'mobile' as const, + device, + providerOsVersion: osVersion, + providerApp: appSource(app, request.projectRoot), + providerDeviceType: deviceType, + providerProject: project ?? DEFAULT_PROJECT, + providerBuild: build ?? request.runId, + ...(sessionName === undefined ? {} : { providerSessionName: sessionName }), + }; + // Not cancellable: the daemon may grant the lease after an interrupt, and only a lease this returns or releases is ever released. + const granted = await createAgentDeviceClient({ stateDir, session: `lease-${request.slot}` }).leases.allocate({ + tenant: PROVIDER, + runId: request.runId, + leaseBackend, + leaseProvider: PROVIDER, + ...selectors, + }); + const scope: LeaseScope = { tenant: granted.tenantId, runId: granted.runId, leaseId: granted.leaseId, leaseBackend, leaseProvider: PROVIDER }; + try { + if (request.signal.aborted) throw new Error('cancelled'); + request.log(`lease ${scope.leaseId}: ${device}, ${request.platform} ${osVersion} (${deviceType}); the session starts on the first command`); + // The worker's client is created with these fields: the scope picks the lease, and the selectors start the session. + return { id: scope.leaseId, client: { stateDir, ...scope, ...selectors } }; + } catch (cause) { + // The engine releases only leases `acquire` returned. + const outcome = await release(scope.leaseId, { stateDir, scope }).then( + () => 'released it', + (releaseCause: unknown) => `releasing it failed (${messageOf(releaseCause)})`, + ); + throw new Error(`lease ${scope.leaseId} was not handed to the run: ${messageOf(cause)}; ${outcome}`, { cause }); + } + }, + async release(lease: DeviceLease): Promise { + const handle = leaseHandle(lease); + if (handle === undefined) throw new Error(`lease ${lease.id} carries no agent-device lease scope to release`); + await release(lease.id, handle); + }, + }; +} + +/** Releases a lease through the daemon that granted it, which ends its TestMu AI session. A lease the daemon no longer knows counts as released. */ +async function releaseLease({ stateDir, scope }: LeaseHandle): Promise { + await createAgentDeviceClient({ stateDir, session: 'release' }).leases.release(scope); +} + +/** `app` as the daemon reads it: an `lt://` id or URL as written, a local path resolved against the project root, never the daemon's working directory. */ +function appSource(app: string, projectRoot: string): string { + if (isAbsolute(app) || /^[a-z][a-z0-9+.-]*:\/\//i.test(app)) return app; + return resolve(projectRoot, app); +} + +/** The daemon and scope `acquire` put on a lease's `client`, when they are there. */ +function leaseHandle(lease: DeviceLease): LeaseHandle | undefined { + const client = lease.client as Record | undefined; + if (client === undefined) return undefined; + const { stateDir, tenant, runId, leaseId, leaseBackend, leaseProvider } = client; + if ( + typeof stateDir !== 'string' || + typeof tenant !== 'string' || + typeof runId !== 'string' || + typeof leaseId !== 'string' || + (leaseBackend !== 'ios-instance' && leaseBackend !== 'android-instance') || + typeof leaseProvider !== 'string' + ) { + return undefined; + } + return { stateDir, scope: { tenant, runId, leaseId, leaseBackend, leaseProvider } }; +} + +function messageOf(cause: unknown): string { + return cause instanceof Error ? cause.message : String(cause); +} diff --git a/packages/testmu/tests/unit/testmu.test.ts b/packages/testmu/tests/unit/testmu.test.ts new file mode 100644 index 000000000..23f8e556c --- /dev/null +++ b/packages/testmu/tests/unit/testmu.test.ts @@ -0,0 +1,284 @@ +/** + * `testmu()` against a stubbed agent-device client: the options it refuses, + * the lease it allocates and hands the worker, the credentials it reads and + * shares with the daemon, and the release on every exit path. + */ + +import { join } from 'node:path'; +import type { DeviceLease, DeviceReleaseContext, DeviceRequest } from '@e2e-dev/mobile'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { testmu, type TestmuOptions } from '../../src/index.ts'; + +interface ClientCall { + readonly config: Record; + readonly operation: 'allocate' | 'release'; + readonly options: Record; +} + +const daemon = { + calls: [] as ClientCall[], + /** Runs inside `allocate`, before it answers. */ + onAllocate: undefined as (() => void) | undefined, + allocateError: undefined as Error | undefined, + /** Errors `release` answers with, in order, before it succeeds. */ + releaseErrors: [] as Error[], +}; + +vi.mock('agent-device', () => ({ + createAgentDeviceClient: (config: Record) => ({ + leases: { + allocate: async (options: Record) => { + daemon.calls.push({ config, operation: 'allocate', options }); + daemon.onAllocate?.(); + if (daemon.allocateError !== undefined) throw daemon.allocateError; + return { leaseId: `lease-${daemon.calls.length}`, tenantId: options['tenant'], runId: options['runId'], backend: options['leaseBackend'], leaseProvider: options['leaseProvider'] }; + }, + release: async (options: Record) => { + daemon.calls.push({ config, operation: 'release', options }); + const error = daemon.releaseErrors.shift(); + if (error !== undefined) throw error; + return { released: true }; + }, + }, + }), +})); + +const ROOT = join('/', 'work', 'shop'); +const env = { LT_USERNAME: 'ada', LT_ACCESS_KEY: 'lt-key' }; +const options: TestmuOptions = { device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: 'https://example.com/app.apk' }; +const saved = { LT_USERNAME: process.env['LT_USERNAME'], LT_ACCESS_KEY: process.env['LT_ACCESS_KEY'] }; + +beforeEach(() => { + Object.assign(daemon, { calls: [], onAllocate: undefined, allocateError: undefined, releaseErrors: [] }); +}); + +afterEach(() => { + for (const [name, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + } +}); + +function request(overrides: Partial = {}): DeviceRequest & { lines: string[] } { + const lines: string[] = []; + return { + platform: 'android', + runId: 'run-1', + targetName: 'android', + slot: 0, + slots: 2, + app: 'com.example.app', + agentDeviceVersion: '0.21.18', + projectRoot: ROOT, + env, + signal: new AbortController().signal, + log: (line) => lines.push(line), + lines, + ...overrides, + }; +} + +const context: DeviceReleaseContext = { runId: 'run-1', targetName: 'android', env, signal: new AbortController().signal, log: () => undefined }; + +const operations = () => daemon.calls.map((call) => call.operation); + +describe('testmu()', () => { + it('is a device provider named testmu that leaves recording to agent-device', () => { + const provider = testmu(options); + expect(provider.name).toBe('testmu'); + expect(provider.record).toBeUndefined(); + }); + + it('allocates an Android lease from a daemon under the project root, with the device selectors and dashboard labels', async () => { + await testmu(options).acquire(request({ slot: 1 })); + expect(daemon.calls).toEqual([ + { + config: { stateDir: join(ROOT, '.e2e', 'testmu', 'run-1'), session: 'lease-1' }, + operation: 'allocate', + options: { + tenant: 'testmu', + runId: 'run-1', + leaseBackend: 'android-instance', + leaseProvider: 'testmu', + platform: 'android', + target: 'mobile', + device: 'Galaxy S22 Ultra 5G', + providerOsVersion: '14', + providerApp: 'https://example.com/app.apk', + providerDeviceType: 'virtual', + providerProject: 'e2e', + providerBuild: 'run-1', + }, + }, + ]); + }); + + it('allocates an iOS lease on a real device with its own project, build, session name, and state directory', async () => { + await testmu({ device: 'iPhone 16', osVersion: '18', app: 'lt://APP123', deviceType: 'real', project: 'shop', build: 'nightly', sessionName: 'checkout', stateDir: 'tmp/devices' }).acquire( + request({ platform: 'ios' }), + ); + expect(daemon.calls[0]?.config['stateDir']).toBe(join(ROOT, 'tmp', 'devices', 'run-1')); + expect(daemon.calls[0]?.options).toMatchObject({ + leaseBackend: 'ios-instance', + platform: 'ios', + device: 'iPhone 16', + providerOsVersion: '18', + providerApp: 'lt://APP123', + providerDeviceType: 'real', + providerProject: 'shop', + providerBuild: 'nightly', + providerSessionName: 'checkout', + }); + }); + + it('resolves a local build against the project root, never the working directory', async () => { + await testmu({ ...options, app: 'build/app.apk' }).acquire(request()); + expect(daemon.calls[0]?.options['providerApp']).toBe(join(ROOT, 'build', 'app.apk')); + await testmu({ ...options, app: join('/', 'builds', 'app.apk') }).acquire(request()); + expect(daemon.calls[1]?.options['providerApp']).toBe(join('/', 'builds', 'app.apk')); + }); + + it('hands the worker the lease scope and the selectors as JSON client configuration', async () => { + const req = request(); + const lease = await testmu(options).acquire(req); + expect(lease).toEqual({ + id: 'lease-1', + client: { + stateDir: join(ROOT, '.e2e', 'testmu', 'run-1'), + tenant: 'testmu', + runId: 'run-1', + leaseId: 'lease-1', + leaseBackend: 'android-instance', + leaseProvider: 'testmu', + platform: 'android', + target: 'mobile', + device: 'Galaxy S22 Ultra 5G', + providerOsVersion: '14', + providerApp: 'https://example.com/app.apk', + providerDeviceType: 'virtual', + providerProject: 'e2e', + providerBuild: 'run-1', + }, + }); + const json = JSON.stringify(lease); + expect(JSON.parse(json)).toEqual(lease); + expect(Buffer.byteLength(json)).toBeLessThan(1024); + expect(json).not.toContain('lt-key'); + expect(req.lines).toEqual(['lease lease-1: Galaxy S22 Ultra 5G, android 14 (virtual); the session starts on the first command']); + }); + + it('shares the run\'s credentials with the daemon it starts', async () => { + delete process.env['LT_USERNAME']; + process.env['LT_ACCESS_KEY'] = 'stale'; + daemon.onAllocate = () => expect([process.env['LT_USERNAME'], process.env['LT_ACCESS_KEY']]).toEqual(['ada', 'lt-key']); + await testmu(options).acquire(request({ env: { LT_USERNAME: ' ada ', LT_ACCESS_KEY: 'lt-key' } })); + expect(operations()).toEqual(['allocate']); + }); + + it.each([ + [{}, 'LT_USERNAME and LT_ACCESS_KEY are not set'], + [{ LT_USERNAME: 'ada' }, 'LT_ACCESS_KEY is not set'], + [{ LT_USERNAME: ' ', LT_ACCESS_KEY: 'lt-key' }, 'LT_USERNAME is not set'], + ])('fails before any daemon starts without credentials in the run\'s environment (%j)', async (runEnv, message) => { + process.env['LT_USERNAME'] = 'from-the-runner'; + process.env['LT_ACCESS_KEY'] = 'from-the-runner'; + await expect(testmu(options).acquire(request({ env: runEnv }))).rejects.toThrow( + `${message}; set LT_USERNAME and LT_ACCESS_KEY to your TestMu AI username and access key in the environment \`e2e run\` starts in`, + ); + expect(daemon.calls).toEqual([]); + }); + + it("refuses the target's app.appPath, since TestMu AI installs `app`", async () => { + await expect(testmu(options).acquire(request({ appPath: join(ROOT, 'build', 'app.apk') }))).rejects.toThrow( + "TestMu AI installs the app from `app`; leave the target's `app.appPath` out", + ); + expect(daemon.calls).toEqual([]); + }); + + it('allocates nothing once the run is interrupted', async () => { + const controller = new AbortController(); + controller.abort(); + await expect(testmu(options).acquire(request({ signal: controller.signal }))).rejects.toThrow('cancelled before a lease was allocated'); + expect(daemon.calls).toEqual([]); + }); + + it('releases a lease granted after an interrupt instead of handing it over', async () => { + const controller = new AbortController(); + daemon.onAllocate = () => controller.abort(); + const req = request({ signal: controller.signal }); + await expect(testmu(options).acquire(req)).rejects.toThrow('lease lease-1 was not handed to the run: cancelled; released it'); + expect(daemon.calls[1]).toEqual({ + config: { stateDir: join(ROOT, '.e2e', 'testmu', 'run-1'), session: 'release' }, + operation: 'release', + options: { tenant: 'testmu', runId: 'run-1', leaseId: 'lease-1', leaseBackend: 'android-instance', leaseProvider: 'testmu' }, + }); + expect(req.lines).toEqual([]); + }); + + it('releases a granted lease when handing it over fails, and says when that release failed too', async () => { + daemon.releaseErrors = [new Error('daemon gone')]; + const req = request({ + log: () => { + throw new Error('reporter closed'); + }, + }); + await expect(testmu(options).acquire(req)).rejects.toThrow('lease lease-1 was not handed to the run: reporter closed; releasing it failed (daemon gone)'); + expect(operations()).toEqual(['allocate', 'release']); + }); + + it('passes an allocation failure through with nothing to release', async () => { + daemon.allocateError = new Error('unknown lease provider testmu'); + await expect(testmu(options).acquire(request())).rejects.toThrow('unknown lease provider testmu'); + expect(operations()).toEqual(['allocate']); + }); + + it('releases a lease through the daemon that granted it, once however often it is asked', async () => { + const provider = testmu(options); + const lease = await provider.acquire(request()); + await Promise.all([provider.release(lease, context), provider.release(lease, context)]); + await provider.release(lease, context); + expect(daemon.calls.slice(1)).toEqual([ + { + config: { stateDir: join(ROOT, '.e2e', 'testmu', 'run-1'), session: 'release' }, + operation: 'release', + options: { tenant: 'testmu', runId: 'run-1', leaseId: 'lease-1', leaseBackend: 'android-instance', leaseProvider: 'testmu' }, + }, + ]); + }); + + it('releases from the lease alone, after a round trip through JSON', async () => { + const lease = JSON.parse(JSON.stringify(await testmu(options).acquire(request({ platform: 'ios' })))) as DeviceLease; + await testmu(options).release(lease, context); + expect(daemon.calls[1]?.options).toEqual({ tenant: 'testmu', runId: 'run-1', leaseId: 'lease-1', leaseBackend: 'ios-instance', leaseProvider: 'testmu' }); + }); + + it('tries a release again after one failed', async () => { + daemon.releaseErrors = [new Error('daemon busy')]; + const provider = testmu(options); + const lease = await provider.acquire(request()); + await expect(provider.release(lease, context)).rejects.toThrow('daemon busy'); + await provider.release(lease, context); + expect(operations()).toEqual(['allocate', 'release', 'release']); + }); + + it('refuses to release a lease without an agent-device scope', async () => { + await expect(testmu(options).release({ id: 'other', client: { stateDir: '/tmp/x' } }, context)).rejects.toThrow('lease other carries no agent-device lease scope to release'); + expect(daemon.calls).toEqual([]); + }); + + it('rejects an option it does not take with INVALID_CONFIG, naming the nearest one', () => { + expect(() => testmu({ ...options, osVerison: '14' } as unknown as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG', message: expect.stringContaining('osVersion') })); + }); + + it.each(['device', 'osVersion', 'app'] as const)('requires `%s` with INVALID_CONFIG', (key) => { + expect(() => testmu({ ...options, [key]: ' ' })).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG', message: `testmu: \`${key}\` is required, as a non-empty string` })); + const { [key]: _, ...rest } = options; + expect(() => testmu(rest as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG' })); + }); + + it('refuses a device type other than virtual or real', () => { + expect(() => testmu({ ...options, deviceType: 'emulator' as 'virtual' })).toThrow( + expect.objectContaining({ code: 'INVALID_CONFIG', message: 'testmu: `deviceType` must be \'virtual\' or \'real\', not "emulator"' }), + ); + }); +}); diff --git a/packages/testmu/tsconfig.build.json b/packages/testmu/tsconfig.build.json new file mode 100644 index 000000000..65d5a8150 --- /dev/null +++ b/packages/testmu/tsconfig.build.json @@ -0,0 +1,11 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "dist", + "noEmit": false, + "declaration": true + }, + "include": ["src/**/*.ts"], + "exclude": ["dist", "node_modules"] +} diff --git a/packages/testmu/tsconfig.json b/packages/testmu/tsconfig.json new file mode 100644 index 000000000..635a4f631 --- /dev/null +++ b/packages/testmu/tsconfig.json @@ -0,0 +1,9 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "rootDir": ".", + "noEmit": true + }, + "include": ["src/**/*.ts", "tests/**/*.ts"], + "exclude": ["dist", "node_modules"] +} diff --git a/packages/testmu/vitest.config.ts b/packages/testmu/vitest.config.ts new file mode 100644 index 000000000..c7516095d --- /dev/null +++ b/packages/testmu/vitest.config.ts @@ -0,0 +1,10 @@ +import { defineConfig } from 'vitest/config'; + +/** Pure unit tests: agent-device's client is a stub, no daemon and no network. */ +export default defineConfig({ + test: { + include: ['tests/unit/**/*.test.ts'], + testTimeout: 30_000, + pool: 'forks', + }, +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 7b810cfb4..e80250465 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -290,6 +290,9 @@ importers: '@e2e-dev/mobile': specifier: workspace:* version: link:../packages/mobile + '@e2e-dev/testmu': + specifier: workspace:* + version: link:../packages/testmu '@e2e-dev/web': specifier: workspace:* version: link:../packages/web @@ -473,6 +476,27 @@ importers: specifier: 5.0.1 version: 5.0.1(@types/node@26.6.2)(vite@8.1.5(@types/node@26.6.2)(esbuild@0.28.2)(jiti@1.21.7)(terser@5.51.2)(tsx@4.23.13)(yaml@2.9.0)) + packages/testmu: + devDependencies: + '@e2e-dev/mobile': + specifier: workspace:* + version: link:../mobile + '@types/node': + specifier: 26.6.2 + version: 26.6.2 + agent-device: + specifier: 0.21.18 + version: 0.21.18(ai@7.0.107(zod@4.6.1)) + e2e: + specifier: workspace:* + version: link:../e2e + typescript: + specifier: 7.0.2 + version: 7.0.2 + vitest: + specifier: 5.0.1 + version: 5.0.1(@types/node@26.6.2)(vite@8.1.5(@types/node@26.6.2)(esbuild@0.28.2)(jiti@1.21.7)(terser@5.51.2)(tsx@4.23.14)(yaml@2.9.0)) + packages/web: devDependencies: '@types/node': From 98c60c1814d6d5d6dfdb0d93ffbd6695576c474e Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 19:55:37 +0530 Subject: [PATCH 02/22] docs(integrations): document @e2e-dev/testmu Add a TestMu AI (formerly LambdaTest) integration page for the testmu() device provider from @e2e-dev/testmu, in the shape of the EAS Simulators page: setup, authentication, installing the app, real devices, an options table, sessions, what the provider does, and limits. The page covers the requirement for an agent-device that includes the testmu provider and the override that makes @e2e-dev/mobile use the same copy, the catalog-exact device names and OS versions (18.0 for a virtual iOS device, 18 for a real one, 14 on Android), the app formats per device, and the limits of agent-device's WebDriver runtime. The config and test examples are shown verbatim and typecheck against the built package. The integrations overview, the hosted devices section of the mobile page, the environment reference (LT_USERNAME, LT_ACCESS_KEY), and the agent skill's setup reference name the package. Co-Authored-By: Claude Opus 5.5 --- docs/docs.json | 3 +- docs/examples/mobile/testmu.config.ts | 35 ++++ docs/examples/mobile/testmu.e2e.ts | 8 + docs/integrations/index.mdx | 3 + docs/integrations/testmu.mdx | 236 ++++++++++++++++++++++++++ docs/mobile.mdx | 6 +- docs/package.json | 1 + docs/reference/environment.mdx | 1 + scripts/check-docs-examples.ts | 2 + skills/e2e/references/setup.md | 6 + 10 files changed, 298 insertions(+), 3 deletions(-) create mode 100644 docs/examples/mobile/testmu.config.ts create mode 100644 docs/examples/mobile/testmu.e2e.ts create mode 100644 docs/integrations/testmu.mdx diff --git a/docs/docs.json b/docs/docs.json index 19b432cee..52905d389 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -203,7 +203,8 @@ "pages": [ "integrations/index", "integrations/kernel", - "integrations/eas" + "integrations/eas", + "integrations/testmu" ] }, { diff --git a/docs/examples/mobile/testmu.config.ts b/docs/examples/mobile/testmu.config.ts new file mode 100644 index 000000000..eacf33ccf --- /dev/null +++ b/docs/examples/mobile/testmu.config.ts @@ -0,0 +1,35 @@ +import type { E2EConfig } from 'e2e'; +import { mobile } from '@e2e-dev/mobile'; +import { testmu } from '@e2e-dev/testmu'; + +const apk = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_android.apk'; + +export default { + targets: [ + { + name: 'android-emulator', + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: apk }), + }), + app: { bundleId: 'com.lambdatest.proverbial' }, + }, + { + name: 'ios-simulator', + engine: mobile({ + platform: 'ios', + device: testmu({ device: 'iPhone 16', osVersion: '18.0', app: './build/MyApp.zip' }), + }), + app: { bundleId: 'com.example.app' }, + }, + { + name: 'android-real', + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Pixel 6', osVersion: '14', app: apk, deviceType: 'real' }), + }), + app: { bundleId: 'com.lambdatest.proverbial' }, + }, + ], + workers: 1, +} satisfies E2EConfig; diff --git a/docs/examples/mobile/testmu.e2e.ts b/docs/examples/mobile/testmu.e2e.ts new file mode 100644 index 000000000..5602626dc --- /dev/null +++ b/docs/examples/mobile/testmu.e2e.ts @@ -0,0 +1,8 @@ +import { test } from '@e2e-dev/mobile'; +import { expect } from 'e2e'; + +test('Proverbial home screen', async ({ app, screen }) => { + await app.open(); + await expect(screen.getByText('Proverbial')).toBeVisible(); + await expect(screen.getByRole('button', 'GEOLOCATION')).toBeVisible(); +}); diff --git a/docs/integrations/index.mdx b/docs/integrations/index.mdx index 10f027b96..b6cebe12a 100644 --- a/docs/integrations/index.mdx +++ b/docs/integrations/index.mdx @@ -16,6 +16,9 @@ integration decides where it comes from. `@e2e-dev/eas`: hosted iOS simulators and Android emulators for mobile targets. + + `@e2e-dev/testmu`: hosted Android emulators, iOS simulators, and real devices for mobile targets. + A service without an integration is one small provider in your project: diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx new file mode 100644 index 000000000..5549491ca --- /dev/null +++ b/docs/integrations/testmu.mdx @@ -0,0 +1,236 @@ +--- +title: TestMu AI +sidebarTitle: TestMu AI +description: Run mobile targets on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices with the testmu() device provider from @e2e-dev/testmu. +--- + +`testmu()` from `@e2e-dev/testmu` is a +[device provider](/mobile#hosted-devices) for +[TestMu AI](https://www.lambdatest.com) (formerly LambdaTest). The run gets +hosted Android emulators, iOS simulators, or real devices, leased when it +starts and released when it ends. agent-device reaches them through its +`testmu` cloud provider, which drives each session over TestMu AI's Appium +hub. Tests, config, and CI stay the same. + + +No Mac, Xcode, or Android SDK on the machine that runs e2e. + + +## Setup + +The provider runs on agent-device's `testmu` provider, so install +`agent-device` beside the package, at a release that includes it: + + +```bash npm +npm install --save-dev @e2e-dev/testmu agent-device@ +``` + +```bash pnpm +pnpm add -D @e2e-dev/testmu agent-device@ +``` + +```bash bun +bun add -d @e2e-dev/testmu agent-device@ +``` + + +`` is an agent-device release that includes the `testmu` provider. +No published release does yet: until one does, install agent-device from a +build that includes it. The package's `agent-device` peer range excludes +only the releases known to lack the provider, so it cannot tell you whether +a newer one has it. + +`@e2e-dev/mobile` pins its own agent-device. The provider starts the daemon +from your copy and the engine's workers send commands through theirs, so +both must be the same agent-device. Override the engine's pin: + + +```json npm +{ + "overrides": { + "agent-device": "$agent-device" + } +} +``` + +```yaml pnpm +# pnpm-workspace.yaml +overrides: + agent-device: $agent-device +``` + +```json bun +{ + "overrides": { + "agent-device": "" + } +} +``` + + +For npm and bun the `overrides` field goes in `package.json`. + +```ts title="e2e.config.ts" +import type { E2EConfig } from 'e2e'; +import { mobile } from '@e2e-dev/mobile'; +import { testmu } from '@e2e-dev/testmu'; + +const apk = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_android.apk'; + +export default { + targets: [ + { + name: 'android-emulator', + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: apk }), + }), + app: { bundleId: 'com.lambdatest.proverbial' }, + }, + { + name: 'ios-simulator', + engine: mobile({ + platform: 'ios', + device: testmu({ device: 'iPhone 16', osVersion: '18.0', app: './build/MyApp.zip' }), + }), + app: { bundleId: 'com.example.app' }, + }, + { + name: 'android-real', + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Pixel 6', osVersion: '14', app: apk, deviceType: 'real' }), + }), + app: { bundleId: 'com.lambdatest.proverbial' }, + }, + ], + workers: 1, +} satisfies E2EConfig; +``` + +`device` and `osVersion` must match TestMu AI's catalog exactly, as the +[capabilities generator](https://www.lambdatest.com/capabilities-generator) +lists them: a virtual iOS device takes a version like `18.0`, a real iOS +device one like `18`, and Android one like `14`. TestMu AI checks them when +the session starts and refuses a name or version it does not list. + +Keep `app.bundleId`: TestMu AI installs the app when the session starts, +and `app.open()` launches the installed bundle id or package, not the upload +name. Leave `app.appPath` out; a target that names one fails its lease. + +## Authenticate + +Set `LT_USERNAME` and `LT_ACCESS_KEY` to your TestMu AI username and access +key in the environment `e2e run` starts in, such as CI secrets. Without +either, the lease fails before anything starts, naming the missing one. + +agent-device's `testmu` runtime reads the credentials from the environment +of the daemon that drives the sessions. The provider starts that daemon from +the runner process, and agent-device starts a daemon with the process's own +environment, so the provider copies both values from the run's environment +into it first. + +## Install the app + +`app` names the build TestMu AI installs on every session: + +- **An app id**: an `lt://` reference to a build already in TestMu AI's app + storage. +- **A URL**: an `https` link to the build. +- **A local path**: a build on the machine that runs e2e, resolved against + the directory of `e2e.config.ts` and uploaded when the session starts. + +The build must suit the device: + +| Device | Build | +| --- | --- | +| Android emulator or real device | `.apk` or `.aab` | +| iOS simulator | a zipped `.app` bundle, not an `.ipa` | +| Real iOS device | a signed `.ipa` | + +## Real devices + +`deviceType: 'real'` leases a real device instead of an emulator or +simulator. Real and virtual devices have separate catalogs, so check +`device` and `osVersion` against the real-device one, and give a real iOS +device a signed `.ipa`. Real devices need an agent-device release whose +`testmu` provider supports them. + +## Options + +| Option | Default | Meaning | +| ------------- | --------------- | ------- | +| `device` | required | Device name exactly as TestMu AI's catalog lists it (`'Galaxy S22 Ultra 5G'`, `'iPhone 16'`). | +| `osVersion` | required | OS version exactly as the catalog lists it for that device: `'14'` on Android, `'18.0'` on a virtual iOS device, `'18'` on a real one. | +| `app` | required | The build TestMu AI installs: an `lt://` app id, an `https` URL, or a local path. See [Install the app](#install-the-app). | +| `deviceType` | `'virtual'` | `'virtual'` for an emulator or simulator, `'real'` for a real device. | +| `project` | `'e2e'` | Dashboard project the sessions are grouped under. | +| `build` | the run id | Dashboard build the sessions are grouped under. | +| `sessionName` | TestMu AI's | Name of every session on the dashboard. | +| `stateDir` | `'.e2e/testmu'` | Directory for the agent-device daemon each run starts, relative to the directory of `e2e.config.ts`. Keep it out of version control. | + +An option not in this table, a missing or empty `device`, `osVersion`, or +`app`, or another `deviceType` fails the config load with `INVALID_CONFIG`. + +## Write a test + +Tests are the same as on a local device: + +```ts title="tests/home.e2e.ts" +import { test } from '@e2e-dev/mobile'; +import { expect } from 'e2e'; + +test('Proverbial home screen', async ({ app, screen }) => { + await app.open(); + await expect(screen.getByText('Proverbial')).toBeVisible(); + await expect(screen.getByRole('button', 'GEOLOCATION')).toBeVisible(); +}); +``` + +## Sessions + +Each target leases up to `workers` devices (`--workers` overrides it), one +per worker slot, before the run's clock starts. A session takes about 30 to +75 seconds to start, allocation and app upload included. TestMu AI +runs as many sessions at once as the plan allows, so keep `workers` times +the number of TestMu AI targets within it. + +The provider logs each lease as it is granted: + +```text +ℹ leasing 1 android device(s) from testmu +ℹ testmu (1 of 1): lease 4f0c…: Galaxy S22 Ultra 5G, android 14 (virtual); the session starts on the first command +ℹ testmu: leased 4f0c… +``` + +The video, device logs, and network logs of every session are on the +TestMu AI dashboard, under the project and build the options name. + +## What the provider does + +- Starts one agent-device daemon per run under `stateDir/` and + allocates a `testmu` lease from it for each worker slot. +- Hands the worker the lease scope and the device selectors as agent-device + client configuration. The worker's first command starts the TestMu AI + session from them, which installs `app`. +- Releases each lease when the run ends, which ends its session. A lease + is released once however often the engine asks, and a failed release can + be tried again. +- Releases a lease it was granted after the run was interrupted, or could + not hand to the run, before failing the lease. + +## Limits + +Sessions run over TestMu AI's Appium hub through agent-device's WebDriver +runtime, so what agent-device does not support there is not available to +tests: + +- `device` settings: permissions, location, network, and appearance. +- System alerts. +- Recording through agent-device. Leave [`video`](/reference/config#video) + off for these targets. TestMu AI records every session itself, and the + video and device logs are on its dashboard. +- Device logs through agent-device; read them on the dashboard. +- Port reverse. To reach a server on your machine or network, use TestMu + AI Tunnel. diff --git a/docs/mobile.mdx b/docs/mobile.mdx index f1f94c8d8..27577eaf6 100644 --- a/docs/mobile.mdx +++ b/docs/mobile.mdx @@ -146,7 +146,8 @@ ends. The engine drives each lease through the agent-device daemon the lease names instead of the local one, so any service that runs an agent-device daemon next to a simulator, an emulator, or a physical device works. Nothing in the framework knows a vendor: for [EAS Simulators](/integrations/eas), -`easSimulators()` from `@e2e-dev/eas` is the provider, and for +`easSimulators()` from `@e2e-dev/eas` is the provider, for +[TestMu AI](/integrations/testmu) `testmu()` from `@e2e-dev/testmu`, and for any other service a provider is a few dozen lines in your project. ```ts @@ -302,7 +303,8 @@ on `app.bundleId`, so traces recorded locally replay on hosted devices. A device cloud agent-device speaks to itself needs no session API of your own: the provider allocates a lease through a daemon it starts for the run and -hands the worker the lease scope as `client`. +hands the worker the lease scope as `client`. `testmu()` from +`@e2e-dev/testmu` works this way for [TestMu AI](/integrations/testmu). ```ts title="device-cloud-provider.ts" import { createAgentDeviceClient } from 'agent-device'; diff --git a/docs/package.json b/docs/package.json index c55d4f486..8050c5c70 100644 --- a/docs/package.json +++ b/docs/package.json @@ -10,6 +10,7 @@ }, "devDependencies": { "@e2e-dev/mobile": "workspace:*", + "@e2e-dev/testmu": "workspace:*", "e2e": "workspace:*", "@e2e-dev/web": "workspace:*", "@types/node": "26.6.2", diff --git a/docs/reference/environment.mdx b/docs/reference/environment.mdx index 767e5d6d3..217ec2186 100644 --- a/docs/reference/environment.mdx +++ b/docs/reference/environment.mdx @@ -103,6 +103,7 @@ What is sent and how to read it is on the [Telemetry](/telemetry) page. | `KERNEL_API_KEY` | value | The API key `kernel()` from `@e2e-dev/kernel` creates browsers with; unset, the browser lease fails. See [Kernel](/integrations/kernel) | | `EXPO_TOKEN` | value | The Expo access token `easSimulators()` from `@e2e-dev/eas` starts and stops simulator sessions with; unset, it uses the eas-cli login in `.expo/state.json` under `HOME` (`USERPROFILE` on Windows), and with neither the device lease fails. See [EAS Simulators](/integrations/eas) | | `HOME`, `USERPROFILE` | value | The home `easSimulators()` finds the eas-cli login under when `EXPO_TOKEN` is unset: `USERPROFILE` on Windows, `HOME` elsewhere, the OS user's home when unset | +| `LT_USERNAME`, `LT_ACCESS_KEY` | value | The TestMu AI username and access key `testmu()` from `@e2e-dev/testmu` leases devices with; it copies them into the runner process's environment for the agent-device daemon it starts, and with either unset the device lease fails. See [TestMu AI](/integrations/testmu) | | `E2E_AGENT_DEVICE_POOL__` | value | Written by `@e2e-dev/mobile` in `prepare` and read by each worker to find its booted device. Not meant to be set by hand | ## What app processes inherit diff --git a/scripts/check-docs-examples.ts b/scripts/check-docs-examples.ts index a19fd5832..aff292df2 100644 --- a/scripts/check-docs-examples.ts +++ b/scripts/check-docs-examples.ts @@ -27,6 +27,8 @@ const EXAMPLES: Record = { 'docs/examples/quickstart/e2e.command.config.ts': 'docs/starting-your-app.mdx', 'docs/examples/mobile/device-provider.ts': 'docs/mobile.mdx', 'docs/examples/mobile/device-cloud-provider.ts': 'docs/mobile.mdx', + 'docs/examples/mobile/testmu.config.ts': 'docs/integrations/testmu.mdx', + 'docs/examples/mobile/testmu.e2e.ts': 'docs/integrations/testmu.mdx', 'docs/examples/web/browser-provider.ts': 'docs/browser.mdx', 'docs/examples/skill/e2e.config.ts': 'skills/e2e/SKILL.md', 'docs/examples/skill/e2e.setup.config.ts': 'skills/e2e/references/setup.md', diff --git a/skills/e2e/references/setup.md b/skills/e2e/references/setup.md index 7c42c1e01..4a4fffb52 100644 --- a/skills/e2e/references/setup.md +++ b/skills/e2e/references/setup.md @@ -299,6 +299,12 @@ export default { the app (omit `app.appPath`). A run must fit one session: `maxDurationMinutes`, absent, is the account's cap (40 on a standard plan). `videoTouches: false` on the engine for video there. + `testmu({ device, osVersion, app })` from `@e2e-dev/testmu` leases TestMu + AI (formerly LambdaTest) emulators, simulators, or real devices + (`deviceType: 'real'`); `device` and `osVersion` match its catalog + exactly, TestMu AI installs `app` (omit `app.appPath`), it reads + `LT_USERNAME` and `LT_ACCESS_KEY`, and needs an agent-device with the + `testmu` provider shared with `@e2e-dev/mobile` (override its pin). - Only a control that appeared or moved with the previous action waits out `transition` (default 500 ms); agent actions settle `settle` ms (default 150) before the next observation, `settle: false` skips it. From 9a797ebdef0f2adc8182da8b2d45a04310e6dca8 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 20:26:28 +0530 Subject: [PATCH 03/22] fix(testmu): keep the lease alive while the session starts agent-device ends a lease after 60 seconds without activity by default, and a command still running does not count as activity. The worker's first command uploads the app, allocates the hosted session, and boots it; on an iOS simulator that takes about 70 seconds, so the lease lapsed mid-start and the engine's prepare failed with "boot failed: Lease is not active" (LEASE_NOT_FOUND). Each lease now asks for agent-device's longest inactivity window (10 minutes), and the provider heartbeats every lease it holds every 2 minutes from the runner process until the lease is released, including the release acquire does itself when it cannot hand a lease to the run. The timer is unref'd so it never keeps the runner alive. A failed heartbeat is logged once per lease and the next one is still tried; it never fails the run. Co-Authored-By: Claude Opus 5.5 --- .changeset/testmu-devices.md | 2 +- docs/integrations/testmu.mdx | 4 ++ packages/testmu/src/provider.ts | 40 +++++++++++ packages/testmu/tests/unit/testmu.test.ts | 81 ++++++++++++++++++++++- 4 files changed, 123 insertions(+), 4 deletions(-) diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md index c19d878ec..f610a0367 100644 --- a/.changeset/testmu-devices.md +++ b/.changeset/testmu-devices.md @@ -2,4 +2,4 @@ "@e2e-dev/testmu": minor --- -`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); the worker's first command starts the TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id), releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); the worker's first command starts the TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id), heartbeats each lease while the run holds it so a session slow to start keeps its lease, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 5549491ca..ddd216fd0 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -211,6 +211,10 @@ TestMu AI dashboard, under the project and build the options name. - Starts one agent-device daemon per run under `stateDir/` and allocates a `testmu` lease from it for each worker slot. +- Keeps each lease alive while the run holds it. A lease asks for + agent-device's longest inactivity window, 10 minutes, and the provider + heartbeats it every 2 minutes, so a session that takes long to start keeps + its lease. A failed heartbeat is logged once and does not fail the run. - Hands the worker the lease scope and the device selectors as agent-device client configuration. The worker's first command starts the TestMu AI session from them, which installs `app`. diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index 0198f890e..876d2a447 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -20,6 +20,16 @@ const DEFAULT_STATE_DIR = '.e2e/testmu'; /** The dashboard project sessions are grouped under when `project` is absent. */ const DEFAULT_PROJECT = 'e2e'; +/** + * The inactivity window each lease asks for, agent-device's longest. Its + * 60-second default lapses while the worker's first command uploads the app + * and starts the session, which can take longer on an iOS simulator. + */ +const LEASE_TTL_MS = 10 * 60_000; + +/** How often the runner heartbeats each lease it holds, well inside `LEASE_TTL_MS`. */ +const HEARTBEAT_INTERVAL_MS = 2 * 60_000; + const DEVICE_TYPES: ReadonlySet = new Set(['virtual', 'real']); /** Every option `testmu()` takes, kept equal to `TestmuOptions` by the compiler. */ @@ -99,7 +109,11 @@ export function testmu(options: TestmuOptions): DeviceProvider { const { device, osVersion, app, project, build, sessionName } = options; /** One release per lease, shared by every caller: the engine's, and `acquire`'s own after a failure. */ const releases = new Map>(); + /** Stops each held lease's heartbeat, by lease id. */ + const heartbeats = new Map void>(); const release = (id: string, handle: LeaseHandle): Promise => { + heartbeats.get(id)?.(); + heartbeats.delete(id); let pending = releases.get(id); if (pending === undefined) { pending = releaseLease(handle); @@ -134,9 +148,11 @@ export function testmu(options: TestmuOptions): DeviceProvider { runId: request.runId, leaseBackend, leaseProvider: PROVIDER, + ttlMs: LEASE_TTL_MS, ...selectors, }); const scope: LeaseScope = { tenant: granted.tenantId, runId: granted.runId, leaseId: granted.leaseId, leaseBackend, leaseProvider: PROVIDER }; + heartbeats.set(scope.leaseId, keepAlive({ stateDir, scope }, request.log)); try { if (request.signal.aborted) throw new Error('cancelled'); request.log(`lease ${scope.leaseId}: ${device}, ${request.platform} ${osVersion} (${deviceType}); the session starts on the first command`); @@ -164,6 +180,30 @@ async function releaseLease({ stateDir, scope }: LeaseHandle): Promise { await createAgentDeviceClient({ stateDir, session: 'release' }).leases.release(scope); } +/** + * Heartbeats a lease until the returned function is called. A command still + * running does not keep its lease alive, so without this a lease can lapse + * while the worker's first command is starting the session. A failed + * heartbeat is logged once and the next one tried; it never fails the run. + */ +function keepAlive({ stateDir, scope }: LeaseHandle, log: (line: string) => void): () => void { + const client = createAgentDeviceClient({ stateDir, session: 'heartbeat' }); + let warned = false; + const timer = setInterval(() => { + client.leases.heartbeat({ ...scope, ttlMs: LEASE_TTL_MS }).catch((cause: unknown) => { + if (warned) return; + warned = true; + try { + log(`lease ${scope.leaseId}: heartbeat failed (${messageOf(cause)}); agent-device ends the lease after ${LEASE_TTL_MS / 60_000} minutes without one`); + } catch { + // The run's log may be closed by now. + } + }); + }, HEARTBEAT_INTERVAL_MS); + timer.unref(); + return () => clearInterval(timer); +} + /** `app` as the daemon reads it: an `lt://` id or URL as written, a local path resolved against the project root, never the daemon's working directory. */ function appSource(app: string, projectRoot: string): string { if (isAbsolute(app) || /^[a-z][a-z0-9+.-]*:\/\//i.test(app)) return app; diff --git a/packages/testmu/tests/unit/testmu.test.ts b/packages/testmu/tests/unit/testmu.test.ts index 23f8e556c..de4c6fecf 100644 --- a/packages/testmu/tests/unit/testmu.test.ts +++ b/packages/testmu/tests/unit/testmu.test.ts @@ -1,7 +1,8 @@ /** * `testmu()` against a stubbed agent-device client: the options it refuses, * the lease it allocates and hands the worker, the credentials it reads and - * shares with the daemon, and the release on every exit path. + * shares with the daemon, the heartbeat that keeps it alive, and the release + * on every exit path. */ import { join } from 'node:path'; @@ -11,7 +12,7 @@ import { testmu, type TestmuOptions } from '../../src/index.ts'; interface ClientCall { readonly config: Record; - readonly operation: 'allocate' | 'release'; + readonly operation: 'allocate' | 'heartbeat' | 'release'; readonly options: Record; } @@ -22,6 +23,8 @@ const daemon = { allocateError: undefined as Error | undefined, /** Errors `release` answers with, in order, before it succeeds. */ releaseErrors: [] as Error[], + /** Errors `heartbeat` answers with, in order, before it succeeds. */ + heartbeatErrors: [] as Error[], }; vi.mock('agent-device', () => ({ @@ -33,6 +36,12 @@ vi.mock('agent-device', () => ({ if (daemon.allocateError !== undefined) throw daemon.allocateError; return { leaseId: `lease-${daemon.calls.length}`, tenantId: options['tenant'], runId: options['runId'], backend: options['leaseBackend'], leaseProvider: options['leaseProvider'] }; }, + heartbeat: async (options: Record) => { + daemon.calls.push({ config, operation: 'heartbeat', options }); + const error = daemon.heartbeatErrors.shift(); + if (error !== undefined) throw error; + return { leaseId: options['leaseId'], tenantId: options['tenant'], runId: options['runId'], backend: options['leaseBackend'] }; + }, release: async (options: Record) => { daemon.calls.push({ config, operation: 'release', options }); const error = daemon.releaseErrors.shift(); @@ -49,10 +58,11 @@ const options: TestmuOptions = { device: 'Galaxy S22 Ultra 5G', osVersion: '14', const saved = { LT_USERNAME: process.env['LT_USERNAME'], LT_ACCESS_KEY: process.env['LT_ACCESS_KEY'] }; beforeEach(() => { - Object.assign(daemon, { calls: [], onAllocate: undefined, allocateError: undefined, releaseErrors: [] }); + Object.assign(daemon, { calls: [], onAllocate: undefined, allocateError: undefined, releaseErrors: [], heartbeatErrors: [] }); }); afterEach(() => { + vi.useRealTimers(); for (const [name, value] of Object.entries(saved)) { if (value === undefined) delete process.env[name]; else process.env[name] = value; @@ -82,6 +92,8 @@ const context: DeviceReleaseContext = { runId: 'run-1', targetName: 'android', e const operations = () => daemon.calls.map((call) => call.operation); +const MINUTE = 60_000; + describe('testmu()', () => { it('is a device provider named testmu that leaves recording to agent-device', () => { const provider = testmu(options); @@ -108,6 +120,7 @@ describe('testmu()', () => { providerDeviceType: 'virtual', providerProject: 'e2e', providerBuild: 'run-1', + ttlMs: 10 * MINUTE, }, }, ]); @@ -266,6 +279,68 @@ describe('testmu()', () => { expect(daemon.calls).toEqual([]); }); + it('heartbeats each lease it holds every two minutes, asking for the longest lease, until it is released', async () => { + vi.useFakeTimers(); + const provider = testmu(options); + const lease = await provider.acquire(request()); + await vi.advanceTimersByTimeAsync(2 * MINUTE - 1); + expect(operations()).toEqual(['allocate']); + await vi.advanceTimersByTimeAsync(1); + expect(daemon.calls[1]).toEqual({ + config: { stateDir: join(ROOT, '.e2e', 'testmu', 'run-1'), session: 'heartbeat' }, + operation: 'heartbeat', + options: { tenant: 'testmu', runId: 'run-1', leaseId: 'lease-1', leaseBackend: 'android-instance', leaseProvider: 'testmu', ttlMs: 10 * MINUTE }, + }); + await vi.advanceTimersByTimeAsync(2 * MINUTE); + expect(operations()).toEqual(['allocate', 'heartbeat', 'heartbeat']); + await provider.release(lease, context); + await vi.advanceTimersByTimeAsync(10 * MINUTE); + expect(operations()).toEqual(['allocate', 'heartbeat', 'heartbeat', 'release']); + }); + + it('stops heartbeating a lease it released because handing it over failed', async () => { + vi.useFakeTimers(); + const controller = new AbortController(); + daemon.onAllocate = () => controller.abort(); + await expect(testmu(options).acquire(request({ signal: controller.signal }))).rejects.toThrow('was not handed to the run'); + await vi.advanceTimersByTimeAsync(10 * MINUTE); + expect(operations()).toEqual(['allocate', 'release']); + }); + + it('heartbeats nothing when the allocation failed', async () => { + vi.useFakeTimers(); + daemon.allocateError = new Error('no capacity'); + await expect(testmu(options).acquire(request())).rejects.toThrow('no capacity'); + await vi.advanceTimersByTimeAsync(10 * MINUTE); + expect(operations()).toEqual(['allocate']); + }); + + it('logs the first failed heartbeat and keeps beating, without failing the run', async () => { + vi.useFakeTimers(); + daemon.heartbeatErrors = [new Error('Lease is not active'), new Error('daemon gone')]; + const req = request(); + await testmu(options).acquire(req); + await vi.advanceTimersByTimeAsync(6 * MINUTE); + expect(operations()).toEqual(['allocate', 'heartbeat', 'heartbeat', 'heartbeat']); + expect(req.lines.slice(1)).toEqual(['lease lease-1: heartbeat failed (Lease is not active); agent-device ends the lease after 10 minutes without one']); + }); + + it('survives a failed heartbeat when the run\'s log is closed', async () => { + vi.useFakeTimers(); + daemon.heartbeatErrors = [new Error('daemon gone')]; + let closed = false; + await testmu(options).acquire( + request({ + log: () => { + if (closed) throw new Error('reporter closed'); + }, + }), + ); + closed = true; + await vi.advanceTimersByTimeAsync(4 * MINUTE); + expect(operations()).toEqual(['allocate', 'heartbeat', 'heartbeat']); + }); + it('rejects an option it does not take with INVALID_CONFIG, naming the nearest one', () => { expect(() => testmu({ ...options, osVerison: '14' } as unknown as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG', message: expect.stringContaining('osVersion') })); }); From 1ef8f7766178fb40359ac0adec7b6394f51c8930 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 20:27:31 +0530 Subject: [PATCH 04/22] fix(mobile): read XCUIElementType-prefixed iOS element types agent-device's native iOS runner reports element types as `Button`, `StaticText`, `NavigationBar`. Its WebDriver runtime, which hosted Appium clouds (TestMu AI, BrowserStack, AWS Device Farm) go through, reports the XCUITest class names instead: `XCUIElementTypeButton`, `XCUIElementTypeStaticText`. normalizeKind kebab-cased those to `xcuielement-type-button`, which maps to no role, so on a real iPhone `getByRole('button', 'CPU Load')` matched nothing and observe listed the node by its raw class name. normalizeKind now drops a leading `XCUIElementType` before kebab-casing, so both spellings land on one vocabulary: buttons, tab bar items as tabs, the navigation bar as the screen title. Only that exact prefix followed by a type name is stripped; Android class names are untouched, and the golden device trees project unchanged. Co-Authored-By: Claude Opus 5.5 --- .changeset/ios-webdriver-element-types.md | 5 ++++ packages/mobile/src/nodes.ts | 11 +++++---- packages/mobile/tests/unit/nodes.test.ts | 30 +++++++++++++++++++++++ 3 files changed, 41 insertions(+), 5 deletions(-) create mode 100644 .changeset/ios-webdriver-element-types.md diff --git a/.changeset/ios-webdriver-element-types.md b/.changeset/ios-webdriver-element-types.md new file mode 100644 index 000000000..18e659f7b --- /dev/null +++ b/.changeset/ios-webdriver-element-types.md @@ -0,0 +1,5 @@ +--- +'@e2e-dev/mobile': patch +--- + +iOS devices driven through agent-device's WebDriver runtime (hosted Appium clouds such as TestMu AI, BrowserStack, and AWS Device Farm) resolve roles like a local simulator does. That runtime reports XCUITest class names (`XCUIElementTypeButton`, `XCUIElementTypeStaticText`) where the native runner reports `Button` and `StaticText`, and the engine read them as unknown types, so `getByRole('button', 'CPU Load')` matched nothing and `observe` listed the node as `xcuielement-type-button`. The `XCUIElementType` prefix is now dropped before the type is mapped, so a button is a `button`, a tab bar's buttons are `tab`s, and the navigation bar titles the screen. diff --git a/packages/mobile/src/nodes.ts b/packages/mobile/src/nodes.ts index ea9e6212c..dcb5f350d 100644 --- a/packages/mobile/src/nodes.ts +++ b/packages/mobile/src/nodes.ts @@ -266,9 +266,10 @@ const ANDROID_TITLE_IDS = [':id/collapsing_toolbar', ':id/action_bar', ':id/tool /** * Platform element type of one raw node as a kebab-case token: XCTest sends - * `NavigationBar` and `StaticText`, Android sends `android.widget.TextView`; - * both read as one vocabulary here, the Android package prefix dropped. - * `role` is the fallback some platforms send instead. + * `NavigationBar` and `StaticText` (`XCUIElementTypeNavigationBar` over + * WebDriver), Android sends `android.widget.TextView`; all read as one + * vocabulary here, the class and package prefixes dropped. `role` is the + * fallback some platforms send instead. */ function kindOf(raw: RawNode): string { return normalizeKind(raw.type ?? raw.role ?? ''); @@ -279,9 +280,9 @@ function isAndroidClass(type: string | undefined): boolean { return type !== undefined && type.includes('.'); } -/** One element-type spelling for `NavigationBar`, `navigation-bar`, and `android.widget.NavigationBar` alike. */ +/** One element-type spelling for `NavigationBar`, `XCUIElementTypeNavigationBar`, `navigation-bar`, and `android.widget.NavigationBar` alike. */ export function normalizeKind(type: string): string { - const simple = type.slice(type.lastIndexOf('.') + 1); + const simple = type.slice(type.lastIndexOf('.') + 1).replace(/^XCUIElementType(?=[A-Z])/, ''); return simple .replaceAll(/([a-z0-9])([A-Z])/g, '$1-$2') .replaceAll(/[\s_]+/g, '-') diff --git a/packages/mobile/tests/unit/nodes.test.ts b/packages/mobile/tests/unit/nodes.test.ts index 45089d0a6..24f5f76fb 100644 --- a/packages/mobile/tests/unit/nodes.test.ts +++ b/packages/mobile/tests/unit/nodes.test.ts @@ -214,6 +214,36 @@ describe('snapshot projection', () => { expect(screenTitle(projected)).toBe('Settings'); }); + it('reads the XCUIElementType class names a WebDriver session reports as the native XCTest types', () => { + const native: RawNode[] = [ + { ref: 'e1', index: 0, depth: 0, type: 'Application', label: 'Profiler', rect: { x: 0, y: 0, width: 390, height: 844 } }, + { ref: 'e2', index: 1, parentIndex: 0, depth: 1, type: 'NavigationBar', identifier: 'Profiler' }, + { ref: 'e3', index: 2, parentIndex: 1, depth: 2, type: 'StaticText', label: 'Profiler' }, + { ref: 'e4', index: 3, parentIndex: 0, depth: 1, type: 'Button', label: 'CPU Load' }, + { ref: 'e5', index: 4, parentIndex: 0, depth: 1, type: 'TextView', label: 'Notes', value: 'idle' }, + { ref: 'e6', index: 5, parentIndex: 0, depth: 1, type: 'TabBar' }, + { ref: 'e7', index: 6, parentIndex: 5, depth: 2, type: 'Button', label: 'Home', selected: true }, + { ref: 'e8', index: 7, parentIndex: 5, depth: 2, type: 'Button', label: 'Settings' }, + ]; + const webDriver = native.map((node) => ({ ...node, type: `XCUIElementType${node.type}` })); + const kindsAndRoles = (nodes: readonly RawNode[]) => project(nodes).index.map((entry) => [entry.kind, entry.node.role]); + expect(kindsAndRoles(webDriver)).toEqual(kindsAndRoles(native)); + const projected = project(webDriver); + expect(projected.index.map((entry) => entry.kind)).toEqual([ + 'application', + 'navigation-bar', + 'static-text', + 'button', + 'text-view', + 'tab-bar', + 'button', + 'button', + ]); + expect(projected.index.map((entry) => entry.node.role)).toEqual(['application', 'navigation', 'text', 'button', 'textbox', 'tablist', 'tab', 'tab']); + expect(projected.viewport).toEqual({ width: 390, height: 844 }); + expect(screenTitle(projected)).toBe('Profiler'); + }); + it('maps iOS composite widgets onto the vocabulary roles a browser reports for them', () => { const projected = project([ { ref: 'e1', index: 0, depth: 0, type: 'Application', label: 'Shop' }, From 2f737830fec84684e575fe92abfc6ecfa1997aca Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:06:20 +0530 Subject: [PATCH 05/22] fix(testmu): describe when the session starts agent-device's `leases.allocate` creates the TestMu AI session before it returns, so the session starts at acquire, not on the worker's first command, and every slot is billed from the moment it is leased. The lease log line, the provider's comments, the README, the docs page, and the changeset now say so, and explain the long lease window and heartbeat by agent-device starting a lease's window before a slow allocation returns. Co-Authored-By: Claude Opus 5.5 --- .changeset/testmu-devices.md | 2 +- docs/integrations/testmu.mdx | 27 +++++++++++++---------- packages/testmu/README.md | 7 +++--- packages/testmu/src/provider.ts | 25 ++++++++++++--------- packages/testmu/tests/unit/testmu.test.ts | 2 +- 5 files changed, 35 insertions(+), 28 deletions(-) diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md index f610a0367..a617dddfa 100644 --- a/.changeset/testmu-devices.md +++ b/.changeset/testmu-devices.md @@ -2,4 +2,4 @@ "@e2e-dev/testmu": minor --- -`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); the worker's first command starts the TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id), heartbeats each lease while the run holds it so a session slow to start keeps its lease, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index ddd216fd0..17c6714fa 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -191,16 +191,18 @@ test('Proverbial home screen', async ({ app, screen }) => { ## Sessions Each target leases up to `workers` devices (`--workers` overrides it), one -per worker slot, before the run's clock starts. A session takes about 30 to -75 seconds to start, allocation and app upload included. TestMu AI -runs as many sessions at once as the plan allows, so keep `workers` times -the number of TestMu AI targets within it. +per worker slot, before the run's clock starts. Allocating a lease starts +its TestMu AI session, which takes about 30 to 75 seconds, app upload +included, so every slot is billed from the moment it is leased until the run +ends, even when no test runs on it. TestMu AI runs as many sessions at once +as the plan allows, so keep `workers` times the number of TestMu AI targets +within it. The provider logs each lease as it is granted: ```text ℹ leasing 1 android device(s) from testmu -ℹ testmu (1 of 1): lease 4f0c…: Galaxy S22 Ultra 5G, android 14 (virtual); the session starts on the first command +ℹ testmu (1 of 1): lease 4f0c…: Galaxy S22 Ultra 5G, android 14 (virtual); session started ℹ testmu: leased 4f0c… ``` @@ -210,14 +212,15 @@ TestMu AI dashboard, under the project and build the options name. ## What the provider does - Starts one agent-device daemon per run under `stateDir/` and - allocates a `testmu` lease from it for each worker slot. -- Keeps each lease alive while the run holds it. A lease asks for - agent-device's longest inactivity window, 10 minutes, and the provider - heartbeats it every 2 minutes, so a session that takes long to start keeps - its lease. A failed heartbeat is logged once and does not fail the run. + allocates a `testmu` lease from it for each worker slot. The allocation + starts the TestMu AI session, which installs `app`. +- Keeps each lease alive while the run holds it. agent-device starts a + lease's inactivity window before a slow allocation returns, so the + provider asks for agent-device's longest window, 10 minutes, and + heartbeats the lease every 2 minutes. A failed heartbeat is logged once + and does not fail the run. - Hands the worker the lease scope and the device selectors as agent-device - client configuration. The worker's first command starts the TestMu AI - session from them, which installs `app`. + client configuration, so every command lands on the lease's session. - Releases each lease when the run ends, which ends its session. A lease is released once however often the engine asks, and a failed release can be tried again. diff --git a/packages/testmu/README.md b/packages/testmu/README.md index 09dea7f87..3a3bab227 100644 --- a/packages/testmu/README.md +++ b/packages/testmu/README.md @@ -41,9 +41,10 @@ export default { It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment. Each worker slot leases one device from an agent-device daemon -the provider starts for the run; the worker's first command starts the -TestMu AI session, which installs `app`, and the lease is released when the -run ends, which ends the session. +the provider starts for the run. Allocating the lease starts the TestMu AI +session, which installs `app`, so every slot is billed from the moment it is +leased, even when no test runs on it. The lease is released when the run +ends, which ends the session. - `device` and `osVersion` must match TestMu AI's catalog exactly: `18.0` for a virtual iOS device, `18` for a real one, `14` on Android. diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index 876d2a447..8a2ffc53f 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -21,9 +21,10 @@ const DEFAULT_STATE_DIR = '.e2e/testmu'; const DEFAULT_PROJECT = 'e2e'; /** - * The inactivity window each lease asks for, agent-device's longest. Its - * 60-second default lapses while the worker's first command uploads the app - * and starts the session, which can take longer on an iOS simulator. + * The inactivity window each lease asks for, agent-device's longest. + * agent-device starts a lease's window before the allocation that uploads the + * app and starts the session returns, so a slow one, as on an iOS simulator, + * can spend most of the 60-second default before the lease is granted. */ const LEASE_TTL_MS = 10 * 60_000; @@ -53,7 +54,7 @@ export interface TestmuOptions { /** * The build TestMu AI installs on every session: an `lt://` app id, an * `https` URL, or a local path, resolved against the project root and - * uploaded when the session starts. + * uploaded when the lease is allocated. */ readonly app: string; /** `'virtual'` (default) for an emulator or simulator, `'real'` for a real device. */ @@ -89,8 +90,9 @@ interface LeaseHandle { * TestMu AI devices for `mobile({ device: testmu({ device, osVersion, app }) })`: * one hosted device per worker slot, leased when the run starts and released * when it ends. Each lease comes from an agent-device daemon the provider - * starts for the run under `stateDir`; the worker's first command starts the - * TestMu AI session, which installs `app`, and releasing the lease ends it. + * starts for the run under `stateDir`; allocating it starts the TestMu AI + * session, which installs `app`, so every slot is billed from the moment it + * is leased, whether or not a test runs on it, and releasing the lease ends it. * It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's * environment, and needs an agent-device with the `testmu` provider. */ @@ -155,8 +157,8 @@ export function testmu(options: TestmuOptions): DeviceProvider { heartbeats.set(scope.leaseId, keepAlive({ stateDir, scope }, request.log)); try { if (request.signal.aborted) throw new Error('cancelled'); - request.log(`lease ${scope.leaseId}: ${device}, ${request.platform} ${osVersion} (${deviceType}); the session starts on the first command`); - // The worker's client is created with these fields: the scope picks the lease, and the selectors start the session. + request.log(`lease ${scope.leaseId}: ${device}, ${request.platform} ${osVersion} (${deviceType}); session started`); + // The worker's client is created with these fields: the scope picks the lease, and the selectors match the session it holds. return { id: scope.leaseId, client: { stateDir, ...scope, ...selectors } }; } catch (cause) { // The engine releases only leases `acquire` returned. @@ -182,9 +184,10 @@ async function releaseLease({ stateDir, scope }: LeaseHandle): Promise { /** * Heartbeats a lease until the returned function is called. A command still - * running does not keep its lease alive, so without this a lease can lapse - * while the worker's first command is starting the session. A failed - * heartbeat is logged once and the next one tried; it never fails the run. + * running does not keep its lease alive, and the allocation may already have + * spent part of the lease's window, so without this a lease can lapse while + * the run holds it. A failed heartbeat is logged once and the next one tried; + * it never fails the run. */ function keepAlive({ stateDir, scope }: LeaseHandle, log: (line: string) => void): () => void { const client = createAgentDeviceClient({ stateDir, session: 'heartbeat' }); diff --git a/packages/testmu/tests/unit/testmu.test.ts b/packages/testmu/tests/unit/testmu.test.ts index de4c6fecf..14f34ce01 100644 --- a/packages/testmu/tests/unit/testmu.test.ts +++ b/packages/testmu/tests/unit/testmu.test.ts @@ -177,7 +177,7 @@ describe('testmu()', () => { expect(JSON.parse(json)).toEqual(lease); expect(Buffer.byteLength(json)).toBeLessThan(1024); expect(json).not.toContain('lt-key'); - expect(req.lines).toEqual(['lease lease-1: Galaxy S22 Ultra 5G, android 14 (virtual); the session starts on the first command']); + expect(req.lines).toEqual(['lease lease-1: Galaxy S22 Ultra 5G, android 14 (virtual); session started']); }); it('shares the run\'s credentials with the daemon it starts', async () => { From a27b1c0c3cbf9d748e488d434b4dd6620be5a231 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:07:11 +0530 Subject: [PATCH 06/22] feat(testmu): name each slot's session by default Without `sessionName`, each slot's session is now named `e2e---` instead of whatever TestMu AI picks, and a given `sessionName` gets `-` appended when the target leases more than one device. Every slot's session then has a name of its own within its build, which is what finding the session again by name relies on. The lease log line names the session. Co-Authored-By: Claude Opus 5.5 --- .changeset/testmu-devices.md | 2 +- docs/integrations/testmu.mdx | 4 ++-- packages/testmu/README.md | 4 +++- packages/testmu/src/provider.ts | 19 ++++++++++++++++--- packages/testmu/tests/unit/testmu.test.ts | 15 +++++++++++++-- 5 files changed, 35 insertions(+), 9 deletions(-) diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md index a617dddfa..e7f92f388 100644 --- a/.changeset/testmu-devices.md +++ b/.changeset/testmu-devices.md @@ -2,4 +2,4 @@ "@e2e-dev/testmu": minor --- -`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 17c6714fa..4af42ef6f 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -167,7 +167,7 @@ device a signed `.ipa`. Real devices need an agent-device release whose | `deviceType` | `'virtual'` | `'virtual'` for an emulator or simulator, `'real'` for a real device. | | `project` | `'e2e'` | Dashboard project the sessions are grouped under. | | `build` | the run id | Dashboard build the sessions are grouped under. | -| `sessionName` | TestMu AI's | Name of every session on the dashboard. | +| `sessionName` | `e2e---` | Name of each session on the dashboard. With more than one worker slot, `-` is appended, so every slot's session has its own name; slots count from 1. Give targets that share a `build` different names. | | `stateDir` | `'.e2e/testmu'` | Directory for the agent-device daemon each run starts, relative to the directory of `e2e.config.ts`. Keep it out of version control. | An option not in this table, a missing or empty `device`, `osVersion`, or @@ -202,7 +202,7 @@ The provider logs each lease as it is granted: ```text ℹ leasing 1 android device(s) from testmu -ℹ testmu (1 of 1): lease 4f0c…: Galaxy S22 Ultra 5G, android 14 (virtual); session started +ℹ testmu (1 of 1): lease 4f0c…: Galaxy S22 Ultra 5G, android 14 (virtual); session e2e-0199…-android-emulator-1 started ℹ testmu: leased 4f0c… ``` diff --git a/packages/testmu/README.md b/packages/testmu/README.md index 3a3bab227..786b989d9 100644 --- a/packages/testmu/README.md +++ b/packages/testmu/README.md @@ -53,7 +53,9 @@ ends, which ends the session. `app.appPath` out. - `deviceType: 'real'` picks a real device (default `'virtual'`). - `project` (default `e2e`), `build` (default the run id), and `sessionName` - label the sessions on the dashboard. + (default `e2e---`) label the sessions on the + dashboard. A given `sessionName` gets `-` appended when the target + has more than one worker slot, so each slot's session has its own name. - `stateDir` (default `.e2e/testmu`) holds each run's daemon. Sessions run over TestMu AI's Appium hub, so agent-device's device settings, diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index 8a2ffc53f..f889fc550 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -63,7 +63,11 @@ export interface TestmuOptions { readonly project?: string | undefined; /** Dashboard build the sessions are grouped under. Defaults to the run id. */ readonly build?: string | undefined; - /** Name of every session on the dashboard; absent, TestMu AI names it. */ + /** + * Name of every session on the dashboard, with `-` appended when the + * target leases more than one device, so each slot's session has its own. + * Defaults to `e2e---`. Slots count from 1. + */ readonly sessionName?: string | undefined; /** Directory for the agent-device daemon each run starts, relative to the project root. Defaults to `.e2e/testmu`. */ readonly stateDir?: string | undefined; @@ -142,7 +146,7 @@ export function testmu(options: TestmuOptions): DeviceProvider { providerDeviceType: deviceType, providerProject: project ?? DEFAULT_PROJECT, providerBuild: build ?? request.runId, - ...(sessionName === undefined ? {} : { providerSessionName: sessionName }), + providerSessionName: slotSessionName(sessionName, request), }; // Not cancellable: the daemon may grant the lease after an interrupt, and only a lease this returns or releases is ever released. const granted = await createAgentDeviceClient({ stateDir, session: `lease-${request.slot}` }).leases.allocate({ @@ -157,7 +161,7 @@ export function testmu(options: TestmuOptions): DeviceProvider { heartbeats.set(scope.leaseId, keepAlive({ stateDir, scope }, request.log)); try { if (request.signal.aborted) throw new Error('cancelled'); - request.log(`lease ${scope.leaseId}: ${device}, ${request.platform} ${osVersion} (${deviceType}); session started`); + request.log(`lease ${scope.leaseId}: ${device}, ${request.platform} ${osVersion} (${deviceType}); session ${selectors.providerSessionName} started`); // The worker's client is created with these fields: the scope picks the lease, and the selectors match the session it holds. return { id: scope.leaseId, client: { stateDir, ...scope, ...selectors } }; } catch (cause) { @@ -207,6 +211,15 @@ function keepAlive({ stateDir, scope }: LeaseHandle, log: (line: string) => void return () => clearInterval(timer); } +/** + * The dashboard name of a slot's session, unique among the run's slots of the + * target so `record` can find the session by it within its build. + */ +function slotSessionName(sessionName: string | undefined, { runId, targetName, slot, slots }: DeviceRequest): string { + if (sessionName === undefined) return `e2e-${runId}-${targetName}-${slot + 1}`; + return slots > 1 ? `${sessionName}-${slot + 1}` : sessionName; +} + /** `app` as the daemon reads it: an `lt://` id or URL as written, a local path resolved against the project root, never the daemon's working directory. */ function appSource(app: string, projectRoot: string): string { if (isAbsolute(app) || /^[a-z][a-z0-9+.-]*:\/\//i.test(app)) return app; diff --git a/packages/testmu/tests/unit/testmu.test.ts b/packages/testmu/tests/unit/testmu.test.ts index 14f34ce01..d17ecba03 100644 --- a/packages/testmu/tests/unit/testmu.test.ts +++ b/packages/testmu/tests/unit/testmu.test.ts @@ -120,6 +120,7 @@ describe('testmu()', () => { providerDeviceType: 'virtual', providerProject: 'e2e', providerBuild: 'run-1', + providerSessionName: 'e2e-run-1-android-2', ttlMs: 10 * MINUTE, }, }, @@ -140,10 +141,19 @@ describe('testmu()', () => { providerDeviceType: 'real', providerProject: 'shop', providerBuild: 'nightly', - providerSessionName: 'checkout', + providerSessionName: 'checkout-1', }); }); + it("names each slot's session after the run, the target, and the slot, and keeps a given name unique per slot", async () => { + const provider = testmu(options); + await provider.acquire(request({ runId: 'run-7', targetName: 'pixel', slot: 0, slots: 3 })); + await provider.acquire(request({ runId: 'run-7', targetName: 'pixel', slot: 2, slots: 3 })); + await testmu({ ...options, sessionName: 'checkout' }).acquire(request({ slot: 2, slots: 3 })); + await testmu({ ...options, sessionName: 'checkout' }).acquire(request({ slot: 0, slots: 1 })); + expect(daemon.calls.map((call) => call.options['providerSessionName'])).toEqual(['e2e-run-7-pixel-1', 'e2e-run-7-pixel-3', 'checkout-3', 'checkout']); + }); + it('resolves a local build against the project root, never the working directory', async () => { await testmu({ ...options, app: 'build/app.apk' }).acquire(request()); expect(daemon.calls[0]?.options['providerApp']).toBe(join(ROOT, 'build', 'app.apk')); @@ -171,13 +181,14 @@ describe('testmu()', () => { providerDeviceType: 'virtual', providerProject: 'e2e', providerBuild: 'run-1', + providerSessionName: 'e2e-run-1-android-1', }, }); const json = JSON.stringify(lease); expect(JSON.parse(json)).toEqual(lease); expect(Buffer.byteLength(json)).toBeLessThan(1024); expect(json).not.toContain('lt-key'); - expect(req.lines).toEqual(['lease lease-1: Galaxy S22 Ultra 5G, android 14 (virtual); session started']); + expect(req.lines).toEqual(['lease lease-1: Galaxy S22 Ultra 5G, android 14 (virtual); session e2e-run-1-android-1 started']); }); it('shares the run\'s credentials with the daemon it starts', async () => { From e1eeefd0c1e6992830775009aa163697d4506509 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:08:25 +0530 Subject: [PATCH 07/22] feat(testmu): accept device-feature options `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` set up the device when its session starts. Each one is allocated under agent-device's matching lease key (`providerDeviceOrientation`, `providerGeoLocation`, `providerTimezone`, `providerLanguage`, `providerLocale`, `providerAppiumVersion`), which its `testmu` provider turns into TestMu AI capabilities. An empty value, or an orientation other than portrait or landscape, fails the config load with INVALID_CONFIG. Co-Authored-By: Claude Opus 5.5 --- .changeset/testmu-devices.md | 2 +- docs/integrations/testmu.mdx | 12 ++++++- packages/testmu/README.md | 3 ++ packages/testmu/src/provider.ts | 43 +++++++++++++++++++++++ packages/testmu/tests/unit/testmu.test.ts | 25 +++++++++++++ 5 files changed, 83 insertions(+), 2 deletions(-) diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md index e7f92f388..0c0b65c89 100644 --- a/.changeset/testmu-devices.md +++ b/.changeset/testmu-devices.md @@ -2,4 +2,4 @@ "@e2e-dev/testmu": minor --- -`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 4af42ef6f..827a171f0 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -169,9 +169,19 @@ device a signed `.ipa`. Real devices need an agent-device release whose | `build` | the run id | Dashboard build the sessions are grouped under. | | `sessionName` | `e2e---` | Name of each session on the dashboard. With more than one worker slot, `-` is appended, so every slot's session has its own name; slots count from 1. Give targets that share a `build` different names. | | `stateDir` | `'.e2e/testmu'` | Directory for the agent-device daemon each run starts, relative to the directory of `e2e.config.ts`. Keep it out of version control. | +| `orientation` | the device's | `'portrait'` or `'landscape'`: the orientation the device starts in. | +| `geoLocation` | TestMu AI's | Country the device's IP address geolocates to, as a country code TestMu AI takes (`'US'`, `'FR'`). | +| `timezone` | the device's | The device's time zone, as TestMu AI takes it (`'UTC+05:30'`). | +| `language` | the device's | The device's language, as a language code (`'fr'`). | +| `locale` | the device's | The device's locale (`'fr_FR'`). | +| `appiumVersion` | TestMu AI's default | Appium version TestMu AI starts for the session (`'2.16.2'`). | + +The device features, `orientation` through `appiumVersion`, are set when the +session starts and hold for the whole session; a test cannot change them. An option not in this table, a missing or empty `device`, `osVersion`, or -`app`, or another `deviceType` fails the config load with `INVALID_CONFIG`. +`app`, another `deviceType` or `orientation`, or an empty device feature +fails the config load with `INVALID_CONFIG`. ## Write a test diff --git a/packages/testmu/README.md b/packages/testmu/README.md index 786b989d9..2a67b8a32 100644 --- a/packages/testmu/README.md +++ b/packages/testmu/README.md @@ -56,6 +56,9 @@ ends, which ends the session. (default `e2e---`) label the sessions on the dashboard. A given `sessionName` gets `-` appended when the target has more than one worker slot, so each slot's session has its own name. +- `orientation` (`'portrait'` or `'landscape'`), `geoLocation`, `timezone`, + `language`, `locale`, and `appiumVersion` set up the device when the + session starts. - `stateDir` (default `.e2e/testmu`) holds each run's daemon. Sessions run over TestMu AI's Appium hub, so agent-device's device settings, diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index f889fc550..3447343fb 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -33,6 +33,18 @@ const HEARTBEAT_INTERVAL_MS = 2 * 60_000; const DEVICE_TYPES: ReadonlySet = new Set(['virtual', 'real']); +const ORIENTATIONS: ReadonlySet = new Set(['portrait', 'landscape']); + +/** Each device-feature option and the agent-device lease key it is allocated under. */ +const DEVICE_FEATURES = { + orientation: 'providerDeviceOrientation', + geoLocation: 'providerGeoLocation', + timezone: 'providerTimezone', + language: 'providerLanguage', + locale: 'providerLocale', + appiumVersion: 'providerAppiumVersion', +} as const satisfies Partial>; + /** Every option `testmu()` takes, kept equal to `TestmuOptions` by the compiler. */ const OPTION_KEYS: readonly string[] = Object.keys({ device: true, @@ -43,6 +55,12 @@ const OPTION_KEYS: readonly string[] = Object.keys({ build: true, sessionName: true, stateDir: true, + orientation: true, + geoLocation: true, + timezone: true, + language: true, + locale: true, + appiumVersion: true, } satisfies Record); /** What `testmu()` takes: the device, its OS version, and the app TestMu AI installs on it. */ @@ -71,6 +89,18 @@ export interface TestmuOptions { readonly sessionName?: string | undefined; /** Directory for the agent-device daemon each run starts, relative to the project root. Defaults to `.e2e/testmu`. */ readonly stateDir?: string | undefined; + /** Orientation the device starts in; absent, the device's default. */ + readonly orientation?: 'portrait' | 'landscape' | undefined; + /** Country the device's IP geolocates to, as a code TestMu AI takes: `US`, `FR`. */ + readonly geoLocation?: string | undefined; + /** The device's time zone, as TestMu AI takes it: `UTC+05:30`. */ + readonly timezone?: string | undefined; + /** The device's language, as a language code: `fr`. */ + readonly language?: string | undefined; + /** The device's locale: `fr_FR`. */ + readonly locale?: string | undefined; + /** Appium version TestMu AI starts for the session; absent, its default for the device. */ + readonly appiumVersion?: string | undefined; } type LeaseBackend = 'ios-instance' | 'android-instance'; @@ -112,6 +142,18 @@ export function testmu(options: TestmuOptions): DeviceProvider { if (!DEVICE_TYPES.has(deviceType)) { throw new ConfigurationError('INVALID_CONFIG', `testmu: \`deviceType\` must be 'virtual' or 'real', not ${JSON.stringify(deviceType)}`); } + const deviceFeatures: Record = {}; + for (const [key, leaseKey] of Object.entries(DEVICE_FEATURES)) { + const value: unknown = options[key as keyof typeof DEVICE_FEATURES]; + if (value === undefined) continue; + if (typeof value !== 'string' || value.trim() === '') { + throw new ConfigurationError('INVALID_CONFIG', `testmu: \`${key}\` must be a non-empty string`); + } + if (key === 'orientation' && !ORIENTATIONS.has(value)) { + throw new ConfigurationError('INVALID_CONFIG', `testmu: \`orientation\` must be 'portrait' or 'landscape', not ${JSON.stringify(value)}`); + } + deviceFeatures[leaseKey] = value; + } const { device, osVersion, app, project, build, sessionName } = options; /** One release per lease, shared by every caller: the engine's, and `acquire`'s own after a failure. */ const releases = new Map>(); @@ -147,6 +189,7 @@ export function testmu(options: TestmuOptions): DeviceProvider { providerProject: project ?? DEFAULT_PROJECT, providerBuild: build ?? request.runId, providerSessionName: slotSessionName(sessionName, request), + ...deviceFeatures, }; // Not cancellable: the daemon may grant the lease after an interrupt, and only a lease this returns or releases is ever released. const granted = await createAgentDeviceClient({ stateDir, session: `lease-${request.slot}` }).leases.allocate({ diff --git a/packages/testmu/tests/unit/testmu.test.ts b/packages/testmu/tests/unit/testmu.test.ts index d17ecba03..ab489a0f1 100644 --- a/packages/testmu/tests/unit/testmu.test.ts +++ b/packages/testmu/tests/unit/testmu.test.ts @@ -154,6 +154,20 @@ describe('testmu()', () => { expect(daemon.calls.map((call) => call.options['providerSessionName'])).toEqual(['e2e-run-7-pixel-1', 'e2e-run-7-pixel-3', 'checkout-3', 'checkout']); }); + it('passes the device features to the allocation and the worker under agent-device\'s keys', async () => { + const lease = await testmu({ ...options, orientation: 'landscape', geoLocation: 'US', timezone: 'UTC+05:30', language: 'fr', locale: 'fr_FR', appiumVersion: '2.16.2' }).acquire(request()); + const features = { + providerDeviceOrientation: 'landscape', + providerGeoLocation: 'US', + providerTimezone: 'UTC+05:30', + providerLanguage: 'fr', + providerLocale: 'fr_FR', + providerAppiumVersion: '2.16.2', + }; + expect(daemon.calls[0]?.options).toMatchObject(features); + expect(lease.client).toMatchObject(features); + }); + it('resolves a local build against the project root, never the working directory', async () => { await testmu({ ...options, app: 'build/app.apk' }).acquire(request()); expect(daemon.calls[0]?.options['providerApp']).toBe(join(ROOT, 'build', 'app.apk')); @@ -362,6 +376,17 @@ describe('testmu()', () => { expect(() => testmu(rest as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG' })); }); + it('refuses an orientation other than portrait or landscape', () => { + expect(() => testmu({ ...options, orientation: 'PORTRAIT' as 'portrait' })).toThrow( + expect.objectContaining({ code: 'INVALID_CONFIG', message: 'testmu: `orientation` must be \'portrait\' or \'landscape\', not "PORTRAIT"' }), + ); + }); + + it.each(['geoLocation', 'timezone', 'language', 'locale', 'appiumVersion'] as const)('refuses an empty or non-string `%s` with INVALID_CONFIG', (key) => { + expect(() => testmu({ ...options, [key]: ' ' })).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG', message: `testmu: \`${key}\` must be a non-empty string` })); + expect(() => testmu({ ...options, [key]: 2 } as unknown as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG' })); + }); + it('refuses a device type other than virtual or real', () => { expect(() => testmu({ ...options, deviceType: 'emulator' as 'virtual' })).toThrow( expect.objectContaining({ code: 'INVALID_CONFIG', message: 'testmu: `deviceType` must be \'virtual\' or \'real\', not "emulator"' }), From 526abdd030e5e0dc8fa4db0f61fcb887363d9ebf Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:11:20 +0530 Subject: [PATCH 08/22] feat(testmu): record attempt video from the TestMu AI session agent-device's cloud WebDriver runtime cannot record, so a run with `video: 'on'` or `'retain-on-failure'` on a TestMu AI target failed when the attempt started its video. The provider now implements `record`: the attempt's video links TestMu AI's own recording of the slot's session. `record` runs in the worker from the lease alone. It reads the build and session name the lease carries and the credentials from the run's environment, and starts at the moment it is called. `stop` finds the session through TestMu AI's sessions API (the newest session with that name in the build's list, paged) and returns the `video_url` of its details as `video/mp4`. The API defaults to https://mobile-api.lambdatest.com/mobile-automation/api/v1 and follows `TESTMU_API_ENDPOINT`, as agent-device does. Every request is bounded by a 15-second timeout and the stop's signal, and errors name the build and the session without credentials or URLs. Co-Authored-By: Claude Opus 5.5 --- .changeset/testmu-devices.md | 2 +- docs/integrations/testmu.mdx | 35 +++- packages/testmu/README.md | 3 +- packages/testmu/src/credentials.ts | 2 +- packages/testmu/src/provider.ts | 26 ++- packages/testmu/src/sessions.ts | 108 ++++++++++ packages/testmu/tests/unit/recording.test.ts | 201 +++++++++++++++++++ packages/testmu/tests/unit/testmu.test.ts | 6 +- skills/e2e/references/setup.md | 1 + 9 files changed, 373 insertions(+), 11 deletions(-) create mode 100644 packages/testmu/src/sessions.ts create mode 100644 packages/testmu/tests/unit/recording.test.ts diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md index 0c0b65c89..d8f87b7f4 100644 --- a/.changeset/testmu-devices.md +++ b/.changeset/testmu-devices.md @@ -2,4 +2,4 @@ "@e2e-dev/testmu": minor --- -`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session; the attempt's start time is recorded), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 827a171f0..6ac6f67e9 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -219,6 +219,36 @@ The provider logs each lease as it is granted: The video, device logs, and network logs of every session are on the TestMu AI dashboard, under the project and build the options name. +## Recordings + +An attempt that records [video](/reference/config#video) links TestMu AI's +recording of its session in the report, instead of recording through +agent-device, which cannot record over TestMu AI's Appium hub: + +```ts title="e2e.config.ts" +export default { + targets: [ + { + name: 'android-emulator', + engine: mobile({ + platform: 'android', + device: testmu({ device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: apk }), + }), + app: { bundleId: 'com.lambdatest.proverbial' }, + video: 'retain-on-failure', + }, + ], +} satisfies E2EConfig; +``` + +TestMu AI records the whole session, from the lease to the end of the run, +so the link is the same video for every attempt on that worker slot. The +report records when the attempt started recording, to find the attempt in +it. The runner never downloads the video; the link is the URL TestMu AI's +sessions API returns for the session, and the provider finds the session by +its build and name when the attempt stops recording. It reads the API at +`TESTMU_API_ENDPOINT` when that is set, as agent-device does. + ## What the provider does - Starts one agent-device daemon per run under `stateDir/` and @@ -245,9 +275,8 @@ tests: - `device` settings: permissions, location, network, and appearance. - System alerts. -- Recording through agent-device. Leave [`video`](/reference/config#video) - off for these targets. TestMu AI records every session itself, and the - video and device logs are on its dashboard. +- Recording through agent-device. An attempt's video links TestMu AI's + recording of the session instead; see [Recordings](#recordings). - Device logs through agent-device; read them on the dashboard. - Port reverse. To reach a server on your machine or network, use TestMu AI Tunnel. diff --git a/packages/testmu/README.md b/packages/testmu/README.md index 2a67b8a32..5e6960157 100644 --- a/packages/testmu/README.md +++ b/packages/testmu/README.md @@ -63,7 +63,8 @@ ends, which ends the session. Sessions run over TestMu AI's Appium hub, so agent-device's device settings, system alerts, recording, device logs, and port reverse are not available -there. TestMu AI records every session itself. +there. An attempt that records video links TestMu AI's recording of the +whole session instead, found by the session's build and name. Full documentation lives at [e2e.tester.army/docs/integrations/testmu](https://e2e.tester.army/docs/integrations/testmu). diff --git a/packages/testmu/src/credentials.ts b/packages/testmu/src/credentials.ts index faa6b987e..0970a7abb 100644 --- a/packages/testmu/src/credentials.ts +++ b/packages/testmu/src/credentials.ts @@ -8,7 +8,7 @@ const LT_USERNAME = 'LT_USERNAME'; const LT_ACCESS_KEY = 'LT_ACCESS_KEY'; -interface TestmuCredentials { +export interface TestmuCredentials { readonly username: string; readonly accessKey: string; } diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index 3447343fb..487b5d1ee 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -8,8 +8,9 @@ import { isAbsolute, resolve } from 'node:path'; import type { DeviceLease, DeviceProvider, DeviceRequest } from '@e2e-dev/mobile'; import { createAgentDeviceClient } from 'agent-device'; -import { ConfigurationError, rejectUnknownKeys } from 'e2e/engine'; +import { ConfigurationError, rejectUnknownKeys, type ProviderRecordContext, type ProviderRecording } from 'e2e/engine'; import { shareWithDaemon, testmuCredentials } from './credentials.ts'; +import { sessionVideoUrl, testmuApiEndpoint, type SessionRef } from './sessions.ts'; /** agent-device's name for TestMu AI, as the lease provider and the tenant. */ const PROVIDER = 'testmu'; @@ -127,7 +128,8 @@ interface LeaseHandle { * starts for the run under `stateDir`; allocating it starts the TestMu AI * session, which installs `app`, so every slot is billed from the moment it * is leased, whether or not a test runs on it, and releasing the lease ends it. - * It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's + * An attempt that records video links TestMu AI's recording of the whole + * session, found by the lease's build and session name. It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's * environment, and needs an agent-device with the `testmu` provider. */ export function testmu(options: TestmuOptions): DeviceProvider { @@ -221,6 +223,17 @@ export function testmu(options: TestmuOptions): DeviceProvider { if (handle === undefined) throw new Error(`lease ${lease.id} carries no agent-device lease scope to release`); await release(lease.id, handle); }, + // Runs in the worker, from the lease alone. TestMu AI records the whole session, not the attempt. + async record(lease: DeviceLease, context: ProviderRecordContext): Promise { + const session = sessionRef(lease); + if (session === undefined) throw new Error(`lease ${lease.id} carries no TestMu AI build and session name to find its recording by`); + const credentials = testmuCredentials(context.env); + const endpoint = testmuApiEndpoint(context.env); + return { + startedAt: new Date().toISOString(), + stop: async ({ signal }) => ({ url: await sessionVideoUrl(endpoint, credentials, session, signal), mediaType: 'video/mp4' }), + }; + }, }; } @@ -287,6 +300,15 @@ function leaseHandle(lease: DeviceLease): LeaseHandle | undefined { return { stateDir, scope: { tenant, runId, leaseId, leaseBackend, leaseProvider } }; } +/** The build and session name `acquire` put on a lease's `client`, when they are there. */ +function sessionRef(lease: DeviceLease): SessionRef | undefined { + const client = lease.client as Record | undefined; + const build = client?.['providerBuild']; + const sessionName = client?.['providerSessionName']; + if (typeof build !== 'string' || build === '' || typeof sessionName !== 'string' || sessionName === '') return undefined; + return { build, sessionName }; +} + function messageOf(cause: unknown): string { return cause instanceof Error ? cause.message : String(cause); } diff --git a/packages/testmu/src/sessions.ts b/packages/testmu/src/sessions.ts new file mode 100644 index 000000000..6ce52921f --- /dev/null +++ b/packages/testmu/src/sessions.ts @@ -0,0 +1,108 @@ +/** + * TestMu AI's mobile automation sessions API over `fetch`, the one + * agent-device's `testmu` provider reads session artifacts from: finds a + * session by its build and name, and reads the URL of its video. + */ + +import type { TestmuCredentials } from './credentials.ts'; + +/** The API's base, as agent-device's `TESTMU_API_ENDPOINT` sets it. */ +const DEFAULT_API_ENDPOINT = 'https://mobile-api.lambdatest.com/mobile-automation/api/v1'; + +const REQUEST_TIMEOUT_MS = 15_000; + +/** Sessions the list asks for at once. */ +const PAGE_SIZE = 50; + +/** Pages the lookup reads before it gives up on a build: the list is newest first, so the session is near the top. */ +const MAX_PAGES = 10; + +/** A session the recording covers, as the lease named it. */ +export interface SessionRef { + readonly build: string; + readonly sessionName: string; +} + +/** The sessions API at `TESTMU_API_ENDPOINT` from the run's environment, else TestMu AI's own; throws when the override is not an http(s) URL. */ +export function testmuApiEndpoint(env: Readonly>): string { + const override = env['TESTMU_API_ENDPOINT']?.trim(); + if (override === undefined || override === '') return DEFAULT_API_ENDPOINT; + if (!URL.canParse(override) || !/^https?:$/.test(new URL(override).protocol)) throw new Error('TESTMU_API_ENDPOINT is not an http(s) URL'); + return override.replace(/\/+$/, ''); +} + +/** + * The video URL of the newest session named `sessionName` in `build`: the + * list finds the session's `test_id`, and its details carry `video_url`. + * Every request is bounded by `REQUEST_TIMEOUT_MS` and `signal`. Errors name + * the build and the session, never the credentials or a URL. + */ +export async function sessionVideoUrl(endpoint: string, credentials: TestmuCredentials, { build, sessionName }: SessionRef, signal: AbortSignal): Promise { + const auth = `Basic ${Buffer.from(`${credentials.username}:${credentials.accessKey}`).toString('base64')}`; + const id = await findSession(endpoint, auth, { build, sessionName }, signal); + const what = `TestMu AI session ${id} (${JSON.stringify(sessionName)}, build ${JSON.stringify(build)})`; + const body = await getJson(new URL(`${endpoint}/sessions/${encodeURIComponent(id)}`), auth, signal, `${what} details`); + const details = asRecord(body['data']); + const videoUrl = details?.['video_url']; + if (typeof videoUrl !== 'string' || !isHttpUrl(videoUrl)) throw new Error(`${what} reports no video URL`); + return videoUrl; +} + +/** The `test_id` of the newest session named `sessionName` in `build`, reading the list a page at a time. */ +async function findSession(endpoint: string, auth: string, { build, sessionName }: SessionRef, signal: AbortSignal): Promise { + const what = `TestMu AI session lookup for build ${JSON.stringify(build)}`; + for (let page = 0; page < MAX_PAGES; page += 1) { + const url = new URL(`${endpoint}/sessions`); + url.searchParams.set('build', build); + url.searchParams.set('limit', String(PAGE_SIZE)); + url.searchParams.set('offset', String(page * PAGE_SIZE)); + const body = await getJson(url, auth, signal, what); + // An empty page comes back as `data: null`. + const rows = body['data'] ?? []; + if (!Array.isArray(rows)) throw new Error(`${what} failed: no session list in the response`); + for (const row of rows) { + const { name, test_id: id } = asRecord(row) ?? {}; + if (name === sessionName && typeof id === 'string' && id !== '') return id; + } + if (rows.length < PAGE_SIZE) break; + } + throw new Error(`TestMu AI has no session named ${JSON.stringify(sessionName)} in build ${JSON.stringify(build)}`); +} + +/** One authenticated GET answered with a JSON object; anything else throws, as `what`, with the HTTP status and the API's message. */ +async function getJson(url: URL, auth: string, signal: AbortSignal, what: string): Promise> { + const timeout = new AbortController(); + const timer = setTimeout(() => timeout.abort(), REQUEST_TIMEOUT_MS); + let response: Response; + let text: string; + try { + response = await fetch(url, { headers: { Authorization: auth, Accept: 'application/json' }, signal: AbortSignal.any([signal, timeout.signal]) }); + text = await response.text(); + } catch (cause) { + if (signal.aborted) throw new Error(`${what} cancelled`, { cause }); + if (timeout.signal.aborted) throw new Error(`${what} got no answer within ${REQUEST_TIMEOUT_MS / 1000} s`, { cause }); + throw new Error(`${what} failed: ${cause instanceof Error ? cause.message : String(cause)}`, { cause }); + } finally { + clearTimeout(timer); + } + let body: Record | undefined; + try { + body = asRecord(JSON.parse(text)); + } catch { + body = undefined; + } + if (body === undefined) throw new Error(`${what} failed: HTTP ${response.status}, not JSON`); + if (!response.ok) { + const message = body['message']; + throw new Error(`${what} failed: HTTP ${response.status}${typeof message === 'string' && message !== '' ? ` (${message.slice(0, 200)})` : ''}`); + } + return body; +} + +function asRecord(value: unknown): Record | undefined { + return typeof value === 'object' && value !== null && !Array.isArray(value) ? (value as Record) : undefined; +} + +function isHttpUrl(value: string): boolean { + return URL.canParse(value) && /^https?:$/.test(new URL(value).protocol); +} diff --git a/packages/testmu/tests/unit/recording.test.ts b/packages/testmu/tests/unit/recording.test.ts new file mode 100644 index 000000000..b918a5f99 --- /dev/null +++ b/packages/testmu/tests/unit/recording.test.ts @@ -0,0 +1,201 @@ +/** + * `testmu().record()` against a fake TestMu AI sessions API behind a stubbed + * `fetch`: the session it finds by build and name, the video it links, the + * endpoint override, and the failures it names without leaking credentials + * or signed URLs. + */ + +import type { DeviceLease } from '@e2e-dev/mobile'; +import type { ProviderRecordContext } from 'e2e/engine'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { testmu, type TestmuOptions } from '../../src/index.ts'; + +vi.mock('agent-device', () => ({ + createAgentDeviceClient: () => { + throw new Error('recording starts no daemon'); + }, +})); + +const API = 'https://mobile-api.lambdatest.com/mobile-automation/api/v1'; +const VIDEO = 'https://videos.example.com/orgId-1/T2/video/video.mp4?X-Amz-Signature=secret-signature'; +const options: TestmuOptions = { device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: 'lt://APP1' }; +const env = { LT_USERNAME: 'ada', LT_ACCESS_KEY: 'lt-key' }; +const lease: DeviceLease = { + id: 'lease-1', + client: { stateDir: '/work/.e2e/testmu/run-1', tenant: 'testmu', runId: 'run-1', leaseId: 'lease-1', providerBuild: 'run-1', providerSessionName: 'e2e-run-1-android-2' }, +}; + +interface Call { + readonly url: URL; + readonly authorization: string | undefined; +} + +const api = { + calls: [] as Call[], + /** Every session the build holds, newest first, as the list returns them. */ + sessions: [] as Record[], + /** Each session's details by `test_id`. */ + details: {} as Record>, + /** A response the next request gets instead of the fake's own. */ + override: undefined as (() => Response | Promise) | undefined, +}; + +beforeEach(() => { + Object.assign(api, { + calls: [], + sessions: [ + { test_id: 'T3', name: 'e2e-run-1-android-1', build_name: 'run-1' }, + { test_id: 'T2', name: 'e2e-run-1-android-2', build_name: 'run-1' }, + { test_id: 'T1', name: 'e2e-run-1-android-2', build_name: 'run-1' }, + ], + details: { T2: { test_id: 'T2', name: 'e2e-run-1-android-2', video_url: VIDEO }, T1: { test_id: 'T1', video_url: 'https://videos.example.com/old.mp4' } }, + override: undefined, + }); + vi.stubGlobal('fetch', async (input: string | URL, init: RequestInit = {}) => { + const url = new URL(input); + api.calls.push({ url, authorization: (init.headers as Record | undefined)?.['Authorization'] }); + init.signal?.throwIfAborted(); + const override = api.override; + if (override !== undefined) { + api.override = undefined; + return override(); + } + const base = new URL(API).pathname; + if (url.pathname === `${base}/sessions`) { + const build = url.searchParams.get('build'); + const limit = Number(url.searchParams.get('limit') ?? '10'); + const offset = Number(url.searchParams.get('offset') ?? '0'); + const rows = api.sessions.filter((row) => row['build_name'] === build).slice(offset, offset + limit); + // An empty page is Go's nil slice: `data: null`. + return Response.json({ status: 'success', data: rows.length === 0 ? null : rows, message: 'Retrieve session list was successful', Meta: { result_set: { count: rows.length } } }); + } + const id = decodeURIComponent(url.pathname.slice(`${base}/sessions/`.length)); + const details = api.details[id]; + if (details === undefined) return Response.json({ status: 'fail', message: 'Not found' }, { status: 404 }); + return Response.json({ status: 'success', data: details, message: 'Retrieve session was successful' }); + }); +}); + +afterEach(() => { + vi.unstubAllGlobals(); + vi.useRealTimers(); +}); + +function context(overrides: Partial = {}): ProviderRecordContext { + return { runId: 'run-1', targetName: 'android', attemptId: 'attempt-1', env, signal: new AbortController().signal, ...overrides }; +} + +const stopContext = (signal = new AbortController().signal) => ({ dir: '/work/.e2e/artifacts/attempt-1/video', signal }); + +/** The error `promise` rejects with; fails the test when it resolves. */ +async function failure(promise: Promise): Promise { + return promise.then( + () => { + throw new Error('expected a failure'); + }, + (cause: unknown) => cause as Error, + ); +} + +async function record(recordLease: DeviceLease = lease, recordContext: ProviderRecordContext = context()) { + const provider = testmu(options); + if (provider.record === undefined) throw new Error('testmu() records nothing'); + return provider.record(recordLease, recordContext); +} + +describe('testmu().record()', () => { + it('starts at the moment it is called and asks TestMu AI nothing until it stops', async () => { + vi.useFakeTimers({ toFake: ['Date'] }); + vi.setSystemTime(new Date('2026-10-02T10:00:00.000Z')); + const recording = await record(); + expect(recording.startedAt).toBe('2026-10-02T10:00:00.000Z'); + expect(api.calls).toEqual([]); + }); + + it("links the video of the newest session with the slot's name in the lease's build", async () => { + const recording = await record(); + await expect(recording.stop(stopContext())).resolves.toEqual({ url: VIDEO, mediaType: 'video/mp4' }); + expect(api.calls.map((call) => call.url.href)).toEqual([`${API}/sessions?build=run-1&limit=50&offset=0`, `${API}/sessions/T2`]); + expect(api.calls.map((call) => call.authorization)).toEqual([`Basic ${Buffer.from('ada:lt-key').toString('base64')}`, `Basic ${Buffer.from('ada:lt-key').toString('base64')}`]); + }); + + it('reads the lease after a round trip through JSON, as the worker gets it', async () => { + const recording = await record(JSON.parse(JSON.stringify(lease)) as DeviceLease); + await expect(recording.stop(stopContext())).resolves.toMatchObject({ url: VIDEO }); + }); + + it('pages through a build holding more sessions than one page', async () => { + api.sessions = [...Array.from({ length: 50 }, (_, index) => ({ test_id: `X${index}`, name: `other-${index}`, build_name: 'run-1' })), ...api.sessions]; + await expect((await record()).stop(stopContext())).resolves.toMatchObject({ url: VIDEO }); + expect(api.calls.map((call) => call.url.searchParams.get('offset'))).toEqual(['0', '50', null]); + }); + + it('asks the API TESTMU_API_ENDPOINT names', async () => { + const recording = await record(lease, context({ env: { ...env, TESTMU_API_ENDPOINT: 'https://stage-mobile-api.lambdatest.com/mobile-automation/api/v1/' } })); + await expect(recording.stop(stopContext())).resolves.toMatchObject({ url: VIDEO }); + expect(api.calls.map((call) => call.url.href)).toEqual([ + 'https://stage-mobile-api.lambdatest.com/mobile-automation/api/v1/sessions?build=run-1&limit=50&offset=0', + 'https://stage-mobile-api.lambdatest.com/mobile-automation/api/v1/sessions/T2', + ]); + }); + + it('names the build and the session when the build has no session by that name', async () => { + api.sessions = api.sessions.filter((row) => row['name'] !== 'e2e-run-1-android-2'); + await expect((await record()).stop(stopContext())).rejects.toThrow('TestMu AI has no session named "e2e-run-1-android-2" in build "run-1"'); + }); + + it('names the session when its details carry no video URL', async () => { + api.details['T2'] = { test_id: 'T2', video_url: 'not a url' }; + await expect((await record()).stop(stopContext())).rejects.toThrow('TestMu AI session T2 ("e2e-run-1-android-2", build "run-1") reports no video URL'); + delete api.details['T2']!['video_url']; + await expect((await record()).stop(stopContext())).rejects.toThrow('reports no video URL'); + }); + + it('reports an HTTP failure with its status and message, never the credentials or the request URL', async () => { + api.override = () => Response.json({ status: 'fail', message: 'Unauthorized' }, { status: 401 }); + const error = await failure((await record()).stop(stopContext())); + expect(error.message).toBe('TestMu AI session lookup for build "run-1" failed: HTTP 401 (Unauthorized)'); + expect(JSON.stringify(error, Object.getOwnPropertyNames(error))).not.toContain('lt-key'); + }); + + it('names the session when its details request fails', async () => { + api.details = {}; + await expect((await record()).stop(stopContext())).rejects.toThrow('TestMu AI session T2 ("e2e-run-1-android-2", build "run-1") details failed: HTTP 404 (Not found)'); + }); + + it('reports a response that is not JSON, or not a session list', async () => { + api.override = () => new Response('Bad gateway', { status: 502 }); + await expect((await record()).stop(stopContext())).rejects.toThrow('TestMu AI session lookup for build "run-1" failed: HTTP 502, not JSON'); + api.override = () => Response.json({ status: 'success', data: { sessions: [] } }); + await expect((await record()).stop(stopContext())).rejects.toThrow('TestMu AI session lookup for build "run-1" failed: no session list in the response'); + }); + + it('gives up on a request TestMu AI does not answer within 15 seconds', async () => { + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }); + vi.stubGlobal('fetch', (input: string | URL, init: RequestInit = {}) => { + api.calls.push({ url: new URL(input), authorization: undefined }); + return new Promise((_, reject) => init.signal?.addEventListener('abort', () => reject(init.signal?.reason))); + }); + const stopped = failure((await record()).stop(stopContext())); + await vi.advanceTimersByTimeAsync(15_000); + expect((await stopped).message).toBe('TestMu AI session lookup for build "run-1" got no answer within 15 s'); + }); + + it('stops asking once the stop is cancelled', async () => { + const controller = new AbortController(); + controller.abort(); + await expect((await record()).stop(stopContext(controller.signal))).rejects.toThrow('TestMu AI session lookup for build "run-1" cancelled'); + }); + + it('refuses to start for a lease without a build and a session name', async () => { + await expect(record({ id: 'other', client: { stateDir: '/tmp/x', providerBuild: 'run-1' } })).rejects.toThrow( + 'lease other carries no TestMu AI build and session name to find its recording by', + ); + }); + + it('refuses to start without credentials or with an endpoint that is not a URL', async () => { + await expect(record(lease, context({ env: {} }))).rejects.toThrow('LT_USERNAME and LT_ACCESS_KEY are not set'); + await expect(record(lease, context({ env: { ...env, TESTMU_API_ENDPOINT: 'mobile-api' } }))).rejects.toThrow('TESTMU_API_ENDPOINT is not an http(s) URL'); + expect(api.calls).toEqual([]); + }); +}); diff --git a/packages/testmu/tests/unit/testmu.test.ts b/packages/testmu/tests/unit/testmu.test.ts index ab489a0f1..120cdd46a 100644 --- a/packages/testmu/tests/unit/testmu.test.ts +++ b/packages/testmu/tests/unit/testmu.test.ts @@ -2,7 +2,7 @@ * `testmu()` against a stubbed agent-device client: the options it refuses, * the lease it allocates and hands the worker, the credentials it reads and * shares with the daemon, the heartbeat that keeps it alive, and the release - * on every exit path. + * on every exit path. `recording.test.ts` covers `record`. */ import { join } from 'node:path'; @@ -95,10 +95,10 @@ const operations = () => daemon.calls.map((call) => call.operation); const MINUTE = 60_000; describe('testmu()', () => { - it('is a device provider named testmu that leaves recording to agent-device', () => { + it("is a device provider named testmu that records through TestMu AI's own session video", () => { const provider = testmu(options); expect(provider.name).toBe('testmu'); - expect(provider.record).toBeUndefined(); + expect(provider.record).toBeTypeOf('function'); }); it('allocates an Android lease from a daemon under the project root, with the device selectors and dashboard labels', async () => { diff --git a/skills/e2e/references/setup.md b/skills/e2e/references/setup.md index b954baf7d..0a1331296 100644 --- a/skills/e2e/references/setup.md +++ b/skills/e2e/references/setup.md @@ -311,6 +311,7 @@ export default { exactly, TestMu AI installs `app` (omit `app.appPath`), it reads `LT_USERNAME` and `LT_ACCESS_KEY`, and needs an agent-device with the `testmu` provider shared with `@e2e-dev/mobile` (override its pin). + An attempt's video links TestMu AI's recording of the whole session. - Only a control that appeared or moved with the previous action waits out `transition` (default 500 ms); agent actions settle `settle` ms (default 150) before the next observation, `settle: false` skips it. From 5c1038876ca1cc8a15b5b8e245d5419f6394038f Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:12:34 +0530 Subject: [PATCH 09/22] chore(testmu): prune old run state Each run leaves `/` behind with its daemon's state and logs, and nothing removed it. The directory is still kept at release, since the daemon exits by itself after 5 idle minutes and its logs help with a failure. Instead, the provider's first acquire removes earlier runs' directories under `stateDir` last modified more than 24 hours ago. It never touches the current run's directory or anything not named like a run id, and a failure leaves the directory and does not fail the lease. Co-Authored-By: Claude Opus 5.5 --- .changeset/testmu-devices.md | 2 +- docs/integrations/testmu.mdx | 8 +- packages/testmu/README.md | 4 +- packages/testmu/src/provider.ts | 49 +++++++++++- packages/testmu/tests/unit/prune.test.ts | 95 ++++++++++++++++++++++++ 5 files changed, 152 insertions(+), 6 deletions(-) create mode 100644 packages/testmu/tests/unit/prune.test.ts diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md index d8f87b7f4..2a895a657 100644 --- a/.changeset/testmu-devices.md +++ b/.changeset/testmu-devices.md @@ -2,4 +2,4 @@ "@e2e-dev/testmu": minor --- -`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`); allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session; the attempt's start time is recorded), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`), in a directory per run that later runs remove once it is more than 24 hours old; allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session; the attempt's start time is recorded), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 6ac6f67e9..317996e91 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -168,7 +168,7 @@ device a signed `.ipa`. Real devices need an agent-device release whose | `project` | `'e2e'` | Dashboard project the sessions are grouped under. | | `build` | the run id | Dashboard build the sessions are grouped under. | | `sessionName` | `e2e---` | Name of each session on the dashboard. With more than one worker slot, `-` is appended, so every slot's session has its own name; slots count from 1. Give targets that share a `build` different names. | -| `stateDir` | `'.e2e/testmu'` | Directory for the agent-device daemon each run starts, relative to the directory of `e2e.config.ts`. Keep it out of version control. | +| `stateDir` | `'.e2e/testmu'` | Directory for the agent-device daemon each run starts, relative to the directory of `e2e.config.ts`. Keep it out of version control. See [What the provider does](#what-the-provider-does) for how long a run's directory is kept. | | `orientation` | the device's | `'portrait'` or `'landscape'`: the orientation the device starts in. | | `geoLocation` | TestMu AI's | Country the device's IP address geolocates to, as a country code TestMu AI takes (`'US'`, `'FR'`). | | `timezone` | the device's | The device's time zone, as TestMu AI takes it (`'UTC+05:30'`). | @@ -254,6 +254,12 @@ its build and name when the attempt stops recording. It reads the API at - Starts one agent-device daemon per run under `stateDir/` and allocates a `testmu` lease from it for each worker slot. The allocation starts the TestMu AI session, which installs `app`. +- Keeps each run's directory, with its daemon's state and logs, after the + run: the daemon exits on its own after 5 idle minutes, and its logs help + explain a failure. Before its first lease, a run removes the directories + of earlier runs under `stateDir` that were last modified more than 24 + hours ago, never its own; only directories named after a run id are + removed, and a failure to remove one does not fail the lease. - Keeps each lease alive while the run holds it. agent-device starts a lease's inactivity window before a slow allocation returns, so the provider asks for agent-device's longest window, 10 minutes, and diff --git a/packages/testmu/README.md b/packages/testmu/README.md index 5e6960157..3d5129d16 100644 --- a/packages/testmu/README.md +++ b/packages/testmu/README.md @@ -59,7 +59,9 @@ ends, which ends the session. - `orientation` (`'portrait'` or `'landscape'`), `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` set up the device when the session starts. -- `stateDir` (default `.e2e/testmu`) holds each run's daemon. +- `stateDir` (default `.e2e/testmu`) holds each run's daemon, in a + directory per run that is kept after the run for its logs. A run removes + earlier runs' directories once they are more than 24 hours old. Sessions run over TestMu AI's Appium hub, so agent-device's device settings, system alerts, recording, device logs, and port reverse are not available diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index 487b5d1ee..1ce19a097 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -5,7 +5,8 @@ * holding them runs each session over TestMu AI's Appium hub. */ -import { isAbsolute, resolve } from 'node:path'; +import { readdir, rm, stat } from 'node:fs/promises'; +import { isAbsolute, join, resolve } from 'node:path'; import type { DeviceLease, DeviceProvider, DeviceRequest } from '@e2e-dev/mobile'; import { createAgentDeviceClient } from 'agent-device'; import { ConfigurationError, rejectUnknownKeys, type ProviderRecordContext, type ProviderRecording } from 'e2e/engine'; @@ -18,6 +19,12 @@ const PROVIDER = 'testmu'; /** Where each run's daemon keeps its state, under the project root. */ const DEFAULT_STATE_DIR = '.e2e/testmu'; +/** How long an earlier run's directory under `stateDir` is kept: its daemon exits after 5 idle minutes, and its logs help with a failure until then. */ +const RUN_STATE_TTL_MS = 24 * 60 * 60_000; + +/** A run id, which names each run's directory under `stateDir`; nothing else there is pruned. */ +const RUN_DIR_NAME = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + /** The dashboard project sessions are grouped under when `project` is absent. */ const DEFAULT_PROJECT = 'e2e'; @@ -88,7 +95,12 @@ export interface TestmuOptions { * Defaults to `e2e---`. Slots count from 1. */ readonly sessionName?: string | undefined; - /** Directory for the agent-device daemon each run starts, relative to the project root. Defaults to `.e2e/testmu`. */ + /** + * Directory for the agent-device daemon each run starts, relative to the + * project root. Defaults to `.e2e/testmu`. Each run keeps its own + * directory in it, and the first lease of a run removes earlier runs' + * directories last modified more than a day before. + */ readonly stateDir?: string | undefined; /** Orientation the device starts in; absent, the device's default. */ readonly orientation?: 'portrait' | 'landscape' | undefined; @@ -161,6 +173,7 @@ export function testmu(options: TestmuOptions): DeviceProvider { const releases = new Map>(); /** Stops each held lease's heartbeat, by lease id. */ const heartbeats = new Map void>(); + let pruning: Promise | undefined; const release = (id: string, handle: LeaseHandle): Promise => { heartbeats.get(id)?.(); heartbeats.delete(id); @@ -178,8 +191,11 @@ export function testmu(options: TestmuOptions): DeviceProvider { async acquire(request: DeviceRequest): Promise { if (request.appPath !== undefined) throw new Error("TestMu AI installs the app from `app`; leave the target's `app.appPath` out"); shareWithDaemon(testmuCredentials(request.env)); + const baseDir = resolve(request.projectRoot, options.stateDir ?? DEFAULT_STATE_DIR); + const stateDir = join(baseDir, request.runId); + pruning ??= pruneEarlierRuns(baseDir, request.runId); + await pruning; if (request.signal.aborted) throw new Error('cancelled before a lease was allocated'); - const stateDir = resolve(request.projectRoot, options.stateDir ?? DEFAULT_STATE_DIR, request.runId); const leaseBackend: LeaseBackend = request.platform === 'ios' ? 'ios-instance' : 'android-instance'; const selectors = { platform: request.platform, @@ -237,6 +253,33 @@ export function testmu(options: TestmuOptions): DeviceProvider { }; } +/** + * Removes the directories earlier runs left under `baseDir` once they are a + * day old, never the current run's. Best effort: a failure leaves them. + */ +async function pruneEarlierRuns(baseDir: string, runId: string): Promise { + const cutoff = Date.now() - RUN_STATE_TTL_MS; + let entries: string[]; + try { + entries = await readdir(baseDir); + } catch { + return; + } + await Promise.all( + entries + .filter((name) => name !== runId && RUN_DIR_NAME.test(name)) + .map(async (name) => { + const dir = join(baseDir, name); + try { + const info = await stat(dir); + if (info.isDirectory() && info.mtimeMs < cutoff) await rm(dir, { recursive: true, force: true }); + } catch { + // Gone already, or not ours to remove. + } + }), + ); +} + /** Releases a lease through the daemon that granted it, which ends its TestMu AI session. A lease the daemon no longer knows counts as released. */ async function releaseLease({ stateDir, scope }: LeaseHandle): Promise { await createAgentDeviceClient({ stateDir, session: 'release' }).leases.release(scope); diff --git a/packages/testmu/tests/unit/prune.test.ts b/packages/testmu/tests/unit/prune.test.ts new file mode 100644 index 000000000..b31222373 --- /dev/null +++ b/packages/testmu/tests/unit/prune.test.ts @@ -0,0 +1,95 @@ +/** + * `testmu()` pruning the state directories earlier runs left under + * `stateDir`: once per provider, only run directories older than a day, + * never the current run's, and never failing the lease. + */ + +import { existsSync, mkdirSync, mkdtempSync, readdirSync, rmSync, utimesSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import type { DeviceRequest } from '@e2e-dev/mobile'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { testmu, type TestmuOptions } from '../../src/index.ts'; + +vi.mock('agent-device', () => ({ + createAgentDeviceClient: () => ({ + leases: { + allocate: async (options: Record) => ({ leaseId: 'lease-1', tenantId: options['tenant'], runId: options['runId'] }), + heartbeat: async () => ({}), + release: async () => ({ released: true }), + }, + }), +})); + +const options: TestmuOptions = { device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: 'lt://APP1', stateDir: 'state' }; +const CURRENT_RUN = '01a0fc76-002f-736c-991a-fa4778d0543e'; +const OLD_RUN = '01a0e000-0000-7000-8000-000000000001'; +const RECENT_RUN = '01a0e000-0000-7000-8000-000000000002'; +/** A file named like a run: only directories are pruned. */ +const RUN_FILE = '01a0e000-0000-7000-8000-000000000003'; +const HOUR = 60 * 60_000; + +let root: string; +let base: string; + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'testmu-prune-')); + base = join(root, 'state'); + mkdirSync(base); +}); + +afterEach(() => { + rmSync(root, { recursive: true, force: true }); +}); + +/** A directory under the state directory, last modified `ageMs` ago, with a daemon log in it. */ +function runDir(name: string, ageMs: number): void { + const dir = join(base, name); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, 'daemon.log'), 'log'); + const time = new Date(Date.now() - ageMs); + utimesSync(dir, time, time); +} + +function request(overrides: Partial = {}): DeviceRequest { + return { + platform: 'android', + runId: CURRENT_RUN, + targetName: 'android', + slot: 0, + slots: 1, + projectRoot: root, + agentDeviceVersion: '0.21.18', + env: { LT_USERNAME: 'ada', LT_ACCESS_KEY: 'lt-key' }, + signal: new AbortController().signal, + log: () => undefined, + ...overrides, + }; +} + +describe('testmu() state directory pruning', () => { + it("removes earlier runs' directories older than a day, and keeps the current run's, recent ones, and anything not named like a run", async () => { + runDir(OLD_RUN, 25 * HOUR); + runDir(RECENT_RUN, 23 * HOUR); + runDir(CURRENT_RUN, 48 * HOUR); + runDir('notes', 48 * HOUR); + writeFileSync(join(base, RUN_FILE), 'a file'); + utimesSync(join(base, RUN_FILE), new Date(Date.now() - 48 * HOUR), new Date(Date.now() - 48 * HOUR)); + await testmu(options).acquire(request()); + expect(readdirSync(base).toSorted()).toEqual([CURRENT_RUN, RECENT_RUN, RUN_FILE, 'notes'].toSorted()); + }); + + it('prunes once per provider, on its first acquire', async () => { + const provider = testmu(options); + await provider.acquire(request()); + runDir(OLD_RUN, 25 * HOUR); + await provider.acquire(request({ slot: 1, slots: 2 })); + expect(existsSync(join(base, OLD_RUN))).toBe(true); + }); + + it('leases a device when there is nothing to prune or pruning fails', async () => { + await expect(testmu({ ...options, stateDir: 'missing' }).acquire(request())).resolves.toMatchObject({ id: 'lease-1' }); + writeFileSync(join(root, 'plain-file'), 'not a directory'); + await expect(testmu({ ...options, stateDir: 'plain-file' }).acquire(request())).resolves.toMatchObject({ id: 'lease-1' }); + }); +}); From d4403338df68b8f96cb8d2f7ebdb3b715ddb8e02 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:12:56 +0530 Subject: [PATCH 10/22] docs(testmu): explain app handling and the daemon's credentials State that `app` is required and that `app.appPath` is refused on purpose, since the provider hands TestMu AI the build itself, while `device.installApp()` with a path still works during a test. State that the provider copies `LT_USERNAME` and `LT_ACCESS_KEY` into the runner's `process.env`, where the agent-device daemon it starts reads them. Co-Authored-By: Claude Opus 5.5 --- docs/integrations/testmu.mdx | 15 ++++++++++----- packages/testmu/README.md | 11 +++++++---- 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 317996e91..ae22df604 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -115,9 +115,13 @@ lists them: a virtual iOS device takes a version like `18.0`, a real iOS device one like `18`, and Android one like `14`. TestMu AI checks them when the session starts and refuses a name or version it does not list. -Keep `app.bundleId`: TestMu AI installs the app when the session starts, -and `app.open()` launches the installed bundle id or package, not the upload -name. Leave `app.appPath` out; a target that names one fails its lease. +`app` is required: the provider hands the build to TestMu AI, which +uploads a local one and installs it when the session starts. Leave the +target's `app.appPath` out; the provider refuses it on purpose, so a target +that names one fails its lease instead of installing the build twice. +`device.installApp()` with a path still installs a build during a test. +Keep `app.bundleId`: `app.open()` launches the installed bundle id or +package, not the upload name. ## Authenticate @@ -128,8 +132,9 @@ either, the lease fails before anything starts, naming the missing one. agent-device's `testmu` runtime reads the credentials from the environment of the daemon that drives the sessions. The provider starts that daemon from the runner process, and agent-device starts a daemon with the process's own -environment, so the provider copies both values from the run's environment -into it first. +environment, so the provider copies `LT_USERNAME` and `LT_ACCESS_KEY` from +the run's environment into the runner's `process.env` first, where the +daemon it starts can read them. ## Install the app diff --git a/packages/testmu/README.md b/packages/testmu/README.md index 3d5129d16..2961aa5eb 100644 --- a/packages/testmu/README.md +++ b/packages/testmu/README.md @@ -40,7 +40,8 @@ export default { ``` It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's -environment. Each worker slot leases one device from an agent-device daemon +environment, and copies both into the runner's `process.env`, where the +agent-device daemon it starts reads them. Each worker slot leases one device from an agent-device daemon the provider starts for the run. Allocating the lease starts the TestMu AI session, which installs `app`, so every slot is billed from the moment it is leased, even when no test runs on it. The lease is released when the run @@ -48,9 +49,11 @@ ends, which ends the session. - `device` and `osVersion` must match TestMu AI's catalog exactly: `18.0` for a virtual iOS device, `18` for a real one, `14` on Android. -- `app` is an `lt://` app id, an `https` URL, or a local path resolved - against the project root. Keep the target's `app.bundleId` and leave - `app.appPath` out. +- `app` is required: an `lt://` app id, an `https` URL, or a local path + resolved against the project root, which TestMu AI installs when the + session starts. Keep the target's `app.bundleId`. The provider refuses + `app.appPath` on purpose, since it hands TestMu AI the build itself; + `device.installApp()` with a path still works during a test. - `deviceType: 'real'` picks a real device (default `'virtual'`). - `project` (default `e2e`), `build` (default the run id), and `sessionName` (default `e2e---`) label the sessions on the From 8125ebd5ee83b315babaaaa062d3f310a321e369 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:16:53 +0530 Subject: [PATCH 11/22] fix(testmu): start the attempt's video at the session's start time A provider recording's `startedAt` is the time of the video's first frame, and TestMu AI's video covers the whole session, so the time `record` was called put every timeline offset off by however long the session had run. `record` now finds the session in TestMu AI's sessions list by build and name, as `stop` did, and takes the row's `start_timestamp` (RFC 3339; a time without a zone is read as UTC) as `startedAt`. `stop` fetches the `video_url` of the session `record` found. Without a usable start time, `startedAt` falls back to when `record` was called. Co-Authored-By: Claude Opus 5.5 --- .changeset/testmu-devices.md | 2 +- docs/integrations/testmu.mdx | 16 +++-- packages/testmu/README.md | 3 +- packages/testmu/src/provider.ts | 14 +++-- packages/testmu/src/sessions.ts | 66 ++++++++++++++------ packages/testmu/tests/unit/recording.test.ts | 59 ++++++++++------- 6 files changed, 103 insertions(+), 57 deletions(-) diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md index 2a895a657..fbe999cbe 100644 --- a/.changeset/testmu-devices.md +++ b/.changeset/testmu-devices.md @@ -2,4 +2,4 @@ "@e2e-dev/testmu": minor --- -`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`), in a directory per run that later runs remove once it is more than 24 hours old; allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session; the attempt's start time is recorded), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`), in a directory per run that later runs remove once it is more than 24 hours old; allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session, starting at the session's start time), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index ae22df604..9c472bf7a 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -247,12 +247,16 @@ export default { ``` TestMu AI records the whole session, from the lease to the end of the run, -so the link is the same video for every attempt on that worker slot. The -report records when the attempt started recording, to find the attempt in -it. The runner never downloads the video; the link is the URL TestMu AI's -sessions API returns for the session, and the provider finds the session by -its build and name when the attempt stops recording. It reads the API at -`TESTMU_API_ENDPOINT` when that is set, as agent-device does. +so the link is the same video for every attempt on that worker slot, and +the video starts when the session did. When the attempt starts recording, +the provider finds the session through TestMu AI's sessions API by its +build and name and takes the session's start time as the video's, so the +report's timeline offsets land on the attempt within it; if TestMu AI +reports no start time, the time the attempt started recording is used +instead. When the attempt stops recording, the link is the video URL the +API returns for the session. The runner never downloads the video. The +provider reads the API at `TESTMU_API_ENDPOINT` when that is set, as +agent-device does. ## What the provider does diff --git a/packages/testmu/README.md b/packages/testmu/README.md index 2961aa5eb..100181aa9 100644 --- a/packages/testmu/README.md +++ b/packages/testmu/README.md @@ -69,7 +69,8 @@ ends, which ends the session. Sessions run over TestMu AI's Appium hub, so agent-device's device settings, system alerts, recording, device logs, and port reverse are not available there. An attempt that records video links TestMu AI's recording of the -whole session instead, found by the session's build and name. +whole session instead, found by the session's build and name and starting +at the session's start time. Full documentation lives at [e2e.tester.army/docs/integrations/testmu](https://e2e.tester.army/docs/integrations/testmu). diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index 1ce19a097..6c5988d81 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -11,7 +11,7 @@ import type { DeviceLease, DeviceProvider, DeviceRequest } from '@e2e-dev/mobile import { createAgentDeviceClient } from 'agent-device'; import { ConfigurationError, rejectUnknownKeys, type ProviderRecordContext, type ProviderRecording } from 'e2e/engine'; import { shareWithDaemon, testmuCredentials } from './credentials.ts'; -import { sessionVideoUrl, testmuApiEndpoint, type SessionRef } from './sessions.ts'; +import { findSession, sessionVideoUrl, testmuApiEndpoint, type SessionRef } from './sessions.ts'; /** agent-device's name for TestMu AI, as the lease provider and the tenant. */ const PROVIDER = 'testmu'; @@ -141,7 +141,8 @@ interface LeaseHandle { * session, which installs `app`, so every slot is billed from the moment it * is leased, whether or not a test runs on it, and releasing the lease ends it. * An attempt that records video links TestMu AI's recording of the whole - * session, found by the lease's build and session name. It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's + * session, found by the lease's build and session name, and starting when + * the session did. It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's * environment, and needs an agent-device with the `testmu` provider. */ export function testmu(options: TestmuOptions): DeviceProvider { @@ -239,15 +240,18 @@ export function testmu(options: TestmuOptions): DeviceProvider { if (handle === undefined) throw new Error(`lease ${lease.id} carries no agent-device lease scope to release`); await release(lease.id, handle); }, - // Runs in the worker, from the lease alone. TestMu AI records the whole session, not the attempt. + // Runs in the worker, from the lease alone. TestMu AI records the whole session, so its video starts when the session did. async record(lease: DeviceLease, context: ProviderRecordContext): Promise { + const calledAt = new Date().toISOString(); const session = sessionRef(lease); if (session === undefined) throw new Error(`lease ${lease.id} carries no TestMu AI build and session name to find its recording by`); const credentials = testmuCredentials(context.env); const endpoint = testmuApiEndpoint(context.env); + const found = await findSession(endpoint, credentials, session, context.signal); return { - startedAt: new Date().toISOString(), - stop: async ({ signal }) => ({ url: await sessionVideoUrl(endpoint, credentials, session, signal), mediaType: 'video/mp4' }), + // Without a usable start time from TestMu AI, the moment recording was asked for is the closest known bound. + startedAt: found.startedAt ?? calledAt, + stop: async ({ signal }) => ({ url: await sessionVideoUrl(endpoint, credentials, found.id, session, signal), mediaType: 'video/mp4' }), }; }, }; diff --git a/packages/testmu/src/sessions.ts b/packages/testmu/src/sessions.ts index 6ce52921f..9f815a943 100644 --- a/packages/testmu/src/sessions.ts +++ b/packages/testmu/src/sessions.ts @@ -1,7 +1,8 @@ /** * TestMu AI's mobile automation sessions API over `fetch`, the one * agent-device's `testmu` provider reads session artifacts from: finds a - * session by its build and name, and reads the URL of its video. + * session by its build and name, with its start time, and reads the URL of + * its video. */ import type { TestmuCredentials } from './credentials.ts'; @@ -31,25 +32,20 @@ export function testmuApiEndpoint(env: Readonly { - const auth = `Basic ${Buffer.from(`${credentials.username}:${credentials.accessKey}`).toString('base64')}`; - const id = await findSession(endpoint, auth, { build, sessionName }, signal); - const what = `TestMu AI session ${id} (${JSON.stringify(sessionName)}, build ${JSON.stringify(build)})`; - const body = await getJson(new URL(`${endpoint}/sessions/${encodeURIComponent(id)}`), auth, signal, `${what} details`); - const details = asRecord(body['data']); - const videoUrl = details?.['video_url']; - if (typeof videoUrl !== 'string' || !isHttpUrl(videoUrl)) throw new Error(`${what} reports no video URL`); - return videoUrl; +/** A session the list found: its `test_id`, and when it started, if the list says. */ +export interface FoundSession { + readonly id: string; + /** ISO timestamp in UTC; `undefined` when the row has no start time or one that does not parse. */ + readonly startedAt: string | undefined; } -/** The `test_id` of the newest session named `sessionName` in `build`, reading the list a page at a time. */ -async function findSession(endpoint: string, auth: string, { build, sessionName }: SessionRef, signal: AbortSignal): Promise { +/** + * The newest session named `sessionName` in `build`, reading the list a page + * at a time. Every request is bounded by `REQUEST_TIMEOUT_MS` and `signal`, + * and errors name the build and the session, never the credentials or a URL. + */ +export async function findSession(endpoint: string, credentials: TestmuCredentials, { build, sessionName }: SessionRef, signal: AbortSignal): Promise { + const auth = basicAuth(credentials); const what = `TestMu AI session lookup for build ${JSON.stringify(build)}`; for (let page = 0; page < MAX_PAGES; page += 1) { const url = new URL(`${endpoint}/sessions`); @@ -61,14 +57,44 @@ async function findSession(endpoint: string, auth: string, { build, sessionName const rows = body['data'] ?? []; if (!Array.isArray(rows)) throw new Error(`${what} failed: no session list in the response`); for (const row of rows) { - const { name, test_id: id } = asRecord(row) ?? {}; - if (name === sessionName && typeof id === 'string' && id !== '') return id; + const { name, test_id: id, start_timestamp: start } = asRecord(row) ?? {}; + if (name === sessionName && typeof id === 'string' && id !== '') return { id, startedAt: utcTimestamp(start) }; } if (rows.length < PAGE_SIZE) break; } throw new Error(`TestMu AI has no session named ${JSON.stringify(sessionName)} in build ${JSON.stringify(build)}`); } +/** The `video_url` of session `id`'s details, bounded and named as `findSession` is. */ +export async function sessionVideoUrl(endpoint: string, credentials: TestmuCredentials, id: string, { build, sessionName }: SessionRef, signal: AbortSignal): Promise { + const what = `TestMu AI session ${id} (${JSON.stringify(sessionName)}, build ${JSON.stringify(build)})`; + const body = await getJson(new URL(`${endpoint}/sessions/${encodeURIComponent(id)}`), basicAuth(credentials), signal, `${what} details`); + const details = asRecord(body['data']); + const videoUrl = details?.['video_url']; + if (typeof videoUrl !== 'string' || !isHttpUrl(videoUrl)) throw new Error(`${what} reports no video URL`); + return videoUrl; +} + +const TIMESTAMP = /^(\d{4}-\d{2}-\d{2})[T ](\d{2}:\d{2}:\d{2}(?:\.\d+)?)(Z|[+-]\d{2}:\d{2})?$/; + +/** + * A `start_timestamp` as an ISO timestamp in UTC. The API formats the + * database time as RFC 3339 with its zone; a time without one is read as UTC, + * the zone the API's database driver reads it in. + */ +function utcTimestamp(value: unknown): string | undefined { + if (typeof value !== 'string') return undefined; + const match = TIMESTAMP.exec(value.trim()); + if (match === null) return undefined; + const [, date, time, zone] = match; + const ms = Date.parse(`${date}T${time}${zone ?? 'Z'}`); + return Number.isNaN(ms) ? undefined : new Date(ms).toISOString(); +} + +function basicAuth({ username, accessKey }: TestmuCredentials): string { + return `Basic ${Buffer.from(`${username}:${accessKey}`).toString('base64')}`; +} + /** One authenticated GET answered with a JSON object; anything else throws, as `what`, with the HTTP status and the API's message. */ async function getJson(url: URL, auth: string, signal: AbortSignal, what: string): Promise> { const timeout = new AbortController(); diff --git a/packages/testmu/tests/unit/recording.test.ts b/packages/testmu/tests/unit/recording.test.ts index b918a5f99..916f5a8f9 100644 --- a/packages/testmu/tests/unit/recording.test.ts +++ b/packages/testmu/tests/unit/recording.test.ts @@ -1,8 +1,8 @@ /** * `testmu().record()` against a fake TestMu AI sessions API behind a stubbed - * `fetch`: the session it finds by build and name, the video it links, the - * endpoint override, and the failures it names without leaking credentials - * or signed URLs. + * `fetch`: the session it finds by build and name, the start time it takes + * from it, the video it links, the endpoint override, and the failures it + * names without leaking credentials or signed URLs. */ import type { DeviceLease } from '@e2e-dev/mobile'; @@ -44,9 +44,9 @@ beforeEach(() => { Object.assign(api, { calls: [], sessions: [ - { test_id: 'T3', name: 'e2e-run-1-android-1', build_name: 'run-1' }, - { test_id: 'T2', name: 'e2e-run-1-android-2', build_name: 'run-1' }, - { test_id: 'T1', name: 'e2e-run-1-android-2', build_name: 'run-1' }, + { test_id: 'T3', name: 'e2e-run-1-android-1', build_name: 'run-1', start_timestamp: '2026-10-02T09:58:05Z' }, + { test_id: 'T2', name: 'e2e-run-1-android-2', build_name: 'run-1', start_timestamp: '2026-10-02T09:58:07Z' }, + { test_id: 'T1', name: 'e2e-run-1-android-2', build_name: 'run-1', start_timestamp: '2026-10-01T08:00:00Z' }, ], details: { T2: { test_id: 'T2', name: 'e2e-run-1-android-2', video_url: VIDEO }, T1: { test_id: 'T1', video_url: 'https://videos.example.com/old.mp4' } }, override: undefined, @@ -104,21 +104,31 @@ async function record(recordLease: DeviceLease = lease, recordContext: ProviderR } describe('testmu().record()', () => { - it('starts at the moment it is called and asks TestMu AI nothing until it stops', async () => { - vi.useFakeTimers({ toFake: ['Date'] }); - vi.setSystemTime(new Date('2026-10-02T10:00:00.000Z')); - const recording = await record(); - expect(recording.startedAt).toBe('2026-10-02T10:00:00.000Z'); - expect(api.calls).toEqual([]); - }); - - it("links the video of the newest session with the slot's name in the lease's build", async () => { + it("starts at the newest session's start time, found by the slot's name in the lease's build, and links its video when it stops", async () => { const recording = await record(); + expect(recording.startedAt).toBe('2026-10-02T09:58:07.000Z'); + expect(api.calls.map((call) => call.url.href)).toEqual([`${API}/sessions?build=run-1&limit=50&offset=0`]); await expect(recording.stop(stopContext())).resolves.toEqual({ url: VIDEO, mediaType: 'video/mp4' }); expect(api.calls.map((call) => call.url.href)).toEqual([`${API}/sessions?build=run-1&limit=50&offset=0`, `${API}/sessions/T2`]); expect(api.calls.map((call) => call.authorization)).toEqual([`Basic ${Buffer.from('ada:lt-key').toString('base64')}`, `Basic ${Buffer.from('ada:lt-key').toString('base64')}`]); }); + it.each([ + ['2026-10-02T09:58:07.25+05:30', '2026-10-02T04:28:07.250Z'], + ['2026-10-02 09:58:07', '2026-10-02T09:58:07.000Z'], + ])('reads the start time %s as %s, a time without a zone as UTC', async (start, iso) => { + api.sessions[1]!['start_timestamp'] = start; + expect((await record()).startedAt).toBe(iso); + }); + + it.each([undefined, null, '', 'yesterday', '2026-13-45T99:00:00Z'])('falls back to the moment it is called without a usable start time (%j)', async (start) => { + vi.useFakeTimers({ toFake: ['Date'] }); + vi.setSystemTime(new Date('2026-10-02T10:00:00.000Z')); + if (start === undefined) delete api.sessions[1]!['start_timestamp']; + else api.sessions[1]!['start_timestamp'] = start; + expect((await record()).startedAt).toBe('2026-10-02T10:00:00.000Z'); + }); + it('reads the lease after a round trip through JSON, as the worker gets it', async () => { const recording = await record(JSON.parse(JSON.stringify(lease)) as DeviceLease); await expect(recording.stop(stopContext())).resolves.toMatchObject({ url: VIDEO }); @@ -139,9 +149,9 @@ describe('testmu().record()', () => { ]); }); - it('names the build and the session when the build has no session by that name', async () => { + it('fails to start, naming the build and the session, when the build has no session by that name', async () => { api.sessions = api.sessions.filter((row) => row['name'] !== 'e2e-run-1-android-2'); - await expect((await record()).stop(stopContext())).rejects.toThrow('TestMu AI has no session named "e2e-run-1-android-2" in build "run-1"'); + await expect(record()).rejects.toThrow('TestMu AI has no session named "e2e-run-1-android-2" in build "run-1"'); }); it('names the session when its details carry no video URL', async () => { @@ -153,7 +163,7 @@ describe('testmu().record()', () => { it('reports an HTTP failure with its status and message, never the credentials or the request URL', async () => { api.override = () => Response.json({ status: 'fail', message: 'Unauthorized' }, { status: 401 }); - const error = await failure((await record()).stop(stopContext())); + const error = await failure(record()); expect(error.message).toBe('TestMu AI session lookup for build "run-1" failed: HTTP 401 (Unauthorized)'); expect(JSON.stringify(error, Object.getOwnPropertyNames(error))).not.toContain('lt-key'); }); @@ -165,9 +175,9 @@ describe('testmu().record()', () => { it('reports a response that is not JSON, or not a session list', async () => { api.override = () => new Response('Bad gateway', { status: 502 }); - await expect((await record()).stop(stopContext())).rejects.toThrow('TestMu AI session lookup for build "run-1" failed: HTTP 502, not JSON'); + await expect(record()).rejects.toThrow('TestMu AI session lookup for build "run-1" failed: HTTP 502, not JSON'); api.override = () => Response.json({ status: 'success', data: { sessions: [] } }); - await expect((await record()).stop(stopContext())).rejects.toThrow('TestMu AI session lookup for build "run-1" failed: no session list in the response'); + await expect(record()).rejects.toThrow('TestMu AI session lookup for build "run-1" failed: no session list in the response'); }); it('gives up on a request TestMu AI does not answer within 15 seconds', async () => { @@ -176,15 +186,16 @@ describe('testmu().record()', () => { api.calls.push({ url: new URL(input), authorization: undefined }); return new Promise((_, reject) => init.signal?.addEventListener('abort', () => reject(init.signal?.reason))); }); - const stopped = failure((await record()).stop(stopContext())); + const started = failure(record()); await vi.advanceTimersByTimeAsync(15_000); - expect((await stopped).message).toBe('TestMu AI session lookup for build "run-1" got no answer within 15 s'); + expect((await started).message).toBe('TestMu AI session lookup for build "run-1" got no answer within 15 s'); }); - it('stops asking once the stop is cancelled', async () => { + it('stops asking once the attempt or the stop is cancelled', async () => { const controller = new AbortController(); controller.abort(); - await expect((await record()).stop(stopContext(controller.signal))).rejects.toThrow('TestMu AI session lookup for build "run-1" cancelled'); + await expect(record(lease, context({ signal: controller.signal }))).rejects.toThrow('TestMu AI session lookup for build "run-1" cancelled'); + await expect((await record()).stop(stopContext(controller.signal))).rejects.toThrow('TestMu AI session T2 ("e2e-run-1-android-2", build "run-1") details cancelled'); }); it('refuses to start for a lease without a build and a session name', async () => { From 74272761a37a8abd331f67093aa81556a3d7ba97 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:25:29 +0530 Subject: [PATCH 12/22] docs: run the TestMu AI example on one app, and fix a list sentence The TestMu AI example config ran its test, which checks the Proverbial sample app, on an iOS simulator target configured for a different app. Every target now runs Proverbial: the iOS target is a real iPhone with TestMu AI's public Proverbial `.ipa`, and the test matches the geolocation button's label on both platforms. An iOS simulator target is shown separately in the build section. The hosted-devices paragraph in the mobile guide names the TestMu AI provider with the same structure as the EAS one. Co-Authored-By: Claude Opus 5.5 --- docs/examples/mobile/testmu.config.ts | 7 ++++--- docs/examples/mobile/testmu.e2e.ts | 3 ++- docs/integrations/testmu.mdx | 19 +++++++++++++++---- docs/mobile.mdx | 5 +++-- 4 files changed, 24 insertions(+), 10 deletions(-) diff --git a/docs/examples/mobile/testmu.config.ts b/docs/examples/mobile/testmu.config.ts index eacf33ccf..cad6e264f 100644 --- a/docs/examples/mobile/testmu.config.ts +++ b/docs/examples/mobile/testmu.config.ts @@ -3,6 +3,7 @@ import { mobile } from '@e2e-dev/mobile'; import { testmu } from '@e2e-dev/testmu'; const apk = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_android.apk'; +const ipa = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_ios.ipa'; export default { targets: [ @@ -15,12 +16,12 @@ export default { app: { bundleId: 'com.lambdatest.proverbial' }, }, { - name: 'ios-simulator', + name: 'ios-real', engine: mobile({ platform: 'ios', - device: testmu({ device: 'iPhone 16', osVersion: '18.0', app: './build/MyApp.zip' }), + device: testmu({ device: 'iPhone 16', osVersion: '18', app: ipa, deviceType: 'real' }), }), - app: { bundleId: 'com.example.app' }, + app: { bundleId: 'proverbial' }, }, { name: 'android-real', diff --git a/docs/examples/mobile/testmu.e2e.ts b/docs/examples/mobile/testmu.e2e.ts index 5602626dc..976edf5ca 100644 --- a/docs/examples/mobile/testmu.e2e.ts +++ b/docs/examples/mobile/testmu.e2e.ts @@ -4,5 +4,6 @@ import { expect } from 'e2e'; test('Proverbial home screen', async ({ app, screen }) => { await app.open(); await expect(screen.getByText('Proverbial')).toBeVisible(); - await expect(screen.getByRole('button', 'GEOLOCATION')).toBeVisible(); + // Android labels the button GEOLOCATION, iOS GeoLocation. + await expect(screen.getByRole('button', /^geolocation$/i)).toBeVisible(); }); diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 9c472bf7a..4db764417 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -77,6 +77,7 @@ import { mobile } from '@e2e-dev/mobile'; import { testmu } from '@e2e-dev/testmu'; const apk = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_android.apk'; +const ipa = 'https://prod-mobile-artefacts.lambdatest.com/assets/docs/proverbial_ios.ipa'; export default { targets: [ @@ -89,12 +90,12 @@ export default { app: { bundleId: 'com.lambdatest.proverbial' }, }, { - name: 'ios-simulator', + name: 'ios-real', engine: mobile({ platform: 'ios', - device: testmu({ device: 'iPhone 16', osVersion: '18.0', app: './build/MyApp.zip' }), + device: testmu({ device: 'iPhone 16', osVersion: '18', app: ipa, deviceType: 'real' }), }), - app: { bundleId: 'com.example.app' }, + app: { bundleId: 'proverbial' }, }, { name: 'android-real', @@ -109,6 +110,9 @@ export default { } satisfies E2EConfig; ``` +Every target runs TestMu AI's Proverbial sample app, from its public +Android and iOS builds, so the test below passes on each one. + `device` and `osVersion` must match TestMu AI's catalog exactly, as the [capabilities generator](https://www.lambdatest.com/capabilities-generator) lists them: a virtual iOS device takes a version like `18.0`, a real iOS @@ -154,6 +158,12 @@ The build must suit the device: | iOS simulator | a zipped `.app` bundle, not an `.ipa` | | Real iOS device | a signed `.ipa` | +An iOS simulator target names a zipped simulator build of the app: + +```ts +device: testmu({ device: 'iPhone 16', osVersion: '18.0', app: './build/MyApp.zip' }), +``` + ## Real devices `deviceType: 'real'` leases a real device instead of an emulator or @@ -199,7 +209,8 @@ import { expect } from 'e2e'; test('Proverbial home screen', async ({ app, screen }) => { await app.open(); await expect(screen.getByText('Proverbial')).toBeVisible(); - await expect(screen.getByRole('button', 'GEOLOCATION')).toBeVisible(); + // Android labels the button GEOLOCATION, iOS GeoLocation. + await expect(screen.getByRole('button', /^geolocation$/i)).toBeVisible(); }); ``` diff --git a/docs/mobile.mdx b/docs/mobile.mdx index e88061289..bab031062 100644 --- a/docs/mobile.mdx +++ b/docs/mobile.mdx @@ -187,8 +187,9 @@ names instead of the local one, so any service that runs an agent-device daemon next to a simulator, an emulator, or a physical device works. Nothing in the framework knows a vendor: for [EAS Simulators](/integrations/eas), `easSimulators()` from `@e2e-dev/eas` is the provider, for -[TestMu AI](/integrations/testmu) `testmu()` from `@e2e-dev/testmu`, and for -any other service a provider is a few dozen lines in your project. +[TestMu AI](/integrations/testmu), `testmu()` from `@e2e-dev/testmu` is the +provider, and for any other service a provider is a few dozen lines in your +project. ```ts export interface DeviceProvider { From 33e370d86ff08f385b604299843feac4b6bd2f24 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:28:00 +0530 Subject: [PATCH 13/22] fix(testmu): keep the run's credentials out of the host's environment agent-device's client starts its local daemon with `process.env` and takes no environment of its own, so the provider copied `LT_USERNAME` and `LT_ACCESS_KEY` into `process.env` and left them there, where a long-lived host would pass them to unrelated child processes. The provider now sets them only around its daemon calls (allocate, heartbeat, and release, any of which may start the daemon) and puts back the previous values once no such call is in flight. A release without credentials in its environment still goes through the daemon that is already running. Co-Authored-By: Claude Opus 5.5 --- docs/integrations/testmu.mdx | 9 ++- docs/reference/environment.mdx | 2 +- packages/testmu/README.md | 7 ++- packages/testmu/src/credentials.ts | 48 ++++++++++++--- packages/testmu/src/provider.ts | 46 +++++++------- packages/testmu/tests/unit/testmu.test.ts | 74 ++++++++++++++++++++++- 6 files changed, 147 insertions(+), 39 deletions(-) diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 4db764417..853b377d6 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -136,9 +136,12 @@ either, the lease fails before anything starts, naming the missing one. agent-device's `testmu` runtime reads the credentials from the environment of the daemon that drives the sessions. The provider starts that daemon from the runner process, and agent-device starts a daemon with the process's own -environment, so the provider copies `LT_USERNAME` and `LT_ACCESS_KEY` from -the run's environment into the runner's `process.env` first, where the -daemon it starts can read them. +environment and takes no other, so while the provider calls the daemon (to +allocate, heartbeat, or release a lease), it sets `LT_USERNAME` and +`LT_ACCESS_KEY` from the run's environment in the runner's `process.env`, +where a daemon the call starts reads them. Once no such call is running, it +puts back the values `process.env` had before, so the credentials do not +stay there for other child processes of a long-lived host. ## Install the app diff --git a/docs/reference/environment.mdx b/docs/reference/environment.mdx index 217ec2186..edfbc44b8 100644 --- a/docs/reference/environment.mdx +++ b/docs/reference/environment.mdx @@ -103,7 +103,7 @@ What is sent and how to read it is on the [Telemetry](/telemetry) page. | `KERNEL_API_KEY` | value | The API key `kernel()` from `@e2e-dev/kernel` creates browsers with; unset, the browser lease fails. See [Kernel](/integrations/kernel) | | `EXPO_TOKEN` | value | The Expo access token `easSimulators()` from `@e2e-dev/eas` starts and stops simulator sessions with; unset, it uses the eas-cli login in `.expo/state.json` under `HOME` (`USERPROFILE` on Windows), and with neither the device lease fails. See [EAS Simulators](/integrations/eas) | | `HOME`, `USERPROFILE` | value | The home `easSimulators()` finds the eas-cli login under when `EXPO_TOKEN` is unset: `USERPROFILE` on Windows, `HOME` elsewhere, the OS user's home when unset | -| `LT_USERNAME`, `LT_ACCESS_KEY` | value | The TestMu AI username and access key `testmu()` from `@e2e-dev/testmu` leases devices with; it copies them into the runner process's environment for the agent-device daemon it starts, and with either unset the device lease fails. See [TestMu AI](/integrations/testmu) | +| `LT_USERNAME`, `LT_ACCESS_KEY` | value | The TestMu AI username and access key `testmu()` from `@e2e-dev/testmu` leases devices with; it sets them in the runner process's environment only while it calls the agent-device daemon it starts, which reads them from there, and with either unset the device lease fails. See [TestMu AI](/integrations/testmu) | | `E2E_AGENT_DEVICE_POOL__` | value | Written by `@e2e-dev/mobile` in `prepare` and read by each worker to find its booted device. Not meant to be set by hand | ## What app processes inherit diff --git a/packages/testmu/README.md b/packages/testmu/README.md index 100181aa9..f2cfbf46c 100644 --- a/packages/testmu/README.md +++ b/packages/testmu/README.md @@ -40,9 +40,10 @@ export default { ``` It authenticates with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's -environment, and copies both into the runner's `process.env`, where the -agent-device daemon it starts reads them. Each worker slot leases one device from an agent-device daemon -the provider starts for the run. Allocating the lease starts the TestMu AI +environment. While it calls the agent-device daemon it starts for the run, +it sets both in the runner's `process.env`, where that daemon reads them, +and puts back the previous values afterwards. Each worker slot leases one +device from that daemon. Allocating the lease starts the TestMu AI session, which installs `app`, so every slot is billed from the moment it is leased, even when no test runs on it. The lease is released when the run ends, which ends the session. diff --git a/packages/testmu/src/credentials.ts b/packages/testmu/src/credentials.ts index 0970a7abb..4d82679dc 100644 --- a/packages/testmu/src/credentials.ts +++ b/packages/testmu/src/credentials.ts @@ -26,16 +26,48 @@ export function testmuCredentials(env: Readonly>): TestmuCredentials | undefined { + const username = envValue(env, LT_USERNAME); + const accessKey = envValue(env, LT_ACCESS_KEY); + return username === undefined || accessKey === undefined ? undefined : { username, accessKey }; +} + +/** Daemon calls in flight inside `withDaemonCredentials`, and the values they replaced. */ +let scopes = 0; +let replaced: { readonly username: string | undefined; readonly accessKey: string | undefined } | undefined; + /** - * Puts the run's credentials in this process's environment, which is where - * the daemon gets them: agent-device's client starts its local daemon with - * `process.env` and takes no environment of its own, while a run's - * environment can differ from `process.env` (a host that passes `env`). - * Every worker of the run already starts with these values. + * Runs a call to the run's daemon with the run's credentials in this + * process's environment, which is where a daemon the call starts gets them: + * agent-device's client starts its local daemon with `process.env` and takes + * no environment of its own, while a run's environment can differ from + * `process.env` (a host that passes `env`). Once no such call is in flight + * the previous values are put back, so the credentials do not linger for + * other child processes of a long-lived host. Every worker of the run + * already starts with these values. */ -export function shareWithDaemon(credentials: TestmuCredentials): void { - if (process.env[LT_USERNAME] !== credentials.username) process.env[LT_USERNAME] = credentials.username; - if (process.env[LT_ACCESS_KEY] !== credentials.accessKey) process.env[LT_ACCESS_KEY] = credentials.accessKey; +export async function withDaemonCredentials(credentials: TestmuCredentials | undefined, call: () => Promise): Promise { + if (credentials === undefined) return call(); + if (scopes === 0) replaced = { username: process.env[LT_USERNAME], accessKey: process.env[LT_ACCESS_KEY] }; + scopes += 1; + process.env[LT_USERNAME] = credentials.username; + process.env[LT_ACCESS_KEY] = credentials.accessKey; + try { + return await call(); + } finally { + scopes -= 1; + if (scopes === 0 && replaced !== undefined) { + restore(LT_USERNAME, replaced.username); + restore(LT_ACCESS_KEY, replaced.accessKey); + replaced = undefined; + } + } +} + +function restore(name: string, value: string | undefined): void { + if (value === undefined) delete process.env[name]; + else process.env[name] = value; } /** A non-empty variable from the run's environment, trimmed, or `undefined`. */ diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index 6c5988d81..b44166fc3 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -7,10 +7,10 @@ import { readdir, rm, stat } from 'node:fs/promises'; import { isAbsolute, join, resolve } from 'node:path'; -import type { DeviceLease, DeviceProvider, DeviceRequest } from '@e2e-dev/mobile'; +import type { DeviceLease, DeviceProvider, DeviceReleaseContext, DeviceRequest } from '@e2e-dev/mobile'; import { createAgentDeviceClient } from 'agent-device'; import { ConfigurationError, rejectUnknownKeys, type ProviderRecordContext, type ProviderRecording } from 'e2e/engine'; -import { shareWithDaemon, testmuCredentials } from './credentials.ts'; +import { optionalTestmuCredentials, testmuCredentials, withDaemonCredentials, type TestmuCredentials } from './credentials.ts'; import { findSession, sessionVideoUrl, testmuApiEndpoint, type SessionRef } from './sessions.ts'; /** agent-device's name for TestMu AI, as the lease provider and the tenant. */ @@ -131,6 +131,8 @@ interface LeaseScope { interface LeaseHandle { readonly stateDir: string; readonly scope: LeaseScope; + /** What a daemon the call starts authenticates with, when the caller has them. */ + readonly credentials: TestmuCredentials | undefined; } /** @@ -191,7 +193,7 @@ export function testmu(options: TestmuOptions): DeviceProvider { name: PROVIDER, async acquire(request: DeviceRequest): Promise { if (request.appPath !== undefined) throw new Error("TestMu AI installs the app from `app`; leave the target's `app.appPath` out"); - shareWithDaemon(testmuCredentials(request.env)); + const credentials = testmuCredentials(request.env); const baseDir = resolve(request.projectRoot, options.stateDir ?? DEFAULT_STATE_DIR); const stateDir = join(baseDir, request.runId); pruning ??= pruneEarlierRuns(baseDir, request.runId); @@ -211,16 +213,18 @@ export function testmu(options: TestmuOptions): DeviceProvider { ...deviceFeatures, }; // Not cancellable: the daemon may grant the lease after an interrupt, and only a lease this returns or releases is ever released. - const granted = await createAgentDeviceClient({ stateDir, session: `lease-${request.slot}` }).leases.allocate({ - tenant: PROVIDER, - runId: request.runId, - leaseBackend, - leaseProvider: PROVIDER, - ttlMs: LEASE_TTL_MS, - ...selectors, - }); + const granted = await withDaemonCredentials(credentials, () => + createAgentDeviceClient({ stateDir, session: `lease-${request.slot}` }).leases.allocate({ + tenant: PROVIDER, + runId: request.runId, + leaseBackend, + leaseProvider: PROVIDER, + ttlMs: LEASE_TTL_MS, + ...selectors, + }), + ); const scope: LeaseScope = { tenant: granted.tenantId, runId: granted.runId, leaseId: granted.leaseId, leaseBackend, leaseProvider: PROVIDER }; - heartbeats.set(scope.leaseId, keepAlive({ stateDir, scope }, request.log)); + heartbeats.set(scope.leaseId, keepAlive({ stateDir, scope, credentials }, request.log)); try { if (request.signal.aborted) throw new Error('cancelled'); request.log(`lease ${scope.leaseId}: ${device}, ${request.platform} ${osVersion} (${deviceType}); session ${selectors.providerSessionName} started`); @@ -228,15 +232,15 @@ export function testmu(options: TestmuOptions): DeviceProvider { return { id: scope.leaseId, client: { stateDir, ...scope, ...selectors } }; } catch (cause) { // The engine releases only leases `acquire` returned. - const outcome = await release(scope.leaseId, { stateDir, scope }).then( + const outcome = await release(scope.leaseId, { stateDir, scope, credentials }).then( () => 'released it', (releaseCause: unknown) => `releasing it failed (${messageOf(releaseCause)})`, ); throw new Error(`lease ${scope.leaseId} was not handed to the run: ${messageOf(cause)}; ${outcome}`, { cause }); } }, - async release(lease: DeviceLease): Promise { - const handle = leaseHandle(lease); + async release(lease: DeviceLease, context: DeviceReleaseContext): Promise { + const handle = leaseHandle(lease, optionalTestmuCredentials(context.env)); if (handle === undefined) throw new Error(`lease ${lease.id} carries no agent-device lease scope to release`); await release(lease.id, handle); }, @@ -285,8 +289,8 @@ async function pruneEarlierRuns(baseDir: string, runId: string): Promise { } /** Releases a lease through the daemon that granted it, which ends its TestMu AI session. A lease the daemon no longer knows counts as released. */ -async function releaseLease({ stateDir, scope }: LeaseHandle): Promise { - await createAgentDeviceClient({ stateDir, session: 'release' }).leases.release(scope); +async function releaseLease({ stateDir, scope, credentials }: LeaseHandle): Promise { + await withDaemonCredentials(credentials, () => createAgentDeviceClient({ stateDir, session: 'release' }).leases.release(scope)); } /** @@ -296,11 +300,11 @@ async function releaseLease({ stateDir, scope }: LeaseHandle): Promise { * the run holds it. A failed heartbeat is logged once and the next one tried; * it never fails the run. */ -function keepAlive({ stateDir, scope }: LeaseHandle, log: (line: string) => void): () => void { +function keepAlive({ stateDir, scope, credentials }: LeaseHandle, log: (line: string) => void): () => void { const client = createAgentDeviceClient({ stateDir, session: 'heartbeat' }); let warned = false; const timer = setInterval(() => { - client.leases.heartbeat({ ...scope, ttlMs: LEASE_TTL_MS }).catch((cause: unknown) => { + withDaemonCredentials(credentials, () => client.leases.heartbeat({ ...scope, ttlMs: LEASE_TTL_MS })).catch((cause: unknown) => { if (warned) return; warned = true; try { @@ -330,7 +334,7 @@ function appSource(app: string, projectRoot: string): string { } /** The daemon and scope `acquire` put on a lease's `client`, when they are there. */ -function leaseHandle(lease: DeviceLease): LeaseHandle | undefined { +function leaseHandle(lease: DeviceLease, credentials: TestmuCredentials | undefined): LeaseHandle | undefined { const client = lease.client as Record | undefined; if (client === undefined) return undefined; const { stateDir, tenant, runId, leaseId, leaseBackend, leaseProvider } = client; @@ -344,7 +348,7 @@ function leaseHandle(lease: DeviceLease): LeaseHandle | undefined { ) { return undefined; } - return { stateDir, scope: { tenant, runId, leaseId, leaseBackend, leaseProvider } }; + return { stateDir, scope: { tenant, runId, leaseId, leaseBackend, leaseProvider }, credentials }; } /** The build and session name `acquire` put on a lease's `client`, when they are there. */ diff --git a/packages/testmu/tests/unit/testmu.test.ts b/packages/testmu/tests/unit/testmu.test.ts index 120cdd46a..513ae680a 100644 --- a/packages/testmu/tests/unit/testmu.test.ts +++ b/packages/testmu/tests/unit/testmu.test.ts @@ -20,6 +20,11 @@ const daemon = { calls: [] as ClientCall[], /** Runs inside `allocate`, before it answers. */ onAllocate: undefined as (() => void) | undefined, + /** What each `allocate`, in order, waits for before it runs `onAllocate`. */ + allocateWaits: [] as Promise[], + /** Runs inside `heartbeat` and `release`, before they answer. */ + onHeartbeat: undefined as (() => void) | undefined, + onRelease: undefined as (() => void) | undefined, allocateError: undefined as Error | undefined, /** Errors `release` answers with, in order, before it succeeds. */ releaseErrors: [] as Error[], @@ -32,18 +37,22 @@ vi.mock('agent-device', () => ({ leases: { allocate: async (options: Record) => { daemon.calls.push({ config, operation: 'allocate', options }); + const wait = daemon.allocateWaits.shift(); + if (wait !== undefined) await wait; daemon.onAllocate?.(); if (daemon.allocateError !== undefined) throw daemon.allocateError; return { leaseId: `lease-${daemon.calls.length}`, tenantId: options['tenant'], runId: options['runId'], backend: options['leaseBackend'], leaseProvider: options['leaseProvider'] }; }, heartbeat: async (options: Record) => { daemon.calls.push({ config, operation: 'heartbeat', options }); + daemon.onHeartbeat?.(); const error = daemon.heartbeatErrors.shift(); if (error !== undefined) throw error; return { leaseId: options['leaseId'], tenantId: options['tenant'], runId: options['runId'], backend: options['leaseBackend'] }; }, release: async (options: Record) => { daemon.calls.push({ config, operation: 'release', options }); + daemon.onRelease?.(); const error = daemon.releaseErrors.shift(); if (error !== undefined) throw error; return { released: true }; @@ -58,7 +67,16 @@ const options: TestmuOptions = { device: 'Galaxy S22 Ultra 5G', osVersion: '14', const saved = { LT_USERNAME: process.env['LT_USERNAME'], LT_ACCESS_KEY: process.env['LT_ACCESS_KEY'] }; beforeEach(() => { - Object.assign(daemon, { calls: [], onAllocate: undefined, allocateError: undefined, releaseErrors: [], heartbeatErrors: [] }); + Object.assign(daemon, { + calls: [], + onAllocate: undefined, + allocateWaits: [], + onHeartbeat: undefined, + onRelease: undefined, + allocateError: undefined, + releaseErrors: [], + heartbeatErrors: [], + }); }); afterEach(() => { @@ -92,6 +110,8 @@ const context: DeviceReleaseContext = { runId: 'run-1', targetName: 'android', e const operations = () => daemon.calls.map((call) => call.operation); +const runnerCredentials = () => [process.env['LT_USERNAME'], process.env['LT_ACCESS_KEY']]; + const MINUTE = 60_000; describe('testmu()', () => { @@ -205,12 +225,60 @@ describe('testmu()', () => { expect(req.lines).toEqual(['lease lease-1: Galaxy S22 Ultra 5G, android 14 (virtual); session e2e-run-1-android-1 started']); }); - it('shares the run\'s credentials with the daemon it starts', async () => { + it("shares the run's credentials with a daemon the allocation starts, and puts back the process's own after it", async () => { delete process.env['LT_USERNAME']; process.env['LT_ACCESS_KEY'] = 'stale'; - daemon.onAllocate = () => expect([process.env['LT_USERNAME'], process.env['LT_ACCESS_KEY']]).toEqual(['ada', 'lt-key']); + daemon.onAllocate = () => expect(runnerCredentials()).toEqual(['ada', 'lt-key']); await testmu(options).acquire(request({ env: { LT_USERNAME: ' ada ', LT_ACCESS_KEY: 'lt-key' } })); expect(operations()).toEqual(['allocate']); + expect(runnerCredentials()).toEqual([undefined, 'stale']); + }); + + it('keeps the credentials in place until every allocation running at once has finished', async () => { + delete process.env['LT_USERNAME']; + delete process.env['LT_ACCESS_KEY']; + let open!: () => void; + daemon.allocateWaits = [Promise.resolve(), new Promise((resolve) => (open = resolve))]; + const seen: unknown[] = []; + daemon.onAllocate = () => seen.push(runnerCredentials()); + const provider = testmu(options); + const first = provider.acquire(request({ slot: 0 })); + const second = provider.acquire(request({ slot: 1 })); + await first; + expect(runnerCredentials()).toEqual(['ada', 'lt-key']); + open(); + await second; + expect(seen).toEqual([ + ['ada', 'lt-key'], + ['ada', 'lt-key'], + ]); + expect(runnerCredentials()).toEqual([undefined, undefined]); + }); + + it('shares the credentials with a daemon a heartbeat or a release starts', async () => { + vi.useFakeTimers(); + process.env['LT_USERNAME'] = 'host'; + delete process.env['LT_ACCESS_KEY']; + const seen: unknown[] = []; + daemon.onHeartbeat = () => seen.push(['heartbeat', ...runnerCredentials()]); + daemon.onRelease = () => seen.push(['release', ...runnerCredentials()]); + const provider = testmu(options); + const lease = await provider.acquire(request()); + await vi.advanceTimersByTimeAsync(2 * MINUTE); + expect(runnerCredentials()).toEqual(['host', undefined]); + await provider.release(lease, { ...context, env: { LT_USERNAME: 'grace', LT_ACCESS_KEY: 'other-key' } }); + expect(seen).toEqual([ + ['heartbeat', 'ada', 'lt-key'], + ['release', 'grace', 'other-key'], + ]); + expect(runnerCredentials()).toEqual(['host', undefined]); + }); + + it('releases a lease without credentials in the release environment, through the daemon already running', async () => { + const provider = testmu(options); + const lease = await provider.acquire(request()); + await provider.release(lease, { ...context, env: {} }); + expect(operations()).toEqual(['allocate', 'release']); }); it.each([ From 4a362a95b7a034b40acefff4a87c507c4f6e2918 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:28:36 +0530 Subject: [PATCH 14/22] fix(testmu): validate every option at config load `project`, `build`, `sessionName`, and `stateDir` were passed through unchecked, so a JavaScript config with `project: 1` or `stateDir: 1` reached the allocation or failed inside `acquire`. They are now checked with the device-feature options when `testmu()` is called, and any of them that is not a non-empty string fails with INVALID_CONFIG. `deviceType` defaults only when it is absent, so `null` is refused as well. Co-Authored-By: Claude Opus 5.5 --- docs/integrations/testmu.mdx | 5 +++-- packages/testmu/src/provider.ts | 26 ++++++++++++++--------- packages/testmu/tests/unit/testmu.test.ts | 19 ++++++++++++----- 3 files changed, 33 insertions(+), 17 deletions(-) diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 853b377d6..853e6ea39 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -198,8 +198,9 @@ The device features, `orientation` through `appiumVersion`, are set when the session starts and hold for the whole session; a test cannot change them. An option not in this table, a missing or empty `device`, `osVersion`, or -`app`, another `deviceType` or `orientation`, or an empty device feature -fails the config load with `INVALID_CONFIG`. +`app`, another `deviceType` or `orientation`, or any other option given as +something other than a non-empty string fails the config load with +`INVALID_CONFIG`. ## Write a test diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index b44166fc3..32fc20c70 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -53,6 +53,9 @@ const DEVICE_FEATURES = { appiumVersion: 'providerAppiumVersion', } as const satisfies Partial>; +/** The options that are strings when given, checked at config load. */ +const OPTIONAL_STRING_KEYS = ['project', 'build', 'sessionName', 'stateDir', ...(Object.keys(DEVICE_FEATURES) as (keyof typeof DEVICE_FEATURES)[])] as const; + /** Every option `testmu()` takes, kept equal to `TestmuOptions` by the compiler. */ const OPTION_KEYS: readonly string[] = Object.keys({ device: true, @@ -155,21 +158,24 @@ export function testmu(options: TestmuOptions): DeviceProvider { throw new ConfigurationError('INVALID_CONFIG', `testmu: \`${key}\` is required, as a non-empty string`); } } - const deviceType = options.deviceType ?? 'virtual'; + // Only an absent value takes the default: `null` from a JavaScript config is refused like any other. + const deviceType = options.deviceType === undefined ? 'virtual' : options.deviceType; if (!DEVICE_TYPES.has(deviceType)) { throw new ConfigurationError('INVALID_CONFIG', `testmu: \`deviceType\` must be 'virtual' or 'real', not ${JSON.stringify(deviceType)}`); } - const deviceFeatures: Record = {}; - for (const [key, leaseKey] of Object.entries(DEVICE_FEATURES)) { - const value: unknown = options[key as keyof typeof DEVICE_FEATURES]; - if (value === undefined) continue; - if (typeof value !== 'string' || value.trim() === '') { + for (const key of OPTIONAL_STRING_KEYS) { + const value: unknown = options[key]; + if (value !== undefined && (typeof value !== 'string' || value.trim() === '')) { throw new ConfigurationError('INVALID_CONFIG', `testmu: \`${key}\` must be a non-empty string`); } - if (key === 'orientation' && !ORIENTATIONS.has(value)) { - throw new ConfigurationError('INVALID_CONFIG', `testmu: \`orientation\` must be 'portrait' or 'landscape', not ${JSON.stringify(value)}`); - } - deviceFeatures[leaseKey] = value; + } + if (options.orientation !== undefined && !ORIENTATIONS.has(options.orientation)) { + throw new ConfigurationError('INVALID_CONFIG', `testmu: \`orientation\` must be 'portrait' or 'landscape', not ${JSON.stringify(options.orientation)}`); + } + const deviceFeatures: Record = {}; + for (const [key, leaseKey] of Object.entries(DEVICE_FEATURES)) { + const value = options[key as keyof typeof DEVICE_FEATURES]; + if (value !== undefined) deviceFeatures[leaseKey] = value; } const { device, osVersion, app, project, build, sessionName } = options; /** One release per lease, shared by every caller: the engine's, and `acquire`'s own after a failure. */ diff --git a/packages/testmu/tests/unit/testmu.test.ts b/packages/testmu/tests/unit/testmu.test.ts index 513ae680a..ac600c9ac 100644 --- a/packages/testmu/tests/unit/testmu.test.ts +++ b/packages/testmu/tests/unit/testmu.test.ts @@ -450,14 +450,23 @@ describe('testmu()', () => { ); }); - it.each(['geoLocation', 'timezone', 'language', 'locale', 'appiumVersion'] as const)('refuses an empty or non-string `%s` with INVALID_CONFIG', (key) => { - expect(() => testmu({ ...options, [key]: ' ' })).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG', message: `testmu: \`${key}\` must be a non-empty string` })); + it.each(['orientation', 'geoLocation', 'timezone', 'language', 'locale', 'appiumVersion'] as const)('refuses an empty or non-string `%s` with INVALID_CONFIG', (key) => { + expect(() => testmu({ ...options, [key]: ' ' } as unknown as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG', message: `testmu: \`${key}\` must be a non-empty string` })); expect(() => testmu({ ...options, [key]: 2 } as unknown as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG' })); + expect(() => testmu({ ...options, [key]: null } as unknown as TestmuOptions)).toThrow(expect.objectContaining({ code: 'INVALID_CONFIG' })); }); - it('refuses a device type other than virtual or real', () => { - expect(() => testmu({ ...options, deviceType: 'emulator' as 'virtual' })).toThrow( - expect.objectContaining({ code: 'INVALID_CONFIG', message: 'testmu: `deviceType` must be \'virtual\' or \'real\', not "emulator"' }), + it.each(['emulator', null])('refuses a device type other than virtual or real (%j)', (deviceType) => { + expect(() => testmu({ ...options, deviceType } as unknown as TestmuOptions)).toThrow( + expect.objectContaining({ code: 'INVALID_CONFIG', message: `testmu: \`deviceType\` must be 'virtual' or 'real', not ${JSON.stringify(deviceType)}` }), ); }); + + it.each(['project', 'build', 'sessionName', 'stateDir'] as const)('refuses an empty or non-string `%s` with INVALID_CONFIG', (key) => { + for (const value of [' ', 1, null]) { + expect(() => testmu({ ...options, [key]: value } as unknown as TestmuOptions)).toThrow( + expect.objectContaining({ code: 'INVALID_CONFIG', message: `testmu: \`${key}\` must be a non-empty string` }), + ); + } + }); }); From 3a45875e81e8b92b4d7636d0a12d9c468ceb08b8 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:37:56 +0530 Subject: [PATCH 15/22] docs(testmu): explain the lease window without agent-device internals The provider's comments, the docs page, and the changeset explained the long lease window by agent-device starting a lease's window before a slow allocation returns, which stops being true once agent-device starts it when the allocation completes. They now say what holds on any agent-device version: the provider allocates with agent-device's longest lease window and heartbeats every 2 minutes, so a lease stays alive however long its session takes to start. Co-Authored-By: Claude Opus 5.5 --- .changeset/testmu-devices.md | 2 +- docs/integrations/testmu.mdx | 10 +++++----- packages/testmu/src/provider.ts | 15 +++++++-------- 3 files changed, 13 insertions(+), 14 deletions(-) diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md index fbe999cbe..4dfa22263 100644 --- a/.changeset/testmu-devices.md +++ b/.changeset/testmu-devices.md @@ -2,4 +2,4 @@ "@e2e-dev/testmu": minor --- -`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`), in a directory per run that later runs remove once it is more than 24 hours old; allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), heartbeats each lease with agent-device's longest inactivity window while the run holds it (agent-device starts that window before a slow allocation returns), links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session, starting at the session's start time), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`), in a directory per run that later runs remove once it is more than 24 hours old; allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), allocates each lease with agent-device's longest inactivity window and heartbeats it every 2 minutes while the run holds it, so a lease stays alive however long its session takes to start, links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session, starting at the session's start time), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 853e6ea39..b52ac35ae 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -284,11 +284,11 @@ agent-device does. of earlier runs under `stateDir` that were last modified more than 24 hours ago, never its own; only directories named after a run id are removed, and a failure to remove one does not fail the lease. -- Keeps each lease alive while the run holds it. agent-device starts a - lease's inactivity window before a slow allocation returns, so the - provider asks for agent-device's longest window, 10 minutes, and - heartbeats the lease every 2 minutes. A failed heartbeat is logged once - and does not fail the run. +- Keeps each lease alive while the run holds it. The provider allocates + with agent-device's longest lease window, 10 minutes, and heartbeats the + lease every 2 minutes, so a lease stays alive however long its session + takes to start, on any agent-device version. A failed heartbeat is logged + once and does not fail the run. - Hands the worker the lease scope and the device selectors as agent-device client configuration, so every command lands on the lease's session. - Releases each lease when the run ends, which ends its session. A lease diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index 32fc20c70..2bf1211ea 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -29,10 +29,10 @@ const RUN_DIR_NAME = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{ const DEFAULT_PROJECT = 'e2e'; /** - * The inactivity window each lease asks for, agent-device's longest. - * agent-device starts a lease's window before the allocation that uploads the - * app and starts the session returns, so a slow one, as on an iOS simulator, - * can spend most of the 60-second default before the lease is granted. + * The inactivity window each lease asks for, agent-device's longest. With + * the heartbeat, a lease stays alive however long its session takes to + * start, whether agent-device starts the window when the allocation begins + * or when it completes. */ const LEASE_TTL_MS = 10 * 60_000; @@ -301,10 +301,9 @@ async function releaseLease({ stateDir, scope, credentials }: LeaseHandle): Prom /** * Heartbeats a lease until the returned function is called. A command still - * running does not keep its lease alive, and the allocation may already have - * spent part of the lease's window, so without this a lease can lapse while - * the run holds it. A failed heartbeat is logged once and the next one tried; - * it never fails the run. + * running does not keep its lease alive, so without this a lease can lapse + * while the run holds it. A failed heartbeat is logged once and the next one + * tried; it never fails the run. */ function keepAlive({ stateDir, scope, credentials }: LeaseHandle, log: (line: string) => void): () => void { const client = createAgentDeviceClient({ stateDir, session: 'heartbeat' }); From 20ad5725cfae1d2bea8c5a540c2c280fd72e5f6a Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:38:18 +0530 Subject: [PATCH 16/22] fix(testmu): refuse an API endpoint that carries credentials A `TESTMU_API_ENDPOINT` with a username or password in it would reach a failed request's error message. `record` now refuses such an endpoint with INVALID_CONFIG and a message that does not repeat the value, and reports an endpoint that is not an http(s) URL as INVALID_CONFIG too. Co-Authored-By: Claude Opus 5.5 --- packages/testmu/src/sessions.ts | 14 ++++++++++++-- packages/testmu/tests/unit/recording.test.ts | 9 +++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/packages/testmu/src/sessions.ts b/packages/testmu/src/sessions.ts index 9f815a943..4cdf46c7d 100644 --- a/packages/testmu/src/sessions.ts +++ b/packages/testmu/src/sessions.ts @@ -5,6 +5,7 @@ * its video. */ +import { ConfigurationError } from 'e2e/engine'; import type { TestmuCredentials } from './credentials.ts'; /** The API's base, as agent-device's `TESTMU_API_ENDPOINT` sets it. */ @@ -24,11 +25,20 @@ export interface SessionRef { readonly sessionName: string; } -/** The sessions API at `TESTMU_API_ENDPOINT` from the run's environment, else TestMu AI's own; throws when the override is not an http(s) URL. */ +/** + * The sessions API at `TESTMU_API_ENDPOINT` from the run's environment, else + * TestMu AI's own. Throws, without repeating the value, when the override is + * not an http(s) URL or carries a username or password, which a failed + * request would otherwise put in its error message. + */ export function testmuApiEndpoint(env: Readonly>): string { const override = env['TESTMU_API_ENDPOINT']?.trim(); if (override === undefined || override === '') return DEFAULT_API_ENDPOINT; - if (!URL.canParse(override) || !/^https?:$/.test(new URL(override).protocol)) throw new Error('TESTMU_API_ENDPOINT is not an http(s) URL'); + if (!URL.canParse(override) || !/^https?:$/.test(new URL(override).protocol)) throw new ConfigurationError('INVALID_CONFIG', 'TESTMU_API_ENDPOINT is not an http(s) URL'); + const url = new URL(override); + if (url.username !== '' || url.password !== '') { + throw new ConfigurationError('INVALID_CONFIG', 'TESTMU_API_ENDPOINT must not carry a username or password; set LT_USERNAME and LT_ACCESS_KEY instead'); + } return override.replace(/\/+$/, ''); } diff --git a/packages/testmu/tests/unit/recording.test.ts b/packages/testmu/tests/unit/recording.test.ts index 916f5a8f9..40c4f1b69 100644 --- a/packages/testmu/tests/unit/recording.test.ts +++ b/packages/testmu/tests/unit/recording.test.ts @@ -209,4 +209,13 @@ describe('testmu().record()', () => { await expect(record(lease, context({ env: { ...env, TESTMU_API_ENDPOINT: 'mobile-api' } }))).rejects.toThrow('TESTMU_API_ENDPOINT is not an http(s) URL'); expect(api.calls).toEqual([]); }); + + it('refuses an endpoint carrying credentials, without repeating them', async () => { + for (const endpoint of ['https://ada:lt-key@mobile-api.lambdatest.com/mobile-automation/api/v1', 'https://lt-key@mobile-api.lambdatest.com']) { + const error = await failure(record(lease, context({ env: { ...env, TESTMU_API_ENDPOINT: endpoint } }))); + expect(error).toMatchObject({ code: 'INVALID_CONFIG', message: 'TESTMU_API_ENDPOINT must not carry a username or password; set LT_USERNAME and LT_ACCESS_KEY instead' }); + expect(JSON.stringify(error, Object.getOwnPropertyNames(error))).not.toContain('lt-key'); + } + expect(api.calls).toEqual([]); + }); }); From 6026844ddd05103236ac71fe83cbb440c08ff73f Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:39:00 +0530 Subject: [PATCH 17/22] fix(testmu): look up the recorded session among the user's own TestMu AI's session list filters by build name across the whole organization, so a build name another user's run shares could find their session. The lookup now also passes `username`, which the API matches against the session's owner, set to `LT_USERNAME`. The docs say that a fixed `build` and `sessionName` shared by overlapping runs is ambiguous, and that the default names avoid it. Co-Authored-By: Claude Opus 5.5 --- docs/integrations/testmu.mdx | 9 ++++++- packages/testmu/README.md | 3 ++- packages/testmu/src/sessions.ts | 6 +++-- packages/testmu/tests/unit/recording.test.ts | 26 +++++++++++--------- 4 files changed, 28 insertions(+), 16 deletions(-) diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index b52ac35ae..6d82cd269 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -265,7 +265,8 @@ TestMu AI records the whole session, from the lease to the end of the run, so the link is the same video for every attempt on that worker slot, and the video starts when the session did. When the attempt starts recording, the provider finds the session through TestMu AI's sessions API by its -build and name and takes the session's start time as the video's, so the +build and name, among the sessions of the `LT_USERNAME` user, taking the +newest that matches, and takes the session's start time as the video's, so the report's timeline offsets land on the attempt within it; if TestMu AI reports no start time, the time the attempt started recording is used instead. When the attempt stops recording, the link is the video URL the @@ -273,6 +274,12 @@ API returns for the session. The runner never downloads the video. The provider reads the API at `TESTMU_API_ENDPOINT` when that is set, as agent-device does. +The default `build` (the run id) and `sessionName` (with the run id in it) +give every session its own name. A fixed `build` and a fixed `sessionName` +shared by runs that overlap name several sessions the same, so an attempt +can link another run's video; give such runs their own `build`, or leave +`sessionName` to its default. + ## What the provider does - Starts one agent-device daemon per run under `stateDir/` and diff --git a/packages/testmu/README.md b/packages/testmu/README.md index f2cfbf46c..5660aa932 100644 --- a/packages/testmu/README.md +++ b/packages/testmu/README.md @@ -71,7 +71,8 @@ Sessions run over TestMu AI's Appium hub, so agent-device's device settings, system alerts, recording, device logs, and port reverse are not available there. An attempt that records video links TestMu AI's recording of the whole session instead, found by the session's build and name and starting -at the session's start time. +at the session's start time. Runs that overlap and share a fixed `build` +and `sessionName` can link each other's videos; the defaults never do. Full documentation lives at [e2e.tester.army/docs/integrations/testmu](https://e2e.tester.army/docs/integrations/testmu). diff --git a/packages/testmu/src/sessions.ts b/packages/testmu/src/sessions.ts index 4cdf46c7d..bbadd53f6 100644 --- a/packages/testmu/src/sessions.ts +++ b/packages/testmu/src/sessions.ts @@ -50,8 +50,8 @@ export interface FoundSession { } /** - * The newest session named `sessionName` in `build`, reading the list a page - * at a time. Every request is bounded by `REQUEST_TIMEOUT_MS` and `signal`, + * The newest session named `sessionName` in `build` that the credentials' + * user started, reading the list a page at a time. Every request is bounded by `REQUEST_TIMEOUT_MS` and `signal`, * and errors name the build and the session, never the credentials or a URL. */ export async function findSession(endpoint: string, credentials: TestmuCredentials, { build, sessionName }: SessionRef, signal: AbortSignal): Promise { @@ -60,6 +60,8 @@ export async function findSession(endpoint: string, credentials: TestmuCredentia for (let page = 0; page < MAX_PAGES; page += 1) { const url = new URL(`${endpoint}/sessions`); url.searchParams.set('build', build); + // The build filter spans the whole organization; a build name another user's run shares would find their session. + url.searchParams.set('username', credentials.username); url.searchParams.set('limit', String(PAGE_SIZE)); url.searchParams.set('offset', String(page * PAGE_SIZE)); const body = await getJson(url, auth, signal, what); diff --git a/packages/testmu/tests/unit/recording.test.ts b/packages/testmu/tests/unit/recording.test.ts index 40c4f1b69..fb96b1ff3 100644 --- a/packages/testmu/tests/unit/recording.test.ts +++ b/packages/testmu/tests/unit/recording.test.ts @@ -44,9 +44,10 @@ beforeEach(() => { Object.assign(api, { calls: [], sessions: [ - { test_id: 'T3', name: 'e2e-run-1-android-1', build_name: 'run-1', start_timestamp: '2026-10-02T09:58:05Z' }, - { test_id: 'T2', name: 'e2e-run-1-android-2', build_name: 'run-1', start_timestamp: '2026-10-02T09:58:07Z' }, - { test_id: 'T1', name: 'e2e-run-1-android-2', build_name: 'run-1', start_timestamp: '2026-10-01T08:00:00Z' }, + { test_id: 'T4', name: 'e2e-run-1-android-2', build_name: 'run-1', username: 'grace', start_timestamp: '2026-10-02T09:59:00Z' }, + { test_id: 'T3', name: 'e2e-run-1-android-1', build_name: 'run-1', username: 'ada', start_timestamp: '2026-10-02T09:58:05Z' }, + { test_id: 'T2', name: 'e2e-run-1-android-2', build_name: 'run-1', username: 'ada', start_timestamp: '2026-10-02T09:58:07Z' }, + { test_id: 'T1', name: 'e2e-run-1-android-2', build_name: 'run-1', username: 'ada', start_timestamp: '2026-10-01T08:00:00Z' }, ], details: { T2: { test_id: 'T2', name: 'e2e-run-1-android-2', video_url: VIDEO }, T1: { test_id: 'T1', video_url: 'https://videos.example.com/old.mp4' } }, override: undefined, @@ -63,9 +64,10 @@ beforeEach(() => { const base = new URL(API).pathname; if (url.pathname === `${base}/sessions`) { const build = url.searchParams.get('build'); + const username = url.searchParams.get('username'); const limit = Number(url.searchParams.get('limit') ?? '10'); const offset = Number(url.searchParams.get('offset') ?? '0'); - const rows = api.sessions.filter((row) => row['build_name'] === build).slice(offset, offset + limit); + const rows = api.sessions.filter((row) => row['build_name'] === build && (username === null || row['username'] === username)).slice(offset, offset + limit); // An empty page is Go's nil slice: `data: null`. return Response.json({ status: 'success', data: rows.length === 0 ? null : rows, message: 'Retrieve session list was successful', Meta: { result_set: { count: rows.length } } }); } @@ -104,12 +106,12 @@ async function record(recordLease: DeviceLease = lease, recordContext: ProviderR } describe('testmu().record()', () => { - it("starts at the newest session's start time, found by the slot's name in the lease's build, and links its video when it stops", async () => { + it("starts at the newest session's start time, found by the slot's name in the lease's build among the user's own, and links its video when it stops", async () => { const recording = await record(); expect(recording.startedAt).toBe('2026-10-02T09:58:07.000Z'); - expect(api.calls.map((call) => call.url.href)).toEqual([`${API}/sessions?build=run-1&limit=50&offset=0`]); + expect(api.calls.map((call) => call.url.href)).toEqual([`${API}/sessions?build=run-1&username=ada&limit=50&offset=0`]); await expect(recording.stop(stopContext())).resolves.toEqual({ url: VIDEO, mediaType: 'video/mp4' }); - expect(api.calls.map((call) => call.url.href)).toEqual([`${API}/sessions?build=run-1&limit=50&offset=0`, `${API}/sessions/T2`]); + expect(api.calls.map((call) => call.url.href)).toEqual([`${API}/sessions?build=run-1&username=ada&limit=50&offset=0`, `${API}/sessions/T2`]); expect(api.calls.map((call) => call.authorization)).toEqual([`Basic ${Buffer.from('ada:lt-key').toString('base64')}`, `Basic ${Buffer.from('ada:lt-key').toString('base64')}`]); }); @@ -117,15 +119,15 @@ describe('testmu().record()', () => { ['2026-10-02T09:58:07.25+05:30', '2026-10-02T04:28:07.250Z'], ['2026-10-02 09:58:07', '2026-10-02T09:58:07.000Z'], ])('reads the start time %s as %s, a time without a zone as UTC', async (start, iso) => { - api.sessions[1]!['start_timestamp'] = start; + api.sessions[2]!['start_timestamp'] = start; expect((await record()).startedAt).toBe(iso); }); it.each([undefined, null, '', 'yesterday', '2026-13-45T99:00:00Z'])('falls back to the moment it is called without a usable start time (%j)', async (start) => { vi.useFakeTimers({ toFake: ['Date'] }); vi.setSystemTime(new Date('2026-10-02T10:00:00.000Z')); - if (start === undefined) delete api.sessions[1]!['start_timestamp']; - else api.sessions[1]!['start_timestamp'] = start; + if (start === undefined) delete api.sessions[2]!['start_timestamp']; + else api.sessions[2]!['start_timestamp'] = start; expect((await record()).startedAt).toBe('2026-10-02T10:00:00.000Z'); }); @@ -135,7 +137,7 @@ describe('testmu().record()', () => { }); it('pages through a build holding more sessions than one page', async () => { - api.sessions = [...Array.from({ length: 50 }, (_, index) => ({ test_id: `X${index}`, name: `other-${index}`, build_name: 'run-1' })), ...api.sessions]; + api.sessions = [...Array.from({ length: 50 }, (_, index) => ({ test_id: `X${index}`, name: `other-${index}`, build_name: 'run-1', username: 'ada' })), ...api.sessions]; await expect((await record()).stop(stopContext())).resolves.toMatchObject({ url: VIDEO }); expect(api.calls.map((call) => call.url.searchParams.get('offset'))).toEqual(['0', '50', null]); }); @@ -144,7 +146,7 @@ describe('testmu().record()', () => { const recording = await record(lease, context({ env: { ...env, TESTMU_API_ENDPOINT: 'https://stage-mobile-api.lambdatest.com/mobile-automation/api/v1/' } })); await expect(recording.stop(stopContext())).resolves.toMatchObject({ url: VIDEO }); expect(api.calls.map((call) => call.url.href)).toEqual([ - 'https://stage-mobile-api.lambdatest.com/mobile-automation/api/v1/sessions?build=run-1&limit=50&offset=0', + 'https://stage-mobile-api.lambdatest.com/mobile-automation/api/v1/sessions?build=run-1&username=ada&limit=50&offset=0', 'https://stage-mobile-api.lambdatest.com/mobile-automation/api/v1/sessions/T2', ]); }); From dae2c2ff235edf47649820a0664be9be0d9c6136 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:39:59 +0530 Subject: [PATCH 18/22] fix(testmu): never start a daemon with another run's credentials Two runs in one host process with different credentials could overlap their daemon calls, and a daemon one of them started would then read the other's credentials from `process.env`. Daemon calls with the same credentials still overlap; a call with other credentials now waits until those have finished. When the last call finishes, a value is put back only if `process.env` still holds the one the provider set, so a change the host made in the meantime stays. Co-Authored-By: Claude Opus 5.5 --- docs/integrations/testmu.mdx | 7 ++- packages/testmu/src/credentials.ts | 66 +++++++++++++++++------ packages/testmu/tests/unit/testmu.test.ts | 32 +++++++++++ 3 files changed, 86 insertions(+), 19 deletions(-) diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 6d82cd269..f5cbf3e9b 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -140,8 +140,11 @@ environment and takes no other, so while the provider calls the daemon (to allocate, heartbeat, or release a lease), it sets `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment in the runner's `process.env`, where a daemon the call starts reads them. Once no such call is running, it -puts back the values `process.env` had before, so the credentials do not -stay there for other child processes of a long-lived host. +puts back the values `process.env` had before, unless the host changed them +in the meantime, so the credentials do not stay there for other child +processes of a long-lived host. In a host that runs several runs with +different credentials at once, their daemon calls take turns, so no daemon +starts with another run's credentials. ## Install the app diff --git a/packages/testmu/src/credentials.ts b/packages/testmu/src/credentials.ts index 4d82679dc..39b756b9c 100644 --- a/packages/testmu/src/credentials.ts +++ b/packages/testmu/src/credentials.ts @@ -33,41 +33,73 @@ export function optionalTestmuCredentials(env: Readonly; + readonly close: () => void; +} + +let scope: CredentialScope | undefined; /** * Runs a call to the run's daemon with the run's credentials in this * process's environment, which is where a daemon the call starts gets them: * agent-device's client starts its local daemon with `process.env` and takes * no environment of its own, while a run's environment can differ from - * `process.env` (a host that passes `env`). Once no such call is in flight + * `process.env` (a host that passes `env`). Calls with the same credentials + * overlap; a call with other credentials waits until they have finished, so + * a daemon never starts under another run's user. Once no call is in flight * the previous values are put back, so the credentials do not linger for * other child processes of a long-lived host. Every worker of the run * already starts with these values. */ export async function withDaemonCredentials(credentials: TestmuCredentials | undefined, call: () => Promise): Promise { if (credentials === undefined) return call(); - if (scopes === 0) replaced = { username: process.env[LT_USERNAME], accessKey: process.env[LT_ACCESS_KEY] }; - scopes += 1; - process.env[LT_USERNAME] = credentials.username; - process.env[LT_ACCESS_KEY] = credentials.accessKey; + const current = await enterScope(credentials); try { return await call(); } finally { - scopes -= 1; - if (scopes === 0 && replaced !== undefined) { - restore(LT_USERNAME, replaced.username); - restore(LT_ACCESS_KEY, replaced.accessKey); - replaced = undefined; - } + leaveScope(current); + } +} + +async function enterScope(credentials: TestmuCredentials): Promise { + // `scope` changes while this waits: another call may open one first. + for (let open = scope; open !== undefined && !sameCredentials(open.credentials, credentials); open = scope) await open.closed; + if (scope === undefined) { + let close!: () => void; + const closed = new Promise((resolve) => (close = resolve)); + scope = { credentials, calls: 0, replaced: { username: process.env[LT_USERNAME], accessKey: process.env[LT_ACCESS_KEY] }, closed, close }; } + // Set on every call, not only the first: the host may have changed them since. + process.env[LT_USERNAME] = credentials.username; + process.env[LT_ACCESS_KEY] = credentials.accessKey; + scope.calls += 1; + return scope; +} + +function sameCredentials(a: TestmuCredentials, b: TestmuCredentials): boolean { + return a.username === b.username && a.accessKey === b.accessKey; +} + +function leaveScope(current: CredentialScope): void { + current.calls -= 1; + if (current.calls > 0) return; + scope = undefined; + restore(LT_USERNAME, current.credentials.username, current.replaced.username); + restore(LT_ACCESS_KEY, current.credentials.accessKey, current.replaced.accessKey); + current.close(); } -function restore(name: string, value: string | undefined): void { - if (value === undefined) delete process.env[name]; - else process.env[name] = value; +/** Puts back `previous`, unless the host changed the value from the one the scope set: that one is the host's. */ +function restore(name: string, set: string, previous: string | undefined): void { + if (process.env[name] !== set) return; + if (previous === undefined) delete process.env[name]; + else process.env[name] = previous; } /** A non-empty variable from the run's environment, trimmed, or `undefined`. */ diff --git a/packages/testmu/tests/unit/testmu.test.ts b/packages/testmu/tests/unit/testmu.test.ts index ac600c9ac..6de0dbf3d 100644 --- a/packages/testmu/tests/unit/testmu.test.ts +++ b/packages/testmu/tests/unit/testmu.test.ts @@ -255,6 +255,38 @@ describe('testmu()', () => { expect(runnerCredentials()).toEqual([undefined, undefined]); }); + it('makes a daemon call with other credentials wait until the calls with the first have finished', async () => { + delete process.env['LT_USERNAME']; + delete process.env['LT_ACCESS_KEY']; + let open!: () => void; + daemon.allocateWaits = [new Promise((resolve) => (open = resolve))]; + const seen: unknown[] = []; + daemon.onAllocate = () => seen.push(runnerCredentials()); + const alice = testmu(options).acquire(request({ env: { LT_USERNAME: 'alice', LT_ACCESS_KEY: 'alice-key' } })); + await vi.waitFor(() => expect(operations()).toEqual(['allocate'])); + const bob = testmu(options).acquire(request({ env: { LT_USERNAME: 'bob', LT_ACCESS_KEY: 'bob-key' } })); + await new Promise((resolve) => setTimeout(resolve, 20)); + expect(operations()).toEqual(['allocate']); + expect(runnerCredentials()).toEqual(['alice', 'alice-key']); + open(); + await Promise.all([alice, bob]); + expect(seen).toEqual([ + ['alice', 'alice-key'], + ['bob', 'bob-key'], + ]); + expect(runnerCredentials()).toEqual([undefined, undefined]); + }); + + it('leaves a value the host changed during a daemon call', async () => { + process.env['LT_USERNAME'] = 'host'; + process.env['LT_ACCESS_KEY'] = 'host-key'; + daemon.onAllocate = () => { + process.env['LT_USERNAME'] = 'changed-by-host'; + }; + await testmu(options).acquire(request()); + expect(runnerCredentials()).toEqual(['changed-by-host', 'host-key']); + }); + it('shares the credentials with a daemon a heartbeat or a release starts', async () => { vi.useFakeTimers(); process.env['LT_USERNAME'] = 'host'; From 431bb4c993ba664c2e20623a4fb17dc22c2253ca Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:45:10 +0530 Subject: [PATCH 19/22] fix(testmu): prune only run directories the provider marked With `stateDir` set to a shared directory such as the system temp directory or the project root, pruning removed any directory named like a run id that was more than a day old, whoever made it. The provider now writes a `.e2e-testmu-run` marker into each run's directory when it leases and touches it on every heartbeat, and pruning removes only marked directories in which neither the marker nor any other top-level entry has changed for 24 hours, so a run still going in another process is kept. The unit tests now use a temporary project root, since acquiring writes the marker there. Co-Authored-By: Claude Opus 5.5 --- .changeset/testmu-devices.md | 2 +- docs/integrations/testmu.mdx | 11 ++-- packages/testmu/README.md | 3 +- packages/testmu/src/provider.ts | 53 +++++++++++++++---- packages/testmu/tests/unit/prune.test.ts | 62 +++++++++++++++++++---- packages/testmu/tests/unit/testmu.test.ts | 29 +++++++---- 6 files changed, 123 insertions(+), 37 deletions(-) diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md index 4dfa22263..937488967 100644 --- a/.changeset/testmu-devices.md +++ b/.changeset/testmu-devices.md @@ -2,4 +2,4 @@ "@e2e-dev/testmu": minor --- -`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`), in a directory per run that later runs remove once it is more than 24 hours old; allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), allocates each lease with agent-device's longest inactivity window and heartbeats it every 2 minutes while the run holds it, so a lease stays alive however long its session takes to start, links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session, starting at the session's start time), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`), in a directory per run that it marks as its own and that later runs remove once nothing in it has changed for 24 hours; allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), allocates each lease with agent-device's longest inactivity window and heartbeats it every 2 minutes while the run holds it, so a lease stays alive however long its session takes to start, links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session, starting at the session's start time), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index f5cbf3e9b..03ace5afd 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -290,10 +290,13 @@ can link another run's video; give such runs their own `build`, or leave starts the TestMu AI session, which installs `app`. - Keeps each run's directory, with its daemon's state and logs, after the run: the daemon exits on its own after 5 idle minutes, and its logs help - explain a failure. Before its first lease, a run removes the directories - of earlier runs under `stateDir` that were last modified more than 24 - hours ago, never its own; only directories named after a run id are - removed, and a failure to remove one does not fail the lease. + explain a failure. The provider marks each run's directory as its own + with a `.e2e-testmu-run` file and touches it on every heartbeat. Before + its first lease, a run removes the marked directories of earlier runs + under `stateDir` in which nothing has changed for more than 24 hours, + never its own. A directory without the marker is never removed, so a + `stateDir` shared with other tools is safe, and a failure to remove one + does not fail the lease. - Keeps each lease alive while the run holds it. The provider allocates with agent-device's longest lease window, 10 minutes, and heartbeats the lease every 2 minutes, so a lease stays alive however long its session diff --git a/packages/testmu/README.md b/packages/testmu/README.md index 5660aa932..78345c534 100644 --- a/packages/testmu/README.md +++ b/packages/testmu/README.md @@ -65,7 +65,8 @@ ends, which ends the session. session starts. - `stateDir` (default `.e2e/testmu`) holds each run's daemon, in a directory per run that is kept after the run for its logs. A run removes - earlier runs' directories once they are more than 24 hours old. + directories earlier runs marked as the provider's once nothing in them + has changed for 24 hours; it never removes a directory it did not mark. Sessions run over TestMu AI's Appium hub, so agent-device's device settings, system alerts, recording, device logs, and port reverse are not available diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index 2bf1211ea..398b300e7 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -5,7 +5,7 @@ * holding them runs each session over TestMu AI's Appium hub. */ -import { readdir, rm, stat } from 'node:fs/promises'; +import { lstat, mkdir, readdir, rm, utimes, writeFile } from 'node:fs/promises'; import { isAbsolute, join, resolve } from 'node:path'; import type { DeviceLease, DeviceProvider, DeviceReleaseContext, DeviceRequest } from '@e2e-dev/mobile'; import { createAgentDeviceClient } from 'agent-device'; @@ -25,6 +25,13 @@ const RUN_STATE_TTL_MS = 24 * 60 * 60_000; /** A run id, which names each run's directory under `stateDir`; nothing else there is pruned. */ const RUN_DIR_NAME = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; +/** + * The file that marks a run's directory as the provider's: only a marked + * directory is ever pruned, since `stateDir` may be shared with other tools. + * Heartbeats keep it fresh while a run holds leases. + */ +const RUN_MARKER = '.e2e-testmu-run'; + /** The dashboard project sessions are grouped under when `project` is absent. */ const DEFAULT_PROJECT = 'e2e'; @@ -101,8 +108,8 @@ export interface TestmuOptions { /** * Directory for the agent-device daemon each run starts, relative to the * project root. Defaults to `.e2e/testmu`. Each run keeps its own - * directory in it, and the first lease of a run removes earlier runs' - * directories last modified more than a day before. + * directory in it, marked as the provider's, and the first lease of a run + * removes earlier runs' marked directories unchanged for more than a day. */ readonly stateDir?: string | undefined; /** Orientation the device starts in; absent, the device's default. */ @@ -204,6 +211,7 @@ export function testmu(options: TestmuOptions): DeviceProvider { const stateDir = join(baseDir, request.runId); pruning ??= pruneEarlierRuns(baseDir, request.runId); await pruning; + await markRun(stateDir); if (request.signal.aborted) throw new Error('cancelled before a lease was allocated'); const leaseBackend: LeaseBackend = request.platform === 'ios' ? 'ios-instance' : 'android-instance'; const selectors = { @@ -267,9 +275,22 @@ export function testmu(options: TestmuOptions): DeviceProvider { }; } +/** Marks a run's directory as the provider's, creating it. Best effort: an unmarked directory is only never pruned. */ +async function markRun(stateDir: string): Promise { + try { + await mkdir(stateDir, { recursive: true }); + await writeFile(join(stateDir, RUN_MARKER), ''); + } catch { + // The daemon reports a state directory it cannot use. + } +} + /** - * Removes the directories earlier runs left under `baseDir` once they are a - * day old, never the current run's. Best effort: a failure leaves them. + * Removes the directories earlier runs left under `baseDir` once nothing in + * them has changed for a day: only directories named after a run id and + * holding the provider's marker, never the current run's. A run still going + * in another process stays, since its heartbeats touch the marker and its + * daemon writes its log. Best effort: a failure leaves them. */ async function pruneEarlierRuns(baseDir: string, runId: string): Promise { const cutoff = Date.now() - RUN_STATE_TTL_MS; @@ -285,30 +306,40 @@ async function pruneEarlierRuns(baseDir: string, runId: string): Promise { .map(async (name) => { const dir = join(baseDir, name); try { - const info = await stat(dir); - if (info.isDirectory() && info.mtimeMs < cutoff) await rm(dir, { recursive: true, force: true }); + if (!(await lstat(dir)).isDirectory() || !(await lstat(join(dir, RUN_MARKER))).isFile()) return; + if ((await newestChange(dir)) < cutoff) await rm(dir, { recursive: true, force: true }); } catch { - // Gone already, or not ours to remove. + // Not marked, gone already, or not ours to remove. } }), ); } +/** The latest modification time of a directory and of each entry directly in it. */ +async function newestChange(dir: string): Promise { + const times = await Promise.all([dir, ...(await readdir(dir)).map((name) => join(dir, name))].map(async (path) => (await lstat(path)).mtimeMs)); + return Math.max(...times); +} + /** Releases a lease through the daemon that granted it, which ends its TestMu AI session. A lease the daemon no longer knows counts as released. */ async function releaseLease({ stateDir, scope, credentials }: LeaseHandle): Promise { await withDaemonCredentials(credentials, () => createAgentDeviceClient({ stateDir, session: 'release' }).leases.release(scope)); } /** - * Heartbeats a lease until the returned function is called. A command still - * running does not keep its lease alive, so without this a lease can lapse - * while the run holds it. A failed heartbeat is logged once and the next one + * Heartbeats a lease until the returned function is called, touching the + * run's marker so pruning never takes a run that is still going. A command + * still running does not keep its lease alive, so without this a lease can + * lapse while the run holds it. A failed heartbeat is logged once and the next one * tried; it never fails the run. */ function keepAlive({ stateDir, scope, credentials }: LeaseHandle, log: (line: string) => void): () => void { const client = createAgentDeviceClient({ stateDir, session: 'heartbeat' }); let warned = false; + const marker = join(stateDir, RUN_MARKER); const timer = setInterval(() => { + const now = new Date(); + utimes(marker, now, now).catch(() => undefined); withDaemonCredentials(credentials, () => client.leases.heartbeat({ ...scope, ttlMs: LEASE_TTL_MS })).catch((cause: unknown) => { if (warned) return; warned = true; diff --git a/packages/testmu/tests/unit/prune.test.ts b/packages/testmu/tests/unit/prune.test.ts index b31222373..e91cbeebd 100644 --- a/packages/testmu/tests/unit/prune.test.ts +++ b/packages/testmu/tests/unit/prune.test.ts @@ -1,10 +1,11 @@ /** * `testmu()` pruning the state directories earlier runs left under - * `stateDir`: once per provider, only run directories older than a day, - * never the current run's, and never failing the lease. + * `stateDir`: once per provider, only directories it marked as its own whose + * newest file is older than a day, never the current run's, and never + * failing the lease; and the marker it writes and keeps fresh. */ -import { existsSync, mkdirSync, mkdtempSync, readdirSync, rmSync, utimesSync, writeFileSync } from 'node:fs'; +import { existsSync, mkdirSync, mkdtempSync, readdirSync, rmSync, statSync, utimesSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import type { DeviceRequest } from '@e2e-dev/mobile'; @@ -27,7 +28,13 @@ const OLD_RUN = '01a0e000-0000-7000-8000-000000000001'; const RECENT_RUN = '01a0e000-0000-7000-8000-000000000002'; /** A file named like a run: only directories are pruned. */ const RUN_FILE = '01a0e000-0000-7000-8000-000000000003'; +/** A directory named like a run that another tool keeps under the same directory. */ +const FOREIGN_RUN = '01a0e000-0000-7000-8000-000000000004'; +/** A marked run, still going in another process, whose daemon wrote its log recently. */ +const LONG_RUN = '01a0e000-0000-7000-8000-000000000005'; +const MARKER = '.e2e-testmu-run'; const HOUR = 60 * 60_000; +const MINUTE = 60_000; let root: string; let base: string; @@ -42,13 +49,23 @@ afterEach(() => { rmSync(root, { recursive: true, force: true }); }); -/** A directory under the state directory, last modified `ageMs` ago, with a daemon log in it. */ -function runDir(name: string, ageMs: number): void { +/** Sets a path's modification time `ageMs` ago. */ +function age(path: string, ageMs: number): void { + const time = new Date(Date.now() - ageMs); + utimesSync(path, time, time); +} + +/** A run directory under the state directory with a daemon log, marked as the provider's unless `marked` is false, every entry `ageMs` old. */ +function runDir(name: string, ageMs: number, { marked = true } = {}): void { const dir = join(base, name); mkdirSync(dir, { recursive: true }); writeFileSync(join(dir, 'daemon.log'), 'log'); - const time = new Date(Date.now() - ageMs); - utimesSync(dir, time, time); + age(join(dir, 'daemon.log'), ageMs); + if (marked) { + writeFileSync(join(dir, MARKER), ''); + age(join(dir, MARKER), ageMs); + } + age(dir, ageMs); } function request(overrides: Partial = {}): DeviceRequest { @@ -68,15 +85,40 @@ function request(overrides: Partial = {}): DeviceRequest { } describe('testmu() state directory pruning', () => { - it("removes earlier runs' directories older than a day, and keeps the current run's, recent ones, and anything not named like a run", async () => { + it("removes earlier runs' marked directories older than a day, and keeps the current run's, recent ones, and anything it did not mark", async () => { runDir(OLD_RUN, 25 * HOUR); runDir(RECENT_RUN, 23 * HOUR); runDir(CURRENT_RUN, 48 * HOUR); + runDir(FOREIGN_RUN, 48 * HOUR, { marked: false }); runDir('notes', 48 * HOUR); writeFileSync(join(base, RUN_FILE), 'a file'); - utimesSync(join(base, RUN_FILE), new Date(Date.now() - 48 * HOUR), new Date(Date.now() - 48 * HOUR)); + age(join(base, RUN_FILE), 48 * HOUR); await testmu(options).acquire(request()); - expect(readdirSync(base).toSorted()).toEqual([CURRENT_RUN, RECENT_RUN, RUN_FILE, 'notes'].toSorted()); + expect(readdirSync(base).toSorted()).toEqual([CURRENT_RUN, FOREIGN_RUN, RECENT_RUN, RUN_FILE, 'notes'].toSorted()); + }); + + it('keeps a marked run whose daemon wrote a file within the day, however old its marker and directory', async () => { + runDir(LONG_RUN, 30 * HOUR); + age(join(base, LONG_RUN, 'daemon.log'), HOUR); + age(join(base, LONG_RUN), 30 * HOUR); + await testmu(options).acquire(request()); + expect(existsSync(join(base, LONG_RUN))).toBe(true); + }); + + it("marks the run's directory when it leases, and touches the marker on every heartbeat", async () => { + vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] }); + try { + const provider = testmu(options); + const lease = await provider.acquire(request()); + const marker = join(base, CURRENT_RUN, MARKER); + expect(statSync(marker).isFile()).toBe(true); + age(marker, 30 * HOUR); + await vi.advanceTimersByTimeAsync(2 * MINUTE); + await vi.waitFor(() => expect(Date.now() - statSync(marker).mtimeMs).toBeLessThan(HOUR)); + await provider.release(lease, { runId: CURRENT_RUN, targetName: 'android', env: {}, signal: new AbortController().signal, log: () => undefined }); + } finally { + vi.useRealTimers(); + } }); it('prunes once per provider, on its first acquire', async () => { diff --git a/packages/testmu/tests/unit/testmu.test.ts b/packages/testmu/tests/unit/testmu.test.ts index 6de0dbf3d..008f82af8 100644 --- a/packages/testmu/tests/unit/testmu.test.ts +++ b/packages/testmu/tests/unit/testmu.test.ts @@ -5,9 +5,11 @@ * on every exit path. `recording.test.ts` covers `record`. */ +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; import { join } from 'node:path'; import type { DeviceLease, DeviceReleaseContext, DeviceRequest } from '@e2e-dev/mobile'; -import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { testmu, type TestmuOptions } from '../../src/index.ts'; interface ClientCall { @@ -20,8 +22,8 @@ const daemon = { calls: [] as ClientCall[], /** Runs inside `allocate`, before it answers. */ onAllocate: undefined as (() => void) | undefined, - /** What each `allocate`, in order, waits for before it runs `onAllocate`. */ - allocateWaits: [] as Promise[], + /** What an `allocate` from a client of that session waits for, once, before it runs `onAllocate`. */ + allocateGates: {} as Record>, /** Runs inside `heartbeat` and `release`, before they answer. */ onHeartbeat: undefined as (() => void) | undefined, onRelease: undefined as (() => void) | undefined, @@ -37,8 +39,9 @@ vi.mock('agent-device', () => ({ leases: { allocate: async (options: Record) => { daemon.calls.push({ config, operation: 'allocate', options }); - const wait = daemon.allocateWaits.shift(); - if (wait !== undefined) await wait; + const gate = daemon.allocateGates[String(config['session'])]; + delete daemon.allocateGates[String(config['session'])]; + if (gate !== undefined) await gate; daemon.onAllocate?.(); if (daemon.allocateError !== undefined) throw daemon.allocateError; return { leaseId: `lease-${daemon.calls.length}`, tenantId: options['tenant'], runId: options['runId'], backend: options['leaseBackend'], leaseProvider: options['leaseProvider'] }; @@ -61,7 +64,12 @@ vi.mock('agent-device', () => ({ }), })); -const ROOT = join('/', 'work', 'shop'); +/** A real directory: the provider writes each run's marker under the project root. */ +const ROOT = mkdtempSync(join(tmpdir(), 'testmu-unit-')); + +afterAll(() => { + rmSync(ROOT, { recursive: true, force: true }); +}); const env = { LT_USERNAME: 'ada', LT_ACCESS_KEY: 'lt-key' }; const options: TestmuOptions = { device: 'Galaxy S22 Ultra 5G', osVersion: '14', app: 'https://example.com/app.apk' }; const saved = { LT_USERNAME: process.env['LT_USERNAME'], LT_ACCESS_KEY: process.env['LT_ACCESS_KEY'] }; @@ -70,7 +78,7 @@ beforeEach(() => { Object.assign(daemon, { calls: [], onAllocate: undefined, - allocateWaits: [], + allocateGates: {}, onHeartbeat: undefined, onRelease: undefined, allocateError: undefined, @@ -238,12 +246,13 @@ describe('testmu()', () => { delete process.env['LT_USERNAME']; delete process.env['LT_ACCESS_KEY']; let open!: () => void; - daemon.allocateWaits = [Promise.resolve(), new Promise((resolve) => (open = resolve))]; + daemon.allocateGates = { 'lease-1': new Promise((resolve) => (open = resolve)) }; const seen: unknown[] = []; daemon.onAllocate = () => seen.push(runnerCredentials()); const provider = testmu(options); const first = provider.acquire(request({ slot: 0 })); const second = provider.acquire(request({ slot: 1 })); + await vi.waitFor(() => expect(operations()).toEqual(['allocate', 'allocate'])); await first; expect(runnerCredentials()).toEqual(['ada', 'lt-key']); open(); @@ -259,12 +268,12 @@ describe('testmu()', () => { delete process.env['LT_USERNAME']; delete process.env['LT_ACCESS_KEY']; let open!: () => void; - daemon.allocateWaits = [new Promise((resolve) => (open = resolve))]; + daemon.allocateGates = { 'lease-0': new Promise((resolve) => (open = resolve)) }; const seen: unknown[] = []; daemon.onAllocate = () => seen.push(runnerCredentials()); const alice = testmu(options).acquire(request({ env: { LT_USERNAME: 'alice', LT_ACCESS_KEY: 'alice-key' } })); await vi.waitFor(() => expect(operations()).toEqual(['allocate'])); - const bob = testmu(options).acquire(request({ env: { LT_USERNAME: 'bob', LT_ACCESS_KEY: 'bob-key' } })); + const bob = testmu(options).acquire(request({ slot: 1, env: { LT_USERNAME: 'bob', LT_ACCESS_KEY: 'bob-key' } })); await new Promise((resolve) => setTimeout(resolve, 20)); expect(operations()).toEqual(['allocate']); expect(runnerCredentials()).toEqual(['alice', 'alice-key']); From ad6a8ea4a702d48483892cd083874261f5bf3369 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Fri, 2 Oct 2026 22:49:36 +0530 Subject: [PATCH 20/22] docs(testmu): bound the lease-window claim and note one account per process A lease's window covers a session that takes up to 10 minutes to start; the heartbeat only takes over once the lease is granted. In a host that runs several TestMu AI accounts at once, daemon calls take turns, so one run's slow allocation can delay another's heartbeats. Co-Authored-By: Claude Opus 5.5 --- .changeset/testmu-devices.md | 2 +- docs/integrations/testmu.mdx | 9 +++++---- packages/testmu/src/provider.ts | 8 ++++---- 3 files changed, 10 insertions(+), 9 deletions(-) diff --git a/.changeset/testmu-devices.md b/.changeset/testmu-devices.md index 937488967..73dbb3afd 100644 --- a/.changeset/testmu-devices.md +++ b/.changeset/testmu-devices.md @@ -2,4 +2,4 @@ "@e2e-dev/testmu": minor --- -`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`), in a directory per run that it marks as its own and that later runs remove once nothing in it has changed for 24 hours; allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), allocates each lease with agent-device's longest inactivity window and heartbeats it every 2 minutes while the run holds it, so a lease stays alive however long its session takes to start, links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session, starting at the session's start time), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. +`@e2e-dev/testmu`: `mobile({ device: testmu({ device, osVersion, app }) })` runs a mobile target on TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real devices (`deviceType: 'real'`), one per worker slot, through agent-device's `testmu` cloud provider. Each slot leases a device from an agent-device daemon the provider starts for the run under `stateDir` (default `.e2e/testmu`), in a directory per run that it marks as its own and that later runs remove once nothing in it has changed for 24 hours; allocating a lease starts its TestMu AI session, which installs `app` (an `lt://` id, an `https` URL, or a local path), so every slot is billed from the moment it is leased, and releasing the lease ends it. It reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and fails the lease before starting anything when either is missing, sets up the device with `orientation`, `geoLocation`, `timezone`, `language`, `locale`, and `appiumVersion` when given, groups sessions on the dashboard under `project` (default `e2e`) and `build` (default the run id) and names each slot's session `sessionName` (default `e2e---`, with `-` appended to a given name when the target has more than one slot), allocates each lease with agent-device's longest inactivity window and heartbeats it every 2 minutes while the run holds it, so a session that takes up to 10 minutes to start keeps its lease, links TestMu AI's recording of the slot's session as the video of an attempt that records one (the whole session, starting at the session's start time), found through TestMu AI's sessions API by the session's build and name, releases a lease it allocated when the run was interrupted, and releases each lease once. It needs an agent-device that includes the `testmu` provider, which no published release does yet, with `@e2e-dev/mobile`'s agent-device overridden to the same version; `agent-device` and `@e2e-dev/mobile` are its peers. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index 03ace5afd..ba972c80d 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -143,8 +143,9 @@ where a daemon the call starts reads them. Once no such call is running, it puts back the values `process.env` had before, unless the host changed them in the meantime, so the credentials do not stay there for other child processes of a long-lived host. In a host that runs several runs with -different credentials at once, their daemon calls take turns, so no daemon -starts with another run's credentials. +different credentials at once, their daemon calls take turns, and one run's +slow allocation can delay another run's heartbeats; run one TestMu AI +account per process where you can. ## Install the app @@ -299,8 +300,8 @@ can link another run's video; give such runs their own `build`, or leave does not fail the lease. - Keeps each lease alive while the run holds it. The provider allocates with agent-device's longest lease window, 10 minutes, and heartbeats the - lease every 2 minutes, so a lease stays alive however long its session - takes to start, on any agent-device version. A failed heartbeat is logged + lease every 2 minutes once it is granted, so a session that takes up to + 10 minutes to start keeps its lease on any agent-device version. A failed heartbeat is logged once and does not fail the run. - Hands the worker the lease scope and the device selectors as agent-device client configuration, so every command lands on the lease's session. diff --git a/packages/testmu/src/provider.ts b/packages/testmu/src/provider.ts index 398b300e7..0923c97de 100644 --- a/packages/testmu/src/provider.ts +++ b/packages/testmu/src/provider.ts @@ -36,10 +36,10 @@ const RUN_MARKER = '.e2e-testmu-run'; const DEFAULT_PROJECT = 'e2e'; /** - * The inactivity window each lease asks for, agent-device's longest. With - * the heartbeat, a lease stays alive however long its session takes to - * start, whether agent-device starts the window when the allocation begins - * or when it completes. + * The inactivity window each lease asks for, agent-device's longest, so a + * session that takes up to 10 minutes to start keeps its lease whether + * agent-device starts the window when the allocation begins or when it + * completes; the heartbeat keeps it alive after that. */ const LEASE_TTL_MS = 10 * 60_000; From 4a6def633217b098bf0960543789852dedb2e2f3 Mon Sep 17 00:00:00 2001 From: amankansal-lt Date: Mon, 5 Oct 2026 13:47:06 +0530 Subject: [PATCH 21/22] chore(testmu): follow upstream's agent-device 0.21.20 pin and Node floor @e2e-dev/mobile now pins agent-device 0.21.20, so testmu develops against the same copy, and its peer floor moves past 0.21.20, which still has no testmu provider. engines.node matches every other @e2e-dev package after the oxc loader change. This also drops the lockfile's dangling agent-device 0.21.18 reference left by the merge. Co-Authored-By: Claude Opus 5.5 --- packages/testmu/package.json | 6 +++--- pnpm-lock.yaml | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/packages/testmu/package.json b/packages/testmu/package.json index f4fc14f91..7fe24c378 100644 --- a/packages/testmu/package.json +++ b/packages/testmu/package.json @@ -56,18 +56,18 @@ }, "peerDependencies": { "@e2e-dev/mobile": ">=0.9.0 <1", - "agent-device": ">0.21.18 <1", + "agent-device": ">0.21.20 <1", "e2e": ">=0.15.0 <1" }, "devDependencies": { "@e2e-dev/mobile": "workspace:*", "@types/node": "26.6.2", - "agent-device": "0.21.18", + "agent-device": "0.21.20", "e2e": "workspace:*", "typescript": "7.0.2", "vitest": "5.0.1" }, "engines": { - "node": ">=22.12.0" + "node": "^22.22.3 || >=24.8.0" } } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index d3293a3ed..b9f2d6fc1 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -537,8 +537,8 @@ importers: specifier: 26.6.2 version: 26.6.2 agent-device: - specifier: 0.21.18 - version: 0.21.18(ai@7.0.107(zod@4.6.1)) + specifier: 0.21.20 + version: 0.21.20(ai@7.0.107(zod@4.6.1)) e2e: specifier: workspace:* version: link:../e2e From fe91e5fe181f3e56350c4a6c496d1e0ff042c695 Mon Sep 17 00:00:00 2001 From: SahilSawLT Date: Mon, 5 Oct 2026 20:02:08 +0530 Subject: [PATCH 22/22] feat(testmu): hosted Chrome and Edge for the web engine with testmuBrowsers() @e2e-dev/testmu/web exports testmuBrowsers(), a BrowserProvider for TestMu AI (formerly LambdaTest) hosted Chrome and Edge on Windows and macOS. A session starts when its CDP websocket opens and ends when the engine closes it, so the provider only builds the CDP URL from the options and LT_USERNAME / LT_ACCESS_KEY (via the package's testmuCredentials()); it calls no API. - One session per worker slot, or per attempt with scope: 'attempt'. - route /puppeteer (default, raw CDP on every machine) or /playwright-cdp (Playwright label; raw CDP only where TestMu AI's newest backend has rolled out). hub cdp.lambdatest.com. - Sessions are named as testmu()'s: build (default the run id, so a run's device and browser sessions share one build) and sessionName (default e2e---); project (default e2e) is sent as LT:Options.project. geoLocation and timezone are options, as on testmu(). - idleTimeout defaults to 600 s. Unknown options, out-of-range values, empty or null strings, and capabilities that set what the provider sets (including browserName, browserVersion and a nested LT:Options) fail with INVALID_CONFIG at config load. - The package root stays device-only; the browser provider lives at @e2e-dev/testmu/web, and @e2e-dev/mobile, @e2e-dev/web and agent-device are optional peers, so neither side's imports or types reach the other. - Docs: integrations/testmu-browsers page with a checked example (docs/examples/web/testmu-browsers.config.ts), nav, card, env, security, browser.mdx, README, AGENTS, skill row; credentials sourcing on both pages. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01KoNxaN1YAAP9PwmHtMm1WX --- .changeset/testmu-browsers.md | 5 + AGENTS.md | 7 + README.md | 2 +- docs/browser.mdx | 5 +- docs/docs.json | 3 +- docs/examples/web/testmu-browsers.config.ts | 13 ++ docs/integrations/index.mdx | 3 + docs/integrations/testmu-browsers.mdx | 171 ++++++++++++++++ docs/integrations/testmu.mdx | 4 + docs/reference/environment.mdx | 2 +- docs/security.mdx | 3 + packages/testmu/README.md | 40 +++- packages/testmu/package.json | 22 +- packages/testmu/src/browsers.ts | 189 +++++++++++++++++ packages/testmu/src/index.ts | 2 + packages/testmu/src/web.ts | 8 + packages/testmu/tests/unit/browsers.test.ts | 212 ++++++++++++++++++++ pnpm-lock.yaml | 3 + scripts/check-docs-examples.ts | 1 + skills/e2e/references/setup.md | 2 +- 20 files changed, 689 insertions(+), 8 deletions(-) create mode 100644 .changeset/testmu-browsers.md create mode 100644 docs/examples/web/testmu-browsers.config.ts create mode 100644 docs/integrations/testmu-browsers.mdx create mode 100644 packages/testmu/src/browsers.ts create mode 100644 packages/testmu/src/web.ts create mode 100644 packages/testmu/tests/unit/browsers.test.ts diff --git a/.changeset/testmu-browsers.md b/.changeset/testmu-browsers.md new file mode 100644 index 000000000..9d02bc331 --- /dev/null +++ b/.changeset/testmu-browsers.md @@ -0,0 +1,5 @@ +--- +"@e2e-dev/testmu": minor +--- + +`@e2e-dev/testmu/web`: `web({ browser: testmuBrowsers() })` runs a web target in TestMu AI (formerly LambdaTest) hosted Chrome or Edge on Windows and macOS, one session per worker slot, or one per attempt with `scope: 'attempt'`. A session starts when its CDP websocket opens and ends when the engine closes it, so the provider only builds the CDP URL (`hub` `cdp.lambdatest.com`) from `browserName`, `browserVersion`, `platform`, `geoLocation`, `timezone`, and further `LT:Options` `capabilities`, with `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment, which never appear in a log line. `route` is `/puppeteer` by default, which serves raw CDP on every machine; `/playwright-cdp` labels sessions as Playwright but serves raw CDP only on machines with TestMu AI's newest backend, which is still rolling out. Sessions are named as `testmu()`'s are: `build` defaults to the run id, so a run's device and browser sessions share one build, `sessionName` to `e2e---`, and `project` (default `e2e`) is sent as `LT:Options.project`. `idleTimeout` defaults to 600 seconds. Unknown options, a `scope` or `route` outside their values, a `hub` with a scheme or path, a non-Chromium `browserName`, an empty or non-string string option, `null` for any option, and `capabilities` that set what the provider sets (`user`, `accessKey`, `build`, `name`, `project`, `platform`, `geoLocation`, `timezone`, `browserName`, `browserVersion`, or a nested `LT:Options`) fail with `INVALID_CONFIG` at config load. The package root stays device-only, and `@e2e-dev/mobile`, `@e2e-dev/web`, and `agent-device` are optional peers, so a project installs only the side it uses. diff --git a/AGENTS.md b/AGENTS.md index 4f761c8c8..493187f2b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,6 +83,13 @@ suites that consume the built packages the way a user would. cloud provider is the vendor client: the package allocates agent-device leases from a daemon it starts per run, so `agent-device` (which must be the same copy `@e2e-dev/mobile` uses) and `@e2e-dev/mobile` are its peers. + `@e2e-dev/testmu/web` exports `testmuBrowsers()`, TestMu AI hosted Chrome + and Edge for the web engine (`BrowserProvider`): a session starts when its + CDP websocket opens and ends when it closes, so it only builds the URL and + calls no API. The root stays device-only, so neither side's types or + imports reach the other. Every + peer (`@e2e-dev/mobile`, `@e2e-dev/web`, `agent-device`) is optional, so a + project installs only the side it uses. - `apps/testbed` (`@e2e-dev/testbed`, private) — dogfood project that consumes the **built** packages like a real user would: the playground app where every runner feature (sessions, routes, downloads, frames, uploads, diff --git a/README.md b/README.md index 65bf73b19..ee0caf714 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ config and an example test. The | [`@e2e-dev/kernel`](https://www.npmjs.com/package/@e2e-dev/kernel) | Kernel hosted browsers for the web engine. | | [`@e2e-dev/eas`](https://www.npmjs.com/package/@e2e-dev/eas) | EAS Simulators hosted iOS simulators and Android emulators for the mobile engine. | | [`@e2e-dev/decision`](https://e2e.tester.army/docs/decision-models) | Decision-model executors for bounded semantic actions and assertions. | -| [`@e2e-dev/testmu`](https://www.npmjs.com/package/@e2e-dev/testmu) | TestMu AI (formerly LambdaTest) hosted Android emulators, iOS simulators, and real devices for the mobile engine. | +| [`@e2e-dev/testmu`](https://www.npmjs.com/package/@e2e-dev/testmu) | TestMu AI (formerly LambdaTest) hosted Android emulators, iOS simulators, and real devices for the mobile engine, and hosted Chrome and Edge for the web engine (`@e2e-dev/testmu/web`). | ## Documentation diff --git a/docs/browser.mdx b/docs/browser.mdx index da6bf58df..1079652bb 100644 --- a/docs/browser.mdx +++ b/docs/browser.mdx @@ -194,8 +194,9 @@ lacks: it asks for a browser when one is needed and gives it back when its scope ends, on every exit path, so a session billed by the minute stops when the run no longer uses it. The engine knows no vendor: for [Kernel](/integrations/kernel), `kernel()` from `@e2e-dev/kernel` is the -provider, and for any other service a provider is a few dozen lines in your -project. Add `viewport: null` when +provider, for [TestMu AI](/integrations/testmu-browsers), +`testmuBrowsers()` from `@e2e-dev/testmu/web` is the provider, and for any other +service a provider is a few dozen lines in your project. Add `viewport: null` when the service shows the window in a live view: the page then [fills the window](/viewports#fill-the-window) instead of a fixed 1280 by 720 area of it. diff --git a/docs/docs.json b/docs/docs.json index 30906c169..cbe567e77 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -210,7 +210,8 @@ "integrations/index", "integrations/kernel", "integrations/eas", - "integrations/testmu" + "integrations/testmu", + "integrations/testmu-browsers" ] }, { diff --git a/docs/examples/web/testmu-browsers.config.ts b/docs/examples/web/testmu-browsers.config.ts new file mode 100644 index 000000000..535f23796 --- /dev/null +++ b/docs/examples/web/testmu-browsers.config.ts @@ -0,0 +1,13 @@ +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { testmuBrowsers } from '@e2e-dev/testmu/web'; + +export default { + targets: [ + { + name: 'testmu-web', + engine: web({ browser: testmuBrowsers(), viewport: null }), + app: { url: 'https://staging.example.com' }, + }, + ], +} satisfies E2EConfig; diff --git a/docs/integrations/index.mdx b/docs/integrations/index.mdx index b6cebe12a..838815e40 100644 --- a/docs/integrations/index.mdx +++ b/docs/integrations/index.mdx @@ -19,6 +19,9 @@ integration decides where it comes from. `@e2e-dev/testmu`: hosted Android emulators, iOS simulators, and real devices for mobile targets. + + `@e2e-dev/testmu`: hosted Chrome and Edge on Windows and macOS for web targets, one session per worker or per test. + A service without an integration is one small provider in your project: diff --git a/docs/integrations/testmu-browsers.mdx b/docs/integrations/testmu-browsers.mdx new file mode 100644 index 000000000..697a5ee65 --- /dev/null +++ b/docs/integrations/testmu-browsers.mdx @@ -0,0 +1,171 @@ +--- +title: TestMu AI browsers +sidebarTitle: TestMu AI browsers +description: Run web targets in TestMu AI's hosted Chrome and Edge with the testmuBrowsers() browser provider from @e2e-dev/testmu/web. +--- + +`testmuBrowsers()` from `@e2e-dev/testmu/web` is a +[browser provider](/browser#hosted-browsers) for +[TestMu AI](https://www.lambdatest.com) (formerly LambdaTest). By default each worker slot gets its +own TestMu AI session for the run; `scope: 'attempt'` gives every test +attempt its own session instead. Tests and CI stay the same; the config +changes only in the target's `browser`. + +## Setup + + +```bash npm +npm install --save-dev @e2e-dev/testmu +``` + +```bash pnpm +pnpm add -D @e2e-dev/testmu +``` + +```bash bun +bun add -d @e2e-dev/testmu +``` + + +```ts title="e2e.config.ts" +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { testmuBrowsers } from '@e2e-dev/testmu/web'; + +export default { + targets: [ + { + name: 'testmu-web', + engine: web({ browser: testmuBrowsers(), viewport: null }), + app: { url: 'https://staging.example.com' }, + }, + ], +} satisfies E2EConfig; +``` + +The browser provider lives at `@e2e-dev/testmu/web`, so a web-only project +needs neither `@e2e-dev/mobile` nor agent-device. The package root exports +the [device provider](/integrations/testmu), which uses both. + +Set `LT_USERNAME` and `LT_ACCESS_KEY` in the environment `e2e run` starts in, +such as CI secrets. To get them, [sign up for TestMu AI](https://accounts.lambdatest.com/register) +or log in to your account, then copy your username and access key from +**Account Settings → Password & Security → Username and Access Key** +([accounts.lambdatest.com/security/username-accesskey](https://accounts.lambdatest.com/security/username-accesskey)). + +Each worker gets its own session, so `workers` sets how +many run at once. Keep it within your TestMu AI plan's parallel sessions: +a session beyond it waits in TestMu AI's queue, and the web engine gives a +session 60 seconds to accept the CDP connection, so one that waits longer +fails its attempt. See [Limits](#limits). + +## Options + +| Option | Default | Meaning | +| ------ | ------- | ------- | +| `scope` | `'worker'` | `'worker'`: one session per worker slot for the run. `'attempt'`: a fresh session per test attempt, retries included. See [Scopes](#scopes). | +| `browserName` | `'Chrome'` | `'Chrome'` or `'MicrosoftEdge'`. The web engine attaches over CDP, so the browser must be Chromium-based; any other value fails the config. | +| `browserVersion` | `'latest'` | Any version TestMu AI offers for the platform, such as `'140'`. | +| `platform` | `'Windows 11'` | A TestMu AI desktop platform, such as `'Windows 10'` or `'macOS Sequoia'`. | +| `project` | `'e2e'` | Sent as `LT:Options.project`, as [`testmu()`](/integrations/testmu) sends its own. | +| `build` | the run id | The dashboard build the sessions are grouped under. The same default as `testmu()`'s, so a run's device and browser sessions share one build. | +| `sessionName` | `e2e---` | Every session's dashboard name. A given name gets `-` appended when the target has more than one worker slot, and `-` in `attempt` scope, so each session has its own. | +| `geoLocation` | none | Country the browser's IP geolocates to, as a code TestMu AI takes: `'US'`, `'FR'`. | +| `timezone` | none | The machine's time zone, as TestMu AI takes it: `'UTC+05:30'`. | +| `route` | `'/puppeteer'` | The hub route that serves the session's CDP endpoint. See [Routes](#routes). | +| `hub` | `'cdp.lambdatest.com'` | The hub host, without a scheme or path. | +| `capabilities` | none | Further `LT:Options` capabilities, such as `video`, `network`, `console`, `idleTimeout`, `resolution`, or `tunnel` and `tunnelName` for [TestMu AI Tunnel](https://www.lambdatest.com/support/docs/testing-locally-hosted-pages/). `idleTimeout` is 600 seconds when absent. `user`, `accessKey`, `build`, `name`, `project`, `platform`, `geoLocation`, `timezone`, `browserName`, `browserVersion`, and a nested `LT:Options` fail the config here: the provider sets them from the environment and the options above. | + +```ts +testmuBrowsers({ + scope: 'attempt', + platform: 'Windows 10', + build: `checkout ${process.env.GITHUB_SHA}`, + capabilities: { video: true, network: true, idleTimeout: 300 }, +}) +``` + +A misspelled option, a `route` other than the two below, or a `hub` with a +scheme or path fails with `INVALID_CONFIG` when the config loads. + +## Scopes + +`scope: 'worker'` keeps one session per worker slot for the whole run and +gives every attempt a new browser context in it, as a local run does: a test +never sees another test's cookies, storage, or permissions. TestMu AI shows +it as one session, named after the run, the target, and the slot, that covers every +test the worker ran. + +`scope: 'attempt'` starts a session per attempt, so each test is its own +TestMu AI session, named after the run, the target, and the attempt. It costs a +session start per test and carries the limits of a +[per-attempt lease](/browser#hosted-browsers): no `headers`, `basicAuth`, or +`userAgent`, no `app.clearState()`, and no [sessions](/authentication). A +dropped connection cannot be resumed. The engine reconnects to the same CDP +URL, which starts a new TestMu AI session under the same name; the engine +sees a different browser, closes it, and fails the attempt with +`CDP recovery failed`. That second session shows in the dashboard and counts +against your plan while it lasts. + +## Routes + +A TestMu AI session starts when its CDP websocket opens, on one of the +hub's routes: + +- `'/puppeteer'` (default) serves a raw CDP endpoint on every TestMu AI + machine. The dashboard labels these sessions Puppeteer. +- `'/playwright-cdp'` labels sessions as Playwright, but serves raw CDP only on + machines that run TestMu AI's newest backend, which is still rolling out. On + a machine without it the session fails to connect, so keep the default + unless you need the Playwright label. + +## Limits + +- **60-second connect.** The web engine waits 60 seconds for the browser to + accept the CDP connection, and `launchTimeout` does not change that. A + session still in TestMu AI's queue, or one that is slow to provision, fails + its attempt with `connectOverCDP: Timeout 60000ms exceeded` + (`LAUNCH_TIMEOUT`). Chrome on Windows typically connects in about 10 to 30 + seconds; Edge sessions can take longer than the limit. +- **About 24 to 30 workers per target in `worker` scope.** The engine hands + every worker's CDP URL to the workers in one environment variable capped at + 16 KB, and each URL carries its capabilities. Past about 30 workers with the + defaults, or 24 with several `capabilities`, the run fails at start with + `ENGINE_FAILURE: the browser leases take ... bytes`. `scope: 'attempt'` leases in each worker and + has no such ceiling. + +## Recordings + +TestMu AI records the session's screen itself when you pass +`capabilities: { video: true }`; watch it in the TestMu AI dashboard. The +provider does not hand that recording to e2e, so an attempt that records +[video](/reference/config#video) gets the web engine's screencast of the +page, as a local run does. + +## Downloads + +TestMu AI's browsers save downloads on their own machines, and the provider +has no way to read them back, so `browser.waitForDownload` fails with +`ENGINE_FAILURE` naming the provider. Assert on the download link or the +request instead. + +## What the provider does + +- Builds the session's CDP URL with the capabilities above and returns it as + the lease. It calls no TestMu AI API: opening the websocket starts the + session and the engine closing it ends the session, so `release` has + nothing to call. +- Names every session as `testmu()` does, `e2e---` unless `sessionName` is given, in the `build`, sends `project` + as `LT:Options.project`, and + logs `TestMu AI session "" in build ""` to the reporter as + each lease is made. +- Reads `LT_USERNAME` and `LT_ACCESS_KEY` from the run's environment and puts + them only in the CDP URL: they never appear in a log line. A missing or + blank one fails the lease with an error naming it, such as + `LT_ACCESS_KEY is not set`; in `worker` scope that ends the run before any + test. +- Sets `idleTimeout` to 600 seconds unless `capabilities` sets one: the hub + otherwise ends a session after 300 seconds without browser traffic, which a + slow step between two browser calls can exceed. The same timeout ends a + session a dead worker held. diff --git a/docs/integrations/testmu.mdx b/docs/integrations/testmu.mdx index ba972c80d..0ea55afbd 100644 --- a/docs/integrations/testmu.mdx +++ b/docs/integrations/testmu.mdx @@ -132,6 +132,10 @@ package, not the upload name. Set `LT_USERNAME` and `LT_ACCESS_KEY` to your TestMu AI username and access key in the environment `e2e run` starts in, such as CI secrets. Without either, the lease fails before anything starts, naming the missing one. +To get them, [sign up for TestMu AI](https://accounts.lambdatest.com/register) +or log in to your account, then copy your username and access key from +**Account Settings → Password & Security → Username and Access Key** +([accounts.lambdatest.com/security/username-accesskey](https://accounts.lambdatest.com/security/username-accesskey)). agent-device's `testmu` runtime reads the credentials from the environment of the daemon that drives the sessions. The provider starts that daemon from diff --git a/docs/reference/environment.mdx b/docs/reference/environment.mdx index 31f38a206..725e90e21 100644 --- a/docs/reference/environment.mdx +++ b/docs/reference/environment.mdx @@ -109,7 +109,7 @@ What is sent and how to read it is on the [Telemetry](/telemetry) page. | `KERNEL_API_KEY` | value | The API key `kernel()` from `@e2e-dev/kernel` creates browsers with; unset, the browser lease fails. See [Kernel](/integrations/kernel) | | `EXPO_TOKEN` | value | The Expo access token `easSimulators()` from `@e2e-dev/eas` starts and stops simulator sessions with; unset, it uses the eas-cli login in `.expo/state.json` under `HOME` (`USERPROFILE` on Windows), and with neither the device lease fails. See [EAS Simulators](/integrations/eas) | | `HOME`, `USERPROFILE` | value | The home `easSimulators()` finds the eas-cli login under when `EXPO_TOKEN` is unset: `USERPROFILE` on Windows, `HOME` elsewhere, the OS user's home when unset | -| `LT_USERNAME`, `LT_ACCESS_KEY` | value | The TestMu AI username and access key `testmu()` from `@e2e-dev/testmu` leases devices with; it sets them in the runner process's environment only while it calls the agent-device daemon it starts, which reads them from there, and with either unset the device lease fails. See [TestMu AI](/integrations/testmu) | +| `LT_USERNAME`, `LT_ACCESS_KEY` | value | The TestMu AI username and access key `testmu()` from `@e2e-dev/testmu` leases devices with; it sets them in the runner process's environment only while it calls the agent-device daemon it starts, which reads them from there, and with either unset the device lease fails. `testmuBrowsers()` puts them only in the CDP URL of each browser session, and with either unset the browser lease fails. Both come from Account Settings → Password & Security → Username and Access Key on your TestMu AI account. See [TestMu AI](/integrations/testmu) and [TestMu AI browsers](/integrations/testmu-browsers) | | `E2E_AGENT_DEVICE_POOL__` | value | Written by `@e2e-dev/mobile` in `prepare` and read by each worker to find its booted device. Not meant to be set by hand | ## What app processes inherit diff --git a/docs/security.mdx b/docs/security.mdx index 4cf0a0d35..8149fce6e 100644 --- a/docs/security.mdx +++ b/docs/security.mdx @@ -289,6 +289,9 @@ Built-in features may connect to: - the GitHub API, only from the opt-in `@e2e-dev/github` reporter - the Kernel API and its hosted browsers, only from [`kernel()`](/integrations/kernel) +- TestMu AI's CDP hub and its hosted browsers, only from + [`testmuBrowsers()`](/integrations/testmu-browsers). The username and + access key travel only in the CDP URL, never in a log line - the Expo API and the simulators' agent-device daemons, only from [`easSimulators()`](/integrations/eas). It authenticates with `EXPO_TOKEN`, else with the eas-cli login in `~/.expo/state.json`: on a CI runner, set diff --git a/packages/testmu/README.md b/packages/testmu/README.md index 78345c534..47d96e820 100644 --- a/packages/testmu/README.md +++ b/packages/testmu/README.md @@ -75,7 +75,45 @@ whole session instead, found by the session's build and name and starting at the session's start time. Runs that overlap and share a fixed `build` and `sessionName` can link each other's videos; the defaults never do. -Full documentation lives at [e2e.tester.army/docs/integrations/testmu](https://e2e.tester.army/docs/integrations/testmu). +## Browsers + +`web({ browser: testmuBrowsers() })` runs a web target in TestMu AI's hosted +Chrome or Edge on Windows and macOS, one session per worker slot, or one per +attempt with `scope: 'attempt'`. It is exported from `@e2e-dev/testmu/web`, +which needs only `@e2e-dev/web`; the package root exports the device +provider. + +```ts title="e2e.config.ts" +import type { E2EConfig } from 'e2e'; +import { web } from '@e2e-dev/web'; +import { testmuBrowsers } from '@e2e-dev/testmu/web'; + +export default { + targets: [ + { + name: 'testmu-web', + engine: web({ browser: testmuBrowsers({ platform: 'Windows 11' }), viewport: null }), + app: { url: 'https://staging.example.com' }, + }, + ], +} satisfies E2EConfig; +``` + +A session starts when its CDP websocket opens and ends when the engine +closes it, so the provider only builds the CDP URL from the options and +`LT_USERNAME`/`LT_ACCESS_KEY`, and calls no API. + +## Credentials + +Both providers read `LT_USERNAME` and `LT_ACCESS_KEY` from the run's +environment. To get them, [sign up for TestMu AI](https://accounts.lambdatest.com/register) +or log in to your account, then copy your username and access key from +**Account Settings → Password & Security → Username and Access Key** +([accounts.lambdatest.com/security/username-accesskey](https://accounts.lambdatest.com/security/username-accesskey)). + +Full documentation lives at [e2e.tester.army/docs/integrations/testmu](https://e2e.tester.army/docs/integrations/testmu) +(devices) and [e2e.tester.army/docs/integrations/testmu-browsers](https://e2e.tester.army/docs/integrations/testmu-browsers) +(browsers). ## License diff --git a/packages/testmu/package.json b/packages/testmu/package.json index 7fe24c378..debb63e89 100644 --- a/packages/testmu/package.json +++ b/packages/testmu/package.json @@ -1,7 +1,7 @@ { "name": "@e2e-dev/testmu", "version": "0.0.0", - "description": "TestMu AI (formerly LambdaTest) for e2e: run mobile targets on hosted Android emulators, iOS simulators, and real devices", + "description": "TestMu AI (formerly LambdaTest) for e2e: run mobile targets on hosted Android emulators, iOS simulators, and real devices, and web targets on hosted Chrome and Edge", "keywords": [ "e2e", "end-to-end", @@ -9,6 +9,9 @@ "mobile-testing", "testmu", "lambdatest", + "browser-testing", + "chrome", + "cdp", "device-cloud", "real-devices", "ios-simulator", @@ -40,6 +43,10 @@ ".": { "types": "./dist/index.d.ts", "default": "./dist/index.js" + }, + "./web": { + "types": "./dist/web.d.ts", + "default": "./dist/web.js" } }, "files": [ @@ -56,11 +63,24 @@ }, "peerDependencies": { "@e2e-dev/mobile": ">=0.9.0 <1", + "@e2e-dev/web": ">=0.11.0 <1", "agent-device": ">0.21.20 <1", "e2e": ">=0.15.0 <1" }, + "peerDependenciesMeta": { + "@e2e-dev/mobile": { + "optional": true + }, + "@e2e-dev/web": { + "optional": true + }, + "agent-device": { + "optional": true + } + }, "devDependencies": { "@e2e-dev/mobile": "workspace:*", + "@e2e-dev/web": "workspace:*", "@types/node": "26.6.2", "agent-device": "0.21.20", "e2e": "workspace:*", diff --git a/packages/testmu/src/browsers.ts b/packages/testmu/src/browsers.ts new file mode 100644 index 000000000..d8d8bcf69 --- /dev/null +++ b/packages/testmu/src/browsers.ts @@ -0,0 +1,189 @@ +/** TestMu AI's hosted Chrome and Edge as a `BrowserProvider` for the web engine. */ + +import type { BrowserLease, BrowserProvider, BrowserProviderScope, BrowserRequest } from '@e2e-dev/web'; +import { ConfigurationError, rejectUnknownKeys } from 'e2e/engine'; +import { testmuCredentials } from './credentials.ts'; + +/** + * The hub route that serves a raw CDP endpoint. `/puppeteer` serves it on + * every machine. `/playwright-cdp` labels sessions as Playwright, but serves + * raw CDP only on machines that run TestMu AI's newest backend, which is + * still rolling out; elsewhere the session fails to connect. + */ +export type TestmuBrowsersRoute = '/puppeteer' | '/playwright-cdp'; + +/** The browsers the web engine can attach to over CDP: Chromium ones. */ +export type TestmuBrowserName = 'Chrome' | 'MicrosoftEdge'; + +export interface TestmuBrowsersOptions { + /** + * `worker` (default): one TestMu AI session per worker slot for the run. + * `attempt`: a fresh session per test attempt, so each test is its own + * TestMu AI session; rules out `headers`, `basicAuth`, and `userAgent`. + */ + readonly scope?: BrowserProviderScope | undefined; + /** CDP route on the hub; `/puppeteer` when absent. `/playwright-cdp` is still rolling out (see `TestmuBrowsersRoute`). */ + readonly route?: TestmuBrowsersRoute | undefined; + /** The hub host, without a scheme or path; `cdp.lambdatest.com` when absent. */ + readonly hub?: string | undefined; + /** `Chrome` when absent. */ + readonly browserName?: TestmuBrowserName | undefined; + /** `latest` when absent. */ + readonly browserVersion?: string | undefined; + /** TestMu AI platform name, such as `Windows 11` (default) or `macOS Sequoia`. */ + readonly platform?: string | undefined; + /** Sent as `LT:Options.project`, as `testmu()` sends its own. Defaults to `e2e`. */ + readonly project?: string | undefined; + /** Dashboard build the sessions are grouped under. Defaults to the run id, as `testmu()`'s, so a run's device and browser sessions share one build. */ + readonly build?: string | undefined; + /** + * Name of every session on the dashboard, with `-` (worker scope, + * more than one slot) or `-` (attempt scope) appended so each + * session has its own. Defaults to `e2e---`. Slots count from 1. + */ + readonly sessionName?: string | undefined; + /** Country the browser's IP geolocates to, as a code TestMu AI takes: `US`, `FR`. */ + readonly geoLocation?: string | undefined; + /** The machine's time zone, as TestMu AI takes it: `UTC+05:30`. */ + readonly timezone?: string | undefined; + /** + * Further `LT:Options` capabilities (`video`, `network`, `console`, + * `idleTimeout`, `tunnel`, ...). `idleTimeout` is 600 seconds when absent. + * `user`, `accessKey`, `build`, `name`, `project`, `platform`, + * `geoLocation`, `timezone`, `browserName`, `browserVersion`, and a nested + * `LT:Options` are not accepted here: the provider sets them, from the + * environment and the options above. + */ + readonly capabilities?: Readonly> | undefined; +} + +const OPTION_KEYS: readonly string[] = Object.keys({ + scope: true, + route: true, + hub: true, + browserName: true, + browserVersion: true, + platform: true, + project: true, + build: true, + sessionName: true, + geoLocation: true, + timezone: true, + capabilities: true, +} satisfies Record); + +const SCOPES: readonly string[] = ['worker', 'attempt'] satisfies BrowserProviderScope[]; +const ROUTES: readonly string[] = ['/puppeteer', '/playwright-cdp'] satisfies TestmuBrowsersRoute[]; +const BROWSERS: readonly string[] = ['Chrome', 'MicrosoftEdge'] satisfies TestmuBrowserName[]; +const PROVIDER_CAPABILITIES = ['user', 'accessKey', 'build', 'name', 'project', 'platform', 'geoLocation', 'timezone', 'browserName', 'browserVersion', 'LT:Options']; + +/** The options that are strings when given, checked at config load as `testmu()` checks its own. */ +const OPTIONAL_STRING_KEYS = ['browserVersion', 'platform', 'project', 'build', 'sessionName', 'geoLocation', 'timezone'] as const; + +/** The `LT:Options.project` sent when `project` is absent, as `testmu()`'s default. */ +const DEFAULT_PROJECT = 'e2e'; +const HOST = /^[A-Za-z0-9.-]+(:\d+)?$/; + +const DEFAULT_HUB = 'cdp.lambdatest.com'; +const DEFAULT_ROUTE: TestmuBrowsersRoute = '/puppeteer'; +// The hub ends a CDP session after 300 s without client traffic, shorter than a slow test's gaps. +const DEFAULT_IDLE_TIMEOUT_SECONDS = 600; + +/** + * TestMu AI browsers for `web({ browser: testmuBrowsers() })`. A TestMu AI + * session is its websocket: connecting to the CDP URL starts it and closing + * the connection ends it, so `acquire` only builds the URL and `release` has + * nothing to call. Every session is named after the target and the slot or + * attempt, inside one build per run. `LT_USERNAME` and `LT_ACCESS_KEY` come + * from the run's environment and never appear in a log line. + */ +export function testmuBrowsers(options: TestmuBrowsersOptions = {}): BrowserProvider { + if (!isRecord(options)) { + throw new ConfigurationError('INVALID_CONFIG', 'testmuBrowsers() options must be an object'); + } + rejectUnknownKeys('testmuBrowsers()', options, OPTION_KEYS); + // Only an absent value takes the default: `null` from a JavaScript config is refused like any other. + const { scope } = options; + const route = options.route === undefined ? DEFAULT_ROUTE : options.route; + const hub = options.hub === undefined ? DEFAULT_HUB : options.hub; + const browserName = options.browserName === undefined ? 'Chrome' : options.browserName; + if (scope !== undefined && !SCOPES.includes(scope)) { + throw new ConfigurationError('INVALID_CONFIG', `testmuBrowsers: \`scope\` must be one of ${SCOPES.join(', ')}, got ${JSON.stringify(scope)}`); + } + if (!ROUTES.includes(route)) { + throw new ConfigurationError('INVALID_CONFIG', `testmuBrowsers: \`route\` must be one of ${ROUTES.join(', ')}, got "${route}"`); + } + if (typeof hub !== 'string' || !HOST.test(hub)) { + throw new ConfigurationError('INVALID_CONFIG', `testmuBrowsers: \`hub\` must be a host such as "${DEFAULT_HUB}", without a scheme or path, got "${hub}"`); + } + if (!BROWSERS.includes(browserName)) { + throw new ConfigurationError('INVALID_CONFIG', `testmuBrowsers: \`browserName\` must be one of ${BROWSERS.join(', ')}, got "${browserName}"`); + } + for (const key of OPTIONAL_STRING_KEYS) { + const value: unknown = options[key]; + if (value !== undefined && (typeof value !== 'string' || value.trim() === '')) { + throw new ConfigurationError('INVALID_CONFIG', `testmuBrowsers: \`${key}\` must be a non-empty string`); + } + } + if (options.capabilities !== undefined && !isRecord(options.capabilities)) { + throw new ConfigurationError('INVALID_CONFIG', 'testmuBrowsers: `capabilities` must be an object of `LT:Options` fields'); + } + const reserved = PROVIDER_CAPABILITIES.filter((key) => options.capabilities !== undefined && key in options.capabilities); + if (reserved.length > 0) { + throw new ConfigurationError( + 'INVALID_CONFIG', + `testmuBrowsers: \`capabilities\` cannot set ${reserved.map((key) => `\`${key}\``).join(', ')}; use the options of the same name, and LT_USERNAME/LT_ACCESS_KEY for credentials`, + ); + } + return { + name: 'testmu-browsers', + ...(scope === undefined ? {} : { scope }), + async acquire(request: BrowserRequest): Promise { + const { username, accessKey } = testmuCredentials(request.env); + const label = request.attemptId ?? `slot ${request.slot + 1} of ${request.slots}`; + const build = options.build ?? request.runId; + const name = sessionNameFor(options.sessionName, request); + const capabilities = { + browserName, + browserVersion: options.browserVersion ?? 'latest', + 'LT:Options': { + idleTimeout: DEFAULT_IDLE_TIMEOUT_SECONDS, + ...options.capabilities, + platform: options.platform ?? 'Windows 11', + project: options.project ?? DEFAULT_PROJECT, + build, + name, + ...(options.geoLocation === undefined ? {} : { geoLocation: options.geoLocation }), + ...(options.timezone === undefined ? {} : { timezone: options.timezone }), + user: username, + accessKey, + }, + }; + request.log(`TestMu AI session "${name}" in build "${build}"`); + return { + // Worker-scope leases share one bounded environment variable, so the id stays short. + id: `${request.targetName}:${label}`, + cdpEndpoint: `wss://${hub}${route}?capabilities=${encodeURIComponent(JSON.stringify(capabilities))}`, + }; + }, + async release(): Promise { + // Closing the CDP connection, which the engine does, ends the TestMu AI session. + }, + }; +} + +/** + * The dashboard name of a session, unique among the run's sessions of the + * target, in `testmu()`'s format: `e2e---` per worker + * slot, or `-` in attempt scope. + */ +function sessionNameFor(sessionName: string | undefined, { runId, targetName, slot, slots, attemptId }: BrowserRequest): string { + const suffix = attemptId ?? String(slot + 1); + if (sessionName === undefined) return `e2e-${runId}-${targetName}-${suffix}`; + return attemptId !== undefined || slots > 1 ? `${sessionName}-${suffix}` : sessionName; +} + +/** True for a plain object. Config runs as JavaScript, so the types alone are no guard. */ +function isRecord(value: unknown): value is object { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} diff --git a/packages/testmu/src/index.ts b/packages/testmu/src/index.ts index 65b060dfa..cdce6162a 100644 --- a/packages/testmu/src/index.ts +++ b/packages/testmu/src/index.ts @@ -2,6 +2,8 @@ * `@e2e-dev/testmu` public surface: `testmu()`, a device provider that leases * TestMu AI (formerly LambdaTest) Android emulators, iOS simulators, and real * devices for `@e2e-dev/mobile` through agent-device's `testmu` provider. + * `testmuBrowsers()`, for TestMu AI's hosted Chrome and Edge on `@e2e-dev/web`, + * is exported from `@e2e-dev/testmu/web`. */ export { testmu } from './provider.ts'; diff --git a/packages/testmu/src/web.ts b/packages/testmu/src/web.ts new file mode 100644 index 000000000..07ae2ccca --- /dev/null +++ b/packages/testmu/src/web.ts @@ -0,0 +1,8 @@ +/** + * `@e2e-dev/testmu/web`: `testmuBrowsers()`, for web targets. It loads neither + * `@e2e-dev/mobile` nor agent-device, and the package root, which exports the + * device provider, does not reference `@e2e-dev/web`. + */ + +export { testmuBrowsers } from './browsers.ts'; +export type { TestmuBrowserName, TestmuBrowsersOptions, TestmuBrowsersRoute } from './browsers.ts'; diff --git a/packages/testmu/tests/unit/browsers.test.ts b/packages/testmu/tests/unit/browsers.test.ts new file mode 100644 index 000000000..b6a3e52ef --- /dev/null +++ b/packages/testmu/tests/unit/browsers.test.ts @@ -0,0 +1,212 @@ +/** + * `testmuBrowsers()` builds the CDP URL a TestMu AI session starts on: the + * capabilities it encodes, credentials read from the run's environment + * only, session and build names, the route and hub options, scope, option + * validation, log lines that never carry the access key, and a release that + * calls nothing. + */ + +import type { BrowserReleaseContext, BrowserRequest } from '@e2e-dev/web'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { testmuBrowsers, type TestmuBrowsersOptions } from '../../src/web.ts'; + +const ACCESS_KEY = 'LT_secret-access-key'; + +interface Capabilities { + browserName: string; + browserVersion: string; + 'LT:Options': Record; +} + +function request(overrides: Partial = {}): BrowserRequest & { lines: string[] } { + const lines: string[] = []; + return { + runId: 'run-1', + targetName: 'chromium', + slot: 0, + slots: 2, + env: { LT_USERNAME: 'alice', LT_ACCESS_KEY: ACCESS_KEY }, + signal: new AbortController().signal, + log: (line: string) => lines.push(line), + lines, + ...overrides, + }; +} + +function decode(cdpEndpoint: string): { url: URL; capabilities: Capabilities } { + const url = new URL(cdpEndpoint); + return { url, capabilities: JSON.parse(url.searchParams.get('capabilities') ?? '{}') as Capabilities }; +} + +beforeEach(() => { + vi.stubGlobal('fetch', () => { + throw new Error('testmuBrowsers() must not call the network'); + }); +}); + +afterEach(() => { + vi.unstubAllGlobals(); + vi.unstubAllEnvs(); +}); + +describe('testmuBrowsers()', () => { + it('builds a /puppeteer CDP URL on cdp.lambdatest.com with the run and target in the names', async () => { + const lease = await testmuBrowsers().acquire(request()); + const { url, capabilities } = decode(lease.cdpEndpoint); + + expect(url.protocol).toBe('wss:'); + expect(url.host).toBe('cdp.lambdatest.com'); + expect(url.pathname).toBe('/puppeteer'); + expect(capabilities).toEqual({ + browserName: 'Chrome', + browserVersion: 'latest', + 'LT:Options': { + idleTimeout: 600, + platform: 'Windows 11', + project: 'e2e', + build: 'run-1', + name: 'e2e-run-1-chromium-1', + user: 'alice', + accessKey: ACCESS_KEY, + }, + }); + expect(lease.id).toBe('chromium:slot 1 of 2'); + }); + + it('names a per-attempt session after the attempt', async () => { + const provider = testmuBrowsers({ scope: 'attempt' }); + const lease = await provider.acquire(request({ attemptId: 'attempt-7' })); + + expect(provider.scope).toBe('attempt'); + expect(decode(lease.cdpEndpoint).capabilities['LT:Options']['name']).toBe('e2e-run-1-chromium-attempt-7'); + expect(lease.id).toBe('chromium:attempt-7'); + }); + + it('leaves scope to the engine default when none is given', () => { + expect('scope' in testmuBrowsers()).toBe(false); + }); + + it('honours the route, hub, browser, platform, build, and extra capabilities', async () => { + const lease = await testmuBrowsers({ + route: '/playwright-cdp', + hub: 'cdp.eu.example.test', + browserName: 'MicrosoftEdge', + browserVersion: '140', + platform: 'macOS Sequoia', + build: 'nightly', + project: 'checkout', + geoLocation: 'FR', + timezone: 'UTC+01:00', + capabilities: { video: true, idleTimeout: 300 }, + }).acquire(request()); + const { url, capabilities } = decode(lease.cdpEndpoint); + + expect(url.host).toBe('cdp.eu.example.test'); + expect(url.pathname).toBe('/playwright-cdp'); + expect(capabilities.browserName).toBe('MicrosoftEdge'); + expect(capabilities.browserVersion).toBe('140'); + expect(capabilities['LT:Options']).toMatchObject({ + platform: 'macOS Sequoia', + project: 'checkout', + build: 'nightly', + geoLocation: 'FR', + timezone: 'UTC+01:00', + video: true, + idleTimeout: 300, + }); + }); + + it('names sessions like testmu() does: a given sessionName gets the slot or attempt appended when needed', async () => { + const nameOf = async (options: TestmuBrowsersOptions, overrides: Partial = {}) => + decode((await testmuBrowsers(options).acquire(request(overrides))).cdpEndpoint).capabilities['LT:Options']['name']; + + expect(await nameOf({ sessionName: 'smoke' }, { slots: 1 })).toBe('smoke'); + expect(await nameOf({ sessionName: 'smoke' }, { slot: 1, slots: 2 })).toBe('smoke-2'); + expect(await nameOf({ sessionName: 'smoke', scope: 'attempt' }, { attemptId: 'attempt-3' })).toBe('smoke-attempt-3'); + expect(await nameOf({}, { slot: 1, slots: 2 })).toBe('e2e-run-1-chromium-2'); + }); + + it('takes the credentials from the run environment, not process.env', async () => { + vi.stubEnv('LT_USERNAME', 'from-process-env'); + vi.stubEnv('LT_ACCESS_KEY', 'from-process-env'); + const options = decode((await testmuBrowsers().acquire(request())).cdpEndpoint).capabilities['LT:Options']; + + expect(options['user']).toBe('alice'); + expect(options['accessKey']).toBe(ACCESS_KEY); + }); + + it('fails the lease when a credential is missing or blank', async () => { + await expect(testmuBrowsers().acquire(request({ env: { LT_USERNAME: 'alice' } }))).rejects.toThrow('LT_ACCESS_KEY is not set'); + await expect(testmuBrowsers().acquire(request({ env: { LT_USERNAME: ' ', LT_ACCESS_KEY: ACCESS_KEY } }))).rejects.toThrow('LT_USERNAME is not set'); + }); + + it.each([ + [{ platfrom: 'macOS Sequoia' }, 'testmuBrowsers() has unknown key "platfrom"; did you mean "platform"?'], + [{ route: 'playwright-cdp' }, '`route` must be one of /puppeteer, /playwright-cdp'], + [{ hub: 'https://cdp.lambdatest.com' }, '`hub` must be a host'], + [{ hub: 'cdp.lambdatest.com/puppeteer' }, '`hub` must be a host'], + [{ browserName: 'Firefox' }, '`browserName` must be one of Chrome, MicrosoftEdge'], + [{ capabilities: { user: 'mallory', accessKey: 'spoofed' } }, '`capabilities` cannot set `user`, `accessKey`'], + [{ capabilities: { build: 'nightly', platform: 'Windows 10' } }, '`capabilities` cannot set `build`, `platform`'], + [{ capabilities: { browserName: 'Firefox', browserVersion: '120' } }, '`capabilities` cannot set `browserName`, `browserVersion`'], + [{ capabilities: { 'LT:Options': { video: true } } }, '`capabilities` cannot set `LT:Options`'], + [{ hub: null }, '`hub` must be a host'], + [{ route: null }, '`route` must be one of'], + [{ browserName: null }, '`browserName` must be one of'], + [{ scope: 'session' }, '`scope` must be one of worker, attempt'], + [{ scope: null }, '`scope` must be one of worker, attempt'], + [{ build: '' }, '`build` must be a non-empty string'], + [{ project: 1 }, '`project` must be a non-empty string'], + [{ sessionName: null }, '`sessionName` must be a non-empty string'], + [{ browserVersion: ' ' }, '`browserVersion` must be a non-empty string'], + [{ capabilities: { project: 'x', geoLocation: 'US' } }, '`capabilities` cannot set `project`, `geoLocation`'], + [null, 'testmuBrowsers() options must be an object'], + [[], 'testmuBrowsers() options must be an object'], + [{ capabilities: null }, '`capabilities` must be an object'], + [{ capabilities: ['video'] }, '`capabilities` must be an object'], + [{ capabilities: 'video' }, '`capabilities` must be an object'], + ])('rejects %j when the provider is created', (options, message) => { + expect(() => testmuBrowsers(options as unknown as TestmuBrowsersOptions)).toThrow( + expect.objectContaining({ code: 'INVALID_CONFIG', message: expect.stringContaining(message) }), + ); + }); + + it('keeps 24 worker-scope leases inside the engine\'s 16 KB hand-off', async () => { + const provider = testmuBrowsers({ capabilities: { video: true, network: true, console: true, tunnel: true, tunnelName: 'ci-tunnel' } }); + const runId = '01a0fcab-e8da-797a-8b35-2168b81385c0'; + const env = { LT_USERNAME: 'some.user.name', LT_ACCESS_KEY: `LT_${'x'.repeat(46)}` }; + const leases = await Promise.all( + Array.from({ length: 24 }, (_, slot) => provider.acquire(request({ runId, targetName: 'checkout-web', slot, slots: 24, env }))), + ); + + expect(Buffer.byteLength(JSON.stringify({ slots: 24, leases }))).toBeLessThan(16 * 1024); + }); + + it('logs the session and build names but never the access key or the URL', async () => { + const req = request(); + await testmuBrowsers().acquire(req); + + expect(req.lines).toEqual(['TestMu AI session "e2e-run-1-chromium-1" in build "run-1"']); + expect(req.lines.join('\n')).not.toContain(ACCESS_KEY); + expect(req.lines.join('\n')).not.toContain('wss://'); + }); + + it('is exported from @e2e-dev/testmu/web only: the package root stays device-only', async () => { + const root: Record = await import('../../src/index.ts'); + expect(Object.keys(root)).toEqual(['testmu']); + }); + + it('releases without calling anything', async () => { + const provider = testmuBrowsers(); + const lease = await provider.acquire(request()); + const context: BrowserReleaseContext = { + runId: 'run-1', + targetName: 'chromium', + env: {}, + signal: new AbortController().signal, + log: () => {}, + }; + + await expect(provider.release(lease, context)).resolves.toBeUndefined(); + }); +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index b9f2d6fc1..546489bd2 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -533,6 +533,9 @@ importers: '@e2e-dev/mobile': specifier: workspace:* version: link:../mobile + '@e2e-dev/web': + specifier: workspace:* + version: link:../web '@types/node': specifier: 26.6.2 version: 26.6.2 diff --git a/scripts/check-docs-examples.ts b/scripts/check-docs-examples.ts index aed7a0aad..41a89cc9e 100644 --- a/scripts/check-docs-examples.ts +++ b/scripts/check-docs-examples.ts @@ -30,6 +30,7 @@ const EXAMPLES: Record = { 'docs/examples/mobile/testmu.config.ts': 'docs/integrations/testmu.mdx', 'docs/examples/mobile/testmu.e2e.ts': 'docs/integrations/testmu.mdx', 'docs/examples/web/browser-provider.ts': 'docs/browser.mdx', + 'docs/examples/web/testmu-browsers.config.ts': 'docs/integrations/testmu-browsers.mdx', 'docs/examples/models/e2e.decision.config.ts': 'docs/decision-models.mdx', 'docs/examples/models/tests/decision.e2e.ts': 'docs/decision-models.mdx', 'docs/examples/skill/e2e.config.ts': 'skills/e2e/SKILL.md', diff --git a/skills/e2e/references/setup.md b/skills/e2e/references/setup.md index 88a4bbbc9..aa161b7f5 100644 --- a/skills/e2e/references/setup.md +++ b/skills/e2e/references/setup.md @@ -180,7 +180,7 @@ start a script that brings them up and serves the app. | Option | Meaning | | --- | --- | -| `browser` | `'chromium'` (default), `'firefox'`, `'webkit'`, or a `BrowserProvider` leasing hosted browsers over CDP (`kernel()` from `@e2e-dev/kernel`, or your own), which implies chromium and excludes `connect`. Scope `'worker'` (default): one browser per worker slot from `prepare` to `finish`; `'attempt'`: one per attempt, with `reconnectEndpoint`'s limits. | +| `browser` | `'chromium'` (default), `'firefox'`, `'webkit'`, or a `BrowserProvider` leasing hosted browsers over CDP (`kernel()` from `@e2e-dev/kernel`, `testmuBrowsers()` from `@e2e-dev/testmu/web`, or your own), which implies chromium and excludes `connect`. Scope `'worker'` (default): one browser per worker slot from `prepare` to `finish`; `'attempt'`: one per attempt, with `reconnectEndpoint`'s limits. | | `viewport` | `{ width, height }`, default 1280x720; `null` follows the browser window (hosted live view, headed run). On a headed hosted browser (Kernel) use `null` and size the service's screen; a fixed size gives a smaller, unmaximized window. | | `connect` | `{ cdpEndpoint }` attaches to a remote Chromium over CDP; both it and `reconnectEndpoint` are resolvers `(signal) => url`, not strings. With `reconnectEndpoint` it rides one persistent default context and reconnects only to the original browser and page. | | `headers` | Sent to the app's site only (Vercel's `x-vercel-protection-bypass`, ngrok's `ngrok-skip-browser-warning`), `agent.act` included; disables the browser HTTP cache and service workers. |