diff --git a/skills/uipath-test/SKILL.md b/skills/uipath-test/SKILL.md index 9330dabd3f..3d492e5759 100644 --- a/skills/uipath-test/SKILL.md +++ b/skills/uipath-test/SKILL.md @@ -1,6 +1,6 @@ --- name: uipath-test -description: "UiPath Test Manager — manage test projects, cases, sets, executions; generate reports. For Orchestrator→uipath-platform. For test automation→uipath-rpa." +description: "UiPath Test Manager — manage test projects, cases, sets, executions; generate reports; package and run external Playwright test suites. For Orchestrator→uipath-platform. For Studio/RPA test automation authoring→uipath-rpa." allowed-tools: Bash, Read, Write, Glob, Grep user-invocable: true --- @@ -29,6 +29,7 @@ UiPath Test Manager is a web application that manages the testing lifecycle of p - **Test case logs** - Logs of a **test case** in an execution. A **testcase** can be navigated from **testcaselogs**. - **Test step logs** — Step-level logs within a **test case log**. - **Test case log assertions** - Assertion steps of a test case log in an execution. +- **External (Playwright) test packages** - A Playwright suite packaged as an external test package and uploaded to Orchestrator. Ingestion auto-creates one test case per Playwright test (no link step) and labels each with `PW_Tag_*`, `PW_Project_*`, `PW_Suite_*`, `PW_File_*`. These run on serverless cloud runtimes — see [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). CLI tool for UiPath Test Manager (`uip tm`). Use `uip tm --help` and `uip tm --help` to discover commands and options. **Always pass `--output json`** on every `uip` command. @@ -85,7 +86,7 @@ Common `uip tm` commands organized by resource type. | `uip tm testcases steps delete --project-key --step-id --yes` | Delete a step. | | `uip tm testcases list-result-history --project-key --test-case-id ` | List test case log result history for a specific test case. Optional `--only-failed`, `--filter`, `--limit`, `--offset`. | | `uip tm testcases run --project-key --test-case-id --name --execution-type ` | Start a new execution for one or more test cases. **Uses `--test-case-id ` (space-separated for multiple).** Optional `--async`, `--folder-key`, `--robot-user-key`, `--machine-key`. | -| `uip tm testcases add --test-set-key --test-case-keys ` | Add test cases to a test set (comma-separated keys). | +| `uip tm testcases add --test-set-key (--test-case-keys … \| --labels …)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels. Both selectors are variadic and **space-separated**: `--test-case-keys DEMO:1 DEMO:2`, `--labels PW_Tag_smoke "PW_Suite_Checkout flow"` (quote names containing spaces). Keys additionally accept the comma form (`DEMO:1,DEMO:2`); **labels do not** — `--labels A,B` is read as one label named `A,B` and matches nothing. Label matching is OR, exact and case-sensitive. The two selectors are mutually exclusive. | | `uip tm testcases remove --test-set-key --test-case-keys ` | Remove test cases from a test set (comma-separated keys). | > **Flag shapes for test case and step identifiers — do not interchange:** @@ -103,10 +104,13 @@ Common `uip tm` commands organized by resource type. | `uip tm testsets update --test-set-key --name ` | Update a test set name or description. | | `uip tm testsets delete --test-set-key ` | Delete a test set by its key. | | `uip tm testsets list-testcases --project-key --test-set-key ` | List test cases assigned to a test set. | -| `uip tm testsets run --test-set-key ` | Run a test set and return the execution ID. Optional `--execution-type ` (default `automated`), `--input-path ` for parameter overrides. | +| `uip tm testsets run --test-set-key ` | Run a test set and return the execution ID. Optional `--execution-type ` (default `automated`), `--input-path ` for parameter overrides. For Playwright test sets, optional `--playwright-projects ` — see the note below. | +| `uip tm testsets playwright-context --test-set-key ` | Probe whether a test set is a Playwright test set: returns `IsPlaywright` plus the available and selected Playwright project names. | > Keys use the format `PROJECT_KEY:NUMBER` (e.g., `INV:42`). To add or remove test cases in a test set, use `uip tm testcases add` / `uip tm testcases remove` — those verbs live under the `testcases` group, not under `testsets`. +> **Playwright test sets:** `--playwright-projects ` (space-separated, case-sensitive `playwright.config` project names) runs only the selected projects and persists the selection on the test set. It requires every test case in the set to come from one Playwright package; unknown names fail fast listing the valid ones. Probe first with `playwright-context` and branch on `IsPlaywright`. Both need a Test Manager with Playwright support and a CLI carrying the external-package commands — [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md) opens with the check to run and what to do when they are absent. + ### Executions Commands | Command | Purpose | @@ -141,7 +145,7 @@ Common `uip tm` commands organized by resource type. | Command | Purpose | |---|---| -| `uip tm report get --execution-id ` | Get a summary report for a completed test execution. Optional `--project-key`, `--test-set-key`, `--query`. | +| `uip tm report get --execution-id (--project-key \| --test-set-key )` | Get a summary report for a completed test execution. One of `--project-key`/`--test-set-key` is required to identify the project (verified: passing only `--execution-id` exits with "Provide --project-key or --test-set-key"). | ### Attachment Commands @@ -156,6 +160,14 @@ Common `uip tm` commands organized by resource type. |---|---| | `uip tm result download --execution-id ` | Download test execution results as JUnit XML. Optional `--project-key`, `--test-set-key`, `--result-path `. | +### Pack Commands (Playwright) + +| Command | Purpose | +|---|---| +| `uip tm pack --project-path --type playwright --project-key --name --package-version -o ` | Pack a Playwright suite into a `.nupkg` external test package. Requires a lockfile and `@playwright/test` in the project. `--package-version` takes a NuGet/SemVer-style version — three numeric parts, optional prerelease suffix (`1.0.0`, `1.0.1-beta.1`); `1.0` or a non-numeric string is rejected. `--project-key` targets the Test Manager project where ingestion auto-creates the test cases; `--no-create-test-cases` skips that; `--dry-run` previews. Upload with `uip or packages upload `. | + +> Packing is offline — no auth needed. The upload → ingestion → label-fill → run pipeline is in [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). + ### Wait Commands | Command | Purpose | @@ -223,7 +235,7 @@ Object labels are tag-style metadata applied to Requirement, TestCase, TestSet, 4. **Cap retries at 3** for any failing `uip` CLI command. After 3 failures, stop and report the error to the user (see Rule — never fall back to direct REST APIs). 5. **Handle empty results** — if a list command returns an empty array, stop and inform the user rather than proceeding with a null key. Exception: a zero-result `tm testcases list --filter` call may be a prefix miss rather than a truly empty dataset — apply Rule 9's prefix fallback to disambiguate; stop only when the fallback also finds nothing. Filtered lookups on other resources (project, customfield) returning empty are real empty results — stop as usual. 6. **Confirm before delete** — always confirm the target resource key with the user before running any `delete` command. All delete commands require `--yes` (or `-y`) to proceed; omitting it exits without deleting. -7. **For operations requiring folder key** — use `uip or folders list -n --all --output json` (run `/uipath-platform` for folder management details). +7. **For operations requiring folder key** — use `uip or folders list -n --all --output json` when the user named a folder; when picking one yourself, list without `--all` so you only get folders you are a member of (run `/uipath-platform` for folder management details). 8. **Discover before assuming** — never guess automation names, folder keys, project IDs, or test case keys. Always run the matching `list` command first (e.g., `uip tm testcases list-automations`, `uip or folders list -n --all`). 9. **Narrow `list` calls server-side when the user names an entity.** When the user provides a name, key, label, or tag, check `uip tm list --help` (or `uip or list --help`) for the narrowing flag the command exposes and pass it on the `list` call. Never list all results and filter client-side — it wastes tokens and misses paginated entries. Applies to every entity across `uip tm` and `uip or`. Exception: `tm testcases list --filter` matches by prefix. A mid-name term returns zero — retry `--filter` with a broader name prefix from context; re-list without `--filter` only when no workable prefix exists, paging through ALL results (`--limit`/`--offset`) before concluding the target is absent. 10. **Set default folder before any `run` command** — `uip tm testcases run` and `uip tm testsets run` both require a default Orchestrator folder on the project. Run `uip tm project set-default-folder --project-key --folder-key --output json` first. Get folder keys with `uip or folders list -n --all --output json`. @@ -305,7 +317,8 @@ If the probe in Rule #2 shows singular subjects, the CLI predates the closed-ver | I want to... | Start here | |---|---| | **Generate a shareable test report** (tester or release manager view) | [references/test-result-report-guide.md](references/test-result-report-guide.md) | -| **Publish a project and link it to a Test Manager test case** | [references/publish-and-link-guide.md](references/publish-and-link-guide.md) | +| **Publish a project and link it to a Test Manager test case** (Studio/RPA) | [references/publish-and-link-guide.md](references/publish-and-link-guide.md) | +| **Pack, ingest, and run a Playwright suite on serverless** (pack → upload → labels → run) | [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md) | ## Anti-patterns @@ -313,3 +326,4 @@ If the probe in Rule #2 shows singular subjects, the CLI predates the closed-ver - **Do NOT proceed if authentication fails** — all Test Manager API calls require a valid bearer token. Fail fast rather than surfacing confusing 401 errors later. - **Do NOT skip the surface probe** (Critical Rule #2). On a pre-rename CLI, post-rename commands fail with `unknown command`; on a post-rename CLI, pre-rename commands fail the same way. The skill targets the post-rename surface and falls back per the [Pre-rename fallbacks](#pre-rename-fallbacks) table. Picking the wrong shape without probing burns a retry on every call. - **Do NOT guess command names — verb-noun composites are required.** The CLI uses explicit verb-noun forms; bare verbs do not exist. Confirm with `uip tm --help --output json`. +- **Do NOT `link-automation` Playwright test cases.** Playwright ingestion links them to the package automatically; the manual link step belongs to the Studio/RPA pipeline only. diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md new file mode 100644 index 0000000000..eb0e0aa8cb --- /dev/null +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -0,0 +1,208 @@ +# Playwright First Mile — Pack, Ingest, and Run on Serverless + +End-to-end pipeline: take a Playwright test suite from a repo to executed results in UiPath Test Manager, using only `uip` commands. This flow is **Playwright-specific** — the pack command, the auto-created test cases, the `PW_*` labels, and the per-project run selection below do NOT apply to UiPath Studio/RPA packages. For Studio test automation use [publish-and-link-guide.md](publish-and-link-guide.md) instead; never mix the two pipelines. + +## Pipeline + +``` +uip tm pack --type playwright → .nupkg with embedded test metadata +uip or packages upload → package on the Orchestrator feed +(Test Manager ingestion, automatic) → test cases auto-create, PW_* labels applied +uip tm testsets create → empty test set +uip tm testcases add --labels → fill it by label +uip tm testsets playwright-context → probe: is this a Playwright test set? which projects? +uip tm testsets run [--playwright-projects ] → execute on serverless +uip tm wait / report / result → outcome +``` + +The key difference from the RPA pipeline: there is **no link step**. Uploading the package is enough — ingestion creates one Test Manager test case per discovered Playwright test, already bound to the package, and labels each with: + +- `PW_Tag_` — one per Playwright tag (`@smoke` → `PW_Tag_smoke`) +- `PW_Project_` — one per Playwright project the test runs in +- `PW_Suite_` / `PW_Path_` — describe-block grouping (name and full chain) +- `PW_File_` — the spec file + +> **Do NOT run `uip tm testcases link-automation` on Playwright test cases.** They are linked by ingestion; manual linking is the RPA pipeline and will corrupt the association. + +> **Reading command output.** `--output json` prints a JSON envelope, but not on its own line: auto-updater chatter (including `Update completed with failures.`, which is unrelated to your command), `Resolved project …` progress lines and trailing telemetry warnings share the same stream. Judge a command by the `Result` field inside the envelope, never by surrounding text, and extract from the first `{` before parsing. + +> **If a command is missing.** Two commands in this pipeline are hidden from `--help`, so `--help` is not a reliable way to tell whether a build has them: `testsets playwright-context` (Step 5) and `run --playwright-projects` (Step 6). Older CLIs answer `unknown command` / `unknown option` for them. Treat them differently: +> +> - **Probe missing** → skip Step 5 and continue. The probe only *reports* whether a test set is Playwright; it does not enable anything, so losing it costs you the pre-check, not the capability. Project scoping still works. +> - **`--playwright-projects` rejected** → this build cannot scope a run to selected projects. Run the test set without the flag (every project in the package's config runs) rather than retrying. + +## Prerequisites + +- A Test Manager project to land the test cases in: `uip tm project list --filter --output json`, or `uip tm project create --name --project-key --output json`. Capture the project key. +- The tenant's Test Manager must have Playwright support enabled (a server-side feature flag). If ingestion never produces test cases (Step 3), this is the first thing to suspect — stop and ask the user. +- The Playwright project directory must contain: + - `package.json` with `@playwright/test` installed (discovery shells out to the project's own `playwright test --list`; no browsers needed), + - a **lockfile** (`package-lock.json` / `yarn.lock` / `pnpm-lock.yaml` / `bun.lock`) — serverless does a deterministic install, + - a `playwright.config` file. + +## Step 1 — Pack + +```bash +uip tm pack --project-path --type playwright \ + --project-key --name \ + --package-version 1.0.0 -o --output json +``` + +- `--project-key` is **required** by default because test-case auto-creation is on; pass `--no-create-test-cases` to pack without it (label metadata stays embedded for later use). +- Preview with `--dry-run` (writes nothing). +- Capture `Data.Output` (the `.nupkg` path) and `Data.TestCount` from the JSON output. `TestCount` is the number of test cases ingestion will create — remember it for Step 3. +- Pack fails loudly when the lockfile or `@playwright/test` is missing — fix the project, do not improvise around it. + +## Step 2 — Upload to Orchestrator + +```bash +uip or packages upload "/.1.0.0.nupkg" --output json +``` + +`--package-version` is a NuGet/SemVer-style version: three numeric parts with an optional prerelease suffix (`1.0.0`, `1.0.1-beta.1`) — `1.0` is rejected. Each upload needs a version the feed does not already have, so before re-packing check what is published and go above it: + +```bash +uip or packages list --search --output json +``` + +The match returns the feed's **latest** version only (one row, `IsLatestVersion: true`) — enough to pick the next version, but it is not a version history. + +## Step 3 — Wait for ingestion + +Ingestion is asynchronous and automatic. Poll until the auto-created test cases appear: + +```bash +uip tm testcases list --project-key --output json +``` + +- Poll **unfiltered**. Do NOT pass `--filter ` — `--filter` matches a test case's name or key by **prefix** (see SKILL.md Rule 9), and an ingested test case is named `" > "`, so a package name never matches: the call stays empty forever and reads as a false "ingestion never happened". +- The rows carry `TestCaseKey` (e.g. `SHIP:1`) plus `Id` (the UUID) and `Name` — count `TestCaseKey`, not `Id`, when polling. +- Ingestion is done when `TestCount` new test cases from Step 1 are present, each named `" > "`. A plain `pack` prints only `Package`, `Output` and `TestCount` — if you want the exact expected names up front, run `--dry-run` first or read `testCases.json` inside the `.nupkg`; otherwise match on the count plus that name shape. +- `TestCount` from Step 1 is one test case per Playwright **test** — it is not multiplied by the number of Playwright projects (2 tests × 2 projects → 2 test cases). +- Ingested test cases show `IsAutomated: false` in list output; that is normal and does not mean ingestion failed. +- Poll every ~10 seconds for up to ~3 minutes. If the expected names never appear, STOP and report — the likely causes are the Playwright feature flag being off for the tenant or a wrong `--project-key`, and neither is fixed by retrying. +- Spot-check the labels landed: `uip tm objectlabel list --project-key --object-type TestCase --filter PW_ --output json` (returns distinct label *names* only). + +## Step 4 — Set the default folder, create a test set, fill it by label + +Set the project's default Orchestrator folder FIRST — both the Step 5 probe and the Step 6 run resolve packages through it (Critical Rule #10): + +```bash +uip or folders list --output json # WITHOUT --all: only folders you are a member of +uip tm project set-default-folder --project-key --folder-key --output json +``` + +- Pick the folder from the **unflagged** `folders list` — `--all` returns every folder *visible* to you, including ones where you have no rights; choosing one of those fails later with `folderNotFoundOrNoAccess`. +The folder needs **two** things, and missing either one fails the run differently: + +- **A Cloud Robots – Serverless machine** — without it job creation 500s and the execution is instantly `Cancelled`. Check with `uip or machines list --folder-key --output json`; if none, create and assign one (one serverless machine per folder): + +```bash +uip or machines create -n --serverless --testing-slots 2 --output json # capture Data.Key +uip or machines assign --folder-key --output json # takes machine KEYS (GUIDs), not names +``` + +- **A folder member who can run unattended** — the run executes as a robot user drawn from the folder, and `uip tm testsets run` has no flag to choose one (only `uip tm testcases run` does), so folder membership is the only lever: + +```bash +uip or users list-in-folder --folder-key --output json +uip or users get --all-fields --output json # needs MayHaveUnattendedSession: true +``` + +Folder, machine and robot-user management is the platform skill's domain — see [/uipath:uipath-platform § orchestrator/setup-environment.md](../../uipath-platform/references/orchestrator/setup-environment.md) for the authoritative flags and the one-serverless-machine-per-folder rule. An ordinary interactive user usually has `MayHaveUnattendedSession: false`, and `uip or users update --allow-unattended` cannot fix that here — it requires a Windows unattended username and password, which serverless does not use. Assign an already-unattended-capable principal instead (a `DirectoryRobot` account is the reliable pick). Creating your own folder is a first-class option when no existing one qualifies: `uip or folders create --output json` (name is positional), then attach a machine and such a user. + +```bash +uip tm testsets create --project-key --name "PW Smoke" --output json +uip tm testcases add --test-set-key --labels "PW_File_" --output json +``` + +- Capture `TestSetKey` from the create output (e.g. `DEMO:10`). +- `--labels` is variadic and space-separated (quote names that contain spaces). Matching is **OR across labels, exact, and case-sensitive** — discover the real names first with `uip tm objectlabel list` rather than guessing. +- `--labels` works with any object label; the `PW_*` labels are simply what ingestion applies. +- Mutually exclusive with `--test-case-keys`; pass exactly one of the two. +- To run the whole suite on one browser: fill by a suite/file label (`PW_Suite_*` or `PW_File_*`) and pass the browser to `--playwright-projects` in Step 6. Filling by `PW_Project_` is for selecting the subset of tests that participate in that project — it does not restrict which browsers run. +- **Labels select *tests*; `--playwright-projects` selects *browsers*.** Filling by `PW_Project_firefox` picks every test that runs in the firefox project (often all of them); it does not make the run firefox-only — that is what the run flag in Step 6 does. To "run only ", label-fill by whatever identifies the tests you want (tag, suite, file) and pass the project name to `--playwright-projects`. +- **Keep one test set = one Playwright package.** Per-project selection (Step 6) requires every test case in the set to come from a single Playwright package; label-filling across packages produces a set that cannot be project-scoped. Labels are NOT package-qualified — in a project holding several Playwright packages, a generic label like `PW_Tag_smoke` matches tests from all of them. There, fill by a package-unique label (`PW_File_`, or a suite name unique to the package) or by explicit `--test-case-keys` from the current ingestion. + +## Step 5 — Probe the Playwright context (when available) + +Before deciding whether `--playwright-projects` applies, ask the server: + +```bash +uip tm testsets playwright-context --test-set-key --output json +``` + +Read the fields off the JSON response rather than relying on the names below staying current — this list describes today's shape, and `--output json` always carries whatever the API returns. + +- `Data.IsPlaywright: true` → the set resolves to one Playwright package; `AvailablePlaywrightProjects` holds the only valid `--playwright-projects` values, and `SelectedPlaywrightProjects` shows any selection already stored on the test set. Both are **comma-joined strings** (`"chromium, firefox"`), not arrays — split on `", "` when scripting; no stored selection is `""`. +- `Data.IsPlaywright: false` → RPA, mixed, manual, or multi-package test set — run it **without** `--playwright-projects`. +- The server never errors on type here, so this is the safe discriminator for automation: probe first, branch on `IsPlaywright`. +- **False negative without a folder:** the probe resolves the package through the project's default folder — if that isn't set (Step 4), a genuine Playwright test set reports `IsPlaywright: false`. Set the default folder before trusting a `false`. + +## Step 6 — Run, optionally per Playwright project + +```bash +uip tm testsets run --test-set-key \ + --playwright-projects chromium --output json +``` + +`--playwright-projects` does not appear in `uip tm testsets run --help` — functional but unlisted, so treat this guide as its reference rather than concluding the build lacks it. Semantics (all enforced with clear errors, nothing silently ignored): + +- Space-separated, case-sensitive names from the package's `playwright.config`. Passing several (`chromium firefox`) runs all of them, but results stay one log per test case — not per browser — so a per-browser breakdown is not available from Test Manager; scope to one project when you need attributable results. Unknown names **fail fast, before anything is persisted**, listing the available projects. +- Valid only when every test case in the set comes from one single Playwright package (see Step 4); fails for Studio/RPA test sets — run those without the flag. +- The selection **persists on the test set** and applies to later runs until changed; omit the flag to reuse the stored selection (or the config's defaults if none was ever stored). +- On a Test Manager without Playwright support the command fails with instructions rather than running incorrectly. + +Omit `--playwright-projects` entirely for a plain run (all config-default projects). + +**Getting the execution id.** Start the run **without** `--wait`: it returns a complete JSON envelope immediately, carrying `ExecutionId` and `Status: Pending` — the cleanest handle for automation. With `--wait` the envelope only arrives at terminal state, so take the id from the `Execution started: (Pending)` progress line — not from `Starting execution for test set …`, whose UUID is the *test set*. + +**Agent-friendly waiting:** a single `--wait` call can sit silent for many minutes, which trips agent-harness watchdogs and shell timeouts. When running as an agent, prefer starting the run **without** `--wait`, then poll in bounded chunks: `uip tm wait --execution-id --timeout 120 --output json` in a loop (or `uip tm executions get-stats` every 30–60 s), so every call returns quickly and progress stays visible. `wait` polls on a fixed 60 s interval, so keep `--timeout` a multiple of that — a shorter timeout can report a timeout for a run that already finished. A `wait` that hits its `--timeout` returns a Failure envelope with `Retry: "RetryWillNotFix"` — for a non-terminal execution that just means "still running"; keep polling, don't treat it as fatal. + +## Step 7 — Results + +- `--wait` on the run blocks until terminal; without it, use `uip tm wait --execution-id --output json`. +- Summary: `uip tm report get --execution-id --project-key --output json` (`--project-key` or `--test-set-key` is required — bare `--execution-id` exits with "Provide --project-key or --test-set-key"). +- Per-test detail: `uip tm executions testcaselogs list --execution-id --project-key --output json`. Judge pass/fail by `Result` (`Passed` / `Failed` / `None`) — a failed test still reports `HasError: false`, so that field is not the outcome. +- JUnit export: `uip tm result download --execution-id --result-path --output json` — counts only for Playwright: the XML names every case after the spec file, not the test title, so use `testcaselogs list` when you need to know *which* test failed. +- **Proving the run was scoped as asked:** `uip tm executions get-stats --execution-id --project-key --output json` returns a `PlaywrightExecutionSnapshot` whose `Projects` is the project list the run actually used (`["chromium"]`). Read only `Projects` from it — its `Version` is two-component (`1.0`), like `list-automations` and `playwright-context`, so none of the three can tell 1.0.1 from 1.0.0; `TestCaseVersion` on the logs is the only field that can. Counting logs proves nothing here — there is one test case log per Playwright **test**, not per test × project, so a two-project run yields the same number of logs as a one-project run. + +Execution happens on UiPath serverless cloud runtimes — no robot or package deployment into the folder is needed beyond the upload in Step 2, but the folder does need its serverless machine assignment (Step 4). + +### When a run produces no results + +Three shapes, one rule: diagnose with the two commands below, then stop — retrying doesn't fix any of them. + +- **`Cancelled` within seconds**, logs pointing at `CreateTestAutomationJobs` / `InternalServerError` → the default folder has no serverless machine assigned (Step 4). +- **`Finished` with `Passed: 0 / Failed: 0 / None: N`** → the run ended without per-test results reaching Test Manager; the test case log's `Info` carries the reason (results upload failed, or the job never started). Note `report get` counts `None` as **`Skipped`**, so 0% here means "results lost", not "tests skipped". +- **Stuck `Pending`** → dispatch worked but nothing is executing. A faulted job may never sync back, so don't sit out the 30-minute wait; check after ~5 minutes. Fixing the folder does **not** rescue an already-pending execution — start a new run after fixing. +- **`Serverless.Runtime.CannotIssueUserTokenDueToUserNotPartOfOrg`** in a log's `Info` → the folder's robot-user pool is bad: the account Test Manager picked is still folder-assigned but no longer valid in the org. Unlike the shapes above this **is** fixable from the CLI — point the project at a folder whose members satisfy Step 4 (or create one), then re-run. + +```bash +uip tm executions testcaselogs list --execution-id --project-key --output json +uip or jobs list --folder-key --output json +``` + +Everything past the Test Manager boundary — job states, machines, folder membership — belongs to [/uipath:uipath-platform § orchestrator/run-jobs.md](../../uipath-platform/references/orchestrator/run-jobs.md) and [§ orchestrator/setup-environment.md](../../uipath-platform/references/orchestrator/setup-environment.md); use those for the job-side detail. A `Duration` of `00:00:00` and an empty `StartTime` on an already-finished execution are normal — not evidence of a broken run. `JobKey` values on the logs prove Test Manager dispatched; jobs `Faulted` with no host machine mean the tenant can't run them. Either way it needs the platform team — report it rather than re-running (there is no CLI verb to cancel a Test Manager execution). + +## Iterating on the suite (and getting a fix back in) + +Changing a test — including fixing one the run just failed — uses the same pipeline with a bumped version: + +```bash +uip tm pack --project-path --type playwright --project-key --name --package-version 1.0.1 -o --output json +uip or packages upload "/.1.0.1.nupkg" --output json +``` + +The version **must** be new — the Orchestrator feed rejects one it already has (Step 2). Ingestion then **updates** the existing test cases in place (matched per test), creates any new ones, and unlinks removed ones, so: + +- **A test set keeps its membership** when the test list is unchanged — nothing to re-add, no new test set, re-run the same `--test-set-key`. Only re-run `uip tm testcases add --labels` if new tests should join. +- **A stored Playwright project selection survives** too, so the re-run stays scoped. + +**Confirming the new package actually ran:** the test case logs carry `TestCaseVersion`, so after the re-run + +```bash +uip tm executions testcaselogs list --execution-id --project-key --output json +``` + +reports the version you just packed (`1.0.1`). That is the reliable check — `list-automations` shows a two-component `PackageVersion` (`1.0`) that cannot distinguish 1.0.1 from 1.0.0. Ingestion of an update is otherwise invisible (same count, same names), so allow ~60–90 s after the upload before re-running; an old `TestCaseVersion` in the results means a stale package, not a bad fix. diff --git a/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml b/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml new file mode 100644 index 0000000000..dc9861cc2e --- /dev/null +++ b/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml @@ -0,0 +1,66 @@ +# Passes on the currently published CLI and stays valid after the external-package +# commands ship: it grades the preparation work and the capability check the skill +# prescribes, not the packaging call itself. Everything happens in the sandbox +# working directory (no tenant state), so nothing needs cleaning up afterwards. +task_id: skill-test-playwright-package-prep-smoke +description: > + Smoke test: agent uses the uipath-test skill to get a Playwright suite ready + for UiPath external test packaging and to report whether this CLI can package + it. Asserts the agent produces a project meeting the skill's stated + prerequisites (a @playwright/test dependency, a lockfile, a playwright + config), checks the CLI's packaging capability instead of guessing, and never + reaches for the Studio/RPA link-automation pipeline. +tags: [uipath-test, smoke, mode:build, lifecycle:generate, feature:test-case] + +initial_prompt: | + I have a Playwright suite I want to run through UiPath Test Manager. Set up a + minimal one in ./pw-suite with two passing tests, get it ready for UiPath + packaging, and tell me whether this CLI can package it. + +success_criteria: + - type: run_command + description: "Playwright config present in the prepared project (any supported extension)" + command: "find pw-suite -maxdepth 1 -name 'playwright.config.*' | grep -q ." + timeout: 30 + expected_exit_code: 0 + weight: 1.0 + pass_threshold: 1.0 + + - type: run_command + description: "Project declares @playwright/test (pack prerequisite)" + command: "grep -q '@playwright/test' pw-suite/package.json" + timeout: 30 + expected_exit_code: 0 + weight: 1.5 + pass_threshold: 1.0 + + - type: run_command + description: "Lockfile committed next to package.json (pack prerequisite)" + command: "ls pw-suite/ | grep -qE 'package-lock.json|yarn.lock|pnpm-lock.yaml|bun.lock'" + timeout: 30 + expected_exit_code: 0 + weight: 1.5 + pass_threshold: 1.0 + + - type: run_command + description: "Two Playwright tests were written" + command: "[ \"$(grep -rho 'test(' pw-suite/tests | wc -l | tr -d ' ')\" -ge 2 ]" + timeout: 30 + expected_exit_code: 0 + weight: 1.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent checked the CLI's packaging capability instead of assuming it" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+pack\b' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_not_executed + description: "Agent did not reach for the Studio/RPA pipeline (link-automation is wrong for external packages)" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+link-automation' + weight: 1.0 + pass_threshold: 1.0