From eeb0bc260f4b44f2cb6616a64100eba235a7f907 Mon Sep 17 00:00:00 2001 From: Aditya Garg Date: Wed, 29 Jul 2026 12:48:28 +0530 Subject: [PATCH 01/24] feat(uipath-test): Playwright first-mile pipeline Teach agents the full Playwright flow on serverless via uip alone: pack --type playwright, upload, automatic ingestion (PW_* labels, no link step), label-based test-set fill, the hidden playwright-context probe as the RPA-vs-Playwright discriminator, and per-project runs with --playwright-projects. New references/playwright-first-mile-guide plus SKILL.md command rows, concept, navigation entry, and anti-patterns (never link-automation Playwright test cases; hidden commands are absent from --help by design). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/SKILL.md | 23 +++- .../references/playwright-first-mile-guide.md | 127 ++++++++++++++++++ 2 files changed, 146 insertions(+), 4 deletions(-) create mode 100644 skills/uipath-test/references/playwright-first-mile-guide.md diff --git a/skills/uipath-test/SKILL.md b/skills/uipath-test/SKILL.md index 9330dabd3f..0086228f4d 100644 --- a/skills/uipath-test/SKILL.md +++ b/skills/uipath-test/SKILL.md @@ -1,6 +1,6 @@ --- name: uipath-test -description: "UiPath Test Manager — manage test projects, cases, sets, executions; generate reports. For Orchestrator→uipath-platform. For test automation→uipath-rpa." +description: "UiPath Test Manager — manage test projects, cases, sets, executions; generate reports; pack and run Playwright suites on serverless (tm pack, PW_* labels, --playwright-projects). For Orchestrator→uipath-platform. For Studio/RPA test automation authoring→uipath-rpa." allowed-tools: Bash, Read, Write, Glob, Grep user-invocable: true --- @@ -29,6 +29,7 @@ UiPath Test Manager is a web application that manages the testing lifecycle of p - **Test case logs** - Logs of a **test case** in an execution. A **testcase** can be navigated from **testcaselogs**. - **Test step logs** — Step-level logs within a **test case log**. - **Test case log assertions** - Assertion steps of a test case log in an execution. +- **External (Playwright) test packages** - A Playwright suite packed with `uip tm pack --type playwright` and uploaded to Orchestrator. Ingestion then auto-creates one test case per Playwright test (no link step) and labels each with `PW_Tag_*`, `PW_Project_*`, `PW_Suite_*`, `PW_File_*`. These run on serverless cloud runtimes — see [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). CLI tool for UiPath Test Manager (`uip tm`). Use `uip tm --help` and `uip tm --help` to discover commands and options. **Always pass `--output json`** on every `uip` command. @@ -85,7 +86,7 @@ Common `uip tm` commands organized by resource type. | `uip tm testcases steps delete --project-key --step-id --yes` | Delete a step. | | `uip tm testcases list-result-history --project-key --test-case-id ` | List test case log result history for a specific test case. Optional `--only-failed`, `--filter`, `--limit`, `--offset`. | | `uip tm testcases run --project-key --test-case-id --name --execution-type ` | Start a new execution for one or more test cases. **Uses `--test-case-id ` (space-separated for multiple).** Optional `--async`, `--folder-key`, `--robot-user-key`, `--machine-key`. | -| `uip tm testcases add --test-set-key --test-case-keys ` | Add test cases to a test set (comma-separated keys). | +| `uip tm testcases add --test-set-key (--test-case-keys \| --labels )` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels (variadic, space-separated; OR-match, exact, case-sensitive). Pass exactly one of the two selectors. | | `uip tm testcases remove --test-set-key --test-case-keys ` | Remove test cases from a test set (comma-separated keys). | > **Flag shapes for test case and step identifiers — do not interchange:** @@ -103,10 +104,13 @@ Common `uip tm` commands organized by resource type. | `uip tm testsets update --test-set-key --name ` | Update a test set name or description. | | `uip tm testsets delete --test-set-key ` | Delete a test set by its key. | | `uip tm testsets list-testcases --project-key --test-set-key ` | List test cases assigned to a test set. | -| `uip tm testsets run --test-set-key ` | Run a test set and return the execution ID. Optional `--execution-type ` (default `automated`), `--input-path ` for parameter overrides. | +| `uip tm testsets run --test-set-key ` | Run a test set and return the execution ID. Optional `--execution-type ` (default `automated`), `--input-path ` for parameter overrides, `--wait` to block until terminal. For Playwright test sets, optional `--playwright-projects ` (hidden from `--help`; see below). | +| `uip tm testsets playwright-context --test-set-key ` | Probe whether a test set is a Playwright test set: returns `IsPlaywright` plus the available/selected Playwright project names. Hidden from `--help` but functional. See [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). | > Keys use the format `PROJECT_KEY:NUMBER` (e.g., `INV:42`). To add or remove test cases in a test set, use `uip tm testcases add` / `uip tm testcases remove` — those verbs live under the `testcases` group, not under `testsets`. +> **Playwright test sets:** `--playwright-projects ` (space-separated, case-sensitive `playwright.config` project names) runs only the selected projects and persists the selection on the test set. It requires every test case in the set to come from one single Playwright package; unknown names fail fast listing the valid ones. Probe first with `uip tm testsets playwright-context` and branch on `IsPlaywright`. Full pipeline: [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). + ### Executions Commands | Command | Purpose | @@ -156,6 +160,14 @@ Common `uip tm` commands organized by resource type. |---|---| | `uip tm result download --execution-id ` | Download test execution results as JUnit XML. Optional `--project-key`, `--test-set-key`, `--result-path `. | +### Pack Commands (Playwright) + +| Command | Purpose | +|---|---| +| `uip tm pack --project-path --type playwright --project-key --name --package-version -o ` | Pack a Playwright suite into a `.nupkg` external test package. Requires a lockfile and `@playwright/test` in the project. `--project-key` targets the TM project where ingestion auto-creates the test cases; `--no-create-test-cases` skips that; `--dry-run` previews. Upload with `uip or packages upload `. | + +> Packing is offline (no auth needed). The upload → ingestion → label-fill → run pipeline is in [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). + ### Wait Commands | Command | Purpose | @@ -305,7 +317,8 @@ If the probe in Rule #2 shows singular subjects, the CLI predates the closed-ver | I want to... | Start here | |---|---| | **Generate a shareable test report** (tester or release manager view) | [references/test-result-report-guide.md](references/test-result-report-guide.md) | -| **Publish a project and link it to a Test Manager test case** | [references/publish-and-link-guide.md](references/publish-and-link-guide.md) | +| **Publish a project and link it to a Test Manager test case** (Studio/RPA) | [references/publish-and-link-guide.md](references/publish-and-link-guide.md) | +| **Pack, ingest, and run a Playwright suite on serverless** (pack → upload → labels → run) | [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md) | ## Anti-patterns @@ -313,3 +326,5 @@ If the probe in Rule #2 shows singular subjects, the CLI predates the closed-ver - **Do NOT proceed if authentication fails** — all Test Manager API calls require a valid bearer token. Fail fast rather than surfacing confusing 401 errors later. - **Do NOT skip the surface probe** (Critical Rule #2). On a pre-rename CLI, post-rename commands fail with `unknown command`; on a post-rename CLI, pre-rename commands fail the same way. The skill targets the post-rename surface and falls back per the [Pre-rename fallbacks](#pre-rename-fallbacks) table. Picking the wrong shape without probing burns a retry on every call. - **Do NOT guess command names — verb-noun composites are required.** The CLI uses explicit verb-noun forms; bare verbs do not exist. Confirm with `uip tm --help --output json`. +- **Do NOT `link-automation` Playwright test cases.** Playwright ingestion links them to the package automatically; the manual link step belongs to the Studio/RPA pipeline only. +- **Do NOT conclude the Playwright commands are missing because `--help` omits them.** `uip tm testsets playwright-context` and `run --playwright-projects` are intentionally hidden until the capability is broadly available — they are functional. Probe by running them; on a Test Manager or CLI without the capability they fail with explicit instructions. diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md new file mode 100644 index 0000000000..16c232620e --- /dev/null +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -0,0 +1,127 @@ +# Playwright First Mile — Pack, Ingest, and Run on Serverless + +End-to-end pipeline: take a Playwright test suite from a repo to executed results in UiPath Test Manager, using only `uip` commands. This flow is **Playwright-specific** — the pack command, the auto-created test cases, the `PW_*` labels, and the per-project run selection below do NOT apply to UiPath Studio/RPA packages. For Studio test automation use [publish-and-link-guide.md](publish-and-link-guide.md) instead; never mix the two pipelines. + +## Pipeline + +``` +uip tm pack --type playwright → .nupkg with embedded test metadata +uip or packages upload → package on the Orchestrator feed +(Test Manager ingestion, automatic) → test cases auto-create, PW_* labels applied +uip tm testsets create → empty test set +uip tm testcases add --labels → fill it by label +uip tm testsets playwright-context → probe: is this a Playwright test set? which projects? +uip tm testsets run [--playwright-projects ] → execute on serverless +uip tm wait / report / result → outcome +``` + +The key difference from the RPA pipeline: there is **no link step**. Uploading the package is enough — ingestion creates one Test Manager test case per discovered Playwright test, already bound to the package, and labels each with: + +- `PW_Tag_` — one per Playwright tag (`@smoke` → `PW_Tag_smoke`) +- `PW_Project_` — one per Playwright project the test runs in +- `PW_Suite_` — describe-block grouping +- `PW_File_` — the spec file + +> **Do NOT run `uip tm testcases link-automation` on Playwright test cases.** They are linked by ingestion; manual linking is the RPA pipeline and will corrupt the association. + +> **Hidden commands.** `uip tm testsets playwright-context` and the `--playwright-projects` flag on `uip tm testsets run` are intentionally hidden from `--help` until the capability is broadly available — they are functional. Do not conclude they don't exist because help doesn't list them; trust this guide and probe by running them. + +## Prerequisites + +- A recent `@uipath/cli` — this flow's commands (`tm pack --type playwright`, `testcases add --labels`, `testsets playwright-context`, `run --playwright-projects`) do not exist on older CLIs and have no pre-rename fallback. If `uip tm pack --help` does not show `--type`, upgrade the CLI before anything else. +- Logged in: `uip login status --output json`. If not, `uip login`. +- A Test Manager project to land the test cases in: `uip tm project list --filter --output json`, or create one with `uip tm project create`. Capture the project key. +- The tenant's Test Manager must have Playwright support enabled (a server-side feature flag). If ingestion never produces test cases (Step 3), this is the first thing to suspect — stop and ask the user. +- The Playwright project directory must contain: + - `package.json` with `@playwright/test` installed (discovery shells out to the project's own `playwright test --list`; no browsers needed), + - a **lockfile** (`package-lock.json` / `yarn.lock` / `pnpm-lock.yaml` / `bun.lock`) — serverless does a deterministic install, + - a `playwright.config` file. + +## Step 1 — Pack + +```bash +uip tm pack --project-path --type playwright \ + --project-key --name \ + --package-version 1.0.0 -o --output json +``` + +- `--project-key` is **required** by default because test-case auto-creation is on; pass `--no-create-test-cases` to pack without it (label metadata stays embedded for later use). +- Preview with `--dry-run` (writes nothing). +- Capture `Data.Output` (the `.nupkg` path) and `Data.TestCount` from the JSON output. `TestCount` is the number of test cases ingestion will create — remember it for Step 3. +- Pack fails loudly when the lockfile or `@playwright/test` is missing — fix the project, do not improvise around it. + +## Step 2 — Upload to Orchestrator + +```bash +uip or packages upload "/.1.0.0.nupkg" --output json +``` + +Each re-upload needs a **new `--package-version`** at pack time — Orchestrator feeds reject an existing version. + +## Step 3 — Wait for ingestion + +Ingestion is asynchronous and automatic. Poll until the auto-created test cases appear: + +```bash +uip tm testcases list --project-key --filter --output json +``` + +- Expect exactly `TestCount` test cases (from Step 1), typically within 1–2 minutes. +- Poll every ~10 seconds, up to ~3 minutes. If nothing appears by then, STOP and report — the likely causes are the Playwright feature flag being off for the tenant or a wrong `--project-key`; both need the user, not retries. +- Verify the labels landed: `uip tm objectlabel list --project-key --object-type TestCase --filter PW_ --output json`. + +## Step 4 — Create a test set and fill it by label + +```bash +uip tm testsets create --project-key --name "PW Smoke" --output json +uip tm testcases add --test-set-key --labels PW_Tag_smoke PW_Project_chromium --output json +``` + +- Capture `TestSetKey` from the create output (e.g. `DEMO:10`). +- `--labels` is variadic and space-separated (quote names that contain spaces). Matching is **OR across labels, exact, and case-sensitive** — discover the real names first with `uip tm objectlabel list` rather than guessing. +- `--labels` works with any object label; the `PW_*` labels are simply what ingestion applies. +- Mutually exclusive with `--test-case-keys`; pass exactly one of the two. +- **Keep one test set = one Playwright package.** Per-project selection (Step 6) requires every test case in the set to come from a single Playwright package; label-filling across packages produces a set that cannot be project-scoped. + +## Step 5 — Probe the Playwright context (recommended) + +Before deciding whether `--playwright-projects` applies, ask the server: + +```bash +uip tm testsets playwright-context --test-set-key --output json +``` + +- `Data.IsPlaywright: true` → the set resolves to one Playwright package; `AvailablePlaywrightProjects` lists the only valid `--playwright-projects` values, and `SelectedPlaywrightProjects` shows any selection already stored on the test set. +- `Data.IsPlaywright: false` → RPA, mixed, manual, or multi-package test set — run it **without** `--playwright-projects`. +- The server never errors on type here, so this is the safe discriminator for automation: probe first, branch on `IsPlaywright`. + +## Step 6 — Run, optionally per Playwright project + +The project needs a default Orchestrator folder before any run (Critical Rule #10): `uip tm project set-default-folder --project-key --folder-key --output json` (folder keys via `uip or folders list -n --all --output json`). + +```bash +uip tm testsets run --test-set-key \ + --playwright-projects chromium firefox --wait --output json +``` + +`--playwright-projects` semantics (all enforced with clear errors, nothing is silently ignored): + +- Space-separated, case-sensitive names from the package's `playwright.config`. Unknown names **fail fast, before anything is persisted**, listing the available projects. +- Valid only when every test case in the set comes from one single Playwright package (see Step 4); fails for Studio/RPA test sets — run those without the flag. +- The selection **persists on the test set** and applies to later runs until changed; omit the flag to reuse the stored selection (or the config's defaults if none was ever stored). +- On a Test Manager without Playwright support the command fails with instructions rather than running incorrectly. + +Omit `--playwright-projects` entirely for a plain run (all config-default projects). + +## Step 7 — Results + +- `--wait` on the run blocks until terminal; without it, use `uip tm wait --execution-id --output json`. +- Summary: `uip tm report get --execution-id --output json`. +- Per-test detail: `uip tm executions testcaselogs list --execution-id --project-key --output json`. +- JUnit export: `uip tm result download --execution-id --result-path --output json`. + +Execution happens on UiPath serverless cloud runtimes — no robot, machine, or folder package deployment is needed beyond the upload in Step 2. + +## Iterating on the suite + +Re-running after test changes is the same pipeline with a bumped version: pack with a new `--package-version`, upload, wait for ingestion to sync. Ingestion **updates** existing test cases (matched per test), creates new ones, and unlinks removed ones — test sets keep their membership for surviving test cases; re-run `uip tm testcases add --labels` if new tests should join a set. From b2eab392889072a7b691433a56b70240d0ef1573 Mon Sep 17 00:00:00 2001 From: Aditya Garg Date: Wed, 29 Jul 2026 13:40:11 +0530 Subject: [PATCH 02/24] fix(uipath-test): correct the first-mile guide from a cold-agent run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A context-free agent ran the whole Playwright pipeline from the skill alone and filed defects; all fixed: the ingestion poll no longer uses --filter (names are " > " — the filter never matches and reads as a false "flag off"); TestCount semantics (per test, not per project); IsAutomated:false is normal on ingested test cases; playwright-context fields are comma-joined strings, not arrays; run --wait prints the execution id in an early log line (30-min default timeout, 60s polls); new stuck-in-Pending triage (JobKeys in testcaselogs prove dispatch; then it's a missing serverless runtime — stop, don't retry); PW_Path_* label documented; Critical Rule 3 notes progress lines precede the JSON envelope. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/SKILL.md | 2 +- .../references/playwright-first-mile-guide.md | 19 +++++++++++++++---- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/skills/uipath-test/SKILL.md b/skills/uipath-test/SKILL.md index 0086228f4d..d96f075d31 100644 --- a/skills/uipath-test/SKILL.md +++ b/skills/uipath-test/SKILL.md @@ -231,7 +231,7 @@ Object labels are tag-style metadata applied to Requirement, TestCase, TestSet, 1. **Always check login first** — run `uip login status --output json` before any Test Manager operation. If not authenticated, run `uip login` to sign in. 2. **Probe the CLI surface once per session, before the first `uip tm` command.** Run `uip tm testcases --help --output json` (any flags accepted). Result `Success` → post-rename CLI; use the command tables above as-is. `unknown command` / non-zero exit → pre-rename CLI; translate via the [Pre-rename fallbacks](#pre-rename-fallbacks) table before each call. Re-probe on any later `unknown command` error. -3. **Always pass `--output json`** to every `uip` command — no exceptions. Structured JSON output is what you need to reason about results reliably, even when you only plan to summarize them back to the user. +3. **Always pass `--output json`** to every `uip` command — no exceptions. Structured JSON output is what you need to reason about results reliably, even when you only plan to summarize them back to the user. Note: stdout may carry human progress lines *before* the JSON envelope (e.g. `Resolved project …`, run/wait progress) — when parsing, extract the JSON object rather than parsing stdout wholesale. 4. **Cap retries at 3** for any failing `uip` CLI command. After 3 failures, stop and report the error to the user (see Rule — never fall back to direct REST APIs). 5. **Handle empty results** — if a list command returns an empty array, stop and inform the user rather than proceeding with a null key. Exception: a zero-result `tm testcases list --filter` call may be a prefix miss rather than a truly empty dataset — apply Rule 9's prefix fallback to disambiguate; stop only when the fallback also finds nothing. Filtered lookups on other resources (project, customfield) returning empty are real empty results — stop as usual. 6. **Confirm before delete** — always confirm the target resource key with the user before running any `delete` command. All delete commands require `--yes` (or `-y`) to proceed; omitting it exits without deleting. diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index 16c232620e..794e4bd40a 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -19,7 +19,7 @@ The key difference from the RPA pipeline: there is **no link step**. Uploading t - `PW_Tag_<tag>` — one per Playwright tag (`@smoke` → `PW_Tag_smoke`) - `PW_Project_<name>` — one per Playwright project the test runs in -- `PW_Suite_<name>` — describe-block grouping +- `PW_Suite_<name>` / `PW_Path_<chain>` — describe-block grouping (name and full chain) - `PW_File_<path>` — the spec file > **Do NOT run `uip tm testcases link-automation` on Playwright test cases.** They are linked by ingestion; manual linking is the RPA pipeline and will corrupt the association. @@ -63,10 +63,12 @@ Each re-upload needs a **new `--package-version`** at pack time — Orchestrator Ingestion is asynchronous and automatic. Poll until the auto-created test cases appear: ```bash -uip tm testcases list --project-key <PROJECT_KEY> --filter <PackageName> --output json +uip tm testcases list --project-key <PROJECT_KEY> --output json ``` -- Expect exactly `TestCount` test cases (from Step 1), typically within 1–2 minutes. +- Poll **unfiltered** and count. Do NOT pass `--filter <PackageName>` — the auto-created test case *names* are `"<suite> > <test title>"`; the package name appears only in the description, which `--filter` does not search, so a package-name filter stays empty forever and reads as a false "ingestion never happened". +- Expect exactly `TestCount` new test cases (from Step 1), typically within 1–2 minutes. `TestCount` is one per Playwright **test**, NOT multiplied by the number of Playwright projects (2 tests × 2 projects → 2 test cases). +- Ingested test cases show `IsAutomated: false` in list output — that is normal and does not mean ingestion failed; the package linkage is real (their execution logs carry `HasLinkedAutomation: true` and Orchestrator job keys). - Poll every ~10 seconds, up to ~3 minutes. If nothing appears by then, STOP and report — the likely causes are the Playwright feature flag being off for the tenant or a wrong `--project-key`; both need the user, not retries. - Verify the labels landed: `uip tm objectlabel list --project-key <PROJECT_KEY> --object-type TestCase --filter PW_ --output json`. @@ -91,7 +93,7 @@ Before deciding whether `--playwright-projects` applies, ask the server: uip tm testsets playwright-context --test-set-key <TEST_SET_KEY> --output json ``` -- `Data.IsPlaywright: true` → the set resolves to one Playwright package; `AvailablePlaywrightProjects` lists the only valid `--playwright-projects` values, and `SelectedPlaywrightProjects` shows any selection already stored on the test set. +- `Data.IsPlaywright: true` → the set resolves to one Playwright package; `AvailablePlaywrightProjects` holds the only valid `--playwright-projects` values, and `SelectedPlaywrightProjects` shows any selection already stored on the test set. Both are **comma-joined strings** (`"chromium, firefox"`), not arrays — split on `", "` when scripting; no stored selection is `""`. - `Data.IsPlaywright: false` → RPA, mixed, manual, or multi-package test set — run it **without** `--playwright-projects`. - The server never errors on type here, so this is the safe discriminator for automation: probe first, branch on `IsPlaywright`. @@ -113,6 +115,8 @@ uip tm testsets run --test-set-key <TEST_SET_KEY> \ Omit `--playwright-projects` entirely for a plain run (all config-default projects). +With `--wait`, the execution id is printed **early, in a progress log line** (`Starting execution for test set …`) — the JSON envelope only arrives at terminal state. If you abort the wait, recover the id from that log line or with `uip tm executions list --project-key <PROJECT_KEY> --output json`. `--wait` polls every 60 s with a default timeout of 30 minutes. + ## Step 7 — Results - `--wait` on the run blocks until terminal; without it, use `uip tm wait --execution-id <EXECUTION_ID> --output json`. @@ -122,6 +126,13 @@ Omit `--playwright-projects` entirely for a plain run (all config-default projec Execution happens on UiPath serverless cloud runtimes — no robot, machine, or folder package deployment is needed beyond the upload in Step 2. +### If the execution stays `Pending` + +A run that never leaves `Pending` almost always means dispatch worked but nothing is executing the jobs — the tenant has no serverless Playwright runtime (or no capacity). Triage before waiting out the full 30-minute timeout: + +1. `uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json` — test case logs carrying Orchestrator `JobKey` values prove dispatch happened; the problem is downstream of Test Manager. +2. Still all-`Pending` with JobKeys after ~5 minutes → STOP and report that the tenant lacks a serverless Playwright runtime. This needs the user/platform team; retrying, re-running, or re-uploading will not help. + ## Iterating on the suite Re-running after test changes is the same pipeline with a bumped version: pack with a new `--package-version`, upload, wait for ingestion to sync. Ingestion **updates** existing test cases (matched per test), creates new ones, and unlinks removed ones — test sets keep their membership for surviving test cases; re-run `uip tm testcases add --labels` if new tests should join a set. From 6abfef9d2f306b40d1417b1d47d828f46645f143 Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Wed, 29 Jul 2026 13:57:25 +0530 Subject: [PATCH 03/24] fix(uipath-test): round-2 cold-run findings on the first-mile guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second context-free agent run (patched guide) was flawless through dispatch; remaining gaps were all post-dispatch and are now covered: the Pending triage checks the Orchestrator jobs themselves (uip or jobs list --folder-key; Faulted + empty HostMachineName = missing serverless runtime), warns that faulted jobs may never sync back (TM stays Pending — don't sit out the 30-min wait), and states there is no CLI cancel verb for a TM execution. Also: labels select tests while --playwright-projects selects browsers (the one guess the agent had to make), and the label spot-check wording no longer overpromises. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../uipath-test/references/playwright-first-mile-guide.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index 794e4bd40a..35e920056d 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -70,7 +70,7 @@ uip tm testcases list --project-key <PROJECT_KEY> --output json - Expect exactly `TestCount` new test cases (from Step 1), typically within 1–2 minutes. `TestCount` is one per Playwright **test**, NOT multiplied by the number of Playwright projects (2 tests × 2 projects → 2 test cases). - Ingested test cases show `IsAutomated: false` in list output — that is normal and does not mean ingestion failed; the package linkage is real (their execution logs carry `HasLinkedAutomation: true` and Orchestrator job keys). - Poll every ~10 seconds, up to ~3 minutes. If nothing appears by then, STOP and report — the likely causes are the Playwright feature flag being off for the tenant or a wrong `--project-key`; both need the user, not retries. -- Verify the labels landed: `uip tm objectlabel list --project-key <PROJECT_KEY> --object-type TestCase --filter PW_ --output json`. +- Spot-check the labels landed: `uip tm objectlabel list --project-key <PROJECT_KEY> --object-type TestCase --filter PW_ --output json` (returns distinct label *names* only — enough to confirm ingestion labeled things, not which test case carries which label). ## Step 4 — Create a test set and fill it by label @@ -83,6 +83,7 @@ uip tm testcases add --test-set-key <TEST_SET_KEY> --labels PW_Tag_smoke PW_Proj - `--labels` is variadic and space-separated (quote names that contain spaces). Matching is **OR across labels, exact, and case-sensitive** — discover the real names first with `uip tm objectlabel list` rather than guessing. - `--labels` works with any object label; the `PW_*` labels are simply what ingestion applies. - Mutually exclusive with `--test-case-keys`; pass exactly one of the two. +- **Labels select *tests*; `--playwright-projects` selects *browsers*.** Filling by `PW_Project_firefox` picks every test that runs in the firefox project (often all of them); it does not make the run firefox-only — that is what the run flag in Step 6 does. To "run only <project>", label-fill by whatever identifies the tests you want (tag, suite, file) and pass the project name to `--playwright-projects`. - **Keep one test set = one Playwright package.** Per-project selection (Step 6) requires every test case in the set to come from a single Playwright package; label-filling across packages produces a set that cannot be project-scoped. ## Step 5 — Probe the Playwright context (recommended) @@ -131,7 +132,9 @@ Execution happens on UiPath serverless cloud runtimes — no robot, machine, or A run that never leaves `Pending` almost always means dispatch worked but nothing is executing the jobs — the tenant has no serverless Playwright runtime (or no capacity). Triage before waiting out the full 30-minute timeout: 1. `uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json` — test case logs carrying Orchestrator `JobKey` values prove dispatch happened; the problem is downstream of Test Manager. -2. Still all-`Pending` with JobKeys after ~5 minutes → STOP and report that the tenant lacks a serverless Playwright runtime. This needs the user/platform team; retrying, re-running, or re-uploading will not help. +2. Check the dispatched jobs themselves: `uip or jobs list --folder-key <FOLDER_KEY> --output json` (a folder flag or `--all-folders` is required). Jobs `Faulted` with an empty `HostMachineName`/`MachineKey` = nothing can execute them — the tenant lacks a serverless Playwright runtime (or capacity). +3. **A faulted job may never sync back** — Test Manager can stay `Pending` indefinitely even though the jobs are already dead, so do not sit out the 30-minute `--wait`; run the two checks above after ~5 minutes of `Pending`. +4. Faulted/missing-runtime → STOP and report to the user/platform team. Retrying, re-running, or re-uploading will not help. There is no CLI cancel verb for a Test Manager execution — leave the execution as-is and note it in your report (`uip or jobs stop` can stop *pending/running* Orchestrator jobs, but does nothing for already-faulted ones). ## Iterating on the suite From 89f5946a7ace8f6e5d0a398b3b14c5a0d4e2547e Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Wed, 29 Jul 2026 15:09:04 +0530 Subject: [PATCH 04/24] fix(uipath-test): round-3 findings from the full-serverless cold run Third context-free run reached Finished/2-passed on serverless; the guide now prevents its two traps: the default folder moves ahead of the playwright-context probe with an explicit false-negative warning (no default folder => a real Playwright set reports IsPlaywright false, and a literal reader would then run every browser), and report get documents its required --project-key/--test-set-key. Also: whole-suite-on-one-browser labeling recipe and a quoting note for label names with spaces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/SKILL.md | 4 ++-- .../references/playwright-first-mile-guide.md | 17 ++++++++++++----- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/skills/uipath-test/SKILL.md b/skills/uipath-test/SKILL.md index d96f075d31..3ce475d481 100644 --- a/skills/uipath-test/SKILL.md +++ b/skills/uipath-test/SKILL.md @@ -86,7 +86,7 @@ Common `uip tm` commands organized by resource type. | `uip tm testcases steps delete --project-key <PROJECT_KEY> --step-id <UUID> --yes` | Delete a step. | | `uip tm testcases list-result-history --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID>` | List test case log result history for a specific test case. Optional `--only-failed`, `--filter`, `--limit`, `--offset`. | | `uip tm testcases run --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID> --name <EXECUTION_NAME> --execution-type <manual\|automated\|none\|mixed>` | Start a new execution for one or more test cases. **Uses `--test-case-id <UUID>` (space-separated for multiple).** Optional `--async`, `--folder-key`, `--robot-user-key`, `--machine-key`. | -| `uip tm testcases add --test-set-key <TEST_SET_KEY> (--test-case-keys <KEY1,KEY2,...> \| --labels <name...>)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels (variadic, space-separated; OR-match, exact, case-sensitive). Pass exactly one of the two selectors. | +| `uip tm testcases add --test-set-key <TEST_SET_KEY> (--test-case-keys <KEY1,KEY2,...> \| --labels <name...>)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels (variadic, space-separated; quote names containing spaces; OR-match, exact, case-sensitive). Pass exactly one of the two selectors. | | `uip tm testcases remove --test-set-key <TEST_SET_KEY> --test-case-keys <KEY1,KEY2,...>` | Remove test cases from a test set (comma-separated keys). | > **Flag shapes for test case and step identifiers — do not interchange:** @@ -145,7 +145,7 @@ Common `uip tm` commands organized by resource type. | Command | Purpose | |---|---| -| `uip tm report get --execution-id <EXECUTION_ID>` | Get a summary report for a completed test execution. Optional `--project-key`, `--test-set-key`, `--query`. | +| `uip tm report get --execution-id <EXECUTION_ID> (--project-key <KEY> \| --test-set-key <KEY>)` | Get a summary report for a completed test execution. One of `--project-key`/`--test-set-key` is required to identify the project. Optional `--query`. | ### Attachment Commands diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index 35e920056d..3caad503eb 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -72,17 +72,25 @@ uip tm testcases list --project-key <PROJECT_KEY> --output json - Poll every ~10 seconds, up to ~3 minutes. If nothing appears by then, STOP and report — the likely causes are the Playwright feature flag being off for the tenant or a wrong `--project-key`; both need the user, not retries. - Spot-check the labels landed: `uip tm objectlabel list --project-key <PROJECT_KEY> --object-type TestCase --filter PW_ --output json` (returns distinct label *names* only — enough to confirm ingestion labeled things, not which test case carries which label). -## Step 4 — Create a test set and fill it by label +## Step 4 — Set the default folder, create a test set, fill it by label + +Set the project's default Orchestrator folder FIRST — both the Step 5 probe and the Step 6 run resolve packages through it (Critical Rule #10): + +```bash +uip or folders list -n <folder-name> --all --output json # Key field = folder key +uip tm project set-default-folder --project-key <PROJECT_KEY> --folder-key <FOLDER_KEY> --output json +``` ```bash uip tm testsets create --project-key <PROJECT_KEY> --name "PW Smoke" --output json -uip tm testcases add --test-set-key <TEST_SET_KEY> --labels PW_Tag_smoke PW_Project_chromium --output json +uip tm testcases add --test-set-key <TEST_SET_KEY> --labels "PW_Suite_<name>" --output json ``` - Capture `TestSetKey` from the create output (e.g. `DEMO:10`). - `--labels` is variadic and space-separated (quote names that contain spaces). Matching is **OR across labels, exact, and case-sensitive** — discover the real names first with `uip tm objectlabel list` rather than guessing. - `--labels` works with any object label; the `PW_*` labels are simply what ingestion applies. - Mutually exclusive with `--test-case-keys`; pass exactly one of the two. +- To run the whole suite on one browser: fill by a suite/file label (`PW_Suite_*` or `PW_File_*`) and pass the browser to `--playwright-projects` in Step 6. Filling by `PW_Project_<name>` is for selecting the subset of tests that participate in that project — it does not restrict which browsers run. - **Labels select *tests*; `--playwright-projects` selects *browsers*.** Filling by `PW_Project_firefox` picks every test that runs in the firefox project (often all of them); it does not make the run firefox-only — that is what the run flag in Step 6 does. To "run only <project>", label-fill by whatever identifies the tests you want (tag, suite, file) and pass the project name to `--playwright-projects`. - **Keep one test set = one Playwright package.** Per-project selection (Step 6) requires every test case in the set to come from a single Playwright package; label-filling across packages produces a set that cannot be project-scoped. @@ -97,11 +105,10 @@ uip tm testsets playwright-context --test-set-key <TEST_SET_KEY> --output json - `Data.IsPlaywright: true` → the set resolves to one Playwright package; `AvailablePlaywrightProjects` holds the only valid `--playwright-projects` values, and `SelectedPlaywrightProjects` shows any selection already stored on the test set. Both are **comma-joined strings** (`"chromium, firefox"`), not arrays — split on `", "` when scripting; no stored selection is `""`. - `Data.IsPlaywright: false` → RPA, mixed, manual, or multi-package test set — run it **without** `--playwright-projects`. - The server never errors on type here, so this is the safe discriminator for automation: probe first, branch on `IsPlaywright`. +- **False negative without a folder:** the probe resolves the package through the project's default folder — if that isn't set (Step 4), a genuine Playwright test set reports `IsPlaywright: false`. Set the default folder before trusting a `false`. ## Step 6 — Run, optionally per Playwright project -The project needs a default Orchestrator folder before any run (Critical Rule #10): `uip tm project set-default-folder --project-key <PROJECT_KEY> --folder-key <FOLDER_KEY> --output json` (folder keys via `uip or folders list -n <folder-name> --all --output json`). - ```bash uip tm testsets run --test-set-key <TEST_SET_KEY> \ --playwright-projects chromium firefox --wait --output json @@ -121,7 +128,7 @@ With `--wait`, the execution id is printed **early, in a progress log line** (`S ## Step 7 — Results - `--wait` on the run blocks until terminal; without it, use `uip tm wait --execution-id <EXECUTION_ID> --output json`. -- Summary: `uip tm report get --execution-id <EXECUTION_ID> --output json`. +- Summary: `uip tm report get --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json` (`--project-key` or `--test-set-key` is required — bare `--execution-id` exits with "Provide --project-key or --test-set-key"). - Per-test detail: `uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json`. - JUnit export: `uip tm result download --execution-id <EXECUTION_ID> --result-path <dir> --output json`. From 8b1abf44b84a70e3ac6c05c9de06bc052a60a14d Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Wed, 29 Jul 2026 17:02:35 +0530 Subject: [PATCH 05/24] fix(uipath-test): bounded-poll waiting guidance for agents A silent multi-minute --wait tripped an agent-harness watchdog twice in the clean-slate cold run; the guide now tells agents to start the run without --wait and poll with bounded 'uip tm wait --timeout' / get-stats calls instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/references/playwright-first-mile-guide.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index 3caad503eb..898803628f 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -125,6 +125,8 @@ Omit `--playwright-projects` entirely for a plain run (all config-default projec With `--wait`, the execution id is printed **early, in a progress log line** (`Starting execution for test set …`) — the JSON envelope only arrives at terminal state. If you abort the wait, recover the id from that log line or with `uip tm executions list --project-key <PROJECT_KEY> --output json`. `--wait` polls every 60 s with a default timeout of 30 minutes. +**Agent-friendly waiting:** a single `--wait` call can sit silent for many minutes, which trips agent-harness watchdogs and shell timeouts. When running as an agent, prefer starting the run **without** `--wait`, then poll in bounded chunks: `uip tm wait --execution-id <EXECUTION_ID> --timeout 120 --output json` in a loop (or `uip tm executions get-stats` every 30–60 s), so every call returns quickly and progress stays visible. + ## Step 7 — Results - `--wait` on the run blocks until terminal; without it, use `uip tm wait --execution-id <EXECUTION_ID> --output json`. From 4e7d46baefd42015073a9438671165a78bc435cc Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Wed, 29 Jul 2026 17:21:18 +0530 Subject: [PATCH 06/24] fix(uipath-test): round-4 findings from the clean-slate cold run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The npm-installed clean-slate run (real uip, no harness crutches) surfaced the results-lost failure mode and several trap-shaped doc errors, all fixed: new Finished-with-None triage (pod ran, upload leg broke; report get shows the lost results as Skipped; retrying keeps faulting — cap and report), the execution id comes from the "Execution started:" line not the test-set line, a wait --timeout expiry returns a misleading RetryWillNotFix envelope (means "still running"), the telemetry-flush slow-exit quirk, and a carve-out so autonomous tasks that authorize project creation don't stall on "ask the user". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/SKILL.md | 4 +++- .../references/playwright-first-mile-guide.md | 10 ++++++++-- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/skills/uipath-test/SKILL.md b/skills/uipath-test/SKILL.md index 3ce475d481..e30196dcc6 100644 --- a/skills/uipath-test/SKILL.md +++ b/skills/uipath-test/SKILL.md @@ -273,7 +273,7 @@ If the probe in Rule #2 shows singular subjects, the CLI predates the closed-ver For more authentication details, run `/uipath-platform`. ### Confirm project scope - Ask the user for the project name or key before any Test Manager call. For multi-project scenarios, collect ALL names or keys in one prompt. Resolve each to a `PROJECT_KEY`: + Ask the user for the project name or key before any Test Manager call (unless the task already names one or authorizes creating a new project). For multi-project scenarios, collect ALL names or keys in one prompt. Resolve each to a `PROJECT_KEY`: ```bash uip tm project list --filter <NAME_OR_KEY> --output json ``` @@ -321,6 +321,8 @@ If the probe in Rule #2 shows singular subjects, the CLI predates the closed-ver | **Pack, ingest, and run a Playwright suite on serverless** (pack → upload → labels → run) | [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md) | +> **Known CLI quirk:** a command can emit its JSON envelope and then sit for a while before exiting (`[Telemetry] flush timed out` warnings). If you have the JSON, the command succeeded — don't let a slow exit be mistaken for a hang by your own timeouts; cap per-command wall time modestly and move on once the envelope is in hand. + ## Anti-patterns - **Do NOT proceed if authentication fails** — all Test Manager API calls require a valid bearer token. Fail fast rather than surfacing confusing 401 errors later. diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index 898803628f..401f1bc83c 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -123,9 +123,9 @@ uip tm testsets run --test-set-key <TEST_SET_KEY> \ Omit `--playwright-projects` entirely for a plain run (all config-default projects). -With `--wait`, the execution id is printed **early, in a progress log line** (`Starting execution for test set …`) — the JSON envelope only arrives at terminal state. If you abort the wait, recover the id from that log line or with `uip tm executions list --project-key <PROJECT_KEY> --output json`. `--wait` polls every 60 s with a default timeout of 30 minutes. +With `--wait`, the execution id is printed **early, in a progress log line** — take it from the `Execution started: <id> (Pending)` line, NOT from `Starting execution for test set …` (that line's UUID is the *test set* id). The JSON envelope only arrives at terminal state. If you abort the wait, recover the id from that log line or with `uip tm executions list --project-key <PROJECT_KEY> --output json`. `--wait` polls every 60 s with a default timeout of 30 minutes. -**Agent-friendly waiting:** a single `--wait` call can sit silent for many minutes, which trips agent-harness watchdogs and shell timeouts. When running as an agent, prefer starting the run **without** `--wait`, then poll in bounded chunks: `uip tm wait --execution-id <EXECUTION_ID> --timeout 120 --output json` in a loop (or `uip tm executions get-stats` every 30–60 s), so every call returns quickly and progress stays visible. +**Agent-friendly waiting:** a single `--wait` call can sit silent for many minutes, which trips agent-harness watchdogs and shell timeouts. When running as an agent, prefer starting the run **without** `--wait`, then poll in bounded chunks: `uip tm wait --execution-id <EXECUTION_ID> --timeout 120 --output json` in a loop (or `uip tm executions get-stats` every 30–60 s), so every call returns quickly and progress stays visible. A `wait` that hits its `--timeout` returns a Failure envelope with `Retry: "RetryWillNotFix"` — for a non-terminal execution that just means "still running"; keep polling, don't treat it as fatal. ## Step 7 — Results @@ -136,6 +136,12 @@ With `--wait`, the execution id is printed **early, in a progress log line** (`S Execution happens on UiPath serverless cloud runtimes — no robot, machine, or folder package deployment is needed beyond the upload in Step 2. +### If the execution Finishes with `Passed: 0 / Failed: 0 / None: N` + +`None` means the pod ran but its **results were never uploaded** to Test Manager — check the test case logs' Info ("pod terminated before results could be uploaded") and `uip or jobs list` (jobs `Faulted` *with* a `HostMachineName`). The tests themselves may even have passed; their outcomes were lost in the runner→Test Manager upload leg (storage/network on the tenant side). Two warnings: +- `report get` counts `None` results as **`Skipped`** — a 0% pass rate here means "results lost", not "tests skipped". +- Retrying re-runs the tests but will keep faulting until the upload path is fixed. Apply the retry cap (Critical Rule #4), then stop and report to the platform team. + ### If the execution stays `Pending` A run that never leaves `Pending` almost always means dispatch worked but nothing is executing the jobs — the tenant has no serverless Playwright runtime (or no capacity). Triage before waiting out the full 30-minute timeout: From e1608bc95a07f2b06285be1c2c06838b6f3963e3 Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Thu, 30 Jul 2026 02:12:53 +0530 Subject: [PATCH 07/24] fix(uipath-test): round-5 findings from the alpha cloud cold run First pure-user run on a real cloud tenant got the whole first mile through and exposed the last environment-shaped gaps: folder choice must come from the unflagged folders list (--all includes folders the user cannot use -> folderNotFoundOrNoAccess), the default folder needs a Cloud Robots - Serverless machine (create/assign documented; missing machine = instantly-Cancelled execution, a state the guide never covered), login refresh is the non-interactive fix for the Logged-in-but-401 session, None-result causes broadened beyond storage/network (identity faults land there too; HostMachineName is not a reliable discriminator), and auto-updater output joins the pre-envelope noise note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/SKILL.md | 4 ++-- .../references/playwright-first-mile-guide.md | 18 +++++++++++++++--- 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/skills/uipath-test/SKILL.md b/skills/uipath-test/SKILL.md index e30196dcc6..a30ffa7987 100644 --- a/skills/uipath-test/SKILL.md +++ b/skills/uipath-test/SKILL.md @@ -235,7 +235,7 @@ Object labels are tag-style metadata applied to Requirement, TestCase, TestSet, 4. **Cap retries at 3** for any failing `uip` CLI command. After 3 failures, stop and report the error to the user (see Rule — never fall back to direct REST APIs). 5. **Handle empty results** — if a list command returns an empty array, stop and inform the user rather than proceeding with a null key. Exception: a zero-result `tm testcases list --filter` call may be a prefix miss rather than a truly empty dataset — apply Rule 9's prefix fallback to disambiguate; stop only when the fallback also finds nothing. Filtered lookups on other resources (project, customfield) returning empty are real empty results — stop as usual. 6. **Confirm before delete** — always confirm the target resource key with the user before running any `delete` command. All delete commands require `--yes` (or `-y`) to proceed; omitting it exits without deleting. -7. **For operations requiring folder key** — use `uip or folders list -n <folder-name> --all --output json` (run `/uipath-platform` for folder management details). +7. **For operations requiring folder key** — when the user named a folder, look it up with `uip or folders list -n <folder-name> --all --output json`; when *choosing* a folder yourself, use the unflagged `uip or folders list --output json` (memberships only — `--all` includes folders you cannot actually use). Run `/uipath-platform` for folder management details. 8. **Discover before assuming** — never guess automation names, folder keys, project IDs, or test case keys. Always run the matching `list` command first (e.g., `uip tm testcases list-automations`, `uip or folders list -n <folder-name> --all`). 9. **Narrow `list` calls server-side when the user names an entity.** When the user provides a name, key, label, or tag, check `uip tm <resource> list --help` (or `uip or <resource> list --help`) for the narrowing flag the command exposes and pass it on the `list` call. Never list all results and filter client-side — it wastes tokens and misses paginated entries. Applies to every entity across `uip tm` and `uip or`. Exception: `tm testcases list --filter` matches by prefix. A mid-name term returns zero — retry `--filter` with a broader name prefix from context; re-list without `--filter` only when no workable prefix exists, paging through ALL results (`--limit`/`--offset`) before concluding the target is absent. 10. **Set default folder before any `run` command** — `uip tm testcases run` and `uip tm testsets run` both require a default Orchestrator folder on the project. Run `uip tm project set-default-folder --project-key <PROJECT_KEY> --folder-key <FOLDER_KEY> --output json` first. Get folder keys with `uip or folders list -n <folder-name> --all --output json`. @@ -321,7 +321,7 @@ If the probe in Rule #2 shows singular subjects, the CLI predates the closed-ver | **Pack, ingest, and run a Playwright suite on serverless** (pack → upload → labels → run) | [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md) | -> **Known CLI quirk:** a command can emit its JSON envelope and then sit for a while before exiting (`[Telemetry] flush timed out` warnings). If you have the JSON, the command succeeded — don't let a slow exit be mistaken for a hang by your own timeouts; cap per-command wall time modestly and move on once the envelope is in hand. +> **Known CLI quirks:** a command can emit its JSON envelope and then sit before exiting (`[Telemetry] flush timed out`) — if you have the JSON, it succeeded. Auto-updater lines ("Checking for updates…") can precede the envelope, same hazard as progress lines (Rule #3). And `login status` can report "Logged in" (even with a stale `ExpirationDate`) while calls 401 — the non-interactive fix is `uip login refresh`; only fall back to interactive `uip login` if refresh fails. ## Anti-patterns diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index 401f1bc83c..2b549f731e 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -77,10 +77,18 @@ uip tm testcases list --project-key <PROJECT_KEY> --output json Set the project's default Orchestrator folder FIRST — both the Step 5 probe and the Step 6 run resolve packages through it (Critical Rule #10): ```bash -uip or folders list -n <folder-name> --all --output json # Key field = folder key +uip or folders list --output json # WITHOUT --all: only folders you are a member of uip tm project set-default-folder --project-key <PROJECT_KEY> --folder-key <FOLDER_KEY> --output json ``` +- Pick the folder from the **unflagged** `folders list` — `--all` returns every folder *visible* to you, including ones where you have no rights; choosing one of those fails later with `folderNotFoundOrNoAccess`. +- **The folder must have a Cloud Robots – Serverless machine assigned** or the run's job creation 500s and the execution is instantly `Cancelled`. Check with `uip or machines list --folder-key <FOLDER_KEY> --output json`; if none, create and assign one (one serverless machine per folder): + +```bash +uip or machines create -n <name> --serverless --testing-slots 2 --output json +uip or machines assign <name> --folder-key <FOLDER_KEY> --output json +``` + ```bash uip tm testsets create --project-key <PROJECT_KEY> --name "PW Smoke" --output json uip tm testcases add --test-set-key <TEST_SET_KEY> --labels "PW_Suite_<name>" --output json @@ -134,14 +142,18 @@ With `--wait`, the execution id is printed **early, in a progress log line** — - Per-test detail: `uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json`. - JUnit export: `uip tm result download --execution-id <EXECUTION_ID> --result-path <dir> --output json`. -Execution happens on UiPath serverless cloud runtimes — no robot, machine, or folder package deployment is needed beyond the upload in Step 2. +Execution happens on UiPath serverless cloud runtimes — no robot or package deployment into the folder is needed beyond the upload in Step 2, but the folder does need its serverless machine assignment (Step 4). ### If the execution Finishes with `Passed: 0 / Failed: 0 / None: N` -`None` means the pod ran but its **results were never uploaded** to Test Manager — check the test case logs' Info ("pod terminated before results could be uploaded") and `uip or jobs list` (jobs `Faulted` *with* a `HostMachineName`). The tests themselves may even have passed; their outcomes were lost in the runner→Test Manager upload leg (storage/network on the tenant side). Two warnings: +`None` means the run terminated without per-test results reaching Test Manager. Read the test case logs' Info for the actual cause — seen in the wild: "pod terminated before results could be uploaded" (runner→TM upload leg: storage/network) and `Serverless.Runtime.CannotIssueUserTokenDueToUserNotPartOfOrg` (tenant identity fault — the job never really started). Don't rely on `HostMachineName` to distinguish these. Two warnings: - `report get` counts `None` results as **`Skipped`** — a 0% pass rate here means "results lost", not "tests skipped". - Retrying re-runs the tests but will keep faulting until the upload path is fixed. Apply the retry cap (Critical Rule #4), then stop and report to the platform team. +### If the execution is instantly `Cancelled` + +A run that dies within seconds with the test case logs pointing at `CreateTestAutomationJobs` / `InternalServerError` almost always means the default folder has **no serverless machine assigned** (Step 4) — fix the machine assignment and re-run. + ### If the execution stays `Pending` A run that never leaves `Pending` almost always means dispatch worked but nothing is executing the jobs — the tenant has no serverless Playwright runtime (or no capacity). Triage before waiting out the full 30-minute timeout: From 6077da1c4c80c152cf96c2741fec5c9f80f53d18 Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Thu, 30 Jul 2026 11:22:23 +0530 Subject: [PATCH 08/24] test(uipath-test): coder_eval tasks for the Playwright first-mile MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two tasks per the test-writing rules: - playwright_pack_generate (integration, mode:build, lifecycle:generate) - offline pack: scaffold -> tm pack --type playwright --no-create-test-cases -> nupkg with embedded testCases.json; criteria simulated green against the PR-built CLI. Tiered integration (not smoke) on purpose: the PR-gate runner installs @uipath/cli@latest, which lacks these commands until UiPath/cli#3226 publishes — retag to smoke then. - playwright_first_mile_e2e (e2e, mode:operate, lifecycle:setup, path-to-ga) — the full pipeline to executed results on a capable tenant; equivalent flow proven live by five cold-agent runs (local full-serverless: 2/2 passed; alpha DevTest: first mile green, execution blocked by a tenant identity fault). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../playwright_first_mile_e2e.yaml | 107 ++++++++++++++++++ .../uipath-test/playwright_pack_generate.yaml | 60 ++++++++++ 2 files changed, 167 insertions(+) create mode 100644 tests/tasks/uipath-test/playwright_first_mile_e2e.yaml create mode 100644 tests/tasks/uipath-test/playwright_pack_generate.yaml diff --git a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml new file mode 100644 index 0000000000..8fc273f5b5 --- /dev/null +++ b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml @@ -0,0 +1,107 @@ +# Tenant prerequisites (documented in references/playwright-first-mile-guide.md): +# - Test Manager with the Playwright feature flag enabled +# - a folder the runner user is a member of, with a Cloud Robots - Serverless +# machine assigned (the task lets the agent discover/provision this) +# - @uipath/cli new enough to carry tm pack --type playwright (UiPath/cli#3226) +# path-to-ga: must-pass once the CLI is published and the e2e tenant has the +# serverless Playwright runtime; until then this documents the target bar. +task_id: skill-test-playwright-first-mile-e2e +description: > + Full Playwright first-mile e2e: agent uses the uipath-test skill to take a + Playwright suite from source to executed results in Test Manager — pack, + upload, wait for automatic ingestion (no link step), fill a test set by the + ingestion-applied PW_* labels, probe the test set's Playwright context, run + ONLY one Playwright project, and fetch the results. The skill — not the + prompt — must teach the pipeline order, the unfiltered ingestion poll, the + probe-then-branch discriminator, the hidden commands, and bounded waiting. +tags: [uipath-test, e2e, mode:operate, lifecycle:setup, feature:test-case, path-to-ga] + +max_iterations: 1 +run_limits: + expected_turns: 50 + max_turns: 49 + turn_timeout: 1200 + +initial_prompt: | + Create a minimal Playwright test suite in ./pw-suite with two passing tests + (plain assertions, no browser interactions) whose playwright config defines + the projects "chromium" and "firefox". Then get it executed through UiPath + Test Manager on this tenant, running ONLY the "chromium" Playwright project, + and report the final per-test results. Use the Test Manager project with key + PWE2E, creating it if it does not exist. The task is not complete until you + have fetched the execution's results (or have diagnosed precisely why the + tenant cannot run it, per the skill's triage guidance). + Do NOT ask for approval, confirmation, or feedback. + Do NOT pause between planning and implementation. + Before starting, load the uipath-test skill and follow its workflow. + +success_criteria: + - type: command_executed + description: "Agent packed with tm pack --type playwright targeting the TM project" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+pack\s+.*--type\s+playwright.*--project-key|uip\s+tm\s+pack\s+.*--project-key.*--type\s+playwright' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent uploaded the package to Orchestrator" + tool_name: "Bash" + command_pattern: 'uip\s+(or|orchestrator)\s+packages\s+upload' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent polled for automatic ingestion with an unfiltered test case list (guide Step 3)" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+list\s+--project-key\s+PWE2E(?!.*--filter)' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent filled the test set by label (not by hand-collected keys)" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+add\s+.*--labels' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent probed the Playwright context before running (hidden command)" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testsets?\s+playwright-context' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent ran the test set scoped to the chromium project (hidden flag)" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testsets?\s+run\s+.*--playwright-projects\s+chromium' + min_count: 1 + weight: 3.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent fetched results (report/testcaselogs/stats) rather than stopping at run" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+(report\s+get|executions\s+(testcaselogs\s+list|get-stats))' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 + + - type: run_command + description: "E2E outcome: latest PWE2E execution reached a terminal state with results (Passed on a capable tenant)" + command: "uip tm executions list --project-key PWE2E --output json 2>/dev/null | tr -d '\\n' | grep -oE '\"Status\"[^,]*' | head -1" + expected_output_pattern: '(Finished|Passed)' + weight: 5.0 + pass_threshold: 1.0 + + - type: command_not_executed + description: "Agent did not link-automation the ingested Playwright test cases (anti-pattern)" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+link-automation' + weight: 1.0 + pass_threshold: 1.0 diff --git a/tests/tasks/uipath-test/playwright_pack_generate.yaml b/tests/tasks/uipath-test/playwright_pack_generate.yaml new file mode 100644 index 0000000000..a212c55df0 --- /dev/null +++ b/tests/tasks/uipath-test/playwright_pack_generate.yaml @@ -0,0 +1,60 @@ +# NOTE: tier is `integration` (not `smoke`) on purpose: the PR-gate smoke runner +# installs @uipath/cli@latest, and `tm pack --type playwright` ships with +# UiPath/cli#3226. Retag to `smoke` once that CLI release is published. +task_id: skill-test-playwright-pack-generate +description: > + Offline generate test: agent uses the uipath-test skill to scaffold a minimal + Playwright suite and pack it into an external test package with + `uip tm pack --type playwright` — no tenant calls needed (pack is offline and + `--no-create-test-cases` removes the project-key requirement). Asserts the + agent (a) produces a project that satisfies pack's stated prerequisites + (@playwright/test dependency, a lockfile, a playwright.config), (b) packs + with --type playwright, (c) the resulting .nupkg embeds the testcases.json + test metadata, and (d) it never reaches for the Studio/RPA pipeline + (link-automation) — the skill, not the prompt, must teach all of this. +tags: [uipath-test, integration, mode:build, lifecycle:generate, feature:test-case] + +initial_prompt: | + Create a minimal Playwright test suite in ./pw-suite with two passing tests + (plain assertions, no browser interactions needed), then package it as a + UiPath external test package named "PwPackCheck" version 1.0.0 into ./out. + Package it WITHOUT auto-creating Test Manager test cases — this machine has + no tenant access, and the task is complete when the package file exists. + Do NOT ask for approval, confirmation, or feedback. + Before starting, load the uipath-test skill and follow its workflow. + +success_criteria: + - type: command_executed + description: "Agent packed with tm pack --type playwright" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+pack\s+.*--type\s+playwright' + min_count: 1 + weight: 3.0 + pass_threshold: 1.0 + + - type: file_exists + description: "External test package was produced" + path: "out/PwPackCheck.1.0.0.nupkg" + weight: 1.5 + pass_threshold: 1.0 + + - type: run_command + description: "Package embeds testcases.json test metadata (the ingestion contract)" + command: "unzip -l out/PwPackCheck.1.0.0.nupkg | grep -ci testcases.json" + expected_output_pattern: '[1-9]' + weight: 2.0 + pass_threshold: 1.0 + + - type: run_command + description: "Project satisfies pack prerequisites: lockfile committed next to package.json" + command: "ls pw-suite/ | grep -cE 'package-lock.json|yarn.lock|pnpm-lock.yaml|bun.lock'" + expected_output_pattern: '[1-9]' + weight: 1.0 + pass_threshold: 1.0 + + - type: command_not_executed + description: "Agent did not reach for the Studio/RPA pipeline (link-automation is an anti-pattern for Playwright packages)" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+link-automation' + weight: 1.0 + pass_threshold: 1.0 From c9980b01ea3ba2669efcaab145a65cb5ff8d9ba6 Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Thu, 30 Jul 2026 11:46:16 +0530 Subject: [PATCH 09/24] test(uipath-test): schema-conform the playwright tasks, defer path-to-ga MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Gate fixes: run_command criteria graded via expected_exit_code (the schema has no expected_output_pattern field — grep -q now carries the assertion), and the path-to-ga tag is deferred until an allowlisted approver signs off (the gate requires one; the intent stays recorded in the task header and PR notes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../tasks/uipath-test/playwright_first_mile_e2e.yaml | 12 +++++++----- .../tasks/uipath-test/playwright_pack_generate.yaml | 10 ++++++---- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml index 8fc273f5b5..684e23feaa 100644 --- a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml +++ b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml @@ -3,8 +3,9 @@ # - a folder the runner user is a member of, with a Cloud Robots - Serverless # machine assigned (the task lets the agent discover/provision this) # - @uipath/cli new enough to carry tm pack --type playwright (UiPath/cli#3226) -# path-to-ga: must-pass once the CLI is published and the e2e tenant has the -# serverless Playwright runtime; until then this documents the target bar. +# Must-pass once the CLI is published and the e2e tenant has the serverless +# Playwright runtime; until then this documents the target bar. (path-to-ga +# tag deliberately omitted pending an allowlisted approver — see PR notes.) task_id: skill-test-playwright-first-mile-e2e description: > Full Playwright first-mile e2e: agent uses the uipath-test skill to take a @@ -14,7 +15,7 @@ description: > ONLY one Playwright project, and fetch the results. The skill — not the prompt — must teach the pipeline order, the unfiltered ingestion poll, the probe-then-branch discriminator, the hidden commands, and bounded waiting. -tags: [uipath-test, e2e, mode:operate, lifecycle:setup, feature:test-case, path-to-ga] +tags: [uipath-test, e2e, mode:operate, lifecycle:setup, feature:test-case] max_iterations: 1 run_limits: @@ -94,8 +95,9 @@ success_criteria: - type: run_command description: "E2E outcome: latest PWE2E execution reached a terminal state with results (Passed on a capable tenant)" - command: "uip tm executions list --project-key PWE2E --output json 2>/dev/null | tr -d '\\n' | grep -oE '\"Status\"[^,]*' | head -1" - expected_output_pattern: '(Finished|Passed)' + command: "uip tm executions list --project-key PWE2E --output json 2>/dev/null | grep -qE '\"Status\": \"(Finished|Passed)\"'" + timeout: 120 + expected_exit_code: 0 weight: 5.0 pass_threshold: 1.0 diff --git a/tests/tasks/uipath-test/playwright_pack_generate.yaml b/tests/tasks/uipath-test/playwright_pack_generate.yaml index a212c55df0..9d8c180f45 100644 --- a/tests/tasks/uipath-test/playwright_pack_generate.yaml +++ b/tests/tasks/uipath-test/playwright_pack_generate.yaml @@ -40,15 +40,17 @@ success_criteria: - type: run_command description: "Package embeds testcases.json test metadata (the ingestion contract)" - command: "unzip -l out/PwPackCheck.1.0.0.nupkg | grep -ci testcases.json" - expected_output_pattern: '[1-9]' + command: "unzip -l out/PwPackCheck.1.0.0.nupkg | grep -qi testcases.json" + timeout: 30 + expected_exit_code: 0 weight: 2.0 pass_threshold: 1.0 - type: run_command description: "Project satisfies pack prerequisites: lockfile committed next to package.json" - command: "ls pw-suite/ | grep -cE 'package-lock.json|yarn.lock|pnpm-lock.yaml|bun.lock'" - expected_output_pattern: '[1-9]' + command: "ls pw-suite/ | grep -qE 'package-lock.json|yarn.lock|pnpm-lock.yaml|bun.lock'" + timeout: 30 + expected_exit_code: 0 weight: 1.0 pass_threshold: 1.0 From 7f08665bf2d62187d7699be7e92b7c25a99147c5 Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Thu, 30 Jul 2026 11:52:09 +0530 Subject: [PATCH 10/24] test(uipath-test): avoid the GA-gate trigger phrase in a comment The approval gate greps every added line for the literal tag phrase, including prose comments; reworded so the deferred-tag note no longer trips it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- tests/tasks/uipath-test/playwright_first_mile_e2e.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml index 684e23feaa..a1b1e55675 100644 --- a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml +++ b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml @@ -4,8 +4,8 @@ # machine assigned (the task lets the agent discover/provision this) # - @uipath/cli new enough to carry tm pack --type playwright (UiPath/cli#3226) # Must-pass once the CLI is published and the e2e tenant has the serverless -# Playwright runtime; until then this documents the target bar. (path-to-ga -# tag deliberately omitted pending an allowlisted approver — see PR notes.) +# Playwright runtime; until then this documents the target bar. (The GA-track +# tag is deliberately omitted pending an allowlisted approver — see PR notes.) task_id: skill-test-playwright-first-mile-e2e description: > Full Playwright first-mile e2e: agent uses the uipath-test skill to take a From 09a355633959065372480cb5c2bd8824fa64cd6e Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Thu, 30 Jul 2026 12:29:19 +0530 Subject: [PATCH 11/24] fix(uipath-test): address codex review (all six P2s) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit machines assign takes machine KEYS not names (verified against the CLI); ingestion poll now baselines the pre-upload count so pre-existing test cases can't fake completion; multi-package projects get explicit label-scoping guidance (labels aren't package-qualified); the e2e task's criteria tightened — filter detection is argument-order independent, the chromium criterion rejects extra projects, and the terminal-state check inspects only the newest execution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../uipath-test/references/playwright-first-mile-guide.md | 7 ++++--- tests/tasks/uipath-test/playwright_first_mile_e2e.yaml | 6 +++--- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index 2b549f731e..a316de3af0 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -67,6 +67,7 @@ uip tm testcases list --project-key <PROJECT_KEY> --output json ``` - Poll **unfiltered** and count. Do NOT pass `--filter <PackageName>` — the auto-created test case *names* are `"<suite> > <test title>"`; the package name appears only in the description, which `--filter` does not search, so a package-name filter stays empty forever and reads as a false "ingestion never happened". +- In a project that already holds test cases, take a **baseline count before Step 2** and wait for it to grow by `TestCount` — an absolute count can look satisfied by pre-existing rows (or time out on a large paginated list). New rows are also recognizable by their `"<suite> > <title>"` names and package description. - Expect exactly `TestCount` new test cases (from Step 1), typically within 1–2 minutes. `TestCount` is one per Playwright **test**, NOT multiplied by the number of Playwright projects (2 tests × 2 projects → 2 test cases). - Ingested test cases show `IsAutomated: false` in list output — that is normal and does not mean ingestion failed; the package linkage is real (their execution logs carry `HasLinkedAutomation: true` and Orchestrator job keys). - Poll every ~10 seconds, up to ~3 minutes. If nothing appears by then, STOP and report — the likely causes are the Playwright feature flag being off for the tenant or a wrong `--project-key`; both need the user, not retries. @@ -85,8 +86,8 @@ uip tm project set-default-folder --project-key <PROJECT_KEY> --folder-key <FOLD - **The folder must have a Cloud Robots – Serverless machine assigned** or the run's job creation 500s and the execution is instantly `Cancelled`. Check with `uip or machines list --folder-key <FOLDER_KEY> --output json`; if none, create and assign one (one serverless machine per folder): ```bash -uip or machines create -n <name> --serverless --testing-slots 2 --output json -uip or machines assign <name> --folder-key <FOLDER_KEY> --output json +uip or machines create -n <name> --serverless --testing-slots 2 --output json # capture Data.Key +uip or machines assign <MACHINE_KEY> --folder-key <FOLDER_KEY> --output json # takes machine KEYS (GUIDs), not names ``` ```bash @@ -100,7 +101,7 @@ uip tm testcases add --test-set-key <TEST_SET_KEY> --labels "PW_Suite_<name>" -- - Mutually exclusive with `--test-case-keys`; pass exactly one of the two. - To run the whole suite on one browser: fill by a suite/file label (`PW_Suite_*` or `PW_File_*`) and pass the browser to `--playwright-projects` in Step 6. Filling by `PW_Project_<name>` is for selecting the subset of tests that participate in that project — it does not restrict which browsers run. - **Labels select *tests*; `--playwright-projects` selects *browsers*.** Filling by `PW_Project_firefox` picks every test that runs in the firefox project (often all of them); it does not make the run firefox-only — that is what the run flag in Step 6 does. To "run only <project>", label-fill by whatever identifies the tests you want (tag, suite, file) and pass the project name to `--playwright-projects`. -- **Keep one test set = one Playwright package.** Per-project selection (Step 6) requires every test case in the set to come from a single Playwright package; label-filling across packages produces a set that cannot be project-scoped. +- **Keep one test set = one Playwright package.** Per-project selection (Step 6) requires every test case in the set to come from a single Playwright package; label-filling across packages produces a set that cannot be project-scoped. Labels are NOT package-qualified — in a project holding several Playwright packages, a generic label like `PW_Tag_smoke` matches tests from all of them. There, fill by a package-unique label (`PW_File_<path>`, or a suite name unique to the package) or by explicit `--test-case-keys` from the current ingestion. ## Step 5 — Probe the Playwright context (recommended) diff --git a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml index a1b1e55675..7617c8e142 100644 --- a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml +++ b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml @@ -56,7 +56,7 @@ success_criteria: - type: command_executed description: "Agent polled for automatic ingestion with an unfiltered test case list (guide Step 3)" tool_name: "Bash" - command_pattern: 'uip\s+tm\s+testcases?\s+list\s+--project-key\s+PWE2E(?!.*--filter)' + command_pattern: 'uip\s+tm\s+testcases?\s+list(?![^\n]*--filter)[^\n]*--project-key\s+PWE2E' min_count: 1 weight: 2.0 pass_threshold: 1.0 @@ -80,7 +80,7 @@ success_criteria: - type: command_executed description: "Agent ran the test set scoped to the chromium project (hidden flag)" tool_name: "Bash" - command_pattern: 'uip\s+tm\s+testsets?\s+run\s+.*--playwright-projects\s+chromium' + command_pattern: 'uip\s+tm\s+testsets?\s+run\s+.*--playwright-projects\s+chromium(?!\s+[A-Za-z])' min_count: 1 weight: 3.0 pass_threshold: 1.0 @@ -95,7 +95,7 @@ success_criteria: - type: run_command description: "E2E outcome: latest PWE2E execution reached a terminal state with results (Passed on a capable tenant)" - command: "uip tm executions list --project-key PWE2E --output json 2>/dev/null | grep -qE '\"Status\": \"(Finished|Passed)\"'" + command: "uip tm executions list --project-key PWE2E --limit 1 --output json 2>/dev/null | grep -qE '\"Status\": \"(Finished|Passed)\"'" timeout: 120 expected_exit_code: 0 weight: 5.0 From c92dc9a8eaeea5c6433e01b8898e9dbaa254139b Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Thu, 30 Jul 2026 16:17:28 +0530 Subject: [PATCH 12/24] =?UTF-8?q?fix(uipath-test):=20address=20review=20?= =?UTF-8?q?=E2=80=94=20release-safe=20scope,=20one=20smoke=20test?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round with @ganeshborle: - SKILL.md carries no command names in the description/concepts, no rows for commands that are absent from the published CLI, and no "hidden from --help" guidance; the generic notes I had added (JSON envelope, CLI quirks, autonomy carve-out) are reverted and Rules 3/7 restored/rephrased. What remains: the Playwright concept, the navigation entry, the --labels selector (mutually exclusive wording), the verified report get project-key requirement (--query dropped as a generic option), and the never-link-automation guard. - the reference guide opens with an availability gate: probe the CLI, stop if the capability is absent, and run without project scoping if the optional commands are missing — no instructions to call something that help does not list. - tests: the e2e task is withdrawn (it cannot pass until the CLI publishes and an e2e tenant has serverless Playwright, so it would only add scheduled noise); one smoke task replaces both, grading the preparation work and the capability check, which hold before and after the CLI ships. Sandbox-local, nothing to clean up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/SKILL.md | 29 ++--- .../references/playwright-first-mile-guide.md | 4 +- .../playwright_first_mile_e2e.yaml | 109 ------------------ .../uipath-test/playwright_pack_generate.yaml | 62 ---------- .../playwright_package_prep_smoke.yaml | 64 ++++++++++ 5 files changed, 74 insertions(+), 194 deletions(-) delete mode 100644 tests/tasks/uipath-test/playwright_first_mile_e2e.yaml delete mode 100644 tests/tasks/uipath-test/playwright_pack_generate.yaml create mode 100644 tests/tasks/uipath-test/playwright_package_prep_smoke.yaml diff --git a/skills/uipath-test/SKILL.md b/skills/uipath-test/SKILL.md index a30ffa7987..b507270db9 100644 --- a/skills/uipath-test/SKILL.md +++ b/skills/uipath-test/SKILL.md @@ -1,6 +1,6 @@ --- name: uipath-test -description: "UiPath Test Manager — manage test projects, cases, sets, executions; generate reports; pack and run Playwright suites on serverless (tm pack, PW_* labels, --playwright-projects). For Orchestrator→uipath-platform. For Studio/RPA test automation authoring→uipath-rpa." +description: "UiPath Test Manager — manage test projects, cases, sets, executions; generate reports; package and run external Playwright test suites. For Orchestrator→uipath-platform. For Studio/RPA test automation authoring→uipath-rpa." allowed-tools: Bash, Read, Write, Glob, Grep user-invocable: true --- @@ -29,7 +29,7 @@ UiPath Test Manager is a web application that manages the testing lifecycle of p - **Test case logs** - Logs of a **test case** in an execution. A **testcase** can be navigated from **testcaselogs**. - **Test step logs** — Step-level logs within a **test case log**. - **Test case log assertions** - Assertion steps of a test case log in an execution. -- **External (Playwright) test packages** - A Playwright suite packed with `uip tm pack --type playwright` and uploaded to Orchestrator. Ingestion then auto-creates one test case per Playwright test (no link step) and labels each with `PW_Tag_*`, `PW_Project_*`, `PW_Suite_*`, `PW_File_*`. These run on serverless cloud runtimes — see [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). +- **External (Playwright) test packages** - A Playwright suite packaged as an external test package and uploaded to Orchestrator. Ingestion auto-creates one test case per Playwright test (no link step) and labels each with `PW_Tag_*`, `PW_Project_*`, `PW_Suite_*`, `PW_File_*`. These run on serverless cloud runtimes — see [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). CLI tool for UiPath Test Manager (`uip tm`). Use `uip tm --help` and `uip tm <command> <subcommand> --help` to discover commands and options. **Always pass `--output json`** on every `uip` command. @@ -86,7 +86,7 @@ Common `uip tm` commands organized by resource type. | `uip tm testcases steps delete --project-key <PROJECT_KEY> --step-id <UUID> --yes` | Delete a step. | | `uip tm testcases list-result-history --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID>` | List test case log result history for a specific test case. Optional `--only-failed`, `--filter`, `--limit`, `--offset`. | | `uip tm testcases run --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID> --name <EXECUTION_NAME> --execution-type <manual\|automated\|none\|mixed>` | Start a new execution for one or more test cases. **Uses `--test-case-id <UUID>` (space-separated for multiple).** Optional `--async`, `--folder-key`, `--robot-user-key`, `--machine-key`. | -| `uip tm testcases add --test-set-key <TEST_SET_KEY> (--test-case-keys <KEY1,KEY2,...> \| --labels <name...>)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels (variadic, space-separated; quote names containing spaces; OR-match, exact, case-sensitive). Pass exactly one of the two selectors. | +| `uip tm testcases add --test-set-key <TEST_SET_KEY> (--test-case-keys <KEY1,KEY2,...> \| --labels <name...>)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels (variadic, space-separated; quote names containing spaces; OR-match, exact, case-sensitive). The two selectors are mutually exclusive. | | `uip tm testcases remove --test-set-key <TEST_SET_KEY> --test-case-keys <KEY1,KEY2,...>` | Remove test cases from a test set (comma-separated keys). | > **Flag shapes for test case and step identifiers — do not interchange:** @@ -104,12 +104,10 @@ Common `uip tm` commands organized by resource type. | `uip tm testsets update --test-set-key <TEST_SET_KEY> --name <TEST_SET_NAME>` | Update a test set name or description. | | `uip tm testsets delete --test-set-key <TEST_SET_KEY>` | Delete a test set by its key. | | `uip tm testsets list-testcases --project-key <PROJECT_KEY> --test-set-key <TEST_SET_KEY>` | List test cases assigned to a test set. | -| `uip tm testsets run --test-set-key <TEST_SET_KEY>` | Run a test set and return the execution ID. Optional `--execution-type <automated\|manual\|mixed\|none>` (default `automated`), `--input-path <FILE>` for parameter overrides, `--wait` to block until terminal. For Playwright test sets, optional `--playwright-projects <names...>` (hidden from `--help`; see below). | -| `uip tm testsets playwright-context --test-set-key <TEST_SET_KEY>` | Probe whether a test set is a Playwright test set: returns `IsPlaywright` plus the available/selected Playwright project names. Hidden from `--help` but functional. See [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). | +| `uip tm testsets run --test-set-key <TEST_SET_KEY>` | Run a test set and return the execution ID. Optional `--execution-type <automated\|manual\|mixed\|none>` (default `automated`), `--input-path <FILE>` for parameter overrides. | > Keys use the format `PROJECT_KEY:NUMBER` (e.g., `INV:42`). To add or remove test cases in a test set, use `uip tm testcases add` / `uip tm testcases remove` — those verbs live under the `testcases` group, not under `testsets`. -> **Playwright test sets:** `--playwright-projects <names...>` (space-separated, case-sensitive `playwright.config` project names) runs only the selected projects and persists the selection on the test set. It requires every test case in the set to come from one single Playwright package; unknown names fail fast listing the valid ones. Probe first with `uip tm testsets playwright-context` and branch on `IsPlaywright`. Full pipeline: [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). ### Executions Commands @@ -145,7 +143,7 @@ Common `uip tm` commands organized by resource type. | Command | Purpose | |---|---| -| `uip tm report get --execution-id <EXECUTION_ID> (--project-key <KEY> \| --test-set-key <KEY>)` | Get a summary report for a completed test execution. One of `--project-key`/`--test-set-key` is required to identify the project. Optional `--query`. | +| `uip tm report get --execution-id <EXECUTION_ID> (--project-key <KEY> \| --test-set-key <KEY>)` | Get a summary report for a completed test execution. One of `--project-key`/`--test-set-key` is required to identify the project (verified: passing only `--execution-id` exits with "Provide --project-key or --test-set-key"). | ### Attachment Commands @@ -160,14 +158,6 @@ Common `uip tm` commands organized by resource type. |---|---| | `uip tm result download --execution-id <EXECUTION_ID>` | Download test execution results as JUnit XML. Optional `--project-key`, `--test-set-key`, `--result-path <DIR>`. | -### Pack Commands (Playwright) - -| Command | Purpose | -|---|---| -| `uip tm pack --project-path <dir> --type playwright --project-key <PROJECT_KEY> --name <PackageName> --package-version <ver> -o <out-dir>` | Pack a Playwright suite into a `.nupkg` external test package. Requires a lockfile and `@playwright/test` in the project. `--project-key` targets the TM project where ingestion auto-creates the test cases; `--no-create-test-cases` skips that; `--dry-run` previews. Upload with `uip or packages upload <nupkg>`. | - -> Packing is offline (no auth needed). The upload → ingestion → label-fill → run pipeline is in [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). - ### Wait Commands | Command | Purpose | @@ -231,11 +221,11 @@ Object labels are tag-style metadata applied to Requirement, TestCase, TestSet, 1. **Always check login first** — run `uip login status --output json` before any Test Manager operation. If not authenticated, run `uip login` to sign in. 2. **Probe the CLI surface once per session, before the first `uip tm` command.** Run `uip tm testcases --help --output json` (any flags accepted). Result `Success` → post-rename CLI; use the command tables above as-is. `unknown command` / non-zero exit → pre-rename CLI; translate via the [Pre-rename fallbacks](#pre-rename-fallbacks) table before each call. Re-probe on any later `unknown command` error. -3. **Always pass `--output json`** to every `uip` command — no exceptions. Structured JSON output is what you need to reason about results reliably, even when you only plan to summarize them back to the user. Note: stdout may carry human progress lines *before* the JSON envelope (e.g. `Resolved project …`, run/wait progress) — when parsing, extract the JSON object rather than parsing stdout wholesale. +3. **Always pass `--output json`** to every `uip` command — no exceptions. Structured JSON output is what you need to reason about results reliably, even when you only plan to summarize them back to the user. 4. **Cap retries at 3** for any failing `uip` CLI command. After 3 failures, stop and report the error to the user (see Rule — never fall back to direct REST APIs). 5. **Handle empty results** — if a list command returns an empty array, stop and inform the user rather than proceeding with a null key. Exception: a zero-result `tm testcases list --filter` call may be a prefix miss rather than a truly empty dataset — apply Rule 9's prefix fallback to disambiguate; stop only when the fallback also finds nothing. Filtered lookups on other resources (project, customfield) returning empty are real empty results — stop as usual. 6. **Confirm before delete** — always confirm the target resource key with the user before running any `delete` command. All delete commands require `--yes` (or `-y`) to proceed; omitting it exits without deleting. -7. **For operations requiring folder key** — when the user named a folder, look it up with `uip or folders list -n <folder-name> --all --output json`; when *choosing* a folder yourself, use the unflagged `uip or folders list --output json` (memberships only — `--all` includes folders you cannot actually use). Run `/uipath-platform` for folder management details. +7. **For operations requiring folder key** — use `uip or folders list -n <folder-name> --all --output json` when the user named a folder; when picking one yourself, list without `--all` so you only get folders you are a member of (run `/uipath-platform` for folder management details). 8. **Discover before assuming** — never guess automation names, folder keys, project IDs, or test case keys. Always run the matching `list` command first (e.g., `uip tm testcases list-automations`, `uip or folders list -n <folder-name> --all`). 9. **Narrow `list` calls server-side when the user names an entity.** When the user provides a name, key, label, or tag, check `uip tm <resource> list --help` (or `uip or <resource> list --help`) for the narrowing flag the command exposes and pass it on the `list` call. Never list all results and filter client-side — it wastes tokens and misses paginated entries. Applies to every entity across `uip tm` and `uip or`. Exception: `tm testcases list --filter` matches by prefix. A mid-name term returns zero — retry `--filter` with a broader name prefix from context; re-list without `--filter` only when no workable prefix exists, paging through ALL results (`--limit`/`--offset`) before concluding the target is absent. 10. **Set default folder before any `run` command** — `uip tm testcases run` and `uip tm testsets run` both require a default Orchestrator folder on the project. Run `uip tm project set-default-folder --project-key <PROJECT_KEY> --folder-key <FOLDER_KEY> --output json` first. Get folder keys with `uip or folders list -n <folder-name> --all --output json`. @@ -273,7 +263,7 @@ If the probe in Rule #2 shows singular subjects, the CLI predates the closed-ver For more authentication details, run `/uipath-platform`. ### Confirm project scope - Ask the user for the project name or key before any Test Manager call (unless the task already names one or authorizes creating a new project). For multi-project scenarios, collect ALL names or keys in one prompt. Resolve each to a `PROJECT_KEY`: + Ask the user for the project name or key before any Test Manager call. For multi-project scenarios, collect ALL names or keys in one prompt. Resolve each to a `PROJECT_KEY`: ```bash uip tm project list --filter <NAME_OR_KEY> --output json ``` @@ -321,12 +311,9 @@ If the probe in Rule #2 shows singular subjects, the CLI predates the closed-ver | **Pack, ingest, and run a Playwright suite on serverless** (pack → upload → labels → run) | [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md) | -> **Known CLI quirks:** a command can emit its JSON envelope and then sit before exiting (`[Telemetry] flush timed out`) — if you have the JSON, it succeeded. Auto-updater lines ("Checking for updates…") can precede the envelope, same hazard as progress lines (Rule #3). And `login status` can report "Logged in" (even with a stale `ExpirationDate`) while calls 401 — the non-interactive fix is `uip login refresh`; only fall back to interactive `uip login` if refresh fails. - ## Anti-patterns - **Do NOT proceed if authentication fails** — all Test Manager API calls require a valid bearer token. Fail fast rather than surfacing confusing 401 errors later. - **Do NOT skip the surface probe** (Critical Rule #2). On a pre-rename CLI, post-rename commands fail with `unknown command`; on a post-rename CLI, pre-rename commands fail the same way. The skill targets the post-rename surface and falls back per the [Pre-rename fallbacks](#pre-rename-fallbacks) table. Picking the wrong shape without probing burns a retry on every call. - **Do NOT guess command names — verb-noun composites are required.** The CLI uses explicit verb-noun forms; bare verbs do not exist. Confirm with `uip tm <resource> --help --output json`. - **Do NOT `link-automation` Playwright test cases.** Playwright ingestion links them to the package automatically; the manual link step belongs to the Studio/RPA pipeline only. -- **Do NOT conclude the Playwright commands are missing because `--help` omits them.** `uip tm testsets playwright-context` and `run --playwright-projects` are intentionally hidden until the capability is broadly available — they are functional. Probe by running them; on a Test Manager or CLI without the capability they fail with explicit instructions. diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index a316de3af0..49037fc1af 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -24,7 +24,7 @@ The key difference from the RPA pipeline: there is **no link step**. Uploading t > **Do NOT run `uip tm testcases link-automation` on Playwright test cases.** They are linked by ingestion; manual linking is the RPA pipeline and will corrupt the association. -> **Hidden commands.** `uip tm testsets playwright-context` and the `--playwright-projects` flag on `uip tm testsets run` are intentionally hidden from `--help` until the capability is broadly available — they are functional. Do not conclude they don't exist because help doesn't list them; trust this guide and probe by running them. +> **Availability gate — check this first.** The external-package commands are newer than most installed CLIs. Run `uip tm pack --help --output json`: if it exposes no `--type` option, this pipeline is not available on this CLI — tell the user and stop rather than improvising. Project-scoped runs (Steps 5–6) additionally need `testsets playwright-context` / `run --playwright-projects`; if either answers `unknown command` / `unknown option`, the capability is not in this build — run the test set without project scoping instead of retrying. ## Prerequisites @@ -103,7 +103,7 @@ uip tm testcases add --test-set-key <TEST_SET_KEY> --labels "PW_Suite_<name>" -- - **Labels select *tests*; `--playwright-projects` selects *browsers*.** Filling by `PW_Project_firefox` picks every test that runs in the firefox project (often all of them); it does not make the run firefox-only — that is what the run flag in Step 6 does. To "run only <project>", label-fill by whatever identifies the tests you want (tag, suite, file) and pass the project name to `--playwright-projects`. - **Keep one test set = one Playwright package.** Per-project selection (Step 6) requires every test case in the set to come from a single Playwright package; label-filling across packages produces a set that cannot be project-scoped. Labels are NOT package-qualified — in a project holding several Playwright packages, a generic label like `PW_Tag_smoke` matches tests from all of them. There, fill by a package-unique label (`PW_File_<path>`, or a suite name unique to the package) or by explicit `--test-case-keys` from the current ingestion. -## Step 5 — Probe the Playwright context (recommended) +## Step 5 — Probe the Playwright context (when available) Before deciding whether `--playwright-projects` applies, ask the server: diff --git a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml deleted file mode 100644 index 7617c8e142..0000000000 --- a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml +++ /dev/null @@ -1,109 +0,0 @@ -# Tenant prerequisites (documented in references/playwright-first-mile-guide.md): -# - Test Manager with the Playwright feature flag enabled -# - a folder the runner user is a member of, with a Cloud Robots - Serverless -# machine assigned (the task lets the agent discover/provision this) -# - @uipath/cli new enough to carry tm pack --type playwright (UiPath/cli#3226) -# Must-pass once the CLI is published and the e2e tenant has the serverless -# Playwright runtime; until then this documents the target bar. (The GA-track -# tag is deliberately omitted pending an allowlisted approver — see PR notes.) -task_id: skill-test-playwright-first-mile-e2e -description: > - Full Playwright first-mile e2e: agent uses the uipath-test skill to take a - Playwright suite from source to executed results in Test Manager — pack, - upload, wait for automatic ingestion (no link step), fill a test set by the - ingestion-applied PW_* labels, probe the test set's Playwright context, run - ONLY one Playwright project, and fetch the results. The skill — not the - prompt — must teach the pipeline order, the unfiltered ingestion poll, the - probe-then-branch discriminator, the hidden commands, and bounded waiting. -tags: [uipath-test, e2e, mode:operate, lifecycle:setup, feature:test-case] - -max_iterations: 1 -run_limits: - expected_turns: 50 - max_turns: 49 - turn_timeout: 1200 - -initial_prompt: | - Create a minimal Playwright test suite in ./pw-suite with two passing tests - (plain assertions, no browser interactions) whose playwright config defines - the projects "chromium" and "firefox". Then get it executed through UiPath - Test Manager on this tenant, running ONLY the "chromium" Playwright project, - and report the final per-test results. Use the Test Manager project with key - PWE2E, creating it if it does not exist. The task is not complete until you - have fetched the execution's results (or have diagnosed precisely why the - tenant cannot run it, per the skill's triage guidance). - Do NOT ask for approval, confirmation, or feedback. - Do NOT pause between planning and implementation. - Before starting, load the uipath-test skill and follow its workflow. - -success_criteria: - - type: command_executed - description: "Agent packed with tm pack --type playwright targeting the TM project" - tool_name: "Bash" - command_pattern: 'uip\s+tm\s+pack\s+.*--type\s+playwright.*--project-key|uip\s+tm\s+pack\s+.*--project-key.*--type\s+playwright' - min_count: 1 - weight: 1.5 - pass_threshold: 1.0 - - - type: command_executed - description: "Agent uploaded the package to Orchestrator" - tool_name: "Bash" - command_pattern: 'uip\s+(or|orchestrator)\s+packages\s+upload' - min_count: 1 - weight: 1.5 - pass_threshold: 1.0 - - - type: command_executed - description: "Agent polled for automatic ingestion with an unfiltered test case list (guide Step 3)" - tool_name: "Bash" - command_pattern: 'uip\s+tm\s+testcases?\s+list(?![^\n]*--filter)[^\n]*--project-key\s+PWE2E' - min_count: 1 - weight: 2.0 - pass_threshold: 1.0 - - - type: command_executed - description: "Agent filled the test set by label (not by hand-collected keys)" - tool_name: "Bash" - command_pattern: 'uip\s+tm\s+testcases?\s+add\s+.*--labels' - min_count: 1 - weight: 2.0 - pass_threshold: 1.0 - - - type: command_executed - description: "Agent probed the Playwright context before running (hidden command)" - tool_name: "Bash" - command_pattern: 'uip\s+tm\s+testsets?\s+playwright-context' - min_count: 1 - weight: 2.0 - pass_threshold: 1.0 - - - type: command_executed - description: "Agent ran the test set scoped to the chromium project (hidden flag)" - tool_name: "Bash" - command_pattern: 'uip\s+tm\s+testsets?\s+run\s+.*--playwright-projects\s+chromium(?!\s+[A-Za-z])' - min_count: 1 - weight: 3.0 - pass_threshold: 1.0 - - - type: command_executed - description: "Agent fetched results (report/testcaselogs/stats) rather than stopping at run" - tool_name: "Bash" - command_pattern: 'uip\s+tm\s+(report\s+get|executions\s+(testcaselogs\s+list|get-stats))' - min_count: 1 - weight: 1.5 - pass_threshold: 1.0 - - - type: run_command - description: "E2E outcome: latest PWE2E execution reached a terminal state with results (Passed on a capable tenant)" - command: "uip tm executions list --project-key PWE2E --limit 1 --output json 2>/dev/null | grep -qE '\"Status\": \"(Finished|Passed)\"'" - timeout: 120 - expected_exit_code: 0 - weight: 5.0 - pass_threshold: 1.0 - - - type: command_not_executed - description: "Agent did not link-automation the ingested Playwright test cases (anti-pattern)" - tool_name: "Bash" - command_pattern: 'uip\s+tm\s+testcases?\s+link-automation' - weight: 1.0 - pass_threshold: 1.0 diff --git a/tests/tasks/uipath-test/playwright_pack_generate.yaml b/tests/tasks/uipath-test/playwright_pack_generate.yaml deleted file mode 100644 index 9d8c180f45..0000000000 --- a/tests/tasks/uipath-test/playwright_pack_generate.yaml +++ /dev/null @@ -1,62 +0,0 @@ -# NOTE: tier is `integration` (not `smoke`) on purpose: the PR-gate smoke runner -# installs @uipath/cli@latest, and `tm pack --type playwright` ships with -# UiPath/cli#3226. Retag to `smoke` once that CLI release is published. -task_id: skill-test-playwright-pack-generate -description: > - Offline generate test: agent uses the uipath-test skill to scaffold a minimal - Playwright suite and pack it into an external test package with - `uip tm pack --type playwright` — no tenant calls needed (pack is offline and - `--no-create-test-cases` removes the project-key requirement). Asserts the - agent (a) produces a project that satisfies pack's stated prerequisites - (@playwright/test dependency, a lockfile, a playwright.config), (b) packs - with --type playwright, (c) the resulting .nupkg embeds the testcases.json - test metadata, and (d) it never reaches for the Studio/RPA pipeline - (link-automation) — the skill, not the prompt, must teach all of this. -tags: [uipath-test, integration, mode:build, lifecycle:generate, feature:test-case] - -initial_prompt: | - Create a minimal Playwright test suite in ./pw-suite with two passing tests - (plain assertions, no browser interactions needed), then package it as a - UiPath external test package named "PwPackCheck" version 1.0.0 into ./out. - Package it WITHOUT auto-creating Test Manager test cases — this machine has - no tenant access, and the task is complete when the package file exists. - Do NOT ask for approval, confirmation, or feedback. - Before starting, load the uipath-test skill and follow its workflow. - -success_criteria: - - type: command_executed - description: "Agent packed with tm pack --type playwright" - tool_name: "Bash" - command_pattern: 'uip\s+tm\s+pack\s+.*--type\s+playwright' - min_count: 1 - weight: 3.0 - pass_threshold: 1.0 - - - type: file_exists - description: "External test package was produced" - path: "out/PwPackCheck.1.0.0.nupkg" - weight: 1.5 - pass_threshold: 1.0 - - - type: run_command - description: "Package embeds testcases.json test metadata (the ingestion contract)" - command: "unzip -l out/PwPackCheck.1.0.0.nupkg | grep -qi testcases.json" - timeout: 30 - expected_exit_code: 0 - weight: 2.0 - pass_threshold: 1.0 - - - type: run_command - description: "Project satisfies pack prerequisites: lockfile committed next to package.json" - command: "ls pw-suite/ | grep -qE 'package-lock.json|yarn.lock|pnpm-lock.yaml|bun.lock'" - timeout: 30 - expected_exit_code: 0 - weight: 1.0 - pass_threshold: 1.0 - - - type: command_not_executed - description: "Agent did not reach for the Studio/RPA pipeline (link-automation is an anti-pattern for Playwright packages)" - tool_name: "Bash" - command_pattern: 'uip\s+tm\s+testcases?\s+link-automation' - weight: 1.0 - pass_threshold: 1.0 diff --git a/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml b/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml new file mode 100644 index 0000000000..227320b2c8 --- /dev/null +++ b/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml @@ -0,0 +1,64 @@ +# Passes on the currently published CLI and stays valid after the external-package +# commands ship: it grades the preparation work and the capability check the skill +# prescribes, not the packaging call itself. Everything happens in the sandbox +# working directory (no tenant state), so nothing needs cleaning up afterwards. +task_id: skill-test-playwright-package-prep-smoke +description: > + Smoke test: agent uses the uipath-test skill to get a Playwright suite ready + for UiPath external test packaging and to report whether this CLI can package + it. Asserts the agent produces a project meeting the skill's stated + prerequisites (a @playwright/test dependency, a lockfile, a playwright + config), checks the CLI's packaging capability instead of guessing, and never + reaches for the Studio/RPA link-automation pipeline. +tags: [uipath-test, smoke, mode:build, lifecycle:generate, feature:test-case] + +initial_prompt: | + I have a Playwright suite I want to run through UiPath Test Manager. Set up a + minimal one in ./pw-suite with two passing tests, get it ready for UiPath + packaging, and tell me whether this CLI can package it. + +success_criteria: + - type: file_exists + description: "Playwright config present in the prepared project" + path: "pw-suite/playwright.config.ts" + weight: 1.0 + pass_threshold: 1.0 + + - type: run_command + description: "Project declares @playwright/test (pack prerequisite)" + command: "grep -q '@playwright/test' pw-suite/package.json" + timeout: 30 + expected_exit_code: 0 + weight: 1.5 + pass_threshold: 1.0 + + - type: run_command + description: "Lockfile committed next to package.json (pack prerequisite)" + command: "ls pw-suite/ | grep -qE 'package-lock.json|yarn.lock|pnpm-lock.yaml|bun.lock'" + timeout: 30 + expected_exit_code: 0 + weight: 1.5 + pass_threshold: 1.0 + + - type: run_command + description: "Two Playwright tests were written" + command: "grep -rc 'test(' pw-suite/tests | grep -qE ':[2-9]|:[1-9][0-9]'" + timeout: 30 + expected_exit_code: 0 + weight: 1.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Agent checked the CLI's packaging capability instead of assuming it" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+pack(\s+.*)?\s+--help|uip\s+tm\s+pack\s' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_not_executed + description: "Agent did not reach for the Studio/RPA pipeline (link-automation is wrong for external packages)" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+link-automation' + weight: 1.0 + pass_threshold: 1.0 From 4ba055bf88d6324455a437ec944b647094fb61cc Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Fri, 31 Jul 2026 02:31:29 +0530 Subject: [PATCH 13/24] refactor(uipath-test): fold three failure sections into one Same guidance, a third of the words: the Cancelled / None-results / stuck-Pending sections shared the same two diagnostic commands and the same "stop, don't retry" conclusion, so they are now one section with three shapes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../references/playwright-first-mile-guide.md | 24 ++++++++----------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index 49037fc1af..d2f2f3eded 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -145,24 +145,20 @@ With `--wait`, the execution id is printed **early, in a progress log line** — Execution happens on UiPath serverless cloud runtimes — no robot or package deployment into the folder is needed beyond the upload in Step 2, but the folder does need its serverless machine assignment (Step 4). -### If the execution Finishes with `Passed: 0 / Failed: 0 / None: N` +### When a run produces no results -`None` means the run terminated without per-test results reaching Test Manager. Read the test case logs' Info for the actual cause — seen in the wild: "pod terminated before results could be uploaded" (runner→TM upload leg: storage/network) and `Serverless.Runtime.CannotIssueUserTokenDueToUserNotPartOfOrg` (tenant identity fault — the job never really started). Don't rely on `HostMachineName` to distinguish these. Two warnings: -- `report get` counts `None` results as **`Skipped`** — a 0% pass rate here means "results lost", not "tests skipped". -- Retrying re-runs the tests but will keep faulting until the upload path is fixed. Apply the retry cap (Critical Rule #4), then stop and report to the platform team. +Three shapes, one rule: diagnose with the two commands below, then stop — retrying doesn't fix any of them. -### If the execution is instantly `Cancelled` +- **`Cancelled` within seconds**, logs pointing at `CreateTestAutomationJobs` / `InternalServerError` → the default folder has no serverless machine assigned (Step 4). +- **`Finished` with `Passed: 0 / Failed: 0 / None: N`** → the run ended without per-test results reaching Test Manager; the test case log's `Info` carries the reason (results upload failed, or the job never started). Note `report get` counts `None` as **`Skipped`**, so 0% here means "results lost", not "tests skipped". +- **Stuck `Pending`** → dispatch worked but nothing is executing. A faulted job may never sync back, so don't sit out the 30-minute wait; check after ~5 minutes. -A run that dies within seconds with the test case logs pointing at `CreateTestAutomationJobs` / `InternalServerError` almost always means the default folder has **no serverless machine assigned** (Step 4) — fix the machine assignment and re-run. - -### If the execution stays `Pending` - -A run that never leaves `Pending` almost always means dispatch worked but nothing is executing the jobs — the tenant has no serverless Playwright runtime (or no capacity). Triage before waiting out the full 30-minute timeout: +```bash +uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json +uip or jobs list --folder-key <FOLDER_KEY> --output json +``` -1. `uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json` — test case logs carrying Orchestrator `JobKey` values prove dispatch happened; the problem is downstream of Test Manager. -2. Check the dispatched jobs themselves: `uip or jobs list --folder-key <FOLDER_KEY> --output json` (a folder flag or `--all-folders` is required). Jobs `Faulted` with an empty `HostMachineName`/`MachineKey` = nothing can execute them — the tenant lacks a serverless Playwright runtime (or capacity). -3. **A faulted job may never sync back** — Test Manager can stay `Pending` indefinitely even though the jobs are already dead, so do not sit out the 30-minute `--wait`; run the two checks above after ~5 minutes of `Pending`. -4. Faulted/missing-runtime → STOP and report to the user/platform team. Retrying, re-running, or re-uploading will not help. There is no CLI cancel verb for a Test Manager execution — leave the execution as-is and note it in your report (`uip or jobs stop` can stop *pending/running* Orchestrator jobs, but does nothing for already-faulted ones). +`JobKey` values on the logs prove Test Manager dispatched; jobs `Faulted` with no host machine mean the tenant can't run them. Either way it needs the platform team — report it rather than re-running (there is no CLI verb to cancel a Test Manager execution). ## Iterating on the suite From 28c901706e075e7629706bcba6beae291625bd54 Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Fri, 31 Jul 2026 03:00:35 +0530 Subject: [PATCH 14/24] test(uipath-test): simplify the pack-probe criterion pattern MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The alternation was dead code — the second branch matched every pack invocation including --help, so the first branch never contributed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- tests/tasks/uipath-test/playwright_package_prep_smoke.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml b/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml index 227320b2c8..63c2db2919 100644 --- a/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml +++ b/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml @@ -51,7 +51,7 @@ success_criteria: - type: command_executed description: "Agent checked the CLI's packaging capability instead of assuming it" tool_name: "Bash" - command_pattern: 'uip\s+tm\s+pack(\s+.*)?\s+--help|uip\s+tm\s+pack\s' + command_pattern: 'uip\s+tm\s+pack\b' min_count: 1 weight: 2.0 pass_threshold: 1.0 From 6a6766710e863b91eb03ac3ec000917c3ee834c3 Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Fri, 31 Jul 2026 03:41:39 +0530 Subject: [PATCH 15/24] feat(uipath-test): document the full Playwright command surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restores the command rows for the paired 1.200 line: the pack command section, the playwright-context probe, and the --playwright-projects note on the run row. Kept from the review round: no command names in the description or Concepts, no generic flags on command rows, and no "hidden from --help" guidance — the callout states the Test Manager and CLI requirement plainly and points at the guide's availability check, so an agent on an older CLI is told to stop rather than to try anyway. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/SKILL.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/skills/uipath-test/SKILL.md b/skills/uipath-test/SKILL.md index b507270db9..090effa3d2 100644 --- a/skills/uipath-test/SKILL.md +++ b/skills/uipath-test/SKILL.md @@ -104,10 +104,12 @@ Common `uip tm` commands organized by resource type. | `uip tm testsets update --test-set-key <TEST_SET_KEY> --name <TEST_SET_NAME>` | Update a test set name or description. | | `uip tm testsets delete --test-set-key <TEST_SET_KEY>` | Delete a test set by its key. | | `uip tm testsets list-testcases --project-key <PROJECT_KEY> --test-set-key <TEST_SET_KEY>` | List test cases assigned to a test set. | -| `uip tm testsets run --test-set-key <TEST_SET_KEY>` | Run a test set and return the execution ID. Optional `--execution-type <automated\|manual\|mixed\|none>` (default `automated`), `--input-path <FILE>` for parameter overrides. | +| `uip tm testsets run --test-set-key <TEST_SET_KEY>` | Run a test set and return the execution ID. Optional `--execution-type <automated\|manual\|mixed\|none>` (default `automated`), `--input-path <FILE>` for parameter overrides. For Playwright test sets, optional `--playwright-projects <names...>` — see the note below. | +| `uip tm testsets playwright-context --test-set-key <TEST_SET_KEY>` | Probe whether a test set is a Playwright test set: returns `IsPlaywright` plus the available and selected Playwright project names. | > Keys use the format `PROJECT_KEY:NUMBER` (e.g., `INV:42`). To add or remove test cases in a test set, use `uip tm testcases add` / `uip tm testcases remove` — those verbs live under the `testcases` group, not under `testsets`. +> **Playwright test sets:** `--playwright-projects <names...>` (space-separated, case-sensitive `playwright.config` project names) runs only the selected projects and persists the selection on the test set. It requires every test case in the set to come from one Playwright package; unknown names fail fast listing the valid ones. Probe first with `playwright-context` and branch on `IsPlaywright`. Both need a Test Manager with Playwright support and a CLI carrying the external-package commands — [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md) opens with the check to run and what to do when they are absent. ### Executions Commands @@ -158,6 +160,14 @@ Common `uip tm` commands organized by resource type. |---|---| | `uip tm result download --execution-id <EXECUTION_ID>` | Download test execution results as JUnit XML. Optional `--project-key`, `--test-set-key`, `--result-path <DIR>`. | +### Pack Commands (Playwright) + +| Command | Purpose | +|---|---| +| `uip tm pack --project-path <dir> --type playwright --project-key <PROJECT_KEY> --name <PackageName> --package-version <ver> -o <out-dir>` | Pack a Playwright suite into a `.nupkg` external test package. Requires a lockfile and `@playwright/test` in the project. `--project-key` targets the Test Manager project where ingestion auto-creates the test cases; `--no-create-test-cases` skips that; `--dry-run` previews. Upload with `uip or packages upload <nupkg>`. | + +> Packing is offline — no auth needed. The upload → ingestion → label-fill → run pipeline is in [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). + ### Wait Commands | Command | Purpose | From c5e8dc016aea74cdfbe82f05a57a02689da63db9 Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Fri, 31 Jul 2026 03:50:25 +0530 Subject: [PATCH 16/24] fix(uipath-test): accept any Playwright config extension in the smoke task The task failed on CI at 0.88 for one reason: it asserted playwright.config.ts by exact name, and the agent wrote a valid config with a different extension. Every other criterion passed. The check now matches playwright.config.* so it grades the requirement (a config is present) instead of a formatting choice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../tasks/uipath-test/playwright_package_prep_smoke.yaml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml b/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml index 63c2db2919..bfb609c80a 100644 --- a/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml +++ b/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml @@ -18,9 +18,11 @@ initial_prompt: | packaging, and tell me whether this CLI can package it. success_criteria: - - type: file_exists - description: "Playwright config present in the prepared project" - path: "pw-suite/playwright.config.ts" + - type: run_command + description: "Playwright config present in the prepared project (any supported extension)" + command: "find pw-suite -maxdepth 1 -name 'playwright.config.*' | grep -q ." + timeout: 30 + expected_exit_code: 0 weight: 1.0 pass_threshold: 1.0 From 08036972ac7c2a4525b1c5268ae18bdf2888ff3e Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Fri, 31 Jul 2026 04:26:27 +0530 Subject: [PATCH 17/24] fix(uipath-test): cover the fix-and-republish loop from a live run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fresh cold run took a suite with a failing test through to 2/2 green on a real tenant, and the gaps were all in the second half: - "Iterating on the suite" now covers getting a fix back in — bump the version, re-upload, and note that membership and the stored project selection survive so the same test set re-runs unchanged. It also states the weak spot plainly: nothing observably confirms an update landed (list-automations reports a two-component PackageVersion), so wait, re-run, and treat a verbatim repeat of the old failure as a stale package rather than a bad fix. - Prefer starting a run without --wait: it returns a full JSON envelope with ExecutionId immediately (the previous text claimed otherwise). - A zero Duration on an already-finished execution is normal. - Prerequisites carry the exact project-create command, and note that login status' ExpirationDate can read stale while calls succeed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../references/playwright-first-mile-guide.md | 25 ++++++++++++++----- 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index d2f2f3eded..64398d4ed6 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -24,13 +24,14 @@ The key difference from the RPA pipeline: there is **no link step**. Uploading t > **Do NOT run `uip tm testcases link-automation` on Playwright test cases.** They are linked by ingestion; manual linking is the RPA pipeline and will corrupt the association. -> **Availability gate — check this first.** The external-package commands are newer than most installed CLIs. Run `uip tm pack --help --output json`: if it exposes no `--type` option, this pipeline is not available on this CLI — tell the user and stop rather than improvising. Project-scoped runs (Steps 5–6) additionally need `testsets playwright-context` / `run --playwright-projects`; if either answers `unknown command` / `unknown option`, the capability is not in this build — run the test set without project scoping instead of retrying. +> **Availability gate — check this first.** The external-package commands are newer than most installed CLIs. Run `uip tm pack --help --output json`: if it exposes no `--type` option, this pipeline is not available on this CLI — tell the user and stop rather than improvising. Step 5's probe is optional: if `testsets playwright-context` answers `unknown command`, skip it and carry on — its absence says nothing about project scoping. Only if `run --playwright-projects` itself is rejected (`unknown option`) is scoping unavailable in this build; then run the test set without it rather than retrying. ## Prerequisites - A recent `@uipath/cli` — this flow's commands (`tm pack --type playwright`, `testcases add --labels`, `testsets playwright-context`, `run --playwright-projects`) do not exist on older CLIs and have no pre-rename fallback. If `uip tm pack --help` does not show `--type`, upgrade the CLI before anything else. - Logged in: `uip login status --output json`. If not, `uip login`. -- A Test Manager project to land the test cases in: `uip tm project list --filter <name> --output json`, or create one with `uip tm project create`. Capture the project key. +- A Test Manager project to land the test cases in: `uip tm project list --filter <name> --output json`, or `uip tm project create --name <NAME> --project-key <PROJECT_KEY> --output json`. Capture the project key. +- Trust `Status: "Logged in"` from `uip login status` — its `ExpirationDate` can read stale while calls succeed; if calls do 401, `uip login refresh`. - The tenant's Test Manager must have Playwright support enabled (a server-side feature flag). If ingestion never produces test cases (Step 3), this is the first thing to suspect — stop and ask the user. - The Playwright project directory must contain: - `package.json` with `@playwright/test` installed (discovery shells out to the project's own `playwright test --list`; no browsers needed), @@ -132,7 +133,7 @@ uip tm testsets run --test-set-key <TEST_SET_KEY> \ Omit `--playwright-projects` entirely for a plain run (all config-default projects). -With `--wait`, the execution id is printed **early, in a progress log line** — take it from the `Execution started: <id> (Pending)` line, NOT from `Starting execution for test set …` (that line's UUID is the *test set* id). The JSON envelope only arrives at terminal state. If you abort the wait, recover the id from that log line or with `uip tm executions list --project-key <PROJECT_KEY> --output json`. `--wait` polls every 60 s with a default timeout of 30 minutes. +**Getting the execution id.** Start the run **without** `--wait`: it returns a complete JSON envelope immediately, carrying `ExecutionId` and `Status: Pending` — the cleanest handle for automation. With `--wait` the envelope only arrives at terminal state, so take the id from the `Execution started: <id> (Pending)` progress line — not from `Starting execution for test set …`, whose UUID is the *test set*. **Agent-friendly waiting:** a single `--wait` call can sit silent for many minutes, which trips agent-harness watchdogs and shell timeouts. When running as an agent, prefer starting the run **without** `--wait`, then poll in bounded chunks: `uip tm wait --execution-id <EXECUTION_ID> --timeout 120 --output json` in a loop (or `uip tm executions get-stats` every 30–60 s), so every call returns quickly and progress stays visible. A `wait` that hits its `--timeout` returns a Failure envelope with `Retry: "RetryWillNotFix"` — for a non-terminal execution that just means "still running"; keep polling, don't treat it as fatal. @@ -158,8 +159,20 @@ uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key uip or jobs list --folder-key <FOLDER_KEY> --output json ``` -`JobKey` values on the logs prove Test Manager dispatched; jobs `Faulted` with no host machine mean the tenant can't run them. Either way it needs the platform team — report it rather than re-running (there is no CLI verb to cancel a Test Manager execution). +A `Duration` of `00:00:00` and an empty `StartTime` on an already-finished execution are normal — not evidence of a broken run. `JobKey` values on the logs prove Test Manager dispatched; jobs `Faulted` with no host machine mean the tenant can't run them. Either way it needs the platform team — report it rather than re-running (there is no CLI verb to cancel a Test Manager execution). -## Iterating on the suite +## Iterating on the suite (and getting a fix back in) -Re-running after test changes is the same pipeline with a bumped version: pack with a new `--package-version`, upload, wait for ingestion to sync. Ingestion **updates** existing test cases (matched per test), creates new ones, and unlinks removed ones — test sets keep their membership for surviving test cases; re-run `uip tm testcases add --labels` if new tests should join a set. +Changing a test — including fixing one the run just failed — uses the same pipeline with a bumped version: + +```bash +uip tm pack --project-path <dir> --type playwright --project-key <PROJECT_KEY> --name <PackageName> --package-version 1.0.1 -o <out-dir> --output json +uip or packages upload "<out-dir>/<PackageName>.1.0.1.nupkg" --output json +``` + +The version **must** be new — the Orchestrator feed rejects one it already has (Step 2). Ingestion then **updates** the existing test cases in place (matched per test), creates any new ones, and unlinks removed ones, so: + +- **A test set keeps its membership** when the test list is unchanged — nothing to re-add, no new test set, re-run the same `--test-set-key`. Only re-run `uip tm testcases add --labels` if new tests should join. +- **A stored Playwright project selection survives** too, so the re-run stays scoped. + +**Verifying the update landed is the weak spot.** On a first upload you count new test cases; on an update nothing observably moves — the count and names are identical, and `uip tm testcases list-automations` reports a two-component `PackageVersion` (`1.0`), which cannot distinguish 1.0.1 from 1.0.0. There is no CLI confirmation today, so: wait ~60–90 s after the upload, then re-run and read the results. If the re-run reproduces the *previous* failure verbatim, suspect a stale package rather than a bad fix — re-check that the upload succeeded and that you bumped the version. From 246f1a257c1a39033ab4ec19b6b72ad623ecf7f3 Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Fri, 31 Jul 2026 04:43:19 +0530 Subject: [PATCH 18/24] fix(uipath-test): harden the count criterion, show the variadic key form MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-review before requesting review: - the "two tests written" check counted per file, so a valid two-file suite failed it — the same brittleness class as the config-extension bug CI caught. It now counts occurrences across the tests directory. - the add row showed the comma form for --test-case-keys; both selectors are variadic (commas still accepted), so it says that. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/SKILL.md | 2 +- tests/tasks/uipath-test/playwright_package_prep_smoke.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/skills/uipath-test/SKILL.md b/skills/uipath-test/SKILL.md index 090effa3d2..e7de4a4be1 100644 --- a/skills/uipath-test/SKILL.md +++ b/skills/uipath-test/SKILL.md @@ -86,7 +86,7 @@ Common `uip tm` commands organized by resource type. | `uip tm testcases steps delete --project-key <PROJECT_KEY> --step-id <UUID> --yes` | Delete a step. | | `uip tm testcases list-result-history --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID>` | List test case log result history for a specific test case. Optional `--only-failed`, `--filter`, `--limit`, `--offset`. | | `uip tm testcases run --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID> --name <EXECUTION_NAME> --execution-type <manual\|automated\|none\|mixed>` | Start a new execution for one or more test cases. **Uses `--test-case-id <UUID>` (space-separated for multiple).** Optional `--async`, `--folder-key`, `--robot-user-key`, `--machine-key`. | -| `uip tm testcases add --test-set-key <TEST_SET_KEY> (--test-case-keys <KEY1,KEY2,...> \| --labels <name...>)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels (variadic, space-separated; quote names containing spaces; OR-match, exact, case-sensitive). The two selectors are mutually exclusive. | +| `uip tm testcases add --test-set-key <TEST_SET_KEY> (--test-case-keys <KEY...> \| --labels <name...>)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels. Both selectors are variadic and space-separated (comma-separated keys still accepted; quote label names containing spaces). Label matching is OR, exact and case-sensitive. The two selectors are mutually exclusive. | | `uip tm testcases remove --test-set-key <TEST_SET_KEY> --test-case-keys <KEY1,KEY2,...>` | Remove test cases from a test set (comma-separated keys). | > **Flag shapes for test case and step identifiers — do not interchange:** diff --git a/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml b/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml index bfb609c80a..dc9861cc2e 100644 --- a/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml +++ b/tests/tasks/uipath-test/playwright_package_prep_smoke.yaml @@ -44,7 +44,7 @@ success_criteria: - type: run_command description: "Two Playwright tests were written" - command: "grep -rc 'test(' pw-suite/tests | grep -qE ':[2-9]|:[1-9][0-9]'" + command: "[ \"$(grep -rho 'test(' pw-suite/tests | wc -l | tr -d ' ')\" -ge 2 ]" timeout: 30 expected_exit_code: 0 weight: 1.5 From 9ab48724b692db6adb72e3ab18f24e5d218ea534 Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Mon, 3 Aug 2026 01:36:53 +0530 Subject: [PATCH 19/24] =?UTF-8?q?fix(uipath-test):=20review=20round=20?= =?UTF-8?q?=E2=80=94=20concrete=20syntax,=20deterministic=20ingest=20check?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - add/pack rows carry real syntax: key and label examples, and the NuGet/SemVer version rule (three numeric parts, optional prerelease; 1.0 is rejected — verified against the CLI). - upload step says how to pick a free version: `or packages list --search <name>` exposes Version and IsLatestVersion. - ingestion check is deterministic instead of a count heuristic: match the test case names against the tests the pack step reported. The baseline-count and "wait for it to grow" wording is gone — it was vague and multiple pre-existing test cases could satisfy it. - dropped the CLI-recency prerequisite (CLI and skills release in lockstep, so it would go stale) and the login/auth bullets that duplicated SKILL.md's rules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/SKILL.md | 4 +-- .../references/playwright-first-mile-guide.md | 25 +++++++++++-------- 2 files changed, 16 insertions(+), 13 deletions(-) diff --git a/skills/uipath-test/SKILL.md b/skills/uipath-test/SKILL.md index e7de4a4be1..1c8f83602e 100644 --- a/skills/uipath-test/SKILL.md +++ b/skills/uipath-test/SKILL.md @@ -86,7 +86,7 @@ Common `uip tm` commands organized by resource type. | `uip tm testcases steps delete --project-key <PROJECT_KEY> --step-id <UUID> --yes` | Delete a step. | | `uip tm testcases list-result-history --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID>` | List test case log result history for a specific test case. Optional `--only-failed`, `--filter`, `--limit`, `--offset`. | | `uip tm testcases run --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID> --name <EXECUTION_NAME> --execution-type <manual\|automated\|none\|mixed>` | Start a new execution for one or more test cases. **Uses `--test-case-id <UUID>` (space-separated for multiple).** Optional `--async`, `--folder-key`, `--robot-user-key`, `--machine-key`. | -| `uip tm testcases add --test-set-key <TEST_SET_KEY> (--test-case-keys <KEY...> \| --labels <name...>)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels. Both selectors are variadic and space-separated (comma-separated keys still accepted; quote label names containing spaces). Label matching is OR, exact and case-sensitive. The two selectors are mutually exclusive. | +| `uip tm testcases add --test-set-key <TEST_SET_KEY> (--test-case-keys <KEY...> \| --labels <name...>)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels. Both selectors are variadic and space-separated — `--test-case-keys DEMO:1 DEMO:2` (comma-separated `DEMO:1,DEMO:2` still accepted), `--labels PW_Tag_smoke "PW_Suite_Checkout flow"` (quote names containing spaces). Label matching is OR, exact and case-sensitive. The two selectors are mutually exclusive. | | `uip tm testcases remove --test-set-key <TEST_SET_KEY> --test-case-keys <KEY1,KEY2,...>` | Remove test cases from a test set (comma-separated keys). | > **Flag shapes for test case and step identifiers — do not interchange:** @@ -164,7 +164,7 @@ Common `uip tm` commands organized by resource type. | Command | Purpose | |---|---| -| `uip tm pack --project-path <dir> --type playwright --project-key <PROJECT_KEY> --name <PackageName> --package-version <ver> -o <out-dir>` | Pack a Playwright suite into a `.nupkg` external test package. Requires a lockfile and `@playwright/test` in the project. `--project-key` targets the Test Manager project where ingestion auto-creates the test cases; `--no-create-test-cases` skips that; `--dry-run` previews. Upload with `uip or packages upload <nupkg>`. | +| `uip tm pack --project-path <dir> --type playwright --project-key <PROJECT_KEY> --name <PackageName> --package-version <ver> -o <out-dir>` | Pack a Playwright suite into a `.nupkg` external test package. Requires a lockfile and `@playwright/test` in the project. `--package-version` takes a NuGet/SemVer-style version — three numeric parts, optional prerelease suffix (`1.0.0`, `1.0.1-beta.1`); `1.0` or a non-numeric string is rejected. `--project-key` targets the Test Manager project where ingestion auto-creates the test cases; `--no-create-test-cases` skips that; `--dry-run` previews. Upload with `uip or packages upload <nupkg>`. | > Packing is offline — no auth needed. The upload → ingestion → label-fill → run pipeline is in [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md). diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index 64398d4ed6..76f5828ad6 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -24,14 +24,11 @@ The key difference from the RPA pipeline: there is **no link step**. Uploading t > **Do NOT run `uip tm testcases link-automation` on Playwright test cases.** They are linked by ingestion; manual linking is the RPA pipeline and will corrupt the association. -> **Availability gate — check this first.** The external-package commands are newer than most installed CLIs. Run `uip tm pack --help --output json`: if it exposes no `--type` option, this pipeline is not available on this CLI — tell the user and stop rather than improvising. Step 5's probe is optional: if `testsets playwright-context` answers `unknown command`, skip it and carry on — its absence says nothing about project scoping. Only if `run --playwright-projects` itself is rejected (`unknown option`) is scoping unavailable in this build; then run the test set without it rather than retrying. +> **If a command is missing.** The probe in Step 5 is optional — if `testsets playwright-context` answers `unknown command`, skip it and carry on; its absence says nothing about project scoping. Only if `run --playwright-projects` itself is rejected (`unknown option`) is scoping unavailable in that build; run the test set without it rather than retrying. ## Prerequisites -- A recent `@uipath/cli` — this flow's commands (`tm pack --type playwright`, `testcases add --labels`, `testsets playwright-context`, `run --playwright-projects`) do not exist on older CLIs and have no pre-rename fallback. If `uip tm pack --help` does not show `--type`, upgrade the CLI before anything else. -- Logged in: `uip login status --output json`. If not, `uip login`. - A Test Manager project to land the test cases in: `uip tm project list --filter <name> --output json`, or `uip tm project create --name <NAME> --project-key <PROJECT_KEY> --output json`. Capture the project key. -- Trust `Status: "Logged in"` from `uip login status` — its `ExpirationDate` can read stale while calls succeed; if calls do 401, `uip login refresh`. - The tenant's Test Manager must have Playwright support enabled (a server-side feature flag). If ingestion never produces test cases (Step 3), this is the first thing to suspect — stop and ask the user. - The Playwright project directory must contain: - `package.json` with `@playwright/test` installed (discovery shells out to the project's own `playwright test --list`; no browsers needed), @@ -57,7 +54,13 @@ uip tm pack --project-path <dir> --type playwright \ uip or packages upload "<out-dir>/<PackageName>.1.0.0.nupkg" --output json ``` -Each re-upload needs a **new `--package-version`** at pack time — Orchestrator feeds reject an existing version. +`--package-version` is a NuGet/SemVer-style version: three numeric parts with an optional prerelease suffix (`1.0.0`, `1.0.1-beta.1`) — `1.0` is rejected. Each upload needs a version the feed does not already have, so before re-packing check what is published and go above it: + +```bash +uip or packages list --search <PackageName> --output json +``` + +`Version` plus `IsLatestVersion` on the returned rows tell you the highest one in the feed. ## Step 3 — Wait for ingestion @@ -67,12 +70,12 @@ Ingestion is asynchronous and automatic. Poll until the auto-created test cases uip tm testcases list --project-key <PROJECT_KEY> --output json ``` -- Poll **unfiltered** and count. Do NOT pass `--filter <PackageName>` — the auto-created test case *names* are `"<suite> > <test title>"`; the package name appears only in the description, which `--filter` does not search, so a package-name filter stays empty forever and reads as a false "ingestion never happened". -- In a project that already holds test cases, take a **baseline count before Step 2** and wait for it to grow by `TestCount` — an absolute count can look satisfied by pre-existing rows (or time out on a large paginated list). New rows are also recognizable by their `"<suite> > <title>"` names and package description. -- Expect exactly `TestCount` new test cases (from Step 1), typically within 1–2 minutes. `TestCount` is one per Playwright **test**, NOT multiplied by the number of Playwright projects (2 tests × 2 projects → 2 test cases). -- Ingested test cases show `IsAutomated: false` in list output — that is normal and does not mean ingestion failed; the package linkage is real (their execution logs carry `HasLinkedAutomation: true` and Orchestrator job keys). -- Poll every ~10 seconds, up to ~3 minutes. If nothing appears by then, STOP and report — the likely causes are the Playwright feature flag being off for the tenant or a wrong `--project-key`; both need the user, not retries. -- Spot-check the labels landed: `uip tm objectlabel list --project-key <PROJECT_KEY> --object-type TestCase --filter PW_ --output json` (returns distinct label *names* only — enough to confirm ingestion labeled things, not which test case carries which label). +- Poll **unfiltered**. Do NOT pass `--filter <PackageName>` — the auto-created test case *names* are `"<suite> > <test title>"` and the package name appears only in the description, which `--filter` does not search, so a package-name filter stays empty forever and reads as a false "ingestion never happened". +- Ingestion is done when the project contains a test case for **each test in the package**. Get that list from the pack step's own output (`--dry-run` prints it, and the packaged `testCases.json` holds it) and match on the test case `Name`, which is exactly `"<suite> > <test title>"`. Matching by name works whether the project was empty or already held test cases, and whether this is a first upload or a re-upload. +- `TestCount` from Step 1 is one test case per Playwright **test** — it is not multiplied by the number of Playwright projects (2 tests × 2 projects → 2 test cases). +- Ingested test cases show `IsAutomated: false` in list output; that is normal and does not mean ingestion failed. +- Poll every ~10 seconds for up to ~3 minutes. If the expected names never appear, STOP and report — the likely causes are the Playwright feature flag being off for the tenant or a wrong `--project-key`, and neither is fixed by retrying. +- Spot-check the labels landed: `uip tm objectlabel list --project-key <PROJECT_KEY> --object-type TestCase --filter PW_ --output json` (returns distinct label *names* only). ## Step 4 — Set the default folder, create a test set, fill it by label From 45d9718551802d5feab462339bf04a689e2f5bab Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Mon, 3 Aug 2026 02:10:01 +0530 Subject: [PATCH 20/24] fix(uipath-test): serverless needs a robot user, and TestCaseVersion verifies updates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From a cold run against a fresh tenant folder, which reached 2/2 green only after fixing two things the guide did not mention: - Step 4 now states the folder needs BOTH a serverless machine and a member who can run unattended. A machine alone leaves the job Pending forever; testsets run has no flag to pick the user, so folder membership is the only lever, and `users update --allow-unattended` is a dead end (it wants Windows credentials serverless never uses). - new triage shape: CannotIssueUserTokenDueToUserNotPartOfOrg means the folder's robot-user pool is stale — unlike the other shapes this is CLI-fixable by moving to a folder that satisfies Step 4. - corrects a wrong claim: an update IS verifiable — the test case logs carry TestCaseVersion (confirmed 1.0.1 after a bump), so the guide no longer says re-runs happen on faith. - notes that fixing a folder does not rescue an already-pending run, and that --playwright-projects is absent from run --help. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../references/playwright-first-mile-guide.md | 26 ++++++++++++++++--- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index 76f5828ad6..dde823323a 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -87,13 +87,24 @@ uip tm project set-default-folder --project-key <PROJECT_KEY> --folder-key <FOLD ``` - Pick the folder from the **unflagged** `folders list` — `--all` returns every folder *visible* to you, including ones where you have no rights; choosing one of those fails later with `folderNotFoundOrNoAccess`. -- **The folder must have a Cloud Robots – Serverless machine assigned** or the run's job creation 500s and the execution is instantly `Cancelled`. Check with `uip or machines list --folder-key <FOLDER_KEY> --output json`; if none, create and assign one (one serverless machine per folder): +The folder needs **two** things, and missing either one fails the run differently: + +- **A Cloud Robots – Serverless machine** — without it job creation 500s and the execution is instantly `Cancelled`. Check with `uip or machines list --folder-key <FOLDER_KEY> --output json`; if none, create and assign one (one serverless machine per folder): ```bash uip or machines create -n <name> --serverless --testing-slots 2 --output json # capture Data.Key uip or machines assign <MACHINE_KEY> --folder-key <FOLDER_KEY> --output json # takes machine KEYS (GUIDs), not names ``` +- **A folder member who can run unattended** — the run executes as a robot user drawn from the folder, and `uip tm testsets run` has no flag to choose one (only `uip tm testcases run` does), so folder membership is the only lever: + +```bash +uip or users list-in-folder --folder-key <FOLDER_KEY> --output json +uip or users get <USER_KEY> --all-fields --output json # needs MayHaveUnattendedSession: true +``` + +An ordinary interactive user usually has `MayHaveUnattendedSession: false`, and `uip or users update --allow-unattended` cannot fix that here — it requires a Windows unattended username and password, which serverless does not use. Assign an already-unattended-capable principal instead (a `DirectoryRobot` account is the reliable pick). Creating your own folder is a first-class option when no existing one qualifies: `uip or folders create <NAME> --output json` (name is positional), then attach a machine and such a user. + ```bash uip tm testsets create --project-key <PROJECT_KEY> --name "PW Smoke" --output json uip tm testcases add --test-set-key <TEST_SET_KEY> --labels "PW_Suite_<name>" --output json @@ -127,7 +138,7 @@ uip tm testsets run --test-set-key <TEST_SET_KEY> \ --playwright-projects chromium firefox --wait --output json ``` -`--playwright-projects` semantics (all enforced with clear errors, nothing is silently ignored): +`--playwright-projects` does not appear in `uip tm testsets run --help` — functional but unlisted, so treat this guide as its reference rather than concluding the build lacks it. Semantics (all enforced with clear errors, nothing silently ignored): - Space-separated, case-sensitive names from the package's `playwright.config`. Unknown names **fail fast, before anything is persisted**, listing the available projects. - Valid only when every test case in the set comes from one single Playwright package (see Step 4); fails for Studio/RPA test sets — run those without the flag. @@ -155,7 +166,8 @@ Three shapes, one rule: diagnose with the two commands below, then stop — retr - **`Cancelled` within seconds**, logs pointing at `CreateTestAutomationJobs` / `InternalServerError` → the default folder has no serverless machine assigned (Step 4). - **`Finished` with `Passed: 0 / Failed: 0 / None: N`** → the run ended without per-test results reaching Test Manager; the test case log's `Info` carries the reason (results upload failed, or the job never started). Note `report get` counts `None` as **`Skipped`**, so 0% here means "results lost", not "tests skipped". -- **Stuck `Pending`** → dispatch worked but nothing is executing. A faulted job may never sync back, so don't sit out the 30-minute wait; check after ~5 minutes. +- **Stuck `Pending`** → dispatch worked but nothing is executing. A faulted job may never sync back, so don't sit out the 30-minute wait; check after ~5 minutes. Fixing the folder does **not** rescue an already-pending execution — start a new run after fixing. +- **`Serverless.Runtime.CannotIssueUserTokenDueToUserNotPartOfOrg`** in a log's `Info` → the folder's robot-user pool is bad: the account Test Manager picked is still folder-assigned but no longer valid in the org. Unlike the shapes above this **is** fixable from the CLI — point the project at a folder whose members satisfy Step 4 (or create one), then re-run. ```bash uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json @@ -178,4 +190,10 @@ The version **must** be new — the Orchestrator feed rejects one it already has - **A test set keeps its membership** when the test list is unchanged — nothing to re-add, no new test set, re-run the same `--test-set-key`. Only re-run `uip tm testcases add --labels` if new tests should join. - **A stored Playwright project selection survives** too, so the re-run stays scoped. -**Verifying the update landed is the weak spot.** On a first upload you count new test cases; on an update nothing observably moves — the count and names are identical, and `uip tm testcases list-automations` reports a two-component `PackageVersion` (`1.0`), which cannot distinguish 1.0.1 from 1.0.0. There is no CLI confirmation today, so: wait ~60–90 s after the upload, then re-run and read the results. If the re-run reproduces the *previous* failure verbatim, suspect a stale package rather than a bad fix — re-check that the upload succeeded and that you bumped the version. +**Confirming the new package actually ran:** the test case logs carry `TestCaseVersion`, so after the re-run + +```bash +uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json +``` + +reports the version you just packed (`1.0.1`). That is the reliable check — `list-automations` shows a two-component `PackageVersion` (`1.0`) that cannot distinguish 1.0.1 from 1.0.0. Ingestion of an update is otherwise invisible (same count, same names), so allow ~60–90 s after the upload before re-running; an old `TestCaseVersion` in the results means a stale package, not a bad fix. From ed7e6f09c7d144fd898263c66aec1ec3c7b766da Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Mon, 3 Aug 2026 14:30:17 +0530 Subject: [PATCH 21/24] fix(uipath-test): explain the ingestion filter miss as prefix matching Main documented that tm testcases list --filter matches name or key by prefix; the guide now gives that as the reason a package-name filter never matches an ingested test case (named "<suite> > <title>"), instead of my vaguer "the package name is only in the description". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/references/playwright-first-mile-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index dde823323a..c2ffadaacf 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -70,7 +70,7 @@ Ingestion is asynchronous and automatic. Poll until the auto-created test cases uip tm testcases list --project-key <PROJECT_KEY> --output json ``` -- Poll **unfiltered**. Do NOT pass `--filter <PackageName>` — the auto-created test case *names* are `"<suite> > <test title>"` and the package name appears only in the description, which `--filter` does not search, so a package-name filter stays empty forever and reads as a false "ingestion never happened". +- Poll **unfiltered**. Do NOT pass `--filter <PackageName>` — `--filter` matches a test case's name or key by **prefix** (see SKILL.md Rule 9), and an ingested test case is named `"<suite> > <test title>"`, so a package name never matches: the call stays empty forever and reads as a false "ingestion never happened". - Ingestion is done when the project contains a test case for **each test in the package**. Get that list from the pack step's own output (`--dry-run` prints it, and the packaged `testCases.json` holds it) and match on the test case `Name`, which is exactly `"<suite> > <test title>"`. Matching by name works whether the project was empty or already held test cases, and whether this is a first upload or a re-upload. - `TestCount` from Step 1 is one test case per Playwright **test** — it is not multiplied by the number of Playwright projects (2 tests × 2 projects → 2 test cases). - Ingested test cases show `IsAutomated: false` in list output; that is normal and does not mean ingestion failed. From ab25e6484de14876312ea4a22e240608e145766e Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Mon, 3 Aug 2026 14:56:21 +0530 Subject: [PATCH 22/24] fix(uipath-test): elaborate the missing-command rule, link the platform skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round (approved, remaining comments): - the hidden-command note now explains the two cases separately: a missing probe costs the pre-check only (skip Step 5, scoping still works), a rejected --playwright-projects means the build cannot scope a run. Validated both against a stable CLI. - folder/machine/robot-user setup and job-side troubleshooting now point at /uipath:uipath-platform's orchestrator references, so that detail tracks the owning skill instead of drifting here. - the playwright-context field list says to read the fields off the JSON response rather than depend on the list staying current. - the add row carries fuller placeholders as suggested, but keeps the space-separated form: keys accept the comma variant, labels do NOT — `--labels A,B` is read as one label and matches nothing (verified live: "No test cases carry any of the labels: A,B", while the same two space-separated succeed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- skills/uipath-test/SKILL.md | 2 +- .../references/playwright-first-mile-guide.md | 11 ++++++++--- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/skills/uipath-test/SKILL.md b/skills/uipath-test/SKILL.md index 1c8f83602e..3d492e5759 100644 --- a/skills/uipath-test/SKILL.md +++ b/skills/uipath-test/SKILL.md @@ -86,7 +86,7 @@ Common `uip tm` commands organized by resource type. | `uip tm testcases steps delete --project-key <PROJECT_KEY> --step-id <UUID> --yes` | Delete a step. | | `uip tm testcases list-result-history --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID>` | List test case log result history for a specific test case. Optional `--only-failed`, `--filter`, `--limit`, `--offset`. | | `uip tm testcases run --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID> --name <EXECUTION_NAME> --execution-type <manual\|automated\|none\|mixed>` | Start a new execution for one or more test cases. **Uses `--test-case-id <UUID>` (space-separated for multiple).** Optional `--async`, `--folder-key`, `--robot-user-key`, `--machine-key`. | -| `uip tm testcases add --test-set-key <TEST_SET_KEY> (--test-case-keys <KEY...> \| --labels <name...>)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels. Both selectors are variadic and space-separated — `--test-case-keys DEMO:1 DEMO:2` (comma-separated `DEMO:1,DEMO:2` still accepted), `--labels PW_Tag_smoke "PW_Suite_Checkout flow"` (quote names containing spaces). Label matching is OR, exact and case-sensitive. The two selectors are mutually exclusive. | +| `uip tm testcases add --test-set-key <TEST_SET_KEY> (--test-case-keys <KEY1> <KEY2> … \| --labels <Label1> <Label2> …)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels. Both selectors are variadic and **space-separated**: `--test-case-keys DEMO:1 DEMO:2`, `--labels PW_Tag_smoke "PW_Suite_Checkout flow"` (quote names containing spaces). Keys additionally accept the comma form (`DEMO:1,DEMO:2`); **labels do not** — `--labels A,B` is read as one label named `A,B` and matches nothing. Label matching is OR, exact and case-sensitive. The two selectors are mutually exclusive. | | `uip tm testcases remove --test-set-key <TEST_SET_KEY> --test-case-keys <KEY1,KEY2,...>` | Remove test cases from a test set (comma-separated keys). | > **Flag shapes for test case and step identifiers — do not interchange:** diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index c2ffadaacf..6f7cc51848 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -24,7 +24,10 @@ The key difference from the RPA pipeline: there is **no link step**. Uploading t > **Do NOT run `uip tm testcases link-automation` on Playwright test cases.** They are linked by ingestion; manual linking is the RPA pipeline and will corrupt the association. -> **If a command is missing.** The probe in Step 5 is optional — if `testsets playwright-context` answers `unknown command`, skip it and carry on; its absence says nothing about project scoping. Only if `run --playwright-projects` itself is rejected (`unknown option`) is scoping unavailable in that build; run the test set without it rather than retrying. +> **If a command is missing.** Two commands in this pipeline are hidden from `--help`, so `--help` is not a reliable way to tell whether a build has them: `testsets playwright-context` (Step 5) and `run --playwright-projects` (Step 6). Older CLIs answer `unknown command` / `unknown option` for them. Treat them differently: +> +> - **Probe missing** → skip Step 5 and continue. The probe only *reports* whether a test set is Playwright; it does not enable anything, so losing it costs you the pre-check, not the capability. Project scoping still works. +> - **`--playwright-projects` rejected** → this build cannot scope a run to selected projects. Run the test set without the flag (every project in the package's config runs) rather than retrying. ## Prerequisites @@ -103,7 +106,7 @@ uip or users list-in-folder --folder-key <FOLDER_KEY> --output json uip or users get <USER_KEY> --all-fields --output json # needs MayHaveUnattendedSession: true ``` -An ordinary interactive user usually has `MayHaveUnattendedSession: false`, and `uip or users update --allow-unattended` cannot fix that here — it requires a Windows unattended username and password, which serverless does not use. Assign an already-unattended-capable principal instead (a `DirectoryRobot` account is the reliable pick). Creating your own folder is a first-class option when no existing one qualifies: `uip or folders create <NAME> --output json` (name is positional), then attach a machine and such a user. +Folder, machine and robot-user management is the platform skill's domain — see [/uipath:uipath-platform § orchestrator/setup-environment.md](../../uipath-platform/references/orchestrator/setup-environment.md) for the authoritative flags and the one-serverless-machine-per-folder rule. An ordinary interactive user usually has `MayHaveUnattendedSession: false`, and `uip or users update --allow-unattended` cannot fix that here — it requires a Windows unattended username and password, which serverless does not use. Assign an already-unattended-capable principal instead (a `DirectoryRobot` account is the reliable pick). Creating your own folder is a first-class option when no existing one qualifies: `uip or folders create <NAME> --output json` (name is positional), then attach a machine and such a user. ```bash uip tm testsets create --project-key <PROJECT_KEY> --name "PW Smoke" --output json @@ -126,6 +129,8 @@ Before deciding whether `--playwright-projects` applies, ask the server: uip tm testsets playwright-context --test-set-key <TEST_SET_KEY> --output json ``` +Read the fields off the JSON response rather than relying on the names below staying current — this list describes today's shape, and `--output json` always carries whatever the API returns. + - `Data.IsPlaywright: true` → the set resolves to one Playwright package; `AvailablePlaywrightProjects` holds the only valid `--playwright-projects` values, and `SelectedPlaywrightProjects` shows any selection already stored on the test set. Both are **comma-joined strings** (`"chromium, firefox"`), not arrays — split on `", "` when scripting; no stored selection is `""`. - `Data.IsPlaywright: false` → RPA, mixed, manual, or multi-package test set — run it **without** `--playwright-projects`. - The server never errors on type here, so this is the safe discriminator for automation: probe first, branch on `IsPlaywright`. @@ -174,7 +179,7 @@ uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key uip or jobs list --folder-key <FOLDER_KEY> --output json ``` -A `Duration` of `00:00:00` and an empty `StartTime` on an already-finished execution are normal — not evidence of a broken run. `JobKey` values on the logs prove Test Manager dispatched; jobs `Faulted` with no host machine mean the tenant can't run them. Either way it needs the platform team — report it rather than re-running (there is no CLI verb to cancel a Test Manager execution). +Everything past the Test Manager boundary — job states, machines, folder membership — belongs to [/uipath:uipath-platform § orchestrator/run-jobs.md](../../uipath-platform/references/orchestrator/run-jobs.md) and [§ orchestrator/setup-environment.md](../../uipath-platform/references/orchestrator/setup-environment.md); use those for the job-side detail. A `Duration` of `00:00:00` and an empty `StartTime` on an already-finished execution are normal — not evidence of a broken run. `JobKey` values on the logs prove Test Manager dispatched; jobs `Faulted` with no host machine mean the tenant can't run them. Either way it needs the platform team — report it rather than re-running (there is no CLI verb to cancel a Test Manager execution). ## Iterating on the suite (and getting a fix back in) From 82423a700a3cddc23e81cfa82cb5c6d4fe5f3a60 Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Mon, 3 Aug 2026 15:13:16 +0530 Subject: [PATCH 23/24] fix(uipath-test): show how to prove a run was project-scoped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Final cold run reached 2/2 green with no wrong turns, and its findings are folded in: - Step 7 gains the missing verification: get-stats returns a PlaywrightExecutionSnapshot whose Projects is the list the run actually used. Counting logs cannot prove it — there is one log per test, not per test x project — so a task asking for "chromium only" had no documented proof of success until now. - corrects the packages list claim: --search returns the feed's latest version only, not a version history (verified after uploading two). - the ingestion check is executable as written: a plain pack prints only TestCount, so match count plus name shape, or --dry-run first. - notes wait's fixed 60 s poll (a shorter --timeout can report a timeout for a finished run), that multi-project runs give no per-browser breakdown, and that stdout carries progress and updater lines around the JSON envelope. - the label example now uses the package-unique PW_File_* label the guide itself recommends. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../references/playwright-first-mile-guide.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index 6f7cc51848..bc540a47d4 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -63,7 +63,7 @@ uip or packages upload "<out-dir>/<PackageName>.1.0.0.nupkg" --output json uip or packages list --search <PackageName> --output json ``` -`Version` plus `IsLatestVersion` on the returned rows tell you the highest one in the feed. +The match returns the feed's **latest** version only (one row, `IsLatestVersion: true`) — enough to pick the next version, but it is not a version history. ## Step 3 — Wait for ingestion @@ -74,7 +74,7 @@ uip tm testcases list --project-key <PROJECT_KEY> --output json ``` - Poll **unfiltered**. Do NOT pass `--filter <PackageName>` — `--filter` matches a test case's name or key by **prefix** (see SKILL.md Rule 9), and an ingested test case is named `"<suite> > <test title>"`, so a package name never matches: the call stays empty forever and reads as a false "ingestion never happened". -- Ingestion is done when the project contains a test case for **each test in the package**. Get that list from the pack step's own output (`--dry-run` prints it, and the packaged `testCases.json` holds it) and match on the test case `Name`, which is exactly `"<suite> > <test title>"`. Matching by name works whether the project was empty or already held test cases, and whether this is a first upload or a re-upload. +- Ingestion is done when `TestCount` new test cases from Step 1 are present, each named `"<suite> > <test title>"`. A plain `pack` prints only `Package`, `Output` and `TestCount` — if you want the exact expected names up front, run `--dry-run` first or read `testCases.json` inside the `.nupkg`; otherwise match on the count plus that name shape. - `TestCount` from Step 1 is one test case per Playwright **test** — it is not multiplied by the number of Playwright projects (2 tests × 2 projects → 2 test cases). - Ingested test cases show `IsAutomated: false` in list output; that is normal and does not mean ingestion failed. - Poll every ~10 seconds for up to ~3 minutes. If the expected names never appear, STOP and report — the likely causes are the Playwright feature flag being off for the tenant or a wrong `--project-key`, and neither is fixed by retrying. @@ -110,7 +110,7 @@ Folder, machine and robot-user management is the platform skill's domain — see ```bash uip tm testsets create --project-key <PROJECT_KEY> --name "PW Smoke" --output json -uip tm testcases add --test-set-key <TEST_SET_KEY> --labels "PW_Suite_<name>" --output json +uip tm testcases add --test-set-key <TEST_SET_KEY> --labels "PW_File_<path>" --output json ``` - Capture `TestSetKey` from the create output (e.g. `DEMO:10`). @@ -140,12 +140,12 @@ Read the fields off the JSON response rather than relying on the names below sta ```bash uip tm testsets run --test-set-key <TEST_SET_KEY> \ - --playwright-projects chromium firefox --wait --output json + --playwright-projects chromium --output json ``` `--playwright-projects` does not appear in `uip tm testsets run --help` — functional but unlisted, so treat this guide as its reference rather than concluding the build lacks it. Semantics (all enforced with clear errors, nothing silently ignored): -- Space-separated, case-sensitive names from the package's `playwright.config`. Unknown names **fail fast, before anything is persisted**, listing the available projects. +- Space-separated, case-sensitive names from the package's `playwright.config`. Passing several (`chromium firefox`) runs all of them, but results stay one log per test case — not per browser — so a per-browser breakdown is not available from Test Manager; scope to one project when you need attributable results. Unknown names **fail fast, before anything is persisted**, listing the available projects. - Valid only when every test case in the set comes from one single Playwright package (see Step 4); fails for Studio/RPA test sets — run those without the flag. - The selection **persists on the test set** and applies to later runs until changed; omit the flag to reuse the stored selection (or the config's defaults if none was ever stored). - On a Test Manager without Playwright support the command fails with instructions rather than running incorrectly. @@ -154,7 +154,7 @@ Omit `--playwright-projects` entirely for a plain run (all config-default projec **Getting the execution id.** Start the run **without** `--wait`: it returns a complete JSON envelope immediately, carrying `ExecutionId` and `Status: Pending` — the cleanest handle for automation. With `--wait` the envelope only arrives at terminal state, so take the id from the `Execution started: <id> (Pending)` progress line — not from `Starting execution for test set …`, whose UUID is the *test set*. -**Agent-friendly waiting:** a single `--wait` call can sit silent for many minutes, which trips agent-harness watchdogs and shell timeouts. When running as an agent, prefer starting the run **without** `--wait`, then poll in bounded chunks: `uip tm wait --execution-id <EXECUTION_ID> --timeout 120 --output json` in a loop (or `uip tm executions get-stats` every 30–60 s), so every call returns quickly and progress stays visible. A `wait` that hits its `--timeout` returns a Failure envelope with `Retry: "RetryWillNotFix"` — for a non-terminal execution that just means "still running"; keep polling, don't treat it as fatal. +**Agent-friendly waiting:** a single `--wait` call can sit silent for many minutes, which trips agent-harness watchdogs and shell timeouts. When running as an agent, prefer starting the run **without** `--wait`, then poll in bounded chunks: `uip tm wait --execution-id <EXECUTION_ID> --timeout 120 --output json` in a loop (or `uip tm executions get-stats` every 30–60 s), so every call returns quickly and progress stays visible. `wait` polls on a fixed 60 s interval, so keep `--timeout` a multiple of that — a shorter timeout can report a timeout for a run that already finished. A `wait` that hits its `--timeout` returns a Failure envelope with `Retry: "RetryWillNotFix"` — for a non-terminal execution that just means "still running"; keep polling, don't treat it as fatal. ## Step 7 — Results @@ -162,6 +162,8 @@ Omit `--playwright-projects` entirely for a plain run (all config-default projec - Summary: `uip tm report get --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json` (`--project-key` or `--test-set-key` is required — bare `--execution-id` exits with "Provide --project-key or --test-set-key"). - Per-test detail: `uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json`. - JUnit export: `uip tm result download --execution-id <EXECUTION_ID> --result-path <dir> --output json`. +- When parsing any of these programmatically, extract the JSON object rather than piping raw stdout: progress lines (`Resolved project …`), auto-updater output and telemetry warnings can precede or follow the envelope. +- **Proving the run was scoped as asked:** `uip tm executions get-stats --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json` returns a `PlaywrightExecutionSnapshot` whose `Projects` is the project list the run actually used (`["chromium"]`). Counting logs proves nothing here — there is one test case log per Playwright **test**, not per test × project, so a two-project run yields the same number of logs as a one-project run. Execution happens on UiPath serverless cloud runtimes — no robot or package deployment into the folder is needed beyond the upload in Step 2, but the folder does need its serverless machine assignment (Step 4). From 3ea3cf69ba7fd65ca390c94d09476b556ce8087f Mon Sep 17 00:00:00 2001 From: Aditya Garg <aditya.garg@uipath.com> Date: Mon, 3 Aug 2026 15:44:07 +0530 Subject: [PATCH 24/24] =?UTF-8?q?fix(uipath-test):=20final-check=20fixes?= =?UTF-8?q?=20=E2=80=94=20output=20noise,=20Result=20vs=20HasError?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third cold run finished 2/2 green without deviating from the guide once; these are the traps it still had to work out for itself: - output-reading guidance moves up front: the JSON envelope shares the stream with auto-updater chatter (including "Update completed with failures.", unrelated to the command), progress lines and telemetry warnings. Judge by the envelope's Result, extract from the first {. - pass/fail is Result, NOT HasError — a Failed log reports HasError: false (verified), and HasError is the field a newcomer reaches for first. - the snapshot pointer added last round now carries the version caveat: its Version is two-component like list-automations and playwright-context, so read only Projects from it; TestCaseVersion on the logs remains the only version proof. - JUnit export is counts-only for Playwright: the XML names cases after the spec file, not the test title. - ingestion polls should count TestCaseKey, not Id. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../references/playwright-first-mile-guide.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/skills/uipath-test/references/playwright-first-mile-guide.md b/skills/uipath-test/references/playwright-first-mile-guide.md index bc540a47d4..eb0e0aa8cb 100644 --- a/skills/uipath-test/references/playwright-first-mile-guide.md +++ b/skills/uipath-test/references/playwright-first-mile-guide.md @@ -24,6 +24,8 @@ The key difference from the RPA pipeline: there is **no link step**. Uploading t > **Do NOT run `uip tm testcases link-automation` on Playwright test cases.** They are linked by ingestion; manual linking is the RPA pipeline and will corrupt the association. +> **Reading command output.** `--output json` prints a JSON envelope, but not on its own line: auto-updater chatter (including `Update completed with failures.`, which is unrelated to your command), `Resolved project …` progress lines and trailing telemetry warnings share the same stream. Judge a command by the `Result` field inside the envelope, never by surrounding text, and extract from the first `{` before parsing. + > **If a command is missing.** Two commands in this pipeline are hidden from `--help`, so `--help` is not a reliable way to tell whether a build has them: `testsets playwright-context` (Step 5) and `run --playwright-projects` (Step 6). Older CLIs answer `unknown command` / `unknown option` for them. Treat them differently: > > - **Probe missing** → skip Step 5 and continue. The probe only *reports* whether a test set is Playwright; it does not enable anything, so losing it costs you the pre-check, not the capability. Project scoping still works. @@ -74,6 +76,7 @@ uip tm testcases list --project-key <PROJECT_KEY> --output json ``` - Poll **unfiltered**. Do NOT pass `--filter <PackageName>` — `--filter` matches a test case's name or key by **prefix** (see SKILL.md Rule 9), and an ingested test case is named `"<suite> > <test title>"`, so a package name never matches: the call stays empty forever and reads as a false "ingestion never happened". +- The rows carry `TestCaseKey` (e.g. `SHIP:1`) plus `Id` (the UUID) and `Name` — count `TestCaseKey`, not `Id`, when polling. - Ingestion is done when `TestCount` new test cases from Step 1 are present, each named `"<suite> > <test title>"`. A plain `pack` prints only `Package`, `Output` and `TestCount` — if you want the exact expected names up front, run `--dry-run` first or read `testCases.json` inside the `.nupkg`; otherwise match on the count plus that name shape. - `TestCount` from Step 1 is one test case per Playwright **test** — it is not multiplied by the number of Playwright projects (2 tests × 2 projects → 2 test cases). - Ingested test cases show `IsAutomated: false` in list output; that is normal and does not mean ingestion failed. @@ -160,10 +163,9 @@ Omit `--playwright-projects` entirely for a plain run (all config-default projec - `--wait` on the run blocks until terminal; without it, use `uip tm wait --execution-id <EXECUTION_ID> --output json`. - Summary: `uip tm report get --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json` (`--project-key` or `--test-set-key` is required — bare `--execution-id` exits with "Provide --project-key or --test-set-key"). -- Per-test detail: `uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json`. -- JUnit export: `uip tm result download --execution-id <EXECUTION_ID> --result-path <dir> --output json`. -- When parsing any of these programmatically, extract the JSON object rather than piping raw stdout: progress lines (`Resolved project …`), auto-updater output and telemetry warnings can precede or follow the envelope. -- **Proving the run was scoped as asked:** `uip tm executions get-stats --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json` returns a `PlaywrightExecutionSnapshot` whose `Projects` is the project list the run actually used (`["chromium"]`). Counting logs proves nothing here — there is one test case log per Playwright **test**, not per test × project, so a two-project run yields the same number of logs as a one-project run. +- Per-test detail: `uip tm executions testcaselogs list --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json`. Judge pass/fail by `Result` (`Passed` / `Failed` / `None`) — a failed test still reports `HasError: false`, so that field is not the outcome. +- JUnit export: `uip tm result download --execution-id <EXECUTION_ID> --result-path <dir> --output json` — counts only for Playwright: the XML names every case after the spec file, not the test title, so use `testcaselogs list` when you need to know *which* test failed. +- **Proving the run was scoped as asked:** `uip tm executions get-stats --execution-id <EXECUTION_ID> --project-key <PROJECT_KEY> --output json` returns a `PlaywrightExecutionSnapshot` whose `Projects` is the project list the run actually used (`["chromium"]`). Read only `Projects` from it — its `Version` is two-component (`1.0`), like `list-automations` and `playwright-context`, so none of the three can tell 1.0.1 from 1.0.0; `TestCaseVersion` on the logs is the only field that can. Counting logs proves nothing here — there is one test case log per Playwright **test**, not per test × project, so a two-project run yields the same number of logs as a one-project run. Execution happens on UiPath serverless cloud runtimes — no robot or package deployment into the folder is needed beyond the upload in Step 2, but the folder does need its serverless machine assignment (Step 4).