Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
eeb0bc2
feat(uipath-test): Playwright first-mile pipeline
addyGarg Jul 29, 2026
b2eab39
fix(uipath-test): correct the first-mile guide from a cold-agent run
addyGarg Jul 29, 2026
6abfef9
fix(uipath-test): round-2 cold-run findings on the first-mile guide
addyGarg Jul 29, 2026
89f5946
fix(uipath-test): round-3 findings from the full-serverless cold run
addyGarg Jul 29, 2026
8b1abf4
fix(uipath-test): bounded-poll waiting guidance for agents
addyGarg Jul 29, 2026
4e7d46b
fix(uipath-test): round-4 findings from the clean-slate cold run
addyGarg Jul 29, 2026
e1608bc
fix(uipath-test): round-5 findings from the alpha cloud cold run
addyGarg Jul 29, 2026
6077da1
test(uipath-test): coder_eval tasks for the Playwright first-mile
addyGarg Jul 30, 2026
c9980b0
test(uipath-test): schema-conform the playwright tasks, defer path-to-ga
addyGarg Jul 30, 2026
7f08665
test(uipath-test): avoid the GA-gate trigger phrase in a comment
addyGarg Jul 30, 2026
09a3556
fix(uipath-test): address codex review (all six P2s)
addyGarg Jul 30, 2026
c92dc9a
fix(uipath-test): address review — release-safe scope, one smoke test
addyGarg Jul 30, 2026
4ba055b
refactor(uipath-test): fold three failure sections into one
addyGarg Jul 30, 2026
28c9017
test(uipath-test): simplify the pack-probe criterion pattern
addyGarg Jul 30, 2026
6a67667
feat(uipath-test): document the full Playwright command surface
addyGarg Jul 30, 2026
c5e8dc0
fix(uipath-test): accept any Playwright config extension in the smoke…
addyGarg Jul 30, 2026
0803697
fix(uipath-test): cover the fix-and-republish loop from a live run
addyGarg Jul 30, 2026
246f1a2
fix(uipath-test): harden the count criterion, show the variadic key form
addyGarg Jul 30, 2026
9ab4872
fix(uipath-test): review round — concrete syntax, deterministic inges…
addyGarg Aug 2, 2026
45d9718
fix(uipath-test): serverless needs a robot user, and TestCaseVersion …
addyGarg Aug 2, 2026
ed7e6f0
fix(uipath-test): explain the ingestion filter miss as prefix matching
addyGarg Aug 3, 2026
ab25e64
fix(uipath-test): elaborate the missing-command rule, link the platfo…
addyGarg Aug 3, 2026
82423a7
fix(uipath-test): show how to prove a run was project-scoped
addyGarg Aug 3, 2026
3ea3cf6
fix(uipath-test): final-check fixes — output noise, Result vs HasError
addyGarg Aug 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 20 additions & 6 deletions skills/uipath-test/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: uipath-test
description: "UiPath Test Manager — manage test projects, cases, sets, executions; generate reports. For Orchestrator→uipath-platform. For test automation→uipath-rpa."
description: "UiPath Test Manager — manage test projects, cases, sets, executions; generate reports; package and run external Playwright test suites. For Orchestrator→uipath-platform. For Studio/RPA test automation authoring→uipath-rpa."
allowed-tools: Bash, Read, Write, Glob, Grep
user-invocable: true
---
Expand Down Expand Up @@ -29,6 +29,7 @@
- **Test case logs** - Logs of a **test case** in an execution. A **testcase** can be navigated from **testcaselogs**.
- **Test step logs** — Step-level logs within a **test case log**.
- **Test case log assertions** - Assertion steps of a test case log in an execution.
- **External (Playwright) test packages** - A Playwright suite packaged as an external test package and uploaded to Orchestrator. Ingestion auto-creates one test case per Playwright test (no link step) and labels each with `PW_Tag_*`, `PW_Project_*`, `PW_Suite_*`, `PW_File_*`. These run on serverless cloud runtimes — see [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md).

CLI tool for UiPath Test Manager (`uip tm`). Use `uip tm --help` and `uip tm <command> <subcommand> --help` to discover commands and options. **Always pass `--output json`** on every `uip` command.

Expand Down Expand Up @@ -85,7 +86,7 @@
| `uip tm testcases steps delete --project-key <PROJECT_KEY> --step-id <UUID> --yes` | Delete a step. |
| `uip tm testcases list-result-history --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID>` | List test case log result history for a specific test case. Optional `--only-failed`, `--filter`, `--limit`, `--offset`. |
| `uip tm testcases run --project-key <PROJECT_KEY> --test-case-id <TEST_CASE_ID> --name <EXECUTION_NAME> --execution-type <manual\|automated\|none\|mixed>` | Start a new execution for one or more test cases. **Uses `--test-case-id <UUID>` (space-separated for multiple).** Optional `--async`, `--folder-key`, `--robot-user-key`, `--machine-key`. |
| `uip tm testcases add --test-set-key <TEST_SET_KEY> --test-case-keys <KEY1,KEY2,...>` | Add test cases to a test set (comma-separated keys). |
| `uip tm testcases add --test-set-key <TEST_SET_KEY> (--test-case-keys <KEY1> <KEY2> … \| --labels <Label1> <Label2> …)` | Add test cases to a test set — by explicit keys, OR every test case carrying at least one of the given labels. Both selectors are variadic and **space-separated**: `--test-case-keys DEMO:1 DEMO:2`, `--labels PW_Tag_smoke "PW_Suite_Checkout flow"` (quote names containing spaces). Keys additionally accept the comma form (`DEMO:1,DEMO:2`); **labels do not** — `--labels A,B` is read as one label named `A,B` and matches nothing. Label matching is OR, exact and case-sensitive. The two selectors are mutually exclusive. |
| `uip tm testcases remove --test-set-key <TEST_SET_KEY> --test-case-keys <KEY1,KEY2,...>` | Remove test cases from a test set (comma-separated keys). |

> **Flag shapes for test case and step identifiers — do not interchange:**
Expand All @@ -103,10 +104,13 @@
| `uip tm testsets update --test-set-key <TEST_SET_KEY> --name <TEST_SET_NAME>` | Update a test set name or description. |
| `uip tm testsets delete --test-set-key <TEST_SET_KEY>` | Delete a test set by its key. |
| `uip tm testsets list-testcases --project-key <PROJECT_KEY> --test-set-key <TEST_SET_KEY>` | List test cases assigned to a test set. |
| `uip tm testsets run --test-set-key <TEST_SET_KEY>` | Run a test set and return the execution ID. Optional `--execution-type <automated\|manual\|mixed\|none>` (default `automated`), `--input-path <FILE>` for parameter overrides. |
| `uip tm testsets run --test-set-key <TEST_SET_KEY>` | Run a test set and return the execution ID. Optional `--execution-type <automated\|manual\|mixed\|none>` (default `automated`), `--input-path <FILE>` for parameter overrides. For Playwright test sets, optional `--playwright-projects <names...>` — see the note below. |
| `uip tm testsets playwright-context --test-set-key <TEST_SET_KEY>` | Probe whether a test set is a Playwright test set: returns `IsPlaywright` plus the available and selected Playwright project names. |

Check warning on line 108 in skills/uipath-test/SKILL.md

View workflow job for this annotation

GitHub Actions / uipath-test

Possibly stale `uip tm testsets playwright-context` (valid prefix: `tm testsets`)

> Keys use the format `PROJECT_KEY:NUMBER` (e.g., `INV:42`). To add or remove test cases in a test set, use `uip tm testcases add` / `uip tm testcases remove` — those verbs live under the `testcases` group, not under `testsets`.

> **Playwright test sets:** `--playwright-projects <names...>` (space-separated, case-sensitive `playwright.config` project names) runs only the selected projects and persists the selection on the test set. It requires every test case in the set to come from one Playwright package; unknown names fail fast listing the valid ones. Probe first with `playwright-context` and branch on `IsPlaywright`. Both need a Test Manager with Playwright support and a CLI carrying the external-package commands — [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md) opens with the check to run and what to do when they are absent.

### Executions Commands

| Command | Purpose |
Expand Down Expand Up @@ -141,7 +145,7 @@

| Command | Purpose |
|---|---|
| `uip tm report get --execution-id <EXECUTION_ID>` | Get a summary report for a completed test execution. Optional `--project-key`, `--test-set-key`, `--query`. |
| `uip tm report get --execution-id <EXECUTION_ID> (--project-key <KEY> \| --test-set-key <KEY>)` | Get a summary report for a completed test execution. One of `--project-key`/`--test-set-key` is required to identify the project (verified: passing only `--execution-id` exits with "Provide --project-key or --test-set-key"). |

### Attachment Commands

Expand All @@ -156,6 +160,14 @@
|---|---|
| `uip tm result download --execution-id <EXECUTION_ID>` | Download test execution results as JUnit XML. Optional `--project-key`, `--test-set-key`, `--result-path <DIR>`. |

### Pack Commands (Playwright)

| Command | Purpose |
|---|---|
| `uip tm pack --project-path <dir> --type playwright --project-key <PROJECT_KEY> --name <PackageName> --package-version <ver> -o <out-dir>` | Pack a Playwright suite into a `.nupkg` external test package. Requires a lockfile and `@playwright/test` in the project. `--package-version` takes a NuGet/SemVer-style version — three numeric parts, optional prerelease suffix (`1.0.0`, `1.0.1-beta.1`); `1.0` or a non-numeric string is rejected. `--project-key` targets the Test Manager project where ingestion auto-creates the test cases; `--no-create-test-cases` skips that; `--dry-run` previews. Upload with `uip or packages upload <nupkg>`. |

> Packing is offline — no auth needed. The upload → ingestion → label-fill → run pipeline is in [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md).

### Wait Commands

| Command | Purpose |
Expand Down Expand Up @@ -223,7 +235,7 @@
4. **Cap retries at 3** for any failing `uip` CLI command. After 3 failures, stop and report the error to the user (see Rule — never fall back to direct REST APIs).
5. **Handle empty results** — if a list command returns an empty array, stop and inform the user rather than proceeding with a null key. Exception: a zero-result `tm testcases list --filter` call may be a prefix miss rather than a truly empty dataset — apply Rule 9's prefix fallback to disambiguate; stop only when the fallback also finds nothing. Filtered lookups on other resources (project, customfield) returning empty are real empty results — stop as usual.
6. **Confirm before delete** — always confirm the target resource key with the user before running any `delete` command. All delete commands require `--yes` (or `-y`) to proceed; omitting it exits without deleting.
7. **For operations requiring folder key** — use `uip or folders list -n <folder-name> --all --output json` (run `/uipath-platform` for folder management details).
7. **For operations requiring folder key** — use `uip or folders list -n <folder-name> --all --output json` when the user named a folder; when picking one yourself, list without `--all` so you only get folders you are a member of (run `/uipath-platform` for folder management details).
8. **Discover before assuming** — never guess automation names, folder keys, project IDs, or test case keys. Always run the matching `list` command first (e.g., `uip tm testcases list-automations`, `uip or folders list -n <folder-name> --all`).
9. **Narrow `list` calls server-side when the user names an entity.** When the user provides a name, key, label, or tag, check `uip tm <resource> list --help` (or `uip or <resource> list --help`) for the narrowing flag the command exposes and pass it on the `list` call. Never list all results and filter client-side — it wastes tokens and misses paginated entries. Applies to every entity across `uip tm` and `uip or`. Exception: `tm testcases list --filter` matches by prefix. A mid-name term returns zero — retry `--filter` with a broader name prefix from context; re-list without `--filter` only when no workable prefix exists, paging through ALL results (`--limit`/`--offset`) before concluding the target is absent.
10. **Set default folder before any `run` command** — `uip tm testcases run` and `uip tm testsets run` both require a default Orchestrator folder on the project. Run `uip tm project set-default-folder --project-key <PROJECT_KEY> --folder-key <FOLDER_KEY> --output json` first. Get folder keys with `uip or folders list -n <folder-name> --all --output json`.
Expand All @@ -235,14 +247,14 @@

| Post-rename (tables above) | Pre-rename equivalent |
|---|---|
| `uip tm testcases <verb>` | `uip tm testcase <verb>` |

Check warning on line 250 in skills/uipath-test/SKILL.md

View workflow job for this annotation

GitHub Actions / uipath-test

Possibly stale `uip tm testcase` (valid prefix: `tm`)
| `uip tm testsets <verb>` | `uip tm testset <verb>` |

Check warning on line 251 in skills/uipath-test/SKILL.md

View workflow job for this annotation

GitHub Actions / uipath-test

Possibly stale `uip tm testset` (valid prefix: `tm`)
| `uip tm executions <verb>` | `uip tm execution <verb>` |

Check warning on line 252 in skills/uipath-test/SKILL.md

View workflow job for this annotation

GitHub Actions / uipath-test

Possibly stale `uip tm execution` (valid prefix: `tm`)
| `uip tm testcases run` | `uip tm testcase execute` |

Check warning on line 253 in skills/uipath-test/SKILL.md

View workflow job for this annotation

GitHub Actions / uipath-test

Possibly stale `uip tm testcase execute` (valid prefix: `tm`)
| `uip tm testsets run` | `uip tm testset execute` |

Check warning on line 254 in skills/uipath-test/SKILL.md

View workflow job for this annotation

GitHub Actions / uipath-test

Possibly stale `uip tm testset execute` (valid prefix: `tm`)
| `uip tm testcases add --test-set-key … --test-case-keys …` | `uip tm testset add-testcases --test-set-key … --test-case-keys …` |

Check warning on line 255 in skills/uipath-test/SKILL.md

View workflow job for this annotation

GitHub Actions / uipath-test

Possibly stale `uip tm testset add-testcases` (valid prefix: `tm`)
| `uip tm testcases remove --test-set-key … --test-case-keys …` | `uip tm testset remove-testcases …` |

Check warning on line 256 in skills/uipath-test/SKILL.md

View workflow job for this annotation

GitHub Actions / uipath-test

Possibly stale `uip tm testset remove-testcases …` (valid prefix: `tm`)
| `uip tm executions testcaselogs list` | `uip tm execution list-testcaselogs` |

Check warning on line 257 in skills/uipath-test/SKILL.md

View workflow job for this annotation

GitHub Actions / uipath-test

Possibly stale `uip tm execution list-testcaselogs` (valid prefix: `tm`)

`uip tm wait`, `tm testcaselog`, `tm report`, `tm result`, `tm attachment`, `tm project`, `tm user`, `tm requirement` are unchanged on both surfaces.

Expand Down Expand Up @@ -305,11 +317,13 @@
| I want to... | Start here |
|---|---|
| **Generate a shareable test report** (tester or release manager view) | [references/test-result-report-guide.md](references/test-result-report-guide.md) |
| **Publish a project and link it to a Test Manager test case** | [references/publish-and-link-guide.md](references/publish-and-link-guide.md) |
| **Publish a project and link it to a Test Manager test case** (Studio/RPA) | [references/publish-and-link-guide.md](references/publish-and-link-guide.md) |
| **Pack, ingest, and run a Playwright suite on serverless** (pack → upload → labels → run) | [references/playwright-first-mile-guide.md](references/playwright-first-mile-guide.md) |


## Anti-patterns

- **Do NOT proceed if authentication fails** — all Test Manager API calls require a valid bearer token. Fail fast rather than surfacing confusing 401 errors later.
- **Do NOT skip the surface probe** (Critical Rule #2). On a pre-rename CLI, post-rename commands fail with `unknown command`; on a post-rename CLI, pre-rename commands fail the same way. The skill targets the post-rename surface and falls back per the [Pre-rename fallbacks](#pre-rename-fallbacks) table. Picking the wrong shape without probing burns a retry on every call.
- **Do NOT guess command names — verb-noun composites are required.** The CLI uses explicit verb-noun forms; bare verbs do not exist. Confirm with `uip tm <resource> --help --output json`.
- **Do NOT `link-automation` Playwright test cases.** Playwright ingestion links them to the package automatically; the manual link step belongs to the Studio/RPA pipeline only.
Loading
Loading