From 772d335a36f1894094e9ce8f728ac0adcf5ddcc3 Mon Sep 17 00:00:00 2001 From: Aditya Garg Date: Fri, 31 Jul 2026 02:32:44 +0530 Subject: [PATCH 1/3] test(uipath-test): Playwright integration and e2e tasks (hold until CLI ships) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two tasks for the Playwright first mile, deliberately kept out of the skill PR because they cannot pass until the external-package commands reach npm `latest` (the eval runner installs @uipath/cli@latest): - ingest-and-select (integration): pack, upload, ingestion, label fill, and the single-package check — stops before execution, so it needs only a Playwright-enabled tenant, no serverless runtime. - first-mile (e2e): the same flow through a chromium-only run to real results. Both grade saved artifacts rather than tenant state, so the agent can delete the project it created without breaking the assertions, and both carry a teardown criterion so runs leave nothing behind. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../playwright_first_mile_e2e.yaml | 87 +++++++++++++++++++ ...ywright_ingest_and_select_integration.yaml | 82 +++++++++++++++++ 2 files changed, 169 insertions(+) create mode 100644 tests/tasks/uipath-test/playwright_first_mile_e2e.yaml create mode 100644 tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml diff --git a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml new file mode 100644 index 0000000000..724be822c2 --- /dev/null +++ b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml @@ -0,0 +1,87 @@ +# Prerequisites (see PR description): +# - a CLI carrying the external-package commands (npm `latest` on 1.200+) +# - the eval tenant's Test Manager has Playwright support enabled +# - a folder the runner is a member of, with a Cloud Robots - Serverless +# machine, and working serverless Playwright execution +task_id: skill-test-playwright-first-mile-e2e +description: > + Full Playwright first mile: package a suite, upload it, let ingestion + auto-create the test cases, fill a test set by label, run only one Playwright + project, and collect the results. Graded on the saved report rather than + tenant state, and the agent cleans the project up afterwards. +tags: [uipath-test, e2e, mode:operate, lifecycle:setup, feature:test-case] + +run_limits: + turn_timeout: 1200 + +initial_prompt: | + I have a Playwright suite I want to run through Test Manager. Create a + minimal one in ./pw-suite with two passing tests and a config defining the + projects "chromium" and "firefox", then get it running from the Test Manager + project PWE2E (create it if needed) on chromium only — not firefox. + Save the final test report to ./report.json, then delete the project so my + tenant stays clean. + Do NOT ask for approval, confirmation, or feedback. + Do NOT pause between planning and implementation. + +success_criteria: + - type: command_executed + description: "Packaged and uploaded the suite" + tool_name: "Bash" + command_pattern: 'uip\s+(or|orchestrator)\s+packages\s+upload' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Filled the test set by label rather than hand-collected keys" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+add\s+.*--labels' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Scoped the run to chromium only" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testsets?\s+run\s+.*--playwright-projects\s+chromium(?!\s+[A-Za-z])' + min_count: 1 + weight: 3.0 + pass_threshold: 1.0 + + - type: file_exists + description: "Saved the final report" + path: "report.json" + weight: 1.5 + pass_threshold: 1.0 + + - type: run_command + description: "Both tests passed — the run reached real results" + command: "grep -qE '\"Passed\":[[:space:]]*2' report.json" + timeout: 30 + expected_exit_code: 0 + weight: 5.0 + pass_threshold: 1.0 + + - type: run_command + description: "Nothing failed or was left without a result" + command: "grep -qE '\"Failed\":[[:space:]]*0' report.json" + timeout: 30 + expected_exit_code: 0 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Cleaned up the project it created" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+project\s+delete\s+.*PWE2E' + min_count: 1 + weight: 1.0 + pass_threshold: 1.0 + + - type: command_not_executed + description: "Did not use the Studio/RPA link-automation pipeline" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+link-automation' + weight: 1.0 + pass_threshold: 1.0 diff --git a/tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml b/tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml new file mode 100644 index 0000000000..8d4f1a424d --- /dev/null +++ b/tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml @@ -0,0 +1,82 @@ +# Prerequisites (see PR description): +# - a CLI carrying the external-package commands (npm `latest` on 1.200+) +# - the eval tenant's Test Manager has Playwright support enabled +# Does NOT need serverless execution — it stops before running the test set. +task_id: skill-test-playwright-ingest-and-select +description: > + Integration test for the Playwright first mile up to (not including) + execution: package a suite, upload it, let ingestion auto-create the test + cases, fill a test set by the ingestion-applied labels, and confirm the set + resolves to a single Playwright package. Grades the recorded context rather + than tenant state, and the agent cleans the project up afterwards. +tags: [uipath-test, integration, mode:operate, lifecycle:setup, feature:test-case] + +initial_prompt: | + I have a Playwright suite I want to run from Test Manager. Create a minimal + one in ./pw-suite with two passing tests and a config defining the projects + "chromium" and "firefox", get it into the Test Manager project PWINT + (create it if needed), and collect its smoke tests into a test set. + Save what Test Manager reports about that test set's Playwright packaging to + ./context.json, then delete the project so my tenant stays clean. + +success_criteria: + - type: command_executed + description: "Packaged the suite as an external test package" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+pack\s+.*--type\s+playwright' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Uploaded the package to Orchestrator" + tool_name: "Bash" + command_pattern: 'uip\s+(or|orchestrator)\s+packages\s+upload' + min_count: 1 + weight: 1.5 + pass_threshold: 1.0 + + - type: command_executed + description: "Waited for ingestion with an unfiltered test case list" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+list(?![^\n]*--filter)[^\n]*--project-key\s+PWINT' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Filled the test set by label rather than hand-collected keys" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+add\s+.*--labels' + min_count: 1 + weight: 2.0 + pass_threshold: 1.0 + + - type: file_exists + description: "Recorded what Test Manager reported about the test set" + path: "context.json" + weight: 1.5 + pass_threshold: 1.0 + + - type: run_command + description: "Test set resolves to a single Playwright package, with its projects listed" + command: "grep -qi 'true' context.json && grep -qi 'chromium' context.json" + timeout: 30 + expected_exit_code: 0 + weight: 3.0 + pass_threshold: 1.0 + + - type: command_executed + description: "Cleaned up the project it created" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+project\s+delete\s+.*PWINT' + min_count: 1 + weight: 1.0 + pass_threshold: 1.0 + + - type: command_not_executed + description: "Did not use the Studio/RPA link-automation pipeline" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+link-automation' + weight: 1.0 + pass_threshold: 1.0 From 41ffc1519ed58a3cd68a3806482e079e54f9b723 Mon Sep 17 00:00:00 2001 From: Aditya Garg Date: Fri, 31 Jul 2026 03:14:47 +0530 Subject: [PATCH 2/3] test(uipath-test): express must-nots as criteria, not lookaheads Two clever regex lookaheads become plain positive+negative criterion pairs, which is how the repo expresses "must not" and reads without decoding: the unfiltered-poll check and the chromium-only check each split into a command_executed plus a command_not_executed. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../tasks/uipath-test/playwright_first_mile_e2e.yaml | 11 +++++++++-- .../playwright_ingest_and_select_integration.yaml | 11 +++++++++-- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml index 724be822c2..fe92751caf 100644 --- a/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml +++ b/tests/tasks/uipath-test/playwright_first_mile_e2e.yaml @@ -42,13 +42,20 @@ success_criteria: pass_threshold: 1.0 - type: command_executed - description: "Scoped the run to chromium only" + description: "Ran the test set scoped to the chromium project" tool_name: "Bash" - command_pattern: 'uip\s+tm\s+testsets?\s+run\s+.*--playwright-projects\s+chromium(?!\s+[A-Za-z])' + command_pattern: 'uip\s+tm\s+testsets?\s+run\s+.*--playwright-projects\s+chromium' min_count: 1 weight: 3.0 pass_threshold: 1.0 + - type: command_not_executed + description: "Did not widen the run to a second Playwright project" + tool_name: "Bash" + command_pattern: '--playwright-projects\s+\S+\s+\S' + weight: 1.0 + pass_threshold: 1.0 + - type: file_exists description: "Saved the final report" path: "report.json" diff --git a/tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml b/tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml index 8d4f1a424d..f8be41b409 100644 --- a/tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml +++ b/tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml @@ -37,13 +37,20 @@ success_criteria: pass_threshold: 1.0 - type: command_executed - description: "Waited for ingestion with an unfiltered test case list" + description: "Waited for ingestion by listing the project's test cases" tool_name: "Bash" - command_pattern: 'uip\s+tm\s+testcases?\s+list(?![^\n]*--filter)[^\n]*--project-key\s+PWINT' + command_pattern: 'uip\s+tm\s+testcases?\s+list\s+.*--project-key\s+PWINT' min_count: 1 weight: 2.0 pass_threshold: 1.0 + - type: command_not_executed + description: "Did not filter that list by package name — the filter never matches ingested test cases" + tool_name: "Bash" + command_pattern: 'uip\s+tm\s+testcases?\s+list\s+.*--filter' + weight: 1.0 + pass_threshold: 1.0 + - type: command_executed description: "Filled the test set by label rather than hand-collected keys" tool_name: "Bash" From 1ff7a879cb720e6598ccf114885abb0159067b2d Mon Sep 17 00:00:00 2001 From: Aditya Garg Date: Fri, 31 Jul 2026 04:43:32 +0530 Subject: [PATCH 3/3] test(uipath-test): tighten the context assertion, note the probe dependency The IsPlaywright check matched a bare "true" anywhere in the file, so an unrelated boolean could satisfy it; it now matches the field. Header records that the assertion reads playwright-context output, which needs UiPath/cli#3336 in the released line. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr --- .../uipath-test/playwright_ingest_and_select_integration.yaml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml b/tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml index f8be41b409..2b85fedc67 100644 --- a/tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml +++ b/tests/tasks/uipath-test/playwright_ingest_and_select_integration.yaml @@ -1,6 +1,8 @@ # Prerequisites (see PR description): # - a CLI carrying the external-package commands (npm `latest` on 1.200+) # - the eval tenant's Test Manager has Playwright support enabled +# - the released CLI registers `testsets playwright-context` on stable +# (UiPath/cli#3336) — this task's assertion reads that command's output # Does NOT need serverless execution — it stops before running the test set. task_id: skill-test-playwright-ingest-and-select description: > @@ -67,7 +69,7 @@ success_criteria: - type: run_command description: "Test set resolves to a single Playwright package, with its projects listed" - command: "grep -qi 'true' context.json && grep -qi 'chromium' context.json" + command: "grep -qiE '\"IsPlaywright\"[^,}]*true' context.json && grep -qi 'chromium' context.json" timeout: 30 expected_exit_code: 0 weight: 3.0