test(uipath-test): Playwright integration and e2e tasks (hold until C… #2401
+185
−0
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| # Prerequisites (see PR description): | ||
| # - a CLI carrying the external-package commands (npm `latest` on 1.200+) | ||
| # - the eval tenant's Test Manager has Playwright support enabled | ||
| # - a folder the runner is a member of, with a Cloud Robots - Serverless | ||
| # machine, and working serverless Playwright execution | ||
| task_id: skill-test-playwright-first-mile-e2e | ||
| description: > | ||
| Full Playwright first mile: package a suite, upload it, let ingestion | ||
| auto-create the test cases, fill a test set by label, run only one Playwright | ||
| project, and collect the results. Graded on the saved report rather than | ||
| tenant state, and the agent cleans the project up afterwards. | ||
| tags: [uipath-test, e2e, mode:operate, lifecycle:setup, feature:test-case] | ||
|
|
||
| run_limits: | ||
| turn_timeout: 1200 | ||
|
|
||
| initial_prompt: | | ||
| I have a Playwright suite I want to run through Test Manager. Create a | ||
| minimal one in ./pw-suite with two passing tests and a config defining the | ||
| projects "chromium" and "firefox", then get it running from the Test Manager | ||
| project PWE2E (create it if needed) on chromium only — not firefox. | ||
| Save the final test report to ./report.json, then delete the project so my | ||
| tenant stays clean. | ||
| Do NOT ask for approval, confirmation, or feedback. | ||
| Do NOT pause between planning and implementation. | ||
|
|
||
| success_criteria: | ||
| - type: command_executed | ||
| description: "Packaged and uploaded the suite" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+(or|orchestrator)\s+packages\s+upload' | ||
| min_count: 1 | ||
| weight: 1.5 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: command_executed | ||
| description: "Filled the test set by label rather than hand-collected keys" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+tm\s+testcases?\s+add\s+.*--labels' | ||
| min_count: 1 | ||
| weight: 1.5 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: command_executed | ||
| description: "Ran the test set scoped to the chromium project" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+tm\s+testsets?\s+run\s+.*--playwright-projects\s+chromium' | ||
| min_count: 1 | ||
| weight: 3.0 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: command_not_executed | ||
| description: "Did not widen the run to a second Playwright project" | ||
| tool_name: "Bash" | ||
| command_pattern: '--playwright-projects\s+\S+\s+\S' | ||
|
There was a problem hiding this comment.
For a correct invocation such as Useful? React with 👍 / 👎. |
||
| weight: 1.0 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: file_exists | ||
| description: "Saved the final report" | ||
| path: "report.json" | ||
| weight: 1.5 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: run_command | ||
| description: "Both tests passed — the run reached real results" | ||
| command: "grep -qE '\"Passed\":[[:space:]]*2' report.json" | ||
|
There was a problem hiding this comment.
If the upload, run, wait, or report retrieval fails, an agent can still write AGENTS.md reference: AGENTS.md:L40-L42 Useful? React with 👍 / 👎. |
||
| timeout: 30 | ||
| expected_exit_code: 0 | ||
| weight: 5.0 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: run_command | ||
| description: "Nothing failed or was left without a result" | ||
| command: "grep -qE '\"Failed\":[[:space:]]*0' report.json" | ||
| timeout: 30 | ||
| expected_exit_code: 0 | ||
| weight: 2.0 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: command_executed | ||
| description: "Cleaned up the project it created" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+tm\s+project\s+delete\s+.*PWE2E' | ||
| min_count: 1 | ||
| weight: 1.0 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: command_not_executed | ||
| description: "Did not use the Studio/RPA link-automation pipeline" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+tm\s+testcases?\s+link-automation' | ||
| weight: 1.0 | ||
| pass_threshold: 1.0 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # Prerequisites (see PR description): | ||
| # - a CLI carrying the external-package commands (npm `latest` on 1.200+) | ||
| # - the eval tenant's Test Manager has Playwright support enabled | ||
| # - the released CLI registers `testsets playwright-context` on stable | ||
| # (UiPath/cli#3336) — this task's assertion reads that command's output | ||
| # Does NOT need serverless execution — it stops before running the test set. | ||
| task_id: skill-test-playwright-ingest-and-select | ||
| description: > | ||
| Integration test for the Playwright first mile up to (not including) | ||
| execution: package a suite, upload it, let ingestion auto-create the test | ||
| cases, fill a test set by the ingestion-applied labels, and confirm the set | ||
| resolves to a single Playwright package. Grades the recorded context rather | ||
| than tenant state, and the agent cleans the project up afterwards. | ||
| tags: [uipath-test, integration, mode:operate, lifecycle:setup, feature:test-case] | ||
|
|
||
| initial_prompt: | | ||
| I have a Playwright suite I want to run from Test Manager. Create a minimal | ||
| one in ./pw-suite with two passing tests and a config defining the projects | ||
| "chromium" and "firefox", get it into the Test Manager project PWINT | ||
| (create it if needed), and collect its smoke tests into a test set. | ||
| Save what Test Manager reports about that test set's Playwright packaging to | ||
| ./context.json, then delete the project so my tenant stays clean. | ||
|
|
||
| success_criteria: | ||
| - type: command_executed | ||
| description: "Packaged the suite as an external test package" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+tm\s+pack\s+.*--type\s+playwright' | ||
| min_count: 1 | ||
| weight: 1.5 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: command_executed | ||
| description: "Uploaded the package to Orchestrator" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+(or|orchestrator)\s+packages\s+upload' | ||
| min_count: 1 | ||
| weight: 1.5 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: command_executed | ||
| description: "Waited for ingestion by listing the project's test cases" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+tm\s+testcases?\s+list\s+.*--project-key\s+PWINT' | ||
| min_count: 1 | ||
| weight: 2.0 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: command_not_executed | ||
| description: "Did not filter that list by package name — the filter never matches ingested test cases" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+tm\s+testcases?\s+list\s+.*--filter' | ||
| weight: 1.0 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: command_executed | ||
| description: "Filled the test set by label rather than hand-collected keys" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+tm\s+testcases?\s+add\s+.*--labels' | ||
| min_count: 1 | ||
| weight: 2.0 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: file_exists | ||
| description: "Recorded what Test Manager reported about the test set" | ||
| path: "context.json" | ||
| weight: 1.5 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: run_command | ||
| description: "Test set resolves to a single Playwright package, with its projects listed" | ||
| command: "grep -qiE '\"IsPlaywright\"[^,}]*true' context.json && grep -qi 'chromium' context.json" | ||
|
There was a problem hiding this comment.
If ingestion or context lookup fails, the agent can create a local file containing AGENTS.md reference: AGENTS.md:L40-L42 Useful? React with 👍 / 👎. |
||
| timeout: 30 | ||
| expected_exit_code: 0 | ||
| weight: 3.0 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: command_executed | ||
| description: "Cleaned up the project it created" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+tm\s+project\s+delete\s+.*PWINT' | ||
| min_count: 1 | ||
| weight: 1.0 | ||
| pass_threshold: 1.0 | ||
|
|
||
| - type: command_not_executed | ||
| description: "Did not use the Studio/RPA link-automation pipeline" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+tm\s+testcases?\s+link-automation' | ||
| weight: 1.0 | ||
| pass_threshold: 1.0 | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When this task overlaps another run or
PWE2Eremains after an earlier failure, the prompt reuses that fixed project and then requires its deletion. The workflow-dispatch runner executes tasks with-j 4in.github/workflows/run-coder-eval.yml, so one run can delete the project while another is ingesting or executing tests, and it can delete a project it did not create. Seed a unique project key per run and clean up only that owned project; the companionPWINTtask needs the same isolation.Useful? React with 👍 / 👎.