test(uipath-test): Playwright integration and e2e tasks (hold until C… - #2401
test(uipath-test): Playwright integration and e2e tasks (hold until C…#2401addyGarg wants to merge 3 commits into
Conversation
…LI ships) Two tasks for the Playwright first mile, deliberately kept out of the skill PR because they cannot pass until the external-package commands reach npm `latest` (the eval runner installs @uipath/cli@latest): - ingest-and-select (integration): pack, upload, ingestion, label fill, and the single-package check — stops before execution, so it needs only a Playwright-enabled tenant, no serverless runtime. - first-mile (e2e): the same flow through a chromium-only run to real results. Both grade saved artifacts rather than tenant state, so the agent can delete the project it created without breaking the assertions, and both carry a teardown criterion so runs leave nothing behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr
Two clever regex lookaheads become plain positive+negative criterion pairs, which is how the repo expresses "must not" and reads without decoding: the unfiltered-poll check and the chromium-only check each split into a command_executed plus a command_not_executed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr
…ndency The IsPlaywright check matched a bare "true" anywhere in the file, so an unrelated boolean could satisfy it; it now matches the field. Header records that the assertion reads playwright-context output, which needs UiPath/cli#3336 in the released line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0138XHhf1Jg5ozDUdbpX19Zr
PR Review: test(uipath-test) Playwright integration and e2e tasks
|
|
Claude finished @addyGarg's task in 3m 37s —— View job Coder-eval task lint (advisory)2 task YAMLs changed; verdicts: 0 Critical, 0 High, 1 Medium, 0 Low, 1 OK. Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge. Evidence of passing run❌ High — PR body explicitly states these tasks "cannot pass until the external-package commands reach npm
Per-task lint
|
|
Split out of #2322 (skill content + the zero-prerequisite smoke task). Also depends on UiPath/cli#3336 — the integration task reads |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ff7a879cb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| - type: run_command | ||
| description: "Both tests passed — the run reached real results" | ||
| command: "grep -qE '\"Passed\":[[:space:]]*2' report.json" |
There was a problem hiding this comment.
If the upload, run, wait, or report retrieval fails, an agent can still write {"Passed":2,"Failed":0} to report.json; the other criteria only prove that commands were attempted, while this grep accepts the fabricated file. The e2e task can therefore report success without reaching real Test Manager results. Verify the execution through a deterministic checker or require and validate the successful uip tm report get output instead.
AGENTS.md reference: AGENTS.md:L40-L42
Useful? React with 👍 / 👎.
| projects "chromium" and "firefox", then get it running from the Test Manager | ||
| project PWE2E (create it if needed) on chromium only — not firefox. | ||
| Save the final test report to ./report.json, then delete the project so my | ||
| tenant stays clean. |
There was a problem hiding this comment.
When this task overlaps another run or PWE2E remains after an earlier failure, the prompt reuses that fixed project and then requires its deletion. The workflow-dispatch runner executes tasks with -j 4 in .github/workflows/run-coder-eval.yml, so one run can delete the project while another is ingesting or executing tests, and it can delete a project it did not create. Seed a unique project key per run and clean up only that owned project; the companion PWINT task needs the same isolation.
Useful? React with 👍 / 👎.
| - type: command_not_executed | ||
| description: "Did not widen the run to a second Playwright project" | ||
| tool_name: "Bash" | ||
| command_pattern: '--playwright-projects\s+\S+\s+\S' |
There was a problem hiding this comment.
For a correct invocation such as uip tm testsets run ... --playwright-projects chromium --output json, this regex matches chromium --output as two project tokens and fails the negative criterion. Since the skill requires --output json on every uip command, the natural conforming command order produces a false negative; constrain the match to actual project arguments rather than arbitrary subsequent tokens.
Useful? React with 👍 / 👎.
|
|
||
| - type: run_command | ||
| description: "Test set resolves to a single Playwright package, with its projects listed" | ||
| command: "grep -qiE '\"IsPlaywright\"[^,}]*true' context.json && grep -qi 'chromium' context.json" |
There was a problem hiding this comment.
If ingestion or context lookup fails, the agent can create a local file containing {"IsPlaywright":true,"project":"chromium"} and satisfy this primary assertion; no criterion even requires testsets playwright-context to be executed successfully. Consequently the integration task can pass without proving that the test set resolves to a Playwright package. Run the lookup in a deterministic checker or otherwise bind context.json to successful CLI output.
AGENTS.md reference: AGENTS.md:L40-L42
Useful? React with 👍 / 👎.
1 participant
What
The Playwright integration and e2e eval tasks for the
uipath-testskill, split out of #2322 so that PR can merge now while these wait for the capability they exercise.playwright_ingest_and_select_integration.yamlplaywright_first_mile_e2e.yamlBoth grade saved artifacts (
context.json,report.json) rather than live tenant state, so the agent can delete the project it created without breaking the assertions — and a stale execution from an earlier run cannot satisfy an outcome check. Both carry a teardown criterion, so scheduled runs leave nothing behind.Why separate
The eval runner installs
@uipath/cli@latestbefore any task runs, and tasks are forbidden from pinning a version (tests/README.md). Until the external-package commands reach npmlatest, these can only fail — and a scheduled task that always fails is noise that also desensitises people to real failures. So the skill and its passing smoke task ship in #2322; these wait here.Do not merge until
latest— checknpm view @uipath/cli versionreports 1.200.x. The 1.200 cut is not sufficient: stable promotion is a separate manual dispatch.playwright-contextoutput, which a stable CLI only registers once that fix ships.Then run each once by hand and merge on green:
SKILLS_REPO_PATH=$(cd .. && pwd) .venv/bin/coder-eval run \ tasks/uipath-test/playwright_ingest_and_select_integration.yaml -e experiments/default.yamlValidation
The flows these tasks assert are proven manually: on a local serverless stack and on alpha
testmanagerdev/DevTest, a context-free agent using only the skill reached 2/2 passed on a chromium-only run, including a fix-and-republish loop. What is unproven is these YAMLs executing undercoder-evalon a capable tenant — hence the manual run in the merge checklist above.The e2e task is a candidate for the
path-to-gatag (must-pass, currently blocked); it is left off so this PR stays mergeable on demand, since that tag needs an allowlisted approval on the head commit.Related