fix(uipath-test): steer workflow-impact task to the working test-set …

[ganeshborle]

…traversal TMHUB-32404

The 2026-08-03 nightly failed this task on codex/gpt-5.6-terra (0.47): the agent chose the test-case-first traversal and called uip tm testcases list-testsets six times, each crashing with testCasesApi.testCasesGetAssignedTestSets is not a function, so test-set membership and execution history were never retrieved.

Root cause is a CLI defect, not a task defect: the generated test-manager-sdk has no such method because
swagger/filtered-swagger.json (the SDK generation input) omits /api/v2/{projectId}/testcases/{id}/assignedtestsets; the swagger operationId is also TestCases_GetTestSetsByTestCaseId, so the call site name never matched a generated method. Tracked separately for UiPath/cli.

claude-sonnet-5 passes this task because it picks the test-set-first traversal (testsets list → testsets list-testcases per set). This change adds a goal-level line to the prompt so every model walks that path — no command or flag names, and criteria are untouched.

Verified: codex/gpt-5.6-terra now scores 1.00 with the same traversal claude uses (GH run 30880820479).

[github-actions]

Claude finished @ganeshborle's task in 2m 8s —— View job

---

Coder-eval task lint (advisory)

1 task YAML changed; verdicts: 0 Critical, 0 High, 0 Medium, 1 Low, 0 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

• ✅ Author confirms passing run: "Verified: codex/gpt-5.6-terra now scores 1.00 with the same traversal claude uses (GH run 30880820479)."

Per-task lint

tests/tasks/uipath-test/integration_developer_workflow_impact.yaml — verdict: Low

Issues:

• [Low] Meaningful coverage: Report content check (lines 86–87) uses broad keyword matching (shipment|carrier|route|test case|impact|workflow) — all terms already in the prompt, so a plausible-sounding report passes without real API data. Sibling integration_release_readiness_qa_lead.yaml shows the stronger pattern: asserting data-grounded signals like project-specific keys (BANK:\d+) and UUIDs that only appear in real API responses.

Suggested fixes:

• Strengthen the run_command at lines 82–90 by adding assertions for fixture-specific identifiers that only appear in real uip tm output — e.g., SHIP:\d+ test-set/test-case keys or a UUID regex for entity IDs returned by the API. This lifts the report check from "domain-relevant prose" to "data obtained from the tenant."

Notes on the change itself:

• The added line (28–29) steers toward the test-set-first traversal at goal level ("go through the project's test sets and check what's in each one") without naming CLI commands or flags — well within the prompt-over-specification carve-outs.
• All 5 command_executed criteria (lines 42–80) use specific patterns with required flags (--project-key, --test-set-key), providing solid traversal-path validation independent of the report.
• No self-report anti-pattern: the report is the task's deliverable, and the dominant validation signal comes from command_executed checks, not the report content.
• No near-duplicate concern: shares the discovery scaffold with integration_release_readiness_qa_lead.yaml but tests a materially distinct operation (workflow-impact analysis vs. release-readiness chain with execution drilldown and coverage-gap detection).

Conclusion

• ⚠ 1 task has issues, max severity Low. Advisory only — not blocking merge. The prompt change is well-scoped and the pre-existing Low finding (loose report content check) is unrelated to this diff.

[github-actions]

Claude finished @ganeshborle's task in 1m 19s —— View job

---

PR Review: fix(uipath-test): steer workflow-impact task to the working test-set traversal

• Read review criteria and project conventions
• Read full diff and changed file
• Check PR against review criteria
• Post review

---

Summary

Single-line prompt tweak in integration_developer_workflow_impact.yaml that adds a directional hint ("go through the project's test sets and check what's in each one") to steer agents toward the test-set-first traversal, avoiding the broken testcases list-testsets CLI path. No criteria, tags, or structural changes.

Change-by-Change Review

1. tests/tasks/uipath-test/integration_developer_workflow_impact.yaml (lines 27-28)

Severity: OK

The change replaces the original single-line bullet with a two-line version that adds a natural-language nudge toward the test-set-first traversal:

-  - Which test sets contain test cases that exercise these two workflows
+  - Which test sets contain test cases that exercise these two workflows —
+    go through the project's test sets and check what's in each one

This is well-crafted:

• Goal-level, not prescriptive: it describes the strategy ("go through test sets") without naming CLI commands or flags, which is consistent with the test-writing anti-pattern rule against hand-holding prompts
• Success criteria unchanged: the existing command_executed checks for testsets list and testsets list-testcases already validate the correct traversal path — no new criteria needed
• Prompt stays minimal: the hint integrates naturally into the existing bullet rather than adding a separate instruction block
• PR body documents the root cause (CLI SDK generation gap) and includes a verified passing run (GH run 30880820479), satisfying the lint expectation for a passing-run claim

No issues with YAML formatting, indentation, or tag validity (uipath-test, integration, mode:diagnose, lifecycle:discover, feature:test-case all conform to the taxonomy).

What's Missing

Nothing identified. This is a scoped prompt-only fix for a known CLI defect workaround — no new capabilities, commands, or criteria are introduced.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No frontmatter changes
E2E Tests	OK	Existing task modified, criteria unchanged
Skill Body	N/A	No skill body changes
References & Assets	N/A	No reference/asset changes
Repo Hygiene	OK	No secrets, no cross-skill deps, change scoped to one file

Issues for Manual Review

None found.

Conclusion

Clean, minimal fix. The prompt hint is well-calibrated — directional enough to avoid the broken CLI path without being prescriptive about commands. PR body provides thorough root-cause analysis and a verified passing run. Approve.

---

· Branch

[vaishalisharma-uipath]

we shouldn't hard code the logic in the prompt. Let's make the cli fix only.