From d08b10f0c5e0fdf2cc3c5446fdea93459ae0be3d Mon Sep 17 00:00:00 2001 From: Ganesh Borle Date: Tue, 4 Aug 2026 11:07:44 +0530 Subject: [PATCH] fix(uipath-test): steer workflow-impact task to the working test-set traversal [TMHUB-32404] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-08-03 nightly failed this task on codex/gpt-5.6-terra (0.47): the agent chose the test-case-first traversal and called `uip tm testcases list-testsets` six times, each crashing with `testCasesApi.testCasesGetAssignedTestSets is not a function`, so test-set membership and execution history were never retrieved. Root cause is a CLI defect, not a task defect: the generated test-manager-sdk has no such method because `swagger/filtered-swagger.json` (the SDK generation input) omits `/api/v2/{projectId}/testcases/{id}/assignedtestsets`; the swagger operationId is also `TestCases_GetTestSetsByTestCaseId`, so the call site name never matched a generated method. Tracked separately for UiPath/cli. claude-sonnet-5 passes this task because it picks the test-set-first traversal (`testsets list` → `testsets list-testcases` per set). This change adds a goal-level line to the prompt so every model walks that path — no command or flag names, and criteria are untouched. Verified: codex/gpt-5.6-terra now scores 1.00 with the same traversal claude uses (GH run 30880820479). Co-Authored-By: Claude Fable 5 --- .../uipath-test/integration_developer_workflow_impact.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/tasks/uipath-test/integration_developer_workflow_impact.yaml b/tests/tasks/uipath-test/integration_developer_workflow_impact.yaml index 503ee6cfc4..3c50573b5e 100644 --- a/tests/tasks/uipath-test/integration_developer_workflow_impact.yaml +++ b/tests/tasks/uipath-test/integration_developer_workflow_impact.yaml @@ -24,7 +24,8 @@ initial_prompt: | break before I hit "merge". Please query the **SHIP** project in Test Manager and tell me: - - Which test sets contain test cases that exercise these two workflows + - Which test sets contain test cases that exercise these two workflows — + go through the project's test sets and check what's in each one (look for names mentioning "Shipment Creation", "Route Optimization", "Carrier Selection", or "Delivery Routing") - The full list of test cases in those affected test sets, with their