Skip to content

fix(tests): unstarve three uipath-agents smoke tasks (max_turns -> 60) - #2413

Open
mjnovice wants to merge 1 commit into
mainfrom
fix/unstarve-agents-smoke-tasks
Open

fix(tests): unstarve three uipath-agents smoke tasks (max_turns -> 60)#2413
mjnovice wants to merge 1 commit into
mainfrom
fix/unstarve-agents-smoke-tasks

Conversation

@mjnovice

Copy link
Copy Markdown
Contributor

Why

The uipath-agents smoke suite is flaking below the 95% gate because three tasks keep hitting MAX_TURNS_EXHAUSTED with high partial scores and zero failed commands — pure turn starvation, the same disease #2349 and #2234 fixed for other tasks.

Evidence from PR #1694's three smoke runs (26/29, 27/29, 27/29 — a different failure set each time) plus PR #1801's run:

Task max_turns Starvation evidence
skill-agents-coded-is-smoke 30 (below the smoke experiment default of 40) Starved twice in a row; score 0.84, 0 failed commands, only the doc-read criterion left
skill-agent-context-prompt-reference 30 Starved on #1694's run and #1801's run; mid-flow at score 0.41 after creating agent.json + the context resource
skill-agent-coded-conversational-langgraph 40 (inherited) Starved on #1694's rerun

What

Raise all three to expected_turns: 40 / max_turns: 60, matching #2349's precedent for eval_custom_evaluator. No prompt or criteria changes.

🤖 Generated with Claude Code

… -> 60)

Three smoke tasks keep hitting MAX_TURNS_EXHAUSTED with high partial
scores and zero failed commands — pure turn starvation, same disease
as #2349/#2234:

- skill-agents-coded-is-smoke (max_turns: 30, below the smoke
  experiment default of 40): starved twice in a row on PR #1694's
  runs at score 0.84 with only the doc-read criterion left.
- skill-agent-context-prompt-reference (max_turns: 30): starved on
  PR #1694's run AND on PR #1801's run — mid-flow at score 0.41
  after creating agent.json + the context resource.
- skill-agent-coded-conversational-langgraph (inherited 40): starved
  on PR #1694's rerun.

Raise all three to expected_turns 40 / max_turns 60, matching the
precedent set by #2349 for eval_custom_evaluator.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Claude finished @mjnovice's task in 2m 51s —— View job


Coder-eval task lint (advisory)

3 task YAMLs changed; verdict counts: 0 Critical, 0 High, 0 Medium, 0 Low, 3 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

High — PR body does not claim the changed tasks have been run and passed with the new max_turns values. The body cites starvation evidence from prior PRs (#1694, #1801) showing the tasks fail at their old limits, but doesn't assert a passing run at max_turns: 60. Please edit the PR description to add a line like:

Ran skill-agents-coded-is-smoke, skill-agent-context-prompt-reference, and skill-agent-coded-conversational-langgraph locally and they passed.

(Honor-system — a verbal claim is sufficient.)

Per-task lint

tests/tasks/uipath-agents/coded/conversational_agent_langgraph/conversational_agent_langgraph.yaml — verdict: OK

No issues. Change is run_limits only (expected_turns: 20→40, added max_turns: 60). Task has strong criteria: command_executed for new/init/run + a run_command Python checker (weight 5.0). Not a near-duplicate of langgraph_classifier — that task is e2e with schema-sync coverage and a classifier workflow; this one tests the conversational (isConversational: true) path with two chat turns.

tests/tasks/uipath-agents/coded/is_smoke/is_smoke.yaml — verdict: OK

No issues. Change is run_limits only (expected_turns: 20→40, max_turns: 30→60). Criteria include skill_triggered, command_executed for the doc-read, file_exists, and a run_command checker at weight 6.0. Not a near-duplicate of is_outlook_send_mail — that task is e2e with multipart edge case and tenant verification; this one is a smoke testing artifact shape without a live tenant.

tests/tasks/uipath-agents/lowcode/context_prompt_reference/context_prompt_reference.yaml — verdict: OK

No issues. Change is run_limits only (expected_turns: 20→40, max_turns: 30→60). Criteria cover the full init→refresh→validate lifecycle plus a run_command checker (weight 5.0) verifying @{contexts.Knowledge} expression tokens. Not a near-duplicate of context_index — that task is e2e testing index-type context resources with solution-level commands; this one exercises the @{}→expression contentToken path with an attachments context.

Within-PR duplicates

No duplicate clusters detected — all three tasks target distinct features (conversational LangGraph, coded IS capability, low-code expression tokens).

Conclusion

✅ All changed tasks pass the rubric. The only finding is the missing passing-run claim in the PR body (advisory).


@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Claude finished @mjnovice's task in 1m 53s —— View job


Reviewing PR #2413

  • Read review criteria and project conventions
  • Run git diff and read full changed files
  • Check for precedent consistency and task YAML validity
  • Post review

Summary

Raises expected_turns / max_turns from 20–30/30–40 to 40/60 on three uipath-agents smoke tasks that are consistently hitting MAX_TURNS_EXHAUSTED with high partial scores and zero failed commands. No prompt or criteria changes — pure turn-budget relief.

Change-by-Change Review

1. tests/tasks/uipath-agents/coded/conversational_agent_langgraph/conversational_agent_langgraph.yaml

Severity: OK

expected_turns raised 20→40, max_turns: 60 added (was absent, so the task previously inherited the experiment default of 200 — the expected_turns: 20 was the effective cap via coder-eval's behavior). The new values are consistent with other tasks in this suite that needed similar relief (bindings_sync, edit_roundtrip).

2. tests/tasks/uipath-agents/coded/is_smoke/is_smoke.yaml

Severity: OK

expected_turns raised 20→40, max_turns raised 30→60. Previously the tightest budget in the suite (max_turns: 30, below even the smoke experiment default). The bump matches the established 40/60 pattern.

3. tests/tasks/uipath-agents/lowcode/context_prompt_reference/context_prompt_reference.yaml

Severity: OK

Same 20→40 / 30→60 change. turn_timeout: 1200 preserved.

What's Missing

Nothing identified. The PR is scoped correctly to the three starving tasks with clear evidence in the description. No prompt or criteria changes needed — the tasks were passing all criteria except they ran out of turns before completion.

Area Ratings

Area Status Notes
Frontmatter N/A No skill frontmatter changed
E2E Tests OK Only run_limits tuned — tags, criteria, prompts untouched
Skill Body N/A No skill docs changed
References & Assets N/A No references or assets changed
Repo Hygiene OK No secrets, no unrelated changes, scoped to the three affected tasks

Issues for Manual Review

None found.

Conclusion

Clean, well-scoped fix. The 40/60 values are consistent with the existing pattern across other uipath-agents tasks that needed similar relief (bindings_sync, edit_roundtrip). The starvation evidence in the PR description is thorough. Approve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant