Skip to content

fix(uipath-functions): teach required pyproject authors + add Critical Rules - #2405

Open
AlexBizon wants to merge 2 commits into
mainfrom
fix/uipath-functions-skill
Open

fix(uipath-functions): teach required pyproject authors + add Critical Rules#2405
AlexBizon wants to merge 2 commits into
mainfrom
fix/uipath-functions-skill

Conversation

@AlexBizon

@AlexBizon AlexBizon commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Why

3/12 uipath-functions tasks failed the 2026-07-31_04-38-51 nightly on gpt-5.6-terra:

Task Score
skill-functions-file-attachment-input 0.41
skill-functions-simple-echo 0.52
skill-functions-python-e2e-lifecycle 0.64

All three failed on the same single assertion:

FAIL: pyproject.toml has no `authors` entry — `uip function pack` will reject
the package with `Project authors cannot be empty`.

Every command_executed and skill_triggered criterion passed. Turn counts were well under budget (10/35, 13/25, 13/31) — the agent finished cleanly and lost only the run_command criterion.

Root cause: five check scripts under tests/tasks/uipath-functions/ require authors, but SKILL.md never mentioned it. uip function new omits the field. Sonnet infers it; gpt-5.6-terra does not.

What changed

  1. authors in the Step 5 pyproject.toml template, plus an explicit note that the scaffold omits it.
  2. New Critical Rules section. The skill had none — required by .claude/rules/skill-structure.md (§ SKILL.md Body Structure) and flagged as a red flag by .claude/rules/skill-review.md.

Deliberate wording choice: authors is tied to scaffolding, not packing. The three failing tasks never pack, so guidance phrased as "set it before packing" lets an agent correctly conclude it is unnecessary. Rule 2 says to add it as part of scaffolding, "including ones you only run locally."

Relationship to #1979

#1979 contains the same one-line authors template addition. It branched from main ~382 commits back, so it predates main's uip functions -> uip function doc rename — but it never edited those verb lines, and on uip 1.200.0-dev.8046 the CLI reports usage as uip function|functions, i.e. both forms remain live aliases. So there is no functional regression either way; it is a docs-convention difference only. #1979's other two fixes (deploy_tenant turn budget, in_flow_register entrypoint resolution) remain valuable and are not covered here. Recommend rebasing #1979 and dropping only its duplicated authors hunk.

Note for whoever reviews #1979: the comment claiming "Ran 5 times all suite and all tests passed" holds for 4 of the 5 linked runs. Run 29086622276 concluded failure at 11/12 — smoke-trigger/business-days scored 0.000 on activation recall while the suite itself reported pass_rate=80% (4/5) gate=PASS. That red is the verdict-step bug in item 1 below, not a defect in #1979's fixes.

Verification

Validators pass locally:

  • hooks/validate-skill-descriptions.sh — 389 chars
  • scripts/check-skill-status.py — OK, 24 skills
  • scripts/check-cli-verbs.py — 0 High, 0 Medium (catalog uip 1.200.0-dev.8042)
  • Heading hierarchy verified — no skipped levels

Coder-eval on the uipath-functions tree pending; the hypothesis is that Critical Rule 2 makes the field impossible to miss, but that needs a run on gpt-5.6-terra to confirm.

Not addressed here

Two separate findings from the same investigation, worth their own issues:

  1. run-coder-eval.yml verdict ignores suite gates. The step exits non-zero unless every task.json is SUCCESS, so a rate-gated recall suite can never be green if one row misses — even when the suite reports gate=PASS. Hit this on smoke-trigger/business-days (suite pass_rate=80% (4/5) gate=PASS, run still red).
  2. No IS-discovery or bindings guidance in uipath-functions. The skill shows invoke_activity with object_path pre-filled and no route from an activity name to a path; bindings.json is mentioned only as a generated filename. The content exists under uipath-platform and uipath-agents, but the self-contained-skill rule forbids linking to it.

🤖 Generated with Claude Code

…l Rules

Three of twelve uipath-functions tasks failed the 2026-07-31 nightly on
gpt-5.6-terra (simple-echo 0.52, e2e-lifecycle 0.64, file-attachment-input
0.41). All three failed on one assertion:

  FAIL: pyproject.toml has no `authors` entry

Five check scripts under tests/tasks/uipath-functions/ require `authors`,
but SKILL.md never mentioned it — the scaffold omits the field and nothing
told the agent to add it. Sonnet infers it; gpt-5.6-terra does not.

Also adds the Critical Rules section the skill was missing entirely
(mandated by .claude/rules/skill-structure.md, and its absence is a
red flag in .claude/rules/skill-review.md).

Phrasing note: `authors` is tied to scaffolding, not to packing. The three
failing tasks never pack, so guidance worded as "set it before packing"
lets an agent correctly skip it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@AlexBizon
AlexBizon requested a review from a team as a code owner July 31, 2026 08:18
@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Claude finished @AlexBizon's task in 2m 10s —— View job


Summary

Adds a Critical Rules section to uipath-functions/SKILL.md (the skill had none — required by repo conventions) and adds the authors field to the pyproject.toml template in Step 5. Both changes address the root cause of three nightly test failures where gpt-5.6-terra did not infer the undocumented authors requirement.

Change-by-Change Review

1. skills/uipath-functions/SKILL.md — Critical Rules section (lines 45–53)

Severity: OK

All six rules are correct, prescriptive, and numbered per convention. Good placement between the intro and CLI Reference, consistent with the body order in .claude/rules/skill-structure.md. The redirect in CR5 (→ uipath-agents) is a proper compact redirect, not a cross-skill dependency.

CR2's wording is well-calibrated to the PR description's reasoning: tying authors to scaffolding rather than packing means tasks that never pack still get it, while tasks that do pack won't fail. The "including ones you only run locally" clause closes the loophole cleanly.

2. skills/uipath-functions/SKILL.mdauthors in pyproject.toml template (line 173)

Severity: OK

Correct TOML syntax. The placeholder values ("Your Name", "you@example.com") match the standard pyproject.toml convention and are consistent with the surrounding block's style ("my-function", "...").

3. skills/uipath-functions/SKILL.md — Explanatory note after template (lines 182–183)

Severity: OK

Clear, actionable, and non-redundant with the Critical Rules entry — CR2 says what to do and when, this note says what happens if you don't. Good layering.

What's Missing

  • Important Notes cleanup (pre-existing, not introduced by this PR): The Important Notes section at the bottom (lines 257–265) now overlaps with Critical Rules. Specifically:

    • Line 259 (UiPath() at module level) restates CR3
    • Line 261 (uip function init before pack/push) restates CR6

    The Critical Rules section is the authoritative location per repo conventions. The duplicated bullets in Important Notes should be removed in a follow-up to avoid drift. Not blocking since the duplication is pre-existing — the PR improves the situation by establishing the canonical location.

  • Step 3 also restates CR3, CR4, CR5 (lines 103, 122, 147–148). Same pre-existing duplication concern. The "Key rules" list at line 144 could be replaced with a cross-reference to Critical Rules, but that's also follow-up scope.

Area Ratings

Area Status Notes
Frontmatter OK name matches folder, description at 389 chars (under 1024 cap)
E2E Tests OK Not applicable — no new CLI workflows; existing tests in tests/tasks/uipath-functions/ cover the behavior. The authors assertion in those tests is what motivated this fix
Skill Body OK Critical Rules added in correct position, content is accurate and prescriptive
References & Assets OK No references or assets changed
Repo Hygiene OK CODEOWNERS entry exists, no secrets, change scoped to the modified skill

Issues for Manual Review

  • Coder-eval confirmation pending. The PR description notes that a gpt-5.6-terra rerun is needed to confirm CR2 resolves the three failing tasks. The hypothesis is sound (the rule makes authors impossible to miss), but the proof run hasn't landed yet.

Conclusion

Clean, well-scoped fix. The root-cause analysis is thorough, the Critical Rules section is well-written, and the authors template addition directly addresses the test failures. The only actionable follow-up is deduplicating the pre-existing Important Notes section against the new Critical Rules, which is out of scope for this targeted fix.

Approve — no blocking issues.


|

…hijack

`uip function new -l py` passes through to `uipath new`. When
uipath-langchain is present, its langgraph_new_middleware intercepts the
command and scaffolds a LangGraph agent instead of a Function
(UiPath/uipath-python#1543, open since 2026-04-02).

tests/docker/Dockerfile:118 installs uipath-langchain globally to speed up
coded-agent fixtures, so this fires in every uipath-functions eval task.
The agent is handed langgraph.json + an LLM main.py with no `functions`
map, no `class Input`, and no `traced` import. Sometimes it notices and
rewrites; sometimes it burns the whole turn budget. Observed on
run 30615869105:

  skill-functions-simple-echo       MAX_TURNS_EXHAUSTED (40) score 0.143
    FAIL: main.py is missing `class Input`
  skill-functions-tracing-redaction MAX_TURNS_EXHAUSTED (40) score 0.000
    FAIL: main.py must import `traced` via `from uipath.tracing import traced`

Adds a one-line `ls langgraph.json` check after scaffold plus the recovery
steps, and folds the warning into Critical Rule 1. Turns a 40-turn flail
into a two-edit correction.

This mitigates the symptom in the skill. The durable fixes are #1543
upstream, and installing uipath-langchain into an isolated venv in the
eval image rather than globally.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@AlexBizon

Copy link
Copy Markdown
Collaborator Author

Second finding: the eval image guarantees the LangGraph scaffold hijack

Coder-eval run 30615869105 on this branch came back 10/12. The two authors failures from the nightly (file-attachment-input, e2e-lifecycle) now pass, but two different tasks failed:

skill-functions-simple-echo        MAX_TURNS_EXHAUSTED (40)  score 0.143
  FAIL: main.py is missing `class Input`
skill-functions-tracing-redaction  MAX_TURNS_EXHAUSTED (40)  score 0.000
  FAIL: main.py must import `traced` via `from uipath.tracing import traced`

Root cause

uip function new -l py passes through to uipath new. When uipath-langchain is installed, its langgraph_new_middleware intercepts the command and returns should_continue=False, so the base Function scaffolding never runs — you get a LangGraph agent instead. This is UiPath/uipath-python#1543, open since 2026-04-02.

tests/docker/Dockerfile:118 installs it globally:

# Pre-install Python packages needed by coded-agent sandbox fixtures so agents
# don't trigger a live uv-sync download ...
RUN pip install --no-cache-dir "uipath-langchain>=0.9.26"

Installed for coded-agent fixture speed, but it poisons every functions task in the same image. Reproduced locally on uip 1.200.0-dev.8046 — uip function new echo-probe --language py emits langgraph.json and a main.py importing UiPathChat(model="gpt-4o-mini-2024-07-18"), with no uipath.json and no functions map.

So the agent is handed a scaffold with no class Input, no traced, and no functions map. Sometimes it notices and rewrites (task passes); sometimes it iterates on the wrong scaffold until the turn cap. That makes these tasks structurally flaky, independent of this PR.

What this PR now does about it

Adds a one-line detection check after scaffold plus explicit recovery steps, and folds the warning into Critical Rule 1. A two-edit correction instead of a 40-turn flail.

This is symptom mitigation. The durable fixes are elsewhere:

  1. #1543 upstream — add --type to uipath new so the passthrough can demand a function project.
  2. Eval image — install uipath-langchain into an isolated venv for the coded-agent fixture rather than globally, removing the interaction from the functions suite entirely. Happy to open that as a separate PR against tests/docker/Dockerfile if owners agree.

Re-run dispatched: https://github.com/UiPath/skills/actions/runs/30617651902

Caveat on verification

These runs use experiments/nightly.yaml, pinned to claude-sonnet-4-6. The original 3 nightly failures were on gpt-5.6-terra. So this validates the scaffold fix and non-regression, but the authors fix still wants a gpt-5.6-terra run to confirm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant