docs(uipath-agents): update eval skills for unified runtime eval commands - #2440
docs(uipath-agents): update eval skills for unified runtime eval commands#2440AAgnihotry wants to merge 3 commits into
Conversation
…ands Rewrites orchestrator-eval-run.md to cover the full `uip or eval` command surface (30 commands across 6 groups): - `execute-and-evaluate` replaces `run-offline-evals` — uses `--process-key` instead of `--package-name`/`--package-version`, requires `--items` and `--evaluators`, removes `--is-low-code-agent` - New CRUD sections: evaluator, eval-set, evaluation (data points) - Run results: list/get/results - Schedules: create/list/get/update/pause/resume/delete - Typical workflow showing the full create-evaluator → create-eval-set → add-data-points → run → check-results → schedule pipeline Updates evaluate.md nav and SKILL.md task table. Companion to UiPath/cli#3402. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Claude finished @AAgnihotry's task in 2m 50s —— View job PR Review: docs(uipath-agents): update eval skills for unified runtime eval commands
SummaryRewrites Change-by-Change Review1.
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | OK | No frontmatter changes beyond the task table |
| E2E Tests | Issue | No test tasks cover orchestrator eval commands (pre-existing gap, not introduced by this PR) |
| Skill Body | OK | Task table row accurately reflects new doc scope |
| References & Assets | OK | All internal links resolve; nav updated consistently; no stale run-offline-evals refs remain anywhere in skills/uipath-agents/ |
| Repo Hygiene | OK | No secrets, no cross-skill dependencies, no personal paths, changes scoped to one skill |
Issues for Manual Review
- CLI command accuracy: The 30 commands documented (evaluator/eval-set/evaluation/schedule CRUD,
execute-and-evaluate, run results) should be verified against the actual CLI implementation inUiPath/cli#3402. Flag names, required vs. optional status, and output codes are taken at face value in this review. - Typical Workflow step 4: The
evaluatorConfig: {}empty object in the inline--evaluatorsJSON may not be valid at runtime — verify whether the API accepts an empty config or requires the full prompt/model/type fields.
Conclusion
Clean, well-structured rewrite that properly replaces the old single-command doc with comprehensive CRUD coverage. The two Low-severity items (workflow step redundancy and missing anti-patterns) are worth addressing but not blocking. No Critical or High issues found. Approve with minor suggestions.
Adds --limit/--offset options and Pagination field to evaluator list, eval-set list, evaluation list, and run list command docs to match CLI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix workflow step redundancy: step 4 now references the eval set
created in step 2 via --eval-set-id instead of duplicating data inline
- Fix empty evaluatorConfig: {} in workflow example — now shows the full
config with prompt, model, and targetOutputKey
- Add Anti-patterns section covering empty configs, same-as-agent, missing
folder-key, CRUD vs inline mixing, and cross-process evaluator reuse
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Updates the uipath-agents documentation to reflect the unified uip or eval runtime evaluation command surface for agents deployed as Orchestrator packages, replacing the older “offline eval” guidance and updating internal navigation to the new reference.
Changes:
- Rewrote
orchestrator-eval-run.mdinto a consolidated reference coveringuip or evalcommand groups (execute-and-evaluate, CRUD, results, schedules) and a typical end-to-end workflow. - Updated the skill’s task table entry to point to the new runtime-eval command set and scope.
- Updated the evaluations navigation to reference the renamed/repurposed Orchestrator runtime eval doc.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| skills/uipath-agents/SKILL.md | Updates the task table entry from “offline evals” to runtime eval management coverage. |
| skills/uipath-agents/references/lowcode/evaluations/orchestrator-eval-run.md | Replaces the prior offline-eval doc with a unified runtime-eval command reference, examples, and troubleshooting. |
| skills/uipath-agents/references/lowcode/evaluations/evaluate.md | Updates nav text to the new Orchestrator runtime eval documentation entry. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Orchestrator Runtime Eval Commands | ||
|
|
||
| Submit offline evaluation runs for low-code agents published as Orchestrator packages. | ||
| Manage and run runtime evaluations for agents published as Orchestrator packages. All commands are scoped by `--process-key` (deployment GUID) and hit the agents runtime API. |
| [--evaluation-criterias <json>] \ | ||
| [--tenant <tenant>] \ | ||
| --output json | ||
| ``` | ||
|
|
||
| | Flag | Required | Description | | ||
| |------|----------|-------------| | ||
| | `--process-key` | Yes | Process key (GUID) | | ||
| | `--eval-set-id` | Yes | Eval set ID (GUID) | | ||
| | `--folder-key` | Yes | Folder key (GUID) | | ||
| | `--name` | Yes | Data point name | | ||
| | `--inputs` | Yes | Input values as JSON | | ||
| | `--expected-output` | No | Expected output as JSON (for output evaluators) | | ||
| | `--expected-behavior` | No | Expected agent behavior (for trajectory evaluators) | | ||
| | `--evaluation-criterias` | No | Per-evaluator criteria overrides as JSON | |
| # 4. Run the eval referencing the eval set | ||
| uip or eval execute-and-evaluate \ | ||
| --process-key "$PROCESS_KEY" \ | ||
| --eval-set-id "$EVAL_SET_ID" \ | ||
| --items '[{"id":"i1","name":"Greeting test","inputs":{"input":"hello"},"expectedOutput":{"content":"Hi there!"},"expectedBehavior":""}]' \ | ||
| --evaluators '[{"id":"'"$EVALUATOR_ID"'","version":"","evaluatorTypeId":"5","evaluatorConfig":{"name":"Semantic","type":5,"category":1,"prompt":"Score 0-100...","model":"gpt-4.1-2025-04-14","targetOutputKey":"*"}}]' \ | ||
| --output json |
Summary
orchestrator-eval-run.mdto cover all 30uip or evalcommands in one doc: execute-and-evaluate, evaluator CRUD, eval-set CRUD, evaluation CRUD, run results, and schedulesrun-offline-evals/--package-name/--is-low-code-agentreferences withexecute-and-evaluate/--process-key/ required--items+--evaluatorsevaluate.mdnav andSKILL.mdtask tableCompanion to UiPath/cli#3402.
Test plan
🤖 Generated with Claude Code