fix(tests,skills): de-flake nightly guardrail failures — valid fixtures, doc gaps, run_limits

[apetraru-uipath]

What changed?

De-flakes the guardrail-tagged failures of nightly run 2026-07-28_04-16-10 (all tagged turn-timeout-too-short). Transcript analysis showed each timeout was a symptom of a concrete defect — fixture schema noise, a never-seeded test resource, or a missing skill-doc branch — not model slowness. Three change areas:

1. tests/tasks/uipath-review/ — fixtures now pass uip agent validate (5e170b1ab)

5 of 7 lowcode review fixtures failed uip agent validate on main, so reviewers saw "review: Grade A" vs "validate: hard errors" and burned 15+ min investigating harness noise instead of the seeded guardrail defect.

• _shared/lowcode_scaffold.py: new content_tokens() / set_message() (rebuild contentTokens to match patched message content), connection_binding() / process_binding() / write_bindings() (emit the packager-shaped bindings_v2 entries), and the scaffold now writes root bindings_v2.json like a real uip agent init project.
• All 7 inject scripts updated to use the helpers; inject_guardrail_audit_logging.py additionally gets a schema-valid integration tool resource (iconUrl, argumentProperties, properties.method/connection/parameters/bodyStructure).
• Result: all 8 lowcode review fixtures return Status=Valid straight from injection (uip 1.200.0), with no reliance on the refresh-before-validate workaround.

2. escalation_app_validation — the schema trap is now real and deterministic (f81ce1617, eec9e91cf)

The task grades "fetch the app's action schema → detect the mismatch → reject", but EscalationWorksApp was never seeded in any tenant — the trap never fired: passes rejected for the wrong reason ("not found") and the nightly failure was the model hunting the missing resource for ~18 min.

• New _fixtures/ValidationSol/: solution shell + EscalationWorksApp as a solution-internal resource (app / appVersion / package / process-webApp JSONs, fixed GUIDs) with a deliberately incompatible actionSchema (generic input/output; Submit/Reject outcomes). No tenant state — fully deterministic.
• escalation_app_validation.yaml: pre_run copies the fixture; prompt becomes brownfield ("solution exists, do NOT re-create it") and bans uip solution resources refresh (refresh destroys the trap either way — it deletes unregistered local resources after a project registration, or regenerates them and wipes the authored schema when a backing project is registered); the advisory uip solution init criterion becomes a gating command_not_executed (same pattern as local_escalation); run_limits gain headroom (max_turns 40→60, turn_timeout 1200→1800, +task_timeout: 2400).

3. skills/ — two doc gaps that caused thinking rabbit holes (bece3e061, ba3303858, part of eec9e91cf)

• uipath-agents/.../escalation.md (+4 lines): terminal-stop rule when the named app is not in the resources list / resources get fails (no retries, alternate endpoints, or re-auth; never wire an escalation against an unresolved app), and a note that --kind/--source/--search filters currently drop Source: Local rows — use the unfiltered uip solution resources list and pass the Key (not the name) to resources get.
• uipath-review/.../coded-guardrails-review.md (+8 lines): in the Actionability check, the catalog verdict IS the evidence — do not verify SDK behavior empirically (importing/executing project dependencies, reading SDK sources, fetching SDK docs); write the finding and the report.

Out of scope: skill-case-artifact-safety-guard (uipath-maestro-case) also failed that night but is a different feature that coincidentally carries the guardrail tag — intentionally untouched.

CLI issues surfaced (not fixable in this repo)

1. uip solution resources list with any filter (--kind, --source local, --search <name>) drops Source: Local rows; resources get <name> fails for local resources (Key-only). Worked around in escalation.md; worth a CLI fix.
2. uip solution resources refresh deletes local resources with no registered backing project, and regenerates (wiping authored actionSchema) when one is registered. Mitigated via the task prompt; worth a CLI look.

How has this been tested?

• All 8 lowcode review fixtures scaffolded from this branch validate Status=Valid on uip 1.200.0 (verified via CLI, no LLM involved).
• Escalation fixture verified via CLI: app appears in the unfiltered resources list as Source=Local; resources get <key> returns the incompatible schema; uip agent init ValidationSol/ReviewAgent registers cleanly alongside it.
• Final escalation_app_validation.yaml verified on both harnesses (Coder Eval v0.9.1, tempdir driver, isolated worktrees): Claude Sonnet 5 SUCCESS 1.0 (243s) and Codex gpt-5.6-terra SUCCESS 1.0 (107s) — both rejecting for the right reason, e.g. Terra: "schema only exposes input, output, Submit/Reject; guardrails require the named review inputs/outputs and Approve/Reject outcomes."
• uipath-review guardrail tasks (action-ineffective, audit-logging, content-safety, misapplied): SUCCESS 1.0 on Claude Sonnet 5 across two independent replicates each, and on Codex gpt-5.6-terra.
• git diff --check, python3 -m py_compile on all edited injects/scaffold, scripts/check-skill-status.py clean; check-skill-verbs.py findings on escalation.md are pre-existing (verified via stash).

Are there any breaking changes?

• None
• Under Feature Flag
• DB migrations
• API removals

[github-actions]

Claude finished @apetraru-uipath's task in 58s —— View job

---

Summary

Bumps run_limits for two guardrail test tasks that timed out in the 2026-07-28 nightly coder_eval run. Changes are limited to numeric timeout/turn-cap values — no prompt, criteria, or structural changes.

Change-by-Change Review

1. tests/tasks/uipath-agents/lowcode/guardrails/escalation_app_validation/escalation_app_validation.yaml

Severity: OK

max_turns 40→60, turn_timeout 1200→1800, new task_timeout: 2400. Precautionary headroom bump — PR author reports the original failure was tail-latency variance, not a systematic gap, and local repro passed cleanly at original limits. The added task_timeout (2400s) exceeds turn_timeout (1800s), which is correct (overall wall-clock cap vs. per-turn cap). expected_turns (20) unchanged, keeping the baseline realistic.

2. tests/tasks/uipath-maestro-case/guardrails/artifact_safety_guard.yaml

Severity: OK

task_timeout/turn_timeout 900→1500, max_turns 120→160. Well-justified — original run died at 904.5s against a 900s cap with 113/120 turns used while still making forward progress. The ~67% timeout increase and ~33% turn increase give adequate headroom without being excessive. task_timeout and turn_timeout remain equal (matching the file's original pattern).

What's Missing

Nothing identified. This is a clean, scoped timeout bump with thorough local repro evidence in the PR description.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill frontmatter changed
E2E Tests	OK	Task structure, tags, and criteria untouched — only run_limits bumped
Skill Body	N/A	No skill docs changed
References & Assets	N/A	No references or assets changed
Repo Hygiene	OK	Changes scoped to the two affected task files; no secrets or path leaks

Issues for Manual Review

None found.

Conclusion

Clean, well-scoped PR. Both limit bumps are justified by nightly failure evidence and validated by local repro on two models. No issues found — approve.

[github-actions]

Claude finished @apetraru-uipath's task in 1m 51s —— View job

---

Coder-eval task lint (advisory)

2 task YAMLs changed; verdict counts: 0 Critical, 0 High, 0 Medium, 0 Low, 2 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

✅ Author confirms passing run: "Reproduced locally against skills main (9711c2c9f) / Coder Eval v0.9.1 … escalation_app_validation: Claude Sonnet 5 → SUCCESS, score 1.0, 266s. Codex Terra → SUCCESS, score 1.0, 123s. artifact_safety_guard: Claude Sonnet 5 → SUCCESS, score 1.0, 553s"

Per-task lint

tests/tasks/uipath-agents/lowcode/guardrails/escalation_app_validation/escalation_app_validation.yaml — verdict: OK

Changes are limited to run_limits (lines 63–67): max_turns 40→60, turn_timeout 1200→1800, added task_timeout: 2400. No prompt, criteria, or structural changes. The task has strong criteria: command_executed checks for key CLI verbs (solution init, agent init, solution resources list, solution resources get) plus a run_command gating on a Python validation script (check_escalation_app_validation.py, weight 5.0). Not a near-duplicate of sibling escalation_app.yaml — that tests the happy-path (compatible app accepted), this tests the negative case (incompatible app rejected).

tests/tasks/uipath-maestro-case/guardrails/artifact_safety_guard.yaml — verdict: OK

Changes are limited to run_limits (lines 207–210): task_timeout 900→1500, max_turns 120→160, turn_timeout 900→1500. No prompt, criteria, or structural changes. Criteria mix is solid: file_exists + json_check with real assertions (action + process task types) + run_command via uip maestro case validate (weight 3.0) + two command_not_executed Rule 13 guardrails (weights 2.0 each). Not a near-duplicate — only task in guardrails/ and exercises a materially distinct concern (artifact assembly method guard) vs. other maestro-case siblings.

Within-PR duplicates

No duplicate clusters detected.

Conclusion

✅ All changed tasks pass the rubric. Evidence of passing run confirmed. Both changes are run-limit-only bumps with clear nightly-failure evidence in the PR body.

---