fix(troubleshoot): re-anchor prompt fault timing to the run so frozen fixtures stop drifting - #2427
fix(troubleshoot): re-anchor prompt fault timing to the run so frozen fixtures stop drifting#2427dmorosanu wants to merge 1 commit into
Conversation
… fixtures stop drifting
|
Claude finished @dmorosanu's task in 3m 10s —— View job Coder-eval task lint (advisory)16 task YAMLs changed (all modifications, no new tasks); 0 Critical, 0 High, 0 Medium, 0 Low, 16 OK. Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge. Evidence of passing run✅ Author confirms passing run: "CI Per-task lintAll 16 changed files modify only
Within-PR duplicatesNo duplicate clusters detected. (No new tasks added — all 16 are modifications to existing scenarios covering distinct failure classes.) Conclusion✅ All changed tasks pass the rubric. Evidence of passing run confirmed. The changes are a clean, mechanical re-anchoring of calendar-relative timing phrases to run-relative phrasing across all 16 prompts — no structural, criteria, or sandbox changes. |
|
Claude finished @dmorosanu's task in 2m 2s —— View job PR Review: fix(troubleshoot): re-anchor prompt fault timing
SummaryRe-anchors 16 troubleshoot scenario prompts from calendar-relative timing ("today", "yesterday", "last night", "this morning") to run-relative timing ("most recent run", "previous run", "recently") so frozen fixture timestamps no longer drift against the prompt's claim. Also updates the Change-by-Change Review1. Activity-package
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | OK | No frontmatter changes — all edits are to initial_prompt, simulation, or RESOLUTION body |
| E2E Tests | OK | No new scenarios; existing scenarios' structure (tags, criteria, sandbox) unchanged |
| Skill Body | N/A | No skill source changes |
| References & Assets | OK | RESOLUTION.md edit is minimal and consistent with the prompt change |
| Repo Hygiene | OK | No secrets, no cross-skill refs, all changes scoped to tests/tasks/uipath-troubleshoot/ |
Issues for Manual Review
None found. The CI validation (16/16 pass) and the local n=3 rerun on the one flaky scenario (rpa-consecutive-system-exceptions — pre-existing mock starvation, unrelated to this change) are documented in the PR body.
Conclusion
Clean, well-scoped change. Every edit is a minimal prompt rewrite from calendar-relative to run-relative timing — no fixture, manifest, criterion, or judge prompt changes. The logon-failure-password-mismatch case correctly preserves the absolute date as evidence while dropping only the false relative framing. The PR body's analysis of the remaining date-word occurrences checks out. No issues found — approve.
1 participant
Problem
Troubleshoot scenario fixtures are frozen snapshots with hardcoded absolute timestamps. Sixteen scenarios paired those fixtures with a prompt that anchors the fault to the calendar - "This started today", "faulted last night", "started failing this morning", "it worked yesterday". The fixture date never moves; the calendar does, so the gap widens every day. It is now 15 to 83 days.
The mock dispatcher's matcher ignores
--created-after/--created-before, so an agent that correctly scopes its query to the window the user described still receives the out-of-window job. From the agent's side that looks like a correlation failure, and the right response is to refuse to attribute the fault to it.That is what happened on
rpa-serverless-robot-units. The agent queried with a today-window, got a job 15 days old, and stopped:It never reached
or jobs get/or jobs logs, where the decisive"Your tenant's assigned Robot Units have been exceeded"message lives, and was then marked down for hedging -0.850against a0.7judge threshold. The behaviour was correct; the fixture was contradictory.Fix
Re-anchor every fault-timing claim to the run or the schedule instead of the calendar:
No fixture, manifest, criterion, judge prompt or RESOLUTION root cause changes. The diagnostic content of every prompt is preserved - prior-success contrasts, recurrence, and the "nothing changed on my side" framing all stay.
logon-failure-password-mismatchis the one nuanced case: its absolute date2026-05-11is evidence (ADPasswordLastSetvs the Orchestrator credential record), so the date stays and only the now-false "yesterday" / "this morning" framing is dropped - in the prompt, the simulator persona, the simulator constraint, and the matching aside inRESOLUTION.md.Scope
16 scenarios, 17 files. A sweep of all 298 task YAMLs found 75 files containing a date word; 59 were left alone deliberately:
excel-lookuprange-formula-cells- "Today's published price" is quoted error text and a workbook column semantic, not a timing claim.uia-napplicationcard-view-generation,is-activities-prerelease-not-found- design-time scenarios with no dated fixtures at all, so there is nothing to contradict.Validation
Agent arm matches the nightly: codex
gpt-5.6-terra.CI
Run Coder Eval- 16/16 succeeded. 7 of the 16 carryskip: trueand the workflow does not pass--include-skipped, so they were run from a throwawayci/eval-pr2427ref with the skip removed (ref deleted afterwards).Local (all 16,
--include-skipped) - 15/16. Nine at 1.000; six at 0.850 (all above the 0.7 judge threshold); one failure,rpa-consecutive-system-exceptionsat 0.400.That failure is pre-existing and unrelated to this change - a prompt-wording edit cannot affect folder-key selection, and the same commit scored 1.000 in CI. Re-run locally at n=3 it passed 3/3 (0.962 / 0.925 / 0.925). Cause: the folders-list fixture returns three folders, the mock ignores
--output-filter, and the agent occasionally takes the firstKey(the personal workspace) instead ofOpsAutomation;or jobs list --folder-key <wrong>then matches no rule andunmocked_default: []reads as "no faulted jobs". One replicate shows the agent self-correcting mid-run - firstb2c8cb22…→unmocked_default, thena5b5c6d7…→ fixture. The 0.400 run is the case where it did not recover.Note for reviewers
The starvation pattern behind that flake is the same one this PR fixes, seen from a different angle - and it is the third instance found:
[]--created-afterwindow → out-of-window job (this PR)--output-filter→ unfiltered list → wrong key →[]In each case the mock ignores a filter flag and
unmocked_default: []withexit 0is indistinguishable from a genuine empty result. Prompt wording only addresses case 2. The general fix - makingunmocked_defaultreturn a non-zero exit so an unmatched or malformed query can never masquerade as "no results" - spans 244+ manifests and is deliberately out of scope here.A durable alternative to this PR's approach would be shifting fixture timestamps relative to the run date at seal time, so "today" is always true and no prompt has to avoid the word. That is a shared-template change affecting every scenario and needs its own validation pass, so it is also not attempted here.