Fix automation run failures - #952
Open
adriandemian wants to merge 3 commits into
Open
Conversation
adriandemian
marked this pull request as ready for review
August 1, 2026 15:47
adriandemian
enabled auto-merge (squash)
August 1, 2026 15:47
adriandemian
disabled auto-merge
August 1, 2026 15:49
* fix: simplify publish grace boolean for clippy nonminimal_bool * feat: Fix current branch CI
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
adriandemian
enabled auto-merge (squash)
August 4, 2026 08:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Automation runs were failing spuriously due to two ordering bugs in the queue and publish-grace logic. Phantom agent runs were created when launch-slot contention occurred after the run row was persisted, and those phantom runs—with newer timestamps—became "latest" and caused the scheduler to abandon healthy runs. Additionally, the publish-grace timeout did not account for durable repair agents legitimately working on the publish path, triggering false timeouts. This PR reorders the queue preflight to reserve the launch slot before persisting, and wires the repair-attempt repository into the scheduler to suspend publish grace while a repair is in flight.
User Impact
Automation runs will no longer spuriously fail due to:
Runs will proceed normally as long as a durable repair attempt is unsettled, and will fail only after grace has elapsed with no active repair.
Technical Context
Root cause #1 — Phantom failed run in queue processing:
chat_service_queue.rspersisted the queued continuation agent run before attempting to reserve the launch slot. When the slot was occupied,fail_queued_agent_runmarked the freshly-created row as failed. Since the phantom run had a newerstarted_atthan the run that actually launched,get_latest_for_conversation(ordered DESC bystarted_at) returned the phantom as "latest," and the scheduler interpreted it as an agent failure and abandoned the automation run.Fix: Reserve the launch slot before persisting the run row. On slot contention, restore the message to the queue front without creating a failed run. On persist failure, explicitly unregister the slot. This ensures only runs that actually launch are persisted.
Root cause #2 — Premature publish grace timeout:
The publish-grace timeout in
scheduler.rstriggered on workspaces inneeds_agentstate when the elapsed time exceeded the grace. It did not check whether a durable repair agent was unsettled on the publish path, causing timeouts while the repair was still working.Fix: Wire
AgentWorkspaceRepairRepositoryinto the scheduler and extract the grace timeout intopublish_repair_grace_exhausted(). The method suspends the grace entirely if a current repair attempt is unsettled, only allowing the grace to trigger when elapsed ≥ grace AND no unsettled repair exists. Repository errors propagate instead of collapsing into "no repair" (fail-closed on reads).Risks / Follow-Ups
None identified. The changes are surgical: queue reordering is isolated to the preflight path, and repair-grace suspension is a conservative check that does not alter the base timeout logic. Regression tests cover both the happy path (grace suspends with unsettled repair) and failure path (grace triggers with no repair).
Test coverage includes
occupied_launch_slot_requeues_message_without_persisting_a_failed_run()to verify launch-slot contention does not create phantom runs and does not steal terminal authority, and two scheduler tests to verify grace behavior in both unsettled and settled repair states.Generated by RalphX
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.