Skip to content

Fix automation run failures - #952

Open
adriandemian wants to merge 3 commits into
mainfrom
ralphx/ralphx/agent-ef006d8b
Open

Fix automation run failures#952
adriandemian wants to merge 3 commits into
mainfrom
ralphx/ralphx/agent-ef006d8b

Conversation

@adriandemian

@adriandemian adriandemian commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Automation runs were failing spuriously due to two ordering bugs in the queue and publish-grace logic. Phantom agent runs were created when launch-slot contention occurred after the run row was persisted, and those phantom runs—with newer timestamps—became "latest" and caused the scheduler to abandon healthy runs. Additionally, the publish-grace timeout did not account for durable repair agents legitimately working on the publish path, triggering false timeouts. This PR reorders the queue preflight to reserve the launch slot before persisting, and wires the repair-attempt repository into the scheduler to suspend publish grace while a repair is in flight.

User Impact

Automation runs will no longer spuriously fail due to:

  • Phantom failed agent runs created by launch-slot contention
  • Premature publish-grace timeouts that orphan workspaces mid-repair

Runs will proceed normally as long as a durable repair attempt is unsettled, and will fail only after grace has elapsed with no active repair.

Technical Context

Root cause #1 — Phantom failed run in queue processing:

chat_service_queue.rs persisted the queued continuation agent run before attempting to reserve the launch slot. When the slot was occupied, fail_queued_agent_run marked the freshly-created row as failed. Since the phantom run had a newer started_at than the run that actually launched, get_latest_for_conversation (ordered DESC by started_at) returned the phantom as "latest," and the scheduler interpreted it as an agent failure and abandoned the automation run.

Fix: Reserve the launch slot before persisting the run row. On slot contention, restore the message to the queue front without creating a failed run. On persist failure, explicitly unregister the slot. This ensures only runs that actually launch are persisted.

Root cause #2 — Premature publish grace timeout:

The publish-grace timeout in scheduler.rs triggered on workspaces in needs_agent state when the elapsed time exceeded the grace. It did not check whether a durable repair agent was unsettled on the publish path, causing timeouts while the repair was still working.

Fix: Wire AgentWorkspaceRepairRepository into the scheduler and extract the grace timeout into publish_repair_grace_exhausted(). The method suspends the grace entirely if a current repair attempt is unsettled, only allowing the grace to trigger when elapsed ≥ grace AND no unsettled repair exists. Repository errors propagate instead of collapsing into "no repair" (fail-closed on reads).

Risks / Follow-Ups

None identified. The changes are surgical: queue reordering is isolated to the preflight path, and repair-grace suspension is a conservative check that does not alter the base timeout logic. Regression tests cover both the happy path (grace suspends with unsettled repair) and failure path (grace triggers with no repair).

Test coverage includes occupied_launch_slot_requeues_message_without_persisting_a_failed_run() to verify launch-slot contention does not create phantom runs and does not steal terminal authority, and two scheduler tests to verify grace behavior in both unsettled and settled repair states.


Generated by RalphX


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@adriandemian
adriandemian marked this pull request as ready for review August 1, 2026 15:47
@adriandemian
adriandemian enabled auto-merge (squash) August 1, 2026 15:47
@adriandemian
adriandemian disabled auto-merge August 1, 2026 15:49
* fix: simplify publish grace boolean for clippy nonminimal_bool

* feat: Fix current branch CI
@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.65217% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src-tauri/src/application/automation/scheduler.rs 95.65% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@adriandemian
adriandemian enabled auto-merge (squash) August 4, 2026 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant