fix(flows): recover orphaned runs and add Stop button - #478
Merged
Merged
Conversation
Scheduled flows could get permanently stuck when finalizeRun failed silently — the lease was released but the run stayed in 'running' status. recoverStaleRunningRuns only checked for expired leases (leaseExpiresAt < now), missing the NULL-lease case entirely. The noActiveRun guard then blocked all future claims for that flow. Two backend fixes: - Expand recoverStaleRunningRuns WHERE to also match NULL leases - Don't release the lease in settleFlowRun when finalizeRun fails; let it expire so the existing recovery path handles it Surface the existing cancel API as a Stop button in three places: flow list, run history header, and individual run cards.
…recovery The test asserted the old WHERE clause shape. Updated to match the new OR condition that also recovers runs with a released (NULL) lease.
albertoperdomo2
force-pushed
the
fix/flow-orphaned-runs-stop-button
branch
from
August 28, 2026 21:41
fe61218 to
88ea307
Compare
archectl-generated deployments never set ARCHE_FLOW_SCHEDULER_MODE, and the web app requires it in production (getFlowSchedulerMode throws when missing), so one-click servers failed at web startup. The Ansible remote path already sets it to daemon. Set ARCHE_FLOW_SCHEDULER_MODE=daemon in renderEnvFile, the bootstrap env heredoc, and the update script (idempotent set_env, so existing deployments converge on next update). Because daemon mode expects a dedicated runner, the generated compose also gains a flows service running the web image's flow-daemon entrypoint — mirroring the Ansible remote template — and the update script brings flows up alongside web.
Multi-step flows failed mid-run with `APIError: Unauthorized: invalid_token` from the internal provider gateway: the runner synced provider access exactly once before the flow started, gateway tokens carry a short TTL (900s default), and a flow could also inherit a partially aged token when it started shortly after interactive activity. `invalid_token` was not retryable, so expiry failed the run terminally. - Force a provider-access refresh at flow start (`force` flag on `ensureProviderAccessFreshForExecution`, threaded through `ensureWorkspaceRunningForExecution`) so flows start with a full-TTL token; the credential-hash check and active-run deferral still apply. - Refresh provider access before each flow node in `executeFlowNodes`. At a step boundary the flow's own message run is finalized, so this only defers when an unrelated run is active — the case where a concurrent sync (which disposes the instance) must not abort in-flight generation. A failed boundary refresh warns and continues. - Classify `invalid_token` as retryable so an expiry that survives the above costs one step re-run after backoff; the retry re-syncs and resumes from the failed node. No changes to token TTL, claims, or issuance. Mid-step refresh without dispose is deliberately left for a follow-up pending verification that OpenCode re-reads auth keys per request. Co-Authored-By: Claude Code <noreply@anthropic.com>
The step-boundary refresh used the freshness-threshold check, which is tuned for interactive cadence: it only fires once the sync is older than TTL minus the 60s skew. Flow steps run for minutes, so boundaries rarely land in that window — an 11-minute step started with 10 minutes of token life left and still failed at expiry (2026-09-02 11:07 run), defeating the between-steps refresh for exactly the flows it was built for. Force the refresh before every step after the first, so each step starts with the full gateway-token TTL; mid-step expiry now requires a single step longer than the TTL, which the retryable classification bounds to one step re-run. The first iteration stays exempt because the run entry points already force a fresh sync before the loop. Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
Author
|
/build |
|
📦 PR Workspace Image Built Successfully Default (amd64): Optional arm64: |
|
📦 PR Image Built Successfully Default (amd64): Optional arm64: |
Cancelling a run flipped only the run row to `cancelled`; its in-flight step stayed `running` forever, so the run history kept rendering a spinner for a dead run. The runner never revisits step rows after cancellation — finalizeRun early-returns and executors skip step updates for cancelled prompts — and stale-recovered runs have no living runner at all. Settle `pending`/`running`/`waiting_for_human` steps to `failed` with the corresponding error in the same service operation that settles the run: both cancel paths (`cancelRunById`, `cancelRunByIdForScope`) and stale-run recovery. Already-final steps are untouched. Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
Author
|
/build |
|
📦 PR Workspace Image Built Successfully Default (amd64): Optional arm64: |
|
📦 PR Image Built Successfully Default (amd64): Optional arm64: |
4 tasks
The workspace Flows section passed `navigateToHistoryOnRun`, so clicking Run on a flow card yanked the user to the run history view; the web list already refreshed in place. Both surfaces now stay on the list. To make staying informative, the list silently re-fetches every 5s while any visible flow has an active run (`running`/`waiting_for_human`), so badges and stop controls track progress without navigating away or flashing the loader. Polling tears itself down once no run is active. Co-Authored-By: Claude Code <noreply@anthropic.com>
The provider-sync deferral checked for active runs before the auth PUTs, but the instance dispose fired after them. A run registering during the PUTs was disposed mid-generation and every such generation died with MessageAbortedError: Aborted — persisted across fresh sessions for as long as syncs kept landing in generation windows (2026-09-02 13:37 incident: six consecutive runs aborted over four minutes, then self-healed when the sync cadence settled). Re-check for active runs immediately before the dispose, after the writes: the keys are already updated by then, so only the destructive dispose is skipped and provider-discovery reload defers to the next sync. Unlike the deferral, this check does not exclude the caller's own run — an in-flight generation must never be disposed underneath itself. Also log dispose decisions and session-family abort targets so runtime MessageAbortedError reports can be correlated with their source. Co-Authored-By: Claude Code <noreply@anthropic.com>
…into fix/flow-orphaned-runs-stop-button
…' into fix/flow-orphaned-runs-stop-button
reapStaleRuns — which fails message runs stuck in `running` past the 35-minute timeout and deletes their locks — is only called by the standalone reaper daemon, so one-click deployments never reaped stale locks (five have been stuck since July). Mirror the Ansible reaper service into renderCompose and bring it up in the update script alongside web and flows. Co-Authored-By: Claude Code <noreply@anthropic.com>
… runs Three follow-ups to the stop/recovery work: - A provider sync whose dispose was skipped (run started during the sync) no longer records a fresh sync state. Recording it made later syncs short-circuit on the fresh timestamp while the instance kept running on stale cached provider discovery — the keys were in storage but never loaded. Withholding the record makes the next idle-boundary sync re-write credentials and dispose. Disposal intentionally disabled (disposeInstance: false) still records. - A prompt whose runtime termination could not be confirmed now fails its tracked message run instead of leaving it `running` for up to 35 minutes, which blocked workspace provider syncs. The flow run itself stays preserved for lease-expiry recovery, per the existing never-finalize-unconfirmed-termination invariant. - flow_no_assistant_output now carries the provider, model, and part type profile (reasoning-only vs tool-only vs empty) as a suffix, and logs the full diagnostics — the empty sub-second responses reported on 2026-09-02 become self-describing. Co-Authored-By: Claude Code <noreply@anthropic.com>
…hanges - session-executor.test.ts had a leftover closing brace from the no-output test rewrite (parse error, broke lint and the suite). - tests/opencode-providers.test.ts mocked @/lib/services without messageRunService, so the new dispose guard threw and the sync returned sync_failed before disposing. Mock hasActiveRunForSlug explicitly. Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
Author
|
/build |
|
📦 PR Workspace Image Built Successfully Default (amd64): Optional arm64: |
|
📦 PR Image Built Successfully Default (amd64): Optional arm64: |
This was referenced Sep 3, 2026
Disposing exits the OpenCode process and Docker restarts the container (5-15s), but the sync returned immediately and callers created sessions against the dead instance — the model produced an assistant message with zero content parts and the step failed with flow_no_assistant_output. Deterministic whenever a flow started more than 14 minutes after the last sync (the refresh threshold), which made scheduled flows fail on the first run after every idle gap. Poll /global/health (bounded 30s, 1s interval, 3s probe timeout) after the dispose and only hand control back to callers once the instance reports healthy; warn if it never does. The SDK exposes no soft-reload alternative to dispose, so the restart cannot be avoided from our side. Co-Authored-By: Claude Code <noreply@anthropic.com>
…wait Add the flows-list requirement (stay mounted after starting a run, silent polling while a run is active) that rode in from the closed list-redirect fix without a spec delta, and task entries for the dispose health wait. Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
Author
|
/build |
|
📦 PR Workspace Image Built Successfully Default (amd64): Optional arm64: |
|
📦 PR Image Built Successfully Default (amd64): Optional arm64: |
Contributor
Author
|
@Inakitajes this is probably the one with higher priority and the once that probably need a more thorough review. |
- Keep active flow controls mounted during silent refresh failures and surface run-card stop errors - Settle stale runs safely and wrap cancellation settlement in a transaction - Harden PR scanner inputs, malformed manifest handling, and manual dispatch checks - Cover silent flow refresh and card Stop errors - Exercise workflow scanner and dispatch regressions - Execute malformed-manifest governance behavior Convoy-Run: 20260915-112714-x8qg
The hardened secret scanner matches quoted password/secret assignments of eight or more characters, which caught two test fixtures added by this branch: - the governance scanner's own test built a literal 'password = ...' sample, so assemble it at runtime instead - the one-click env test reused production-length placeholder secrets; shorten them to obvious placeholders The scanner itself is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Flow-execution reliability umbrella: the stop button plus fixes for eight defects found while testing flow runs in production between 2026-08-31 and 2026-09-02. Each problem is described inline below with its fix. Absorbs #497 and #500 (both will be closed when this merges).
1. The scheduler never starts on
archectldeploymentsWeb containers crash-loop with:
archectlnever wrote that variable, while the Ansible path sets it — so scheduled flows and stale-run recovery never ran on one-click deployments at all.Fix:
renderEnvFile, the bootstrap heredoc, and an idempotentset_envin the update script all writeARCHE_FLOW_SCHEDULER_MODE=daemon; the compose stack gains aflowsservice runningsrc/flow-daemon.ts, and the update script recreates it alongsideweb.2. Gateway tokens expired mid-flow (
APIError: Unauthorized: invalid_token)The runner synced provider access once per flow, gateway tokens live 15 min by default, and two real runs died: a 17-minute run whose token expired between steps, and a run that started 7 minutes after an interactive visit and inherited a partially aged token.
invalid_tokenwas not retryable, so expiry failed the whole run terminally.Fix: force a provider-access refresh at flow start (full-TTL token at step 1), force it again between steps — a freshness-threshold check is useless here because multi-minute steps rarely land in a threshold window — and classify
invalid_tokenas retryable so any residual expiry costs one step re-run instead of the run. No token-scheme changes: same short TTL, same per-user/workspace/provider scoping, refreshes serialized by the existing per-slug lock.3. Cancel aborted unrelated generations, then the guard for it created a stale-config trap
Cancelling a run was followed by a 4-minute cascade: six fresh runs each aborted within seconds (
MessageAbortedError: Aborted), self-healing only after the sync cadence settled. Root cause: provider syncs end withPOST /instance/dispose(which aborts any in-flight generation), and the active-run deferral was checked before the multi-second credential PUTs — a run registering during that window was disposed mid-generation.Fix: re-check active runs immediately before the dispose, after the writes. Because that skip can leave the instance on stale cached provider discovery, a skipped dispose now withholds the sync-state record, so the next idle-boundary sync re-writes credentials and disposes instead of short-circuiting on a fresh timestamp. Dispose decisions and session-family abort targets are logged so any future
MessageAbortedErroris correlatable to its source.Additionally: after a dispose the sync now waits for
/global/healthto report healthy (bounded 30s) before returning — dispose exits the OpenCode process and Docker restarts it (5–15s), so returning immediately handed callers a dead container. That produced deterministicflow_no_assistant_outputfailures with empty assistant messages (partTypes: []) on the first flow after any >14-minute idle gap; the one success in the cascade was luck. The SDK exposes no soft-reload alternative to dispose, so the restart itself cannot be avoided from our side.4.
termination_unconfirmedleaked message runs for 35 minutesWhen an abort could not be confirmed idle within its 10s window, the flow run is deliberately preserved (unconfirmed termination must never be finalized — the generation may still be winding down), but the tracked message run stayed
runningtoo, and its lock blocked provider syncs for the whole 35-minute message-run timeout. Five message runs stuck since July followed exactly this shape.Fix: the message run is marked failed with the termination cause (it is definitively over — abort was requested); the flow run itself still waits for lease-expiry recovery.
5. Recovery existed but had no worker on one-click deployments
Stale-run recovery fires inside the scheduler tick, so bug #1 silently disabled it; and message-run reaping (
reapStaleRuns) is called only by the standalone reaper daemon, which the archectl compose never deployed.Fix (with #1): the archectl stack now also runs the
reaperservice (src/reaper-daemon.ts), and the update script brings upweb flows reaper. Stale locks and orphaned runs now self-heal everywhere.6. Five message-run locks stuck since July 2026
Legacy evidence of bug #4. The reaper (above) prevents recurrence; to clear the existing rows immediately, run against production:
7.
flow_no_assistant_outputwith sub-second empty responsesFour steps failed with completed sessions but no assistant text (0.3–1.1s responses, no provider error). Provider auth was verified healthy, so the likely cause is model-side (reasoning-only or tool-only replies). The error was undiagnostic.
Fix: the error now carries
<provider>/<model>:<partTypes>(e.g.flow_no_assistant_output:openrouter/glm-5.2:reasoning+step-finish) and logs the full profile (agent, message counts). Root-causing the model behavior itself needs live session inspection and remains an ops follow-up.8. Stop button UX
The flows list previously redirected to run history on Run; it now stays in place and silently refreshes every 5s while a run is active. Cancelling a run settles its in-flight step records (previously the step spinner rendered forever on a
cancelledrun).OpenSpec
flow-run-recovery-and-stop,flow-gateway-token-refresh, andarchectl-flow-scheduler-modechanges ride on this branch; all validate--strict.Test plan
apps/web: 508 files / 5,199 tests passing (including the roottests/directory)gofmt/go vet/go test ./...ininfra/one-click— 17 tests, including rendered-compose assertions for theflowsandreaperservices andARCHE_FLOW_SCHEDULER_MODEPost-merge checklist
executeFlowNodesrewrite composes with the forced boundary refresh