fix(flows): keep gateway tokens fresh across flow execution - #500
Closed
albertoperdomo2 wants to merge 3 commits into
Closed
albertoperdomo2 wants to merge 3 commits into
albertoperdomo2 wants to merge 3 commits into
Conversation
Multi-step flows failed mid-run with `APIError: Unauthorized: invalid_token` from the internal provider gateway: the runner synced provider access exactly once before the flow started, gateway tokens carry a short TTL (900s default), and a flow could also inherit a partially aged token when it started shortly after interactive activity. `invalid_token` was not retryable, so expiry failed the run terminally. - Force a provider-access refresh at flow start (`force` flag on `ensureProviderAccessFreshForExecution`, threaded through `ensureWorkspaceRunningForExecution`) so flows start with a full-TTL token; the credential-hash check and active-run deferral still apply. - Refresh provider access before each flow node in `executeFlowNodes`. At a step boundary the flow's own message run is finalized, so this only defers when an unrelated run is active — the case where a concurrent sync (which disposes the instance) must not abort in-flight generation. A failed boundary refresh warns and continues. - Classify `invalid_token` as retryable so an expiry that survives the above costs one step re-run after backoff; the retry re-syncs and resumes from the failed node. No changes to token TTL, claims, or issuance. Mid-step refresh without dispose is deliberately left for a follow-up pending verification that OpenCode re-reads auth keys per request. Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
Author
|
/build |
|
📦 PR Workspace Image Built Successfully Default (amd64): Optional arm64: |
|
📦 PR Image Built Successfully Default (amd64): Optional arm64: |
6 tasks
Contributor
Author
|
/rerun |
The step-boundary refresh used the freshness-threshold check, which is tuned for interactive cadence: it only fires once the sync is older than TTL minus the 60s skew. Flow steps run for minutes, so boundaries rarely land in that window — an 11-minute step started with 10 minutes of token life left and still failed at expiry (2026-09-02 11:07 run), defeating the between-steps refresh for exactly the flows it was built for. Force the refresh before every step after the first, so each step starts with the full gateway-token TTL; mid-step expiry now requires a single step longer than the TTL, which the retryable classification bounds to one step re-run. The first iteration stays exempt because the run entry points already force a fresh sync before the loop. Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
Author
|
/build |
|
📦 PR Workspace Image Built Successfully Default (amd64): Optional arm64: |
|
📦 PR Image Built Successfully Default (amd64): Optional arm64: |
6 tasks
Contributor
Author
|
Superseded by #478, which absorbs the token-freshness work — forced refresh at flow start, forced between-steps refresh (the threshold check skips for multi-minute steps), and retryable |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Multi-step flows fail mid-run with
APIError: Unauthorized: invalid_tokenfrom the internal provider gateway. Earlier steps succeed; a later step is rejected once the gateway token — TTL 900s by default (ARCHE_GATEWAY_TOKEN_TTL_SECONDS) — expires.Two failure shapes, both observed on 2026-08-31 (Codebase Hunter flow runs):
shouldRefreshProviderAccessthreshold) and inherited a token with <8 min of TTL left, which expired mid-run.During execution tokens were never refreshed, and
invalid_tokenwas absent fromisRetryableFlowRunError, making expiry a terminal run failure even though retries resume from the failed node and re-sync on the way in.Changes
ensureProviderAccessFreshForExecutiongainsforce, which bypasses only the freshness-age skip; the provider-sync lock, credential-hash check, and active-run deferral still apply. Threaded throughensureWorkspaceRunningForExecution(slug, userId, { forceProviderRefresh: true })from both flow entry points (executeClaimedFlowRun,resumeClaimedFlowRun). Only the already-running path honors the flag — a freshly started workspace was just synced.executeFlowNodescallsensureProviderAccessFreshForExecutionbefore each node, after the cancellation and lease checks. At a boundary the flow's own message run is finalized, so the deferral fires only for unrelated active runs — exactly when a concurrent sync (which disposes the OpenCode instance) must not abort in-flight generation. A failed boundary refresh logs a warning and does not fail the run; genuine auth failures still surface from the step itself.invalid_tokenadded toisRetryableFlowRunError, so an expiry that slips through costs one step re-run after backoff instead of failing the run.Security posture
No new token class and no TTL/claims/issuance changes: tokens stay short-lived and scoped per user, workspace, provider, and credential version. Refreshes serialize through the existing per-slug sync lock, and
forceis set only by server-side flow paths, never from user input.OpenSpec
Adds
flow-gateway-token-refreshchange with aflow-executionspec delta (start-of-run freshness, step-boundary freshness, retryable gateway auth failures).openspec validate flow-gateway-token-refresh --strictpasses.Test plan
providers.test.ts— forced refresh runs despite a fresh matching sync record; forced refresh still defers while the workspace has active runssession-execution.test.ts—forceProviderRefreshthreaded through asforce: trueon the running pathrunner.test.ts— flow start passes the force flag; one refresh per node in a two-node flow; run still succeeds when the boundary refresh rejectsretry-policy.test.ts—APIError: Unauthorized: invalid_tokenclassified retryablepnpm exec vitest run— 146 tests across the touched and transitively affected suites pass; eslint clean on changed filesNon-goals / follow-up
Mid-step refresh without the instance dispose (
disposeInstance: falsealready exists) is left for a follow-up: it only helps if OpenCode re-reads auth keys per request instead of caching them at provider creation — needs one verification against a live instance. Operators running long single steps can raiseARCHE_GATEWAY_TOKEN_TTL_SECONDSabove the longest expected step in the meantime.Depends on #497 for scheduled flows to trigger at all on
archectldeployments, but is independently correct for manual runs and the Ansible path.