[freeqaz]

Problem

• One transient provider error fails a deployment for good. The dedicated Codex startup model probe retries only MODEL_PROBE_TIMEOUT, once, inside a 61 s budget. Any other failure is held at once. The worker then marks the deployment failed_permanent on attempt 1. So a single provider 5xx, overload or dropped stream fails a redeploy permanently (finding 1576).
• A misleading 404 every 30 s. While a failure is held, the wrapper's held-failure server answers the Agent Gateway's authenticated WebSocket upgrade with 404 not_found. The Gateway logs that 404 every 30 s, which looks like a routing fault (finding 1577).

Change

• Probe retry (runtime-entrypoints.ts). A transient failure now gets the single retry that a timeout gets, within the same 61 s budget and with the same 1 s delay. It is one retry in total: a timeout followed by a transient failure holds. Transient means:
  • a Codex error/turn.failed whose message carries a provider 5xx status, server_error, overload, "at capacity" or "high demand";
  • a dropped stream (stream disconnected), error sending request, a failed/reset/refused/closed connection, or a broken pipe;
  • a probe process killed from outside (SIGKILL/SIGTERM with no spawn error, so not its own timeout or an output overflow).
• Same-attempt veto. A transient failure is only a candidate while the attempt's events are scanned. A later deterministic failure in the same attempt (a tool event, a rate limit) still holds at once.
• What still holds at once. A message with a rate-limit or credential marker, or a refused Reconnecting... notice, is never transient. Auth rejection, no model, rate limit/billing, malformed output, tool events, no reply and nonzero exits also stay immediate. The worker's permanent-failure rule is unchanged, and so is how credentials reach the probe.
• Persisted cause. The held cause is the last attempt's classified cause. When the check was retried (after a timeout or a transient failure) and that cause has a detail token, the detail ends -attempt2, for example turn-failed-attempt2. Codes without a cause, WRAPPER_ERROR and a detail-less PROCESS_EXIT show no marker. The codex.model_probe log line keeps each attempt's own cause and number. Compute's closed cause vocabulary (runtime-failure-cause.ts) accepts exactly that one suffix on a valid token. The contract and DB already allow ^[A-Za-z0-9_-]{1,32}$, and the longest suffixed token is 31 characters, so there is no contract or schema change.
• Held-failure server. An authenticated request other than GET /openclaw/runtime/status now gets 503 {"error":"runtime_failure_held","code":"<held code>"} (no-store) instead of 404. A missing or wrong bearer still gets 401, and the status route is unchanged. Compute's handshake-first harnessStatus treats a refused upgrade the same way as before. The doc comment and integration test are updated.
• Docs. harness-execution.md (net zero words) and agent-deployment-diagnostics.md describe the shared retry, the -attempt2 suffix and the 503.
• Finding 1578 stays open. The persisted cause can't carry providerStatus/providerCode without a contract and migration change. The cause is additionalProperties: false in the OpenAPI schema, and the occ DB CHECK allows only kind/detail.

Verification

• New tests fail on main. Running codex-native-startup, openshell-harness-status and plugin-compute against main's runtime-entrypoints.ts and runtime-failure-cause.ts gives 34 failures:

  • each transient class recovering on attempt 2;
  • each class failing twice and holding <detail>-attempt2;
  • transient then deterministic (tool-event-attempt2), transient then 401 (AUTHENTICATION_FAILED), transient then timeout;
  • timeout then transient (held with no second retry), and the remaining-budget timeout;
  • the 503 assertions, including a real-socket WebSocket upgrade against the real wrapper;
  • the vocabulary cases.

• Mixed events within one attempt (review thread):

  • transient then tool event holds tool-event;
  • transient then final 429 holds rate_limit;
  • transient then 401 gives AUTHENTICATION_FAILED;
  • transient then a completed reply is still retried;
  • two transients in one attempt get one retry, and attempt 1 reports the first;
  • two transients then a tool event hold tool-event;
  • a valid Reconnecting... notice before or after a transient error is still accepted, and the attempt is retried.

  The first two fail on the previous head.

• Deterministic cases hold immediately, with the fixture asserting that no retry timer is set: missing model (404 model_not_found), a 500 naming the API key, a rate limit naming an overload, a probe killed for oversized output (error-ENOBUFS), plus the existing tool-event, no-reply, rate-limit, auth, malformed-output and exit cases.

• Mutants: 26 of 26 killed. They cover:

  • the same-attempt veto (candidate returned at once; last candidate kept instead of the first; candidate dropped);
  • the retry count (two transient retries; transient never retried);
  • the transient retry ignoring the budget;
  • each transient class marker: 5xx, server_error, overloaded, at capacity, high demand, stream disconnected, error sending request, connection, broken pipe, SIGKILL, SIGTERM;
  • any process error counting as transient;
  • each exclusion: rate limit, credential marker, Reconnecting;
  • the missing -attempt2 marker;
  • 404 restored, and the 503 body without the code;
  • the vocabulary allowing any -attemptN, or no suffix.

• node --test on codex-native-startup, codex-stderr-filter, kubernetes-compute, kubernetes-runtime-credentials, openshell-harness-status, plugin-compute, runtime-logs-content, integration/codex-model-probe and integration/sandbox-driver-startup: 814 pass, 2 skipped (environment-gated), 0 fail.

• integration/codex-model-probe (Docker, run in CI's Images and Packaging lane) now expects an external SIGKILL to get the one retry: killed twice holds signal-SIGKILL-attempt2, and a new case is killed once then succeeds. This host's Docker rejects the test's tmpfs uid mount option, so it could not run locally.

• The touched tests ran 20/20 at 2 CPUs (taskset -c 0,1).

• tsc --build is clean. prettier --check and eslint --max-warnings 0 are clean on the touched files.

• No new conflicts with open runtime PRs (git merge-tree, fix head vs main). For each PR the conflicted-file set is identical, and Enable dedicated OpenClaw from Console with pinned native runtime #1685's one hunk in sandbox-driver-startup.test.mjs is unchanged:

  PR	vs main	vs this PR
  Credential Proxy Implementation #2487	3 files	same 3 files
  Enable Codex OAuth with OpenShell-owned credentials #1559	clean	clean
  Enable dedicated OpenClaw from Console with pinned native runtime #1685	3 files	same 3 files
  feat(runtime): apply Gateway tool policy to dedicated OCE deployments #2633	clean	clean

• Internal review: approve with nits. Applied: doc wording for -attempt2 (it also follows a timeout retry, and it applies to all dedicated Codex), plus the PR body clarifications above. A second internal review of the same-attempt veto delta also approved with nits: stronger tests were applied, and the duplicate classification call on the error branch is kept so the lines Credential Proxy Implementation #2487 edits stay untouched. Left as follow-ups: the console's rate_limit/billing text lookup for suffixed tokens (a console change that needs Storybook media) and the contract's detail description (needs OpenAPI regeneration).

Risks

• An out-of-memory kill is retried. A probe killed with SIGKILL for lack of memory gets one more attempt and will likely fail again. That attempt stays inside the same budget.
• A deterministic failure can look transient. If its message also contains a transient marker, it now costs one extra probe of up to 30 s within the existing budget. The worst case is a slower hold with the same final code.
• Console text after a retry. The console's special text for rate_limit/billing doesn't match a suffixed rate_limit-attempt2, which only happens when a transient failure is followed by a rate limit. It then falls back to the generic label with the token shown.
• Credential Proxy Implementation #2487 needs a follow-up when it lands. It disables the timeout retry for providerTools: "none" profiles, and should apply the same guard to the new transient retry line beside it. A note is on the board.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review blocked

Automated review did not run, so no review verdict was produced.

Reason: The input-safety check rejected material in this revision. No detected value, path, or scanner output is reproduced here.

Automatic review is on hold, including after source changes. Request a fresh re-review after resolving the failure; maintainers can also release the hold through an intentional scanner-policy update.

Next step: If this is a genuine credential, remove and rotate it. If it is an intentional test fixture, a maintainer must review and qualify it.

View the workflow run.

[freeqaz-openai]

One correctness fix before landing: a later deterministic failure in the first buffered attempt must veto the new transient retry. I reproduced the order dependence in the actual wrapper and verified a small correction; the existing 84 owning tests pass with it. This is independent of a real OpenShell rollout or the deferred Console copy.