📶 fix: Ride Out Transient Worker Status Failures Within the Worker's Lease

[danny-avila]

Summary

Before each turn that uses an attached (BYOM) code environment, LibreChat polls the Code API for the worker's status. Any failure of that one HTTP request, including a connect timeout at the proxy in front of the Code API, was treated as the machine being gone: the turn failed with "The attached code environment is unavailable. Reconnect the machine and try again." The poller also shares one in-flight request across concurrent turns, so a single blip failed every turn starting at that moment.

On one deployment the worker stayed connected and kept completing calls every minute, yet 5 turns failed within two minutes because 2 status requests from the cluster timed out connecting to Cloudflare's edge. That was 3 failed status requests out of 4,788 in a day, and each one cost the user a turn.

A transient transport failure (timeout, network error, or a 5xx from the Code API or its proxy) says nothing about the worker. The worker's own heartbeat lease does. The poller now remembers the last ready observation per base URL, worker and credential. When a poll fails transiently while that observation's lease is still running, the remembered status answers for it, with the lease time that remains. With no valid observation, the poll is retried once. Rejections (4xx), malformed responses and non-ready statuses fail as before and discard the remembered status. The bypassCache path, which validates a workspace selection before it is persisted, is unchanged and still gets exactly one fresh observation.

How it works

poll (shared cache as before)
  fetch status
    ok        -> remember it if ready with a lease; else forget
    4xx/invalid -> forget, throw
    timeout / network / 5xx
      remembered ready status whose lease has not run out -> return it (lease shortened to what remains), warn
      otherwise -> fetch once more
bypassCache (selection validation): one fetch, no retry, no remembered status

The remembered lease is measured from when the request started, so it never outlives the lease the Code API reported. If the worker really disconnected, its lease runs out within about a minute and polls fail normally, and any workspace call made in the meantime still fails with its own typed error at execution.

Type of change

• Bug fix

Testing

Tested environments/configuration:

• Attached BYOM environment reached through a Cloudflare tunnel; the failure pattern above came from its traces.

Automated tests:

• packages/api: npx jest src/code (402 passed, 1 skipped). New cases in bridge.spec.ts cover each behaviour against the real poller with a scripted fetch:
  • the remembered status answers within the lease;
  • a proxy 5xx counts as transient;
  • one retry when nothing is remembered;
  • failure after the lease runs out;
  • a rejection is never covered and clears the remembered status, on both paths;
  • a non-ready status clears it;
  • bypassCache never uses it;
  • remembered statuses stay separate per credential.
• An existing capacity test that used a 503 now uses a 403, because 5xx is transient by design.
• npx tsc --noEmit in packages/api; eslint on the changed files.

[danny-avila]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-28T20:15:52.291368Z	e2ff70e	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: e2ff70e252

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".