🔁 fix: Retry Code API Rate Limits That Reject Before a Workspace Call Starts

[danny-avila]

Summary

When several agents share one attached (BYOM) code environment, a burst of workspace calls trips the Code API's per-user rate limiter, which returns 429 with a typed body ({"error":"rate_limited","retry_after_seconds":N}) and a Retry-After header. LibreChat surfaced every one of those as a failed tool call ("Workspace tool request rejected (upstreamStatus: 429)"), so the model saw a hard failure and often gave up or improvised. On one deployment 18 of 72 workspace calls (25%) failed this way during a parallel-agent burst.

The limiter runs as middleware ahead of the Code API's workspace router, so a typed 429 means the operation was never assigned or started, the same guarantee a 503 WORKSPACE_QUEUE_TIMEOUT already carries. This change treats both as admission rejections, but bounds their retries separately: 503 uses the workspace queue horizon, while typed 429s use the existing endpoints.agents.codeApiMaxRetryWaitMs budget (20 seconds by default; 0 disables rate-limit retries), even when capacity retries are disabled or exhausted. Both remain subject to the caller HTTP deadline and its reserved execution time. The client prefers Retry-After, falls back to the typed body’s retry_after_seconds when a proxy drops or mangles that header, and caps each wait at 30 seconds. When a budget runs out, the error tells the model the operation was not started and to slow down. Anything else under 429 or 503, including an untyped or truncated body, keeps an unknown outcome and is never retried.

Related to LibreChat-AI/code-interpreter#267 (BYOM defaults and parallel-agent limits).

How it works

response not ok
  getWorkspaceAdmissionRejection(status, body, truncated)
    503 + {"code":"WORKSPACE_QUEUE_TIMEOUT"}  -> 'queue_timeout'
    429 + {"error":"rate_limited"}            -> 'rate_limited'   (new)
    anything else / truncated / unparsable    -> undefined -> throw, never retried
  503 -> queue horizon (maxQueueWaitMs)
  429 -> actual rate-limit waits (codeApiMaxRetryWaitMs)
    wait Retry-After or body retry_after_seconds (<= 30 s)
    re-mint credentials and resend the identical body

Queue waits never consume the rate-limit wait budget, and an accepted rate-limit wait earns one retry even if its timer fires late. An unaffordable rate-limit hint fails immediately; later 429s cannot borrow time from a spent budget. The three request sources (Bash, file operations, and repository instructions) use the same request config. Each retry obtains fresh credentials; cached instructions still require authorization. A protected preview/edit retains the capacity retry horizon across both requests, but a successful preview always permits one hash-guarded edit attempt even when that horizon is spent. Untyped 429s and ambiguous failures are never retried.

Type of change

• Bug fix

Testing

Tested environments/configuration:

• Focused tests use a simulated attached BYOM transport. The Code API contract was checked against indexed code-interpreter main (337ddd60a8ea): service/src/middleware/limits.ts emits the typed 429 body and service/src/workspace-tools/index.ts installs the limiter before the router. A live BYOM request was not rerun for this head.

Automated tests:

• packages/api: 523 passed across src/code/workspace.spec.ts, src/code/instructions.spec.ts, src/code/command.spec.ts, src/agents/handlers.spec.ts, src/agents/__tests__/initialize.test.ts, and src/utils/code.spec.ts. Coverage includes isolated 503/429 horizons, timer jitter, cancellation, HTTP execution reserve, malformed delay hints, input limits, authorization refresh, and protected edit preview-to-mutation transitions.
• api: 308 passed across server/services/Files/Code/process.spec.js and server/services/__tests__/ToolService.spec.js, including both definitions-only and eager request-config propagation.
• packages/api: no-emit TypeScript check and production package build. npm run static-checks -- --against origin/dev passed all affected PR-wide gates, including the circular-dependency scan. Local Lighthouse was not run because Chrome is unavailable; the PR has a Lighthouse CI lane.

[danny-avila]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-28T16:44:57.214220Z	de32233	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

LibreChat/packages/api/src/code/workspace.ts

Lines 940 to 942 in 3247167

	const delayMs = Math.min(
	workspaceAdmissionRetryDelay(response.headers.get('Retry-After')),
	retryDeadlineAt - Date.now(),

Honor the rate-limit delay from the response body

When a proxy drops or mangles Retry-After, this passes only the missing header to workspaceAdmissionRetryDelay, so every typed 429 is retried after the one-second default instead of its retry_after_seconds value. The Code API body field is explicitly the proxy-safe fallback already supported by getCodeApiRetryAfterMs in packages/api/src/utils/code.ts; ignoring it here can generate repeated premature requests and exhaust the retry horizon before the limiter permits another call. Parse the buffered body’s delay when the header is unusable.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[lia-by-librechat]

Follow-up ready at b1565d86240f7b7c9fd49a45ec481e66f7f13a5e: typed workspace 429 retries now honor the configured Code API rate-limit wait budget independently of queue waits, and use the typed response body's retry_after_seconds if Retry-After is missing or invalid. Added coverage for disabled/exhausted budgets, the body-delay fallback, and mixed 429/503 sequences.

[lia-by-librechat]

Follow-up head 9c406fdfb1c3434ffa515d1de7bf4a3da50ad8f3 removes the circular import reported by CI while preserving the shared Code API retry default. The circular-dependency gate now passes locally. Focused new-head tests and TypeScript checks are running alongside CI.

[lia-by-librechat]

Latest pushed head de32233b99d9fcdf1ca9c03597c8967b52a6e17d also fixes the API declaration emit error from the prior head. The packages/api build and circular-dependency check pass locally; final-head focused tests and typecheck are running now.

[danny-avila]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: de32233b99

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[lia-by-librechat]

Review handoff for exact pushed head c19a73114cdb21d9f24c6712cd56d24716275d4a (base dev). This head separates the 503 capacity horizon from the 429 Code API wait budget, honors an already-approved wait despite timer jitter, and passes the configured budget to repository-instruction reads in both initialization paths. Subsystem sweep: protected preview/edit still gets one hash-guarded edit attempt when its queue retry horizon has elapsed; auth refresh, cancellation, mixed 429/503 transitions, malformed delay hints, invalid limits, and caller HTTP reserve have focused coverage. Local checks passed: 523 focused TypeScript tests, 308 legacy adapter tests, packages/api no-emit typecheck and build, circular-dependency scan, and PR-wide Static Checks against origin/dev. CI is running on this head. Maintainer review is needed for this exact SHA; a Lia App comment cannot trigger Codex.