Repository navigation
♨️ feat: Prewarm Stateful Code Sandboxes with Cold-Boot UX Feedback - #14239
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a best-effort “prewarm” for stateful Code API sandboxes and threads a new SSE event through to the UI so users see explicit cold-boot feedback (“Starting sandbox environment”) instead of a generic running state on first tool use.
Changes:
- Introduces a per-conversation sandbox warm/cold tracker with fire-and-forget prewarm requests to the Code API.
- Emits and handles a new
on_sandbox_startingstep event, driving a new Recoil atom family to update tool-call progress labels. - Refreshes sandbox “warm window” tracking after real sandbox-bound tool invocations.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/data-provider/src/types/runs.ts | Adds StepEvents.ON_SANDBOX_STARTING and a typed payload for client consumption. |
| packages/api/src/agents/prewarm.ts | Implements sandbox prewarm + warm-window tracking (maybePrewarmCodeSandbox, shouldSignalSandboxStart, markSandboxReady). |
| packages/api/src/agents/prewarm.spec.ts | Adds unit tests covering gating, dedupe, expiry/refire, kill switch, and failure paths. |
| packages/api/src/agents/index.ts | Re-exports the new prewarm module API from the agents package surface. |
| packages/api/src/agents/handlers.ts | Marks a sandbox session “ready” after sandbox-bound tool invocations to keep warm-window tracking accurate. |
| client/src/store/sandbox.ts | Adds sandboxStartingByToolCallId Recoil atom family for per-tool-call “sandbox starting” state. |
| client/src/store/index.ts | Exports the new sandbox store module. |
| client/src/locales/en/translation.json | Adds com_ui_sandbox_starting localization string. |
| client/src/hooks/SSE/useStepHandler.ts | Handles ON_SANDBOX_STARTING SSE events and clears the state on run-step completion. |
| client/src/components/Chat/Messages/Content/Parts/ExecuteCode.tsx | Swaps in-progress label to “Starting sandbox environment” when the sandbox-starting atom is set. |
| client/src/components/Chat/Messages/Content/Parts/BashCall.tsx | Swaps in-progress label to “Starting sandbox environment” (unless still “Writing command…”). |
| client/src/components/Chat/Messages/Content/Part.tsx | Threads toolCallId down into tool-call parts so UI state can be keyed per tool call. |
| api/server/controllers/agents/client.js | Kicks off prewarm at the start of chat completion to overlap sandbox boot with model generation. |
| api/server/controllers/agents/callbacks.js | Emits on_sandbox_starting for code tool calls when a stateful sandbox appears cold. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const statefulAgent = { id: 'agent_stateful', statefulCodeSessions: true }; | ||
| const plainAgent = { id: 'agent_plain', statefulCodeSessions: false }; | ||
|
|
||
| function agents(...list: Array<Record<string, unknown>>): PrewarmParams['agents'] { | ||
| return list as unknown as PrewarmParams['agents']; | ||
| } |
There was a problem hiding this comment.
Fixed in 22d052b — replaced the Record<string, unknown> + double cast with an explicit TestAgent shape and a single cast to the library type.
| resetSandboxStateForTests(); | ||
| process.env.LIBRECHAT_CODE_BASEURL = 'http://code.test/v1'; | ||
| delete process.env.CODE_SANDBOX_PREWARM; | ||
| delete process.env.CODE_SANDBOX_COLD_AFTER_MS; |
There was a problem hiding this comment.
Fixed in 22d052b — CODEAPI_JWT_ENABLED and CODEAPI_AUTH_PROVIDER are now cleared in beforeEach, so the suite can't become order-dependent on JWT-minting state from other tests.
| * running state. No-op on stateless deployments ({@link shouldSignalSandboxStart} | ||
| * only tracks conversations that fired a prewarm). |
There was a problem hiding this comment.
Fixed in 22d052b — the JSDoc now describes the current semantics precisely: the signal fires only while a fired prewarm remains unresolved; both real-exec completions and prewarm completions clear it. (The underlying logic was also rewritten onto TTL cache markers in 88aa531, which made the original wording moot.)
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ac984cc0b2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ...authHeaders, | ||
| }, | ||
| body: JSON.stringify({ lang: 'bash', code: 'true', runtime_session_hint: conversationId }), | ||
| signal: AbortSignal.timeout(PREWARM_REQUEST_TIMEOUT_MS), |
There was a problem hiding this comment.
When the Code API /exec response has a body, fetch resolves as soon as headers arrive, but this path never reads or cancels that body before calling markSandboxReady. Under cold starts or any streaming/slow body from /exec, the conversation can be marked warm while the prewarm request is still in flight, suppressing the new on_sandbox_starting signal for a tool call that races the boot; it also leaves the response body/socket for undici to clean up later during prewarm bursts. Please drain or cancel the response body before marking the sandbox ready.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 26f5665 — the response body is now drained (await response.arrayBuffer()) before the ok-check and markSandboxReady, so the warm marker waits for the exec to fully complete and the socket is released promptly.
| ({ set }) => | ||
| (toolCallId: string): void => { | ||
| knownSandboxAtomKeys.current.add(toolCallId); | ||
| set(sandboxStartingByToolCallId(toolCallId), true); |
There was a problem hiding this comment.
When a stream is cancelled, errors, or the user switches conversations before ON_RUN_STEP_COMPLETED, this setter has already created a sandboxStartingByToolCallId atom and recorded its key, but clearStepMaps and the conversation-switch cleanup never reset knownSandboxAtomKeys. That leaves one atom alive per interrupted code call and can show a stale “Starting sandbox environment” if a later provider reuses the same tool_call_id (for example call_0); please clear these sandbox keys at the same cleanup boundary used for the step maps.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 26f5665 — added resetSandboxAtoms and invoked it in clearStepMaps (the same cleanup boundary as the step maps). Unlike the subagent atoms deliberately preserved there for post-run auditing, these flags are transient status, so an interrupted cold boot can no longer leak a stale label onto a reused tool_call_id.
| event: StepEvents.ON_SANDBOX_STARTING, | ||
| data: { tool_call_id: toolCall.id, runId: metadata?.run_id }, |
There was a problem hiding this comment.
In resumable/Redis streams, this new UI-only event is not represented in aggregated message content, and GenerationJobManager.trackReplayEvent only persists OAuth replay events. If a client reconnects or refreshes while the sandbox is still cold-booting, the sync payload can replay the run step but drops on_sandbox_starting, so the code card falls back to the generic running label for exactly the cold-start gap this event is meant to explain; please add this event to the replay/pending-event path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Acknowledged, but leaving as-is intentionally: on_sandbox_starting is ephemeral status scoped to the live stream. Persisting it into the replay path would need expiry logic to avoid the opposite bug — replaying a stale "starting" label after the boot already finished — and the cost of dropping it on reconnect is exactly the pre-PR behavior (generic running label) for a few seconds in a rare race. Happy to revisit if resumable streams grow a TTL'd transient-event channel.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 26f5665d70
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| await response.arrayBuffer().catch(() => undefined); | ||
| if (!response.ok) { | ||
| throw new Error(`prewarm exec returned ${response.status}`); | ||
| } | ||
| markSandboxReady(conversationId); |
There was a problem hiding this comment.
When the Code API returns 2xx headers but the body stream is aborted or times out, arrayBuffer() rejects; this catch swallows it and the following markSandboxReady records the VM as warm even though the exec did not finish. That suppresses both retry prewarms and the cold-boot label for the first real code call until the warm window expires.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 7300e20 — a 2xx response's body must now drain successfully before markSandboxReady; a drain failure propagates as a prewarm failure (retry + cold signal preserved). The non-2xx path still best-effort drains without masking the status error. Added a regression test for the aborted-body case.
| if (tc.runtimeSessionHint != null && tc.runtimeSessionHint !== '') { | ||
| markSandboxReady(tc.runtimeSessionHint); |
There was a problem hiding this comment.
This ready marker only runs after the generic tool.invoke path, but the branch just above returns early for host-handled read_file, create_file, and edit_file; those handlers also pass tc.runtimeSessionHint into sandbox read/write calls. When one of those is the first successful sandbox access after a failed or in-flight prewarm, the state remains cold, so the next bash/execute_code step can still emit on_sandbox_starting and retry prewarm even though the sandbox was already warmed.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 7300e20 — the host-handled early-return path (skill/read_file/create_file/edit_file) now refreshes the warm window on success when tc.runtimeSessionHint is present, mirroring the generic invoke path's marker.
| await emitEvent(res, streamId, { event, data }); | ||
| await maybeEmitSandboxStarting(res, streamId, data, metadata); |
There was a problem hiding this comment.
This only adds the cold-boot event on top-level ON_RUN_STEP events. Code calls made inside a subagent arrive through the ON_SUBAGENT_UPDATE handler as phase: 'run_step' envelopes, so their execute_code/bash_tool cards never receive on_sandbox_starting and continue to show the generic running label during the first cold boot.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Deferring this one: subagent code steps render through the SubagentCall ticker/dialog (folded content parts), not the atom-driven ExecuteCode/BashCall cards, so emitting the event for child steps has no consumer without a separate ticker integration. The cold-boot cost is also typically already paid by the parent conversation's prewarm before a subagent spawns. Happy to do the ticker integration as a follow-up if desired.
| await emitEvent(res, streamId, { | ||
| event: StepEvents.ON_SANDBOX_STARTING, | ||
| data: { tool_call_id: toolCall.id, runId: metadata?.run_id }, |
There was a problem hiding this comment.
This emits on_sandbox_starting only as a live stream event. In the resumable path, replay state is limited to persisted content plus explicit replay events, and this new event is neither content nor added to the replay set, so a tab reload/reconnect after the cold-boot event but before tool completion reconstructs the in-flight tool card without the sandbox-starting label.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Same rationale as the round-1 thread on this line: the event is intentionally ephemeral live-stream status. Persisting it for replay needs TTL/invalidation to avoid replaying a stale 'starting' label after the boot completed, and the reconnect-during-boot gap degrades to the pre-PR generic label for a few seconds. Deferring until resumable streams have a transient-event channel.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7300e20636
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const now = Date.now(); | ||
| const state = sandboxStateByConversation.get(conversationId); | ||
| const withinWarmWindow = state?.readyAt != null && now - state.readyAt <= coldAfterMs(); | ||
| const prewarmInFlight = state != null && now - state.firedAt <= PREWARM_INFLIGHT_COOLDOWN_MS; |
There was a problem hiding this comment.
When CODE_SANDBOX_COLD_AFTER_MS is configured below the hard-coded 120s cooldown, a sandbox can already be considered cold while prewarmInFlight is still true because markSandboxReady() refreshes firedAt on every successful execution. In that setup (for example a 30s sandbox lifetime), the next turn between 30s and 120s skips prewarming even though shouldSignalSandboxStart() now reports cold, so the first code call pays the cold boot despite the override meant to match shorter session lifetimes.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 4d201cb — markSandboxReady no longer refreshes firedAt, so the in-flight cooldown now only tracks an actual fired prewarm that no completion has caught up with (readyAt < firedAt). A cold-after window shorter than the cooldown can no longer suppress a needed prewarm; covered by a new test with CODE_SANDBOX_COLD_AFTER_MS=30000.
| handlerResult.status === 'success' && | ||
| tc.runtimeSessionHint != null && | ||
| tc.runtimeSessionHint !== '' | ||
| ) { | ||
| markSandboxReady(tc.runtimeSessionHint); |
There was a problem hiding this comment.
When a stateful run attaches runtimeSessionHint to create_file/edit_file because they are code-session participants, successful skills/... file writes take this host-handled branch without touching the Code API sandbox, but this condition still records the sandbox as warm. If the prewarm failed or is still booting and the first tool is a skill-file authoring call, later execute_code/bash_tool calls can skip both prewarm and the cold-boot signal for the full warm window even though no sandbox request has completed.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 4d201cb — the host-handled marker is now gated on isSandboxFileAuthoringCall (and success + hint), so skill-path writes and skill/read_file calls that resolve without touching the Code API no longer record the sandbox as warm. Under-marking is the safe direction: worst case is a redundant cold-boot label.
| state.firedAt = now; | ||
| return; | ||
| } | ||
| sandboxStateByConversation.set(conversationId, { firedAt: now, readyAt: now }); |
There was a problem hiding this comment.
With CODE_SANDBOX_PREWARM=false, maybePrewarmCodeSandbox() never creates an entry, but a later successful stateful code execution still calls markSandboxReady() and this branch creates one. After the cold window expires, shouldSignalSandboxStart() starts emitting the new cold-boot label on subsequent turns even though the kill switch was supposed to revert behavior to the previous generic running state.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 4d201cb — shouldSignalSandboxStart now returns false when CODE_SANDBOX_PREWARM=false, so the kill switch reverts the entire feature (prewarm + label) to baseline behavior. Covered by a new test.
|
@codex review |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d201cb5cb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| event: StepEvents.ON_SANDBOX_STARTING, | ||
| data: { tool_call_id: toolCall.id, runId: metadata?.run_id }, |
There was a problem hiding this comment.
In resumable/Redis streams, emitChunk only reconstructs run steps/content on sync; UI-only events have to be carried through the replay path (GenerationJobManager.trackReplayEvent, which currently only keeps OAuth replay events). Emitting on_sandbox_starting only as a normal chunk means that if the tab reloads or reconnects after this event but before on_run_step_completed, the tool card is rebuilt without setting the sandbox-starting atom, so the cold-boot feedback disappears during the long boot. Please add this event to the replay/resume path or derive the flag during sync.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Deferring (third raise of this item — see the two earlier threads on this line for full rationale): the event is intentionally ephemeral live-stream status, and persisting it for replay without TTL/invalidation would replay stale 'starting' labels after boot completion. Reconnect-during-boot degrades to the pre-PR generic label. Will revisit when resumable streams have a transient-event channel.
| async function maybeEmitSandboxStarting(res, streamId, data, metadata) { | ||
| const conversationId = metadata?.thread_id; | ||
| if (!conversationId || !shouldSignalSandboxStart(conversationId)) { | ||
| return; | ||
| } |
There was a problem hiding this comment.
When the first stateful code call happens inside a subagent, the child run-step is forwarded as an ON_SUBAGENT_UPDATE envelope (handled later in this file), not through this top-level ON_RUN_STEP helper, so no on_sandbox_starting event is emitted for that tool call. Because the prewarm gate explicitly walks subagent configs, delegated code use can still hit the cold-boot path while the UI keeps showing only generic subagent/tool progress. Please inspect run_step subagent envelopes and forward equivalent sandbox-starting state for their code tool calls.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Deferring (raised in the previous round as well): subagent code steps render via the SubagentCall ticker (folded content parts), which has no consumer for this atom — forwarding the event without that renderer integration is inert. Tracked as a follow-up together with the file-authoring renderer below.
| if (!toolCall?.id || name == null || !isCodeSessionToolName(name)) { | ||
| continue; |
There was a problem hiding this comment.
For a stateful run whose first sandbox-bound action is create_file or edit_file, this filter never emits the cold-boot event because isCodeSessionToolName(name) is called without the host file-authoring tool set, so it only matches the Code execution tools. Those host tools are explicitly registered as codeSessionToolNames and refreshed with markSandboxReady after success, meaning they can pay the same first sandbox boot but keep showing only the generic creating/editing label. Please include the sandbox-routed file-authoring names in this signal path as well.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid nuance, deferring for the same renderer-consumer reason as the subagent thread: file-authoring calls render via FileAuthoringCall, which doesn't consume the sandbox-starting atom, so emitting for those names alone changes nothing visible. The right follow-up extends both the signal (pass the host file-authoring tool set to isCodeSessionToolName here) and the FileAuthoringCall/SubagentCall renderers together. Scoped v1 to the execute_code/bash_tool cards deliberately.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d201cb5cb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const toolCalls = data?.stepDetails?.tool_calls ?? []; | ||
| for (const toolCall of toolCalls) { | ||
| const name = toolCall?.name ?? toolCall?.function?.name; |
There was a problem hiding this comment.
When the code tool runs inside a subagent, its run-step is delivered through the visible ON_SUBAGENT_UPDATE path (phase: 'run_step', with the nested step under data.data as handled by subagentPhaseToGraphEvent), but only top-level ON_RUN_STEP payloads call this helper. In conversations where the stateful code tool is reachable only from a subagent, prewarm still runs because the gate walks subagents, yet the nested ExecuteCode/BashCall card never receives on_sandbox_starting, so the new cold-boot UX feedback is missing exactly for those subagent tool calls.
Useful? React with 👍 / 👎.
|
|
||
| async function sendPrewarmRequest(req: ServerRequest, conversationId: string): Promise<void> { | ||
| const authHeaders = await getCodeApiAuthHeaders(req); | ||
| const response = await fetch(`${getCodeBaseURL()}/exec`, { |
There was a problem hiding this comment.
In deployments that rely on process.env.proxy to reach LIBRECHAT_CODE_BASEURL, the real Code API calls go through the shared axios client/proxy path (for example readSandboxFile posts to /exec with the configured axios client and code-server agents), but this new prewarm uses global fetch, which does not use that axios proxy configuration. In that environment every prewarm fails while the actual tool execution still succeeds, so the feature silently never warms the sandbox and users continue to pay the first-call cold boot.
Useful? React with 👍 / 👎.
| ({ set }) => | ||
| (toolCallId: string): void => { | ||
| knownSandboxAtomKeys.current.add(toolCallId); | ||
| set(sandboxStartingByToolCallId(toolCallId), true); |
There was a problem hiding this comment.
When a standard SSE stream is closed or the user navigates away while the sandbox-starting flag is set, this atom can survive because it is keyed only by tool_call_id and is reset only by tool completion or clearStepMaps; the non-resumable useSSE cleanup path dispatches cancel without calling clearStepMaps, and the hook unmount cleanup only resets subagent atoms. If a later tool call reuses a common id such as call_0, its card can incorrectly start with the stale “Starting sandbox environment” label even in another conversation.
Useful? React with 👍 / 👎.
|
Follow-up commit: prewarm/warm state moved from a per-process map onto @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 88aa531705
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const [ready, inflight] = await Promise.all([ | ||
| cache.get(readyKey(conversationId)), | ||
| cache.get(inflightKey(conversationId)), | ||
| ]); | ||
| return inflight != null && ready == null; |
There was a problem hiding this comment.
When Redis backs standardCache, a transient Redis error makes either cache.get reject here. maybeEmitSandboxStarting awaits shouldSignalSandboxStart from the ON_RUN_STEP handler, so in Redis-backed deployments this optional cold-boot label can fail the streaming tool step instead of degrading to the normal UI. Please catch/log cache read failures and return false so code execution is not coupled to sandbox-state cache availability.
Useful? React with 👍 / 👎.
| if (tc.runtimeSessionHint != null && tc.runtimeSessionHint !== '') { | ||
| void markSandboxReady(tc.runtimeSessionHint); |
There was a problem hiding this comment.
In Redis-backed deployments, markSandboxReady can reject on set/delete errors, but this hot-path call intentionally discards the promise after a successful sandbox tool invocation. That turns a best-effort warm-window refresh into an unhandled rejection (same pattern in the sandbox-authoring branch above), which can trip Node's unhandled-rejection handling even though the user's tool call already succeeded. Please attach a catch here or make markSandboxReady swallow/log cache errors internally.
Useful? React with 👍 / 👎.
| await emitEvent(res, streamId, { | ||
| event: StepEvents.ON_SANDBOX_STARTING, | ||
| data: { tool_call_id: toolCall.id, runId: metadata?.run_id }, |
There was a problem hiding this comment.
When this event goes through GenerationJobManager (streamId set), it is not represented in runSteps or aggregated content, and the manager's replay path currently persists only OAuth replay events. In Redis resumable mode, a client that refreshes or reconnects after this emit but before the tool completes will never receive the sandbox-starting atom update, so the new cold-boot UX silently falls back to the generic running label. Please persist/replay this transient event or include equivalent state in the resume payload.
Useful? React with 👍 / 👎.
| } else if (stepEvent.event === StepEvents.ON_SANDBOX_STARTING) { | ||
| setSandboxStarting(stepEvent.data.tool_call_id); |
There was a problem hiding this comment.
Using only tool_call_id as the Recoil key lets independent tool calls collide when providers or parallel agents reuse IDs such as call_0; setting or clearing the flag for one call then affects every rendered code/bash card with the same id. This is especially visible when two parallel agent/tool steps overlap and the first completion clears the label for the second while its sandbox is still starting. Please include a step/message/run scoped component in the key (and in the backend event if needed) instead of keying solely by tool id.
Useful? React with 👍 / 👎.
| function sandboxCache(): Keyv { | ||
| if (!cacheInstance) { | ||
| cacheInstance = standardCache(CacheKeys.SANDBOX_PREWARM); |
There was a problem hiding this comment.
In deployments without Redis, this namespace becomes a memoized in-process Keyv map, but it is created without a namespace TTL and is not registered with the cache sweeper. The per-entry TTLs are therefore only enforced when the same conversation key is read again; one-off ready:/inflight: markers for old stateful conversations can remain in the process map indefinitely. Please add an explicit bounded cleanup/default TTL for the process-local fallback.
Useful? React with 👍 / 👎.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 88aa531705
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| */ | ||
| async function maybeEmitSandboxStarting(res, streamId, data, metadata) { | ||
| const conversationId = metadata?.thread_id; | ||
| if (!conversationId || !(await shouldSignalSandboxStart(conversationId))) { |
There was a problem hiding this comment.
When Redis/Keyv is configured and the sandbox prewarm cache read rejects, this awaited shouldSignalSandboxStart call propagates out of the ON_RUN_STEP handler, so a transient cache issue in a nonessential UX signal can interrupt the tool-call stream after the run step is emitted. Since the sandbox-starting event is cosmetic, wrap this check in a try/catch and skip the signal on cache failure rather than letting the chat run fail.
Useful? React with 👍 / 👎.
| * branch may resolve without touching the Code API, and | ||
| * under-marking only costs a redundant cold-boot label. */ | ||
| if ( | ||
| isSandboxFileAuthoringCall && |
There was a problem hiding this comment.
When read_file falls back to the sandbox for paths like /mnt/data/..., handleSandboxFileFallback does call the Code API with runtime_session_hint, but this shortcut returns before the generic tool.invoke marker and this condition only covers file authoring. If the prewarm failed or is still marked in flight and the first successful sandbox operation is read_file, the in-flight marker remains until its TTL, so later code tools in the same conversation can falsely emit the cold-boot label even though the sandbox just responded.
Useful? React with 👍 / 👎.
| ); | ||
| } | ||
| } else if (stepEvent.event === StepEvents.ON_SANDBOX_STARTING) { | ||
| setSandboxStarting(stepEvent.data.tool_call_id); |
There was a problem hiding this comment.
In parallel or subagent runs where separate model calls can reuse per-run tool IDs such as call_0, this drops the runId that the backend sends and keys Recoil state only by tool_call_id. One on_sandbox_starting event can therefore flip every visible code card with the same id, and the first completion can clear the label for another still-booting tool; include the run/step identity in the atom key and pass that same key to the card.
Useful? React with 👍 / 👎.
| * parallel with generation so the first execute_code/bash call lands | ||
| * on a warm VM. No-op unless a reachable agent resolved | ||
| * `statefulCodeSessions`. */ | ||
| maybePrewarmCodeSandbox({ |
There was a problem hiding this comment.
This prewarm only runs on the fresh chatCompletion path; resumeCompletion rebuilds and resumes the graph without setting an in-flight marker or starting a new prewarm. If a run pauses for approval/ask before a code tool and the user resumes after the sandbox cold window, the approved tool cold-boots with only the generic running label, so the new UX does not cover a normal HITL resume path.
Useful? React with 👍 / 👎.
|
|
||
| async function sendPrewarmRequest(req: ServerRequest, conversationId: string): Promise<void> { | ||
| const authHeaders = await getCodeApiAuthHeaders(req); | ||
| const response = await fetch(`${getCodeBaseURL()}/exec`, { |
There was a problem hiding this comment.
Existing Code API requests in the file/code services use the shared axios client plus codeServerHttpAgent/codeServerHttpsAgent, which preserves deployments that rely on the configured proxy/agent behavior. This native fetch bypasses that path, so in proxied Code API deployments normal execute/read/write calls can work while every prewarm fails and the optimization never takes effect; reuse the same client or configured fetch dispatcher here.
Useful? React with 👍 / 👎.
…ibreChat-AI#14239) * ♨️ feat: Prewarm Stateful Code Sandboxes with Cold-Boot UX Feedback * 🧹 fix: Drain Prewarm Response + Reset Sandbox Atoms on Stream Cleanup * 🚿 fix: Propagate Prewarm Drain Failures + Warm Marker for Host File Tools * 🌡️ fix: Decouple Prewarm In-Flight State from Warm Refreshes + Precise Ready Gates * ☁️ refactor: Redis-Backed Sandbox Prewarm State via standardCache * 🧪 chore: Hermetic Prewarm Spec + Accurate Signal JSDoc (Copilot review)
…ibreChat-AI#14239) * ♨️ feat: Prewarm Stateful Code Sandboxes with Cold-Boot UX Feedback * 🧹 fix: Drain Prewarm Response + Reset Sandbox Atoms on Stream Cleanup * 🚿 fix: Propagate Prewarm Drain Failures + Warm Marker for Host File Tools * 🌡️ fix: Decouple Prewarm In-Flight State from Warm Refreshes + Precise Ready Gates * ☁️ refactor: Redis-Backed Sandbox Prewarm State via standardCache * 🧪 chore: Hermetic Prewarm Spec + Accurate Signal JSDoc (Copilot review)
Summary
I added a fire-and-forget prewarm for stateful code sandboxes plus live UI feedback while a sandbox cold-boots, so the first
execute_code/bash_toolcall in a conversation no longer looks hung. Measured on a real Lambda MicroVM + S3 backend, a fresh session costs ~4s of boot before the tool result streams (far worse on heavy first imports), and today the UI shows only a generic running state with no explanation. Builds on thestateful_code_sessionscapability (#14150).packages/api/src/agents/prewarm.ts:maybePrewarmCodeSandboxposts{ lang: 'bash', code: 'true', runtime_session_hint: conversationId }to the Code API with the samegetCodeApiAuthHeaders(req)the real executors use, so the boot lands on the exact session the tools will hit. Gated onanyAgentHasStatefulSessions(walks subagents), deduped per conversation via TTL markers in the standard cache (CacheKeys.SANDBOX_PREWARM) — Redis-backed and replica-safe when Redis is configured, transparently process-local otherwise.chatCompletioninapi/server/controllers/agents/client.js, so the VM boots in parallel with model generation — the model usually takes longer than the boot to emit a tool call, making the cold start invisible in the common case.on_sandbox_startingSSE event from theON_RUN_STEPhandler incallbacks.jsfor code-execution tool calls when the conversation's sandbox has not confirmed ready (shouldSignalSandboxStart); stateless deployments never track a conversation and never emit, keeping existing behavior byte-identical.handlers.tsviamarkSandboxReady(tc.runtimeSessionHint)after each sandbox-boundtool.invoke, so the signal stays accurate without tool-name inspection.ExecuteCode/BashCallthrough a newsandboxStartingByToolCallIdRecoil atom family (mirroring the subagent progress-ticker pattern): the in-progress label swaps to "Starting sandbox environment" and clears on run-step completion.StepEvents.ON_SANDBOX_STARTING+SandboxStartingEventtolibrechat-data-provider, threadedtoolCallIdthroughPart.tsx, and added thecom_ui_sandbox_startinglocalization key.CODE_SANDBOX_PREWARM=falsekill switch andCODE_SANDBOX_COLD_AFTER_MS(default 2,100,000 ms, matching the backend's idle+suspend window) for deployments with different session lifetimes.CODE_SANDBOX_COLD_AFTER_MSoverrides.Prewarm cost note: each prewarm is one trivial
bash trueexec (plus its checkpoint write) per conversation per cold window — only on deployments that opted intostateful_code_sessions, and only for agents with the builder toggle on.Change Type
Testing
Verified against a live stateful Code API (lambda-microvm backend + S3 checkpoints, affinity mode):
cd packages/api && npx jest prewarm.spec— 12/12 pass;npx jest statefulCodeSessions— 4/4 pass;cd packages/data-provider && npx jest runs— 10/10 pass.npx tsc --noEmitclean inclient/; ESLint clean across all touched files.stateful_code_sessions+ the agent toggle, start a new conversation, ask for code execution immediately — the tool call should show "Starting sandbox environment" only when the model beats the boot; a second message should run warm with the standard label; withCODE_SANDBOX_PREWARM=falsebehavior reverts to baseline.Test Configuration:
feat/sandbox-backend-seamservice,CODEAPI_SANDBOX_BACKEND=lambda-microvm,CODEAPI_RUNTIME_SESSION_MODE=affinity, S3 checkpointsLIBRECHAT_CODE_BASEURLpointed at the stateful Code API;librechat.yamlagents capabilities includestateful_code_sessionsChecklist