Repository navigation
fix(compute): size the gateway for its first turn and test it without swap - #3822
Conversation
… swap Since the OpenClaw pin of #2982 an embedded Gateway is ready at 1.5-1.6 GiB and peaks at 2.6-2.8 GiB on its first turn, when OpenClaw starts most of its worker threads (finding 2700). The runtime image probe tests capped it at --memory 2g, which Docker pairs with as much swap again; CI runners swap, so the tests passed while a swapless host OOM-killed the Gateway. - Image probe tests share one Gateway memory setting, --memory and --memory-swap both 4g, so swap can no longer hide growth. - Gateway sizing in the production example, the profile renderer and the local Kubernetes stack: requests 2560Mi (was 1792Mi), limits 4Gi (was 3Gi). The 3Gi limit left about 10% headroom over a stub turn. - Breaking-change notice for Installations with a lower Gateway limit.
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
Free's clanker here. Image Runtime Startup 2 on a remote test VM at bd4b4fe, with the Gateway at |
clawsweeper
Bot
added
clawsweeper:intake:bug
|
Codex review: blocked before merge. What this changesThis PR raises Gateway memory requests to 2560Mi and limits to 4Gi, and runs runtime probes without swap. Example: An embedded Gateway answers its first chat turn on a host without swap.
Review scores
ProductKind: Bug fix · Worth it: Yes · Fix scope: Complete Merge readiness⛔ Blocked before merge - 1 item remains Keep this PR open: it addresses measured Gateway memory growth through the existing sizing owners, and no definite introduced code defect was found. Priority: P2 Before merge
FindingsNone. Tests
Agent review detailsHow this fits togetherInstallation generation supplies resource budgets to Kubernetes Compute, which applies them to Agent workloads; runtime-image tests exercise the production wrappers against a stand-in model provider. flowchart LR
A[Installation profile or development setup] --> B[Gateway resource configuration]
B --> C[Kubernetes Compute]
C --> D[Agent Gateway Pod]
E[Runtime image probes] --> F[Container with swap disabled]
F --> G[Production Gateway wrapper]
Technical reviewBest possible solution: Land the focused sizing and swap-control change after verifying the documented update of an existing Installation. Do we have a high-confidence way to reproduce the issue? The author supplies a concrete swapless first-turn OOM scenario and comparative cgroup measurements, corroborated by the earlier diagnostic discussion; this review did not execute the runtime. Is this the best way to solve the issue? Updating the existing sizing owners and disabling swap in their runtime probes is proportionate to the measured footprint; this provides headroom rather than repairing OpenClaw's underlying memory growth. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 9d9e8dd27b7b. Provenance checked
TestingProof path: in-process harness. SecurityNone. EvidenceWhat I checked:
LabelsLabel changes:
Label justifications:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment Reviewed October 27, 2026, 2:08 AM ET / 06:08 UTC. |
freeqaz
left a comment
freeqaz
left a comment
There was a problem hiding this comment.
Free's clanker here (an AI agent working on Free's behalf). Independent review of head f4c24f3.
Verdict: no blocking issues. The sizing is consistent, the numbers follow from the measurements, and the stricter test setting catches the regression it targets.
Sizing consistency. I searched the repository for 1792Mi, 3Gi, 1.75 GiB, 2.2 GiB, 3 GiB, 2560Mi, 3328Mi, 3.25 GiB and 2.5 GiB. Every Gateway statement now reads 2560Mi/4Gi: the production example (deploy/examples/production/installation.yaml:75,78), the renderer (scripts/render-installation-profile.mjs:1858), occdev (internal/occdev/kubernetes.go:381), the reference, the guides, the Compose flow, and the conformance, renderer and Go tests. No old value is left. The Harness (768Mi/6Gi), container defaults and Helm values are unchanged. The Helm chart has no Gateway default. The dedicated Codex total adds up: 2560 + 768 = 3328 Mi = 3.25 GiB.
Measurements. The request is 2.5 GiB against 2.29-2.38 GiB settled between turns, so it still covers what an Agent holds. The limit is 4 GiB against a 2.53-2.78 GiB first-turn peak (no swap), about 1.2 GiB of headroom. 3Gi would leave about 10%.
Breaking-change notice. It follows the file's format (What breaks / Who is affected / How to tell / Steps, marked untested), is dated 2026-10-27 and is placed newest first. Suite Audit and Static Checks are green on this head, so the word limits hold.
Test helper. Both probe launchers spread the shared gatewayMemoryArguments, which set --memory 4g --memory-swap 4g: runEmbeddedGatewayProbe (tests/helpers/runtime-image-model-probe.mjs:237) and runStartupProbeScenario (tests/helpers/runtime-image-startup-probe.mjs:402). No caller still passes memory. productionGatewayMemoryLimit and startupProbeMemoryLimit are gone and nothing imports them. The other runtime-image tests start containers without a memory limit, as before.
Mutant. I set the shared limit back to 2g, so --memory 2g --memory-swap 2g, and ran Image Runtime Startup 2 on a remote test VM that has swap like CI. It failed: all three embedded-Gateway cases in runtime-image-gateway-unclean-restart.test.mjs hit fetch failed / SocketError: other side closed after about 18-20 s. That is finding 2692's OOM signature. The other files passed. The no-swap flag is what makes the test catch this.
CI cost. Image Runtime Startup 1/2/3 took 542/452/420 s on this head. The last six main runs took 343-541 / 463-514 / 405-609 s, so there is no slowdown. Image Runtime Startup 2 runs up to 4 files with 2 tests each. Gateways that used to swap past 2 GiB now stay resident at up to about 2.8 GiB each, which is roughly 22 GiB in the worst case. That is fine on the 32-vCPU runner, and the lane is green.
Nits (non-blocking):
docs/guides/deploy/installation-profiles.md:47: the row covers "embedded or dedicated", but only the embedded Gateway was re-measured. The dedicated Gateway's old measurements (1.2-1.6 GiB) were dropped. Either say the dedicated Gateway was not re-measured, or measure it, because a dedicated Codex Agent now reserves 768 Mi more on that basis.tests/integration/runtime-image-model-probe.test.mjs: the CPU-contention case runs eightnodehogs inside the Gateway container. That uses about 0.4 GiB of the 4 GiB, so the margin over the peak is smaller there. It passed on the head (Image Runtime Startup 3), so this is something to watch rather than change.- Pre-existing, for a follow-up:
tests/helpers/harness-topology-k3d-real.mjs:915still limits the real dedicated Gateway to2Gi, while the profiles page says a dedicated Gateway serving native admin chat was OOM-killed at2Gi. The k3d lanes are green on this head.
Problem
Finding 2700. Since the OpenClaw pin update in #2982, an embedded Gateway needs much more memory on its first chat turn:
memory.peak), first turnruntime-image-gateway-unclean-restart, whose turn client runs in the container)--memory 2g --memory-swap 2g--memory 3g --memory-swap 3gTurns 2-5 add nothing, and with no turn the Gateway stays at about 1.7 GiB, so this is a one-time cost of the first turn, not a per-turn leak. The growth is in the V8 heaps of OpenClaw's
worker_threadsisolates (about 25 on the new pin, about 18 on the old one); the main thread's heap stays at 0.3-0.5 GiB.The runtime image probe tests capped the Gateway at
--memory 2g. Docker pairs that with as much again in swap, and CI runners swap, so the tests passed while the Gateway swapped 0.25-1 GiB. On a host without swap the same tests OOM-killed the Gateway ("fetch failed: other side closed"). Kubernetes nodes usually run without swap, and the production example's3GiGateway limit leaves about 10% headroom over a stub turn.Change
tests/helpers/runtime-image-startup.mjs:--memory 4g --memory-swap 4g, the production limit with no swap, so swap can no longer hide growth. The unused per-callmemoryoption is gone.requests.memory1792Mi -> 2560Mi andlimits.memory3Gi -> 4Gi in the production example, the profile renderer, the local Kubernetes stack (internal/occdev), their tests, and the sizing docs. Requests still cover use between turns (now 2.3-2.4 GiB); the limit leaves about 30% headroom over the first-turn peak.Verification
process.memoryUsage()of every isolate every 500 ms (scratch probe, not committed).runtime-image-gateway-unclean-restartand the rest of Image Runtime Startup 2 on a remote test VM: see the comment below.profile-renderer.test.mjs67/67; thekubernetes-computeconformance case for the production example;go test ./internal/occdev -run TestDevelopmentInstallationSizesAgentsFromMeasuredUse.Risks