[freeqaz]

Problem

On a remote test VM, Image Runtime Startup 2 fails all three runtime-image-gateway-unclean-restart tests since #2982: the first Gateway's first chat turn ends with "fetch failed: other side closed" because Docker kills the Gateway container at its 2 GiB memory limit (OOMKilled: true). GitHub CI passes on the same commits. We need the same numbers from a CI runner to explain the difference.

Change

Diagnostic only. The first Gateway's turn records the container's memory.current, CPU throttling, memory limit and heap limit every 2 s, then the test fails with a one-line DIAG2692 summary in the job log.

Verification

None needed beyond the job log of Image Runtime Startup 2 on this PR.

Risks

None. Draft; it is closed without merging.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs changes before merge.

What this changes

This PR samples Gateway memory and swap during the first chat turn, then deliberately fails the test with a diagnostic summary.

Example: The Docker recovery case sends “Reply with READY.” to a Gateway limited to 2 GiB.

• Before: The test checks the answer and proceeds to kill and restart the Gateway.
• After: The test samples memory and swap every two seconds, reports container OOM status, and stops with DIAG2692.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	Useful temporary instrumentation, with incomplete CPU accounting and no claim to resolve the underlying OOM.
Proof confidence	🦐 gold shrimp (3/6)	This maintainer-authored, test-only diagnostic changes no production behavior; CI metadata confirms execution, but raw diagnostic logs could not be inspected because their storage host was blocked. The inspected managed-mode screenshot belongs to the linked earlier PR and does not demonstrate memory sampling; its video preparation failed because ffprobe was unavailable.
Patch quality	🐚 platinum hermit (4/6)	1 actionable review finding remain.

Product

Kind: Test only · Worth it: Yes · Fix scope: Complete
User problem: A remote test VM loses the Gateway during its first chat turn while CI reportedly succeeds.
Reason: The collaborator has authorized a narrow diagnostic comparison; the PR deliberately offers no production fix and explicitly prohibits merging.

Merge readiness

⛔ Needs changes before merge - 1 item remains

Keep this PR open for maintainer triage and preserve its explicit do-not-merge hold; it is a useful temporary investigation, with one missing diagnostic measurement.

Priority: P2
Reviewed head: f4b09a650a9385c150dba67d40fe34f92903794f

Before merge

• Capture CPU throttling counters around the first turn (P3) - The diagnostic promises CPU-throttling measurements, but cpu.max contains only quota and period, and the sampler reads only memory and swap. A throttled and an unthrottled turn can therefore produce indistinguishable CPU evidence. Read cpu.stat before and after the turn, or include its throttling counters in the samples, so the CI-versus-VM comparison contains the advertised measurement.

Findings

• [P3] Capture CPU throttling counters around the first turn — tests/integration/runtime-image-gateway-unclean-restart.test.mjs:224-227

Agent review details

How this fits together

The runtime-image integration suite starts the production Gateway wrapper against a stand-in model provider; this probe reads container accounting around its first HTTP chat request.

flowchart LR
  A[Image Runtime Startup 2] --> B[Startup scenario helper]
  B --> C[Gateway runtime container]
  C --> D[Stand-in model provider]
  E[Memory and swap sampler] --> C
  C --> F[DIAG2692 failure summary]
  E --> F

Loading

Technical review

Best possible solution:

Use the temporary probe to compare resource accounting, then let the maintainer retire it and carry any confirmed runtime repair in a separate PR.

Do we have a high-confidence way to reproduce the issue?

The remote-VM OOM is reported with a concrete first-turn trigger and OOMKilled result, but this review did not reproduce it or verify raw CI resource measurements.

Is this the best way to solve the issue?

Extending the existing container scenario is a proportionate diagnostic approach, provided it captures the claimed throttling data and remains outside main.

Full review comments:

• [P3] Capture CPU throttling counters around the first turn — tests/integration/runtime-image-gateway-unclean-restart.test.mjs:224-227
  The diagnostic promises CPU-throttling measurements, but cpu.max contains only quota and period, and the sampler reads only memory and swap. A throttled and an unthrottled turn can therefore produce indistinguishable CPU evidence. Read cpu.stat before and after the turn, or include its throttling counters in the samples, so the CI-versus-VM comparison contains the advertised measurement.
  Confidence: 0.99

Overall correctness: patch is correct
Overall confidence: 0.93

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 34d3848633a1.

Provenance checked

• First-turn assertions and Docker recovery handoff changes intended behavior with a stated reason (feat(runtime): apply Gateway tool policy to dedicated OCE deployments #2633: The existing workflow verifies first-turn readiness and registers the predecessor for verified recovery; the earlier test extraction preserved the readiness regression scenario.)

Testing

Proof path: none.

Security

None.

Evidence

What I checked:

• Explicit diagnostic scope and maintainer direction: The collaborator-authored PR describes a remote-VM first-turn OOM investigation, requests comparison with CI, and explicitly says to close without merging; repository review policy requires leaving disposition to maintainers. (f4b09a650a93)
• Verified introduced change: The pinned introduction changes only the existing restart test: it replaces first-turn handoff registration with resource sampling and an unconditional diagnostic exception. (tests/integration/runtime-image-gateway-unclean-restart.test.mjs:208, f4b09a650a93)
• CPU throttling is absent from the diagnostic: The facts command reads cpu.max, while periodic samples read only memory.current and memory.swap.current; no cpu.stat throttling counters reach DIAG2692. (tests/integration/runtime-image-gateway-unclean-restart.test.mjs:217, f4b09a650a93)
• Current-main boundary: Main retains the first-turn assertions and recovery workflow; the shared scenario still applies a fixed 2g test-container limit. (tests/helpers/runtime-image-startup-probe.mjs:50, 34d3848633a1)
• CI attribution and inspection limit: Run 40451693341 targets the reviewed head. Runtime failures are expected from the unconditional exception; static-job metadata separately identifies Check formatting as failed. Raw job-log retrieval was blocked at productionresultssa16.blob.core.windows.net, so its diagnostic values and formatting attribution remain unverified. (f4b09a650a93)
• Feature history and existing memory work: History traces the readiness regression test through fix(runtime): hold gateway readiness until agent database inspection ends #3231, its extraction through perf(test): split the unclean-stop gateway case and run image startup cases concurrently #3347, and recovery extensions through feat(runtime): apply Gateway tool policy to dedicated OCE deployments #2633. fix(compute): give gateways a 3Gi memory limit #866 raised configured deployment limits to 3Gi for an earlier OOM report; it does not settle this fixed-2g test investigation. Content-based history searches encountered unavailable promisor blobs. (tests/integration/runtime-image-gateway-unclean-restart.test.mjs:191, 34d3848633a1)

Labels

Label changes:

• add P2: This is a bounded runtime-test investigation without evidence of an emergency in deployed installations.
• add merge-risk: 🚨 automation: Merging the unconditional diagnostic exception would permanently fail three selected runtime cases; the author explicitly accepts this only on a branch that must not merge.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Label justifications:

• P2: This is a bounded runtime-test investigation without evidence of an emergency in deployed installations.
• merge-risk: 🚨 automation: Merging the unconditional diagnostic exception would permanently fail three selected runtime cases; the author explicitly accepts this only on a branch that must not merge.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 27, 2026, 1:00 AM ET / 05:00 UTC.

[freeqaz]

Diagnostic probe done: CI runners have 11 GiB swap, and this test's Gateway container sits at its 2 GiB limit and swaps 0.25-1 GiB during its first turn. A test VM without swap OOM-kills it instead. Closing without merging.