Skip to content

fix(console): name a sandbox driver outage and confirm a full pod quota - #2686

Merged
freeqaz merged 8 commits into
mainfrom
lows-1017e
Oct 17, 2026
Merged

freeqaz merged 8 commits into
mainfrom
lows-1017e

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Three low findings from an audit of #2674 and #2675:

  • 1623. NAMESPACE_POD_QUOTA_EXCEEDED came from the Deployment's ReplicaFailure condition. That condition keeps the first refusal since the last successful create, and the next create can wait up to 1000 s of ReplicaSet backoff. So "The Namespace's Pod quota is full" could stay up for about 16 minutes after a user stopped an Agent. It also kept naming the Pod quota after a CPU or memory quota became the real blocker.
  • 1624. During a Sandbox Driver outage, GET /installation reports openclaw.executionModes: []. Create Agent then showed the permanent refusal ("its selected Sandbox Driver refuses Embedded execution ... ask an administrator to select a Sandbox Driver ..."). Admission answers a retryable 503 in the same state.
  • 1625. Docs and hints still promised both OpenClaw modes. create-and-deploy.md said "OpenClaw starts Embedded". Before discovery, the console hints said "OpenClaw supports Dedicated or Embedded execution". The capability description said the list is what Agent save accepts, but save accepts Dedicated without a qualifying Sandbox Driver; only deploy refuses it.

Change

  • Kubernetes Compute (1623). The quota check now reads the ResourceQuota named in the refusal and reports NAMESPACE_POD_QUOTA_EXCEEDED only while that quota's status still shows used >= hard for pods or count/pods. A freed Pod, a raised limit, a CPU-only blocker, a deleted quota or an unreadable quota reports nothing, so the plain wait or WORKLOAD_UNSCHEDULABLE shows instead. Each quota is read at most once per pass, keyed by plane, Namespace and name. No new permissions: the worker's tenant role already has get on resourcequotas, in both charts, because it reconciles openclaw-quota.
  • Capability (1624). capabilities.openclaw gains an optional sandboxDriverUnavailable: true, present only when the selected Sandbox Driver is unavailable. It is additive and follows the nativeWorkers "present only when" pattern. Agent save answers 503 for every harness in that state, so Create Agent now says, for OpenClaw and Codex alike: "Agents cannot be created right now: this installation's Sandbox Driver is unavailable. Reload this page to check again, or ask an administrator if it persists." It keeps Create disabled and no longer tells anyone to reconfigure. The harness and mode hints say the same.
  • Copy and docs (1625). The pre-discovery hints no longer promise a mode. They now say: "OpenClaw uses Dedicated execution with a Sandbox Driver that provisions it, and Embedded execution without a Sandbox Driver" and "With OpenAI its execution mode depends on the installation's Sandbox Driver". The schema description says the list is the modes an Agent can deploy in, and that save refuses only Embedded with a selected Sandbox Driver. create-and-deploy.md, harness-execution.md and deployment.md are updated. OpenAPI and api.md are regenerated.
  • Test gaps from the audit.
    • New test: the quota check reads only a revision's own workload Deployments. That is both Deployments for dedicated Codex, and only the Gateway for Embedded OpenClaw or a Sandbox-provisioned Harness.
    • The Secret Driver outage provisioning test now cancels its queued retry in fixture teardown, not inline, so the cancel also runs when an earlier assert fails. The teardown still closes the app and pool if that cancel fails.

Verification

  • tests/conformance/kubernetes-compute.test.mjs, configuration-occ.test.mjs, occ-api-security.test.mjs: 439/439 pass.
  • tests/browser/console-create-agent-provisioning-400.test.mjs and console-agents.test.mjs: 57/57 pass.
  • tests/integration/postgres-agent-provisioning.test.mjs, run against a prepared PostgreSQL (before and after the review changes): 54/54 pass. A probe that fails the outage test right after its first attempt leaves the work failed_permanent / PROVISIONING_CANCELLED, so teardown cancels it. With the old inline cancel, that work stayed queued.
  • tsc --build, eslint on the touched files, prettier, openapi:check. Word counts (real counter): harness-execution 2465, create-and-deploy 2483, deployment 2480.
  • Mutants, each killed by the new assertions:
    • quota check: always-true quota check; > instead of >=; unparsable quantity counted as full; only pods (no count/pods); fixed quota name instead of the refused one; read error treated as full; missing hard limit treated as 0; no read dedupe; dedupe key without the Namespace. One mutant survives: dropping the plane from the dedupe key. The fixture serves both planes from one store, so it cannot tell them apart.
    • Deployments read: dropped Harness-Deployment filter; dropped Embedded role filter.
    • server flag: flag never set; flag always set.
    • console: flag ignored; outage refusal limited to OpenClaw; outage refusal dropped (Preset copy shown instead); outage harness hint dropped; outage mode hint dropped; old pre-discovery mode hint; old pre-discovery harness hint.
  • Touched browser and quota tests: 20/20 at 2 CPUs (taskset -c 0,1), on the code before and after the review changes.
  • Internal review: approve with nits. Applied: the outage copy covers Codex, quota reads are deduplicated, the teardown is guarded, and the deployment doc wording is restored. Not applied: listing resourcequotas get in the custom-RBAC docs (the chart roles already grant it, and a missing grant only drops the sentence), and checking quotas when the first refusal was for CPU (a known gap, not a regression).
  • Storybook: new story Pages/Create Agent / OpenClaw during a Sandbox Driver outage (createOpenclawSandboxDriverOutage). Media below.

Story createOpenclawSandboxDriverOutage (media commit a419d1d09019, removed before squash):

Create Agent during a Sandbox Driver outage: harness hint and alert
Create Agent during a Sandbox Driver outage: execution mode hint
walkthrough

Risks

  • Low. The quota sentence can now drop out when a custom tenant role lacks get on resourcequotas. In that case the wait falls back to the plain or unschedulable wording; it is never wrong.
  • The new capability field is optional and additive. Older consoles ignore it, and newer consoles treat a server without it as before.

…ver outage

The Pod quota pending reason now requires the refused ResourceQuota's status to
still show every Pod used (finding 1623). GET /installation marks an empty
OpenClaw mode list caused by a Sandbox Driver outage with
openclaw.sandboxDriverUnavailable, and Create Agent says to retry instead of
reconfiguring (finding 1624). Pre-discovery hints, the capability description
and docs no longer promise both OpenClaw modes (finding 1625). Tests cover the
Embedded and Sandbox-provisioned Harness skips of the quota check and move the
provisioning outage test's retry cancellation into teardown.
@freeqaz
freeqaz requested a review from a team as a code owner October 17, 2026 03:48
@clawsweeper

clawsweeper Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@freeqaz-openai

freeqaz-openai commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Update on fc61ab314: the new refusal correctly distinguishes a temporary driver outage. The two surrounding labels still say an Embedded draft can be created. I reproduced #2684 in sandboxed Chromium on the exact published Create module, using this PR’s operator-managed Preset browser case and its three routed Installation responses. When no mode is available, when this Preset needs Embedded but only Dedicated is available, and during the driver outage, the refusal and disabled Create were correct; the summary and footer still read “Runtime details · Embedded” and “Embedded Agents are saved as drafts.”

The small patch below lets your existing, specific refusal own the explanation. The summary says only that the selected mode is unavailable; the contradictory draft/deploy promise clears. No second copy of the driver, Preset, or permission decision. The new assertions failed on the published UI and passed with this patch. A separate positive control returned Embedded as genuinely available and confirmed the original summary and draft explanation remain. This is a browser copy result with local fixture and controlled Installation shapes, not a live provider, write-path, or whole-PR claim.

Separately, the #2685 polling change combined cleanly with your earlier b8305847 in the three overlapping source files. No CI or review rerun is requested by this note.

Applyable UI + regression assertion diff against fc61ab3
--- a/apps/controller/src/console/agents/create.mjs
+++ b/apps/controller/src/console/agents/create.mjs
@@ -1707,7 +1707,9 @@
       openclawModeUnavailable.textContent = modeRefusal;
     }
     openclawModeUnavailable.hidden = !modeRefusal;
-    runtimeModeSummary.textContent = ` · ${mode.value === "dedicated" ? "Dedicated" : "Embedded"}`;
+    runtimeModeSummary.textContent = modeRefusal
+      ? " · Execution mode unavailable"
+      : ` · ${mode.value === "dedicated" ? "Dedicated" : "Embedded"}`;
     const modeHint = form.querySelector("#execution-mode-hint");
     if (modeHint) {
       modeHint.textContent = sandboxDriverOutage
@@ -1721,11 +1723,13 @@
               : "Codex uses Dedicated execution. OpenClaw uses Dedicated execution with a Sandbox Driver that provisions it, and Embedded execution without a Sandbox Driver. Slack requires Dedicated execution.";
     }
     if (capabilityDiscoveryDone) {
-      const nextStatus = shouldProvision()
-        ? "Dedicated Agents are provisioned and deployed when created."
-        : mode.value === "embedded"
-          ? "Embedded Agents are saved as drafts. Deploy them from the Agent page after creation."
-          : "This installation creates draft Agents for later deployment.";
+      const nextStatus = modeRefusal
+        ? ""
+        : shouldProvision()
+          ? "Dedicated Agents are provisioned and deployed when created."
+          : mode.value === "embedded"
+            ? "Embedded Agents are saved as drafts. Deploy them from the Agent page after creation."
+            : "This installation creates draft Agents for later deployment.";
       if (capabilityStatus.textContent !== nextStatus) {
         capabilityStatus.textContent = nextStatus;
       }
--- a/tests/browser/console-create-agent-provisioning-400.test.mjs
+++ b/tests/browser/console-create-agent-provisioning-400.test.mjs
@@ -1102,6 +1102,11 @@
       .filter({ hasText: /operator-managed credentials|^Agents cannot be created / });
     await alert.waitFor();
     assert.equal(await alert.textContent(), text, JSON.stringify(openclaw));
+    assert.match(
+      await page.locator(".launch-runtime summary").innerText(),
+      /Execution mode unavailable/,
+    );
+    assert.equal(await page.locator("#create-agent-form > [role=status]").innerText(), "");
     assert.equal(await page.getByLabel("Execution mode").inputValue(), "embedded");
     assert.equal(
       await page.getByRole("button", { name: "Create Agent", exact: true }).isDisabled(),

Review follow-ups: Agent save answers 503 for Codex too while the selected
Sandbox Driver is unavailable, so the outage alert and hints cover both
harnesses and Create Agent stays disabled. Quota reads are deduplicated per
pass by plane, Namespace and name; the provisioning test teardown closes the
app and pool even if a cancellation assert fails; the deployment doc keeps
"gateway connection".
@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 17, 2026
@freeqaz
freeqaz temporarily deployed to integration-qa-pr October 17, 2026 04:05 — with GitHub Actions Inactive
@freeqaz
freeqaz temporarily deployed to integration-qa-pr October 17, 2026 04:05 — with GitHub Actions Inactive
@clawsweeper

clawsweeper Bot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(console): name a sandbox driver outage and confirm a full pod quota This is item 1/1 in the current shard. Shard 0/1.

This temporary status tracks the active review worker. The completed review will appear in the durable ClawSweeper review comment.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@freeqaz
freeqaz merged commit 761bf51 into main Oct 17, 2026
45 of 46 checks passed
@freeqaz
freeqaz deleted the lows-1017e branch October 17, 2026 04:18

This branch had an error being deployed

1 failed deployment
integration-qa-pr — fc61ab31 Deployed Oct 17, 2026 by freeqaz via QA advisory (OpenShell) #1707
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants