Skip to content

fix(compute): name a full Namespace Pod quota instead of a bare provisioning wait - #2674

Merged
freeqaz merged 6 commits into
mainfrom
fix-1617
Oct 17, 2026
Merged

freeqaz merged 6 commits into
mainfrom
fix-1617

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

With the tenant Namespace's ResourceQuota at pods 10/10, Create Agent sat on "Provisioning Agent…" for 60+ s and the Agent page said only REVISION_INCOMPLETE / "Waiting for the runtime to become ready." The revision's ReplicaSets reported FailedCreate: exceeded quota: openclaw-quota every few seconds, but nothing surfaced it (dogfood finding D629).

Change

  • Kubernetes Compute. When a revision is unready, the driver reads the ReplicaFailure condition on the revision's Deployments: the Harness Deployment, and the gateway Deployment. The Deployment controller copies this condition from the ReplicaSet. If the condition is True with reason FailedCreate, and the quota refusal's requested: list names pods or count/pods, the observation reports pendingReason: NAMESPACE_POD_QUOTA_EXCEEDED. A full quota takes precedence over WORKLOAD_UNSCHEDULABLE. A held runtime failure still wins. Each Deployment is read on its own: an unreadable one reports nothing and does not hide the other's refusal. A Harness that a SandboxDriver provisions has no Harness Deployment, so only the gateway is read. The revision stays pending, since the quota can free up: no new timers, caps or outcomes.
  • No new cluster permissions. The driver reads Deployments, which it already manages. It never reads ReplicaSets, which its RBAC does not grant.
  • Worker and API. NAMESPACE_POD_QUOTA_EXCEEDED joins REVISION_PENDING_CODES (the existing pending path, like REVISION_UNSCHEDULABLE). Deployment progress carries fixed copy: "The Namespace's Pod quota is full. Stop or delete an Agent, or ask an administrator to raise the quota." It never shows the quota's name or numbers. The Agent page already renders progress.lastAttempt, so it now shows that sentence instead of the bare wait.
  • Create Agent. Once a running provisioning job has handed off its first revision, each poll also reads that deployment's status. When the deployment's last attempt is NAMESPACE_POD_QUOTA_EXCEEDED, the status line shows its sentence. That read uses the normal deployment route, so it needs read access to the Agent. A refused read, or any other wait, keeps the bare "Provisioning Agent…" line.
  • Docs: docs/reference/agents/deployment.md lists the new code. Storybook: Pages/Create Agent → Initial deployment waits on a full Pod quota (createDeploymentPodQuotaFull).

Verification

  • tests/conformance/kubernetes-compute.test.mjs adds "a pending dedicated deploy names a full Namespace Pod quota". It uses the fake Kubernetes fixture with ReplicaFailure/FailedCreate quota conditions on the Harness and gateway Deployments, covering pods, mixed pods,requests.cpu, and count/pods. The negatives are a CPU-only refusal (with pods only in used:), status False, another reason, another condition type, a non-quota refusal, and a missing message. It also pins precedence over an unschedulable Pod, a held runtime failure leading instead, and the per-Deployment read. Full file: 367/367 pass.

  • tests/integration/postgres-worker-agent-revision.test.mjs: the pending-reason case adds a third Agent whose Compute reports the new reason. The case asserts the recorded code and copy. Full file: 50/50 against a disposable PostgreSQL.

  • tests/browser/console-agents-2.test.mjs adds "Dedicated Agent creation names a full Namespace Pod quota while it waits". It records every status-line write. A readable deployment writes the sentence, then the bare line once the wait changes. A refused deployment read (404) writes only the bare lines. Deployment reads happen only after handoff. Full file: 19/19.

  • Mutation check: 21 hand mutants, 19 killed. They covered the refusal parse, pods/count/pods, status, reason and type checks, Harness-only and gateway-only reads, the per-Deployment read, precedence over unschedulable and under a runtime failure, worker mapping, progress copy, the console code filter, the revision guard and error handling. Two mutants survive, and neither changes an outcome. One drops the console's agentId guard: a job never carries revisionId without agentId. The other drops the SandboxDriver filter, which only saves a GET that would return 404.

  • Internal review: nothing blocking; low findings and nits applied (per-Deployment read, SandboxDriver skip, comment on the first refusal, runtime-failure precedence test, docs wording).

  • New tests 20/20 each at 2 CPUs (taskset -c 0,1). tsc --build clean. eslint and prettier clean on touched files.

  • No k3d test: k3d cannot run on this box (rootless podman without the cpuset delegation). There is no cheap existing quota fixture in the k3d suites. The fake-cluster case reproduces the exact condition the Deployment controller writes.

  • Media: console fixture at 1440×900 and 390×844 (simulated UI proof, not live backend proof). Storybook story: Pages/Create Agent → Initial deployment waits on a full Pod quota.

    Create Agent, desktop: the provisioning line names the quota.
    Create Agent names the full Pod quota (1440)

    Create Agent, mobile:
    Create Agent names the full Pod quota (390)

    Agent page deployment record:
    Agent page shows NAMESPACE_POD_QUOTA_EXCEEDED

    Before the deployment hands off, the line is still the bare wait: screenshot. Video: Create Agent walkthrough.

Risks

…sioning wait

Kubernetes Compute reads the ReplicaFailure condition that the Deployment
controller copies from a revision's ReplicaSet. A FailedCreate whose quota
refusal lists pods (or count/pods) reports pendingReason
NAMESPACE_POD_QUOTA_EXCEEDED, which the worker records under the same code
with fixed copy. Create Agent shows that copy while its first deployment
waits, when the reader can read the deployment.
…Storybook state

The browser case now includes a job that created its Agent but has not
handed off a revision, so a missing revision guard is caught. The
deployment reference lists NAMESPACE_POD_QUOTA_EXCEEDED, and Create Agent
gains an 'Initial deployment waits on a full Pod quota' story.
@freeqaz
freeqaz requested a review from a team as a code owner October 17, 2026 02:43
@clawsweeper

clawsweeper Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 17, 2026
…der Harness

Review follow-ups for the Pod quota reason:
- an unreadable Deployment no longer hides the other one's refusal;
- a Harness that a SandboxDriver provisions has no Harness Deployment to read;
- the comment notes that ReplicaFailure keeps the first refusal;
- tests pin that a held runtime failure leads, and the per-Deployment read;
- the deployment reference keeps 'with the deployment's age'.
@freeqaz
freeqaz temporarily deployed to integration-qa-pr October 17, 2026 02:58 — with GitHub Actions Inactive
@freeqaz
freeqaz temporarily deployed to integration-qa-pr October 17, 2026 02:58 — with GitHub Actions Inactive
@freeqaz-openai

Copy link
Copy Markdown
Collaborator

Free's clanker here. One small diagnostic gap on 28ff7a155: the native first-Agent timeout would still print lastAttemptCode: "unknown" for the new quota refusal. I ran this head’s actual deploymentProgressForWork and passed its result to this head’s actual failNativeProvisioningDeadline; this is independent of whether #2677 lands.

Please add only NAMESPACE_POD_QUOTA_EXCEEDED to deploymentProgressCodes in tests/helpers/local-first-agent-native.mjs, with a case that expects the fixed code and still fails the deadline. I verified that exact addition preserves the quota code, keeps REVISION_UNSCHEDULABLE unchanged, and still reduces arbitrary peer text to unknown without printing it. No extra request or raw quota message is needed.

This doesn’t establish that quota caused the earlier hosted timeout, and it isn’t a request for a broad shared enum or another cluster run. It just means the diagnostic can tell us if the controller actually reports the condition this PR adds.

@freeqaz-openai freeqaz-openai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please keep the optional quota explanation from delaying the authoritative provisioning poll. At 28ff7a155, if the deployment read sends headers but stalls in its response body, Create waits for the API client’s 15-second deadline and then waits the normal one-second poll interval. I observed 15,993 ms before the provisioning GET could report the job had already succeeded. This is a bounded delay, not a claim that provisioning itself was stuck.

I exercised this head’s actual waitForProvisioning, namedDeploymentWait, status-text helper and unmodified createApiClient against a loopback HTTP fixture, starting with an already-handed-off job. Only two GETs in each case; no write, cluster, credential or provider. The small patch below starts the poll interval first and gives only the optional read that interval as a deadline:

Detail response When the next provisioning GET arrived Visible status before success
Body stalled after headers 996 ms, detail connection closed at 994 ms Ordinary provisioning status
Prompt named quota 988 ms Quota explanation preserved
Prompt 404 987 ms Ordinary provisioning status

Please add the stalled-detail/terminal-job ordering as a regression control; it needn’t assert a precise wall-clock millisecond. This is a scoped Console finding, independent of the first-Agent diagnostic allowlist suggestion. It neither diagnoses the earlier hosted failure as quota nor reviews the whole PR.

Tested narrow poll-budget correction, against 28ff7a1
--- a/apps/controller/src/console/agents/create.mjs
+++ b/apps/controller/src/console/agents/create.mjs
@@ -199,6 +199,8 @@
 // Agent; a reader without it, or any other wait, keeps the bare line.
 const NAMED_DEPLOYMENT_WAITS = new Set(["NAMESPACE_POD_QUOTA_EXCEEDED"]);
 
+const provisioningPollMs = 1_000;
+
 async function namedDeploymentWait(request, namespaceId, job) {
   if (
     typeof job.agentId !== "string" ||
@@ -211,6 +213,7 @@
   try {
     const deployment = await request(
       `${namespacePath(namespaceId)}/agents/${encodeURIComponent(job.agentId)}/deployments/${encodeURIComponent(job.revisionId)}`,
+      { signal: AbortSignal.timeout(provisioningPollMs) },
     );
     const attempt = deployment?.progress?.lastAttempt;
     return NAMED_DEPLOYMENT_WAITS.has(attempt?.code) &&
@@ -231,6 +234,8 @@
   let started = false;
   while (current?.status === "queued" || current?.status === "running") {
     started ||= current.status === "running";
+    // Read an optional explanation within the poll interval, never ahead of a second full wait.
+    const nextPoll = wait(provisioningPollMs);
     setText(
       status,
       (await namedDeploymentWait(request, namespaceId, current)) ??
@@ -239,7 +244,7 @@
     if (typeof current.url !== "string" || !current.url) {
       throw new Error("Provisioning status URL is unavailable.");
     }
-    await wait(1_000);
+    await nextPoll;
     const next = await request(current.url);
     current = next.provisioning ?? next;
   }

@clawsweeper

clawsweeper Bot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(compute): name a full Namespace Pod quota instead of a bare provisioning wait This is item 1/1 in the current shard. Shard 0/1.

This temporary status tracks the active review worker. The completed review will appear in the durable ClawSweeper review comment.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

This branch had an error being deployed

1 failed deployment
integration-qa-pr — 28ff7a15 Deployed Oct 17, 2026 by freeqaz via QA advisory (OpenShell) #1677
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants