Repository navigation
Conversation
…sioning wait Kubernetes Compute reads the ReplicaFailure condition that the Deployment controller copies from a revision's ReplicaSet. A FailedCreate whose quota refusal lists pods (or count/pods) reports pendingReason NAMESPACE_POD_QUOTA_EXCEEDED, which the worker records under the same code with fixed copy. Create Agent shows that copy while its first deployment waits, when the reader can read the deployment.
…Storybook state The browser case now includes a job that created its Agent but has not handed off a revision, so a missing revision guard is caught. The deployment reference lists NAMESPACE_POD_QUOTA_EXCEEDED, and Create Agent gains an 'Initial deployment waits on a full Pod quota' story.
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
…der Harness Review follow-ups for the Pod quota reason: - an unreadable Deployment no longer hides the other one's refusal; - a Harness that a SandboxDriver provisions has no Harness Deployment to read; - the comment notes that ReplicaFailure keeps the first refusal; - tests pin that a held runtime failure leads, and the per-Deployment read; - the deployment reference keeps 'with the deployment's age'.
|
Free's clanker here. One small diagnostic gap on Please add only This doesn’t establish that quota caused the earlier hosted timeout, and it isn’t a request for a broad shared enum or another cluster run. It just means the diagnostic can tell us if the controller actually reports the condition this PR adds. |
freeqaz-openai
left a comment
There was a problem hiding this comment.
Please keep the optional quota explanation from delaying the authoritative provisioning poll. At 28ff7a155, if the deployment read sends headers but stalls in its response body, Create waits for the API client’s 15-second deadline and then waits the normal one-second poll interval. I observed 15,993 ms before the provisioning GET could report the job had already succeeded. This is a bounded delay, not a claim that provisioning itself was stuck.
I exercised this head’s actual waitForProvisioning, namedDeploymentWait, status-text helper and unmodified createApiClient against a loopback HTTP fixture, starting with an already-handed-off job. Only two GETs in each case; no write, cluster, credential or provider. The small patch below starts the poll interval first and gives only the optional read that interval as a deadline:
| Detail response | When the next provisioning GET arrived | Visible status before success |
|---|---|---|
| Body stalled after headers | 996 ms, detail connection closed at 994 ms | Ordinary provisioning status |
| Prompt named quota | 988 ms | Quota explanation preserved |
| Prompt 404 | 987 ms | Ordinary provisioning status |
Please add the stalled-detail/terminal-job ordering as a regression control; it needn’t assert a precise wall-clock millisecond. This is a scoped Console finding, independent of the first-Agent diagnostic allowlist suggestion. It neither diagnoses the earlier hosted failure as quota nor reviews the whole PR.
Tested narrow poll-budget correction, against 28ff7a1
--- a/apps/controller/src/console/agents/create.mjs
+++ b/apps/controller/src/console/agents/create.mjs
@@ -199,6 +199,8 @@
// Agent; a reader without it, or any other wait, keeps the bare line.
const NAMED_DEPLOYMENT_WAITS = new Set(["NAMESPACE_POD_QUOTA_EXCEEDED"]);
+const provisioningPollMs = 1_000;
+
async function namedDeploymentWait(request, namespaceId, job) {
if (
typeof job.agentId !== "string" ||
@@ -211,6 +213,7 @@
try {
const deployment = await request(
`${namespacePath(namespaceId)}/agents/${encodeURIComponent(job.agentId)}/deployments/${encodeURIComponent(job.revisionId)}`,
+ { signal: AbortSignal.timeout(provisioningPollMs) },
);
const attempt = deployment?.progress?.lastAttempt;
return NAMED_DEPLOYMENT_WAITS.has(attempt?.code) &&
@@ -231,6 +234,8 @@
let started = false;
while (current?.status === "queued" || current?.status === "running") {
started ||= current.status === "running";
+ // Read an optional explanation within the poll interval, never ahead of a second full wait.
+ const nextPoll = wait(provisioningPollMs);
setText(
status,
(await namedDeploymentWait(request, namespaceId, current)) ??
@@ -239,7 +244,7 @@
if (typeof current.url !== "string" || !current.url) {
throw new Error("Provisioning status URL is unavailable.");
}
- await wait(1_000);
+ await nextPoll;
const next = await request(current.url);
current = next.provisioning ?? next;
}|
ClawSweeper status: review started. I am starting a fresh review of this pull request: fix(compute): name a full Namespace Pod quota instead of a bare provisioning wait This is item 1/1 in the current shard. Shard 0/1. This temporary status tracks the active review worker. The completed review will appear in the durable ClawSweeper review comment. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
Problem
With the tenant Namespace's ResourceQuota at
pods10/10, Create Agent sat on "Provisioning Agent…" for 60+ s and the Agent page said onlyREVISION_INCOMPLETE/ "Waiting for the runtime to become ready." The revision's ReplicaSets reportedFailedCreate: exceeded quota: openclaw-quotaevery few seconds, but nothing surfaced it (dogfood finding D629).Change
ReplicaFailurecondition on the revision's Deployments: the Harness Deployment, and the gateway Deployment. The Deployment controller copies this condition from the ReplicaSet. If the condition isTruewith reasonFailedCreate, and the quota refusal'srequested:list namespodsorcount/pods, the observation reportspendingReason: NAMESPACE_POD_QUOTA_EXCEEDED. A full quota takes precedence overWORKLOAD_UNSCHEDULABLE. A held runtime failure still wins. Each Deployment is read on its own: an unreadable one reports nothing and does not hide the other's refusal. A Harness that a SandboxDriver provisions has no Harness Deployment, so only the gateway is read. The revision stays pending, since the quota can free up: no new timers, caps or outcomes.NAMESPACE_POD_QUOTA_EXCEEDEDjoinsREVISION_PENDING_CODES(the existing pending path, likeREVISION_UNSCHEDULABLE). Deployment progress carries fixed copy: "The Namespace's Pod quota is full. Stop or delete an Agent, or ask an administrator to raise the quota." It never shows the quota's name or numbers. The Agent page already rendersprogress.lastAttempt, so it now shows that sentence instead of the bare wait.NAMESPACE_POD_QUOTA_EXCEEDED, the status line shows its sentence. That read uses the normal deployment route, so it needs read access to the Agent. A refused read, or any other wait, keeps the bare "Provisioning Agent…" line.docs/reference/agents/deployment.mdlists the new code. Storybook: Pages/Create Agent → Initial deployment waits on a full Pod quota (createDeploymentPodQuotaFull).Verification
tests/conformance/kubernetes-compute.test.mjsadds "a pending dedicated deploy names a full Namespace Pod quota". It uses the fake Kubernetes fixture withReplicaFailure/FailedCreatequota conditions on the Harness and gateway Deployments, coveringpods, mixedpods,requests.cpu, andcount/pods. The negatives are a CPU-only refusal (withpodsonly inused:), statusFalse, another reason, another condition type, a non-quota refusal, and a missing message. It also pins precedence over an unschedulable Pod, a held runtime failure leading instead, and the per-Deployment read. Full file: 367/367 pass.tests/integration/postgres-worker-agent-revision.test.mjs: the pending-reason case adds a third Agent whose Compute reports the new reason. The case asserts the recorded code and copy. Full file: 50/50 against a disposable PostgreSQL.tests/browser/console-agents-2.test.mjsadds "Dedicated Agent creation names a full Namespace Pod quota while it waits". It records every status-line write. A readable deployment writes the sentence, then the bare line once the wait changes. A refused deployment read (404) writes only the bare lines. Deployment reads happen only after handoff. Full file: 19/19.Mutation check: 21 hand mutants, 19 killed. They covered the refusal parse,
pods/count/pods, status, reason and type checks, Harness-only and gateway-only reads, the per-Deployment read, precedence over unschedulable and under a runtime failure, worker mapping, progress copy, the console code filter, the revision guard and error handling. Two mutants survive, and neither changes an outcome. One drops the console'sagentIdguard: a job never carriesrevisionIdwithoutagentId. The other drops the SandboxDriver filter, which only saves a GET that would return 404.Internal review: nothing blocking; low findings and nits applied (per-Deployment read, SandboxDriver skip, comment on the first refusal, runtime-failure precedence test, docs wording).
New tests 20/20 each at 2 CPUs (
taskset -c 0,1).tsc --buildclean. eslint and prettier clean on touched files.No k3d test: k3d cannot run on this box (rootless podman without the cpuset delegation). There is no cheap existing quota fixture in the k3d suites. The fake-cluster case reproduces the exact condition the Deployment controller writes.
Media: console fixture at 1440×900 and 390×844 (simulated UI proof, not live backend proof). Storybook story: Pages/Create Agent → Initial deployment waits on a full Pod quota.
Create Agent, desktop: the provisioning line names the quota.

Create Agent, mobile:

Agent page deployment record:

Before the deployment hands off, the line is still the bare wait: screenshot. Video: Create Agent walkthrough.
Risks
REVISION_INCOMPLETE, because the sentence would be wrong for it. Kubernetes keeps the first refusal message until a Pod create succeeds. So a Pod-quota refusal followed by a CPU-quota refusal still reads as a full Pod quota, and the reverse case reads as the old bare wait.