[Kimiyu-186]

Single-cluster Kubernetes currently creates a separate Gateway namespace for each tenant. This change places dedicated Gateways, Harnesses and canonical tenant storage in the same created or adopted namespace. Pods, ServiceAccounts, private PVCs, node placement and exact Agent/revision network peers remain separate. Harness credential projections contain selected model credentials and transport authentication, never Gateway passwords or channel credentials. The experimental two-cluster profile keeps its separate control target.

Namespace workload managers are trusted for both roles; quotas and namespace-wide operations affect both. This is the intended namespace boundary.

Existing split-layout installations cannot upgrade in place. Configured API composition (including development) and worker startup preflight check all storage-namespace pages and explicitly refuses the old layout before tenant reconciliation. It preserves existing namespaces and state, and the operator reference explains what must be retained. This PR does not migrate private volumes or UID-bound credential references. Removing old labels or namespaces is not a migration.

Every worker, including development workers using Docker or SSH Compute, now runs the selected Driver's optional preflight and refuses startup if it fails. Configured development API composition runs it too. This covers old k3d layouts before reconciliation. The unconfigured Docker entrypoint retains its early read-only check before opening the database pool.

Concurrent legacy password delivery now re-reads a Secret after HTTP 409 and accepts only the exact-owned password with identical bytes. Foreign or conflicting sources still fail. Main's bounded runtime-image snapshot checks and worker liveness configuration are retained alongside shared-namespace assertions and configurable worker settings.

Validation at 71a1cedb (main merged at ad15e741; previous merge c0409318 passed full CI):

• Focused Compute, runtime credential, Secret, Configuration and preflight contracts: 271 passed, 0 failed, 1 macOS argument-size skip.
• Real Kubernetes fixture scenarios: lifecycle/isolation, adopted namespace preservation, provisioning and authenticated PostgreSQL API/worker deployment all pass without skips. The follow-up run passed the first three cases; the API/worker case passes separately (165.6 s) after shortening a new synthetic label value that exceeded Kubernetes' 63-byte limit. Both lifecycle and configured development startup callers check legacy upgrade refusal.
• Production bootstrap/authentication, session persistence across restart and revocation: 2 passed, 0 failed, 0 skipped.
• Negative controls: the new concurrent-delivery case fails with HTTP 409 before the fix; the paginated upgrade case fails with a missing expected rejection before the startup guard. Reverting API and worker startup individually causes the real caller test to fail at the respective missing-rejection assertion; fixed sources were restored and VM/local hashes verified.
• Typecheck, focused ESLint/Prettier, workspace/module boundaries, flow validation, documentation length/links and spec checks pass.

Earlier native acceptance on 94364ae9 covered real OpenClaw/Codex model turns, files and durable private state, attachments, Memory, Skills, mode changes, Secret rotation/rebinding, repeated API restarts and sibling-preserving deletion. Those receipts predate this merge of main; they are not a fresh native-runtime run of this revision. The current fixture results are listed above. Final required CI passes on 71a1cedb, tested as GitHub merge 39e6c50a64cc7b048411a267cd0542e21ef67225 (run 37160853744). The final k3d configuration lane reports 22 passed, 0 failed, 0 skipped, including all four Kubernetes Compute scenarios in one run.

Proof boundaries: disposable single-cluster Kubernetes, not a Helm installation, production node-pool isolation or real two-cluster execution. External Slack and OTLP acceptance are not claimed. See TASK-0043 for source and historical verification evidence.

Main refresh: 36fb4afa merges pinned main dd934c39, resolving seven conflicted files without rewriting history. Shared placement and legacy upgrade assertions follow main's new lifecycle test file and common helper. Runtime credential RBAC diagnostics, scheduling-event filtering and bounded volume cleanup from main are retained. Focused and adjacent local contracts: 537 passed, 0 failed, 1 macOS skip; lint, format, docs/links and workspace/module checks pass. Local typecheck cannot complete against the older installed dependency graph (main adds a direct Fastify compiler dependency); final clean CI passes (run 37351785476), including typecheck. The run tests merge SHA 353f2ce1e2b5310f3cf2a3ca3831a3b4a425479c with parents dd934c39 and 36fb4afa. Both k3d lanes confirm all four core lifecycle/adoption/provisioning/API-worker scenarios passed with no skips; installation/deployment/chat/stop/start smoke also passes. Earlier validation above applies to 71a1cedb, not this refreshed head.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review blocked

Automated review did not run, so no review verdict was produced.

Reason: The input-safety check rejected material in this revision. No detected value, path, or scanner output is reproduced here.

Automatic review is on hold, including after source changes. Request a fresh re-review after resolving the failure; maintainers can also release the hold through an intentional scanner-policy update.

Next step: If this is a genuine credential, remove and rotate it. If it is an intentional test fixture, a maintainer must review and qualify it.

View the workflow run.

[freeqaz]

Free's clanker here (an AI agent reviewing on Free's behalf).

I re-reviewed the branch at fcdfb37da against the earlier head 8f546066d. Apart from the merges of main, the new commits are c5a83e5f5 (separate transport and Gateway password Secrets in both execution modes, with legacy combined Secrets still accepted), 5260adaed (plugin-status fixtures) and fcdfb37da (plan notes). The credential change looks sound: it reads legacy combined sources, rejects conflicting passwords, copies the legacy password into the separate owned Secret without modifying the old Secret, and deletion removes both names. CI Required passes on fcdfb37da and the branch merges cleanly into current main.

Still open: upgrade of existing single-cluster installs (blocking)

The new commits do not change this path.

• After this PR, ensureNamespace labels the single-cluster tenant namespace openclaw.dev/gateway-namespace=<id> (apps/controller/src/drivers/compute/kubernetes/index.ts, around line 2986).
• An install upgraded from the split layout still has oce-gateways-<hash> with the same label.
• resolveKubernetesControlNamespace (around line 1136) lists namespaces by that label and requires exactly one. With two matches it throws OwnershipFailure("Canonical tenant storage requires exactly one namespace."), so Kubernetes Secret and Configuration storage discovery fails for every existing tenant after the upgrade.
• deleteGatewayNamespace returns true without action when there is no execution cluster. The old Gateway namespace, its PVCs and its Secrets are left behind, and nothing reads them anymore. The existing Gateways' private state is not carried over either.

The plan says "Existing split-layout resources and volumes are not migrated or deleted." That holds only for fresh installs. For an install that already runs, one of these is needed:

1. a migration step that moves the canonical Secrets/Configuration and Gateway state into the tenant namespace and retires oce-gateways-<hash>;
2. discovery that prefers the legacy split namespace while it exists, plus a documented cleanup path; or
3. an explicit upgrade block with operator instructions.

Whichever you pick, a regression test that starts from the split layout would cover it.

Still open: isolation boundary (needs maintainer/security sign-off)

Gateways (holding the Gateway password, transport token and canonical tenant credentials) now share a Kubernetes namespace with Harness Pods that run untrusted code. Isolation now rests only on per-Pod service accounts, RBAC and NetworkPolicy, and any namespace-scoped grant or policy gap now reaches Gateway Secrets. c5a83e5f5 also adds another Secret (gateway-password-<suffix>) in that shared namespace. This is a deliberate design change in the plan. I'm not calling it a bug, but it should get an explicit sign-off from a maintainer who owns the runtime isolation model before this lands.

Minor (non-blocking)

• ensureLegacyGatewayPassword creates the separate Secret without handling a concurrent create (409). The next delivery would converge, but treating "already exists with the same bytes" as success would avoid a spurious deploy failure.
• It runs on every Harness auth delivery, which adds two Secret GETs per deployment. That's fine, just noting it.

[freeqaz]

Free's clanker here (an AI agent reviewing on Free's behalf).

The only new commit since my last comment, 94364ae9f, is test-only: it makes the k3d test helper (tests/helpers/harness-topology-k3d-real.mjs:878-911) follow the current API Pod across repeated restarts. It does not touch the single-cluster upgrade break or the Gateway/Harness isolation concern, so both are still open. CI Required passes on 94364ae9f, but the branch now has a merge conflict with main in tests/integration/kubernetes-compute-real.test.mjs.

[freeqaz]

Free's clanker here (an AI agent reviewing on Free's behalf).

Re-review at head 71a1cedb2. The upgrade break from my earlier review is resolved; two scope notes and one optional nit remain.

• Upgrade break (earlier review): resolved by explicit refusal. Single-cluster preflight now pages every namespace labelled openclaw.dev/gateway-namespace and refuses a legacy oce-gateways-<hash> namespace that lacks openclaw.dev/namespace. Main strips that label from Gateway namespaces, so the check matches real split-layout installs. Both the API and the worker refuse before tenant reconciliation, so nothing gets a duplicate label or orphaned storage. The new conformance cases (a later page, plus an adopted shared namespace that is accepted) and the real-cluster caller regression cover it. No automatic migration is offered; the plan and the reference doc say so.
• Main. git merge-tree --write-tree origin/main 71a1cedb2 is clean against current main 1e7060af0. Your merge c04093189 resolved the six docs files that our recent word-limit trims also touched: harness-execution-topology.md, native-service-account-credential-delivery.md, local-kubernetes-development.md, kubernetes-compute/networking-and-isolation.md, kubernetes-compute/storage-and-credentials.md and docs/testing/kubernetes.md. On the merged tree every changed doc is under 2500 words; the largest are kubernetes-compute.md at 2467 and harness-execution-topology.md at 2462. The old front-matter conflict with fix(runtime): give the startup model probe room for reasoning models #920 is also gone.
• CI. CI Required is green on 71a1cedb2.

Scope notes for the maintainer:

1. 71a1cedb2 goes beyond the shared-namespace change. Every worker, not only production ones, now runs compute.preflight(), and configured development API composition does too. That is what makes the refusal cover development k3d installs. It also means a development worker on Docker or SSH Compute now fails startup when that preflight fails. This is reasonable, but it is a behaviour change worth one line in the PR body.
2. The PR is now about +1984/-1125 across 49 files. That is a large review surface for one landing.

Optional nit: in an unconfigured development worker, worker.mjs:124-127 already runs the Docker driver's preflight(), and ControllerWorker.start() now runs it a second time (ping, version and two image inspects). It is read-only and cheap, but the explicit call in worker.mjs could now be dropped.

[freeqaz]

Free's clanker here (an AI agent reviewing on Free's behalf).

I checked the new merge of main, 36fb4af. The conflict resolution in the real-k3d tests keeps main's coverage. kubernetes-compute-real, kubernetes-compute-driver-real and kubernetes-plugin-status-real have the same top-level tests as main, and kubernetes-runtime-credentials goes from 18 to 22, the extra four being this PR's shared-namespace regressions. The branch merges cleanly with current main (dd934c3), and CI Required is green on this head.

My 10-03 verdict stands. The upgrade break is resolved, and the two scope notes for the maintainer still apply:

• every worker now runs compute.preflight(), including development workers on Docker and SSH;
• the review surface is large (about +2.0k/-1.1k across 51 files).

[freeqaz]

Free's clanker here (an AI agent reviewing on Free's behalf).

A heads-up from upgrading our dogfood install (single cluster, released layout, three Namespaces with Agents) to main after this landed. The new API and worker refuse to start while split-layout tenants exist, as the kubernetes-compute reference describes. The upgrade path doesn't catch that yet:

1. upgrade-checklist.md and production-upgrade.md don't mention it. Only reference/drivers/kubernetes-compute.md does, so an operator who follows the checklist finds out only when the new Pods won't start. docs(deploy): flag split-layout upgrades and fix routing namespace notes #1415 added the note to the local k3d guide; both upgrade guides are at their word cap.
2. The image helper's startup preflight (scripts/upgrade-startup-preflight.mjs) only loads the Installation and doesn't run the Compute check. On a split-layout install it scales OCC down before the new release refuses to start, and production-upgrade-recovery.md has no path for that case.
3. There's no supported way to move existing tenants to the shared layout. We deleted the old tenants through OCC on the old release, following the RWX legacy procedure, and then the in-place upgrade went cleanly: API gap 18.3 s, both new Agent kinds answered.

We're preparing a small fix for 1 and 2: run the Compute check in the helper's preflight before it stops anything, and add a warning to the checklist. Whether 3 should be a migration or a documented "discard first" step is your call.

[freeqaz]

Free's clanker here (an AI agent working for @freeqaz). Heads-up for @Kimiyu-186: we're designing an in-place migration off the split oce-gateways-<hash> layout. Every released single-cluster install with a Namespace has that layout, so the startup refusal added here currently blocks their upgrade unless they delete all Namespaces first.

The shape under review: an offline step in scripts/upgrade-production-images, run after quiescence and before the new API/worker start.

• Copy the canonical Secrets, Configuration ConfigMaps, ServiceAccount credentials and dedicated runtime credentials into the tenant namespace.
• Rebind the dedicated gateway-state-* PVs.
• Relocate the occ.secrets backend namespace/UID rows using a migrator-only function.
• Relabel, then redeploy the Agents that were running.

It keeps the refusal for anyone who doesn't run the migration. We'll link the PR or design decision here. If you already have a migration in mind or see a trap in this approach, we'd like to hear it.

[freeqaz]

Free's clanker here (an AI agent reviewing on Free's behalf).

A heads-up about upgrades after this PR. Releases before it created one oce-gateways-<hash> namespace per ready Namespace in a single cluster. That namespace holds the Gateway Deployment, its gateway-state PVC and the canonical Secrets. The startup preflight added here refuses that layout. So today, every released single-cluster install with at least one Namespace cannot upgrade, unless it deletes its Namespaces. The upgrade helper's candidate preflight stops safely before quiescence, but there is no path forward.

#1643 proposes the smaller fix:

• Keep existing split-layout tenants on their old Gateway namespace, detected strictly. The driver never creates that namespace for new tenants and deletes it with the tenant.
• Narrow the preflight to refuse only a Namespace that two namespaces claim as storage.

A full migration into the shared namespace is deferred. It would have to move PVCs and UID-bound credential references.

#1643 reverses part of plan 0043's choice, so it is a draft and will not be merged without your or Free's agreement. Feedback on whether you'd rather keep the refusal and own a migration is very welcome.

[freeqaz]

Free's clanker here (an AI agent working on Free's behalf).

@Kimiyu-186, an FYI on the upgrade path from the 2026-09-28 release to this PR's shared tenant namespace: #1979 adds an in-place "adopt" step, and plan 0043's single target is kept.

• Each released oce-gateways-<hash> storage namespace becomes the tenant namespace for its Namespace. Compute accepts that name as a tenant namespace only while it still carries its own openclaw.dev/gateway-namespace label, so the two-cluster profile still refuses a tenant-labelled control namespace.
• An operator script moves the old Harness namespace's claims (PersistentVolume rebind), copies its Agent Secrets and archives its routes, then deletes the old namespace once the new release runs.

We dropped the earlier alternative of keeping old-layout tenants in place (#1643), so nothing here reverses this PR's design. Review of the Compute change in #1979 is welcome if you have time.