Repository navigation
fix(compute): preserve native Gateway state across restarts - #3041
Conversation
Extract the persistence and generated-worker-profile restart repair from PR #1982, with its original tests and focused documentation. Trim the owning pages below the documentation word limit. Original-commit: 5ed7304 Signed-off-by: sallyom <somalley@redhat.com> Co-authored-by: sallyom <11166065+sallyom@users.noreply.github.com> Signed-off-by: roboclaw-bot <309084314+roboclaw-bot@users.noreply.github.com>
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
clawsweeper
Bot
added
clawsweeper:intake:bug
|
Codex review: needs maintainer review before merge. What this changesThis PR retains Gateway client credentials on the existing private PVC and permits reuse of an identical generated native worker profile. Example: A dedicated native Gateway restarts with its generated dedicated-native profile.
Review scores
ProductKind: Bug fix · Worth it: Yes · Fix scope: Complete Merge readiness✅ Ready for maintainer review Keep this PR open: current main still needs the restart repair, and review found no actionable introduced defect. Priority: P2 Before mergeNone. FindingsNone. Agent review detailsHow this fits togetherKubernetes Compute renders Agent-owned Gateway storage and startup programs; the Gateway loads retained state and routes workspace and model operations to its dedicated Harness. flowchart TB
A[Agent configuration] --> B[Kubernetes Compute]
B --> C[Gateway startup wrapper]
B --> D[Private Gateway PVC]
D --> C
C --> E[Gateway]
E --> F[Dedicated Harness]
Technical reviewBest possible solution: Retain the client store through the existing private-storage owner and preserve exact generated-profile equality at startup. Do we have a high-confidence way to reproduce the issue? Current main rejects an already-generated worker profile unconditionally; the published native control run independently records client-store failure after Gateway replacement. Is this the best way to solve the issue? The patch uses existing storage and startup owners, preserves conflicting-profile rejection, and avoids a parallel persistence or authorization mechanism. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against bca5b199b037. Provenance checked
TestingProof path: shipped entry point. SecurityNone. EvidenceWhat I checked:
Likely related people:
LabelsLabel changes:
Label justifications:
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment HistoryReviewed October 20, 2026, 7:03 PM ET / 23:03 UTC (Revision 3). |
Preserve main native-admin return-flow documentation while keeping the client-state and exact-worker-profile fixes separately reviewable for PR #1982 qualification. Co-authored-by: sallyom <11166065+sallyom@users.noreply.github.com> Signed-off-by: roboclaw-bot <309084314+roboclaw-bot@users.noreply.github.com>
|
Fresh Linux/k3d native OpenClaw + OpenShell e2e passed: 1 passed, 0 failed, 0 skipped with this persistence/startup repair applied to #1982. Tested revisions: #3041 repair ./scripts/k3d test --harness openclawCompleted The passing run exercised successive real model turns and workspace reads/writes for two Agents, cross-Agent enrollment/device rejection, natural setup-token expiry with a fresh positive control, node revocation, refused workspace/model operations, and a healthy peer. The security operations also assert unchanged Gateway/Harness Pod identities and restart counts. This does not claim arbitrary existing-Agent migration coverage or revocation persistence across another Pod replacement. Qualifications: two disclosed local test-only adapters were required: native-only preparation omits the unused Codex credential requirement, and the rootless Linux TLS relay uses an exact high loopback port. Authority and secret-exposure assertions were retained. NetworkPolicies remained enabled with the legacy iptables backend. The native runtime image was reused by immutable reference; the controller was rebuilt from the combined candidate. Full sanitized image/source receipts and both adapter diffs are in the detailed #1982 proof. Focused Compute regressions: 2 passed, 0 failed/skipped. Lint, changed-file formatting, workspace isolation, and whitespace checks passed. The current #3041 head is Sanitized native e2e assertion transcript |
Why the native k3d run needs this repair, while default Codex/OpenShell testing did not expose itThe two changes in #3041 have different causes. I inspected the OCE code at 1. Client persistence: a runtime-layout mismatch, not a native-only credential directoryThe native k3d helper selects the sibling OpenClaw source checkout. Without that selector, preparation builds the OCE runtime Dockerfile, whose default pin is At the default pin, device identities and device-auth operations use the shared OpenClaw state database, under The VM source includes upstream #174599, commit Before #3041, OCE retained
That matches Sally’s actual #1982-only failure: the native model probe was READY and the worker connected; the failure occurred after Gateway Pod replacement. The combined #1982 + #3041 run passed. This observation specifically supports retaining the new client store; it does not independently prove that both repairs were triggered in that same VM attempt. Why this was not seen in default Codex/OpenShell runs: they do not, by default, select that sibling source checkout, so they do not exercise the same client-store layout. There is also a lifecycle difference: eligible Codex deployments use a mutable workspace-node binding, while dedicated native deployments exclude that path and deliver the worker identity/profile through the Gateway Pod spec. This makes Gateway replacement during native enrollment relevant sooner. Codex is not inherently immune: a Gateway using the new client-store runtime can lose it on replacement too if only The new client schema contains device identities/private keys, role and Gateway-origin device tokens, native execution socket credentials, and the migration receipt. It is not merely a harmless cache. However, local identity/token custody was already persistent in the older shared-state layout. This mount preserves that continuity across the upstream storage split; it is more precise than describing it as entirely new credential persistence. Server-side paired-device authorization/revocation remains a separate owner; this source distinction does not replace final-effect/replacement proof. 2. Worker-profile restart: genuinely native-specificOnly dedicated native OpenClaw receives For a native Gateway that reuses its writable configuration after a container restart, the old wrapper rejects its own existing generated profile. #3041 permits deep equality with the complete expected profile but still rejects conflicting device/settings. The previously recorded base-fails/head-passes conformance comparison and real-image startup/schema case support this independent repair. A fresh-start-only test does not exercise reuse of generated configuration. Evidence limits and dispositionSally’s combined Linux/k3d proof is now available: one pass, no failures/skips, with the source/image receipts and two disclosed test-only adapters. It targets #3041’s This also separates the VM’s later persistence failure from hosted CI’s earlier native authentication 401 and Codex WebSocket 502. The storage diagnosis does not establish the cause of those hosted failures. No code, credentials, image pins, assertions or review holds were changed during this investigation. |
Gateway replacement authority qualification: PASSThe requested native runtime replacement test now passes: 1 passed, 0 failed, 0 skipped. This extends the earlier combined e2e, which deliberately kept Pods unchanged during the authority assertions.
Revisions and environment: #3041 repair ./scripts/k3d test --harness openclawStarted Scope and retained authority: the selected runtime’s client schema supports private device identities, role/origin-scoped device token caches, native execution socket credentials, and migration receipts. Server-side revocation is separately persisted with the paired device, and token verification rejects a revoked entry. This run qualifies fresh provisioning and replacement with already-written current-layout state; it does not claim general cross-version migration, recovery of already-lost credentials, embedded/Codex execution, OpenShift, or all storage classes. The only new changes were local test-only qualification assertions, included below; no additional production repair was used. The same two disclosed environment adapters from the earlier proof remain: native-only preparation does not require an unused Codex token, and the rootless TLS relay uses an exact high loopback port. NetworkPolicies remained enabled with legacy iptables. Immutable runtime/controller images were reused. An initial qualification attempt failed in the new helper because it tried to capture the worker token after revocation had cleared the cache; moving capture before revocation produced this pass. No production code or authority assertions were weakened. Sanitized assertion transcriptSource, immutable images, and result receipt{
"runReceipt": {
"oceCommit": "e583b67b506ce3e6a7a0889003a02f4688db44d0",
"openclawCommit": "4c7a1654f8f4a1dd2ecab4c421419f1093bc2d65",
"openShellVersion": "0.1.4",
"model": "gpt-6-luna",
"engine": "podman",
"command": [
"./scripts/k3d",
"test",
"--harness",
"openclaw"
],
"startedAt": "2026-10-20T22:23:56Z",
"nativeBuildHeapMb": 6144,
"qualification": "Gateway replacement with retained valid, foreign, and revoked authority",
"testFileSha256": "358cf6779a5a2e0d91d39ceeb4a281658911dc167369dce1729a6855fca5a2da",
"iptablesBackend": "legacy",
"freshClusterAndDatabase": true,
"productionPatchesBeyondPR": [
"06ae1e0e13b7517abaa933911d367ac8c8fa2336"
],
"basePr1982Commit": "73c39aed60cab99aa642b282ed99b74e48ce4b2a",
"repairPr3041Commit": "06ae1e0e13b7517abaa933911d367ac8c8fa2336",
"controllerSourceCommit": "e583b67b506ce3e6a7a0889003a02f4688db44d0",
"imageReuse": "Existing exact-source immutable runtime/controller builds; imported into the new cluster by standard preparation",
"imageSelectorWrapperSha256": "0e0dbcb88b7b14757e4bee9fe22b4c67fac152ac6c66a2da4934ec9079c1f6b1",
"rootlessRelayPatchSha256": "74fdf329a4e0dfe9f98c7497372f8b0b18613b83fc7c045a543d0c993e16ac17",
"preparationPatchSha256": "dca5b2a998a8d25a7560cb4d48650058576a90ae700622a38ba464e7847e3f3f",
"exitCode": 0,
"elapsedSeconds": 1341.11,
"finishedAt": "2026-10-20T22:46:17Z"
},
"counts": {
"tests": 1,
"pass": 1,
"fail": 0,
"skipped": 0
},
"checks": {
"syntax": 0,
"format": 0,
"lint": 0,
"whitespace": 0
},
"images": [
{
"kind": "k3d-image",
"name": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-kubernetes-gateway-image-56a0a6394c97:local",
"sourceImage": "localhost/openclaw-ci-image-openclaw-ci-local-e72c0affcd5f/runtime@sha256:8950cea722cd6a6afdad7bec803b3db9b754a2efc43f5c87f3667773798de98e",
"reference": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-kubernetes-gateway-image-56a0a6394c97@sha256:7d930c9455551e0570b102510093587fe5130e55d6383150f696911aea774b6a"
},
{
"kind": "k3d-image",
"name": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-production-controller-image-1bcbd9ebc1af:local",
"sourceImage": "localhost/pr1982-3041/controller@sha256:5beaa58447142dd58825a3e99eb99b966af8c035f02c0f95bee76f76e5c6d2a8",
"reference": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-production-controller-image-1bcbd9ebc1af@sha256:37a5dba34735e2a7e6133842946e273dbd834fd90a553f24611ad8e9cb9c4f6e"
},
{
"kind": "k3d-image",
"name": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-openshell-gateway-image-47dd84e28776:local",
"sourceImage": "ghcr.io/nvidia/openshell/gateway:5a7728c304b5528e954e9ae900cf21f84e1a12b1@sha256:bdb4fe248070032c88be78f029dad9325cb698480e90486e1413a674a5d8edc8",
"reference": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-openshell-gateway-image-47dd84e28776@sha256:f8b547a91cf53642c0fbad5691b1739f17d545283fcfc6844bf922cec4c18017"
},
{
"kind": "k3d-image",
"name": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-openshell-supervisor-image-79a0e9d2d34e:local",
"sourceImage": "ghcr.io/nvidia/openshell/supervisor:5a7728c304b5528e954e9ae900cf21f84e1a12b1@sha256:aa681d8264af6ee860e38349ed3c76609a51778eb6d0f02743df03787b80a290",
"reference": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-openshell-supervisor-image-79a0e9d2d34e@sha256:4a80d5151a87906a86c0a06c330b17f0cef558d51b9a52bb58161dd95fd1ecac"
},
{
"kind": "k3d-image",
"name": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-openshell-sandbox-image-516bb527b5d2:local",
"sourceImage": "ghcr.io/nvidia/openshell/sandbox:5a7728c304b5528e954e9ae900cf21f84e1a12b1@sha256:ca00c094aeaa60681b8196cda18ab3c5dd5f7d782d4c36798256427b67599070",
"reference": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-openshell-sandbox-image-516bb527b5d2@sha256:6e5696d10b0df98d44cda8e789d815fda97e95ec44c224c750d278c3509b6c39"
},
{
"kind": "k3d-image",
"name": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-keycloak-image-917c9812706b:local",
"sourceImage": "quay.io/keycloak/keycloak:26.7.5@sha256:37dbaf6f0722c9ec246335f36e1ef8b2e6cb960f7c27e0d8c615121a3d475a85",
"reference": "localhost/openclaw-k8s-opencl-eeed62ce09b9/occ-test-keycloak-image-917c9812706b@sha256:fcbdc620064b5115738e24a1f57b0f54a4fc16a85ff6879a680225cddfa36fc4"
}
],
"testFileSha256": "358cf6779a5a2e0d91d39ceeb4a281658911dc167369dce1729a6855fca5a2da"
}Complete additional test-only replacement qualification patchApply on top of the previously disclosed native-only preparation/rootless relay adapters. This patch is retained for handoff; it has not been pushed to either PR branch. --- a/tests/integration/sandbox-driver-openshell-k3d-real.test.mjs
+++ b/tests/integration/sandbox-driver-openshell-k3d-real.test.mjs
@@ -4237,6 +4237,150 @@
context.diagnostic("native-authority " + JSON.stringify({ stage, ...observations }));
}
+// This qualification reads the real client database without exporting its contents.
+// Only the worker handshake probe below retains credentials in test-process memory.
+async function nativeClientState(topology, worker = false) {
+ const pod = worker ? topology.harnessPod : topology.gatewayPod;
+ const script = `
+const { DatabaseSync } = require("node:sqlite");
+const { createHash } = require("node:crypto");
+const { join } = require("node:path");
+const worker = process.argv[1] === "worker";
+const root = worker ? process.env.OPENCLAW_NODE_STATE_DIR : process.env.OPENCLAW_STATE_DIR;
+const db = new DatabaseSync(join(root, "client", "credentials.sqlite"), { readOnly: true });
+const select = (sql) => db.prepare(sql).all();
+const identities = select("SELECT identity_key, device_id, public_key_pem, private_key_pem FROM device_identities ORDER BY identity_key");
+const tokens = select("SELECT device_id, role, token, scopes_json FROM device_auth_tokens ORDER BY device_id, role");
+const origins = select("SELECT gateway_scope, device_id, role, token, scopes_json FROM gateway_origin_device_tokens ORDER BY gateway_scope, device_id, role");
+const receipt = select("SELECT version, phase, source_store FROM client_credential_migration ORDER BY version");
+const digest = (value) => createHash("sha256").update(JSON.stringify(value)).digest("hex");
+const result = {
+ identityCount: identities.length, tokenCount: tokens.length, originTokenCount: origins.length,
+ identitiesDigest: digest(identities), tokensDigest: digest(tokens), originsDigest: digest(origins),
+ receiptDigest: digest(receipt), receiptComplete: receipt.length === 1 && receipt[0].phase === "complete",
+};
+if (worker) {
+ const token = [...tokens, ...origins].find((row) => row.role === "node");
+ const identity = identities.find((row) => row.device_id === token?.device_id);
+ if (!identity || !token) throw new Error("Enrolled worker client authority missing");
+ result.deviceIdentity = { deviceId: identity.device_id, publicKeyPem: identity.public_key_pem, privateKeyPem: identity.private_key_pem };
+ result.deviceToken = token.token;
+}
+db.close();
+process.stdout.write(JSON.stringify(result));
+`;
+ const output = await kubectl(
+ "exec",
+ pod.metadata.name,
+ "--namespace",
+ worker ? topology.placement : topology.gatewayPlacement,
+ "--container",
+ worker ? "agent" : "gateway",
+ "--",
+ "node",
+ "-e",
+ script,
+ worker ? "worker" : "gateway",
+ );
+ return JSON.parse(output);
+}
+
+async function replaceNativeGateway(context, topology) {
+ const old = await resource("pod", topology.gatewayPod.metadata.name, topology.gatewayPlacement);
+ const claims = (pod) =>
+ pod.spec.volumes
+ .filter((v) => v.persistentVolumeClaim)
+ .map((v) => v.persistentVolumeClaim.claimName)
+ .sort();
+ const claimNames = claims(old);
+ assert.ok(claimNames.length > 0, "Gateway replacement must retain its actual PVC");
+ const claimsBefore = await Promise.all(
+ claimNames.map((name) => resource("pvc", name, topology.gatewayPlacement)),
+ );
+ await kubectl(
+ "delete",
+ "pod",
+ old.metadata.name,
+ "--namespace",
+ topology.gatewayPlacement,
+ "--wait=true",
+ "--timeout=90s",
+ );
+ const replacement = await waitFor(
+ "replacement Gateway ready with a new Pod UID",
+ async () => {
+ const pods = (await resources("pods", topology.gatewayPlacement)).filter(
+ (pod) =>
+ pod.metadata.labels?.["openclaw.dev/workload-role"] === "gateway" &&
+ pod.metadata.labels?.["openclaw.dev/agent"] === topology.agent.id &&
+ pod.metadata.deletionTimestamp === undefined &&
+ pod.metadata.uid !== old.metadata.uid &&
+ pod.status?.conditions?.some(
+ (condition) => condition.type === "Ready" && condition.status === "True",
+ ),
+ );
+ return pods.length === 1 ? pods[0] : undefined;
+ },
+ 180_000,
+ );
+ assert.deepEqual(claims(replacement), claimNames);
+ for (const previous of claimsBefore) {
+ const current = await resource("pvc", previous.metadata.name, topology.gatewayPlacement);
+ assert.equal(
+ current.metadata.uid,
+ previous.metadata.uid,
+ "replacement must reuse the same PVC object",
+ );
+ assert.equal(
+ current.spec.volumeName,
+ previous.spec.volumeName,
+ "replacement must reuse the same PV",
+ );
+ }
+ nativeAuthorityTrace(context, "gateway-replaced-same-pvc", {
+ previousPodUid: old.metadata.uid,
+ replacementPodUid: replacement.metadata.uid,
+ claimUids: claimsBefore.map((claim) => claim.metadata.uid),
+ });
+ return { ...topology, gatewayPod: replacement };
+}
+
+async function nativeWorkerEffects(topology, nonce) {
+ const script = `
+const fs = require("node:fs");
+const path = require("node:path");
+const crypto = require("node:crypto");
+const nonce = process.argv[1];
+const workspace = process.env.OPENCLAW_WORKSPACE_DIR;
+const content = fs.readFileSync(path.join(workspace, "AGENTS.md"));
+let nonceFound = false;
+const walk = (root) => {
+ for (const entry of fs.readdirSync(root, { withFileTypes: true })) {
+ const file = path.join(root, entry.name);
+ if (entry.isDirectory()) walk(file);
+ else if (entry.isFile() && fs.readFileSync(file).includes(nonce)) nonceFound = true;
+ }
+};
+walk(process.env.OPENCLAW_NODE_STATE_DIR);
+process.stdout.write(JSON.stringify({ workspaceDigest: crypto.createHash("sha256").update(content).digest("hex"), nonceFound }));
+`;
+ return JSON.parse(
+ await kubectl(
+ "exec",
+ topology.harnessPod.metadata.name,
+ "--namespace",
+ topology.placement,
+ "--container",
+ "agent",
+ "--",
+ "node",
+ "-e",
+ script,
+ nonce,
+ ),
+ );
+}
+
async function assertNativeAuthorityChain(context, topology, positive) {
const adminA = await connectNativeAdministrator(context, topology);
const issueSetup = async () => {
@@ -4371,6 +4515,11 @@
topology.gatewayPod.metadata.name,
topology.gatewayPlacement,
);
+ // Capture real credentials while still valid: the native client clears them on revocation.
+ const revokedClient = await nativeClientState(topology, true);
+ assert.equal(revokedClient.deviceIdentity.deviceId, deviceId);
+ const validClient = await nativeClientState(peer, true);
+ assert.equal(validClient.deviceIdentity.deviceId, peerNodeId);
const revoked = await adminA("device.token.revoke", { deviceId, role: "node" });
assert.equal(revoked.deviceId, deviceId);
assert.equal(revoked.role, "node");
@@ -4438,6 +4587,140 @@
);
assert.equal((await adminA("node.describe", { nodeId: deviceId })).connected, false);
nativeAuthorityTrace(context, "revocation-complete-peer-still-working");
+
+ // Persist revocation BEFORE replacing either Gateway. A surviving healthy peer
+ // distinguishes authority rejection from a broken replacement or model provider.
+ const stateA = await nativeClientState(topology);
+ const stateB = await nativeClientState(peer);
+ assert.equal(stateA.receiptComplete, true);
+ assert.equal(stateB.receiptComplete, true);
+ const workspacePath = (item) =>
+ `/namespaces/${item.namespaceId}/agents/${item.agent.id}/workspace/files/AGENTS.md`;
+ const peerFile = await peer.request("GET", workspacePath(peer));
+ assert.equal(peerFile.status, 200);
+ const retained = await nativeWorkerEffects(topology, "unissued-nonce-" + randomUUID());
+ const replacementA = await replaceNativeGateway(context, topology);
+ const replacementB = await replaceNativeGateway(context, peer);
+ assert.deepEqual(
+ await nativeClientState(replacementA),
+ stateA,
+ "A must load its unchanged prior client store and receipt",
+ );
+ assert.deepEqual(
+ await nativeClientState(replacementB),
+ stateB,
+ "B must load its unchanged prior client store and receipt",
+ );
+ nativeAuthorityTrace(context, "prior-client-state-loaded", {
+ agents: 2,
+ migrationReceiptsComplete: true,
+ identities: [stateA.identityCount, stateB.identityCount],
+ tokens: [stateA.tokenCount, stateB.tokenCount],
+ });
+ const replacementAdminA = await connectNativeAdministrator(context, replacementA);
+ const replacementAdminB = await connectNativeAdministrator(context, replacementB);
+ const requesterA = await createNativeTurnRequester(context, replacementA);
+ const requesterB = await createNativeTurnRequester(context, replacementB);
+ await waitFor("valid prior worker reconnects after replacement", async () =>
+ (await replacementAdminB("node.describe", { nodeId: peerNodeId })).connected === true
+ ? true
+ : undefined,
+ );
+ assert.equal((await replacementAdminA("node.describe", { nodeId: deviceId })).connected, false);
+ assert.deepEqual(
+ (await peer.request("GET", workspacePath(peer))).data,
+ peerFile.data,
+ "the peer's previously written workspace must remain readable",
+ );
+ await assertNativeTurn(requesterB, peerTurns.sessions[0]);
+ await assertNativeOpenClawWorkspaceFiles(replacementB);
+ nativeAuthorityTrace(context, "valid-prior-session-and-workspace-after-replacement");
+
+ const nodeProbe = { clientName: "node-host", mode: "node", role: "node", scopes: [] };
+ const revokedDenial = await expectNativeAuthorityDenied(
+ {
+ ...nodeProbe,
+ url: requesterA.nodeUrl,
+ deviceIdentity: revokedClient.deviceIdentity,
+ deviceToken: revokedClient.deviceToken,
+ },
+ "AUTH_DEVICE_TOKEN_MISMATCH",
+ );
+ const foreignDenial = await expectNativeAuthorityDenied(
+ {
+ ...nodeProbe,
+ url: requesterB.nodeUrl,
+ deviceIdentity: revokedClient.deviceIdentity,
+ deviceToken: revokedClient.deviceToken,
+ },
+ "AUTH_DEVICE_TOKEN_MISMATCH",
+ );
+ // A's credentials must not be rejected everywhere merely because they are revoked:
+ // B's still-valid real worker credential must also fail at the foreign Gateway A.
+ const validForeignDenial = await expectNativeAuthorityDenied(
+ {
+ ...nodeProbe,
+ url: requesterA.nodeUrl,
+ deviceIdentity: validClient.deviceIdentity,
+ deviceToken: validClient.deviceToken,
+ },
+ "AUTH_DEVICE_TOKEN_MISMATCH",
+ );
+ nativeAuthorityTrace(context, "retained-revoked-and-foreign-authority-denied-after-replacement", {
+ revokedDenial,
+ foreignDenial,
+ validForeignDenial,
+ });
+ const writeNonce = "replacement-revoked-write-" + randomUUID();
+ for (const method of ["GET", "PUT"]) {
+ const result = await topology.request(
+ method,
+ workspacePath(topology),
+ method === "PUT" ? { content: writeNonce } : undefined,
+ );
+ assert.equal(result.status, 503, "replacement must not restore revoked workspace access");
+ }
+ const rejectedTurn = await requesterA.turn(positive.sessions[0]);
+ assert.ok(rejectedTurn.status >= 500 && rejectedTurn.status <= 599);
+ assert.ok(rejectedTurn.payload.error);
+ assert.equal(
+ /capacity|billing|quota|rate.limit/iu.test(rejectedTurn.payload.error.message ?? ""),
+ false,
+ );
+ assert.equal(JSON.stringify(rejectedTurn.payload).includes(rejectedTurn.nonce), false);
+ const effects = await nativeWorkerEffects(topology, rejectedTurn.nonce);
+ assert.equal(
+ effects.workspaceDigest,
+ retained.workspaceDigest,
+ "revoked write must not reach the Harness",
+ );
+ assert.equal(
+ effects.nonceFound,
+ false,
+ "revoked model request must not reach retained worker state",
+ );
+ const writeEffects = await nativeWorkerEffects(topology, writeNonce);
+ assert.equal(writeEffects.nonceFound, false);
+ await assertNativeTurn(requesterB, peerTurns.sessions[0]);
+ assert.equal((await replacementAdminA("node.describe", { nodeId: deviceId })).connected, false);
+ for (const item of [topology, peer]) {
+ const observed = await resource("pod", item.harnessPod.metadata.name, item.placement);
+ assert.equal(observed.metadata.uid, item.harnessPod.metadata.uid);
+ assert.deepEqual(
+ observed.status.containerStatuses.map(({ name, restartCount }) => ({ name, restartCount })),
+ item.harnessPod.status.containerStatuses.map(({ name, restartCount }) => ({
+ name,
+ restartCount,
+ })),
+ "Gateway replacement must not restart or replace the retained Harness",
+ );
+ }
+ nativeAuthorityTrace(context, "replacement-authority-final-effects-verified", {
+ revokedWorkspaceWriteLanded: false,
+ revokedModelReachedWorker: false,
+ revokedModelStatus: rejectedTurn.status,
+ healthyPeerCompleted: true,
+ });
}
async function holdNativeOpenClawDemo(context, topology) { |
|
@clawsweeper re-review The requested Gateway replacement authority proof now passes: 1 passed, 0 failed, 0 skipped. Sanitized transcript, exact revisions/image digests, and complete qualification patch. The PR body now records the retained-state, valid-session, revoked/foreign-authority and final-effect results, plus the combined-candidate and current-layout scope limits. No review holds were manually cleared. |
|
🦞👀 Re-review progress:
|
clawsweeper
Bot
added
clawsweeper:intake:bug
clawsweeper
Bot
added
clawsweeper:intake:bug
|
Free's clanker here (an AI agent working on Free's behalf). Review of head
Still open before a merge: the ClawSweeper item on existing-state compatibility (an existing Agent's pre-change PVC through the controller cutover). A re-review of this head is running. This PR persists the client credential store across Gateway replacement, and its scope includes the Compute startup wrapper, so @freeqaz-openai should make the landing call; I am not merging it. |
Problem
A dedicated native Gateway rejects the worker profile that its own wrapper generated when the container restarts. Gateway Pod replacement also discards the client credential store while retaining the database whose migration receipts refer to that store.
This extracts Sally’s commit
5ed73046b9808e4759e47c33d736d97eb7cc0166from PR #1982, as requested. It is based directly on main, not on #1982.Relationship to #1982
Sally’s local k3d testing reports that #1982 alone fails and needs this fix. This PR is therefore a functional prerequisite for #1982’s native e2e qualification, while remaining separately reviewable and based on main. Sally’s completed combined native proof now reports 1 passed, 0 failed/skipped. Its #1982-only control failed after Gateway replacement with the client migration error, while the combined candidate passed. This identifies the client-persistence dependency; the exact-profile restart repair has its own regression evidence. Qualification and landing order is #3041, then #1982. Retained-state and foreign/revoked-authority proof requirements remain in force.
Why native testing exposed this
The source-backed investigation distinguishes two causes. Native k3d defaults to building a sibling OpenClaw checkout; the passing VM used
4c7a1654with upstream #174599’s separate client credential store. OCE’s defaultf8386bc7image stores this material in its already-persistent shared-state database. With the new runtime, retaining onlystate/losesclient/credentials.sqliteon replacement, and the runtime refuses that inconsistent state. Codex is not immune on the new layout; no same-image Codex replacement reproduction is claimed.The second fix is native-specific: only the dedicated native worker receives the generated worker-profile setting. Codex does not execute that branch. A native container restart can read its generated writable configuration and incorrectly reject the profile it just generated; exact equality now permits that case without admitting conflicting settings.
The new client store includes device private identities, role/Gateway-origin token caches, native execution socket credentials and migration receipts. These are sensitive contents, but equivalent local identity/token material was already persistent in the older shared-state layout. Server-side paired-device authority is distinct; source ownership does not substitute for revocation/replacement proof.
Change
clientsubdirectory at/home/node/.openclaw/clientbeside retained Gateway state. The client store and its migration receipt survive Pod replacement.Verification
Current head:
97ecaacb07449e8332da9fc3f910af0f692c8409. Merged maine8185dac805b737363c7178d3068b310a53a7c24without rewriting history. Resolved the native-admin flow conflict by retaining main’s session-expiry return flow and both histories; removed repeated authorization prose while preserving Manual Notes. The production repair and regression code are unchanged from06ae1e0e1. Refreshed focused conformance: 2 passed, 0 failed/skipped; full ESLint, repository-wide Prettier, workspace/flow/whitespace checks passed. Independent P0–P3 source review found no actionable findings. New-head hosted CI is pending; earlier results below stay tied to their recorded head.06ae1e0e13b7517abaa933911d367ac8c8fa2336. Hosted docs checks report 2,489 words for the flow and 2,494 for the storage reference; both pass the 2,500-word limit. Each remains one coherent existing workflow/reference, within the review band.ebab02221eb2ccd9d04775d4e8bd2980aeb3ce8band a pass for “runtime image validates the configuration dedicated native OpenClaw renders.” This executes both entrypoints, exact-profile Gateway restart and conflicting-device rejection in the selected image. Its Harness model probe is substituted in a credential-free, networkless container: this is startup/schema proof, not real model execution or existing-Agent replacement proof.06ae1e0e1and97ecaacb0; this is not an exact-head execution of the newer full tree. That earlier run did not qualify authority across a further replacement; the follow-up below now supplies that specific evidence. General cross-version/storage compatibility remains outside the proof. No general cross-version migration success is claimed.Gateway replacement authority proof
Additional fresh Linux/k3d qualification passed: 1 passed, 0 failed, 0 skipped. Replaced both Gateway Pods on their same PVCs after recording real worker credentials and revoking one node. Prior client identities and complete migration receipts survived; the valid peer continued an existing session and read its existing workspace. Retained revoked credentials and foreign-Agent credentials were causally rejected after replacement. The revoked workspace write did not land and the denied model nonce did not reach retained Harness state; the healthy peer completed another real turn. Both Harnesses stayed unchanged.
The comment includes the exact revisions, image digests, sanitized assertion transcript, and complete local test-only qualification patch. Tested combined source remains
e583b67b5(#1982 plus this PR's06ae1e0e1repair), using OpenClaw4c7a1654; this is not an exact-head run of97ecaacb0. It qualifies the exercised current-layout native flow, not general cross-version migration, every credential table, embedded/Codex execution, OpenShift, or all storage classes. Changed-file lint/format, syntax and whitespace checks passed. No additional production changes or authority bypasses were used.Risks
Persisting the client directory extends its contents’ lifetime across Gateway replacement. The authority-bearing contents and runtime-version difference are identified in the source investigation above. The additional real-runtime proof above now qualifies valid prior-state loading and rejection of revoked and foreign-Agent authority across Gateway replacement for the selected native flow, with exact source revisions and image digests. General cross-version migration, other credential categories and storage backends remain outside this evidence; review must assess those scope limits. The local startup regression alone does not establish these outcomes.
This PR keeps those proof obligations separate from #1982’s test-only native authority coverage. No merge or environment-policy change is requested by publication.
Contributor: @sallyom (original implementation). Extraction, documentation cleanup and verification by @roboclaw-bot.
Work sessions
View the OpenClaw team session