Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/full-integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -389,7 +389,7 @@ jobs:
runs-on: ubuntu-24.04
needs: preflight
if: inputs.lane == 'all' || inputs.lane == 'openshell'
timeout-minutes: 120
timeout-minutes: 160
environment: integration-openshell
env:
NODE_BASE_IMAGE: ${{ vars.CONTAINER_NODE_BASE_IMAGE }}
Expand All @@ -413,7 +413,7 @@ jobs:
profile: full
node-version: ${{ env.NODE_VERSION }}
artifact-prefix: full-results
job-timeout-minutes: 120
job-timeout-minutes: 160

k3d-otel:
name: k3d OTEL
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/qa-advisory.yml
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ jobs:
needs: eligibility
# OpenShell requires an enabled Landlock LSM, absent on the Blacksmith kernel.
runs-on: ubuntu-24.04
timeout-minutes: 120
timeout-minutes: 160
environment: integration-qa-pr
env:
NODE_BASE_IMAGE: ${{ vars.CONTAINER_NODE_BASE_IMAGE }}
Expand Down Expand Up @@ -163,4 +163,4 @@ jobs:
profile: full
node-version: "24.16.0"
artifact-prefix: qa-advisory
job-timeout-minutes: 120
job-timeout-minutes: 160
2 changes: 1 addition & 1 deletion docs/testing/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,7 @@ itself remains manual.
| Docker Model Runtime; k3d Model Runtime | [Docker](docker.md) and [Kubernetes](kubernetes.md#kubernetes-model-turns-and-secrets) real models, Pod replacement, Secret rotation and first-Agent reuse | [Docker](../../scripts/ci/test-suites/docker-model.json), [k3d](../../scripts/ci/test-suites/k3d-model.json) | No |
| Gateway Routing; Production TUI | [Envoy/OCC routing](kubernetes.md#private-gateway-routing), [Helm/TUI version cutover](kubernetes.md#production-tui-tests) | [Routing](../../scripts/ci/test-suites/gateway-routing.json), [TUI](../../scripts/ci/test-suites/production-tui.json) | No |
| Slack Live; Provider Account | [Live Slack isolation, issued credentials and model turns](slack.md) | [Slack](../../scripts/ci/test-suites/slack.json), [Account](../../scripts/ci/test-suites/provider-account.json) | No |
| OpenShell | [Sandbox enforcement and first-Agent flows](openshell.md) | [OpenShell](../../scripts/ci/test-suites/openshell.json) | With `full-qa` |
| OpenShell | [Sandbox enforcement, native authority, and first-Agent flows](openshell.md) | [OpenShell](../../scripts/ci/test-suites/openshell.json) | With `full-qa` |
| Local Kubernetes Installation | [Installation/recovery, including Keycloak](kubernetes.md#local-kubernetes-installation) | [Installation](../../scripts/ci/test-suites/dev-up-k3d.json) | With `full-qa` |
| Logging Collector | [Collector forwarding and recovery](local.md#metrics-and-observability-tests) | [Collector](../../scripts/ci/test-suites/logging-collector.json) | Yes: CI |
| k3d OTEL; Helper Timeout | [Real runtime OTLP logs](local.md#kubernetes-observability-acceptance); 300-second worker readiness deadline | [OTEL](../../scripts/ci/test-suites/k3d-otel.json), [Timeout](../../scripts/ci/test-suites/helper-timeout.json) | No |
Expand Down
23 changes: 23 additions & 0 deletions docs/testing/openshell-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,29 @@ network, and credential enforcement. Exact workload identity, approved mounts
and privileges, source withdrawal, replacement, and cleanup remain required.
Missing prerequisites fail rather than skip.

## Native authority

The native `--harness openclaw` case exercises production provider delivery. Its separate
`tests/integration/sandbox-driver-openshell-native-k3d-real.test.mjs` entry runs
in the registered OpenShell lane with `OPENAI_API_KEY`; the Codex entry cannot
substitute for it. It provisions independent Agents A and B using the same
model source. Each must read/write its workspace and complete fresh and retained
model sessions. Compute observes held startup failures through the temporary
provider exposure and withdraws that exposure once the node enrolls.

A-issued setup and signed-device authority must work at A and fail at B. An
unused setup expires on the real Gateway clock; a fresh positive control still
connects. Auth-only probes connect through loopback-forwarded `/node`; they do not
prove NetworkPolicy enforcement or substitute for the running Harness.

Revoking A's actual Compute-enrolled node must disconnect it, reject workspace
GET/PUT and stop a previously successful retained model session. B must remain
functional, with A's Gateway/Harness Pod identities and restart counts unchanged.
Transport errors and worker-capacity failures cannot stand in for revocation.
These checks require the selected upstream OpenClaw source to implement setup
issuance/status, signed device authentication and retained-worker revocation.
Source inspection and ordinary CI do not establish that this live case ran.

## Agent network policy

The policy scenario uses the production HTTP admission path, PostgreSQL worker,
Expand Down
17 changes: 11 additions & 6 deletions docs/testing/openshell.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,8 @@ The Codex account must permit `:danger-full-access`; OpenShell supplies the oute
sandbox. The live probe checks the effective profile before starting inference
and fails immediately if managed policy selects a restricted sandbox.

The lane includes both setup tests, the Sandbox test, and first-Agent tests for
both control-plane variants. It fails when selected prerequisites are missing.
The lane includes both setup tests, separate Codex and native authority Sandbox
entries, and first-Agent tests for both control-plane variants. It fails when selected prerequisites are missing.
Run cleanup even if preparation or a test fails:

```sh
Expand All @@ -130,8 +130,8 @@ node scripts/ci/cleanup.mjs --state "$OCC_OPENSHELL_TEST_ROOT/state.json"

The Codex Sandbox case and guided native OpenClaw first-Agent case use production
Drivers, but neither qualifies a production installation. The separate native
Sandbox case uses a test-only compatibility bridge. Use the k3d helper below to
run that case. The [Sandbox Driver reference](../reference/drivers/openshell-sandbox.md#qualification-contract)
Sandbox case also uses production Driver delivery. Use the k3d helper below to
run its [authority checks](openshell-coverage.md#native-authority). The [Sandbox Driver reference](../reference/drivers/openshell-sandbox.md#qualification-contract)
describes the remaining limits.

Codex also exercises Python deny/grant/allow/revoke, loaded-version confirmation
Expand All @@ -149,10 +149,14 @@ the native test and builds the sibling `../openclaw` checkout. Set
./scripts/k3d demo --harness openclaw
```

Both commands use the test-only bridge. `test` requires an authorized
Both commands use unmodified Compute requirements and production Driver delivery.
`test` requires an authorized
`OPENAI_API_KEY`; `demo` prompts for one if it is not set. `demo` leaves the
proven environment running and opens the Control UI and OCE console on loopback.
It leaves the dedicated worker slot free for the first browser session.
The native test reserves two worker slots and 6 GiB per Harness, with a 32 GiB
Namespace quota for two Agents and provider workloads. Genuine setup expiry
adds up to ten minutes; the test has a 40-minute cap.
Run `./scripts/k3d down --harness openclaw` to remove it. Codex and native OpenClaw
use separate helper-owned state; without `--harness`, `copy` selects the active
demo and `down` removes both environments.
Expand All @@ -179,7 +183,8 @@ directory when finished.
and Kubernetes-only first-Agent stacks. It first proves Codex with a `codex-pat`
source, then creates a separate `openai` source from `OPENAI_API_KEY` for guided
native OpenClaw provisioning. The native case uses the pinned runtime and
production Drivers without the k3d helper's compatibility bridge.
production Drivers. The separate k3d Sandbox case covers cross-Agent authority
and node revocation rather than session continuity across replacement.

The case edits `USER.md` through the workspace API and verifies its marker through
real model execution and `exec`. That turn writes a unique file in its session
Expand Down
4 changes: 4 additions & 0 deletions scripts/ci/test-suites/openshell.json
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,10 @@
"path": "tests/integration/sandbox-driver-openshell-k3d-real.test.mjs",
"expectedTests": ["OpenShell enforces provider delivery and authenticated Codex exposure"]
},
{
"path": "tests/integration/sandbox-driver-openshell-native-k3d-real.test.mjs",
"expectedTests": ["OpenShell enforces native provider delivery and Gateway authority"]
},
{
"path": "tests/integration/local-first-agent-openshell-real.test.mjs",
"expectedTests": [
Expand Down
115 changes: 115 additions & 0 deletions tests/helpers/native-gateway-authority.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
import assert from "node:assert/strict";
import { createHash, createPublicKey, generateKeyPairSync, sign } from "node:crypto";
import { createRequire } from "node:module";
import { pathToFileURL } from "node:url";

// Resolve the same published client as the controller, not a second protocol implementation.
const controllerRequire = createRequire(
new URL("../../apps/controller/package.json", import.meta.url),
);

export function createProbeDeviceIdentity() {
const { publicKey, privateKey } = generateKeyPairSync("ed25519");
const publicKeyPem = publicKey.export({ type: "spki", format: "pem" }).toString();
return {
deviceId: createHash("sha256").update(rawPublicKey(publicKeyPem)).digest("hex"),
publicKeyPem,
privateKeyPem: privateKey.export({ type: "pkcs8", format: "pem" }).toString(),
};
}

function rawPublicKey(pem) {
const der = createPublicKey(pem).export({ type: "spki", format: "der" });
const prefix = Buffer.from("302a300506032b6570032100", "hex");
assert.equal(der.length, prefix.length + 32);
assert.ok(der.subarray(0, prefix.length).equals(prefix));
return der.subarray(prefix.length);
}

// No persistent device store: every probe explicitly supplies its authority. The real
// Gateway validates the client's signed connect challenge and issues the device token.
export async function connectNativeGateway(options) {
const { GatewayClient } = await import(
pathToFileURL(controllerRequire.resolve("@openclaw/gateway-client")).href
);
let accept;
let refuse;
const hello = new Promise((resolve, reject) => {
accept = resolve;
refuse = reject;
});
const client = new GatewayClient({
...options,
minProtocol: 4,
maxProtocol: 4,
hostDeps: {
signDevicePayload: (key, payload) =>
sign(null, Buffer.from(payload), key).toString("base64url"),
publicKeyRawBase64UrlFromPem: (key) => rawPublicKey(key).toString("base64url"),
loadDeviceAuthToken: () => null,
storeDeviceAuthToken: () => {},
clearDeviceAuthToken: () => {},
// Handshake failures are reported below as codes only, never raw transport errors.
logDebug: () => {},
logError: () => {},
},
onHelloOk: accept,
onConnectError: refuse,
});
const timer = setTimeout(
() => refuse(new Error("Gateway authority handshake timed out")),
15_000,
);
const close = async () => {
client.stop();
await client.stopAndWait({ timeoutMs: 1_000 });
};
try {
client.start();
const connected = await hello;
return { client, hello: connected, close };
} catch (error) {
await close();
// Do not retain the raw error/cause: URLs, setup bearers and device tokens are private.
const rejected = new Error("Native Gateway rejected the authority handshake");
rejected.code = error?.code;
rejected.detailCode = error?.details?.code;
throw rejected;
} finally {
clearTimeout(timer);
}
}

export async function expectNativeAuthorityDenied(options, detailCode) {
let admitted;
let failure;
try {
admitted = await connectNativeGateway(options);
} catch (error) {
failure = error;
} finally {
await admitted?.close();
}
assert.ok(failure, "the other Agent or expired authority must not connect");
assert.equal(failure.code, "INVALID_REQUEST");
assert.equal(failure.detailCode, detailCode, "require the Gateway's causal auth rejection");
return { code: failure.code, detailCode: failure.detailCode };
}

export function decodeNativeSetup(setup) {
assert.equal(setup.access, "node");
assert.ok(typeof setup.setupId === "string" && setup.setupId.length > 0);
const encoded = setup.setupCode.toLowerCase().startsWith("oc-pair://")
? setup.setupCode.slice("oc-pair://".length)
: setup.setupCode;
let envelope;
try {
envelope = JSON.parse(Buffer.from(encoded, "base64url").toString("utf8"));
} catch {
throw new Error("The Gateway issued an invalid native setup envelope");
}
assert.equal(envelope.expiresAtMs, setup.expiresAtMs);
assert.ok(typeof envelope.bootstrapToken === "string" && envelope.bootstrapToken.length > 0);
assert.ok(Number.isSafeInteger(setup.expiresAtMs) && setup.expiresAtMs > Date.now());
return envelope;
}
27 changes: 12 additions & 15 deletions tests/helpers/openshell-kubernetes-real.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -346,14 +346,7 @@ export function createOpenShellInstallationConfiguration({
"limits.cpu": "16",
"limits.memory": "8Gi",
};
configuration.drivers.compute.configuration.servicePrincipalCredentials =
process.env.OCC_TEST_OPENSHELL_HARNESS === "openclaw"
? {
mode: "projectedServiceAccountToken",
audience: "openclaw-enterprise",
expirationSeconds: 3600,
}
: { mode: "disabled" };
configuration.drivers.compute.configuration.servicePrincipalCredentials = { mode: "disabled" };
// The integration replaces this placeholder transport with each namespace's port-forward.
configuration.backend = [
{
Expand Down Expand Up @@ -1060,11 +1053,17 @@ export function createOpenShellKubernetesFixture({
return pod;
}

async function assertProviderOwnedHarness(namespace, revision, sandbox, pod) {
async function assertProviderOwnedHarness(
namespace,
revision,
sandbox,
pod,
serviceAccountName = openShellAgentName(revision.agentId),
) {
assert.equal(sandbox.metadata.namespace, namespace);
assert.equal(pod.metadata.namespace, namespace);
assert.ok(sandbox.metadata.name, "Sandbox must have a stable resource name.");
assert.equal(pod.spec.serviceAccountName, openShellAgentName(revision.agentId));
assert.equal(pod.spec.serviceAccountName, serviceAccountName);
assert.equal(pod.metadata.labels?.["openclaw.dev/namespace"], revision.namespaceId);
assert.equal(pod.metadata.labels?.["openclaw.dev/agent"], revision.agentId);
assert.equal(pod.metadata.labels?.["openclaw.dev/revision"], revision.id);
Expand Down Expand Up @@ -1099,7 +1098,7 @@ export function createOpenShellKubernetesFixture({
assert.match(
environment.find(({ name }) => name === "APP_TOKEN_SHA")?.value ?? "",
/^[a-f0-9]{64}$/,
"the OpenShell Harness requires only its app-server token verifier.",
"the OpenShell Harness requires its transport/status token verifier.",
);
return container;
}
Expand Down Expand Up @@ -1225,7 +1224,7 @@ export function createOpenShellKubernetesFixture({
assert.equal(claims.sub, `system:serviceaccount:${namespace}:${pod.spec.serviceAccountName}`);
}

function assertApprovedOpenShellPrivileges(pod, { compatibilityBridge = false } = {}) {
function assertApprovedOpenShellPrivileges(pod) {
assert.equal(
pod.spec.runtimeClassName,
openShellRuntimeClass,
Expand All @@ -1249,9 +1248,7 @@ export function createOpenShellKubernetesFixture({
undefined,
"OpenShell v0.1.4 must keep its network supervisor outside the workload Pod.",
);
const container = compatibilityBridge
? pod.spec.containers.find(({ name }) => name === "agent")
: harnessContainer(pod);
const container = harnessContainer(pod);
assert.ok(container, "the provider-owned Pod must contain its Agent container.");
assert.equal(container.securityContext?.allowPrivilegeEscalation, false);
assert.deepEqual(container.securityContext?.capabilities?.drop, ["ALL"]);
Expand Down
53 changes: 53 additions & 0 deletions tests/integration/ci-openshell.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,30 @@ test("OpenShell host probes retain the exposed hostname while connecting to loop
assert.equal(observedHost, `${serviceHostname}:${address.port}`);
});

test("OpenShell privilege checks require the Harness status verifier", () => {
const { assertApprovedOpenShellPrivileges: check } = createOpenShellKubernetesFixture({});
const pod = {
spec: {
runtimeClassName: "openshell-sandbox",
securityContext: { runAsUser: 10001, runAsGroup: 10001 },
containers: [
{
name: "agent",
env: [{ name: "APP_TOKEN_SHA", value: "a".repeat(64) }],
securityContext: { allowPrivilegeEscalation: false, capabilities: { drop: ["ALL"] } },
},
],
},
};
// Native held-failure status and Codex transport both require a verifier,
// while the raw token must stay outside the Harness Pod spec.
assert.doesNotThrow(() => check(pod));
pod.spec.containers[0].env = [];
assert.throws(() => check(pod), /transport\/status token verifier/);
pod.spec.containers[0].env = [{ name: "APP_SERVER_TOKEN", value: "synthetic-token" }];
assert.throws(() => check(pod), /raw app-server token must stay outside/);
});

test("OpenShell probe observes its command after the turn ends, or preserves the unknown outcome", async (t) => {
// Run the actual in-Pod client and probe decision. Only kubectl transport and the
// app-server peer are fixtures. WebSocket message order, not a sleep, creates the boundary.
Expand Down Expand Up @@ -536,6 +560,35 @@ test("OpenShell probe waits for a command that outlives its turn up to the fixtu
}
});

test("OpenShell fixture disables ServicePrincipal credentials for both Harness selections", () => {
const previous = process.env.OCC_TEST_OPENSHELL_HARNESS;
try {
for (const harness of ["codex", "openclaw"]) {
process.env.OCC_TEST_OPENSHELL_HARNESS = harness;
const configuration = createOpenShellInstallationConfiguration({
authentication: {},
platformNamespace: "openclaw-system",
gatewayImage: "gateway-fixture",
codexImage: "codex-fixture",
cluster: { name: "fixture" },
});
// The real Driver rejects projected identity. Every fixture caller must
// receive the supported mode without patching Compute requirements later.
assert.deepEqual(
configuration.drivers.compute.configuration.servicePrincipalCredentials,
{ mode: "disabled" },
harness,
);
}
} finally {
if (previous === undefined) {
delete process.env.OCC_TEST_OPENSHELL_HARNESS;
} else {
process.env.OCC_TEST_OPENSHELL_HARNESS = previous;
}
}
});

test("a turn the model answered at capacity runs again only for its own final error before any action", async (t) => {
// Finding 1946. The actual in-Pod client, kubectl error path and retry; only kubectl
// transport, the app-server peer and the retry's sleep are fixtures. Message order, not
Expand Down
Loading
Loading