Skip to content

test(secrets): pin Secret and credential-source checks that no test detected - #1401

Merged
freeqaz merged 2 commits into
mainfrom
test-gaps-4-sec
Oct 5, 2026
Merged

freeqaz merged 2 commits into
mainfrom
test-gaps-4-sec

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

Problem

A mutation sweep of the Secret and credential-source paths (105 hand-written mutants over OCC
core, PostgreSQL and in-memory State, the HTTP handlers, the Kubernetes Secret Driver, the
OpenShell Credential Gateway, the worker's runtime materialization and the repository credential
broker) found checks that no test detects. Each mutant ran against every non-PG file that reaches
the mutated file (95) and the 22 postgres-platform files plus postgres-worker-agent-revision,
each PG file on a fresh clone of its prepared database. 58 of 105 were killed.

Change

Test-only. Each new assertion kills one or more surviving mutants:

Mutant Check removed or weakened Killed by
S07, S08 core Secret value check (NUL, lone surrogate, 65,536-byte limit) on create and update secret-occ
S10 backend copy removed when the Secret's metadata does not commit secret-occ
S11 an update refuses a Secret owned by another Secret Driver secret-occ
B11 Harness Secret backend identity drift fails deploy secret-occ
C10 a deletion that wins against a registration removes the late gateway copy; no audit credential-source-occ
C12 only a ready credential source can be updated credential-source-occ
C14, C15 core read authorization; per-source list filtering credential-source-occ
C21 a failing gateway status read reports a fixed reason, not the gateway's error text credential-source-occ
K03, K05 Kubernetes Secret: foreign Namespace annotation; record owned by another Secret Driver kubernetes-secret-driver
P01, P10 Namespace-scoped findSecret/lockSecret and binding availability for a foreign Secret ID postgres-secret-state
P03 an Agent's Harness binding retains its Secret (hasReferences) postgres-secret-state
H06, H07, H08 credential source create, update and delete each commit one value-free audit row over HTTP secret-api

The PG assertions extend existing tests, so scripts/ci/test-suites is unchanged.

Kills by area, before -> after: Secret core 8 -> 12/14, HTTP 4 -> 7/9, bindings and Harness
auth 10 -> 11/16, credential sources 16 -> 21/21, State 8 -> 11/12, Kubernetes Secret Driver
6 -> 8/9. Total 58 -> 76 of 105. The OpenShell gateway, worker projection and broker
survivors are triaged separately.

Verification

Risks

Test-only. The credential-source race test drives the deletion from inside the test gateway's
registerSource, so it is deterministic. The HTTP test uses an injected clock to pass the
registration fence. The recording gateway in secret-api duplicates part of the conformance
gateway; moving both into a shared helper is left for later, because #913 and #851 edit that
fixture.

…etected

Mutation sweep of Secret and credential-source paths (test-gaps-4). Each new
assertion kills a mutant that survived every relevant non-PG and PG file:

- secret-occ: core Secret value checks on create and update (NUL, lone
  surrogate, byte limit) before the driver sees the value; a duplicate-name
  commit failure removes the backend copy; an update refuses a Secret owned by
  another Secret Driver; Harness Secret backend identity drift fails deploy.
- credential-source-occ: a deletion that wins against a registration removes
  the late gateway copy and commits no audit; only a ready source updates;
  core read authorization and per-source list filtering; a failing gateway
  status read reports a fixed reason.
- kubernetes-secret-driver: foreign Namespace annotation and a record owned by
  another Secret Driver are refused.
- postgres-secret-state: Namespace-scoped Secret lookups and binding checks
  for a foreign Secret ID; an Agent's Harness binding retains its Secret.
- secret-api: credential source create, update and delete each commit one
  value-free audit row.
Past-the-fence clock step matches the registration fence (71 s), comments sit
over the assertions they describe, the race test checks committed audit once,
and the deleting-source copy check is labelled as a precondition.
@freeqaz
freeqaz requested a review from a team as a code owner October 5, 2026 17:13
@clawsweeper

clawsweeper Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P3 Low-risk cleanup, docs, polish, ergonomics, or speculative feature. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Oct 5, 2026
@clawsweeper

clawsweeper Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed October 5, 2026, 1:16 PM ET / 17:16 UTC.

ClawSweeper review

What this changes

Adds regression tests for Secret validation, ownership, cleanup, tenant-scoped storage, credential-source authorization, and value-free HTTP audit records.

Merge readiness

✅ Ready for maintainer review

The added checks protect distinct gaps in existing coverage; no blocking patch defect was found. Repository policy keeps this PR open for maintainer triage.

Likely related people: freeqaz, mrunalp, and kevinlin-openai are medium-confidence routing candidates based on prior main-branch work.

Priority: P3
Reviewed head: d2a03df7f14938fb4d60fd871fdcf896b6993443

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) Focused contract coverage with credible regression triggers and no blocking finding; execution results remain author-reported.
Proof confidence 🌊 off-meta tidepool Not applicable: Repository State designates the author MEMBER, so ordinary external-contributor proof is exempt. The test-only patch changes no production authority or stored-data contract; reported PostgreSQL runs supplement source inspection, while gateway and Kubernetes doubles prove only their stated contract scope.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: Repository State designates the author MEMBER, so ordinary external-contributor proof is exempt. The test-only patch changes no production authority or stored-data contract; reported PostgreSQL runs supplement source inspection, while gateway and Kubernetes doubles prove only their stated contract scope.
Evidence reviewed 8 items Verified introduced scope: The pinned main-to-head delta changes only five test files, with 406 added and 11 removed lines. Raw test-merge parents match the pinned main and original head; its result has the same test-only scope.
Controller owns the tested decisions: Existing production code validates Secret values before Driver calls, registers rollback cleanup, refuses mismatched Driver selection, handles registration/deletion races, restricts updates to ready sources, filters reads through IAM, and sanitizes gateway status failures. The added doubles do not replace these decisions.
Real storage boundary: The shared exercise runs through both memory and PostgreSQL adapters. Added foreign-Secret checks exercise namespace-scoped lookup and binding availability, while the Harness-reference assertion occurs before revision creation. PostgreSQL queries implement these invariants rather than reproducing them in a fixture.
Findings None None.
Security None None.

How this fits together

OpenClaw Control Plane authorizes Secret and credential-source requests, stores their metadata, and delegates credential custody to selected Drivers. These resources supply credentials to Agent runtimes while keeping values out of public responses and audit records.

flowchart TB
  A[Authenticated requests] --> B[Control Plane authorization]
  B --> C[Secret and source lifecycle]
  C --> D[Metadata and audit storage]
  C --> E[Secret Driver]
  E --> F[Credential Gateway]
  F --> G[Agent runtime credentials]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production versus test LOC production +0/-0; tests +406/-11, net +395 The additional coverage introduces no production code or public API surface.

Technical review

Best possible solution:

Retain focused regression coverage at the existing controller, Driver, HTTP, and storage boundaries without adding production seams.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR strengthens existing tests rather than fixing a reported runtime failure; mutant failures and repeated successful runs are author-reported.

Is this the best way to solve the issue?

Yes: the assertions exercise existing behavior owners and extend shared storage coverage without adding test-only production interfaces.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 029c3bb67505.

Labels

Label changes:

  • add P3: This is test-only maintenance protecting established behavior without changing runtime semantics.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: Repository State designates the author MEMBER, so ordinary external-contributor proof is exempt. The test-only patch changes no production authority or stored-data contract; reported PostgreSQL runs supplement source inspection, while gateway and Kubernetes doubles prove only their stated contract scope.

Label justifications:

  • P3: This is test-only maintenance protecting established behavior without changing runtime semantics.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: Repository State designates the author MEMBER, so ordinary external-contributor proof is exempt. The test-only patch changes no production authority or stored-data contract; reported PostgreSQL runs supplement source inspection, while gateway and Kubernetes doubles prove only their stated contract scope.

Evidence

What I checked:

  • Verified introduced scope: The pinned main-to-head delta changes only five test files, with 406 added and 11 removed lines. Raw test-merge parents match the pinned main and original head; its result has the same test-only scope. (d2a03df7f149)
  • Controller owns the tested decisions: Existing production code validates Secret values before Driver calls, registers rollback cleanup, refuses mismatched Driver selection, handles registration/deletion races, restricts updates to ready sources, filters reads through IAM, and sanitizes gateway status failures. The added doubles do not replace these decisions. (packages/occ/src/index.ts:3452, 029c3bb67505)
  • Real storage boundary: The shared exercise runs through both memory and PostgreSQL adapters. Added foreign-Secret checks exercise namespace-scoped lookup and binding availability, while the Harness-reference assertion occurs before revision creation. PostgreSQL queries implement these invariants rather than reproducing them in a fixture. (tests/integration/postgres-secret-state.test.mjs:257, d2a03df7f149)
  • HTTP audit coverage: The new case uses the existing controller application, authenticated requests, native IAM, and actual audit sink. Its gateway records credential copies; production handlers commit create, update, and delete audit events. Advancing the controller clock by 71 seconds respects the documented 70-second registration fence. (tests/integration/secret-api.test.mjs:1162, d2a03df7f149)
  • Existing Kubernetes ownership contract: The added adversarial inputs exercise the actual Kubernetes Secret Driver's record Driver-ID check and namespace annotation check through resolve and transient value access. The fake Kubernetes client supplies backend records, so these cases establish Driver contract coverage rather than live-cluster proof. (apps/controller/src/drivers/secret/kubernetes/index.ts:334, 029c3bb67505)
  • Documented behavior and existing CI selection: Current references define exact-resource read filtering, ready-only updates, sanitized status failures, and deletion fencing. Existing baseline and PostgreSQL suite manifests already select every touched file; no test-selection change is necessary. (docs/reference/credential-sources.md:61, 029c3bb67505)

Likely related people:

  • freeqaz: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • mrunalp: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • kevinlin-openai: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@freeqaz
freeqaz merged commit d48e3b3 into main Oct 5, 2026
25 checks passed
@freeqaz
freeqaz deleted the test-gaps-4-sec branch October 5, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P3 Low-risk cleanup, docs, polish, ergonomics, or speculative feature. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant