Skip to content

fix(test): make the k3d-model lane report only tests that run - #3308

Merged
freeqaz merged 1 commit into
mainfrom
fix-k3d-model-suite
Oct 23, 2026
Merged

freeqaz merged 1 commit into
mainfrom
fix-k3d-model-suite

Conversation

@freeqaz

@freeqaz freeqaz commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The k3d-model lane reports two issues on every run, even when every case passes, which hides real failures:

  • missing-expected-test: scripts/ci/test-suites/k3d-model.json expects "production dedicated OpenClaw runs worker-local inference and tools in an enrolled Harness pod". No test has ever had that name. 10af5b8 (dedicated native OpenClaw harness) added the expectation and a helper branch (arrangeProductionTopology harnessId, assertNativeWorkerTurn) but no test. No test passes harnessId: "openclaw", so dedicated native OpenClaw on Kubernetes Compute has no real-cluster test. The openshell lane covers native OpenClaw only through the OpenShell sandbox driver. That gap is tracked separately; this PR only stops the lane from reporting a test that does not exist.
  • unexpected-skip: "candidate dedicated Skill source uploads honor default and denied node-write policy" skips unless OCC_TEST_SKILL_SOURCE_LIFECYCLE=1, which the lane never sets, and the runner no longer accepts declared skips. fix: authorize dedicated Skill lifecycle storage paths #843 added the gate for candidate images with paired-node Skill uploads. build(runtime): adopt paired-node Skill lifecycle and Codex 0.160.0 #1414 put that support in the default runtime three days later, so the gate only hides the test. It is the only real-cluster coverage of Skill uploads under default and denied node-write policy.

Change

  • Remove the expectation that no test backs.
  • Select the Skill source case with requiresProductionCluster, like the other production cases. Rename it "production dedicated Skill source uploads ..." and list it in the manifest.
  • The topology teardown conformance test no longer sets the removed env. It still requires all six tests in the file to reach arrangeProductionTopology.
  • docs/testing/kubernetes.md: count six non-Slack cases, note that the Skill case needs local zip, and drop the opt-in recipe.

Verification

  • node scripts/ci/run-tests.mjs audit: passed, 0 issues.
  • A scan of every suite manifest found no other expected test name that the file cannot produce. Template literals, + concatenation and re-exported files are resolved. Every other env-gated skip in a mapped file is set by the lane's env, requiredEnv or scripts/ci/prepare.mjs. So this was the only mismatch of either kind.
  • Conformance every production topology test stops its worker before the platform Namespace delete: passes 20/20 at 2 CPUs. With the old gate restored and no env set, it fails ("arranged a topology in every test"), so it now guards the case.
  • Running harness-topology-k3d-real.test.mjs unselected skips all six cases with the standard prerequisites message.
  • The lane itself needs model keys and was not dispatched for this branch.

Risks

  • The Skill source case now runs in k3d-model. It has not run in CI since fix: authorize dedicated Skill lifecycle storage paths #843, so the next lane run may expose a real regression that the skip was hiding. That is the point of the change. It adds one dedicated topology and four deploys (enable, deny, restore, original), bounded at 20 minutes. The file's per-test timeouts now sum to 85 minutes, up from 65, against the lane's 90-minute runner budget. Real runtimes are far shorter, but a run where every case is slow could hit the runner deadline before a test's own timeout fires.
  • The case shells out to zip on the runner. ubuntu-24.04 hosted runners include it.
  • Draft Draft implementation: managed outbound HTTPS for dedicated Gateways #2518 edits the adjacent lines of k3d-model.json and keeps the dead expectation. Whichever lands second needs a trivial merge.

The k3d-model manifest expected "production dedicated OpenClaw runs
worker-local inference and tools in an enrolled Harness pod", which no
test ever defined: 10af5b8 added the expectation without the test, and
dedicated native OpenClaw is proven by the openshell lane. Drop it.

The Skill source upload case skipped unless
OCC_TEST_SKILL_SOURCE_LIFECYCLE=1. #843 gated it for candidate images;
#1414 put paired-node Skill uploads in the default runtime three days
later, so the gate only made the lane report an unexpected skip. Select it
like the other production cases, drop "candidate" from its name, and
list it in the manifest. The teardown conformance test no longer needs
the env to reach it, and the Kubernetes testing guide counts six cases.

Findings 1757 and 1758.
@clawsweeper

clawsweeper Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

@freeqaz
freeqaz requested a review from a team as a code owner October 23, 2026 13:19
@freeqaz
freeqaz merged commit b5dd6eb into main Oct 23, 2026
46 checks passed
@freeqaz
freeqaz deleted the fix-k3d-model-suite branch October 23, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant