Skip to content

ci: pin the CTS and ZeroKMS hosts from repo variables - #1199

Closed
tobyhede wants to merge 1 commit into
mainfrom
ci/pin-cts-zerokms-hosts
Closed

tobyhede wants to merge 1 commit into
mainfrom
ci/pin-cts-zerokms-hosts

Conversation

@tobyhede

Copy link
Copy Markdown
Contributor

Summary

Every CI job that talks to the live CipherStash service has failed on authentication since the CI workspace moved to crn:us-east-1.aws:EXRSSGWBYTIXCXWK, with 401 Unauthorized: Authorization failed: Unknown credential id=Some("ZHWW54P57PJOIP6E"). The access key is valid, but it was issued by the ap-southeast-2 CTS (CipherStash's token service). With no CS_CTS_HOST set, the client works out the CTS host from the CRN's region and asks the us-east-1 CTS, which has never seen the key. This PR passes both service hosts explicitly from new repo variables, the same way proxy CI does.

Changes

  • Workflows: every job that passes vars.CS_WORKSPACE_CRN now also passes CS_CTS_HOST: ${{ vars.CS_CTS_HOST }} and CS_ZEROKMS_HOST: ${{ vars.CS_ZEROKMS_HOST }}, in the same form the job already uses for the CRN (a job env: map, a step env: map, or a .env file written by a run: step).
  • Go live tests (tests-golang.yml): also pass STACK_ENCRYPT_TEST_CTS_HOST and STACK_ENCRYPT_TEST_ZEROKMS_URL. TestLiveAutoCredentialsFromTheEnvironment clears CS_CTS_HOST and only restores it from those names.
  • tests.yml e2e-tests: the hardcoded ap-southeast-2 pair is replaced. Its ZeroKMS half pointed at the wrong region for this workspace.
  • integration-protect-ffi.yml: the comment explaining why the hosts were deliberately not set is rewritten, because this PR reverses that decision.
  • New guard, scripts/__tests__/cs-hosts-pinned.test.mjs: fails if a job carries the CRN without both host variables, or if any workflow hardcodes a CTS or ZeroKMS URL.

Verification

  • pnpm exec vitest run --config scripts/vitest.config.mjs: 76 files, 1368 tests passed (29 skipped).
  • Ran the new guard with the workflow changes stashed: it failed 15 jobs. With the changes: all pass.
  • The repo variables are set: CS_CTS_HOST=https://ap-southeast-2.aws.cts.cipherstashmanaged.net, CS_ZEROKMS_HOST=https://us-east-1.aws.zerokms.cipherstashmanaged.net.
  • Not yet verified against the live service. This PR's own credentialed jobs are the first run with the pins.

Review notes

Merging this does not finish the workspace move. Still outstanding, outside this PR:

  • The Dependabot secret store's CLERK_MACHINE_TOKEN and CLERK_MACHINE_TOKEN_B (dated 2026-10-12) are rejected by Clerk, so lock-context suites fail on every Dependabot PR. They need the Actions-store values.
  • The Clerk instance must be registered on the new workspace's OIDC-providers page, or the lock-context suites fail on every branch.
  • The EQL SQLx constants SELECTOR (tests/sqlx/src/fixtures/v3_doc_integer.rs) and SEL_HELLO_OP (tests/sqlx/src/fixtures/v3_ste_vec.rs) are keyed to the workspace keyset and need re-pinning from the candidates their drift guards print.
  • Open question for the CTS owners: if every us-east-1 workspace authenticates through the ap-southeast-2 CTS, then the CRN-region discovery in cts-common's CtsServiceDiscovery::endpoint gives customers the wrong host too.

No changeset: CI configuration and a repo-tooling test only.

The CI workspace moved to crn:us-east-1.aws:EXRSSGWBYTIXCXWK. Its access
key was issued by the ap-southeast-2 CTS, but with no CS_CTS_HOST the
client derives the CTS host from the CRN's region and asks the us-east-1
CTS, which answers "401 Unknown credential" for a valid key. Every
credentialed job has failed on auth since the move.

Every job that passes vars.CS_WORKSPACE_CRN now also passes
vars.CS_CTS_HOST and vars.CS_ZEROKMS_HOST, the same way proxy CI pins
them. The Go live suite gets them as STACK_ENCRYPT_TEST_CTS_HOST and
STACK_ENCRYPT_TEST_ZEROKMS_URL, because its cleanEnv clears CS_CTS_HOST.
The e2e-tests job's hardcoded ap-southeast-2 pair is replaced: its
ZeroKMS half pointed at the wrong region for this workspace.

scripts/__tests__/cs-hosts-pinned.test.mjs fails on a job that carries
the CRN without both host variables, and on any literal host URL.
@changeset-bot

changeset-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 8da5170

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@tobyhede tobyhede closed this Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant