From 186eaa4a2c37ca40c0f2e53684db15175405166a Mon Sep 17 00:00:00 2001 From: Felix Kotschenreuther Date: Mon, 21 Sep 2026 10:02:56 +0200 Subject: [PATCH 1/2] feat: host-portable planning, a credential helper, and tofu-aware id resolution MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four changes the tier-0 OpenTofu cutover needs, and one that makes a single config usable across an estate whose instances differ. **#178 — a right this host does not have warns instead of aborting.** A declared right absent from the active permission catalog took down planning for everything else, so one config could not serve two instances with different modules installed (13 of 113 rights do not exist on eqrm-dev). It is now skipped with a warning naming the right, the declaration and both catalog versions — never granted, never revoked — while everything else still plans. A name no catalog ct knows defines stays a hard error, which is the case that guard was written for; ct's bundled catalog is the oracle that tells the two apart, so the verdict needs a per-instance capture to be active. `--strict-catalog` on plan and apply restores the old behaviour. Same split for a `preserveUnknown` dimension. **#179 — `ct auth token`, a credential helper.** Emits the ChurchTools SESSION rather than the personal login token: it expires, `ct auth logout` kills it, and a leaked copy dies in hours instead of being a permanent admin credential. The credential goes to stdout and nothing else does, a failure writes nothing there, and a terminal is refused without `--allow-tty`. A cross-process brake bounds login handshakes per host (3s spacing, 20/hour) so calling this on every tofu run cannot burst into ChurchTools' login rate limit. **#180 — a reference is not a declaration.** `ct state rm` refused a key the config only referenced, which made `--force` mandatory for 49 of 50 tier-0 entries and suppressed the check for the case it is meant to catch. Declaredness is now decided from the resources the config actually declares, matched on type and key. References are reported, not refused — except a group, which is managed-only with no live catalog to fall back to, and is refused in its own words. **#181 — resolving what OpenTofu now owns.** Once tier-0 leaves ct's state, the references left behind fall back to matching keys against live names, which cannot work for keys that were never name-derived (`status_unbekannt` vs `Unbekannt`), and numeric ids are not portable (39 of 43 differ between hosts). ct now resolves through a committed `.ct/ids..json` — after its own state, before the live catalog — written by `ct export tf` and refreshed by `ct ids sync --tofu-state -` from a `tofu state pull`, so no S3 client enters ct. The map is host-checked on load, and a sync never replaces a populated map with an empty one. Closes #178, #179, #180, #181 --- README.md | 20 +- docs/README.md | 1 + docs/handbuch/blueprints.md | 2 +- docs/handbuch/field-definitions.md | 2 +- docs/handbuch/group-member-fields.md | 2 +- docs/handbuch/permissions.md | 59 ++++- docs/opentofu-migration.md | 171 ++++++++++++++ src/api/ctClient.ts | 58 ++++- src/api/session.ts | 8 +- src/application/contracts.ts | 2 +- src/application/operations/auth.ts | 104 ++++++++- src/application/operations/export-tf.ts | 30 ++- src/application/operations/ids.ts | 144 ++++++++++++ src/application/operations/plan.ts | 25 +- src/application/operations/state.ts | 107 +++++++-- src/auth/loginThrottle.ts | 117 ++++++++++ src/commands/apply.ts | 6 + src/commands/auth.ts | 39 +++- src/commands/export-tf.ts | 8 + src/commands/ids.ts | 87 +++++++ src/commands/plan.ts | 11 + src/config/context.ts | 8 +- src/index.ts | 2 + src/permissions/catalog.ts | 83 +++++++ src/permissions/plan.ts | 84 ++++++- src/resolve/idMap.ts | 185 +++++++++++++++ src/resolve/resolver.ts | 45 ++++ src/resolve/tfstate.ts | 84 +++++++ tests/auth-token-command.test.ts | 131 +++++++++++ tests/export/export-tf.test.ts | 39 ++++ tests/ids-sync.test.ts | 162 +++++++++++++ tests/login-throttle.test.ts | 120 ++++++++++ ...permission-catalog-host-difference.test.ts | 221 ++++++++++++++++++ tests/state-rm-command.test.ts | 61 ++++- tests/tofu-id-map.test.ts | 204 ++++++++++++++++ 35 files changed, 2396 insertions(+), 36 deletions(-) create mode 100644 docs/opentofu-migration.md create mode 100644 src/application/operations/ids.ts create mode 100644 src/auth/loginThrottle.ts create mode 100644 src/commands/ids.ts create mode 100644 src/resolve/idMap.ts create mode 100644 src/resolve/tfstate.ts create mode 100644 tests/auth-token-command.test.ts create mode 100644 tests/ids-sync.test.ts create mode 100644 tests/login-throttle.test.ts create mode 100644 tests/permission-catalog-host-difference.test.ts create mode 100644 tests/tofu-id-map.test.ts diff --git a/README.md b/README.md index 6ce00bf..19de06e 100644 --- a/README.md +++ b/README.md @@ -237,6 +237,10 @@ ct plan # diff the config against ChurchTools (read-only) ct plan --format markdown # plain-language review report (German by default) ct apply # create + update in dependency order (confirm + backup first) ct refresh --group # make ChurchTools re-evaluate one auto-group now + +ct export tf # render managed state as OpenTofu HCL + import blocks +ct ids sync --tofu-state - # refresh the key->id map ct resolves tofu-owned resources through +ct auth token --env dev # hand a short-lived session to another tool (credential helper) ``` Run without `--token`, `ct auth login` asks how you want to authenticate: **username and @@ -250,12 +254,19 @@ export `CT_HOST` and `CT_LOGINTOKEN` there. `state rm` is the inverse of `adopt`, and only of `adopt`: it removes the entry from the state file, makes no HTTP call, and leaves the resource in place in -ChurchTools, now unmanaged. It refuses a key the config still declares — that +ChurchTools, now unmanaged. It refuses a key the config still **declares** — that would make the next plan propose creating a resource that already exists — so delete the declaration first, or pass `--force` to do both in one change. -"Declares" covers permission declarations too, not only resources: a key named -by a `ct.groupRole` domain or a group scope is just as broken to remove, and the -refusal is what keeps that from surfacing one command later as a plan error. + +A key the config only **references** is a different matter and is removed: a +logical reference asks the host for an id, so it never proposes creating the +resource it points at. `state rm` says how many references stay behind, because +they now resolve against the live instance by name (or through the +[id map](docs/opentofu-migration.md)) rather than from state. The one reference +that still blocks removal is a **group**: groups are managed-only, with no live +catalog to fall back to, so dropping a group a `ct.groupRole` domain or a group +scope names would make the next plan fail to resolve it — the refusal is what +keeps that from surfacing one command later. `apply` reconciles **creates and updates** only, saving state after each action (crash-safe / resumable). It **never deletes**: a resource dropped from the @@ -336,6 +347,7 @@ only; it carries no instance data. - [`docs/configuration.md`](docs/configuration.md) — writing the config: keys, portable references, escape hatches - [`docs/environments.md`](docs/environments.md) · [`docs/ci.md`](docs/ci.md) — multi-instance and automation +- [`docs/opentofu-migration.md`](docs/opentofu-migration.md) — living alongside terraform-provider-churchtools: the committed id map, and `ct auth token` as a credential helper - [`docs/handbuch/`](docs/handbuch/) — the **generic ChurchTools reference** (permissions, dynamic groups, field definitions, blueprints), published into the Handbuch - [`docs/api-coverage.md`](docs/api-coverage.md) · [`docs/runbook-manual-surface.md`](docs/runbook-manual-surface.md) — what the API supports, and what still has to be done by hand - [`docs/README.md`](docs/README.md) — how the docs are organised and how pages stay in sync with the code diff --git a/docs/README.md b/docs/README.md index d6a750c..15cb546 100644 --- a/docs/README.md +++ b/docs/README.md @@ -52,6 +52,7 @@ A page with no code behaviour to track declares `sources: []` plus a | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | [`adoption-contract.md`](adoption-contract.md) | What else comes along when `ct adopt` adopts one resource — the five categories and their defaults | | [`api-coverage.md`](api-coverage.md) | Which ChurchTools endpoints support which CRUD verbs | +| [`opentofu-migration.md`](opentofu-migration.md) | Living alongside terraform-provider-churchtools: the committed id map, and `ct auth token` | | [`group-field-decisions.md`](group-field-decisions.md) | Which group fields are managed vs. left to the CT UI, and why | | [`runbook-manual-surface.md`](runbook-manual-surface.md) | What `ct` cannot automate today — where the write path is missing, and the manual steps around it | | `superpowers/` | Historical implementation plans; kept as a record, never published | diff --git a/docs/handbuch/blueprints.md b/docs/handbuch/blueprints.md index b0aef90..adaaf49 100644 --- a/docs/handbuch/blueprints.md +++ b/docs/handbuch/blueprints.md @@ -4,7 +4,7 @@ sources: - src/config/context.ts - src/engine/graph.ts - src/engine/hierarchy.ts -sources_hash: 6f4be8d3a93113ce +sources_hash: b183fe0075bd7ad3 reviewed: 2026-08-28 --- diff --git a/docs/handbuch/field-definitions.md b/docs/handbuch/field-definitions.md index f82f074..1a1a3b1 100644 --- a/docs/handbuch/field-definitions.md +++ b/docs/handbuch/field-definitions.md @@ -3,7 +3,7 @@ title: Field definitions & security levels sources: - src/commands/get.ts - src/api/ctClient.ts -sources_hash: 659b65504025ebb8 +sources_hash: 5c991e89fd5b6b9b reviewed: 2026-08-13 --- diff --git a/docs/handbuch/group-member-fields.md b/docs/handbuch/group-member-fields.md index 1da01b5..67ff8c2 100644 --- a/docs/handbuch/group-member-fields.md +++ b/docs/handbuch/group-member-fields.md @@ -1,5 +1,5 @@ --- -sources_hash: c18b710bff24503d +sources_hash: 83e743cd6cf4caf3 title: Group member fields sources: - src/engine/member-fields.ts diff --git a/docs/handbuch/permissions.md b/docs/handbuch/permissions.md index 4b2c46a..0a0e151 100644 --- a/docs/handbuch/permissions.md +++ b/docs/handbuch/permissions.md @@ -7,7 +7,7 @@ sources: - src/resolve/resolver.ts - src/resolve/refs.ts - src/config/context.ts -sources_hash: c3a6740c4d4bc134 +sources_hash: 1ccb85af4d54a491 reviewed: 2026-08-28 --- @@ -152,7 +152,10 @@ Each line shows the name, its numeric `authId`, and whether it's `scoped` when grants are resolved to tuples against the catalog, after the authed fetch — with a "did you mean" hint drawn from same-module names. (Config evaluation only checks a grant's _shape_: `module:right` string or -`{ right, scope }`; it does not resolve the name against the catalog.) +`{ right, scope }`; it does not resolve the name against the catalog.) A name the +**active** catalog does not define but ct's bundled catalog does is a different +case — a host difference, not a typo — and is skipped rather than fatal; see +[A right this host does not have](#a-right-this-host-does-not-have-178). ## Catalog lifecycle & staleness (#25) @@ -212,6 +215,50 @@ records the version the catalog was captured from. On every `plan`/`apply`: Both are warnings, not errors: the plan still runs and the exit code stays success. `ct permissions catalog --refresh` is the fix for both. +### A right this host does not have (#178) + +An estate's instances do not all have the same modules installed, and one +declarative config is supposed to serve all of them. So a **declared** right +that the active catalog does not define is not automatically an error: + +- the name is in **ct's bundled catalog** but not in this host's capture → a real + right this instance does not have (a module that is not installed). `ct` prints + a warning naming the right, the declaration and both catalog versions, and + **skips that one grant on this host** — it is never granted and never revoked. + Everything else in the same declaration still plans. +- the name is in **no** catalog `ct` has ever seen → still a hard error with the + "did you mean" hint. That is a typo, or a right ChurchTools deleted, and it is + the case the error was written for. + +The same split applies to a [`preserveUnknown`](#partial-ownership-preserveunknown-opt-in-102) +dimension: one no right on this host scopes by is reported and ignored, one no +catalog knows is still rejected at config-evaluation time. + +This mirrors what `ct` has always done in the other direction — a _live_ grant +whose `authId` the catalog cannot name is reported and left alone. Say what you +cannot manage, manage the rest. + +Note the precondition: the "it exists elsewhere" verdict needs a **per-instance +capture** to be active. Without one, the active catalog _is_ the bundled snapshot, +so "missing here" and "missing everywhere" are the same statement and every +absence stays fatal. This is one more reason to commit +`.ct/permission-catalog..json` per host. + +```bash +ct plan --env dev +# ! group_role "implementierung_churchtools_mitglied": right "jpmFlowManager:view" is absent +# from this host's permission catalog (this host's catalog: ChurchTools 3.137.0-RC13, 221 rights) +# — skipped for this host — never granted, never revoked. ct's bundled catalog +# (ChurchTools 3.134.0) defines it, so this reads as a module this instance does not have. +# Pass --strict-catalog to fail on it instead. + +ct plan --env dev --strict-catalog # every unresolvable declaration is an error again +``` + +`--strict-catalog` is available on `ct plan` and `ct apply`. Exit codes are +unchanged either way: a skip is not a pending change, so `--detailed-exitcode` +still reports 0 for a clean plan. + ## `domainId` semantics The two DSL functions manage two different ChurchTools "domain types," and @@ -489,6 +536,14 @@ therefore a cross-environment misgrant, and because declaring a domain makes `ct` _own_ it, the wrong-scope grant also revokes whatever is really there on the other host. The typed reference makes one config plan clean on both. +Between those two sits one more source, for a repo mid-migration: the committed +**OpenTofu id map** (`.ct/ids..json`, #181). It is consulted after managed +state and before the live catalog, and it exists because a resource that moved to +`terraform-provider-churchtools` leaves ct's state entirely — at which point a +reference like `personStatus: "status_unbekannt"` has nothing left to resolve +against, since ct's keys are not derived from the live names the catalog matches +on. See [Living alongside terraform-provider-churchtools](https://github.com/eqrm/ct-cli/blob/main/docs/opentofu-migration.md). + Resolution mirrors the domain-reference rules: managed state first, the live master-data catalog second, and a target **declared in this same config** resolves to a _pending_ scope re-resolved at apply time. A reference resolved diff --git a/docs/opentofu-migration.md b/docs/opentofu-migration.md new file mode 100644 index 0000000..eec76fe --- /dev/null +++ b/docs/opentofu-migration.md @@ -0,0 +1,171 @@ +# Living alongside terraform-provider-churchtools + +The TypeScript DSL is frozen and the tier-0 resources — campuses, group types, +Bereiche, person statuses, comment viewers — are moving to +[`terraform-provider-churchtools`](https://github.com/eqrm/terraform-provider-churchtools). +`ct export tf` produces the HCL and the import blocks for that move. This page +covers the two things needed to live in the in-between state, where OpenTofu owns +tier-0 and `ct` still owns everything above it. + +## The id map: references that outlive their declarations (#181) + +When tier-0 leaves `ct.config.ts`, its **references** stay: several hundred +`campus: "mainz"`, `personStatus: "status_unbekannt"`, `{ groupType: "struktur" }` +on groups, grants and rulesets. Those are logical references — they ask the host +for an id rather than carrying one — so they look like they should survive. + +They do not, on their own. `ct` resolves a logical reference from its own state +first and falls back to matching the key against the live object's **name**. While +the resource was in ct's state the key resolved exactly; with the state entry gone, +only the name fallback is left, and ct's tier-0 keys were never name-derived: + +| key | live name | +| ------------------ | --------------------------- | +| `status_unbekannt` | `Unbekannt` | +| `status_5_core` | `5 - Core` | +| `egc` | `Equippers Germany Central` | + +No slug of those names produces those keys, so removing the state entries makes +`ct plan` fail to resolve them. Pinning numeric ids is not an escape either: one +config serves two hosts, and on the eqrm estate 39 of 43 tier-0 ids differ between +them. + +So `ct` reads a committed **id map** — the exact `key → id` table for this host: + +``` +.ct/ids..json +``` + +```json +{ + "$meta": { + "host": "https://eqrm.church.tools", + "source": "ct export tf", + "generatedAt": "2026-09-21", + "entries": 50 + }, + "campus": { "mainz": { "id": 0 } }, + "person-status": { "status_unbekannt": { "id": 0 }, "status_5_core": { "id": 6 } } +} +``` + +It sits **between** the two existing sources: after ct's own managed state (ct +never stops trusting what it owns) and before the live catalog (an exact table +beats a name guess). A repo without a map behaves exactly as before. + +`ct plan` names the map it loaded in its header, next to the permission catalog: + +``` +permission catalog: .ct/permission-catalog.eqrm.church.tools.json +tofu id map: .ct/ids.eqrm.church.tools.json +``` + +**Commit the map, one per host.** It is host-checked on load — a map whose `$meta.host` +does not match the resolved host is rejected rather than applied, because 39 of 43 +ids differing means a foreign map would resolve every reference to a real, +_wrong_ resource, which nothing downstream could detect. + +### Writing it + +`ct export tf` writes it from the state it is exporting, so the cutover itself +needs no extra step: + +```bash +ct export tf --env prod # tofu/*.tf, tofu/imports.tf, and .ct/ids..json +ct export tf --env prod --no-ids # …without the map, if your repo generates it another way +``` + +The map is written under `.ct/`, not into the tofu output directory: it is `ct`'s +input, not part of the root module tofu reads. + +### Keeping it current + +An export can only describe what `ct` still holds. Once tier-0 belongs to tofu, +tofu is the only place new ids appear — a campus created by `tofu apply` exists in +no ct state file. `ct ids sync` reads tofu's own state instead: + +```bash +tofu state pull | ct ids sync --env prod --tofu-state - # any backend, no S3 client in ct +ct ids sync --env prod --tofu-state terraform.tfstate # or a local file +ct ids sync --env prod --tofu-state terraform.tfstate --dry-run +ct ids list --env prod # what ct would resolve through +``` + +`sync` reports every id it adds, changes or drops, and **refuses to replace a +populated map with an empty one** — the likeliest cause of an empty read is the +wrong workspace, and overwriting 50 working ids with nothing would break every +reference at once. Provider resource types `ct` has no mapping for (a +`churchtools_group`, another provider's resources in a shared state) are reported +and skipped. + +One subtlety: `ct export tf` relabels keys that are not valid HCL identifiers +(`3_groupactive` → `g_3_groupactive`), and that mapping is many-to-one, so it +cannot be inverted by rule. The map records the label alongside the key, and +`ct ids sync` uses the existing map to translate a tofu address back to the ct +key. Without a previous map, the label is taken as the key — correct for every +key that needed no relabelling. + +Reading tofu's **remote** state directly was considered and rejected: it would put +an S3 backend, its credentials and an AWS SDK inside a CLI whose every other read +is ChurchTools. Piping `tofu state pull` keeps the backend tofu's problem. + +## `ct auth token`: a credential helper (#179) + +The provider needs credentials, and its `token` attribute wants a ChurchTools +personal login token — which is permanent, cannot be scoped or rotated by +ChurchTools, and is an admin credential on prod. Writing one to a `.env` file for +the provider to read is the thing worth avoiding. + +`ct auth token` hands over the **session** that token buys instead: + +```bash +$ ct auth token --env dev | jq +{ + "operation": "auth", + "action": "token", + "environment": "dev", + "host": "https://eqrm-dev.church.tools", + "cookie": "ChurchTools_eqrm-dev=…", + "csrfToken": "…", + "expiresAt": "2026-09-22T06:12:00.000Z", + "source": "cache" +} + +$ ct auth token --env dev --raw # the bare cookie, for command substitution +``` + +Why the session rather than the token: it expires, `ct auth logout --env dev` +kills it, and a copy that leaks into a `tofu` debug log or a CI artifact is dead +within hours instead of being the permanent admin credential forever. The login +token itself never leaves the Keychain. + +The contract: + +- the credential goes to **stdout and nothing else does** — every message, + warning and Keychain prompt is on stderr, so `$(ct auth token --raw)` is safe; +- a failure writes **nothing** to stdout and exits non-zero, with the remedy + named (`ct auth login --env `); +- printing to a **terminal is refused** unless `--allow-tty` — a credential in + your scrollback defeats the point of it being short-lived; +- `expiresAt` is `ct`'s reuse ceiling (12h), not a promise from ChurchTools: + treat a 401 as "ask again", which is cheap because this command answers from + the Keychain-cached session. + +Calling it on every `tofu` run is the intended usage. Sessions are cached per host +in the Keychain (macOS), so that is normally zero network calls; when a handshake +_is_ needed, a cross-process brake keeps it from becoming a burst: + +- handshakes against one host are spaced at least 3s apart (waited out, not an + error); +- more than 20 in a rolling hour is refused, naming when the window frees up — + that is a runaway loop, and hammering a throttled instance only lengthens the + outage for everyone on it; +- `CT_NO_LOGIN_THROTTLE=1` disables it, for a CI job that knows it runs alone. + +The counter lives in `$XDG_CACHE_HOME/ct-cli/login-throttle..json` and holds +nothing but timestamps. Deleting it, or being unable to write it, simply means no +throttle. + +**CI is unaffected and needs none of this.** A CI job passes the token explicitly +from a GitHub secret, which is already storage-free; this path exists for local +development, where the alternative was a token on disk. diff --git a/src/api/ctClient.ts b/src/api/ctClient.ts index 969355c..d7d0b86 100644 --- a/src/api/ctClient.ts +++ b/src/api/ctClient.ts @@ -78,13 +78,31 @@ const DEFAULT_PAGE_LIMIT = 100; * exactly as it always has. */ export interface SessionCache { - load(host: string, token: string): Promise<{ cookie: string; csrfToken: string } | null>; + /** + * `obtainedAt` (epoch ms) is optional and purely informational: it lets a caller that HANDS THE + * SESSION ON say when it was bought, and therefore how long it may still be reused (#179). + * A cache that does not track it simply omits it. + */ + load( + host: string, + token: string, + ): Promise<{ cookie: string; csrfToken: string; obtainedAt?: number } | null>; save(host: string, token: string, session: { cookie: string; csrfToken: string }): Promise; drop(host: string): Promise; } +/** + * A cross-process brake on the login handshake (#179). Injected for the same reason + * {@link SessionCache} is: a client built in a test must not touch the developer's disk, and a client + * without one behaves exactly as it always has. + */ +export interface LoginThrottleGate { + acquire(host: string): Promise; +} + export interface CtClientOptions { sessionCache?: SessionCache; + loginThrottle?: LoginThrottleGate; } /** Human-readable "wait this long" for a 429, from `Retry-After` when the server sent one. */ @@ -105,6 +123,9 @@ export class CtClient { private ctVersion: string | null = null; /** Kept so an expired session can be re-bought without the caller having to notice (#145). */ private loginToken: string | null = null; + /** When the live session was bought, and whether this process bought it (#179). */ + private sessionObtainedAt: number | null = null; + private sessionSource: "cache" | "handshake" | null = null; /** Re-entrancy guards: no self-heal while a login (or a resume probe) is already in flight. */ private loggingIn = false; private resuming = false; @@ -202,7 +223,7 @@ export class CtClient { if (!cache) { return null; } - let cached: { cookie: string; csrfToken: string } | null = null; + let cached: { cookie: string; csrfToken: string; obtainedAt?: number } | null = null; try { cached = await cache.load(this.config.host, loginToken); } catch { @@ -213,6 +234,8 @@ export class CtClient { } this.cookie = cached.cookie; this.csrfToken = cached.csrfToken; + this.sessionObtainedAt = cached.obtainedAt ?? null; + this.sessionSource = "cache"; this.resuming = true; try { return await this.get("/whoami"); @@ -241,7 +264,36 @@ export class CtClient { } } + /** + * The live session, for a caller whose whole job is to hand it to another tool (`ct auth token`, + * #179). Returns `null` before the handshake has run. + * + * This is the ONE deliberate exit from the rule that the cookie never leaves this object: a + * ChurchTools session expires and can be dropped (`ct auth logout`), while the personal login + * token it was bought with is permanent and, on prod, an admin credential. Handing out the session + * is what lets the OpenTofu provider authenticate without the permanent secret ever leaving the + * Keychain. Nothing here logs or formats it — `sessionCredential` is read only by a caller that has + * already decided where the value is allowed to go. + */ + sessionCredential(): { + cookie: string; + csrfToken: string; + obtainedAt: number; + source: "cache" | "handshake"; + } | null { + if (!this.cookie || this.csrfToken === null) return null; + return { + cookie: this.cookie, + csrfToken: this.csrfToken, + obtainedAt: this.sessionObtainedAt ?? Date.now(), + source: this.sessionSource ?? "handshake", + }; + } + private async performLogin(loginToken: string): Promise { + // Before the request, not after: the point is to not ADD to a burst that is already in progress, + // including one started by a different `ct` process (#179). + await this.options.loginThrottle?.acquire(this.config.host); // The token rides as a URL query param (it lands in the server's access logs). This is // unavoidable for this token class: the handshake above is documented to require the // `login_token` query param — an `Authorization` header yields a null CSRF token and breaks @@ -271,6 +323,8 @@ export class CtClient { if (!this.cookie) { throw new CtApiError("Login succeeded but no session cookie was returned", res.status, null); } + this.sessionObtainedAt = Date.now(); + this.sessionSource = "handshake"; await this.refreshCsrfToken(); // Keep the freshly bought session for the NEXT invocation. Best-effort: a store // that refuses (no Keychain, locked Keychain) must not fail the command. diff --git a/src/api/session.ts b/src/api/session.ts index 92000b0..8825104 100644 --- a/src/api/session.ts +++ b/src/api/session.ts @@ -5,6 +5,7 @@ import { CtClient, type WhoAmI } from "./ctClient.js"; import { readCredentials } from "../auth/tokenStore.js"; import { keychainSessionCache } from "../auth/sessionStore.js"; +import { fileLoginThrottle } from "../auth/loginThrottle.js"; import { normalizeHost, resolveConfig } from "../config.js"; export interface AuthedSession { @@ -48,7 +49,12 @@ export async function authedSession(): Promise { // The session cache is keyed by the SAME host the binding check above just cleared, // so a cached session can only ever be replayed against the instance it came from (#30/#145). - const client = new CtClient(config, { sessionCache: keychainSessionCache() }); + const client = new CtClient(config, { + sessionCache: keychainSessionCache(), + // The session cache removes most handshakes; the throttle bounds the ones it cannot — a cold + // Linux/CI run, `ct auth token` called per `tofu` run, two invocations in parallel (#179). + loginThrottle: fileLoginThrottle(), + }); const me = await client.authenticate(token); // Hard-fail below the minimum CT version before any command reads or writes — // a stale instance half-applies (tier-0 writes succeed, hierarchy endpoints 404). diff --git a/src/application/contracts.ts b/src/application/contracts.ts index 508c5a0..1680402 100644 --- a/src/application/contracts.ts +++ b/src/application/contracts.ts @@ -3,7 +3,7 @@ export type JsonPrimitive = string | number | boolean | null; export type JsonValue = JsonPrimitive | JsonValue[] | { [key: string]: JsonValue }; export type OperationName = - "plan" | "apply" | "coverage" | "adopt" | "state" | "refresh" | "destroy" | "auth" | "export-tf"; + "plan" | "apply" | "coverage" | "adopt" | "state" | "refresh" | "destroy" | "auth" | "export-tf" | "ids"; /** Common project selection accepted by CLI and, later, HTTP adapters. */ export interface ProjectRequest { diff --git a/src/application/operations/auth.ts b/src/application/operations/auth.ts index 9589318..a3c8342 100644 --- a/src/application/operations/auth.ts +++ b/src/application/operations/auth.ts @@ -4,7 +4,7 @@ import { CtClient, type WhoAmI } from "../../api/ctClient.js"; import { formatError } from "../../api/format.js"; import { meetsMinVersion, MIN_CT_VERSION, type CtInfo } from "../../api/version.js"; import { checkAllEnvAuth, type EnvAuthStatus } from "../../auth/status.js"; -import { keychainSessionCache } from "../../auth/sessionStore.js"; +import { keychainSessionCache, SESSION_MAX_AGE_MS } from "../../auth/sessionStore.js"; import { clearCredentials, readToken, @@ -151,6 +151,108 @@ export async function runAuthLogout( }; } +export interface AuthTokenRequest { + cwd?: string; + environment?: string; +} + +/** + * A ChurchTools SESSION, for handing to another tool (#179). + * + * Deliberately not the login token. The OpenTofu provider declares `token` as required, so the + * tier-0 cutover meant writing a personal ChurchTools login token to local disk — a permanent, + * full-privilege credential (an admin one on prod) that ChurchTools offers no way to scope or + * rotate. The session bought with it is the one short-lived credential in the system: it expires, + * `ct auth logout` drops it, and a copy that leaks into a `tofu` debug log or a CI artifact is dead + * within hours rather than forever. So that is what `ct auth token` emits, and the token itself never + * leaves the Keychain. + */ +export interface AuthTokenResult { + operation: "auth"; + action: "token"; + environment: string | null; + host: string; + cookie: string; + csrfToken: string; + /** + * When `ct` will stop reusing this session (its `obtainedAt` + the 12h reuse ceiling in + * sessionStore). A CEILING, not a promise: ChurchTools does not advertise its session lifetime, so + * the server may end the session sooner. A consumer should treat a 401 as "ask again", not as an + * error — which is also why this command is cheap to call on every run. + */ + expiresAt: string; + /** `cache` — reused an existing session; `handshake` — one login handshake was spent to buy it. */ + source: "cache" | "handshake"; +} + +export interface AuthTokenDependencies { + project?: ProjectResolutionDependencies; + resolveProject?: typeof resolveProject; + readToken?: typeof readToken; + authedSession?: () => Promise; + env?: NodeJS.ProcessEnv; + cwd?: () => string; +} + +/** + * Resolve host + session for one environment, buying a session only if there is no reusable one. + * + * Returns the credential; it never prints, logs or formats it — every caller decides where the value + * is allowed to go (the CLI adapter puts it on stdout alone, and refuses a terminal). + */ +export async function runAuthToken( + request: AuthTokenRequest = {}, + dependencies: AuthTokenDependencies = {}, +): Promise { + const env = dependencies.env ?? process.env; + const cwd = resolve(dependencies.cwd?.() ?? process.cwd(), request.cwd ?? "."); + let project; + try { + project = await (dependencies.resolveProject ?? resolveProject)( + { cwd, environment: request.environment }, + { ...dependencies.project, env }, + ); + } catch (cause) { + if (!(cause instanceof MissingHostError)) throw cause; + throw new CtApplicationError( + "AUTH_REQUIRED", + "No host resolved. Pass --env , or run `ct auth login --host --token `.", + { cause }, + ); + } + // Checked before the network: "no credential for this host" is the case a credential helper hits + // most often, and it must fail with the remedy rather than with a login error. + if (!(await (dependencies.readToken ?? readToken)(project.host))) { + throw new CtApplicationError( + "AUTH_REQUIRED", + request.environment + ? `No token for ${project.host}. Run \`ct auth login --env ${request.environment}\`.` + : `No token for ${project.host}. Run \`ct auth login --host ${project.host} --token \`.`, + { details: { host: project.host } }, + ); + } + const { client } = await (dependencies.authedSession ?? authedSession)(); + const session = client.sessionCredential(); + if (!session) { + throw new CtApplicationError( + "AUTH_REQUIRED", + `Authenticated against ${project.host} but no session cookie was captured, so there is nothing ` + + `to hand over. Re-run \`ct auth login\` for this host.`, + { details: { host: project.host } }, + ); + } + return { + operation: "auth", + action: "token", + environment: project.environment, + host: project.host, + cookie: session.cookie, + csrfToken: session.csrfToken, + expiresAt: new Date(session.obtainedAt + SESSION_MAX_AGE_MS).toISOString(), + source: session.source, + }; +} + /** Return authentication identity and source metadata without ever returning a token. */ export async function runAuthStatus( request: AuthStatusRequest = {}, diff --git a/src/application/operations/export-tf.ts b/src/application/operations/export-tf.ts index 58fdfd5..2191da3 100644 --- a/src/application/operations/export-tf.ts +++ b/src/application/operations/export-tf.ts @@ -5,6 +5,8 @@ import { assertLabelsUnique, hclLabel, hclType, renderResource } from "../../exp import { renderImports, type ImportTarget } from "../../export/imports.js"; import { EXPORTABLE_TYPES, fileForType, OWNED_FILES } from "../../export/layout.js"; import { renderVersions } from "../../export/provider.js"; +import { CATALOG_DIR } from "../../permissions/catalog-store.js"; +import { writeIdMap, type IdMapEntry } from "../../resolve/idMap.js"; import { loadState } from "../../state/state.js"; import type { CtWarning, OperationResult, ProjectRequest } from "../contracts.js"; import { resolveProject } from "../project.js"; @@ -26,6 +28,15 @@ export interface ExportTfRequest extends ProjectRequest { * file outright — worse than overwriting it, because nothing hints at why. */ writeVersions?: boolean; + /** + * Also write `.ct/ids..json`, the committed key→id map ct resolves leftover references + * through once these resources are no longer in its state (#181). Default true: every export is a + * step towards ct not declaring these any more, and that is exactly when the map becomes load-bearing. + * + * Written OUTSIDE `outDir`, beside the per-instance permission catalog, because it is ct's input, + * not part of the tofu root module. + */ + writeIds?: boolean; } export interface ExportTfValue { @@ -35,6 +46,8 @@ export interface ExportTfValue { relabelled: { key: string; label: string }[]; /** Managed types the provider has no mapping for yet, with how many were skipped. */ skipped: { type: string; count: number }[]; + /** Path of the id map written beside the export, or null when `--no-ids` was passed. */ + idMapPath: string | null; } export type ExportTfResult = OperationResult; @@ -90,6 +103,7 @@ export async function runExportTf(request: ExportTfRequest): Promise(); const imports: ImportTarget[] = []; const relabelled: { key: string; label: string }[] = []; + const idEntries: IdMapEntry[] = []; for (const resource of [...selected].sort((a, b) => { const left = address(a.type, a.key); @@ -107,6 +121,15 @@ export async function runExportTf(request: ExportTfRequest): Promise ({ code: "EXPORT_TYPE_UNSUPPORTED", message: @@ -145,7 +173,7 @@ export async function runExportTf(request: ExportTfRequest): Promise; + +export interface IdsListValue { + path: string | null; + entries: IdMapEntry[]; +} + +export type IdsListResult = OperationResult; + +export interface IdsOperationDependencies { + project?: ProjectResolutionDependencies; + resolveProject?: typeof resolveProject; + loadIdMap?: typeof loadIdMap; + writeIdMap?: typeof writeIdMap; + readStdin?: () => Promise; +} + +async function readAllStdin(): Promise { + const chunks: Buffer[] = []; + for await (const chunk of process.stdin) chunks.push(Buffer.from(chunk)); + return Buffer.concat(chunks).toString("utf8"); +} + +function entryId(entry: { type: string; key: string }): string { + return `${entry.type}\u0000${entry.key}`; +} + +export async function runIdsSync( + request: IdsSyncRequest, + dependencies: IdsOperationDependencies = {}, +): Promise { + const project = await (dependencies.resolveProject ?? resolveProject)(request, dependencies.project); + const dir = join(project.cwd, CATALOG_DIR); + const raw = + request.tofuState === "-" + ? await (dependencies.readStdin ?? readAllStdin)() + : await readFile(request.tofuState, "utf8"); + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch (err) { + throw new Error( + `Could not read the tofu state from ${request.tofuState === "-" ? "stdin" : request.tofuState}: ` + + `not valid JSON (${(err as Error).message}).`, + ); + } + // The EXISTING map is the source of the label→key mapping: the exporter relabels keys that are not + // valid HCL identifiers, many-to-one, so a tofu address can only be mapped back to its ct key with + // the map that recorded the relabelling. Without one, a label is taken as the key — correct for + // every key that needed no relabelling, which is all of them on a normal instance. + const previous = await (dependencies.loadIdMap ?? loadIdMap)(project.host, dir); + const { entries, unmapped, serial } = readTfState(parsed, previous?.keysByLabel ?? new Map()); + + const previousById = new Map((previous?.entries ?? []).map((e) => [entryId(e), e])); + const nextById = new Map(entries.map((e) => [entryId(e), e])); + const added = entries.filter((e) => !previousById.has(entryId(e))); + const changed = entries + .map((entry) => ({ entry, previousId: previousById.get(entryId(entry))?.id })) + .filter((row): row is { entry: IdMapEntry; previousId: number } => { + return row.previousId !== undefined && row.previousId !== row.entry.id; + }); + // Dropped from tofu's state, so ct must stop claiming to resolve them — keeping a stale id would + // silently point every reference at whatever now holds that number. + const removed = (previous?.entries ?? []).filter((e) => !nextById.has(entryId(e))); + + const warnings: CtWarning[] = []; + if (unmapped.length > 0) { + warnings.push({ + code: "IDS_TYPE_UNSUPPORTED", + message: + `Ignored ${unmapped.length} provider resource type(s) ct has no mapping for: ${unmapped.join(", ")}. ` + + `References to those cannot resolve through the id map — keep them managed by ct, or use a numeric id.`, + details: { types: unmapped }, + }); + } + if (entries.length === 0) { + warnings.push({ + code: "IDS_EMPTY", + message: + `The tofu state holds no churchtools resources ct can map, so the id map would be empty. ` + + `Check you pulled the state for ${project.host} (\`tofu state pull\` in the right workspace).`, + }); + } + + const written = !request.dryRun && entries.length > 0; + const path = written + ? await (dependencies.writeIdMap ?? writeIdMap)(project.host, entries, dir, "ct ids sync") + : (previous?.path ?? join(dir, `ids.${project.host}.json`)); + + return { + operation: "ids", + project, + warnings, + value: { path, entries, added, changed, removed, serial, written }, + }; +} + +export async function runIdsList( + request: ProjectRequest = {}, + dependencies: IdsOperationDependencies = {}, +): Promise { + const project = await (dependencies.resolveProject ?? resolveProject)(request, dependencies.project); + const map = await (dependencies.loadIdMap ?? loadIdMap)(project.host, join(project.cwd, CATALOG_DIR)); + return { + operation: "ids", + project, + warnings: [], + value: { path: map?.path ?? null, entries: map?.entries ?? [] }, + }; +} diff --git a/src/application/operations/plan.ts b/src/application/operations/plan.ts index 86b6fb7..3ca9e54 100644 --- a/src/application/operations/plan.ts +++ b/src/application/operations/plan.ts @@ -3,8 +3,10 @@ import { authedSession, type AuthedSession } from "../../api/session.js"; import { loadConfig } from "../../config/load.js"; import { buildPlan } from "../../engine/build.js"; import { summarize, type Plan, type PlanAction } from "../../engine/types.js"; +import { setStrictCatalog } from "../../permissions/catalog.js"; import { CATALOG_DIR, loadHostCatalog } from "../../permissions/catalog-store.js"; import { buildPermissionPlan, type PermissionPlanItem } from "../../permissions/plan.js"; +import { loadIdMap } from "../../resolve/idMap.js"; import { Resolver } from "../../resolve/resolver.js"; import { loadState, type State } from "../../state/state.js"; import type { CtClient } from "../../api/ctClient.js"; @@ -12,7 +14,17 @@ import type { CtWarning, OperationResult, ProjectRequest } from "../contracts.js import { noopObserver, type OperationObserver } from "../ports.js"; import { resolveProject, type ProjectResolutionDependencies } from "../project.js"; -export type PlanRequest = ProjectRequest; +export interface PlanRequest extends ProjectRequest { + /** + * `--strict-catalog` (#178): treat a declared right (or `preserveUnknown` dimension) the active + * permission catalog does not define as a hard error, even when ct's bundled catalog defines it. + * + * The default is the skip-with-warning posture, which is what lets ONE config serve two instances + * with different modules installed. This flag is for a repo that would rather planning failed than + * have any declaration silently not apply. + */ + strictCatalog?: boolean; +} export interface PlanSummary { resources: Record; @@ -41,6 +53,8 @@ export interface PlanValue { */ buildWarnings: string[]; permissionCatalogPath: string | null; + /** The committed OpenTofu id map this plan resolved through (#181), when the repo has one. */ + tofuIdMapPath: string | null; } export type PlanResult = OperationResult; @@ -51,6 +65,7 @@ export interface PlanOperationDependencies { project?: ProjectResolutionDependencies; resolveProject?: typeof resolveProject; loadHostCatalog?: typeof loadHostCatalog; + loadIdMap?: typeof loadIdMap; loadConfig?: typeof loadConfig; loadState?: typeof loadState; authedSession?: () => Promise; @@ -104,10 +119,16 @@ export async function buildPlanContext( const project = await (dependencies.resolveProject ?? resolveProject)(request, dependencies.project); observer.emit({ type: "phase-started", phase: "load-project" }); + // Set BEFORE the catalog and the config load: the config's own `preserveUnknown` validation reads + // it at eval time (config/context.ts), and it must describe the catalog that is about to be loaded. + setStrictCatalog(request.strictCatalog ?? false); const catalogPath = await (dependencies.loadHostCatalog ?? loadHostCatalog)( project.host, join(project.cwd, CATALOG_DIR), ); + // Loaded beside the permission catalog and from the same directory: both are committed, per-host + // artefacts a consumer repo keeps under `.ct/`. + const idMap = await (dependencies.loadIdMap ?? loadIdMap)(project.host, join(project.cwd, CATALOG_DIR)); const { resources: desired, permissions, @@ -120,6 +141,7 @@ export async function buildPlanContext( state, desired, host: project.host, + idMap, }); observer.emit({ type: "phase-started", phase: "build-plan" }); @@ -158,6 +180,7 @@ export async function buildPlanContext( stateHost: state.host, buildWarnings: resourceResult.warnings ?? [], permissionCatalogPath: catalogPath, + tofuIdMapPath: idMap?.path ?? null, }, }, }; diff --git a/src/application/operations/state.ts b/src/application/operations/state.ts index 4ec60a5..c633049 100644 --- a/src/application/operations/state.ts +++ b/src/application/operations/state.ts @@ -1,6 +1,7 @@ import { loadConfig } from "../../config/load.js"; import { resourceType } from "../../resources/registry.js"; import { collectRefs, isRef, type Ref } from "../../resolve/refs.js"; +import { refKindResolvesLive } from "../../resolve/resolver.js"; import { loadState, saveState, type ManagedResource } from "../../state/state.js"; import type { CtWarning, OperationResult, ProjectRequest } from "../contracts.js"; import { InMemoryMutationLock } from "../prepared-operation-store.js"; @@ -47,34 +48,82 @@ export async function listState( }; } -async function declaredKeys( +/** + * What the config DECLARES, and what it merely REFERENCES (#180). + * + * These were one set, and that made `ct state rm` refuse a key the config only points at. Same state + * file, same config: `campus/horgen` (0 occurrences) was accepted, while `campus/mainz` — declared + * nowhere, but named by `campus: "mainz"` on 60 other resources — was refused as "still declared". + * + * A reference is not a declaration. A logical ref asks the HOST for the id (`verify` labels exactly + * these "logical ref, resolved live — not compared"); it never asks `ct` to create the referenced + * object, so removing the state entry cannot make the next plan propose creating it. Only a real + * declaration can do that, which is what the guard was written to catch. + * + * Declarations are keyed by `type\0key`, not by key alone: the guard exists to predict what the next + * plan would do to a specific resource, and a `campus` declaration says nothing about what removing a + * `group-type` of the same name would cause. + * + * One class of reference DOES still block removal, for the guard's original reason rather than a + * spelling one: a ref whose kind has no live master-data catalog (a `group` — groups are managed-only) + * can be resolved from nothing but ct's state, so dropping the entry it names makes the very next + * plan a hard error instead of a create proposal. Those are refused, and said so in their own words. + */ +interface ConfigUsage { + declared: Set; + /** key → how many times the config references it. Reported, never a refusal. */ + referenceCounts: Map; + /** Keys referenced by at least one ref kind that can only resolve from managed state. */ + stateOnlyRefs: Set; +} + +function declarationId(type: string, key: string): string { + return `${type}\u0000${key}`; +} + +async function configUsage( configPath: string, dependencies: StateOperationDependencies, -): Promise> { +): Promise { const { resources, permissions } = await (dependencies.loadConfig ?? loadConfig)(configPath); - const keys = new Set(resources.map((resource) => resource.key)); + const declared = new Set(resources.map((resource) => declarationId(resource.type, resource.key))); + const referenceCounts = new Map(); + const stateOnlyRefs = new Set(); + const countRef = (key: string, resolvesLive: boolean): void => { + referenceCounts.set(key, (referenceCounts.get(key) ?? 0) + 1); + if (!resolvesLive) stateOnlyRefs.add(key); + }; const addRef = (ref: Ref): void => { - if (ref.kind === "group-role") keys.add(ref.group); - else if (ref.kind === "group-type-role") keys.add(ref.groupType); - else if (ref.kind === "group-member-field") keys.add(ref.group); - else keys.add(ref.key); + // The compound kinds address a permission DOMAIN, and the key they carry here is the GROUP or + // GROUP TYPE the domain hangs off — so each is classified by the kind of that target, not by the + // compound kind itself. A group_role domain needs its group in managed state; a group_type_role + // domain resolves its type from `/group/grouptypes` like any other group-type ref. + if (ref.kind === "group-role") countRef(ref.group, refKindResolvesLive("group")); + else if (ref.kind === "group-member-field") countRef(ref.group, refKindResolvesLive("group")); + else if (ref.kind === "group-type-role") countRef(ref.groupType, refKindResolvesLive("group-type")); + else countRef(ref.key, refKindResolvesLive(ref.kind)); }; + // Refs reachable from the resources themselves (`campus: "mainz"` on a group) as well as from the + // permission set — the old check only walked the latter, so the count it reported was partial too. + for (const ref of collectRefs(resources)) addRef(ref); for (const ref of collectRefs(permissions)) addRef(ref); for (const permission of permissions) { for (const grant of permission.grants) { if (typeof grant === "string" || !Array.isArray(grant.scope)) continue; for (const entry of grant.scope) { - if (typeof entry === "string" && entry.length > 0) keys.add(entry); + // A BARE STRING in a scope list is the historical group dimension (`resolveScope` looks it + // up in managed state and nowhere else), so it is state-only by construction. + if (typeof entry === "string" && entry.length > 0) countRef(entry, false); else if (entry !== null && typeof entry === "object" && !isRef(entry)) { const values = Object.values(entry as Record); if (values.length === 1 && typeof values[0] === "string" && values[0].length > 0) { - keys.add(values[0]); + countRef(values[0], false); } } } } } - return keys; + return { declared, referenceCounts, stateOnlyRefs }; } export async function removeStateEntry( @@ -102,16 +151,48 @@ export async function removeStateEntry( const warnings: CtWarning[] = []; if (!request.force) { try { - const declared = await declaredKeys(project.configPath, dependencies); - if (declared.has(request.key)) { + const { declared, referenceCounts, stateOnlyRefs } = await configUsage( + project.configPath, + dependencies, + ); + if (stateOnlyRefs.has(request.key)) { + throw new Error( + `"${request.key}" is still referenced by the config as a resource that only ct's state can ` + + `resolve (a group has no live master-data catalog to fall back to), so removing it would make ` + + `the next plan fail to resolve those references. Remove the references first, or pass --force ` + + `if you are deleting both in the same change.`, + ); + } + if (declared.has(declarationId(request.type, request.key))) { throw new Error( `"${request.key}" is still declared in the config, so removing it from state would make the next ` + `plan propose CREATING a resource that already exists on this host. Remove the ` + `declaration first, or pass --force if you are deleting both in the same change.`, ); } + // Referenced but not declared — allowed, and worth saying out loud: every one of those refs + // now resolves against the LIVE host by name, which is a different resolution path than the + // state lookup it had a moment ago (see #181). A key whose live name does not slug back to it + // will fail the next plan, and this is the last point at which that is cheap to notice. + const references = referenceCounts.get(request.key) ?? 0; + if (references > 0) { + warnings.push({ + code: "STILL_REFERENCED", + message: + `The config references "${request.key}" ${references} time(s) but does not declare it, so it is ` + + `removed from state. Those references now resolve against ${project.host} by name — check the ` + + `next \`ct plan\` resolves them, or pin them to a numeric id.`, + details: { key: request.key, type: request.type, references }, + }); + } } catch (caught) { - if (caught instanceof Error && caught.message.includes("is still declared in the config")) + // Both refusals above are verdicts, not read failures — re-throw them rather than degrade + // into "could not read the config, removing anyway". + if ( + caught instanceof Error && + (caught.message.includes("is still declared in the config") || + caught.message.includes("is still referenced by the config")) + ) throw caught; warnings.push({ code: "CONFIG_UNREADABLE", diff --git a/src/auth/loginThrottle.ts b/src/auth/loginThrottle.ts new file mode 100644 index 0000000..bb7fcce --- /dev/null +++ b/src/auth/loginThrottle.ts @@ -0,0 +1,117 @@ +/** + * A cross-process brake on the ChurchTools LOGIN handshake (#179). + * + * ChurchTools rate-limits logins per instance, and `ct` is one-shot: nothing inside a single + * invocation can see how many handshakes the last minute already spent. The session cache (#145) + * removes most of them, but it cannot remove the ones that matter here — `ct auth token` is designed + * to be called by another tool on every run, and on Linux/CI there is no session cache at all, so + * every invocation starts cold. Two parallel `tofu` runs, or a loop in a script, and the instance + * answers 429 for everybody, including the humans. + * + * So the count lives in a file: one per host, holding nothing but timestamps. + * + * - A handshake closer than {@link MIN_INTERVAL_MS} to the previous one WAITS for the remainder. + * A short sleep is strictly better than a 429 — the caller gets a session either way. + * - More than {@link MAX_PER_HOUR} in a rolling hour THROWS, naming when the window frees up. That + * is a runaway loop, not a burst, and continuing to hammer a throttled instance only lengthens + * the outage for everyone on it. + * + * Deliberately NOT in the Keychain: there is no secret here (see the blob shape below), and a + * Keychain read is an ACL prompt away from being the very thing that makes the CLI unusable in a + * script. A cache file that a user deletes, or that never appears at all, simply means no throttle — + * every read and write is best-effort, and only the rolling cap ever fails a command. + */ +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { hostSlug } from "../permissions/catalog-store.js"; + +/** Minimum spacing between two handshakes against one host. Waited out, never an error. */ +export const MIN_INTERVAL_MS = 3_000; +/** Handshakes per rolling hour per host before `ct` refuses to add to the pile. */ +export const MAX_PER_HOUR = 20; +const HOUR_MS = 60 * 60 * 1000; + +/** What a throttle needs from its caller. Injected so a client in a test never touches the disk. */ +export interface LoginThrottle { + acquire(host: string): Promise; +} + +/** Nothing here is a credential: a host label and the times ct last logged in to it. */ +interface ThrottleRecord { + host: string; + recent: number[]; +} + +export function throttleDir(env: NodeJS.ProcessEnv = process.env): string { + const base = env.XDG_CACHE_HOME?.trim() || join(homedir(), ".cache"); + return join(base, "ct-cli"); +} + +export function throttlePath(host: string, env: NodeJS.ProcessEnv = process.env): string { + return join(throttleDir(env), `login-throttle.${hostSlug(host)}.json`); +} + +function parseRecord(raw: string, host: string, now: number): ThrottleRecord { + try { + const parsed = JSON.parse(raw) as Partial; + if (parsed.host !== host || !Array.isArray(parsed.recent)) return { host, recent: [] }; + // A timestamp in the FUTURE (a clock change, a copied file) is dropped rather than trusted: + // keeping it would hold the brake on for as long as the skew lasts. + return { host, recent: parsed.recent.filter((t) => typeof t === "number" && t <= now) }; + } catch { + return { host, recent: [] }; + } +} + +const defaultSleep = (ms: number): Promise => new Promise((resolve) => setTimeout(resolve, ms)); + +export interface FileThrottleOptions { + env?: NodeJS.ProcessEnv; + sleep?: (ms: number) => Promise; + now?: () => number; +} + +/** + * The real, file-backed throttle. `CT_NO_LOGIN_THROTTLE=1` disables it — for a CI job that knows it + * runs alone and would rather fail fast than sleep. + */ +export function fileLoginThrottle(options: FileThrottleOptions = {}): LoginThrottle { + const env = options.env ?? process.env; + const sleep = options.sleep ?? defaultSleep; + const now = options.now ?? (() => Date.now()); + return { + async acquire(host: string): Promise { + if (env.CT_NO_LOGIN_THROTTLE === "1") return; + const path = throttlePath(host, env); + let record: ThrottleRecord = { host, recent: [] }; + try { + record = parseRecord(await readFile(path, "utf8"), host, now()); + } catch { + // No file, an unreadable one, a read-only home — none of those is a reason to refuse a login. + } + const recent = record.recent.filter((t) => now() - t < HOUR_MS).sort((a, b) => a - b); + if (recent.length >= MAX_PER_HOUR) { + const freesAt = new Date(recent[0]! + HOUR_MS); + throw new Error( + `Refusing to log in to ${host} again: ${recent.length} login handshakes in the last hour ` + + `(ct's own limit, to stay under ChurchTools' rate limit). The oldest ages out at ` + + `${freesAt.toISOString()}. If a script is looping, fix that first; set ` + + `CT_NO_LOGIN_THROTTLE=1 to override.`, + ); + } + const last = recent[recent.length - 1]; + if (last !== undefined) { + const wait = MIN_INTERVAL_MS - (now() - last); + if (wait > 0) await sleep(wait); + } + const next: ThrottleRecord = { host, recent: [...recent, now()] }; + try { + await mkdir(throttleDir(env), { recursive: true }); + await writeFile(path, `${JSON.stringify(next)}\n`, "utf8"); + } catch { + // Best-effort: an unwritable cache means no throttle, never a failed command. + } + }, + }; +} diff --git a/src/commands/apply.ts b/src/commands/apply.ts index 4fd4cfb..95bf0db 100644 --- a/src/commands/apply.ts +++ b/src/commands/apply.ts @@ -21,6 +21,7 @@ interface ApplyOptions { backupDir?: string; autoApprove?: boolean; refresh?: boolean; + strictCatalog?: boolean; } // Retain this command-module export for callers that used it before the application extraction. @@ -35,6 +36,10 @@ export function applyCommand(): Command { .option("--confirm-env ", "confirm a protected env non-interactively (must match --env exactly)") .option("--backup-dir ", "directory for the pre-apply backup (or set CT_BACKUP_DIR)") .option("-y, --auto-approve", "skip the confirmation prompt") + .option( + "--strict-catalog", + "fail when a declared right or preserveUnknown dimension is absent from this host's permission catalog (default: skip it with a warning)", + ) .option( "--refresh", "after a successful apply, POST /dynamicgroups/{id}/refresh for each changed dynamic group (per-group only)", @@ -49,6 +54,7 @@ export function applyCommand(): Command { environment: opts.env, backupDir: opts.backupDir, refresh: opts.refresh, + strictCatalog: opts.strictCatalog, // No wall-clock expiry: the confirmation below blocks on stdin for as long as the // operator needs to read the rendered diff (#156 review). Staleness is caught by the // state fingerprint at execute time, not by a timer. diff --git a/src/commands/auth.ts b/src/commands/auth.ts index 9d1bad3..df971ed 100644 --- a/src/commands/auth.ts +++ b/src/commands/auth.ts @@ -1,5 +1,5 @@ import { Command } from "commander"; -import { runAuthLogin, runAuthLogout, runAuthStatus } from "../application/operations/auth.js"; +import { runAuthLogin, runAuthLogout, runAuthStatus, runAuthToken } from "../application/operations/auth.js"; import { normalizeHost } from "../config.js"; import { isSecureStorageAvailable } from "../auth/tokenStore.js"; import { bootstrapLoginToken } from "../auth/login.js"; @@ -119,6 +119,43 @@ export function authCommand(): Command { } }); + cmd + .command("token") + .description("Print a short-lived ChurchTools session for another tool (credential helper)") + .option("-e, --env ", "environment profile from ct.envs.json (targets that host)") + .option("--raw", "print only the session cookie, with no JSON envelope") + .option("--allow-tty", "print the credential even though stdout is a terminal") + .action(async (opts: { env?: string; raw?: boolean; allowTty?: boolean }) => { + // A credential is for a pipe, not for scrollback: a terminal keeps it in the buffer, in a + // `script` capture and in whatever the user pastes next. The whole point of emitting the + // SESSION rather than the login token is to shorten a leak's life — printing it where it will + // be kept works against that, so it takes an explicit flag (#179). + if (process.stdout.isTTY && !opts.allowTty) { + error( + "Refusing to print a credential to a terminal. Pipe it (e.g. `ct auth token --env dev | jq`), " + + "or pass --allow-tty if you really want it on screen.", + ); + process.exitCode = 1; + return; + } + try { + const result = await runAuthToken({ environment: opts.env }); + // stdout carries the credential and NOTHING else, so `$(ct auth token --raw)` is safe. Every + // line below — and every warning, prompt and progress message anywhere in ct — is on stderr. + process.stdout.write(opts.raw ? `${result.cookie}\n` : `${JSON.stringify(result)}\n`); + info( + `${result.host}${result.environment ? ` (env ${result.environment})` : ""} — session from ` + + `${result.source === "cache" ? "the keychain cache" : "a fresh login handshake"}, ` + + `reusable until ${result.expiresAt}.`, + ); + } catch (caught) { + // Nothing is written to stdout on failure: a consumer that reads stdout for a credential + // must get an empty stream, never a diagnostic it might mistake for one. + error(formatError(caught)); + process.exitCode = 1; + } + }); + cmd .command("logout") .description("Remove the stored host + login token") diff --git a/src/commands/export-tf.ts b/src/commands/export-tf.ts index d49c60d..e095e45 100644 --- a/src/commands/export-tf.ts +++ b/src/commands/export-tf.ts @@ -1,3 +1,4 @@ +import { relative } from "node:path"; import { Command } from "commander"; import { runExportTf } from "../application/operations/export-tf.js"; import { info, warn } from "../ui.js"; @@ -8,6 +9,7 @@ interface ExportTfOptions { only?: string[]; out: string; versions: boolean; + ids: boolean; } export function exportCommand(): Command { @@ -21,6 +23,10 @@ export function exportCommand(): Command { "--no-versions", "do not write or prune versions.tf — use when your repo owns it (e.g. to pin a provider version)", ) + .option( + "--no-ids", + "do not write .ct/ids..json — the key→id map ct resolves leftover references through once these resources leave its state", + ) .action(async (opts: ExportTfOptions) => { const { value, warnings } = await runExportTf({ statePath: opts.state, @@ -28,11 +34,13 @@ export function exportCommand(): Command { only: opts.only, outDir: opts.out, writeVersions: opts.versions, + writeIds: opts.ids, }); for (const file of value.files) info(`wrote ${opts.out}/${file}`); for (const r of value.relabelled) { info(`relabelled "${r.key}" -> "${r.label}" (HCL references must be identifiers)`); } + if (value.idMapPath) info(`wrote ${relative(process.cwd(), value.idMapPath)}`); info(`${value.imported} resources exported`); // Printed last, after the success line, so the gap is the final thing on // screen rather than scrolled off above a list of written files. diff --git a/src/commands/ids.ts b/src/commands/ids.ts new file mode 100644 index 0000000..0a1dcb0 --- /dev/null +++ b/src/commands/ids.ts @@ -0,0 +1,87 @@ +import { relative } from "node:path"; +import { Command } from "commander"; +import { runIdsList, runIdsSync } from "../application/operations/ids.js"; +import { error, formatError, info, out, success, warn } from "../ui.js"; + +/** + * `ct ids` — the committed OpenTofu id map (#181). + * + * The map is what keeps `campus: "mainz"` and `personStatus: "status_unbekannt"` resolving after + * those resources move to terraform-provider-churchtools and leave ct's state. `ct export tf` writes + * it during the cutover; this command keeps it current afterwards, when tofu is the only place new + * ids appear. + */ +export function idsCommand(): Command { + const cmd = new Command("ids").description( + "Maintain the committed key→id map ct resolves OpenTofu-owned resources through", + ); + + cmd + .command("sync") + .description("Rewrite .ct/ids..json from an OpenTofu state file (or stdin)") + .requiredOption( + "--tofu-state ", + 'path to a terraform.tfstate, or "-" for stdin (e.g. `tofu state pull | ct ids sync -e prod --tofu-state -`)', + ) + .option("-e, --env ", "environment profile from ct.envs.json (decides which host's map)") + .option("--dry-run", "report what would change without writing the map") + .action(async (opts: { tofuState: string; env?: string; dryRun?: boolean }) => { + try { + const { value, warnings, project } = await runIdsSync({ + tofuState: opts.tofuState, + environment: opts.env, + dryRun: opts.dryRun, + }); + const where = relative(project.cwd, value.path); + for (const entry of value.added) info(`+ ${entry.type}.${entry.key} = ${entry.id}`); + for (const { entry, previousId } of value.changed) { + info(`~ ${entry.type}.${entry.key} = ${entry.id} (was ${previousId})`); + } + for (const entry of value.removed) info(`- ${entry.type}.${entry.key} (was ${entry.id})`); + for (const w of warnings) warn(w.message); + const serial = value.serial === null ? "" : ` (tofu state serial ${value.serial})`; + if (value.written) { + success(`${where}: ${value.entries.length} ids from ${project.host}${serial}.`); + } else { + info(`Would write ${where}: ${value.entries.length} ids from ${project.host}${serial}.`); + } + // An empty read never overwrites a good map — the likeliest cause is the wrong workspace, + // and replacing 50 working ids with nothing would break every reference at once. + if (!value.written && !opts.dryRun) process.exitCode = 1; + } catch (caught) { + error(formatError(caught)); + process.exitCode = 1; + } + }); + + cmd + .command("list") + .description("Show the id map ct would resolve through for this host") + .option("-e, --env ", "environment profile from ct.envs.json (decides which host's map)") + .option("--json", "emit the entries as JSON") + .action(async (opts: { env?: string; json?: boolean }) => { + try { + const { value, project } = await runIdsList({ environment: opts.env }); + if (opts.json) { + out(value.entries); + return; + } + if (!value.path) { + info( + `No id map for ${project.host}. One is written by \`ct export tf\`, or by ` + + `\`ct ids sync --tofu-state \`.`, + ); + return; + } + info(`${relative(project.cwd, value.path)} — ${value.entries.length} ids for ${project.host}`); + for (const entry of value.entries) { + process.stdout.write(`${entry.type}.${entry.key} = ${entry.id}\n`); + } + } catch (caught) { + error(formatError(caught)); + process.exitCode = 1; + } + }); + + return cmd; +} diff --git a/src/commands/plan.ts b/src/commands/plan.ts index ef62fac..4d4ac0e 100644 --- a/src/commands/plan.ts +++ b/src/commands/plan.ts @@ -24,6 +24,7 @@ interface PlanOptions { outputBase?: string; locale?: string; detailedExitcode?: boolean; + strictCatalog?: boolean; } function collectFormat(value: string, previous: string[]): string[] { @@ -100,6 +101,10 @@ export function planCommand(): Command { ) .option("--output-base ", "write selected formats as .txt/.json/.md") .option("--locale ", "Markdown language: de-DE or en", "de-DE") + .option( + "--strict-catalog", + "fail when a declared right or preserveUnknown dimension is absent from this host's permission catalog (default: skip it with a warning)", + ) .option( "--detailed-exitcode", "Terraform-style exit code: 0 = no changes, 1 = error, 2 = changes pending (resource or permission)", @@ -112,12 +117,18 @@ export function planCommand(): Command { configPath: opts.config, statePath: opts.state, environment: opts.env, + strictCatalog: opts.strictCatalog, }); const { project, value } = result; const catalogPath = value.permissionCatalogPath ? relative(project.cwd, value.permissionCatalogPath) : null; if (catalogPath) info(`permission catalog: ${catalogPath}`); + // Said out loud for the same reason the catalog path is: a reference resolving through a + // committed map rather than through state or a live name is worth knowing before reading a diff. + if (value.tofuIdMapPath) { + info(`tofu id map: ${relative(project.cwd, value.tofuIdMapPath)}`); + } // Additive on top of the raw plan/permissions (#24) — existing consumers of `plan`/ // `permissions` are unaffected. Every projection below consumes this one computation. diff --git a/src/config/context.ts b/src/config/context.ts index 3b1c3e9..6177825 100644 --- a/src/config/context.ts +++ b/src/config/context.ts @@ -23,7 +23,7 @@ import { import { collectRefs } from "../resolve/refs.js"; import type { DomainType } from "../permissions/grants.js"; import type { DesiredPermission, Grant, PreserveUnknown } from "../permissions/types.js"; -import { KNOWN_SCOPE_FIELDS } from "../permissions/catalog.js"; +import { scopeFieldVerdict } from "../permissions/catalog.js"; import { GROUP_STATUS_NO_CATALOG, isRef, ref, refKey, type Ref } from "../resolve/refs.js"; import { normalizeScopeEntry } from "../permissions/scope.js"; import { conventionalRulesetRef, isCallerAssignedId, knownFields } from "../resources/registry.js"; @@ -281,7 +281,11 @@ function normalizePreserveUnknown( `default, or name the dimensions to leave alone.`, ); } - const unknown = value.filter((d) => !KNOWN_SCOPE_FIELDS.has(d)); + // A dimension the ACTIVE catalog lacks but ct's bundled catalog knows is a host difference, not a + // typo (#178): the module that scopes by it is not installed here. It stays in the declaration and + // is reported by `buildPermissionPlan`; only a name no catalog defines is still fatal, because that + // is the case this guard was written for — a typo silently preserving nothing. + const unknown = value.filter((d) => scopeFieldVerdict(d) === "unknown"); if (unknown.length > 0) { throw new Error( `${domainType} "${input.key}": "preserveUnknown" names ${unknown.length === 1 ? "a scope dimension" : "scope dimensions"} ` + diff --git a/src/index.ts b/src/index.ts index 4c911d7..b628fc4 100644 --- a/src/index.ts +++ b/src/index.ts @@ -5,6 +5,7 @@ import { getCommand } from "./commands/get.js"; import { adoptCommand } from "./commands/adopt.js"; import { stateCommand } from "./commands/state.js"; import { exportCommand } from "./commands/export-tf.js"; +import { idsCommand } from "./commands/ids.js"; import { coverageCommand } from "./commands/coverage.js"; import { permissionsCommand } from "./commands/permissions.js"; import { reportCommand } from "./commands/report.js"; @@ -36,6 +37,7 @@ export function buildProgram(): Command { program.addCommand(adoptCommand()); program.addCommand(stateCommand()); program.addCommand(exportCommand()); + program.addCommand(idsCommand()); program.addCommand(coverageCommand()); program.addCommand(permissionsCommand()); program.addCommand(reportCommand()); diff --git a/src/permissions/catalog.ts b/src/permissions/catalog.ts index 6bd6e19..20f1552 100644 --- a/src/permissions/catalog.ts +++ b/src/permissions/catalog.ts @@ -43,6 +43,36 @@ function splitCatalog(data: unknown): { rights: Record; me const bundled = splitCatalog(catalogData); +/** + * The rights in the catalog BUNDLED with this release, kept reachable after {@link useCatalog} has + * pointed everything else at a per-instance capture (#178). + * + * It is the only evidence `ct` has for the difference between the two ways a declared right can be + * absent from the active catalog: + * + * - absent here too → the name is a typo, or a right ChurchTools deleted. A hard error, as before. + * - present here → the name is real, this HOST just does not have it (a module that is not + * installed). Skipped with a warning, so ONE inapplicable declaration cannot + * take down planning for the 100 that do apply. + * + * That asymmetry is deliberate: the bundled snapshot is a real instance's full right set, so it is a + * usable "does this name exist anywhere ct knows" oracle even though it is the wrong oracle for + * "what can this host do". + */ +export const BUNDLED_CATALOG: Readonly> = bundled.rights; + +/** The ChurchTools version the bundled snapshot was captured against, for a warning that has to name + * BOTH catalogs (#178). `null` on a legacy catalog with no `$meta`. */ +export const BUNDLED_CATALOG_VERSION: string | null = bundled.meta?.ctVersion ?? null; + +/** Scope dimensions the bundled catalog knows — the same oracle as {@link BUNDLED_CATALOG}, for + * `preserveUnknown` dimensions (#178). */ +export const BUNDLED_SCOPE_FIELDS: ReadonlySet = new Set( + Object.values(bundled.rights) + .map((e) => e.scopeField) + .filter((f): f is string => f != null), +); + /** * The permission catalog: name → authId bridge. Bundled at build time (see the module header), and * REPLACEABLE at runtime by a per-instance capture (#105 — see {@link useCatalog}). @@ -120,6 +150,59 @@ export function useCatalog(data: unknown, opts: { perInstance?: boolean } = {}): /** Restore the catalog bundled with this release. Exists so tests can undo {@link useCatalog}. */ export function useBundledCatalog(): void { useCatalog(catalogData); + STRICT_CATALOG = false; +} + +/** + * `--strict-catalog` (#178): restore the pre-#178 behaviour, where ANY declaration the active + * catalog cannot resolve is a hard error rather than a skip. + * + * A module-level flag rather than a threaded option because the two places that need it sit at + * opposite ends of the pipeline — config EVALUATION (`preserveUnknown` dimensions, deep inside a + * user's `ct.config.ts` callback) and plan BUILDING — and the catalog they consult is already + * process-wide mutable state (`CATALOG` is swapped by {@link useCatalog}). Threading a flag through + * both would add a parameter to every layer in between without making anything more honest. + */ +export let STRICT_CATALOG = false; + +export function setStrictCatalog(strict: boolean): void { + STRICT_CATALOG = strict; +} + +/** + * Why a name is not in the active catalog (#178). + * + * - `known` — it is; carry on. + * - `host-missing` — the active catalog is a capture from THIS host and lacks it, but the bundled + * catalog has it: a real right this instance does not have. Skip + warn. + * - `unknown` — no catalog `ct` has ever seen defines it: a typo or a deleted right. Hard error. + * + * `host-missing` requires a per-instance capture to be active. Without one the active catalog IS the + * bundled snapshot, so "missing from the active catalog" and "missing everywhere" are the same + * statement, and the verdict stays `unknown` — a repo that has not captured its host's catalog gets + * exactly today's behaviour, including today's errors. + */ +export type CatalogVerdict = "known" | "host-missing" | "unknown"; + +export function catalogVerdict(name: string): CatalogVerdict { + if (CATALOG[name]) return "known"; + if (!STRICT_CATALOG && CATALOG_IS_PER_INSTANCE && BUNDLED_CATALOG[name]) return "host-missing"; + return "unknown"; +} + +/** The same verdict for a scope DIMENSION rather than a right (#178, `preserveUnknown`). */ +export function scopeFieldVerdict(dimension: string): CatalogVerdict { + if (KNOWN_SCOPE_FIELDS.has(dimension)) return "known"; + if (!STRICT_CATALOG && CATALOG_IS_PER_INSTANCE && BUNDLED_SCOPE_FIELDS.has(dimension)) + return "host-missing"; + return "unknown"; +} + +/** How to describe the active catalog in a warning: where it came from and how big it is. */ +export function describeCatalog(): string { + if (!CATALOG_META) return `${Object.keys(CATALOG).length} rights`; + const scope = CATALOG_IS_PER_INSTANCE ? "this host's catalog" : "ct's bundled catalog"; + return `${scope}: ChurchTools ${CATALOG_META.ctVersion}, ${Object.keys(CATALOG).length} rights`; } export function resolveAuthId(name: string): CatalogEntry { diff --git a/src/permissions/plan.ts b/src/permissions/plan.ts index 957a013..04bacb7 100644 --- a/src/permissions/plan.ts +++ b/src/permissions/plan.ts @@ -10,6 +10,10 @@ import type { State } from "../state/state.js"; import type { DesiredResource } from "../engine/types.js"; import { resolveAuthId, + catalogVerdict, + describeCatalog, + scopeFieldVerdict, + BUNDLED_CATALOG_VERSION, CATALOG_META, CATALOG_IS_PER_INSTANCE, KNOWN_AUTH_IDS, @@ -49,6 +53,26 @@ export interface PermissionPlanItem { diff: GrantDiff; } +/** + * A declaration this HOST cannot act on, because the right (or `preserveUnknown` dimension) it names + * is absent from the host's own permission catalog while ct's bundled catalog has it (#178). + * + * Reported, never fatal. An estate whose instances have different modules installed cannot express + * the difference in one declarative file, so a `jpmFlowManager:*` right that is correct on prod used + * to make the whole plan — including the 100 rights that DO apply — abort on dev. The mirror case has + * always been handled this way: a live grant whose authId the catalog cannot name is reported and + * left untouched. This is the same posture in the other direction — say what you cannot manage, + * manage the rest. + */ +export interface CatalogSkip { + domainType: DomainType; + /** The declaration's key, e.g. the group_role key. */ + key: string; + /** What was skipped: a right name, or a `preserveUnknown` scope dimension. */ + name: string; + kind: "right" | "dimension"; +} + /** * Fan out each grant to (authId, dataId) tuples. ChurchTools reads a scoped grant back as * ONE ROW PER dataId with a scalar `dataId` (see `normalizeActual`), so a desired tuple with @@ -61,9 +85,18 @@ export function desiredTuples( state: State, declaredGroupKeys: ReadonlySet = new Set(), scopeRefs: ScopeRefMap = new Map(), + onSkip?: (skip: CatalogSkip) => void, ): GrantTuple[] { return p.grants.flatMap((g): GrantTuple[] => { const name = typeof g === "string" ? g : g.right; + // A right this host's catalog does not define, but ct's bundled catalog does (#178): emit NO + // tuple at all. No tuple means no PUT, and — because the host cannot have a live row under a + // name it does not define — nothing this declaration could have owned goes unclaimed, so + // nothing lands in toDelete either. The grant is simply not this host's business. + if (catalogVerdict(name) === "host-missing") { + onSkip?.({ domainType: p.domainType, key: p.key, name, kind: "right" }); + return []; + } const entry = resolveAuthId(name); if (typeof g === "string") { // A scoped right declared as a bare string would emit `dataId: []` — a silent GLOBAL grant. @@ -206,6 +239,32 @@ async function resolveDomainIds( return resolved; } +/** + * One line per skipped declaration (#178), naming the right, the declaration, and BOTH catalogs — + * the host's (what it can do) and the bundled one (where the name came from). Without the second + * half the warning reads like a typo; with it, it reads like the module difference it is. + */ +function renderSkip(skip: CatalogSkip): string { + // A skip can only arise while a per-instance capture is active AND the bundled catalog has the name + // (that is what `catalogVerdict` decided), so the provenance clause always has something to say. + const provenance = BUNDLED_CATALOG_VERSION + ? ` ct's bundled catalog (ChurchTools ${BUNDLED_CATALOG_VERSION}) defines it, so this reads as a ` + + `module this instance does not have.` + : ""; + const what = + skip.kind === "right" + ? `right "${skip.name}" is absent from this host's permission catalog` + : `"preserveUnknown" names the scope dimension "${skip.name}", which no right in this host's permission catalog scopes by`; + const consequence = + skip.kind === "right" + ? "skipped for this host — never granted, never revoked" + : "ignored for this host — it can preserve nothing here"; + return ( + `${skip.domainType} "${skip.key}": ${what} (${describeCatalog()}) — ${consequence}.${provenance} ` + + `Pass --strict-catalog to fail on it instead.` + ); +} + export async function buildPermissionPlan( client: PermissionReader, state: State, @@ -217,6 +276,22 @@ export async function buildPermissionPlan( const items: PermissionPlanItem[] = []; const fetchErrors: string[] = []; const warnings: string[] = []; + // Deduped by declaration+name: a right declared on two roles is two skips, the same right listed + // twice on one role is one. + const skips = new Map(); + const noteSkip = (skip: CatalogSkip): void => { + skips.set(`${skip.domainType}:${skip.key}:${skip.kind}:${skip.name}`, skip); + }; + // `preserveUnknown` dimensions are validated at config-eval time (config/context.ts), which keeps + // a typo fatal — but a dimension that exists only on the OTHER host cannot be judged there without + // making the config non-portable, so it is passed through and reported here instead (#178). + for (const p of permissions) { + if (!Array.isArray(p.preserveUnknown)) continue; + for (const dimension of p.preserveUnknown) { + if (scopeFieldVerdict(dimension) === "host-missing") + noteSkip({ domainType: p.domainType, key: p.key, name: dimension, kind: "dimension" }); + } + } // Catalog staleness (#25/#105): the catalog is a snapshot captured against one CT version. If the // live instance reports a different version, right names/authIds/scopeFields may have drifted — // warn (never fail) so the diff is trusted-but-verified. @@ -280,7 +355,7 @@ export async function buildPermissionPlan( // domainId is irrelevant to desiredTuples (it only reads key/domainType/grants); pass the // pending Ref through so the shape stays a valid DesiredPermission. diff: diffGrants( - desiredTuples({ ...p, domainId: p.pendingDomain }, state, declaredGroupKeys, scopeRefs), + desiredTuples({ ...p, domainId: p.pendingDomain }, state, declaredGroupKeys, scopeRefs, noteSkip), [], ), }); @@ -327,7 +402,7 @@ export async function buildPermissionPlan( domainType: p.domainType, domainId: p.domainId, diff: diffGrants( - desiredTuples(p, state, declaredGroupKeys, scopeRefs), + desiredTuples(p, state, declaredGroupKeys, scopeRefs, noteSkip), knownActual, preservePredicateFor(p.preserveUnknown), // The unknown-authId guard (#25) deliberately does NOT apply here: an unnameable right can @@ -336,5 +411,10 @@ export async function buildPermissionPlan( ), }); } + for (const skip of [...skips.values()].sort((a, b) => + `${a.key}${a.name}` < `${b.key}${b.name}` ? -1 : 1, + )) { + warnings.push(renderSkip(skip)); + } return { items, fetchErrors, warnings }; } diff --git a/src/resolve/idMap.ts b/src/resolve/idMap.ts new file mode 100644 index 0000000..19b48ef --- /dev/null +++ b/src/resolve/idMap.ts @@ -0,0 +1,185 @@ +/** + * The OpenTofu id map: a committed, per-host `key -> id` table for resources ct no longer declares + * (#181). + * + * ## The blocker this exists for + * + * The tier-0 cutover moves campuses, group types, departments, person statuses and comment viewers + * to `terraform-provider-churchtools`, which means their tables leave `ct.config.ts` AND their + * entries leave `ct-state..json`. Hundreds of REFERENCES stay behind (`campus: "mainz"`, + * `personStatus: "status_unbekannt"`), and those still have to resolve. + * + * `verify` calls them "logical ref, resolved live — not compared", which is true but only as a + * FALLBACK: while the object was in ct's state, the key resolved from state; with the state entry + * gone, resolution falls back to matching the key against the live object's NAME. ct's tier-0 keys + * were never name-derived, so that fallback cannot work for them — structurally, not by accident: + * + * key live name + * status_unbekannt Unbekannt + * status_5_core 5 - Core + * egc Equippers Germany Central + * + * No slug of those names produces those keys. And a literal id is not an option either: one config + * serves two hosts, and 39 of 43 tier-0 ids differ between them. + * + * ## Why a committed file rather than reading tofu's remote state + * + * After the cutover, tofu's state IS the authoritative key→id map, and it keys on exactly the + * logical names the config already uses (`churchtools_person_status.status_unbekannt` came from + * `status_unbekannt`). Reading it directly, though, would put an S3 backend — credentials, a network + * dependency, one AWS SDK — inside a CLI whose other reads are all ChurchTools. So the map is a + * file: written by `ct export tf` from the state it is exporting, refreshed from a `tofu state pull` + * by `ct ids sync`, and committed per host so portability survives. + * + * ## Where it sits in resolution + * + * AFTER ct's own managed state (ct never stops trusting what it owns) and BEFORE the live catalog + * (the map is exact, the catalog match is a name guess). See `Resolver.resolve`. + */ +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { CATALOG_DIR, hostSlug } from "../permissions/catalog-store.js"; + +/** One resolvable resource: its ct key, its id on this host, and the HCL label tofu knows it by. */ +export interface IdMapEntry { + type: string; + key: string; + id: number; + /** + * The HCL label the exporter gave it, when that differs from the key (`3_groupactive` → + * `g_3_groupactive`). Retained so `ct ids sync` can map a tofu resource address BACK to the ct key + * it came from — the relabelling is many-to-one, so it cannot be re-derived from the label alone. + */ + label?: string; +} + +export interface IdMapMeta { + host: string; + source: string; + generatedAt: string; + entries: number; + [k: string]: unknown; +} + +/** The map as the resolver consumes it: exact lookups, plus the label index `ids sync` needs. */ +export interface IdMap { + path: string; + meta: IdMapMeta | null; + entries: IdMapEntry[]; + /** `type\0key` → id. */ + ids: ReadonlyMap; + /** `type\0label` → ct key, for reading a tofu state back. */ + keysByLabel: ReadonlyMap; +} + +function entryKey(type: string, key: string): string { + return `${type}\u0000${key}`; +} + +export function idMapPath(host: string, dir: string = CATALOG_DIR): string { + return join(dir, `ids.${hostSlug(host)}.json`); +} + +/** + * The file shape: `$meta` plus one object per resource type, exactly like the permission catalog's + * layout, so the two committed artefacts read the same way in a diff. + * + * { "$meta": {...}, "campus": { "mainz": { "id": 0 } }, "person-status": { "status_5_core": { "id": 6 } } } + */ +export function renderIdMap(host: string, entries: readonly IdMapEntry[], source: string): string { + const byType: Record> = {}; + // Sorted by type then key: a committed file has to be byte-stable across runs, or every export + // shows up as a diff (same reason the HCL export sorts). + for (const entry of [...entries].sort((a, b) => + entryKey(a.type, a.key) < entryKey(b.type, b.key) ? -1 : 1, + )) { + (byType[entry.type] ??= {})[entry.key] = + entry.label !== undefined && entry.label !== entry.key + ? { id: entry.id, label: entry.label } + : { id: entry.id }; + } + const meta: IdMapMeta = { + host, + source, + generatedAt: new Date().toISOString().slice(0, 10), + entries: entries.length, + }; + return `${JSON.stringify({ $meta: meta, ...byType }, null, 1)}\n`; +} + +export function parseIdMap(raw: string, path: string, host: string): IdMap { + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch (err) { + throw new Error(`Malformed id map ${path}: not valid JSON (${(err as Error).message}).`); + } + if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) { + throw new Error(`Malformed id map ${path}: expected a JSON object at the top level.`); + } + const { $meta, ...types } = parsed as { $meta?: IdMapMeta } & Record; + // Host-checked like the session cache (#30) and asserted like the state file: a map applied to the + // wrong instance would resolve every reference to a plausible id on the wrong host, which is worse + // than not resolving at all — nothing downstream could tell. + if ($meta && typeof $meta.host === "string" && hostSlug($meta.host) !== hostSlug(host)) { + throw new Error( + `Id map ${path} was generated for ${$meta.host}, but the resolved host is ${host}. ` + + `Re-generate it for this host (\`ct export tf --env \`) — applying another host's ids ` + + `would resolve every reference to the wrong resource.`, + ); + } + const entries: IdMapEntry[] = []; + for (const [type, rows] of Object.entries(types)) { + if (typeof rows !== "object" || rows === null || Array.isArray(rows)) { + throw new Error(`Malformed id map ${path}: "${type}" is not an object of key → { id }.`); + } + for (const [key, value] of Object.entries(rows as Record)) { + const row = value as { id?: unknown; label?: unknown }; + if (typeof row?.id !== "number" || !Number.isFinite(row.id)) { + throw new Error(`Malformed id map ${path}: ${type} "${key}" has no numeric id.`); + } + entries.push({ + type, + key, + id: row.id, + ...(typeof row.label === "string" ? { label: row.label } : {}), + }); + } + } + return { + path, + meta: $meta ?? null, + entries, + ids: new Map(entries.map((e) => [entryKey(e.type, e.key), e.id])), + keysByLabel: new Map(entries.map((e) => [entryKey(e.type, e.label ?? e.key), e.key])), + }; +} + +/** + * Load this host's map, or `null` when there is none (every reference then resolves exactly as it + * does today). A malformed or foreign map THROWS rather than being skipped: a repo that committed + * one is relying on it, and silently resolving without it is how the cutover breaks unnoticed. + */ +export async function loadIdMap(host: string, dir: string = CATALOG_DIR): Promise { + const path = idMapPath(host, dir); + let raw: string; + try { + raw = await readFile(path, "utf8"); + } catch (err) { + if ((err as { code?: string }).code === "ENOENT") return null; + throw err; + } + return parseIdMap(raw, path, host); +} + +export async function writeIdMap( + host: string, + entries: readonly IdMapEntry[], + dir: string = CATALOG_DIR, + source = "ct export tf", +): Promise { + await mkdir(dir, { recursive: true }); + const path = idMapPath(host, dir); + await writeFile(path, renderIdMap(host, entries, source), "utf8"); + return path; +} diff --git a/src/resolve/resolver.ts b/src/resolve/resolver.ts index 0c91814..54698b2 100644 --- a/src/resolve/resolver.ts +++ b/src/resolve/resolver.ts @@ -7,6 +7,12 @@ * state resolves to a {@link PendingRef} (its id is only known after the * resource tier applies — re-resolved at apply time, mirroring the permission * scope pattern in src/permissions/scope.ts). + * 1b. The committed OpenTofu id map for this host (`.ct/ids..json`, #181), an EXACT + * `key -> id` table for resources that moved to terraform-provider-churchtools and are therefore + * no longer in ct's state. After managed state (ct never stops trusting what it owns) and before + * the catalog below (an exact table beats a name guess) — and it is what keeps a reference like + * `personStatus: "status_unbekannt"` resolving at all, since ct's tier-0 keys are not derived + * from the live names the catalog matches on. * 2. Live catalog master data, matched by `slug(name) === key` with an exact-name * secondary: campus → /campuses, group-type → /group/grouptypes, role-def → /group/roles. * Each catalog is fetched at most once per run and cached by a `Map`, @@ -27,6 +33,7 @@ */ import type { CtClient } from "../api/ctClient.js"; import type { State } from "../state/state.js"; +import type { IdMap } from "./idMap.js"; import type { DesiredResource } from "../engine/types.js"; import { slug } from "../resources/registry.js"; import { @@ -127,6 +134,19 @@ const CATALOG_PATH: Partial> = { "role-def": "/group/roles", }; +/** + * Can a reference of this kind resolve WITHOUT ct's managed state — i.e. does it have a live + * master-data catalog to fall back to (#180)? + * + * Read by `ct state rm`, which has to predict what removing a state entry does to the NEXT plan. A + * `campus`/`person-status`/`group-type` ref falls back to the catalog and keeps resolving; a `group` + * ref has no catalog at all (groups are managed-only), so removing the group it names turns the next + * plan into a hard error. Same table, one question, so the two cannot drift. + */ +export function refKindResolvesLive(kind: RefKind): boolean { + return CATALOG_PATH[kind] !== undefined; +} + interface CatalogRecord { id: number; name?: string; @@ -146,6 +166,11 @@ export interface ResolverDeps { desired: DesiredResource[]; /** Host label for error messages. Defaults to `state.host`. */ host?: string; + /** + * The committed OpenTofu id map for this host (#181), when one exists. Loaded by the command layer + * (like the per-instance permission catalog) so the resolver stays filesystem-free. + */ + idMap?: Pick | null; } /** @@ -197,6 +222,7 @@ export class Resolver { private readonly client: Pick & Partial>; private readonly state: State; private readonly host: string; + private readonly idMap: Pick | null; private readonly catalogs = new Map>(); /** Per-group role list cache (group_role domain resolution), keyed by group id, fetched at most once. */ private readonly groupRoleLists = new Map>(); @@ -229,6 +255,7 @@ export class Resolver { this.client = deps.client; this.state = deps.state; this.host = deps.host ?? deps.state.host; + this.idMap = deps.idMap ?? null; for (const d of deps.desired) { let set = this.declaredByType.get(d.type); if (!set) { @@ -267,6 +294,13 @@ export class Resolver { if (managed && managed.type === type) return managed.id; if (this.declaredByType.get(type)?.has(r.key)) return pendingRef(r); } + // (1b) the OpenTofu id map (#181): an exact key→id table for what tofu now owns. Consulted only + // for kinds that HAVE a managed resource type — the map is written from resources ct exported, + // so a kind ct never manages can never be in it, and asking would only slow the miss down. + if (type !== undefined) { + const mapped = this.idMap?.ids.get(`${type}\u0000${r.key}`); + if (mapped !== undefined) return mapped; + } // (2) live catalog if (CATALOG_PATH[r.kind] !== undefined) return this.resolveFromCatalog(r, site); // (3) hard error @@ -636,6 +670,17 @@ export class Resolver { const where = catalog ? `no managed resource and no live ${r.kind} at ${catalog} matches key "${r.key}"` : `no managed ${r.kind} named "${r.key}" is declared or adopted`; + // A repo mid-cutover has the map loaded and the key missing from it — by far the likeliest + // reason a reference that used to resolve stops (#181), and the one remedy the generic advice + // below does not name. + if (this.idMap) { + return new Error( + `Cannot resolve ${refLabel(r)} referenced at ${site} on ${this.host}: ${where}, and ` + + `${this.idMap.path} has no ${r.kind} "${r.key}" either. If OpenTofu owns it now, refresh the ` + + `map (\`tofu state pull | ct ids sync --tofu-state -\`); otherwise declare/adopt it, fix the ` + + `key/name, or use a numeric id.`, + ); + } return new Error( `Cannot resolve ${refLabel(r)} referenced at ${site} on ${this.host}: ${where}. ` + `Declare/adopt it, fix the key/name, or use a numeric id.`, diff --git a/src/resolve/tfstate.ts b/src/resolve/tfstate.ts new file mode 100644 index 0000000..3d5ca5b --- /dev/null +++ b/src/resolve/tfstate.ts @@ -0,0 +1,84 @@ +/** + * Reading an OpenTofu state file into an {@link IdMapEntry} list (#181) — PURE. + * + * `ct export tf` can write the id map from the state it is exporting, but only for as long as ct + * still holds those entries. Once tier-0 belongs to tofu, tofu is the only place new ids appear: a + * campus created by `tofu apply` exists in no ct state file, so an export cannot know about it. That + * is what this reader is for — `tofu state pull | ct ids sync -e prod --tofu-state -` refreshes the + * map from the authoritative source without ct ever learning to speak S3. + * + * Kept pure (a parsed JSON value in, entries out) so it can be tested against fixture state without + * a filesystem, like the HCL renderer it mirrors. + */ +import { hclType } from "../export/hcl.js"; +import { EXPORTABLE_TYPES } from "../export/layout.js"; +import type { IdMapEntry } from "./idMap.js"; + +/** Terraform resource type → ct resource type, derived from the exporter's own table so the two + * directions cannot drift. */ +const CT_TYPE_BY_HCL_TYPE: ReadonlyMap = new Map( + EXPORTABLE_TYPES.map((ctType) => [hclType(ctType), ctType] as const), +); + +export interface TfStateReadResult { + entries: IdMapEntry[]; + /** Provider resources in the state whose type ct has no mapping for — reported, never fatal. */ + unmapped: string[]; + /** Terraform's own serial, if present, so a sync can say which state version it read. */ + serial: number | null; +} + +interface TfResource { + mode?: string; + type?: string; + name?: string; + instances?: { attributes?: Record }[]; +} + +/** + * Turn a parsed `terraform.tfstate` into id-map entries. + * + * `keysByLabel` maps a `type\0label` back to the ct key it was exported from. The exporter relabels + * keys that are not valid HCL identifiers (`3_groupactive` → `g_3_groupactive`) and that mapping is + * many-to-one, so it cannot be inverted by rule — without the previous map's help, the label IS the + * key, which is correct for every key that needed no relabelling. + */ +export function readTfState( + parsed: unknown, + keysByLabel: ReadonlyMap = new Map(), +): TfStateReadResult { + if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) { + throw new Error("Malformed tofu state: expected a JSON object at the top level."); + } + const state = parsed as { resources?: unknown; serial?: unknown }; + if (state.resources !== undefined && !Array.isArray(state.resources)) { + throw new Error('Malformed tofu state: "resources" is not an array.'); + } + const entries: IdMapEntry[] = []; + const unmapped = new Set(); + for (const raw of (state.resources ?? []) as TfResource[]) { + // `data` blocks describe reads, not managed objects: their id is whatever the data source + // happened to return, and nothing in the config keys off it. + if (raw.mode !== undefined && raw.mode !== "managed") continue; + if (typeof raw.type !== "string" || typeof raw.name !== "string") continue; + const ctType = CT_TYPE_BY_HCL_TYPE.get(raw.type); + if (!ctType) { + // Anything the provider manages that ct has no resource type for — a churchtools_group once + // the provider grows one, or another provider's resources in a shared state. + if (raw.type.startsWith("churchtools_")) unmapped.add(raw.type); + continue; + } + const attributes = raw.instances?.[0]?.attributes ?? {}; + const id = Number(attributes.id); + // A resource in state with no numeric id is mid-create or tainted; an unresolvable reference is + // a better outcome than a reference resolved to NaN. + if (!Number.isFinite(id)) continue; + const key = keysByLabel.get(`${ctType}\u0000${raw.name}`) ?? raw.name; + entries.push(key === raw.name ? { type: ctType, key, id } : { type: ctType, key, id, label: raw.name }); + } + return { + entries, + unmapped: [...unmapped].sort(), + serial: typeof state.serial === "number" ? state.serial : null, + }; +} diff --git a/tests/auth-token-command.test.ts b/tests/auth-token-command.test.ts new file mode 100644 index 0000000..bbcbbae --- /dev/null +++ b/tests/auth-token-command.test.ts @@ -0,0 +1,131 @@ +/** + * `ct auth token` — a credential helper for the OpenTofu provider (#179). + * + * The provider declares `token` as required, so the tier-0 cutover meant writing a personal + * ChurchTools login token to local disk: permanent, unscopable, and an admin credential on prod. + * What this command emits instead is the SESSION that token buys — the one credential in the system + * that expires and can be revoked (`ct auth logout`), so a copy that leaks into a `tofu` debug log + * dies in hours rather than never. + * + * The contract under test is the one a `credential_process`-style consumer depends on: + * the credential on stdout and NOTHING else there, everything human on stderr, and an empty stdout + * plus a non-zero exit when nothing resolves. + */ +import { describe, it, expect, beforeEach, afterEach, vi } from "vitest"; +import type * as TokenStore from "../src/auth/tokenStore.js"; + +const authedSession = vi.fn(); +const LOGIN_TOKEN = "permanent-login-token-xyz"; +const readToken = vi.fn(async () => LOGIN_TOKEN); +vi.mock("../src/api/session.js", () => ({ authedSession })); +vi.mock("../src/auth/tokenStore.js", async (importOriginal) => ({ + ...(await importOriginal()), + readToken, +})); + +const { authCommand } = await import("../src/commands/auth.js"); + +const HOST = "https://mychurch.church.tools"; +const originalHost = process.env.CT_HOST; +let stdout: string[]; +let stderr: string[]; + +function session(overrides: Partial<{ cookie: string; source: "cache" | "handshake" }> = {}): void { + authedSession.mockResolvedValue({ + me: { id: 1 }, + client: { + sessionCredential: () => ({ + cookie: overrides.cookie ?? "ChurchTools_mychurch=abc", + csrfToken: "csrf-1", + obtainedAt: Date.parse("2026-09-21T00:00:00.000Z"), + source: overrides.source ?? "cache", + }), + }, + }); +} + +async function run(args: string[]): Promise { + await authCommand().parseAsync(["token", ...args], { from: "user" }); +} + +beforeEach(() => { + process.env.CT_HOST = HOST; + stdout = []; + stderr = []; + vi.spyOn(process.stdout, "write").mockImplementation((chunk) => { + stdout.push(String(chunk)); + return true; + }); + vi.spyOn(process.stderr, "write").mockImplementation((chunk) => { + stderr.push(String(chunk)); + return true; + }); + // A terminal is refused on purpose; every test below is the piped case unless it says otherwise. + Object.defineProperty(process.stdout, "isTTY", { value: false, configurable: true }); + process.exitCode = 0; + readToken.mockResolvedValue(LOGIN_TOKEN); + session(); +}); + +afterEach(() => { + vi.restoreAllMocks(); + process.exitCode = 0; + if (originalHost === undefined) delete process.env.CT_HOST; + else process.env.CT_HOST = originalHost; +}); + +describe("ct auth token", () => { + it("puts the session on stdout and nothing else there", async () => { + await run([]); + expect(stdout).toHaveLength(1); + const payload = JSON.parse(stdout[0]!) as Record; + expect(payload).toMatchObject({ + host: HOST, + cookie: "ChurchTools_mychurch=abc", + csrfToken: "csrf-1", + source: "cache", + }); + // The reuse ceiling, so a consumer can cache it rather than call ct on every single request. + expect(payload.expiresAt).toBe("2026-09-21T12:00:00.000Z"); + // The permanent credential is never part of the answer. + expect(stdout[0]).not.toContain(LOGIN_TOKEN); + }); + + it("emits the bare cookie under --raw, so command substitution is safe", async () => { + await run(["--raw"]); + expect(stdout).toEqual(["ChurchTools_mychurch=abc\n"]); + // Progress belongs on stderr: `$(ct auth token --raw)` must capture the credential alone. + expect(stderr.join("")).toContain(HOST); + expect(stderr.join("")).not.toContain("ChurchTools_mychurch=abc"); + }); + + it("writes nothing to stdout and exits non-zero when no credential resolves", async () => { + readToken.mockResolvedValue(null as never); + await run([]); + expect(stdout).toEqual([]); + expect(process.exitCode).toBe(1); + // The remedy, not just the refusal — this is the case a helper hits most often. + expect(stderr.join("")).toMatch(/ct auth login/); + }); + + it("refuses a terminal, where the credential would stay in the scrollback", async () => { + Object.defineProperty(process.stdout, "isTTY", { value: true, configurable: true }); + await run([]); + expect(stdout).toEqual([]); + expect(process.exitCode).toBe(1); + expect(stderr.join("")).toMatch(/Refusing to print a credential to a terminal/); + }); + + it("prints to a terminal when explicitly asked", async () => { + Object.defineProperty(process.stdout, "isTTY", { value: true, configurable: true }); + await run(["--allow-tty", "--raw"]); + expect(stdout).toEqual(["ChurchTools_mychurch=abc\n"]); + expect(process.exitCode).toBe(0); + }); + + it("says whether a login handshake was spent", async () => { + session({ source: "handshake" }); + await run(["--raw"]); + expect(stderr.join("")).toMatch(/fresh login handshake/); + }); +}); diff --git a/tests/export/export-tf.test.ts b/tests/export/export-tf.test.ts index 8125f50..4dcff0d 100644 --- a/tests/export/export-tf.test.ts +++ b/tests/export/export-tf.test.ts @@ -211,3 +211,42 @@ describe("writeVersions: false", () => { expect(files).toContain("versions.tf"); }); }); + +/** + * The id map written beside the export (#181): the same key→id table, for ct's own resolution once + * these resources leave its state. Written under `.ct/`, not into the tofu root module — it is ct's + * input, not part of what tofu reads. + */ +describe("runExportTf — the id map", () => { + it("writes one entry per exported resource, keyed by host", async () => { + const result = await exportInto({ + mainz: row("campus", "mainz", 0), + struktur: row("group-type", "struktur", 9), + jugend: row("group", "jugend", 99), // not exportable → not in the map either + }); + expect(result.value.idMapPath).toBe(join(dir, ".ct", "ids.example.church.tools.json")); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map.campus).toEqual({ mainz: { id: 0 } }); + expect(map["group-type"]).toEqual({ struktur: { id: 9 } }); + expect(map.group).toBeUndefined(); + expect(map.$meta).toMatchObject({ host: HOST, source: "ct export tf", entries: 2 }); + }); + + it("records the HCL relabelling, which cannot be inverted by rule", async () => { + const result = await exportInto({ "3_active": row("group-type", "3_active", 4) }); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map["group-type"]).toEqual({ "3_active": { id: 4, label: "g_3_active" } }); + }); + + it("writes nothing under .ct when --no-ids is passed", async () => { + await stateWith({ mainz: row("campus", "mainz", 0) }); + const result = await runExportTf({ + cwd: dir, + statePath: "ct-state.json", + outDir: "tofu", + writeIds: false, + }); + expect(result.value.idMapPath).toBeNull(); + await expect(readdir(join(dir, ".ct"))).rejects.toThrow(/ENOENT/); + }); +}); diff --git a/tests/ids-sync.test.ts b/tests/ids-sync.test.ts new file mode 100644 index 0000000..555401f --- /dev/null +++ b/tests/ids-sync.test.ts @@ -0,0 +1,162 @@ +/** + * `ct ids sync` — refreshing the id map from OpenTofu's own state (#181). + * + * `ct export tf` can only write what ct still holds. After the cutover, tofu is the only place new + * ids appear: a campus created by `tofu apply` exists in no ct state file, so no export will ever + * mention it. Reading tofu's state closes that loop — from a file or stdin, so it works against any + * backend (the real one is S3) without ct learning to speak S3. + */ +import { describe, it, expect, afterEach, beforeEach, vi } from "vitest"; +import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { runIdsList, runIdsSync } from "../src/application/operations/ids.js"; +import { writeIdMap } from "../src/resolve/idMap.js"; + +const HOST = "https://eqrm.church.tools"; +let dir: string; + +function tfstate(resources: unknown[], serial = 3): string { + return JSON.stringify({ version: 4, serial, resources }); +} + +const CAMPUS_MAINZ = { + mode: "managed", + type: "churchtools_campus", + name: "mainz", + instances: [{ attributes: { id: 0 } }], +}; + +beforeEach(async () => { + dir = await mkdtemp(join(tmpdir(), "ct-ids-sync-")); + await mkdir(join(dir, ".ct"), { recursive: true }); + vi.stubEnv("CT_HOST", HOST); +}); + +afterEach(async () => { + vi.unstubAllEnvs(); + await rm(dir, { recursive: true, force: true }); +}); + +async function sync(state: string, opts: { dryRun?: boolean } = {}) { + const path = join(dir, "terraform.tfstate"); + await writeFile(path, state, "utf8"); + return runIdsSync({ cwd: dir, tofuState: path, dryRun: opts.dryRun }); +} + +describe("runIdsSync", () => { + it("writes the map from the state and reports what moved", async () => { + await writeIdMap( + HOST, + [ + { type: "campus", key: "mainz", id: 0 }, + { type: "campus", key: "horgen", id: 4 }, + ], + join(dir, ".ct"), + ); + const result = await sync( + tfstate([ + CAMPUS_MAINZ, + { ...CAMPUS_MAINZ, name: "egc", instances: [{ attributes: { id: 3 } }] }, + { + mode: "managed", + type: "churchtools_person_status", + name: "status_5_core", + instances: [{ attributes: { id: 6 } }], + }, + ]), + ); + expect(result.value.written).toBe(true); + expect(result.value.serial).toBe(3); + expect(result.value.added.map((e) => e.key)).toEqual(["egc", "status_5_core"]); + // Gone from tofu's state, so ct must stop claiming to resolve it: a kept id would point every + // reference at whatever holds that number now. + expect(result.value.removed.map((e) => e.key)).toEqual(["horgen"]); + const map = JSON.parse(await readFile(result.value.path, "utf8")); + expect(map.campus).toEqual({ egc: { id: 3 }, mainz: { id: 0 } }); + expect(map.$meta).toMatchObject({ host: HOST, source: "ct ids sync", entries: 3 }); + }); + + it("reports an id that changed under the same key", async () => { + await writeIdMap(HOST, [{ type: "campus", key: "mainz", id: 6 }], join(dir, ".ct")); + const result = await sync(tfstate([CAMPUS_MAINZ])); + expect(result.value.changed).toEqual([{ entry: { type: "campus", key: "mainz", id: 0 }, previousId: 6 }]); + }); + + it("maps a relabelled resource back to its ct key via the existing map", async () => { + await writeIdMap( + HOST, + [{ type: "group-type", key: "3_active", id: 4, label: "g_3_active" }], + join(dir, ".ct"), + ); + const result = await sync( + tfstate([ + { + mode: "managed", + type: "churchtools_group_type", + name: "g_3_active", + instances: [{ attributes: { id: 4 } }], + }, + ]), + ); + expect(result.value.entries).toEqual([ + { type: "group-type", key: "3_active", id: 4, label: "g_3_active" }, + ]); + expect(result.value.added).toEqual([]); + }); + + it("reads the state from stdin, so `tofu state pull | ct ids sync` works on any backend", async () => { + const result = await runIdsSync( + { cwd: dir, tofuState: "-" }, + { readStdin: async () => tfstate([CAMPUS_MAINZ]) }, + ); + expect(result.value.entries).toEqual([{ type: "campus", key: "mainz", id: 0 }]); + expect(result.value.written).toBe(true); + }); + + it("never replaces a good map with an empty one", async () => { + // The likeliest cause is the wrong workspace, and overwriting 50 working ids with nothing would + // break every reference at once — the command reports and refuses instead. + await writeIdMap(HOST, [{ type: "campus", key: "mainz", id: 0 }], join(dir, ".ct")); + const result = await sync(tfstate([])); + expect(result.value.written).toBe(false); + expect(result.warnings.map((w) => w.code)).toContain("IDS_EMPTY"); + const map = JSON.parse(await readFile(result.value.path, "utf8")); + expect(map.campus).toEqual({ mainz: { id: 0 } }); + }); + + it("warns about provider resources ct has no type for", async () => { + const result = await sync( + tfstate([ + CAMPUS_MAINZ, + { mode: "managed", type: "churchtools_group", name: "kids", instances: [{ attributes: { id: 7 } }] }, + ]), + ); + const warning = result.warnings.find((w) => w.code === "IDS_TYPE_UNSUPPORTED"); + expect(warning?.message).toContain("churchtools_group"); + }); + + it("writes nothing under --dry-run", async () => { + const result = await sync(tfstate([CAMPUS_MAINZ]), { dryRun: true }); + expect(result.value.written).toBe(false); + expect(result.value.added.map((e) => e.key)).toEqual(["mainz"]); + await expect(readFile(join(dir, ".ct", "ids.eqrm.church.tools.json"), "utf8")).rejects.toThrow(/ENOENT/); + }); + + it("says where to look rather than parsing garbage", async () => { + await expect(sync("not json")).rejects.toThrow(/not valid JSON/); + }); +}); + +describe("runIdsList", () => { + it("reports no map rather than an empty one", async () => { + const result = await runIdsList({ cwd: dir }); + expect(result.value).toEqual({ path: null, entries: [] }); + }); + + it("lists what the resolver would use", async () => { + await writeIdMap(HOST, [{ type: "campus", key: "mainz", id: 0 }], join(dir, ".ct")); + const result = await runIdsList({ cwd: dir }); + expect(result.value.entries).toEqual([{ type: "campus", key: "mainz", id: 0 }]); + }); +}); diff --git a/tests/login-throttle.test.ts b/tests/login-throttle.test.ts new file mode 100644 index 0000000..63316ad --- /dev/null +++ b/tests/login-throttle.test.ts @@ -0,0 +1,120 @@ +/** + * The login-handshake brake (#179). + * + * `ct` is one-shot, so nothing inside an invocation knows how many handshakes the last minute + * already spent — and `ct auth token` is designed to be called by another tool on every run, with no + * session cache at all off macOS. ChurchTools rate-limits logins per instance, so the count has to + * outlive the process: it lives in a cache file holding nothing but timestamps. + */ +import { describe, it, expect, afterEach } from "vitest"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + fileLoginThrottle, + MAX_PER_HOUR, + MIN_INTERVAL_MS, + throttleDir, + throttlePath, +} from "../src/auth/loginThrottle.js"; + +const HOST = "https://mychurch.church.tools"; +let dir: string | undefined; + +function cacheDir(): NodeJS.ProcessEnv { + dir = mkdtempSync(join(tmpdir(), "ct-throttle-")); + const env = { XDG_CACHE_HOME: dir }; + mkdirSync(throttleDir(env), { recursive: true }); // so a test can seed the file the throttle reads + return env; +} + +afterEach(() => { + if (dir) rmSync(dir, { recursive: true, force: true }); + dir = undefined; +}); + +describe("fileLoginThrottle", () => { + it("keeps one file per host, holding no credential", async () => { + const env = cacheDir(); + await fileLoginThrottle({ env, now: () => 1000 }).acquire(HOST); + const path = throttlePath(HOST, env); + expect(path).toContain("login-throttle.mychurch.church.tools.json"); + expect(JSON.parse(readFileSync(path, "utf8"))).toEqual({ host: HOST, recent: [1000] }); + }); + + it("waits out the minimum spacing rather than letting the instance answer 429", async () => { + const env = cacheDir(); + const slept: number[] = []; + const sleep = async (ms: number): Promise => void slept.push(ms); + const throttle = fileLoginThrottle({ env, sleep, now: () => 1000 }); + await throttle.acquire(HOST); + expect(slept).toEqual([]); + // A second handshake 500ms later waits the remaining 2.5s — a short sleep, not a failure: the + // caller gets its session either way. + await fileLoginThrottle({ env, sleep, now: () => 1500 }).acquire(HOST); + expect(slept).toEqual([MIN_INTERVAL_MS - 500]); + }); + + it("refuses once a rolling hour is full, naming when the window frees up", async () => { + const env = cacheDir(); + const start = Date.parse("2026-09-21T10:00:00.000Z"); + writeFileSync( + throttlePath(HOST, env), + JSON.stringify({ host: HOST, recent: Array.from({ length: MAX_PER_HOUR }, (_, i) => start + i) }), + "utf8", + ); + await expect(fileLoginThrottle({ env, now: () => start + 60_000 }).acquire(HOST)).rejects.toThrow( + /2026-09-21T11:00:00.000Z/, + ); + }); + + it("forgets handshakes older than the window", async () => { + const env = cacheDir(); + const start = Date.parse("2026-09-21T10:00:00.000Z"); + writeFileSync( + throttlePath(HOST, env), + JSON.stringify({ host: HOST, recent: Array.from({ length: MAX_PER_HOUR }, (_, i) => start + i) }), + "utf8", + ); + const later = start + 2 * 60 * 60 * 1000; + await fileLoginThrottle({ env, now: () => later }).acquire(HOST); + expect(JSON.parse(readFileSync(throttlePath(HOST, env), "utf8")).recent).toEqual([later]); + }); + + it("drops a timestamp from the future instead of holding the brake on", async () => { + // A clock change or a copied cache file would otherwise throttle every login until the skew ran + // out — an unreadable cache must degrade to "no throttle", never to "no logins". + const env = cacheDir(); + const now = Date.parse("2026-09-21T10:00:00.000Z"); + writeFileSync( + throttlePath(HOST, env), + JSON.stringify({ host: HOST, recent: [now + 10 * 60 * 1000] }), + "utf8", + ); + const slept: number[] = []; + await fileLoginThrottle({ env, sleep: async (ms) => void slept.push(ms), now: () => now }).acquire(HOST); + expect(slept).toEqual([]); + }); + + it("does nothing under CT_NO_LOGIN_THROTTLE=1", async () => { + const env = { ...cacheDir(), CT_NO_LOGIN_THROTTLE: "1" }; + const start = Date.parse("2026-09-21T10:00:00.000Z"); + writeFileSync( + throttlePath(HOST, env), + JSON.stringify({ host: HOST, recent: Array.from({ length: MAX_PER_HOUR }, (_, i) => start + i) }), + "utf8", + ); + await expect(fileLoginThrottle({ env, now: () => start + 1 }).acquire(HOST)).resolves.toBeUndefined(); + }); + + it("never fails a login because the cache is unwritable", async () => { + // No such directory, and none can be created under a file — the throttle is a courtesy, and a + // courtesy that breaks the tool is a bug. + const file = join(mkdtempSync(join(tmpdir(), "ct-throttle-")), "not-a-dir"); + writeFileSync(file, "x", "utf8"); + dir = file; + await expect( + fileLoginThrottle({ env: { XDG_CACHE_HOME: file }, now: () => 1 }).acquire(HOST), + ).resolves.toBeUndefined(); + }); +}); diff --git a/tests/permission-catalog-host-difference.test.ts b/tests/permission-catalog-host-difference.test.ts new file mode 100644 index 0000000..0c3315b --- /dev/null +++ b/tests/permission-catalog-host-difference.test.ts @@ -0,0 +1,221 @@ +/** + * A right absent from THIS host's catalog is a host difference, not a config error (#178). + * + * One config serves an estate whose instances do not all have the same modules installed. Before + * this, a declared right the active catalog did not define aborted the whole plan: 13 of 113 rights + * did not exist on dev (Flow and Report are not installed there), so planning dev failed entirely — + * including for the 100 rights that do apply. `ct` already handles the mirror case exactly the right + * way (a LIVE grant whose authId the catalog cannot name is reported and left untouched, never + * revoked); this pins the same posture in the declaring direction. + * + * The oracle for "is this a real right?" is ct's BUNDLED catalog, which is why every test here loads + * a per-instance capture first: without one, the active catalog IS the bundled one and "missing here" + * and "missing everywhere" are the same statement. + */ +import { describe, it, expect, afterEach, vi } from "vitest"; +import { buildPermissionPlan, desiredTuples } from "../src/permissions/plan.js"; +import { + CATALOG, + catalogVerdict, + scopeFieldVerdict, + setStrictCatalog, + useBundledCatalog, + useCatalog, +} from "../src/permissions/catalog.js"; +import { evaluateConfig } from "../src/config/context.js"; +import type { State } from "../src/state/state.js"; + +/** A right the bundled catalog defines — the stand-in for `jpmFlowManager:*` on a Flow-less host. */ +const ABSENT_HERE = "churchgroup:administer groups"; +/** A dimension the bundled catalog defines, for the `preserveUnknown` half of the same story. */ +const ABSENT_DIMENSION_HERE = "cdb_comment_viewer"; + +const state: State = { version: 1, host: "h", resources: {} }; + +/** + * A per-instance capture standing in for eqrm-dev: it has ONE right, and deliberately not the one + * the config below declares. + */ +function useHostCatalogWithout(): void { + useCatalog( + { + $meta: { + capturedFrom: "dev.church.tools", + ctVersion: "3.137.0-RC13", + capturedAt: "2026-09-21", + rightCount: 1, + }, + "churchcore:administer settings": { authId: 1, scopeField: null, revocable: true, desc: "" }, + }, + { perInstance: true }, + ); +} + +afterEach(() => { + useBundledCatalog(); // the catalog is process-global — never leak a capture into another test +}); + +describe("catalogVerdict", () => { + it("separates a host difference from a typo", () => { + useHostCatalogWithout(); + expect(catalogVerdict("churchcore:administer settings")).toBe("known"); + expect(catalogVerdict(ABSENT_HERE)).toBe("host-missing"); + // Nothing ct has ever seen defines this, so it is a typo or a right ChurchTools deleted — + // the case the hard error was written for, and the one `churchreport:edit masterdata` is. + expect(catalogVerdict("jpmFlowManager:nosuchright")).toBe("unknown"); + }); + + it("keeps every absence fatal while the bundled catalog is the active one", () => { + // No capture loaded: "absent from the active catalog" and "absent from every catalog" are the + // same statement, so a repo that has not captured its host's catalog sees today's behaviour. + expect(CATALOG[ABSENT_HERE]).toBeDefined(); + expect(catalogVerdict("churchgroup:nosuchright")).toBe("unknown"); + }); + + it("makes --strict-catalog fatal again", () => { + useHostCatalogWithout(); + setStrictCatalog(true); + expect(catalogVerdict(ABSENT_HERE)).toBe("unknown"); + expect(scopeFieldVerdict(ABSENT_DIMENSION_HERE)).toBe("unknown"); + }); +}); + +describe("desiredTuples", () => { + it("emits no tuple for a right this host does not have, and reports it", () => { + useHostCatalogWithout(); + const skips: unknown[] = []; + const tuples = desiredTuples( + { + key: "mitglied", + domainType: "group_role", + domainId: 5, + grants: ["churchcore:administer settings", ABSENT_HERE], + }, + state, + new Set(), + new Map(), + (skip) => skips.push(skip), + ); + // No tuple is the whole point: no tuple means no PUT, and because the host cannot carry a live + // row under a name it does not define, nothing lands in toDelete either. + expect(tuples).toEqual([{ authId: 1, dataId: [], type: "grant" }]); + expect(skips).toEqual([{ domainType: "group_role", key: "mitglied", name: ABSENT_HERE, kind: "right" }]); + }); + + it("still throws for a name no catalog defines", () => { + useHostCatalogWithout(); + expect(() => + desiredTuples( + { key: "mitglied", domainType: "group_role", domainId: 5, grants: ["churchreport:no such right"] }, + state, + ), + ).toThrow(/Unknown permission/); + }); + + it("throws for a host-missing right under --strict-catalog", () => { + useHostCatalogWithout(); + setStrictCatalog(true); + expect(() => + desiredTuples({ key: "mitglied", domainType: "group_role", domainId: 5, grants: [ABSENT_HERE] }, state), + ).toThrow(/Unknown permission/); + }); +}); + +describe("buildPermissionPlan", () => { + it("plans everything else and warns once per skipped declaration", async () => { + useHostCatalogWithout(); + const client = { get: vi.fn(async () => []) }; + const { items, warnings } = await buildPermissionPlan(client as never, state, [ + { + key: "mitglied", + domainType: "group_role", + domainId: 5, + grants: ["churchcore:administer settings", ABSENT_HERE], + }, + ]); + // The declaration is still planned — the skip costs it one grant, not the whole role. + expect(items[0]?.diff.toPut).toEqual([{ authId: 1, dataId: [], type: "grant" }]); + const skipWarning = warnings.find((w) => w.includes(ABSENT_HERE)); + expect(skipWarning).toMatch(/group_role "mitglied"/); + expect(skipWarning).toMatch(/absent from this host's permission catalog/); + // Both catalogs are named: without the bundled half the warning reads like a typo. + expect(skipWarning).toMatch(/3\.137\.0-RC13/); + expect(skipWarning).toMatch(/bundled catalog/); + expect(skipWarning).toMatch(/never granted, never revoked/); + expect(skipWarning).toMatch(/--strict-catalog/); + }); + + it("does not revoke the live grants of a domain whose declaration was partly skipped", async () => { + useHostCatalogWithout(); + // The host grants authId 1 directly; the declaration names it plus a right the host lacks. + const client = { + get: vi.fn(async () => [ + { + domainType: "group_role", + domainId: 5, + authId: 1, + dataId: null, + type: "grant", + meta: { modifiedPid: 9 }, + }, + ]), + }; + const { items } = await buildPermissionPlan(client as never, state, [ + { + key: "mitglied", + domainType: "group_role", + domainId: 5, + grants: ["churchcore:administer settings", ABSENT_HERE], + }, + ]); + expect(items[0]?.diff.toPut).toEqual([]); + expect(items[0]?.diff.toDelete).toEqual([]); + }); + + it("reports a preserveUnknown dimension this host has no right for", async () => { + useHostCatalogWithout(); + const client = { get: vi.fn(async () => []) }; + const { warnings } = await buildPermissionPlan(client as never, state, [ + { + key: "mitglied", + domainType: "group_role", + domainId: 5, + grants: ["churchcore:administer settings"], + preserveUnknown: [ABSENT_DIMENSION_HERE], + }, + ]); + const warning = warnings.find((w) => w.includes(ABSENT_DIMENSION_HERE)); + expect(warning).toMatch(/preserveUnknown/); + expect(warning).toMatch(/can preserve nothing here/); + }); +}); + +describe("config evaluation", () => { + it("accepts a preserveUnknown dimension that exists on the other host", async () => { + useHostCatalogWithout(); + const { permissions } = await evaluateConfig((ct) => { + ct.groupRole({ + key: "mitglied", + id: 5, + grants: ["churchcore:administer settings"], + preserveUnknown: [ABSENT_DIMENSION_HERE], + }); + }); + // Kept verbatim in the declaration: the verdict belongs to the plan, which can name the host. + expect(permissions[0]?.preserveUnknown).toEqual([ABSENT_DIMENSION_HERE]); + }); + + it("still rejects a dimension no catalog defines", async () => { + useHostCatalogWithout(); + await expect( + evaluateConfig((ct) => { + ct.groupRole({ + key: "mitglied", + id: 5, + grants: ["churchcore:administer settings"], + preserveUnknown: ["cdb_nosuchdimension"], + }); + }), + ).rejects.toThrow(/no right in the permission catalog scopes by/); + }); +}); diff --git a/tests/state-rm-command.test.ts b/tests/state-rm-command.test.ts index 72e6f79..ce644fe 100644 --- a/tests/state-rm-command.test.ts +++ b/tests/state-rm-command.test.ts @@ -15,6 +15,7 @@ import { mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { emptyState, loadState, saveState } from "../src/state/state.js"; +import { removeStateEntry } from "../src/application/operations/state.js"; const authedSession = vi.fn(async () => { throw new Error("ct state rm must never contact ChurchTools"); @@ -109,25 +110,81 @@ describe("ct state rm (#122)", () => { // A key can be named by a PERMISSION declaration without being declared as a resource. A // resources-only guard waves those through, and the breakage surfaces one command later as a // `ct plan` hard error ("does not resolve to a managed group") — after the state file was written. + // + // Still refused after #180, which stopped treating a reference as a declaration: a GROUP reference + // is the one class that genuinely cannot survive the removal, because groups are managed-only and + // have no live catalog to fall back to. The refusal now says that instead of claiming the key is + // declared — it is not — so the two cases can be told apart. it("refuses a key a permission DOMAIN still names, not just a declared resource", async () => { await writeFile( configPath, `export default (ct) => { ct.groupRole({ key: "youth_leiter", group: "youth", role: "Leiter", grants: ["churchcore:administer settings"] }); };`, ); await expect(run(["group", "youth", "--state", statePath])).rejects.toThrow( - /still declared in the config/, + /still referenced by the config as a resource that only ct's state can resolve/, ); const state = await loadState(statePath, HOST); expect(state.resources.youth).toBeDefined(); }); + // #180: the tier-0 OpenTofu cutover leaves 232 `campus:`/`groupType:`/`department:` REFERENCES + // behind in the config with the declarations gone. Those are logical refs — they ask the host for + // the id — so `--force` was being demanded for 49 of 50 entries, which suppresses the check for the + // one case it is actually meant to catch. + it("removes a campus the config only references, and says the refs now resolve live", async () => { + await writeFile( + configPath, + `export default (ct) => { ct.group({ key: "kids", name: "Kids", campus: "mainz", groupTypeId: 1 }); };`, + ); + const result = await removeStateEntry({ type: "campus", key: "mainz", statePath, configPath }); + expect(result.value.removed).toBe(true); + const warning = result.warnings.find((w) => w.code === "STILL_REFERENCED"); + expect(warning?.message).toMatch(/references "mainz" 1 time\(s\) but does not declare it/); + expect(warning?.message).toMatch(/resolve against https:\/\/mychurch\.church\.tools by name/); + const state = await loadState(statePath, HOST); + expect(state.resources.mainz).toBeUndefined(); + }); + + // The guard's original purpose, unchanged: a real declaration means the next plan would propose + // CREATING something that already exists on this host. + it("still refuses a campus the config DECLARES", async () => { + await writeFile(configPath, `export default (ct) => { ct.campus({ key: "mainz", name: "Mainz" }); };`); + await expect(run(["campus", "mainz", "--state", statePath])).rejects.toThrow( + /still declared in the config/, + ); + const state = await loadState(statePath, HOST); + expect(state.resources.mainz).toBeDefined(); + }); + + // Declarations are matched on type AND key: a `campus` declaration says nothing about what + // removing a group-type of the same name would do. + it("does not let a declaration of another type block a removal", async () => { + await writeFile(configPath, `export default (ct) => { ct.campus({ key: "mainz", name: "Mainz" }); };`); + const state = await loadState(statePath, HOST); + state.resources["mainz-type"] = { + type: "group-type", + id: 12, + key: "mainz", + fields: {}, + adoptedAt: "t", + updatedAt: "t", + }; + // Re-key the entry under the colliding key so the removal targets group-type/mainz. + delete state.resources.mainz; + state.resources.mainz = { ...state.resources["mainz-type"]! }; + delete state.resources["mainz-type"]; + await saveState(statePath, state); + await run(["group-type", "mainz", "--state", statePath]); + expect((await loadState(statePath, HOST)).resources.mainz).toBeUndefined(); + }); + it("refuses a key a permission SCOPE still names", async () => { await writeFile( configPath, `export default (ct) => { ct.groupRole({ key: "p", id: 77, grants: [{ right: "churchgroup:view group", scope: ["youth"] }] }); };`, ); await expect(run(["group", "youth", "--state", statePath])).rejects.toThrow( - /still declared in the config/, + /still referenced by the config as a resource that only ct's state can resolve/, ); const state = await loadState(statePath, HOST); expect(state.resources.youth).toBeDefined(); diff --git a/tests/tofu-id-map.test.ts b/tests/tofu-id-map.test.ts new file mode 100644 index 0000000..c318242 --- /dev/null +++ b/tests/tofu-id-map.test.ts @@ -0,0 +1,204 @@ +/** + * Resolving OpenTofu-owned resources through a committed id map (#181). + * + * This is the last step of the tier-0 cutover, and the one that could not be done by hand. When the + * five tier-0 tables leave `ct.config.ts` AND their 50 entries leave `ct-state..json`, the + * references that stay behind fall back to matching the key against the live object's NAME — and ct's + * tier-0 keys were never name-derived, so for person statuses that can never work: + * + * key live name + * status_unbekannt Unbekannt + * status_5_core 5 - Core + * + * A literal id is no escape either: one config serves two hosts and 39 of 43 tier-0 ids differ. + * So the map carries the exact key→id table, per host, committed. + */ +import { describe, it, expect, afterEach } from "vitest"; +import { mkdirSync, mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { idMapPath, loadIdMap, parseIdMap, renderIdMap, writeIdMap } from "../src/resolve/idMap.js"; +import { readTfState } from "../src/resolve/tfstate.js"; +import { Resolver } from "../src/resolve/resolver.js"; +import { ref } from "../src/resolve/refs.js"; +import type { State } from "../src/state/state.js"; + +const HOST = "https://eqrm.church.tools"; +let dir: string | undefined; + +const state: State = { version: 1, host: HOST, resources: {} }; + +afterEach(() => { + if (dir) rmSync(dir, { recursive: true, force: true }); + dir = undefined; +}); + +function tempDir(): string { + dir = mkdtempSync(join(tmpdir(), "ct-ids-")); + mkdirSync(join(dir, ".ct"), { recursive: true }); + return join(dir, ".ct"); +} + +/** The tier-0 shape the exporter produces, with one relabelled key. */ +const ENTRIES = [ + { type: "campus", key: "mainz", id: 0 }, + { type: "person-status", key: "status_unbekannt", id: 0 }, + { type: "person-status", key: "status_5_core", id: 6 }, + { type: "group-type", key: "3_groupactive", id: 9, label: "g_3_groupactive" }, +]; + +describe("the id map file", () => { + it("is one file per host, beside the per-instance permission catalog", () => { + expect(idMapPath(HOST, ".ct")).toBe(".ct/ids.eqrm.church.tools.json"); + expect(idMapPath("https://eqrm-dev.church.tools/", ".ct")).toBe(".ct/ids.eqrm-dev.church.tools.json"); + }); + + it("renders byte-stably, sorted by type then key", () => { + const first = renderIdMap(HOST, ENTRIES, "ct export tf"); + const shuffled = renderIdMap(HOST, [...ENTRIES].reverse(), "ct export tf"); + expect(first).toBe(shuffled); + const parsed = JSON.parse(first) as Record; + expect(Object.keys(parsed)).toEqual(["$meta", "campus", "group-type", "person-status"]); + // The label rides along only where it differs — a relabelling is many-to-one, so it cannot be + // re-derived from the label when reading a tofu state back. + expect(parsed["group-type"]).toEqual({ "3_groupactive": { id: 9, label: "g_3_groupactive" } }); + expect(parsed.campus).toEqual({ mainz: { id: 0 } }); + }); + + it("id 0 survives the round trip", async () => { + // `campus/mainz` is id 0 on prod and `person-status/status_unbekannt` is 0 too: any code that + // treats a falsy id as "missing" breaks precisely the references this feature exists for. + const d = tempDir(); + await writeIdMap(HOST, ENTRIES, d); + const loaded = await loadIdMap(HOST, d); + expect(loaded?.ids.get("campus\u0000mainz")).toBe(0); + expect(loaded?.ids.get("person-status\u0000status_unbekannt")).toBe(0); + }); + + it("returns null when the repo has no map, so nothing changes for anyone else", async () => { + expect(await loadIdMap(HOST, tempDir())).toBeNull(); + }); + + it("refuses a map generated for another host", () => { + // 39 of 43 tier-0 ids differ between the two hosts, so applying dev's map to prod would resolve + // every reference to a real, wrong resource — undetectable downstream. + const raw = renderIdMap("https://eqrm-dev.church.tools", ENTRIES, "ct export tf"); + expect(() => parseIdMap(raw, ".ct/ids.json", HOST)).toThrow(/was generated for https:\/\/eqrm-dev/); + }); + + it("refuses a malformed entry rather than resolving to NaN", () => { + const raw = JSON.stringify({ campus: { mainz: { id: "0" } } }); + expect(() => parseIdMap(raw, ".ct/ids.json", HOST)).toThrow(/campus "mainz" has no numeric id/); + }); +}); + +describe("Resolver", () => { + const client = { + get: async () => [], + getAll: async () => ({ data: [{ id: 99, name: "Unbekannt" }] }), + }; + + it("resolves a key the live name cannot produce", async () => { + const map = parseIdMap(renderIdMap(HOST, ENTRIES, "ct export tf"), ".ct/ids.json", HOST); + const resolver = new Resolver({ client: client as never, state, desired: [], host: HOST, idMap: map }); + expect(await resolver.resolve(ref.personStatus("status_5_core"), "site")).toBe(6); + expect(await resolver.resolve(ref.campus("mainz"), "site")).toBe(0); + }); + + it("still prefers ct's own state over the map", async () => { + // ct never stops trusting what it owns: a resource still in ct's state is ct's, and a stale map + // entry must not overrule it. + const owned: State = { + version: 1, + host: HOST, + resources: { + mainz: { type: "campus", id: 42, key: "mainz", fields: {}, adoptedAt: "t", updatedAt: "t" }, + }, + }; + const map = parseIdMap(renderIdMap(HOST, ENTRIES, "ct export tf"), ".ct/ids.json", HOST); + const resolver = new Resolver({ + client: client as never, + state: owned, + desired: [], + host: HOST, + idMap: map, + }); + expect(await resolver.resolve(ref.campus("mainz"), "site")).toBe(42); + }); + + it("prefers the map over a live name match", async () => { + // The map is exact; the catalog match is a guess at a slug. `status_unbekannt` would resolve to + // the live "Unbekannt" row (#99) only by accident of naming — and does not on the real instance. + const map = parseIdMap( + renderIdMap(HOST, [{ type: "person-status", key: "unbekannt", id: 7 }], "ct export tf"), + ".ct/ids.json", + HOST, + ); + const resolver = new Resolver({ client: client as never, state, desired: [], host: HOST, idMap: map }); + expect(await resolver.resolve(ref.personStatus("unbekannt"), "site")).toBe(7); + }); + + it("names the map, and how to refresh it, when a key is in neither", async () => { + const map = parseIdMap(renderIdMap(HOST, ENTRIES, "ct export tf"), ".ct/ids.json", HOST); + const resolver = new Resolver({ client: client as never, state, desired: [], host: HOST, idMap: map }); + await expect(resolver.resolve(ref.campus("horgen"), "site")).rejects.toThrow( + /\.ct\/ids\.json has no campus "horgen" either.*ct ids sync/s, + ); + }); +}); + +describe("readTfState", () => { + const tfstate = { + version: 4, + serial: 12, + resources: [ + { + mode: "managed", + type: "churchtools_person_status", + name: "status_unbekannt", + instances: [{ attributes: { id: 0, name: "Unbekannt" } }], + }, + { + mode: "managed", + type: "churchtools_group_type", + name: "g_3_groupactive", + instances: [{ attributes: { id: 9 } }], + }, + // A data source describes a read, not a managed object. + { + mode: "data", + type: "churchtools_campus", + name: "elsewhere", + instances: [{ attributes: { id: 5 } }], + }, + // Mid-create or tainted: no id to map. + { mode: "managed", type: "churchtools_campus", name: "pending", instances: [{ attributes: {} }] }, + // A provider resource ct has no type for — reported, never fatal. + { mode: "managed", type: "churchtools_group", name: "kids", instances: [{ attributes: { id: 3 } }] }, + { mode: "managed", type: "aws_s3_bucket", name: "state", instances: [{ attributes: { id: "b" } }] }, + ], + }; + + it("maps managed churchtools resources back to ct keys", () => { + // tofu's label is mapped back through the previous map, because the exporter's relabelling + // (`3_groupactive` → `g_3_groupactive`) is many-to-one and cannot be inverted by rule. + const previous = parseIdMap(renderIdMap(HOST, ENTRIES, "ct export tf"), ".ct/ids.json", HOST); + const { entries, unmapped, serial } = readTfState(tfstate, previous.keysByLabel); + expect(entries).toEqual([ + { type: "person-status", key: "status_unbekannt", id: 0 }, + { type: "group-type", key: "3_groupactive", id: 9, label: "g_3_groupactive" }, + ]); + expect(unmapped).toEqual(["churchtools_group"]); + expect(serial).toBe(12); + }); + + it("takes the label as the key when no previous map recorded a relabelling", () => { + const { entries } = readTfState(tfstate); + expect(entries.map((e) => e.key)).toEqual(["status_unbekannt", "g_3_groupactive"]); + }); + + it("rejects something that is not a tofu state", () => { + expect(() => readTfState([])).toThrow(/expected a JSON object/); + expect(() => readTfState({ resources: {} })).toThrow(/"resources" is not an array/); + }); +}); From 0a8283d0cbba73e21a2c23caf560550459074b4e Mon Sep 17 00:00:00 2001 From: Felix Kotschenreuther Date: Mon, 21 Sep 2026 10:18:54 +0200 Subject: [PATCH 2/2] fix: stop the id map losing ids it is the only record of MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings on #183, each reproduced against the built binary before being fixed and pinned by a test that fails without it. The id map is the only thing resolving a leftover `campus:` or `personStatus:` reference once tier-0 leaves ct's state, and its keys are not name-derived, so a dropped entry is a hard plan error rather than a degraded guess. Three ways it was dropped: - a partial `ct export tf --only ` rewrote the whole file from that run's entries, so exporting one type at a time discarded every type it had not reached yet. Types the run covered are now rewritten wholesale, types it never selected are carried over. - a full export against the POST-CUTOVER state — tier-0 gone from the config and the state file, which is the end state the map exists for — found nothing and wrote `entries: 0` over a good map. An export that maps nothing now keeps what is there and says so, matching the guard `ct ids sync` already had. - `Number(attributes.id)` accepted null, "", [] and false as 0, and 0 is a real ChurchTools id, so a mid-create or tainted tofu resource became a reference silently resolving to the wrong live object. A `for_each` block likewise mapped its block label to one arbitrary instance id and reported every real key as removed; both are now refused and reported. Also, from the same review: - the login throttle's hourly cap gates every ct command, not just `ct auth token`, and there is no session cache off macOS — so 21 ct invocations in an hour on Linux CI hard-failed where they had always worked. Cap raised to 120: the 3s spacing is what protects the instance, this is the backstop for a loop that keeps going. The unlocked read-modify-write, and the fact that CI is *not* exempt, are now stated rather than implied away. - `ct ids sync` refused to run while the map it exists to replace was malformed; it now regenerates and warns. - `--dry-run` exited 0 on the empty-state condition the real run exits 1 on, so a CI gate built on it passed exactly when it should fail. - `--tofu-state` resolved against process.cwd() rather than the project cwd, unlike every other path in that operation. - `--strict-catalog` set a process global nothing reset, which in the HTTP adapter contracts.ts anticipates would leave every later plan strict; now restored in a finally. - `ct state rm` checked the key-only state-only-reference refusal before the type-and-key declaredness one, so a key that was both got the vaguer message. --- docs/opentofu-migration.md | 33 +++++- src/application/operations/export-tf.ts | 64 +++++++++++- src/application/operations/ids.ts | 46 +++++++-- src/application/operations/plan.ts | 127 +++++++++++++----------- src/application/operations/state.ts | 18 ++-- src/auth/loginThrottle.ts | 26 ++++- src/commands/ids.ts | 11 +- src/resolve/tfstate.ts | 29 +++++- tests/export/export-tf.test.ts | 54 ++++++++++ tests/ids-sync.test.ts | 23 +++++ tests/tofu-id-map.test.ts | 52 ++++++++++ 11 files changed, 394 insertions(+), 89 deletions(-) diff --git a/docs/opentofu-migration.md b/docs/opentofu-migration.md index eec76fe..98974fc 100644 --- a/docs/opentofu-migration.md +++ b/docs/opentofu-migration.md @@ -78,6 +78,19 @@ ct export tf --env prod --no-ids # …without the map, if your repo generates The map is written under `.ct/`, not into the tofu output directory: it is `ct`'s input, not part of the root module tofu reads. +A partial export (`--only campus`) rewrites only the types it was asked for and +carries the rest of the map over untouched — otherwise a type-at-a-time cutover +would drop the ids of every type it had not reached yet. Within a type the +rewrite is wholesale: a resource that has left `ct`'s state has left `ct`'s +ownership, and its id goes with it. + +An export that maps **nothing** leaves an existing map alone and says so, rather +than emptying it. That is not an edge case but the end state: once tier-0 is gone +from `ct.config.ts` and `ct-state..json`, every `ct export tf` exports zero +resources, and the map it would overwrite is the only thing still resolving the +references that stayed behind. From that point on the map is refreshed from tofu, +with `ct ids sync` — not from `ct`. + ### Keeping it current An export can only describe what `ct` still holds. Once tier-0 belongs to tofu, @@ -157,15 +170,25 @@ _is_ needed, a cross-process brake keeps it from becoming a burst: - handshakes against one host are spaced at least 3s apart (waited out, not an error); -- more than 20 in a rolling hour is refused, naming when the window frees up — +- more than 120 in a rolling hour is refused, naming when the window frees up — that is a runaway loop, and hammering a throttled instance only lengthens the outage for everyone on it; - `CT_NO_LOGIN_THROTTLE=1` disables it, for a CI job that knows it runs alone. The counter lives in `$XDG_CACHE_HOME/ct-cli/login-throttle..json` and holds nothing but timestamps. Deleting it, or being unable to write it, simply means no -throttle. - -**CI is unaffected and needs none of this.** A CI job passes the token explicitly -from a GitHub secret, which is already storage-free; this path exists for local +throttle. It is read-modify-written without a lock, so it bounds a _sequence_ of +invocations rather than a simultaneous burst: two `ct` processes starting at the +same instant are spaced no better than not at all. + +**The brake sits in the login handshake, so it covers every `ct` command**, not +just `ct auth token` — and on Linux and Windows there is no session cache, so +there each invocation is one handshake. That is why the hourly cap is 120 rather +than a number sized for a credential helper alone: a pipeline should never reach +it, while a runaway loop passes it in about six minutes. A CI job that runs more +`ct` invocations than that against one host in an hour should set +`CT_NO_LOGIN_THROTTLE=1`. + +A CI job otherwise needs none of this: it passes the token explicitly from a +GitHub secret, which is already storage-free. This path exists for local development, where the alternative was a token on disk. diff --git a/src/application/operations/export-tf.ts b/src/application/operations/export-tf.ts index 2191da3..98aa157 100644 --- a/src/application/operations/export-tf.ts +++ b/src/application/operations/export-tf.ts @@ -6,7 +6,7 @@ import { renderImports, type ImportTarget } from "../../export/imports.js"; import { EXPORTABLE_TYPES, fileForType, OWNED_FILES } from "../../export/layout.js"; import { renderVersions } from "../../export/provider.js"; import { CATALOG_DIR } from "../../permissions/catalog-store.js"; -import { writeIdMap, type IdMapEntry } from "../../resolve/idMap.js"; +import { loadIdMap, writeIdMap, type IdMapEntry } from "../../resolve/idMap.js"; import { loadState } from "../../state/state.js"; import type { CtWarning, OperationResult, ProjectRequest } from "../contracts.js"; import { resolveProject } from "../project.js"; @@ -66,6 +66,33 @@ function address(type: string, key: string): string { return `${hclType(type)}.${hclLabel(key)}`; } +/** + * Keep the id-map entries a PARTIAL export never looked at (#181). + * + * `--only campus` describes campuses and nothing else, but the map it writes is the whole file — so + * rewriting it from this run's entries alone drops every `person-status`, `group-type` and + * `department` id the previous export put there. Those are exactly the entries that keep a leftover + * `personStatus: "status_unbekannt"` resolving once tier-0 leaves ct's state, and their keys are not + * name-derived, so the live-name fallback cannot cover for them: the next `ct plan` hard-errors. + * + * So a type this run COVERED is rewritten wholesale (a resource that left state has left ct's + * ownership, and its id must go with it), while a type it never selected is carried over untouched. + * A full export covers every exportable type, so it carries nothing and never reads the old map — + * which is also why a corrupt or foreign map can only fail the partial case, the one that cannot do + * its job without reading it. + */ +async function carryOverUncoveredIds( + host: string, + dir: string, + covered: readonly string[], + fresh: readonly IdMapEntry[], +): Promise { + if (EXPORTABLE_TYPES.every((type) => covered.includes(type))) return [...fresh]; + const previous = await loadIdMap(host, dir); + if (!previous) return [...fresh]; + return [...previous.entries.filter((entry) => !covered.includes(entry.type)), ...fresh]; +} + export async function runExportTf(request: ExportTfRequest): Promise { // resolveProject gives the env profile's state path AND the host the state // must belong to; loadState asserts that pairing, so an export can never @@ -157,10 +184,26 @@ export async function runExportTf(request: ExportTfRequest): Promise 0) { + idMapPath = existing.path; + idMapKept = true; + } else { + idMapPath = await writeIdMap(project.host, entries, catalogDir); + } + } const warnings: CtWarning[] = skipped.map(({ type, count }) => ({ code: "EXPORT_TYPE_UNSUPPORTED", @@ -170,6 +213,17 @@ export async function runExportTf(request: ExportTfRequest): Promise> = null; + let previousUnreadable: string | null = null; + try { + previous = await (dependencies.loadIdMap ?? loadIdMap)(project.host, dir); + } catch (err) { + previousUnreadable = (err as Error).message; + } + const { entries, unmapped, multiInstance, serial } = readTfState( + parsed, + previous?.keysByLabel ?? new Map(), + ); const previousById = new Map((previous?.entries ?? []).map((e) => [entryId(e), e])); const nextById = new Map(entries.map((e) => [entryId(e), e])); @@ -98,6 +112,26 @@ export async function runIdsSync( const removed = (previous?.entries ?? []).filter((e) => !nextById.has(entryId(e))); const warnings: CtWarning[] = []; + if (previousUnreadable) { + warnings.push({ + code: "IDS_PREVIOUS_UNREADABLE", + message: + `The existing id map could not be read, so this sync rewrites it from scratch: ` + + `${previousUnreadable} Keys the exporter relabelled fall back to their HCL label.`, + details: { reason: previousUnreadable }, + }); + } + if (multiInstance.length > 0) { + warnings.push({ + code: "IDS_MULTI_INSTANCE", + message: + `Ignored ${multiInstance.length} resource block(s) with more than one instance ` + + `(${multiInstance.join(", ")}): a for_each/count block's label is not a resource key, so ct ` + + `cannot tell which id belongs to which key. Declare those resources individually, or pin the ` + + `references to numeric ids.`, + details: { blocks: multiInstance }, + }); + } if (unmapped.length > 0) { warnings.push({ code: "IDS_TYPE_UNSUPPORTED", @@ -119,7 +153,7 @@ export async function runIdsSync( const written = !request.dryRun && entries.length > 0; const path = written ? await (dependencies.writeIdMap ?? writeIdMap)(project.host, entries, dir, "ct ids sync") - : (previous?.path ?? join(dir, `ids.${project.host}.json`)); + : (previous?.path ?? idMapPath(project.host, dir)); return { operation: "ids", diff --git a/src/application/operations/plan.ts b/src/application/operations/plan.ts index 3ca9e54..b0b78ae 100644 --- a/src/application/operations/plan.ts +++ b/src/application/operations/plan.ts @@ -3,7 +3,7 @@ import { authedSession, type AuthedSession } from "../../api/session.js"; import { loadConfig } from "../../config/load.js"; import { buildPlan } from "../../engine/build.js"; import { summarize, type Plan, type PlanAction } from "../../engine/types.js"; -import { setStrictCatalog } from "../../permissions/catalog.js"; +import { STRICT_CATALOG, setStrictCatalog } from "../../permissions/catalog.js"; import { CATALOG_DIR, loadHostCatalog } from "../../permissions/catalog-store.js"; import { buildPermissionPlan, type PermissionPlanItem } from "../../permissions/plan.js"; import { loadIdMap } from "../../resolve/idMap.js"; @@ -121,67 +121,78 @@ export async function buildPlanContext( observer.emit({ type: "phase-started", phase: "load-project" }); // Set BEFORE the catalog and the config load: the config's own `preserveUnknown` validation reads // it at eval time (config/context.ts), and it must describe the catalog that is about to be loaded. + // + // Restored in the `finally` below, so the flag lives no longer than the build that asked for it. + // For a one-shot CLI that is merely tidy, but `contracts.ts` anticipates an HTTP adapter, and in a + // long-lived process one `--strict-catalog` plan would otherwise leave EVERY later plan strict — + // a setting silently outliving its request, on the exact flag whose whole job is to decide whether + // a plan fails or warns. + const previousStrict = STRICT_CATALOG; setStrictCatalog(request.strictCatalog ?? false); - const catalogPath = await (dependencies.loadHostCatalog ?? loadHostCatalog)( - project.host, - join(project.cwd, CATALOG_DIR), - ); - // Loaded beside the permission catalog and from the same directory: both are committed, per-host - // artefacts a consumer repo keeps under `.ct/`. - const idMap = await (dependencies.loadIdMap ?? loadIdMap)(project.host, join(project.cwd, CATALOG_DIR)); - const { - resources: desired, - permissions, - configDir, - } = await (dependencies.loadConfig ?? loadConfig)(project.configPath); - const state = await (dependencies.loadState ?? loadState)(project.statePath, project.host); - const { client } = await (dependencies.authedSession ?? authedSession)(); - const resolver = (dependencies.createResolver ?? ((options) => new Resolver(options)))({ - client, - state, - desired, - host: project.host, - idMap, - }); - - observer.emit({ type: "phase-started", phase: "build-plan" }); - const [resourceResult, permissionResult] = await Promise.all([ - (dependencies.buildPlan ?? buildPlan)(client, state, desired, { configDir, resolver }), - (dependencies.buildPermissionPlan ?? buildPermissionPlan)( + try { + const catalogPath = await (dependencies.loadHostCatalog ?? loadHostCatalog)( + project.host, + join(project.cwd, CATALOG_DIR), + ); + // Loaded beside the permission catalog and from the same directory: both are committed, per-host + // artefacts a consumer repo keeps under `.ct/`. + const idMap = await (dependencies.loadIdMap ?? loadIdMap)(project.host, join(project.cwd, CATALOG_DIR)); + const { + resources: desired, + permissions, + configDir, + } = await (dependencies.loadConfig ?? loadConfig)(project.configPath); + const state = await (dependencies.loadState ?? loadState)(project.statePath, project.host); + const { client } = await (dependencies.authedSession ?? authedSession)(); + const resolver = (dependencies.createResolver ?? ((options) => new Resolver(options)))({ client, state, - permissions, desired, - resolver, - client.version ?? undefined, - ), - ]); - const fetchErrors = [...resourceResult.fetchErrors, ...permissionResult.fetchErrors]; - const warnings: CtWarning[] = permissionResult.warnings.map((message) => ({ - code: "PERMISSION_CATALOG", - message, - })); + host: project.host, + idMap, + }); - return { - client, - state, - actual: resourceResult.actual, - result: { - operation: "plan", - project, - warnings, - value: { - plan: resourceResult.plan, - permissions: permissionResult.items, - summary: summarizePlan(resourceResult.plan, permissionResult.items), - complete: fetchErrors.length === 0, - fetchErrors, - churchToolsVersion: client.version, - stateHost: state.host, - buildWarnings: resourceResult.warnings ?? [], - permissionCatalogPath: catalogPath, - tofuIdMapPath: idMap?.path ?? null, + observer.emit({ type: "phase-started", phase: "build-plan" }); + const [resourceResult, permissionResult] = await Promise.all([ + (dependencies.buildPlan ?? buildPlan)(client, state, desired, { configDir, resolver }), + (dependencies.buildPermissionPlan ?? buildPermissionPlan)( + client, + state, + permissions, + desired, + resolver, + client.version ?? undefined, + ), + ]); + const fetchErrors = [...resourceResult.fetchErrors, ...permissionResult.fetchErrors]; + const warnings: CtWarning[] = permissionResult.warnings.map((message) => ({ + code: "PERMISSION_CATALOG", + message, + })); + + return { + client, + state, + actual: resourceResult.actual, + result: { + operation: "plan", + project, + warnings, + value: { + plan: resourceResult.plan, + permissions: permissionResult.items, + summary: summarizePlan(resourceResult.plan, permissionResult.items), + complete: fetchErrors.length === 0, + fetchErrors, + churchToolsVersion: client.version, + stateHost: state.host, + buildWarnings: resourceResult.warnings ?? [], + permissionCatalogPath: catalogPath, + tofuIdMapPath: idMap?.path ?? null, + }, }, - }, - }; + }; + } finally { + setStrictCatalog(previousStrict); + } } diff --git a/src/application/operations/state.ts b/src/application/operations/state.ts index c633049..ac73431 100644 --- a/src/application/operations/state.ts +++ b/src/application/operations/state.ts @@ -155,6 +155,17 @@ export async function removeStateEntry( project.configPath, dependencies, ); + // Declaredness is checked FIRST because it is the more specific verdict: it is matched on + // type AND key, while the state-only reference check below can only match on key. A key that + // is both declared and referenced would otherwise be refused with the vaguer of the two + // messages, naming a consequence that is not the main one. + if (declared.has(declarationId(request.type, request.key))) { + throw new Error( + `"${request.key}" is still declared in the config, so removing it from state would make the next ` + + `plan propose CREATING a resource that already exists on this host. Remove the ` + + `declaration first, or pass --force if you are deleting both in the same change.`, + ); + } if (stateOnlyRefs.has(request.key)) { throw new Error( `"${request.key}" is still referenced by the config as a resource that only ct's state can ` + @@ -163,13 +174,6 @@ export async function removeStateEntry( `if you are deleting both in the same change.`, ); } - if (declared.has(declarationId(request.type, request.key))) { - throw new Error( - `"${request.key}" is still declared in the config, so removing it from state would make the next ` + - `plan propose CREATING a resource that already exists on this host. Remove the ` + - `declaration first, or pass --force if you are deleting both in the same change.`, - ); - } // Referenced but not declared — allowed, and worth saying out loud: every one of those refs // now resolves against the LIVE host by name, which is a different resolution path than the // state lookup it had a moment ago (see #181). A key whose live name does not slug back to it diff --git a/src/auth/loginThrottle.ts b/src/auth/loginThrottle.ts index bb7fcce..645bf10 100644 --- a/src/auth/loginThrottle.ts +++ b/src/auth/loginThrottle.ts @@ -16,6 +16,16 @@ * is a runaway loop, not a burst, and continuing to hammer a throttled instance only lengthens * the outage for everyone on it. * + * ## What it does not do + * + * The counter is read, modified and written without a lock, so it bounds a SEQUENCE of invocations, + * not a simultaneous burst: two processes that read the same file at the same moment compute the + * same wait, sleep it together, and the second write drops the first one's timestamp. Truly parallel + * handshakes are therefore spaced no better than not at all, and the hourly count under-reports them. + * Holding a lock across a network handshake is its own hazard (a crashed `ct` would wedge every later + * one), and the 429 this exists to avoid is driven by sustained rate, not by two coincident logins — + * so the honest bound is the useful one, and it is stated here rather than implied away. + * * Deliberately NOT in the Keychain: there is no secret here (see the blob shape below), and a * Keychain read is an ACL prompt away from being the very thing that makes the CLI unusable in a * script. A cache file that a user deletes, or that never appears at all, simply means no throttle — @@ -28,8 +38,20 @@ import { hostSlug } from "../permissions/catalog-store.js"; /** Minimum spacing between two handshakes against one host. Waited out, never an error. */ export const MIN_INTERVAL_MS = 3_000; -/** Handshakes per rolling hour per host before `ct` refuses to add to the pile. */ -export const MAX_PER_HOUR = 20; +/** + * Handshakes per rolling hour per host before `ct` refuses to add to the pile. + * + * Sized for a CI pipeline, not for `ct auth token` alone, because the gate sits in + * `CtClient.performLogin` and therefore covers EVERY ct command. On Linux and Windows there is no + * session cache at all (`sessionStore` is Keychain-only by design), so on CI each invocation costs + * one handshake: at 20/hour a pipeline whose 21st `ct plan`/`ct get`/`ct apply` ran inside the hour + * would start failing on ct's own error, somewhere it had always worked. + * + * The number that matters for the instance is the SPACING above — it already caps a runaway loop at + * 20 handshakes a minute — so this is the backstop for a loop that keeps going, not the primary + * brake. A loop reaches 120 in about six minutes; a real pipeline does not reach it at all. + */ +export const MAX_PER_HOUR = 120; const HOUR_MS = 60 * 60 * 1000; /** What a throttle needs from its caller. Injected so a client in a test never touches the disk. */ diff --git a/src/commands/ids.ts b/src/commands/ids.ts index 0a1dcb0..42a7c5d 100644 --- a/src/commands/ids.ts +++ b/src/commands/ids.ts @@ -42,12 +42,21 @@ export function idsCommand(): Command { const serial = value.serial === null ? "" : ` (tofu state serial ${value.serial})`; if (value.written) { success(`${where}: ${value.entries.length} ids from ${project.host}${serial}.`); + } else if (value.entries.length === 0) { + // Not "wrote 0 ids": the empty-map guard below REFUSES. A dry run has to name the outcome + // it would reach; a real run has to name the one it did reach, in the tense it happened in. + const verb = opts.dryRun ? "Would refuse to write" : "Refused to write"; + info(`${verb} ${where}: the state maps to no ids${serial}.`); } else { info(`Would write ${where}: ${value.entries.length} ids from ${project.host}${serial}.`); } // An empty read never overwrites a good map — the likeliest cause is the wrong workspace, // and replacing 50 working ids with nothing would break every reference at once. - if (!value.written && !opts.dryRun) process.exitCode = 1; + // + // The dry run exits non-zero on the SAME condition, rather than only the real run: a CI gate + // built on `--dry-run` exists to catch exactly this, and an exit code that disagrees with the + // run it is predicting would pass the check precisely when the real thing would fail it. + if (value.entries.length === 0) process.exitCode = 1; } catch (caught) { error(formatError(caught)); process.exitCode = 1; diff --git a/src/resolve/tfstate.ts b/src/resolve/tfstate.ts index 3d5ca5b..806d832 100644 --- a/src/resolve/tfstate.ts +++ b/src/resolve/tfstate.ts @@ -24,6 +24,8 @@ export interface TfStateReadResult { entries: IdMapEntry[]; /** Provider resources in the state whose type ct has no mapping for — reported, never fatal. */ unmapped: string[]; + /** `for_each`/`count` blocks, whose label is not a resource key — reported, never guessed at. */ + multiInstance: string[]; /** Terraform's own serial, if present, so a sync can say which state version it read. */ serial: number | null; } @@ -32,7 +34,7 @@ interface TfResource { mode?: string; type?: string; name?: string; - instances?: { attributes?: Record }[]; + instances?: { attributes?: Record; index_key?: unknown }[]; } /** @@ -56,6 +58,7 @@ export function readTfState( } const entries: IdMapEntry[] = []; const unmapped = new Set(); + const multiInstance = new Set(); for (const raw of (state.resources ?? []) as TfResource[]) { // `data` blocks describe reads, not managed objects: their id is whatever the data source // happened to return, and nothing in the config keys off it. @@ -68,10 +71,25 @@ export function readTfState( if (raw.type.startsWith("churchtools_")) unmapped.add(raw.type); continue; } - const attributes = raw.instances?.[0]?.attributes ?? {}; - const id = Number(attributes.id); - // A resource in state with no numeric id is mid-create or tainted; an unresolvable reference is - // a better outcome than a reference resolved to NaN. + const instances = raw.instances ?? []; + // A `for_each`/`count` block is ONE resource with many instances, and its `name` is the block's + // (`campuses`), not any resource's key. Taking `[0]` would map that one label to one arbitrary + // id and report every real key as removed — and because the result is non-empty, the + // empty-map guard in `ct ids sync` would not catch it. Report it and map nothing. + if (instances.length > 1 || instances[0]?.index_key !== undefined) { + multiInstance.add(`${raw.type}.${raw.name}`); + continue; + } + const attributes = instances[0]?.attributes ?? {}; + // NOT `Number(attributes.id)`: `Number(null)`, `Number("")`, `Number([])` and `Number(false)` are + // all 0, and `Number.isFinite(0)` is true — so a resource that is mid-create, tainted or + // hand-edited would be written into the map as id 0. Zero is a REAL ChurchTools id here (the + // comment-viewer "Alle", and campus 0 in ct's own fixtures), so the reference would resolve + // silently to the wrong live object instead of failing. Only a genuine number or numeric string. + const rawId = attributes.id; + if (typeof rawId !== "number" && !(typeof rawId === "string" && rawId.trim() !== "")) continue; + const id = Number(rawId); + // Mid-create or tainted: an unresolvable reference is a better outcome than a wrong one. if (!Number.isFinite(id)) continue; const key = keysByLabel.get(`${ctType}\u0000${raw.name}`) ?? raw.name; entries.push(key === raw.name ? { type: ctType, key, id } : { type: ctType, key, id, label: raw.name }); @@ -79,6 +97,7 @@ export function readTfState( return { entries, unmapped: [...unmapped].sort(), + multiInstance: [...multiInstance].sort(), serial: typeof state.serial === "number" ? state.serial : null, }; } diff --git a/tests/export/export-tf.test.ts b/tests/export/export-tf.test.ts index 4dcff0d..9ff453f 100644 --- a/tests/export/export-tf.test.ts +++ b/tests/export/export-tf.test.ts @@ -238,6 +238,60 @@ describe("runExportTf — the id map", () => { expect(map["group-type"]).toEqual({ "3_active": { id: 4, label: "g_3_active" } }); }); + // A partial export describes the types it was asked for, but the map it writes is the whole file. + // Rewriting it from one run's entries dropped every id the previous export put there — and a + // `personStatus: "status_unbekannt"` reference cannot fall back to the live name (the key is not + // name-derived), so the next `ct plan` hard-errors on a map the operator never meant to shrink. + it("carries over the types a --only export never looked at", async () => { + const rows = { + mainz: row("campus", "mainz", 4), + status_unbekannt: row("person-status", "status_unbekannt", 9), + }; + await exportInto(rows); + const result = await exportInto(rows, ["campus"]); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map.campus).toEqual({ mainz: { id: 4 } }); + expect(map["person-status"]).toEqual({ status_unbekannt: { id: 9 } }); + expect(map.$meta).toMatchObject({ entries: 2 }); + }); + + // The other half of the same rule: a type the export DID cover is rewritten wholesale, because a + // resource that left ct's state has left ct's ownership and its id must go with it. + it("drops an entry whose resource left state, for a type the export covered", async () => { + await exportInto({ + mainz: row("campus", "mainz", 4), + horgen: row("campus", "horgen", 5), + status_unbekannt: row("person-status", "status_unbekannt", 9), + }); + const result = await exportInto({ mainz: row("campus", "mainz", 4) }, ["campus"]); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map.campus).toEqual({ mainz: { id: 4 } }); + expect(map["person-status"]).toEqual({ status_unbekannt: { id: 9 } }); + }); + + // The end state the map exists to serve: tier-0 has left ct.config.ts AND ct-state..json, so + // a full `ct export tf` (a `make export`, a CI regen, a second env) selects every type, finds + // nothing, and would write `entries: 0` over the only thing still resolving every leftover + // `campus:` / `personStatus:` reference. An export that exported nothing has learned nothing. + it("never empties a populated map, however complete the export", async () => { + await exportInto({ + mainz: row("campus", "mainz", 4), + status_unbekannt: row("person-status", "status_unbekannt", 9), + }); + const result = await exportInto({}); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map.campus).toEqual({ mainz: { id: 4 } }); + expect(map["person-status"]).toEqual({ status_unbekannt: { id: 9 } }); + expect(result.warnings.map((w) => w.code)).toContain("IDS_KEPT"); + }); + + it("still writes an empty map when there was nothing to keep", async () => { + const result = await exportInto({}); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map.$meta).toMatchObject({ entries: 0 }); + expect(result.warnings.map((w) => w.code)).not.toContain("IDS_KEPT"); + }); + it("writes nothing under .ct when --no-ids is passed", async () => { await stateWith({ mainz: row("campus", "mainz", 0) }); const result = await runExportTf({ diff --git a/tests/ids-sync.test.ts b/tests/ids-sync.test.ts index 555401f..2111f18 100644 --- a/tests/ids-sync.test.ts +++ b/tests/ids-sync.test.ts @@ -141,11 +141,34 @@ describe("runIdsSync", () => { expect(result.value.written).toBe(false); expect(result.value.added.map((e) => e.key)).toEqual(["mainz"]); await expect(readFile(join(dir, ".ct", "ids.eqrm.church.tools.json"), "utf8")).rejects.toThrow(/ENOENT/); + // The path it REPORTS is the one it would have written. Built from the raw host, it read + // `.ct/ids.https:/eqrm.church.tools.json` — a filename that exists nowhere, in the one mode + // whose entire output is "here is what I would do". + expect(result.value.path).toBe(join(dir, ".ct", "ids.eqrm.church.tools.json")); }); it("says where to look rather than parsing garbage", async () => { await expect(sync("not json")).rejects.toThrow(/not valid JSON/); }); + + // `loadIdMap` throws on a malformed or foreign-host map, and this is the one command able to + // replace one. Refusing to run until the broken file is deleted by hand makes the repair tool + // need the repair. + it("regenerates over a malformed map instead of refusing to run", async () => { + await writeFile(join(dir, ".ct", "ids.eqrm.church.tools.json"), "not json at all", "utf8"); + const result = await sync(tfstate([CAMPUS_MAINZ])); + expect(result.value.written).toBe(true); + expect(result.value.entries.map((e) => e.key)).toEqual(["mainz"]); + expect(result.warnings.map((w) => w.code)).toContain("IDS_PREVIOUS_UNREADABLE"); + }); + + // Every other path in this operation is anchored to project.cwd; this one read from process.cwd(), + // so an embedded or HTTP caller passing `cwd` read a different file than it asked for. + it("resolves a relative --tofu-state against the project cwd", async () => { + await writeFile(join(dir, "terraform.tfstate"), tfstate([CAMPUS_MAINZ]), "utf8"); + const result = await runIdsSync({ cwd: dir, tofuState: "terraform.tfstate" }); + expect(result.value.entries.map((e) => e.key)).toEqual(["mainz"]); + }); }); describe("runIdsList", () => { diff --git a/tests/tofu-id-map.test.ts b/tests/tofu-id-map.test.ts index c318242..f4119bd 100644 --- a/tests/tofu-id-map.test.ts +++ b/tests/tofu-id-map.test.ts @@ -202,3 +202,55 @@ describe("readTfState", () => { expect(() => readTfState({ resources: {} })).toThrow(/"resources" is not an array/); }); }); + +/** + * Reading a tofu state defensively (#181 review): the two ways `readTfState` could put a + * CONFIDENTLY WRONG row into a committed map, which is worse than putting none there. + */ +describe("readTfState — ids it must refuse to guess", () => { + const managed = (name: string, id: unknown) => ({ + mode: "managed", + type: "churchtools_campus", + name, + instances: [{ attributes: { id } }], + }); + + // `Number(null)`, `Number("")`, `Number([])` and `Number(false)` are all 0, and `Number.isFinite(0)` + // is true — so the finite check alone let a mid-create or tainted resource through as id 0. Zero is + // a REAL ChurchTools id, so the reference would have resolved silently to the wrong live object. + it("does not turn a null or empty id into the real id 0", () => { + const { entries } = readTfState({ + resources: [managed("mainz", null), managed("koeln", ""), managed("bonn", [])], + }); + expect(entries).toEqual([]); + }); + + it("still accepts a genuine 0 and a numeric string", () => { + const { entries } = readTfState({ resources: [managed("mainz", 0), managed("koeln", "7")] }); + expect(entries).toEqual([ + { type: "campus", key: "mainz", id: 0 }, + { type: "campus", key: "koeln", id: 7 }, + ]); + }); + + // A for_each/count block is ONE resource with many instances, and its name is the block's, not any + // resource's key. Taking instances[0] mapped that label to one arbitrary id and reported every real + // key as removed — and being non-empty, it would sail past the empty-map guard in `ct ids sync`. + it("reports a for_each block instead of mapping its label to one arbitrary id", () => { + const { entries, multiInstance } = readTfState({ + resources: [ + { + mode: "managed", + type: "churchtools_campus", + name: "campuses", + instances: [ + { index_key: "mainz", attributes: { id: 4 } }, + { index_key: "koeln", attributes: { id: 5 } }, + ], + }, + ], + }); + expect(entries).toEqual([]); + expect(multiInstance).toEqual(["churchtools_campus.campuses"]); + }); +});