diff --git a/README.md b/README.md index 6ce00bf..19de06e 100644 --- a/README.md +++ b/README.md @@ -237,6 +237,10 @@ ct plan # diff the config against ChurchTools (read-only) ct plan --format markdown # plain-language review report (German by default) ct apply # create + update in dependency order (confirm + backup first) ct refresh --group # make ChurchTools re-evaluate one auto-group now + +ct export tf # render managed state as OpenTofu HCL + import blocks +ct ids sync --tofu-state - # refresh the key->id map ct resolves tofu-owned resources through +ct auth token --env dev # hand a short-lived session to another tool (credential helper) ``` Run without `--token`, `ct auth login` asks how you want to authenticate: **username and @@ -250,12 +254,19 @@ export `CT_HOST` and `CT_LOGINTOKEN` there. `state rm` is the inverse of `adopt`, and only of `adopt`: it removes the entry from the state file, makes no HTTP call, and leaves the resource in place in -ChurchTools, now unmanaged. It refuses a key the config still declares — that +ChurchTools, now unmanaged. It refuses a key the config still **declares** — that would make the next plan propose creating a resource that already exists — so delete the declaration first, or pass `--force` to do both in one change. -"Declares" covers permission declarations too, not only resources: a key named -by a `ct.groupRole` domain or a group scope is just as broken to remove, and the -refusal is what keeps that from surfacing one command later as a plan error. + +A key the config only **references** is a different matter and is removed: a +logical reference asks the host for an id, so it never proposes creating the +resource it points at. `state rm` says how many references stay behind, because +they now resolve against the live instance by name (or through the +[id map](docs/opentofu-migration.md)) rather than from state. The one reference +that still blocks removal is a **group**: groups are managed-only, with no live +catalog to fall back to, so dropping a group a `ct.groupRole` domain or a group +scope names would make the next plan fail to resolve it — the refusal is what +keeps that from surfacing one command later. `apply` reconciles **creates and updates** only, saving state after each action (crash-safe / resumable). It **never deletes**: a resource dropped from the @@ -336,6 +347,7 @@ only; it carries no instance data. - [`docs/configuration.md`](docs/configuration.md) — writing the config: keys, portable references, escape hatches - [`docs/environments.md`](docs/environments.md) · [`docs/ci.md`](docs/ci.md) — multi-instance and automation +- [`docs/opentofu-migration.md`](docs/opentofu-migration.md) — living alongside terraform-provider-churchtools: the committed id map, and `ct auth token` as a credential helper - [`docs/handbuch/`](docs/handbuch/) — the **generic ChurchTools reference** (permissions, dynamic groups, field definitions, blueprints), published into the Handbuch - [`docs/api-coverage.md`](docs/api-coverage.md) · [`docs/runbook-manual-surface.md`](docs/runbook-manual-surface.md) — what the API supports, and what still has to be done by hand - [`docs/README.md`](docs/README.md) — how the docs are organised and how pages stay in sync with the code diff --git a/docs/README.md b/docs/README.md index d6a750c..15cb546 100644 --- a/docs/README.md +++ b/docs/README.md @@ -52,6 +52,7 @@ A page with no code behaviour to track declares `sources: []` plus a | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | [`adoption-contract.md`](adoption-contract.md) | What else comes along when `ct adopt` adopts one resource — the five categories and their defaults | | [`api-coverage.md`](api-coverage.md) | Which ChurchTools endpoints support which CRUD verbs | +| [`opentofu-migration.md`](opentofu-migration.md) | Living alongside terraform-provider-churchtools: the committed id map, and `ct auth token` | | [`group-field-decisions.md`](group-field-decisions.md) | Which group fields are managed vs. left to the CT UI, and why | | [`runbook-manual-surface.md`](runbook-manual-surface.md) | What `ct` cannot automate today — where the write path is missing, and the manual steps around it | | `superpowers/` | Historical implementation plans; kept as a record, never published | diff --git a/docs/handbuch/blueprints.md b/docs/handbuch/blueprints.md index b0aef90..adaaf49 100644 --- a/docs/handbuch/blueprints.md +++ b/docs/handbuch/blueprints.md @@ -4,7 +4,7 @@ sources: - src/config/context.ts - src/engine/graph.ts - src/engine/hierarchy.ts -sources_hash: 6f4be8d3a93113ce +sources_hash: b183fe0075bd7ad3 reviewed: 2026-08-28 --- diff --git a/docs/handbuch/field-definitions.md b/docs/handbuch/field-definitions.md index f82f074..1a1a3b1 100644 --- a/docs/handbuch/field-definitions.md +++ b/docs/handbuch/field-definitions.md @@ -3,7 +3,7 @@ title: Field definitions & security levels sources: - src/commands/get.ts - src/api/ctClient.ts -sources_hash: 659b65504025ebb8 +sources_hash: 5c991e89fd5b6b9b reviewed: 2026-08-13 --- diff --git a/docs/handbuch/group-member-fields.md b/docs/handbuch/group-member-fields.md index 1da01b5..67ff8c2 100644 --- a/docs/handbuch/group-member-fields.md +++ b/docs/handbuch/group-member-fields.md @@ -1,5 +1,5 @@ --- -sources_hash: c18b710bff24503d +sources_hash: 83e743cd6cf4caf3 title: Group member fields sources: - src/engine/member-fields.ts diff --git a/docs/handbuch/permissions.md b/docs/handbuch/permissions.md index 4b2c46a..0a0e151 100644 --- a/docs/handbuch/permissions.md +++ b/docs/handbuch/permissions.md @@ -7,7 +7,7 @@ sources: - src/resolve/resolver.ts - src/resolve/refs.ts - src/config/context.ts -sources_hash: c3a6740c4d4bc134 +sources_hash: 1ccb85af4d54a491 reviewed: 2026-08-28 --- @@ -152,7 +152,10 @@ Each line shows the name, its numeric `authId`, and whether it's `scoped` when grants are resolved to tuples against the catalog, after the authed fetch — with a "did you mean" hint drawn from same-module names. (Config evaluation only checks a grant's _shape_: `module:right` string or -`{ right, scope }`; it does not resolve the name against the catalog.) +`{ right, scope }`; it does not resolve the name against the catalog.) A name the +**active** catalog does not define but ct's bundled catalog does is a different +case — a host difference, not a typo — and is skipped rather than fatal; see +[A right this host does not have](#a-right-this-host-does-not-have-178). ## Catalog lifecycle & staleness (#25) @@ -212,6 +215,50 @@ records the version the catalog was captured from. On every `plan`/`apply`: Both are warnings, not errors: the plan still runs and the exit code stays success. `ct permissions catalog --refresh` is the fix for both. +### A right this host does not have (#178) + +An estate's instances do not all have the same modules installed, and one +declarative config is supposed to serve all of them. So a **declared** right +that the active catalog does not define is not automatically an error: + +- the name is in **ct's bundled catalog** but not in this host's capture → a real + right this instance does not have (a module that is not installed). `ct` prints + a warning naming the right, the declaration and both catalog versions, and + **skips that one grant on this host** — it is never granted and never revoked. + Everything else in the same declaration still plans. +- the name is in **no** catalog `ct` has ever seen → still a hard error with the + "did you mean" hint. That is a typo, or a right ChurchTools deleted, and it is + the case the error was written for. + +The same split applies to a [`preserveUnknown`](#partial-ownership-preserveunknown-opt-in-102) +dimension: one no right on this host scopes by is reported and ignored, one no +catalog knows is still rejected at config-evaluation time. + +This mirrors what `ct` has always done in the other direction — a _live_ grant +whose `authId` the catalog cannot name is reported and left alone. Say what you +cannot manage, manage the rest. + +Note the precondition: the "it exists elsewhere" verdict needs a **per-instance +capture** to be active. Without one, the active catalog _is_ the bundled snapshot, +so "missing here" and "missing everywhere" are the same statement and every +absence stays fatal. This is one more reason to commit +`.ct/permission-catalog..json` per host. + +```bash +ct plan --env dev +# ! group_role "implementierung_churchtools_mitglied": right "jpmFlowManager:view" is absent +# from this host's permission catalog (this host's catalog: ChurchTools 3.137.0-RC13, 221 rights) +# — skipped for this host — never granted, never revoked. ct's bundled catalog +# (ChurchTools 3.134.0) defines it, so this reads as a module this instance does not have. +# Pass --strict-catalog to fail on it instead. + +ct plan --env dev --strict-catalog # every unresolvable declaration is an error again +``` + +`--strict-catalog` is available on `ct plan` and `ct apply`. Exit codes are +unchanged either way: a skip is not a pending change, so `--detailed-exitcode` +still reports 0 for a clean plan. + ## `domainId` semantics The two DSL functions manage two different ChurchTools "domain types," and @@ -489,6 +536,14 @@ therefore a cross-environment misgrant, and because declaring a domain makes `ct` _own_ it, the wrong-scope grant also revokes whatever is really there on the other host. The typed reference makes one config plan clean on both. +Between those two sits one more source, for a repo mid-migration: the committed +**OpenTofu id map** (`.ct/ids..json`, #181). It is consulted after managed +state and before the live catalog, and it exists because a resource that moved to +`terraform-provider-churchtools` leaves ct's state entirely — at which point a +reference like `personStatus: "status_unbekannt"` has nothing left to resolve +against, since ct's keys are not derived from the live names the catalog matches +on. See [Living alongside terraform-provider-churchtools](https://github.com/eqrm/ct-cli/blob/main/docs/opentofu-migration.md). + Resolution mirrors the domain-reference rules: managed state first, the live master-data catalog second, and a target **declared in this same config** resolves to a _pending_ scope re-resolved at apply time. A reference resolved diff --git a/docs/opentofu-migration.md b/docs/opentofu-migration.md new file mode 100644 index 0000000..98974fc --- /dev/null +++ b/docs/opentofu-migration.md @@ -0,0 +1,194 @@ +# Living alongside terraform-provider-churchtools + +The TypeScript DSL is frozen and the tier-0 resources — campuses, group types, +Bereiche, person statuses, comment viewers — are moving to +[`terraform-provider-churchtools`](https://github.com/eqrm/terraform-provider-churchtools). +`ct export tf` produces the HCL and the import blocks for that move. This page +covers the two things needed to live in the in-between state, where OpenTofu owns +tier-0 and `ct` still owns everything above it. + +## The id map: references that outlive their declarations (#181) + +When tier-0 leaves `ct.config.ts`, its **references** stay: several hundred +`campus: "mainz"`, `personStatus: "status_unbekannt"`, `{ groupType: "struktur" }` +on groups, grants and rulesets. Those are logical references — they ask the host +for an id rather than carrying one — so they look like they should survive. + +They do not, on their own. `ct` resolves a logical reference from its own state +first and falls back to matching the key against the live object's **name**. While +the resource was in ct's state the key resolved exactly; with the state entry gone, +only the name fallback is left, and ct's tier-0 keys were never name-derived: + +| key | live name | +| ------------------ | --------------------------- | +| `status_unbekannt` | `Unbekannt` | +| `status_5_core` | `5 - Core` | +| `egc` | `Equippers Germany Central` | + +No slug of those names produces those keys, so removing the state entries makes +`ct plan` fail to resolve them. Pinning numeric ids is not an escape either: one +config serves two hosts, and on the eqrm estate 39 of 43 tier-0 ids differ between +them. + +So `ct` reads a committed **id map** — the exact `key → id` table for this host: + +``` +.ct/ids..json +``` + +```json +{ + "$meta": { + "host": "https://eqrm.church.tools", + "source": "ct export tf", + "generatedAt": "2026-09-21", + "entries": 50 + }, + "campus": { "mainz": { "id": 0 } }, + "person-status": { "status_unbekannt": { "id": 0 }, "status_5_core": { "id": 6 } } +} +``` + +It sits **between** the two existing sources: after ct's own managed state (ct +never stops trusting what it owns) and before the live catalog (an exact table +beats a name guess). A repo without a map behaves exactly as before. + +`ct plan` names the map it loaded in its header, next to the permission catalog: + +``` +permission catalog: .ct/permission-catalog.eqrm.church.tools.json +tofu id map: .ct/ids.eqrm.church.tools.json +``` + +**Commit the map, one per host.** It is host-checked on load — a map whose `$meta.host` +does not match the resolved host is rejected rather than applied, because 39 of 43 +ids differing means a foreign map would resolve every reference to a real, +_wrong_ resource, which nothing downstream could detect. + +### Writing it + +`ct export tf` writes it from the state it is exporting, so the cutover itself +needs no extra step: + +```bash +ct export tf --env prod # tofu/*.tf, tofu/imports.tf, and .ct/ids..json +ct export tf --env prod --no-ids # …without the map, if your repo generates it another way +``` + +The map is written under `.ct/`, not into the tofu output directory: it is `ct`'s +input, not part of the root module tofu reads. + +A partial export (`--only campus`) rewrites only the types it was asked for and +carries the rest of the map over untouched — otherwise a type-at-a-time cutover +would drop the ids of every type it had not reached yet. Within a type the +rewrite is wholesale: a resource that has left `ct`'s state has left `ct`'s +ownership, and its id goes with it. + +An export that maps **nothing** leaves an existing map alone and says so, rather +than emptying it. That is not an edge case but the end state: once tier-0 is gone +from `ct.config.ts` and `ct-state..json`, every `ct export tf` exports zero +resources, and the map it would overwrite is the only thing still resolving the +references that stayed behind. From that point on the map is refreshed from tofu, +with `ct ids sync` — not from `ct`. + +### Keeping it current + +An export can only describe what `ct` still holds. Once tier-0 belongs to tofu, +tofu is the only place new ids appear — a campus created by `tofu apply` exists in +no ct state file. `ct ids sync` reads tofu's own state instead: + +```bash +tofu state pull | ct ids sync --env prod --tofu-state - # any backend, no S3 client in ct +ct ids sync --env prod --tofu-state terraform.tfstate # or a local file +ct ids sync --env prod --tofu-state terraform.tfstate --dry-run +ct ids list --env prod # what ct would resolve through +``` + +`sync` reports every id it adds, changes or drops, and **refuses to replace a +populated map with an empty one** — the likeliest cause of an empty read is the +wrong workspace, and overwriting 50 working ids with nothing would break every +reference at once. Provider resource types `ct` has no mapping for (a +`churchtools_group`, another provider's resources in a shared state) are reported +and skipped. + +One subtlety: `ct export tf` relabels keys that are not valid HCL identifiers +(`3_groupactive` → `g_3_groupactive`), and that mapping is many-to-one, so it +cannot be inverted by rule. The map records the label alongside the key, and +`ct ids sync` uses the existing map to translate a tofu address back to the ct +key. Without a previous map, the label is taken as the key — correct for every +key that needed no relabelling. + +Reading tofu's **remote** state directly was considered and rejected: it would put +an S3 backend, its credentials and an AWS SDK inside a CLI whose every other read +is ChurchTools. Piping `tofu state pull` keeps the backend tofu's problem. + +## `ct auth token`: a credential helper (#179) + +The provider needs credentials, and its `token` attribute wants a ChurchTools +personal login token — which is permanent, cannot be scoped or rotated by +ChurchTools, and is an admin credential on prod. Writing one to a `.env` file for +the provider to read is the thing worth avoiding. + +`ct auth token` hands over the **session** that token buys instead: + +```bash +$ ct auth token --env dev | jq +{ + "operation": "auth", + "action": "token", + "environment": "dev", + "host": "https://eqrm-dev.church.tools", + "cookie": "ChurchTools_eqrm-dev=…", + "csrfToken": "…", + "expiresAt": "2026-09-22T06:12:00.000Z", + "source": "cache" +} + +$ ct auth token --env dev --raw # the bare cookie, for command substitution +``` + +Why the session rather than the token: it expires, `ct auth logout --env dev` +kills it, and a copy that leaks into a `tofu` debug log or a CI artifact is dead +within hours instead of being the permanent admin credential forever. The login +token itself never leaves the Keychain. + +The contract: + +- the credential goes to **stdout and nothing else does** — every message, + warning and Keychain prompt is on stderr, so `$(ct auth token --raw)` is safe; +- a failure writes **nothing** to stdout and exits non-zero, with the remedy + named (`ct auth login --env `); +- printing to a **terminal is refused** unless `--allow-tty` — a credential in + your scrollback defeats the point of it being short-lived; +- `expiresAt` is `ct`'s reuse ceiling (12h), not a promise from ChurchTools: + treat a 401 as "ask again", which is cheap because this command answers from + the Keychain-cached session. + +Calling it on every `tofu` run is the intended usage. Sessions are cached per host +in the Keychain (macOS), so that is normally zero network calls; when a handshake +_is_ needed, a cross-process brake keeps it from becoming a burst: + +- handshakes against one host are spaced at least 3s apart (waited out, not an + error); +- more than 120 in a rolling hour is refused, naming when the window frees up — + that is a runaway loop, and hammering a throttled instance only lengthens the + outage for everyone on it; +- `CT_NO_LOGIN_THROTTLE=1` disables it, for a CI job that knows it runs alone. + +The counter lives in `$XDG_CACHE_HOME/ct-cli/login-throttle..json` and holds +nothing but timestamps. Deleting it, or being unable to write it, simply means no +throttle. It is read-modify-written without a lock, so it bounds a _sequence_ of +invocations rather than a simultaneous burst: two `ct` processes starting at the +same instant are spaced no better than not at all. + +**The brake sits in the login handshake, so it covers every `ct` command**, not +just `ct auth token` — and on Linux and Windows there is no session cache, so +there each invocation is one handshake. That is why the hourly cap is 120 rather +than a number sized for a credential helper alone: a pipeline should never reach +it, while a runaway loop passes it in about six minutes. A CI job that runs more +`ct` invocations than that against one host in an hour should set +`CT_NO_LOGIN_THROTTLE=1`. + +A CI job otherwise needs none of this: it passes the token explicitly from a +GitHub secret, which is already storage-free. This path exists for local +development, where the alternative was a token on disk. diff --git a/src/api/ctClient.ts b/src/api/ctClient.ts index 969355c..d7d0b86 100644 --- a/src/api/ctClient.ts +++ b/src/api/ctClient.ts @@ -78,13 +78,31 @@ const DEFAULT_PAGE_LIMIT = 100; * exactly as it always has. */ export interface SessionCache { - load(host: string, token: string): Promise<{ cookie: string; csrfToken: string } | null>; + /** + * `obtainedAt` (epoch ms) is optional and purely informational: it lets a caller that HANDS THE + * SESSION ON say when it was bought, and therefore how long it may still be reused (#179). + * A cache that does not track it simply omits it. + */ + load( + host: string, + token: string, + ): Promise<{ cookie: string; csrfToken: string; obtainedAt?: number } | null>; save(host: string, token: string, session: { cookie: string; csrfToken: string }): Promise; drop(host: string): Promise; } +/** + * A cross-process brake on the login handshake (#179). Injected for the same reason + * {@link SessionCache} is: a client built in a test must not touch the developer's disk, and a client + * without one behaves exactly as it always has. + */ +export interface LoginThrottleGate { + acquire(host: string): Promise; +} + export interface CtClientOptions { sessionCache?: SessionCache; + loginThrottle?: LoginThrottleGate; } /** Human-readable "wait this long" for a 429, from `Retry-After` when the server sent one. */ @@ -105,6 +123,9 @@ export class CtClient { private ctVersion: string | null = null; /** Kept so an expired session can be re-bought without the caller having to notice (#145). */ private loginToken: string | null = null; + /** When the live session was bought, and whether this process bought it (#179). */ + private sessionObtainedAt: number | null = null; + private sessionSource: "cache" | "handshake" | null = null; /** Re-entrancy guards: no self-heal while a login (or a resume probe) is already in flight. */ private loggingIn = false; private resuming = false; @@ -202,7 +223,7 @@ export class CtClient { if (!cache) { return null; } - let cached: { cookie: string; csrfToken: string } | null = null; + let cached: { cookie: string; csrfToken: string; obtainedAt?: number } | null = null; try { cached = await cache.load(this.config.host, loginToken); } catch { @@ -213,6 +234,8 @@ export class CtClient { } this.cookie = cached.cookie; this.csrfToken = cached.csrfToken; + this.sessionObtainedAt = cached.obtainedAt ?? null; + this.sessionSource = "cache"; this.resuming = true; try { return await this.get("/whoami"); @@ -241,7 +264,36 @@ export class CtClient { } } + /** + * The live session, for a caller whose whole job is to hand it to another tool (`ct auth token`, + * #179). Returns `null` before the handshake has run. + * + * This is the ONE deliberate exit from the rule that the cookie never leaves this object: a + * ChurchTools session expires and can be dropped (`ct auth logout`), while the personal login + * token it was bought with is permanent and, on prod, an admin credential. Handing out the session + * is what lets the OpenTofu provider authenticate without the permanent secret ever leaving the + * Keychain. Nothing here logs or formats it — `sessionCredential` is read only by a caller that has + * already decided where the value is allowed to go. + */ + sessionCredential(): { + cookie: string; + csrfToken: string; + obtainedAt: number; + source: "cache" | "handshake"; + } | null { + if (!this.cookie || this.csrfToken === null) return null; + return { + cookie: this.cookie, + csrfToken: this.csrfToken, + obtainedAt: this.sessionObtainedAt ?? Date.now(), + source: this.sessionSource ?? "handshake", + }; + } + private async performLogin(loginToken: string): Promise { + // Before the request, not after: the point is to not ADD to a burst that is already in progress, + // including one started by a different `ct` process (#179). + await this.options.loginThrottle?.acquire(this.config.host); // The token rides as a URL query param (it lands in the server's access logs). This is // unavoidable for this token class: the handshake above is documented to require the // `login_token` query param — an `Authorization` header yields a null CSRF token and breaks @@ -271,6 +323,8 @@ export class CtClient { if (!this.cookie) { throw new CtApiError("Login succeeded but no session cookie was returned", res.status, null); } + this.sessionObtainedAt = Date.now(); + this.sessionSource = "handshake"; await this.refreshCsrfToken(); // Keep the freshly bought session for the NEXT invocation. Best-effort: a store // that refuses (no Keychain, locked Keychain) must not fail the command. diff --git a/src/api/session.ts b/src/api/session.ts index 92000b0..8825104 100644 --- a/src/api/session.ts +++ b/src/api/session.ts @@ -5,6 +5,7 @@ import { CtClient, type WhoAmI } from "./ctClient.js"; import { readCredentials } from "../auth/tokenStore.js"; import { keychainSessionCache } from "../auth/sessionStore.js"; +import { fileLoginThrottle } from "../auth/loginThrottle.js"; import { normalizeHost, resolveConfig } from "../config.js"; export interface AuthedSession { @@ -48,7 +49,12 @@ export async function authedSession(): Promise { // The session cache is keyed by the SAME host the binding check above just cleared, // so a cached session can only ever be replayed against the instance it came from (#30/#145). - const client = new CtClient(config, { sessionCache: keychainSessionCache() }); + const client = new CtClient(config, { + sessionCache: keychainSessionCache(), + // The session cache removes most handshakes; the throttle bounds the ones it cannot — a cold + // Linux/CI run, `ct auth token` called per `tofu` run, two invocations in parallel (#179). + loginThrottle: fileLoginThrottle(), + }); const me = await client.authenticate(token); // Hard-fail below the minimum CT version before any command reads or writes — // a stale instance half-applies (tier-0 writes succeed, hierarchy endpoints 404). diff --git a/src/application/contracts.ts b/src/application/contracts.ts index 508c5a0..1680402 100644 --- a/src/application/contracts.ts +++ b/src/application/contracts.ts @@ -3,7 +3,7 @@ export type JsonPrimitive = string | number | boolean | null; export type JsonValue = JsonPrimitive | JsonValue[] | { [key: string]: JsonValue }; export type OperationName = - "plan" | "apply" | "coverage" | "adopt" | "state" | "refresh" | "destroy" | "auth" | "export-tf"; + "plan" | "apply" | "coverage" | "adopt" | "state" | "refresh" | "destroy" | "auth" | "export-tf" | "ids"; /** Common project selection accepted by CLI and, later, HTTP adapters. */ export interface ProjectRequest { diff --git a/src/application/operations/auth.ts b/src/application/operations/auth.ts index 9589318..a3c8342 100644 --- a/src/application/operations/auth.ts +++ b/src/application/operations/auth.ts @@ -4,7 +4,7 @@ import { CtClient, type WhoAmI } from "../../api/ctClient.js"; import { formatError } from "../../api/format.js"; import { meetsMinVersion, MIN_CT_VERSION, type CtInfo } from "../../api/version.js"; import { checkAllEnvAuth, type EnvAuthStatus } from "../../auth/status.js"; -import { keychainSessionCache } from "../../auth/sessionStore.js"; +import { keychainSessionCache, SESSION_MAX_AGE_MS } from "../../auth/sessionStore.js"; import { clearCredentials, readToken, @@ -151,6 +151,108 @@ export async function runAuthLogout( }; } +export interface AuthTokenRequest { + cwd?: string; + environment?: string; +} + +/** + * A ChurchTools SESSION, for handing to another tool (#179). + * + * Deliberately not the login token. The OpenTofu provider declares `token` as required, so the + * tier-0 cutover meant writing a personal ChurchTools login token to local disk — a permanent, + * full-privilege credential (an admin one on prod) that ChurchTools offers no way to scope or + * rotate. The session bought with it is the one short-lived credential in the system: it expires, + * `ct auth logout` drops it, and a copy that leaks into a `tofu` debug log or a CI artifact is dead + * within hours rather than forever. So that is what `ct auth token` emits, and the token itself never + * leaves the Keychain. + */ +export interface AuthTokenResult { + operation: "auth"; + action: "token"; + environment: string | null; + host: string; + cookie: string; + csrfToken: string; + /** + * When `ct` will stop reusing this session (its `obtainedAt` + the 12h reuse ceiling in + * sessionStore). A CEILING, not a promise: ChurchTools does not advertise its session lifetime, so + * the server may end the session sooner. A consumer should treat a 401 as "ask again", not as an + * error — which is also why this command is cheap to call on every run. + */ + expiresAt: string; + /** `cache` — reused an existing session; `handshake` — one login handshake was spent to buy it. */ + source: "cache" | "handshake"; +} + +export interface AuthTokenDependencies { + project?: ProjectResolutionDependencies; + resolveProject?: typeof resolveProject; + readToken?: typeof readToken; + authedSession?: () => Promise; + env?: NodeJS.ProcessEnv; + cwd?: () => string; +} + +/** + * Resolve host + session for one environment, buying a session only if there is no reusable one. + * + * Returns the credential; it never prints, logs or formats it — every caller decides where the value + * is allowed to go (the CLI adapter puts it on stdout alone, and refuses a terminal). + */ +export async function runAuthToken( + request: AuthTokenRequest = {}, + dependencies: AuthTokenDependencies = {}, +): Promise { + const env = dependencies.env ?? process.env; + const cwd = resolve(dependencies.cwd?.() ?? process.cwd(), request.cwd ?? "."); + let project; + try { + project = await (dependencies.resolveProject ?? resolveProject)( + { cwd, environment: request.environment }, + { ...dependencies.project, env }, + ); + } catch (cause) { + if (!(cause instanceof MissingHostError)) throw cause; + throw new CtApplicationError( + "AUTH_REQUIRED", + "No host resolved. Pass --env , or run `ct auth login --host --token `.", + { cause }, + ); + } + // Checked before the network: "no credential for this host" is the case a credential helper hits + // most often, and it must fail with the remedy rather than with a login error. + if (!(await (dependencies.readToken ?? readToken)(project.host))) { + throw new CtApplicationError( + "AUTH_REQUIRED", + request.environment + ? `No token for ${project.host}. Run \`ct auth login --env ${request.environment}\`.` + : `No token for ${project.host}. Run \`ct auth login --host ${project.host} --token \`.`, + { details: { host: project.host } }, + ); + } + const { client } = await (dependencies.authedSession ?? authedSession)(); + const session = client.sessionCredential(); + if (!session) { + throw new CtApplicationError( + "AUTH_REQUIRED", + `Authenticated against ${project.host} but no session cookie was captured, so there is nothing ` + + `to hand over. Re-run \`ct auth login\` for this host.`, + { details: { host: project.host } }, + ); + } + return { + operation: "auth", + action: "token", + environment: project.environment, + host: project.host, + cookie: session.cookie, + csrfToken: session.csrfToken, + expiresAt: new Date(session.obtainedAt + SESSION_MAX_AGE_MS).toISOString(), + source: session.source, + }; +} + /** Return authentication identity and source metadata without ever returning a token. */ export async function runAuthStatus( request: AuthStatusRequest = {}, diff --git a/src/application/operations/export-tf.ts b/src/application/operations/export-tf.ts index 58fdfd5..98aa157 100644 --- a/src/application/operations/export-tf.ts +++ b/src/application/operations/export-tf.ts @@ -5,6 +5,8 @@ import { assertLabelsUnique, hclLabel, hclType, renderResource } from "../../exp import { renderImports, type ImportTarget } from "../../export/imports.js"; import { EXPORTABLE_TYPES, fileForType, OWNED_FILES } from "../../export/layout.js"; import { renderVersions } from "../../export/provider.js"; +import { CATALOG_DIR } from "../../permissions/catalog-store.js"; +import { loadIdMap, writeIdMap, type IdMapEntry } from "../../resolve/idMap.js"; import { loadState } from "../../state/state.js"; import type { CtWarning, OperationResult, ProjectRequest } from "../contracts.js"; import { resolveProject } from "../project.js"; @@ -26,6 +28,15 @@ export interface ExportTfRequest extends ProjectRequest { * file outright — worse than overwriting it, because nothing hints at why. */ writeVersions?: boolean; + /** + * Also write `.ct/ids..json`, the committed key→id map ct resolves leftover references + * through once these resources are no longer in its state (#181). Default true: every export is a + * step towards ct not declaring these any more, and that is exactly when the map becomes load-bearing. + * + * Written OUTSIDE `outDir`, beside the per-instance permission catalog, because it is ct's input, + * not part of the tofu root module. + */ + writeIds?: boolean; } export interface ExportTfValue { @@ -35,6 +46,8 @@ export interface ExportTfValue { relabelled: { key: string; label: string }[]; /** Managed types the provider has no mapping for yet, with how many were skipped. */ skipped: { type: string; count: number }[]; + /** Path of the id map written beside the export, or null when `--no-ids` was passed. */ + idMapPath: string | null; } export type ExportTfResult = OperationResult; @@ -53,6 +66,33 @@ function address(type: string, key: string): string { return `${hclType(type)}.${hclLabel(key)}`; } +/** + * Keep the id-map entries a PARTIAL export never looked at (#181). + * + * `--only campus` describes campuses and nothing else, but the map it writes is the whole file — so + * rewriting it from this run's entries alone drops every `person-status`, `group-type` and + * `department` id the previous export put there. Those are exactly the entries that keep a leftover + * `personStatus: "status_unbekannt"` resolving once tier-0 leaves ct's state, and their keys are not + * name-derived, so the live-name fallback cannot cover for them: the next `ct plan` hard-errors. + * + * So a type this run COVERED is rewritten wholesale (a resource that left state has left ct's + * ownership, and its id must go with it), while a type it never selected is carried over untouched. + * A full export covers every exportable type, so it carries nothing and never reads the old map — + * which is also why a corrupt or foreign map can only fail the partial case, the one that cannot do + * its job without reading it. + */ +async function carryOverUncoveredIds( + host: string, + dir: string, + covered: readonly string[], + fresh: readonly IdMapEntry[], +): Promise { + if (EXPORTABLE_TYPES.every((type) => covered.includes(type))) return [...fresh]; + const previous = await loadIdMap(host, dir); + if (!previous) return [...fresh]; + return [...previous.entries.filter((entry) => !covered.includes(entry.type)), ...fresh]; +} + export async function runExportTf(request: ExportTfRequest): Promise { // resolveProject gives the env profile's state path AND the host the state // must belong to; loadState asserts that pairing, so an export can never @@ -90,6 +130,7 @@ export async function runExportTf(request: ExportTfRequest): Promise(); const imports: ImportTarget[] = []; const relabelled: { key: string; label: string }[] = []; + const idEntries: IdMapEntry[] = []; for (const resource of [...selected].sort((a, b) => { const left = address(a.type, a.key); @@ -107,6 +148,15 @@ export async function runExportTf(request: ExportTfRequest): Promise 0) { + idMapPath = existing.path; + idMapKept = true; + } else { + idMapPath = await writeIdMap(project.host, entries, catalogDir); + } + } + const warnings: CtWarning[] = skipped.map(({ type, count }) => ({ code: "EXPORT_TYPE_UNSUPPORTED", message: @@ -142,10 +213,21 @@ export async function runExportTf(request: ExportTfRequest): Promise; + +export interface IdsListValue { + path: string | null; + entries: IdMapEntry[]; +} + +export type IdsListResult = OperationResult; + +export interface IdsOperationDependencies { + project?: ProjectResolutionDependencies; + resolveProject?: typeof resolveProject; + loadIdMap?: typeof loadIdMap; + writeIdMap?: typeof writeIdMap; + readStdin?: () => Promise; +} + +async function readAllStdin(): Promise { + const chunks: Buffer[] = []; + for await (const chunk of process.stdin) chunks.push(Buffer.from(chunk)); + return Buffer.concat(chunks).toString("utf8"); +} + +function entryId(entry: { type: string; key: string }): string { + return `${entry.type}\u0000${entry.key}`; +} + +export async function runIdsSync( + request: IdsSyncRequest, + dependencies: IdsOperationDependencies = {}, +): Promise { + const project = await (dependencies.resolveProject ?? resolveProject)(request, dependencies.project); + const dir = join(project.cwd, CATALOG_DIR); + const raw = + request.tofuState === "-" + ? await (dependencies.readStdin ?? readAllStdin)() + : await readFile(resolve(project.cwd, request.tofuState), "utf8"); + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch (err) { + throw new Error( + `Could not read the tofu state from ${request.tofuState === "-" ? "stdin" : request.tofuState}: ` + + `not valid JSON (${(err as Error).message}).`, + ); + } + // The EXISTING map is the source of the label→key mapping: the exporter relabels keys that are not + // valid HCL identifiers, many-to-one, so a tofu address can only be mapped back to its ct key with + // the map that recorded the relabelling. Without one, a label is taken as the key — correct for + // every key that needed no relabelling, which is all of them on a normal instance. + // + // Read SOFTLY: `loadIdMap` throws on a malformed or foreign-host map, and this is the one command + // able to replace one. Refusing to run until the broken file is deleted by hand makes the repair + // tool need the repair. The cost of continuing is only that relabelled keys fall back to their + // labels, so the failure is named rather than swallowed. + let previous: Awaited> = null; + let previousUnreadable: string | null = null; + try { + previous = await (dependencies.loadIdMap ?? loadIdMap)(project.host, dir); + } catch (err) { + previousUnreadable = (err as Error).message; + } + const { entries, unmapped, multiInstance, serial } = readTfState( + parsed, + previous?.keysByLabel ?? new Map(), + ); + + const previousById = new Map((previous?.entries ?? []).map((e) => [entryId(e), e])); + const nextById = new Map(entries.map((e) => [entryId(e), e])); + const added = entries.filter((e) => !previousById.has(entryId(e))); + const changed = entries + .map((entry) => ({ entry, previousId: previousById.get(entryId(entry))?.id })) + .filter((row): row is { entry: IdMapEntry; previousId: number } => { + return row.previousId !== undefined && row.previousId !== row.entry.id; + }); + // Dropped from tofu's state, so ct must stop claiming to resolve them — keeping a stale id would + // silently point every reference at whatever now holds that number. + const removed = (previous?.entries ?? []).filter((e) => !nextById.has(entryId(e))); + + const warnings: CtWarning[] = []; + if (previousUnreadable) { + warnings.push({ + code: "IDS_PREVIOUS_UNREADABLE", + message: + `The existing id map could not be read, so this sync rewrites it from scratch: ` + + `${previousUnreadable} Keys the exporter relabelled fall back to their HCL label.`, + details: { reason: previousUnreadable }, + }); + } + if (multiInstance.length > 0) { + warnings.push({ + code: "IDS_MULTI_INSTANCE", + message: + `Ignored ${multiInstance.length} resource block(s) with more than one instance ` + + `(${multiInstance.join(", ")}): a for_each/count block's label is not a resource key, so ct ` + + `cannot tell which id belongs to which key. Declare those resources individually, or pin the ` + + `references to numeric ids.`, + details: { blocks: multiInstance }, + }); + } + if (unmapped.length > 0) { + warnings.push({ + code: "IDS_TYPE_UNSUPPORTED", + message: + `Ignored ${unmapped.length} provider resource type(s) ct has no mapping for: ${unmapped.join(", ")}. ` + + `References to those cannot resolve through the id map — keep them managed by ct, or use a numeric id.`, + details: { types: unmapped }, + }); + } + if (entries.length === 0) { + warnings.push({ + code: "IDS_EMPTY", + message: + `The tofu state holds no churchtools resources ct can map, so the id map would be empty. ` + + `Check you pulled the state for ${project.host} (\`tofu state pull\` in the right workspace).`, + }); + } + + const written = !request.dryRun && entries.length > 0; + const path = written + ? await (dependencies.writeIdMap ?? writeIdMap)(project.host, entries, dir, "ct ids sync") + : (previous?.path ?? idMapPath(project.host, dir)); + + return { + operation: "ids", + project, + warnings, + value: { path, entries, added, changed, removed, serial, written }, + }; +} + +export async function runIdsList( + request: ProjectRequest = {}, + dependencies: IdsOperationDependencies = {}, +): Promise { + const project = await (dependencies.resolveProject ?? resolveProject)(request, dependencies.project); + const map = await (dependencies.loadIdMap ?? loadIdMap)(project.host, join(project.cwd, CATALOG_DIR)); + return { + operation: "ids", + project, + warnings: [], + value: { path: map?.path ?? null, entries: map?.entries ?? [] }, + }; +} diff --git a/src/application/operations/plan.ts b/src/application/operations/plan.ts index 86b6fb7..b0b78ae 100644 --- a/src/application/operations/plan.ts +++ b/src/application/operations/plan.ts @@ -3,8 +3,10 @@ import { authedSession, type AuthedSession } from "../../api/session.js"; import { loadConfig } from "../../config/load.js"; import { buildPlan } from "../../engine/build.js"; import { summarize, type Plan, type PlanAction } from "../../engine/types.js"; +import { STRICT_CATALOG, setStrictCatalog } from "../../permissions/catalog.js"; import { CATALOG_DIR, loadHostCatalog } from "../../permissions/catalog-store.js"; import { buildPermissionPlan, type PermissionPlanItem } from "../../permissions/plan.js"; +import { loadIdMap } from "../../resolve/idMap.js"; import { Resolver } from "../../resolve/resolver.js"; import { loadState, type State } from "../../state/state.js"; import type { CtClient } from "../../api/ctClient.js"; @@ -12,7 +14,17 @@ import type { CtWarning, OperationResult, ProjectRequest } from "../contracts.js import { noopObserver, type OperationObserver } from "../ports.js"; import { resolveProject, type ProjectResolutionDependencies } from "../project.js"; -export type PlanRequest = ProjectRequest; +export interface PlanRequest extends ProjectRequest { + /** + * `--strict-catalog` (#178): treat a declared right (or `preserveUnknown` dimension) the active + * permission catalog does not define as a hard error, even when ct's bundled catalog defines it. + * + * The default is the skip-with-warning posture, which is what lets ONE config serve two instances + * with different modules installed. This flag is for a repo that would rather planning failed than + * have any declaration silently not apply. + */ + strictCatalog?: boolean; +} export interface PlanSummary { resources: Record; @@ -41,6 +53,8 @@ export interface PlanValue { */ buildWarnings: string[]; permissionCatalogPath: string | null; + /** The committed OpenTofu id map this plan resolved through (#181), when the repo has one. */ + tofuIdMapPath: string | null; } export type PlanResult = OperationResult; @@ -51,6 +65,7 @@ export interface PlanOperationDependencies { project?: ProjectResolutionDependencies; resolveProject?: typeof resolveProject; loadHostCatalog?: typeof loadHostCatalog; + loadIdMap?: typeof loadIdMap; loadConfig?: typeof loadConfig; loadState?: typeof loadState; authedSession?: () => Promise; @@ -104,61 +119,80 @@ export async function buildPlanContext( const project = await (dependencies.resolveProject ?? resolveProject)(request, dependencies.project); observer.emit({ type: "phase-started", phase: "load-project" }); - const catalogPath = await (dependencies.loadHostCatalog ?? loadHostCatalog)( - project.host, - join(project.cwd, CATALOG_DIR), - ); - const { - resources: desired, - permissions, - configDir, - } = await (dependencies.loadConfig ?? loadConfig)(project.configPath); - const state = await (dependencies.loadState ?? loadState)(project.statePath, project.host); - const { client } = await (dependencies.authedSession ?? authedSession)(); - const resolver = (dependencies.createResolver ?? ((options) => new Resolver(options)))({ - client, - state, - desired, - host: project.host, - }); - - observer.emit({ type: "phase-started", phase: "build-plan" }); - const [resourceResult, permissionResult] = await Promise.all([ - (dependencies.buildPlan ?? buildPlan)(client, state, desired, { configDir, resolver }), - (dependencies.buildPermissionPlan ?? buildPermissionPlan)( + // Set BEFORE the catalog and the config load: the config's own `preserveUnknown` validation reads + // it at eval time (config/context.ts), and it must describe the catalog that is about to be loaded. + // + // Restored in the `finally` below, so the flag lives no longer than the build that asked for it. + // For a one-shot CLI that is merely tidy, but `contracts.ts` anticipates an HTTP adapter, and in a + // long-lived process one `--strict-catalog` plan would otherwise leave EVERY later plan strict — + // a setting silently outliving its request, on the exact flag whose whole job is to decide whether + // a plan fails or warns. + const previousStrict = STRICT_CATALOG; + setStrictCatalog(request.strictCatalog ?? false); + try { + const catalogPath = await (dependencies.loadHostCatalog ?? loadHostCatalog)( + project.host, + join(project.cwd, CATALOG_DIR), + ); + // Loaded beside the permission catalog and from the same directory: both are committed, per-host + // artefacts a consumer repo keeps under `.ct/`. + const idMap = await (dependencies.loadIdMap ?? loadIdMap)(project.host, join(project.cwd, CATALOG_DIR)); + const { + resources: desired, + permissions, + configDir, + } = await (dependencies.loadConfig ?? loadConfig)(project.configPath); + const state = await (dependencies.loadState ?? loadState)(project.statePath, project.host); + const { client } = await (dependencies.authedSession ?? authedSession)(); + const resolver = (dependencies.createResolver ?? ((options) => new Resolver(options)))({ client, state, - permissions, desired, - resolver, - client.version ?? undefined, - ), - ]); - const fetchErrors = [...resourceResult.fetchErrors, ...permissionResult.fetchErrors]; - const warnings: CtWarning[] = permissionResult.warnings.map((message) => ({ - code: "PERMISSION_CATALOG", - message, - })); + host: project.host, + idMap, + }); - return { - client, - state, - actual: resourceResult.actual, - result: { - operation: "plan", - project, - warnings, - value: { - plan: resourceResult.plan, - permissions: permissionResult.items, - summary: summarizePlan(resourceResult.plan, permissionResult.items), - complete: fetchErrors.length === 0, - fetchErrors, - churchToolsVersion: client.version, - stateHost: state.host, - buildWarnings: resourceResult.warnings ?? [], - permissionCatalogPath: catalogPath, + observer.emit({ type: "phase-started", phase: "build-plan" }); + const [resourceResult, permissionResult] = await Promise.all([ + (dependencies.buildPlan ?? buildPlan)(client, state, desired, { configDir, resolver }), + (dependencies.buildPermissionPlan ?? buildPermissionPlan)( + client, + state, + permissions, + desired, + resolver, + client.version ?? undefined, + ), + ]); + const fetchErrors = [...resourceResult.fetchErrors, ...permissionResult.fetchErrors]; + const warnings: CtWarning[] = permissionResult.warnings.map((message) => ({ + code: "PERMISSION_CATALOG", + message, + })); + + return { + client, + state, + actual: resourceResult.actual, + result: { + operation: "plan", + project, + warnings, + value: { + plan: resourceResult.plan, + permissions: permissionResult.items, + summary: summarizePlan(resourceResult.plan, permissionResult.items), + complete: fetchErrors.length === 0, + fetchErrors, + churchToolsVersion: client.version, + stateHost: state.host, + buildWarnings: resourceResult.warnings ?? [], + permissionCatalogPath: catalogPath, + tofuIdMapPath: idMap?.path ?? null, + }, }, - }, - }; + }; + } finally { + setStrictCatalog(previousStrict); + } } diff --git a/src/application/operations/state.ts b/src/application/operations/state.ts index 4ec60a5..ac73431 100644 --- a/src/application/operations/state.ts +++ b/src/application/operations/state.ts @@ -1,6 +1,7 @@ import { loadConfig } from "../../config/load.js"; import { resourceType } from "../../resources/registry.js"; import { collectRefs, isRef, type Ref } from "../../resolve/refs.js"; +import { refKindResolvesLive } from "../../resolve/resolver.js"; import { loadState, saveState, type ManagedResource } from "../../state/state.js"; import type { CtWarning, OperationResult, ProjectRequest } from "../contracts.js"; import { InMemoryMutationLock } from "../prepared-operation-store.js"; @@ -47,34 +48,82 @@ export async function listState( }; } -async function declaredKeys( +/** + * What the config DECLARES, and what it merely REFERENCES (#180). + * + * These were one set, and that made `ct state rm` refuse a key the config only points at. Same state + * file, same config: `campus/horgen` (0 occurrences) was accepted, while `campus/mainz` — declared + * nowhere, but named by `campus: "mainz"` on 60 other resources — was refused as "still declared". + * + * A reference is not a declaration. A logical ref asks the HOST for the id (`verify` labels exactly + * these "logical ref, resolved live — not compared"); it never asks `ct` to create the referenced + * object, so removing the state entry cannot make the next plan propose creating it. Only a real + * declaration can do that, which is what the guard was written to catch. + * + * Declarations are keyed by `type\0key`, not by key alone: the guard exists to predict what the next + * plan would do to a specific resource, and a `campus` declaration says nothing about what removing a + * `group-type` of the same name would cause. + * + * One class of reference DOES still block removal, for the guard's original reason rather than a + * spelling one: a ref whose kind has no live master-data catalog (a `group` — groups are managed-only) + * can be resolved from nothing but ct's state, so dropping the entry it names makes the very next + * plan a hard error instead of a create proposal. Those are refused, and said so in their own words. + */ +interface ConfigUsage { + declared: Set; + /** key → how many times the config references it. Reported, never a refusal. */ + referenceCounts: Map; + /** Keys referenced by at least one ref kind that can only resolve from managed state. */ + stateOnlyRefs: Set; +} + +function declarationId(type: string, key: string): string { + return `${type}\u0000${key}`; +} + +async function configUsage( configPath: string, dependencies: StateOperationDependencies, -): Promise> { +): Promise { const { resources, permissions } = await (dependencies.loadConfig ?? loadConfig)(configPath); - const keys = new Set(resources.map((resource) => resource.key)); + const declared = new Set(resources.map((resource) => declarationId(resource.type, resource.key))); + const referenceCounts = new Map(); + const stateOnlyRefs = new Set(); + const countRef = (key: string, resolvesLive: boolean): void => { + referenceCounts.set(key, (referenceCounts.get(key) ?? 0) + 1); + if (!resolvesLive) stateOnlyRefs.add(key); + }; const addRef = (ref: Ref): void => { - if (ref.kind === "group-role") keys.add(ref.group); - else if (ref.kind === "group-type-role") keys.add(ref.groupType); - else if (ref.kind === "group-member-field") keys.add(ref.group); - else keys.add(ref.key); + // The compound kinds address a permission DOMAIN, and the key they carry here is the GROUP or + // GROUP TYPE the domain hangs off — so each is classified by the kind of that target, not by the + // compound kind itself. A group_role domain needs its group in managed state; a group_type_role + // domain resolves its type from `/group/grouptypes` like any other group-type ref. + if (ref.kind === "group-role") countRef(ref.group, refKindResolvesLive("group")); + else if (ref.kind === "group-member-field") countRef(ref.group, refKindResolvesLive("group")); + else if (ref.kind === "group-type-role") countRef(ref.groupType, refKindResolvesLive("group-type")); + else countRef(ref.key, refKindResolvesLive(ref.kind)); }; + // Refs reachable from the resources themselves (`campus: "mainz"` on a group) as well as from the + // permission set — the old check only walked the latter, so the count it reported was partial too. + for (const ref of collectRefs(resources)) addRef(ref); for (const ref of collectRefs(permissions)) addRef(ref); for (const permission of permissions) { for (const grant of permission.grants) { if (typeof grant === "string" || !Array.isArray(grant.scope)) continue; for (const entry of grant.scope) { - if (typeof entry === "string" && entry.length > 0) keys.add(entry); + // A BARE STRING in a scope list is the historical group dimension (`resolveScope` looks it + // up in managed state and nowhere else), so it is state-only by construction. + if (typeof entry === "string" && entry.length > 0) countRef(entry, false); else if (entry !== null && typeof entry === "object" && !isRef(entry)) { const values = Object.values(entry as Record); if (values.length === 1 && typeof values[0] === "string" && values[0].length > 0) { - keys.add(values[0]); + countRef(values[0], false); } } } } } - return keys; + return { declared, referenceCounts, stateOnlyRefs }; } export async function removeStateEntry( @@ -102,16 +151,52 @@ export async function removeStateEntry( const warnings: CtWarning[] = []; if (!request.force) { try { - const declared = await declaredKeys(project.configPath, dependencies); - if (declared.has(request.key)) { + const { declared, referenceCounts, stateOnlyRefs } = await configUsage( + project.configPath, + dependencies, + ); + // Declaredness is checked FIRST because it is the more specific verdict: it is matched on + // type AND key, while the state-only reference check below can only match on key. A key that + // is both declared and referenced would otherwise be refused with the vaguer of the two + // messages, naming a consequence that is not the main one. + if (declared.has(declarationId(request.type, request.key))) { throw new Error( `"${request.key}" is still declared in the config, so removing it from state would make the next ` + `plan propose CREATING a resource that already exists on this host. Remove the ` + `declaration first, or pass --force if you are deleting both in the same change.`, ); } + if (stateOnlyRefs.has(request.key)) { + throw new Error( + `"${request.key}" is still referenced by the config as a resource that only ct's state can ` + + `resolve (a group has no live master-data catalog to fall back to), so removing it would make ` + + `the next plan fail to resolve those references. Remove the references first, or pass --force ` + + `if you are deleting both in the same change.`, + ); + } + // Referenced but not declared — allowed, and worth saying out loud: every one of those refs + // now resolves against the LIVE host by name, which is a different resolution path than the + // state lookup it had a moment ago (see #181). A key whose live name does not slug back to it + // will fail the next plan, and this is the last point at which that is cheap to notice. + const references = referenceCounts.get(request.key) ?? 0; + if (references > 0) { + warnings.push({ + code: "STILL_REFERENCED", + message: + `The config references "${request.key}" ${references} time(s) but does not declare it, so it is ` + + `removed from state. Those references now resolve against ${project.host} by name — check the ` + + `next \`ct plan\` resolves them, or pin them to a numeric id.`, + details: { key: request.key, type: request.type, references }, + }); + } } catch (caught) { - if (caught instanceof Error && caught.message.includes("is still declared in the config")) + // Both refusals above are verdicts, not read failures — re-throw them rather than degrade + // into "could not read the config, removing anyway". + if ( + caught instanceof Error && + (caught.message.includes("is still declared in the config") || + caught.message.includes("is still referenced by the config")) + ) throw caught; warnings.push({ code: "CONFIG_UNREADABLE", diff --git a/src/auth/loginThrottle.ts b/src/auth/loginThrottle.ts new file mode 100644 index 0000000..645bf10 --- /dev/null +++ b/src/auth/loginThrottle.ts @@ -0,0 +1,139 @@ +/** + * A cross-process brake on the ChurchTools LOGIN handshake (#179). + * + * ChurchTools rate-limits logins per instance, and `ct` is one-shot: nothing inside a single + * invocation can see how many handshakes the last minute already spent. The session cache (#145) + * removes most of them, but it cannot remove the ones that matter here — `ct auth token` is designed + * to be called by another tool on every run, and on Linux/CI there is no session cache at all, so + * every invocation starts cold. Two parallel `tofu` runs, or a loop in a script, and the instance + * answers 429 for everybody, including the humans. + * + * So the count lives in a file: one per host, holding nothing but timestamps. + * + * - A handshake closer than {@link MIN_INTERVAL_MS} to the previous one WAITS for the remainder. + * A short sleep is strictly better than a 429 — the caller gets a session either way. + * - More than {@link MAX_PER_HOUR} in a rolling hour THROWS, naming when the window frees up. That + * is a runaway loop, not a burst, and continuing to hammer a throttled instance only lengthens + * the outage for everyone on it. + * + * ## What it does not do + * + * The counter is read, modified and written without a lock, so it bounds a SEQUENCE of invocations, + * not a simultaneous burst: two processes that read the same file at the same moment compute the + * same wait, sleep it together, and the second write drops the first one's timestamp. Truly parallel + * handshakes are therefore spaced no better than not at all, and the hourly count under-reports them. + * Holding a lock across a network handshake is its own hazard (a crashed `ct` would wedge every later + * one), and the 429 this exists to avoid is driven by sustained rate, not by two coincident logins — + * so the honest bound is the useful one, and it is stated here rather than implied away. + * + * Deliberately NOT in the Keychain: there is no secret here (see the blob shape below), and a + * Keychain read is an ACL prompt away from being the very thing that makes the CLI unusable in a + * script. A cache file that a user deletes, or that never appears at all, simply means no throttle — + * every read and write is best-effort, and only the rolling cap ever fails a command. + */ +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import { hostSlug } from "../permissions/catalog-store.js"; + +/** Minimum spacing between two handshakes against one host. Waited out, never an error. */ +export const MIN_INTERVAL_MS = 3_000; +/** + * Handshakes per rolling hour per host before `ct` refuses to add to the pile. + * + * Sized for a CI pipeline, not for `ct auth token` alone, because the gate sits in + * `CtClient.performLogin` and therefore covers EVERY ct command. On Linux and Windows there is no + * session cache at all (`sessionStore` is Keychain-only by design), so on CI each invocation costs + * one handshake: at 20/hour a pipeline whose 21st `ct plan`/`ct get`/`ct apply` ran inside the hour + * would start failing on ct's own error, somewhere it had always worked. + * + * The number that matters for the instance is the SPACING above — it already caps a runaway loop at + * 20 handshakes a minute — so this is the backstop for a loop that keeps going, not the primary + * brake. A loop reaches 120 in about six minutes; a real pipeline does not reach it at all. + */ +export const MAX_PER_HOUR = 120; +const HOUR_MS = 60 * 60 * 1000; + +/** What a throttle needs from its caller. Injected so a client in a test never touches the disk. */ +export interface LoginThrottle { + acquire(host: string): Promise; +} + +/** Nothing here is a credential: a host label and the times ct last logged in to it. */ +interface ThrottleRecord { + host: string; + recent: number[]; +} + +export function throttleDir(env: NodeJS.ProcessEnv = process.env): string { + const base = env.XDG_CACHE_HOME?.trim() || join(homedir(), ".cache"); + return join(base, "ct-cli"); +} + +export function throttlePath(host: string, env: NodeJS.ProcessEnv = process.env): string { + return join(throttleDir(env), `login-throttle.${hostSlug(host)}.json`); +} + +function parseRecord(raw: string, host: string, now: number): ThrottleRecord { + try { + const parsed = JSON.parse(raw) as Partial; + if (parsed.host !== host || !Array.isArray(parsed.recent)) return { host, recent: [] }; + // A timestamp in the FUTURE (a clock change, a copied file) is dropped rather than trusted: + // keeping it would hold the brake on for as long as the skew lasts. + return { host, recent: parsed.recent.filter((t) => typeof t === "number" && t <= now) }; + } catch { + return { host, recent: [] }; + } +} + +const defaultSleep = (ms: number): Promise => new Promise((resolve) => setTimeout(resolve, ms)); + +export interface FileThrottleOptions { + env?: NodeJS.ProcessEnv; + sleep?: (ms: number) => Promise; + now?: () => number; +} + +/** + * The real, file-backed throttle. `CT_NO_LOGIN_THROTTLE=1` disables it — for a CI job that knows it + * runs alone and would rather fail fast than sleep. + */ +export function fileLoginThrottle(options: FileThrottleOptions = {}): LoginThrottle { + const env = options.env ?? process.env; + const sleep = options.sleep ?? defaultSleep; + const now = options.now ?? (() => Date.now()); + return { + async acquire(host: string): Promise { + if (env.CT_NO_LOGIN_THROTTLE === "1") return; + const path = throttlePath(host, env); + let record: ThrottleRecord = { host, recent: [] }; + try { + record = parseRecord(await readFile(path, "utf8"), host, now()); + } catch { + // No file, an unreadable one, a read-only home — none of those is a reason to refuse a login. + } + const recent = record.recent.filter((t) => now() - t < HOUR_MS).sort((a, b) => a - b); + if (recent.length >= MAX_PER_HOUR) { + const freesAt = new Date(recent[0]! + HOUR_MS); + throw new Error( + `Refusing to log in to ${host} again: ${recent.length} login handshakes in the last hour ` + + `(ct's own limit, to stay under ChurchTools' rate limit). The oldest ages out at ` + + `${freesAt.toISOString()}. If a script is looping, fix that first; set ` + + `CT_NO_LOGIN_THROTTLE=1 to override.`, + ); + } + const last = recent[recent.length - 1]; + if (last !== undefined) { + const wait = MIN_INTERVAL_MS - (now() - last); + if (wait > 0) await sleep(wait); + } + const next: ThrottleRecord = { host, recent: [...recent, now()] }; + try { + await mkdir(throttleDir(env), { recursive: true }); + await writeFile(path, `${JSON.stringify(next)}\n`, "utf8"); + } catch { + // Best-effort: an unwritable cache means no throttle, never a failed command. + } + }, + }; +} diff --git a/src/commands/apply.ts b/src/commands/apply.ts index 4fd4cfb..95bf0db 100644 --- a/src/commands/apply.ts +++ b/src/commands/apply.ts @@ -21,6 +21,7 @@ interface ApplyOptions { backupDir?: string; autoApprove?: boolean; refresh?: boolean; + strictCatalog?: boolean; } // Retain this command-module export for callers that used it before the application extraction. @@ -35,6 +36,10 @@ export function applyCommand(): Command { .option("--confirm-env ", "confirm a protected env non-interactively (must match --env exactly)") .option("--backup-dir ", "directory for the pre-apply backup (or set CT_BACKUP_DIR)") .option("-y, --auto-approve", "skip the confirmation prompt") + .option( + "--strict-catalog", + "fail when a declared right or preserveUnknown dimension is absent from this host's permission catalog (default: skip it with a warning)", + ) .option( "--refresh", "after a successful apply, POST /dynamicgroups/{id}/refresh for each changed dynamic group (per-group only)", @@ -49,6 +54,7 @@ export function applyCommand(): Command { environment: opts.env, backupDir: opts.backupDir, refresh: opts.refresh, + strictCatalog: opts.strictCatalog, // No wall-clock expiry: the confirmation below blocks on stdin for as long as the // operator needs to read the rendered diff (#156 review). Staleness is caught by the // state fingerprint at execute time, not by a timer. diff --git a/src/commands/auth.ts b/src/commands/auth.ts index 9d1bad3..df971ed 100644 --- a/src/commands/auth.ts +++ b/src/commands/auth.ts @@ -1,5 +1,5 @@ import { Command } from "commander"; -import { runAuthLogin, runAuthLogout, runAuthStatus } from "../application/operations/auth.js"; +import { runAuthLogin, runAuthLogout, runAuthStatus, runAuthToken } from "../application/operations/auth.js"; import { normalizeHost } from "../config.js"; import { isSecureStorageAvailable } from "../auth/tokenStore.js"; import { bootstrapLoginToken } from "../auth/login.js"; @@ -119,6 +119,43 @@ export function authCommand(): Command { } }); + cmd + .command("token") + .description("Print a short-lived ChurchTools session for another tool (credential helper)") + .option("-e, --env ", "environment profile from ct.envs.json (targets that host)") + .option("--raw", "print only the session cookie, with no JSON envelope") + .option("--allow-tty", "print the credential even though stdout is a terminal") + .action(async (opts: { env?: string; raw?: boolean; allowTty?: boolean }) => { + // A credential is for a pipe, not for scrollback: a terminal keeps it in the buffer, in a + // `script` capture and in whatever the user pastes next. The whole point of emitting the + // SESSION rather than the login token is to shorten a leak's life — printing it where it will + // be kept works against that, so it takes an explicit flag (#179). + if (process.stdout.isTTY && !opts.allowTty) { + error( + "Refusing to print a credential to a terminal. Pipe it (e.g. `ct auth token --env dev | jq`), " + + "or pass --allow-tty if you really want it on screen.", + ); + process.exitCode = 1; + return; + } + try { + const result = await runAuthToken({ environment: opts.env }); + // stdout carries the credential and NOTHING else, so `$(ct auth token --raw)` is safe. Every + // line below — and every warning, prompt and progress message anywhere in ct — is on stderr. + process.stdout.write(opts.raw ? `${result.cookie}\n` : `${JSON.stringify(result)}\n`); + info( + `${result.host}${result.environment ? ` (env ${result.environment})` : ""} — session from ` + + `${result.source === "cache" ? "the keychain cache" : "a fresh login handshake"}, ` + + `reusable until ${result.expiresAt}.`, + ); + } catch (caught) { + // Nothing is written to stdout on failure: a consumer that reads stdout for a credential + // must get an empty stream, never a diagnostic it might mistake for one. + error(formatError(caught)); + process.exitCode = 1; + } + }); + cmd .command("logout") .description("Remove the stored host + login token") diff --git a/src/commands/export-tf.ts b/src/commands/export-tf.ts index d49c60d..e095e45 100644 --- a/src/commands/export-tf.ts +++ b/src/commands/export-tf.ts @@ -1,3 +1,4 @@ +import { relative } from "node:path"; import { Command } from "commander"; import { runExportTf } from "../application/operations/export-tf.js"; import { info, warn } from "../ui.js"; @@ -8,6 +9,7 @@ interface ExportTfOptions { only?: string[]; out: string; versions: boolean; + ids: boolean; } export function exportCommand(): Command { @@ -21,6 +23,10 @@ export function exportCommand(): Command { "--no-versions", "do not write or prune versions.tf — use when your repo owns it (e.g. to pin a provider version)", ) + .option( + "--no-ids", + "do not write .ct/ids..json — the key→id map ct resolves leftover references through once these resources leave its state", + ) .action(async (opts: ExportTfOptions) => { const { value, warnings } = await runExportTf({ statePath: opts.state, @@ -28,11 +34,13 @@ export function exportCommand(): Command { only: opts.only, outDir: opts.out, writeVersions: opts.versions, + writeIds: opts.ids, }); for (const file of value.files) info(`wrote ${opts.out}/${file}`); for (const r of value.relabelled) { info(`relabelled "${r.key}" -> "${r.label}" (HCL references must be identifiers)`); } + if (value.idMapPath) info(`wrote ${relative(process.cwd(), value.idMapPath)}`); info(`${value.imported} resources exported`); // Printed last, after the success line, so the gap is the final thing on // screen rather than scrolled off above a list of written files. diff --git a/src/commands/ids.ts b/src/commands/ids.ts new file mode 100644 index 0000000..42a7c5d --- /dev/null +++ b/src/commands/ids.ts @@ -0,0 +1,96 @@ +import { relative } from "node:path"; +import { Command } from "commander"; +import { runIdsList, runIdsSync } from "../application/operations/ids.js"; +import { error, formatError, info, out, success, warn } from "../ui.js"; + +/** + * `ct ids` — the committed OpenTofu id map (#181). + * + * The map is what keeps `campus: "mainz"` and `personStatus: "status_unbekannt"` resolving after + * those resources move to terraform-provider-churchtools and leave ct's state. `ct export tf` writes + * it during the cutover; this command keeps it current afterwards, when tofu is the only place new + * ids appear. + */ +export function idsCommand(): Command { + const cmd = new Command("ids").description( + "Maintain the committed key→id map ct resolves OpenTofu-owned resources through", + ); + + cmd + .command("sync") + .description("Rewrite .ct/ids..json from an OpenTofu state file (or stdin)") + .requiredOption( + "--tofu-state ", + 'path to a terraform.tfstate, or "-" for stdin (e.g. `tofu state pull | ct ids sync -e prod --tofu-state -`)', + ) + .option("-e, --env ", "environment profile from ct.envs.json (decides which host's map)") + .option("--dry-run", "report what would change without writing the map") + .action(async (opts: { tofuState: string; env?: string; dryRun?: boolean }) => { + try { + const { value, warnings, project } = await runIdsSync({ + tofuState: opts.tofuState, + environment: opts.env, + dryRun: opts.dryRun, + }); + const where = relative(project.cwd, value.path); + for (const entry of value.added) info(`+ ${entry.type}.${entry.key} = ${entry.id}`); + for (const { entry, previousId } of value.changed) { + info(`~ ${entry.type}.${entry.key} = ${entry.id} (was ${previousId})`); + } + for (const entry of value.removed) info(`- ${entry.type}.${entry.key} (was ${entry.id})`); + for (const w of warnings) warn(w.message); + const serial = value.serial === null ? "" : ` (tofu state serial ${value.serial})`; + if (value.written) { + success(`${where}: ${value.entries.length} ids from ${project.host}${serial}.`); + } else if (value.entries.length === 0) { + // Not "wrote 0 ids": the empty-map guard below REFUSES. A dry run has to name the outcome + // it would reach; a real run has to name the one it did reach, in the tense it happened in. + const verb = opts.dryRun ? "Would refuse to write" : "Refused to write"; + info(`${verb} ${where}: the state maps to no ids${serial}.`); + } else { + info(`Would write ${where}: ${value.entries.length} ids from ${project.host}${serial}.`); + } + // An empty read never overwrites a good map — the likeliest cause is the wrong workspace, + // and replacing 50 working ids with nothing would break every reference at once. + // + // The dry run exits non-zero on the SAME condition, rather than only the real run: a CI gate + // built on `--dry-run` exists to catch exactly this, and an exit code that disagrees with the + // run it is predicting would pass the check precisely when the real thing would fail it. + if (value.entries.length === 0) process.exitCode = 1; + } catch (caught) { + error(formatError(caught)); + process.exitCode = 1; + } + }); + + cmd + .command("list") + .description("Show the id map ct would resolve through for this host") + .option("-e, --env ", "environment profile from ct.envs.json (decides which host's map)") + .option("--json", "emit the entries as JSON") + .action(async (opts: { env?: string; json?: boolean }) => { + try { + const { value, project } = await runIdsList({ environment: opts.env }); + if (opts.json) { + out(value.entries); + return; + } + if (!value.path) { + info( + `No id map for ${project.host}. One is written by \`ct export tf\`, or by ` + + `\`ct ids sync --tofu-state \`.`, + ); + return; + } + info(`${relative(project.cwd, value.path)} — ${value.entries.length} ids for ${project.host}`); + for (const entry of value.entries) { + process.stdout.write(`${entry.type}.${entry.key} = ${entry.id}\n`); + } + } catch (caught) { + error(formatError(caught)); + process.exitCode = 1; + } + }); + + return cmd; +} diff --git a/src/commands/plan.ts b/src/commands/plan.ts index ef62fac..4d4ac0e 100644 --- a/src/commands/plan.ts +++ b/src/commands/plan.ts @@ -24,6 +24,7 @@ interface PlanOptions { outputBase?: string; locale?: string; detailedExitcode?: boolean; + strictCatalog?: boolean; } function collectFormat(value: string, previous: string[]): string[] { @@ -100,6 +101,10 @@ export function planCommand(): Command { ) .option("--output-base ", "write selected formats as .txt/.json/.md") .option("--locale ", "Markdown language: de-DE or en", "de-DE") + .option( + "--strict-catalog", + "fail when a declared right or preserveUnknown dimension is absent from this host's permission catalog (default: skip it with a warning)", + ) .option( "--detailed-exitcode", "Terraform-style exit code: 0 = no changes, 1 = error, 2 = changes pending (resource or permission)", @@ -112,12 +117,18 @@ export function planCommand(): Command { configPath: opts.config, statePath: opts.state, environment: opts.env, + strictCatalog: opts.strictCatalog, }); const { project, value } = result; const catalogPath = value.permissionCatalogPath ? relative(project.cwd, value.permissionCatalogPath) : null; if (catalogPath) info(`permission catalog: ${catalogPath}`); + // Said out loud for the same reason the catalog path is: a reference resolving through a + // committed map rather than through state or a live name is worth knowing before reading a diff. + if (value.tofuIdMapPath) { + info(`tofu id map: ${relative(project.cwd, value.tofuIdMapPath)}`); + } // Additive on top of the raw plan/permissions (#24) — existing consumers of `plan`/ // `permissions` are unaffected. Every projection below consumes this one computation. diff --git a/src/config/context.ts b/src/config/context.ts index 3b1c3e9..6177825 100644 --- a/src/config/context.ts +++ b/src/config/context.ts @@ -23,7 +23,7 @@ import { import { collectRefs } from "../resolve/refs.js"; import type { DomainType } from "../permissions/grants.js"; import type { DesiredPermission, Grant, PreserveUnknown } from "../permissions/types.js"; -import { KNOWN_SCOPE_FIELDS } from "../permissions/catalog.js"; +import { scopeFieldVerdict } from "../permissions/catalog.js"; import { GROUP_STATUS_NO_CATALOG, isRef, ref, refKey, type Ref } from "../resolve/refs.js"; import { normalizeScopeEntry } from "../permissions/scope.js"; import { conventionalRulesetRef, isCallerAssignedId, knownFields } from "../resources/registry.js"; @@ -281,7 +281,11 @@ function normalizePreserveUnknown( `default, or name the dimensions to leave alone.`, ); } - const unknown = value.filter((d) => !KNOWN_SCOPE_FIELDS.has(d)); + // A dimension the ACTIVE catalog lacks but ct's bundled catalog knows is a host difference, not a + // typo (#178): the module that scopes by it is not installed here. It stays in the declaration and + // is reported by `buildPermissionPlan`; only a name no catalog defines is still fatal, because that + // is the case this guard was written for — a typo silently preserving nothing. + const unknown = value.filter((d) => scopeFieldVerdict(d) === "unknown"); if (unknown.length > 0) { throw new Error( `${domainType} "${input.key}": "preserveUnknown" names ${unknown.length === 1 ? "a scope dimension" : "scope dimensions"} ` + diff --git a/src/index.ts b/src/index.ts index 4c911d7..b628fc4 100644 --- a/src/index.ts +++ b/src/index.ts @@ -5,6 +5,7 @@ import { getCommand } from "./commands/get.js"; import { adoptCommand } from "./commands/adopt.js"; import { stateCommand } from "./commands/state.js"; import { exportCommand } from "./commands/export-tf.js"; +import { idsCommand } from "./commands/ids.js"; import { coverageCommand } from "./commands/coverage.js"; import { permissionsCommand } from "./commands/permissions.js"; import { reportCommand } from "./commands/report.js"; @@ -36,6 +37,7 @@ export function buildProgram(): Command { program.addCommand(adoptCommand()); program.addCommand(stateCommand()); program.addCommand(exportCommand()); + program.addCommand(idsCommand()); program.addCommand(coverageCommand()); program.addCommand(permissionsCommand()); program.addCommand(reportCommand()); diff --git a/src/permissions/catalog.ts b/src/permissions/catalog.ts index 6bd6e19..20f1552 100644 --- a/src/permissions/catalog.ts +++ b/src/permissions/catalog.ts @@ -43,6 +43,36 @@ function splitCatalog(data: unknown): { rights: Record; me const bundled = splitCatalog(catalogData); +/** + * The rights in the catalog BUNDLED with this release, kept reachable after {@link useCatalog} has + * pointed everything else at a per-instance capture (#178). + * + * It is the only evidence `ct` has for the difference between the two ways a declared right can be + * absent from the active catalog: + * + * - absent here too → the name is a typo, or a right ChurchTools deleted. A hard error, as before. + * - present here → the name is real, this HOST just does not have it (a module that is not + * installed). Skipped with a warning, so ONE inapplicable declaration cannot + * take down planning for the 100 that do apply. + * + * That asymmetry is deliberate: the bundled snapshot is a real instance's full right set, so it is a + * usable "does this name exist anywhere ct knows" oracle even though it is the wrong oracle for + * "what can this host do". + */ +export const BUNDLED_CATALOG: Readonly> = bundled.rights; + +/** The ChurchTools version the bundled snapshot was captured against, for a warning that has to name + * BOTH catalogs (#178). `null` on a legacy catalog with no `$meta`. */ +export const BUNDLED_CATALOG_VERSION: string | null = bundled.meta?.ctVersion ?? null; + +/** Scope dimensions the bundled catalog knows — the same oracle as {@link BUNDLED_CATALOG}, for + * `preserveUnknown` dimensions (#178). */ +export const BUNDLED_SCOPE_FIELDS: ReadonlySet = new Set( + Object.values(bundled.rights) + .map((e) => e.scopeField) + .filter((f): f is string => f != null), +); + /** * The permission catalog: name → authId bridge. Bundled at build time (see the module header), and * REPLACEABLE at runtime by a per-instance capture (#105 — see {@link useCatalog}). @@ -120,6 +150,59 @@ export function useCatalog(data: unknown, opts: { perInstance?: boolean } = {}): /** Restore the catalog bundled with this release. Exists so tests can undo {@link useCatalog}. */ export function useBundledCatalog(): void { useCatalog(catalogData); + STRICT_CATALOG = false; +} + +/** + * `--strict-catalog` (#178): restore the pre-#178 behaviour, where ANY declaration the active + * catalog cannot resolve is a hard error rather than a skip. + * + * A module-level flag rather than a threaded option because the two places that need it sit at + * opposite ends of the pipeline — config EVALUATION (`preserveUnknown` dimensions, deep inside a + * user's `ct.config.ts` callback) and plan BUILDING — and the catalog they consult is already + * process-wide mutable state (`CATALOG` is swapped by {@link useCatalog}). Threading a flag through + * both would add a parameter to every layer in between without making anything more honest. + */ +export let STRICT_CATALOG = false; + +export function setStrictCatalog(strict: boolean): void { + STRICT_CATALOG = strict; +} + +/** + * Why a name is not in the active catalog (#178). + * + * - `known` — it is; carry on. + * - `host-missing` — the active catalog is a capture from THIS host and lacks it, but the bundled + * catalog has it: a real right this instance does not have. Skip + warn. + * - `unknown` — no catalog `ct` has ever seen defines it: a typo or a deleted right. Hard error. + * + * `host-missing` requires a per-instance capture to be active. Without one the active catalog IS the + * bundled snapshot, so "missing from the active catalog" and "missing everywhere" are the same + * statement, and the verdict stays `unknown` — a repo that has not captured its host's catalog gets + * exactly today's behaviour, including today's errors. + */ +export type CatalogVerdict = "known" | "host-missing" | "unknown"; + +export function catalogVerdict(name: string): CatalogVerdict { + if (CATALOG[name]) return "known"; + if (!STRICT_CATALOG && CATALOG_IS_PER_INSTANCE && BUNDLED_CATALOG[name]) return "host-missing"; + return "unknown"; +} + +/** The same verdict for a scope DIMENSION rather than a right (#178, `preserveUnknown`). */ +export function scopeFieldVerdict(dimension: string): CatalogVerdict { + if (KNOWN_SCOPE_FIELDS.has(dimension)) return "known"; + if (!STRICT_CATALOG && CATALOG_IS_PER_INSTANCE && BUNDLED_SCOPE_FIELDS.has(dimension)) + return "host-missing"; + return "unknown"; +} + +/** How to describe the active catalog in a warning: where it came from and how big it is. */ +export function describeCatalog(): string { + if (!CATALOG_META) return `${Object.keys(CATALOG).length} rights`; + const scope = CATALOG_IS_PER_INSTANCE ? "this host's catalog" : "ct's bundled catalog"; + return `${scope}: ChurchTools ${CATALOG_META.ctVersion}, ${Object.keys(CATALOG).length} rights`; } export function resolveAuthId(name: string): CatalogEntry { diff --git a/src/permissions/plan.ts b/src/permissions/plan.ts index 957a013..04bacb7 100644 --- a/src/permissions/plan.ts +++ b/src/permissions/plan.ts @@ -10,6 +10,10 @@ import type { State } from "../state/state.js"; import type { DesiredResource } from "../engine/types.js"; import { resolveAuthId, + catalogVerdict, + describeCatalog, + scopeFieldVerdict, + BUNDLED_CATALOG_VERSION, CATALOG_META, CATALOG_IS_PER_INSTANCE, KNOWN_AUTH_IDS, @@ -49,6 +53,26 @@ export interface PermissionPlanItem { diff: GrantDiff; } +/** + * A declaration this HOST cannot act on, because the right (or `preserveUnknown` dimension) it names + * is absent from the host's own permission catalog while ct's bundled catalog has it (#178). + * + * Reported, never fatal. An estate whose instances have different modules installed cannot express + * the difference in one declarative file, so a `jpmFlowManager:*` right that is correct on prod used + * to make the whole plan — including the 100 rights that DO apply — abort on dev. The mirror case has + * always been handled this way: a live grant whose authId the catalog cannot name is reported and + * left untouched. This is the same posture in the other direction — say what you cannot manage, + * manage the rest. + */ +export interface CatalogSkip { + domainType: DomainType; + /** The declaration's key, e.g. the group_role key. */ + key: string; + /** What was skipped: a right name, or a `preserveUnknown` scope dimension. */ + name: string; + kind: "right" | "dimension"; +} + /** * Fan out each grant to (authId, dataId) tuples. ChurchTools reads a scoped grant back as * ONE ROW PER dataId with a scalar `dataId` (see `normalizeActual`), so a desired tuple with @@ -61,9 +85,18 @@ export function desiredTuples( state: State, declaredGroupKeys: ReadonlySet = new Set(), scopeRefs: ScopeRefMap = new Map(), + onSkip?: (skip: CatalogSkip) => void, ): GrantTuple[] { return p.grants.flatMap((g): GrantTuple[] => { const name = typeof g === "string" ? g : g.right; + // A right this host's catalog does not define, but ct's bundled catalog does (#178): emit NO + // tuple at all. No tuple means no PUT, and — because the host cannot have a live row under a + // name it does not define — nothing this declaration could have owned goes unclaimed, so + // nothing lands in toDelete either. The grant is simply not this host's business. + if (catalogVerdict(name) === "host-missing") { + onSkip?.({ domainType: p.domainType, key: p.key, name, kind: "right" }); + return []; + } const entry = resolveAuthId(name); if (typeof g === "string") { // A scoped right declared as a bare string would emit `dataId: []` — a silent GLOBAL grant. @@ -206,6 +239,32 @@ async function resolveDomainIds( return resolved; } +/** + * One line per skipped declaration (#178), naming the right, the declaration, and BOTH catalogs — + * the host's (what it can do) and the bundled one (where the name came from). Without the second + * half the warning reads like a typo; with it, it reads like the module difference it is. + */ +function renderSkip(skip: CatalogSkip): string { + // A skip can only arise while a per-instance capture is active AND the bundled catalog has the name + // (that is what `catalogVerdict` decided), so the provenance clause always has something to say. + const provenance = BUNDLED_CATALOG_VERSION + ? ` ct's bundled catalog (ChurchTools ${BUNDLED_CATALOG_VERSION}) defines it, so this reads as a ` + + `module this instance does not have.` + : ""; + const what = + skip.kind === "right" + ? `right "${skip.name}" is absent from this host's permission catalog` + : `"preserveUnknown" names the scope dimension "${skip.name}", which no right in this host's permission catalog scopes by`; + const consequence = + skip.kind === "right" + ? "skipped for this host — never granted, never revoked" + : "ignored for this host — it can preserve nothing here"; + return ( + `${skip.domainType} "${skip.key}": ${what} (${describeCatalog()}) — ${consequence}.${provenance} ` + + `Pass --strict-catalog to fail on it instead.` + ); +} + export async function buildPermissionPlan( client: PermissionReader, state: State, @@ -217,6 +276,22 @@ export async function buildPermissionPlan( const items: PermissionPlanItem[] = []; const fetchErrors: string[] = []; const warnings: string[] = []; + // Deduped by declaration+name: a right declared on two roles is two skips, the same right listed + // twice on one role is one. + const skips = new Map(); + const noteSkip = (skip: CatalogSkip): void => { + skips.set(`${skip.domainType}:${skip.key}:${skip.kind}:${skip.name}`, skip); + }; + // `preserveUnknown` dimensions are validated at config-eval time (config/context.ts), which keeps + // a typo fatal — but a dimension that exists only on the OTHER host cannot be judged there without + // making the config non-portable, so it is passed through and reported here instead (#178). + for (const p of permissions) { + if (!Array.isArray(p.preserveUnknown)) continue; + for (const dimension of p.preserveUnknown) { + if (scopeFieldVerdict(dimension) === "host-missing") + noteSkip({ domainType: p.domainType, key: p.key, name: dimension, kind: "dimension" }); + } + } // Catalog staleness (#25/#105): the catalog is a snapshot captured against one CT version. If the // live instance reports a different version, right names/authIds/scopeFields may have drifted — // warn (never fail) so the diff is trusted-but-verified. @@ -280,7 +355,7 @@ export async function buildPermissionPlan( // domainId is irrelevant to desiredTuples (it only reads key/domainType/grants); pass the // pending Ref through so the shape stays a valid DesiredPermission. diff: diffGrants( - desiredTuples({ ...p, domainId: p.pendingDomain }, state, declaredGroupKeys, scopeRefs), + desiredTuples({ ...p, domainId: p.pendingDomain }, state, declaredGroupKeys, scopeRefs, noteSkip), [], ), }); @@ -327,7 +402,7 @@ export async function buildPermissionPlan( domainType: p.domainType, domainId: p.domainId, diff: diffGrants( - desiredTuples(p, state, declaredGroupKeys, scopeRefs), + desiredTuples(p, state, declaredGroupKeys, scopeRefs, noteSkip), knownActual, preservePredicateFor(p.preserveUnknown), // The unknown-authId guard (#25) deliberately does NOT apply here: an unnameable right can @@ -336,5 +411,10 @@ export async function buildPermissionPlan( ), }); } + for (const skip of [...skips.values()].sort((a, b) => + `${a.key}${a.name}` < `${b.key}${b.name}` ? -1 : 1, + )) { + warnings.push(renderSkip(skip)); + } return { items, fetchErrors, warnings }; } diff --git a/src/resolve/idMap.ts b/src/resolve/idMap.ts new file mode 100644 index 0000000..19b48ef --- /dev/null +++ b/src/resolve/idMap.ts @@ -0,0 +1,185 @@ +/** + * The OpenTofu id map: a committed, per-host `key -> id` table for resources ct no longer declares + * (#181). + * + * ## The blocker this exists for + * + * The tier-0 cutover moves campuses, group types, departments, person statuses and comment viewers + * to `terraform-provider-churchtools`, which means their tables leave `ct.config.ts` AND their + * entries leave `ct-state..json`. Hundreds of REFERENCES stay behind (`campus: "mainz"`, + * `personStatus: "status_unbekannt"`), and those still have to resolve. + * + * `verify` calls them "logical ref, resolved live — not compared", which is true but only as a + * FALLBACK: while the object was in ct's state, the key resolved from state; with the state entry + * gone, resolution falls back to matching the key against the live object's NAME. ct's tier-0 keys + * were never name-derived, so that fallback cannot work for them — structurally, not by accident: + * + * key live name + * status_unbekannt Unbekannt + * status_5_core 5 - Core + * egc Equippers Germany Central + * + * No slug of those names produces those keys. And a literal id is not an option either: one config + * serves two hosts, and 39 of 43 tier-0 ids differ between them. + * + * ## Why a committed file rather than reading tofu's remote state + * + * After the cutover, tofu's state IS the authoritative key→id map, and it keys on exactly the + * logical names the config already uses (`churchtools_person_status.status_unbekannt` came from + * `status_unbekannt`). Reading it directly, though, would put an S3 backend — credentials, a network + * dependency, one AWS SDK — inside a CLI whose other reads are all ChurchTools. So the map is a + * file: written by `ct export tf` from the state it is exporting, refreshed from a `tofu state pull` + * by `ct ids sync`, and committed per host so portability survives. + * + * ## Where it sits in resolution + * + * AFTER ct's own managed state (ct never stops trusting what it owns) and BEFORE the live catalog + * (the map is exact, the catalog match is a name guess). See `Resolver.resolve`. + */ +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { CATALOG_DIR, hostSlug } from "../permissions/catalog-store.js"; + +/** One resolvable resource: its ct key, its id on this host, and the HCL label tofu knows it by. */ +export interface IdMapEntry { + type: string; + key: string; + id: number; + /** + * The HCL label the exporter gave it, when that differs from the key (`3_groupactive` → + * `g_3_groupactive`). Retained so `ct ids sync` can map a tofu resource address BACK to the ct key + * it came from — the relabelling is many-to-one, so it cannot be re-derived from the label alone. + */ + label?: string; +} + +export interface IdMapMeta { + host: string; + source: string; + generatedAt: string; + entries: number; + [k: string]: unknown; +} + +/** The map as the resolver consumes it: exact lookups, plus the label index `ids sync` needs. */ +export interface IdMap { + path: string; + meta: IdMapMeta | null; + entries: IdMapEntry[]; + /** `type\0key` → id. */ + ids: ReadonlyMap; + /** `type\0label` → ct key, for reading a tofu state back. */ + keysByLabel: ReadonlyMap; +} + +function entryKey(type: string, key: string): string { + return `${type}\u0000${key}`; +} + +export function idMapPath(host: string, dir: string = CATALOG_DIR): string { + return join(dir, `ids.${hostSlug(host)}.json`); +} + +/** + * The file shape: `$meta` plus one object per resource type, exactly like the permission catalog's + * layout, so the two committed artefacts read the same way in a diff. + * + * { "$meta": {...}, "campus": { "mainz": { "id": 0 } }, "person-status": { "status_5_core": { "id": 6 } } } + */ +export function renderIdMap(host: string, entries: readonly IdMapEntry[], source: string): string { + const byType: Record> = {}; + // Sorted by type then key: a committed file has to be byte-stable across runs, or every export + // shows up as a diff (same reason the HCL export sorts). + for (const entry of [...entries].sort((a, b) => + entryKey(a.type, a.key) < entryKey(b.type, b.key) ? -1 : 1, + )) { + (byType[entry.type] ??= {})[entry.key] = + entry.label !== undefined && entry.label !== entry.key + ? { id: entry.id, label: entry.label } + : { id: entry.id }; + } + const meta: IdMapMeta = { + host, + source, + generatedAt: new Date().toISOString().slice(0, 10), + entries: entries.length, + }; + return `${JSON.stringify({ $meta: meta, ...byType }, null, 1)}\n`; +} + +export function parseIdMap(raw: string, path: string, host: string): IdMap { + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch (err) { + throw new Error(`Malformed id map ${path}: not valid JSON (${(err as Error).message}).`); + } + if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) { + throw new Error(`Malformed id map ${path}: expected a JSON object at the top level.`); + } + const { $meta, ...types } = parsed as { $meta?: IdMapMeta } & Record; + // Host-checked like the session cache (#30) and asserted like the state file: a map applied to the + // wrong instance would resolve every reference to a plausible id on the wrong host, which is worse + // than not resolving at all — nothing downstream could tell. + if ($meta && typeof $meta.host === "string" && hostSlug($meta.host) !== hostSlug(host)) { + throw new Error( + `Id map ${path} was generated for ${$meta.host}, but the resolved host is ${host}. ` + + `Re-generate it for this host (\`ct export tf --env \`) — applying another host's ids ` + + `would resolve every reference to the wrong resource.`, + ); + } + const entries: IdMapEntry[] = []; + for (const [type, rows] of Object.entries(types)) { + if (typeof rows !== "object" || rows === null || Array.isArray(rows)) { + throw new Error(`Malformed id map ${path}: "${type}" is not an object of key → { id }.`); + } + for (const [key, value] of Object.entries(rows as Record)) { + const row = value as { id?: unknown; label?: unknown }; + if (typeof row?.id !== "number" || !Number.isFinite(row.id)) { + throw new Error(`Malformed id map ${path}: ${type} "${key}" has no numeric id.`); + } + entries.push({ + type, + key, + id: row.id, + ...(typeof row.label === "string" ? { label: row.label } : {}), + }); + } + } + return { + path, + meta: $meta ?? null, + entries, + ids: new Map(entries.map((e) => [entryKey(e.type, e.key), e.id])), + keysByLabel: new Map(entries.map((e) => [entryKey(e.type, e.label ?? e.key), e.key])), + }; +} + +/** + * Load this host's map, or `null` when there is none (every reference then resolves exactly as it + * does today). A malformed or foreign map THROWS rather than being skipped: a repo that committed + * one is relying on it, and silently resolving without it is how the cutover breaks unnoticed. + */ +export async function loadIdMap(host: string, dir: string = CATALOG_DIR): Promise { + const path = idMapPath(host, dir); + let raw: string; + try { + raw = await readFile(path, "utf8"); + } catch (err) { + if ((err as { code?: string }).code === "ENOENT") return null; + throw err; + } + return parseIdMap(raw, path, host); +} + +export async function writeIdMap( + host: string, + entries: readonly IdMapEntry[], + dir: string = CATALOG_DIR, + source = "ct export tf", +): Promise { + await mkdir(dir, { recursive: true }); + const path = idMapPath(host, dir); + await writeFile(path, renderIdMap(host, entries, source), "utf8"); + return path; +} diff --git a/src/resolve/resolver.ts b/src/resolve/resolver.ts index 0c91814..54698b2 100644 --- a/src/resolve/resolver.ts +++ b/src/resolve/resolver.ts @@ -7,6 +7,12 @@ * state resolves to a {@link PendingRef} (its id is only known after the * resource tier applies — re-resolved at apply time, mirroring the permission * scope pattern in src/permissions/scope.ts). + * 1b. The committed OpenTofu id map for this host (`.ct/ids..json`, #181), an EXACT + * `key -> id` table for resources that moved to terraform-provider-churchtools and are therefore + * no longer in ct's state. After managed state (ct never stops trusting what it owns) and before + * the catalog below (an exact table beats a name guess) — and it is what keeps a reference like + * `personStatus: "status_unbekannt"` resolving at all, since ct's tier-0 keys are not derived + * from the live names the catalog matches on. * 2. Live catalog master data, matched by `slug(name) === key` with an exact-name * secondary: campus → /campuses, group-type → /group/grouptypes, role-def → /group/roles. * Each catalog is fetched at most once per run and cached by a `Map`, @@ -27,6 +33,7 @@ */ import type { CtClient } from "../api/ctClient.js"; import type { State } from "../state/state.js"; +import type { IdMap } from "./idMap.js"; import type { DesiredResource } from "../engine/types.js"; import { slug } from "../resources/registry.js"; import { @@ -127,6 +134,19 @@ const CATALOG_PATH: Partial> = { "role-def": "/group/roles", }; +/** + * Can a reference of this kind resolve WITHOUT ct's managed state — i.e. does it have a live + * master-data catalog to fall back to (#180)? + * + * Read by `ct state rm`, which has to predict what removing a state entry does to the NEXT plan. A + * `campus`/`person-status`/`group-type` ref falls back to the catalog and keeps resolving; a `group` + * ref has no catalog at all (groups are managed-only), so removing the group it names turns the next + * plan into a hard error. Same table, one question, so the two cannot drift. + */ +export function refKindResolvesLive(kind: RefKind): boolean { + return CATALOG_PATH[kind] !== undefined; +} + interface CatalogRecord { id: number; name?: string; @@ -146,6 +166,11 @@ export interface ResolverDeps { desired: DesiredResource[]; /** Host label for error messages. Defaults to `state.host`. */ host?: string; + /** + * The committed OpenTofu id map for this host (#181), when one exists. Loaded by the command layer + * (like the per-instance permission catalog) so the resolver stays filesystem-free. + */ + idMap?: Pick | null; } /** @@ -197,6 +222,7 @@ export class Resolver { private readonly client: Pick & Partial>; private readonly state: State; private readonly host: string; + private readonly idMap: Pick | null; private readonly catalogs = new Map>(); /** Per-group role list cache (group_role domain resolution), keyed by group id, fetched at most once. */ private readonly groupRoleLists = new Map>(); @@ -229,6 +255,7 @@ export class Resolver { this.client = deps.client; this.state = deps.state; this.host = deps.host ?? deps.state.host; + this.idMap = deps.idMap ?? null; for (const d of deps.desired) { let set = this.declaredByType.get(d.type); if (!set) { @@ -267,6 +294,13 @@ export class Resolver { if (managed && managed.type === type) return managed.id; if (this.declaredByType.get(type)?.has(r.key)) return pendingRef(r); } + // (1b) the OpenTofu id map (#181): an exact key→id table for what tofu now owns. Consulted only + // for kinds that HAVE a managed resource type — the map is written from resources ct exported, + // so a kind ct never manages can never be in it, and asking would only slow the miss down. + if (type !== undefined) { + const mapped = this.idMap?.ids.get(`${type}\u0000${r.key}`); + if (mapped !== undefined) return mapped; + } // (2) live catalog if (CATALOG_PATH[r.kind] !== undefined) return this.resolveFromCatalog(r, site); // (3) hard error @@ -636,6 +670,17 @@ export class Resolver { const where = catalog ? `no managed resource and no live ${r.kind} at ${catalog} matches key "${r.key}"` : `no managed ${r.kind} named "${r.key}" is declared or adopted`; + // A repo mid-cutover has the map loaded and the key missing from it — by far the likeliest + // reason a reference that used to resolve stops (#181), and the one remedy the generic advice + // below does not name. + if (this.idMap) { + return new Error( + `Cannot resolve ${refLabel(r)} referenced at ${site} on ${this.host}: ${where}, and ` + + `${this.idMap.path} has no ${r.kind} "${r.key}" either. If OpenTofu owns it now, refresh the ` + + `map (\`tofu state pull | ct ids sync --tofu-state -\`); otherwise declare/adopt it, fix the ` + + `key/name, or use a numeric id.`, + ); + } return new Error( `Cannot resolve ${refLabel(r)} referenced at ${site} on ${this.host}: ${where}. ` + `Declare/adopt it, fix the key/name, or use a numeric id.`, diff --git a/src/resolve/tfstate.ts b/src/resolve/tfstate.ts new file mode 100644 index 0000000..806d832 --- /dev/null +++ b/src/resolve/tfstate.ts @@ -0,0 +1,103 @@ +/** + * Reading an OpenTofu state file into an {@link IdMapEntry} list (#181) — PURE. + * + * `ct export tf` can write the id map from the state it is exporting, but only for as long as ct + * still holds those entries. Once tier-0 belongs to tofu, tofu is the only place new ids appear: a + * campus created by `tofu apply` exists in no ct state file, so an export cannot know about it. That + * is what this reader is for — `tofu state pull | ct ids sync -e prod --tofu-state -` refreshes the + * map from the authoritative source without ct ever learning to speak S3. + * + * Kept pure (a parsed JSON value in, entries out) so it can be tested against fixture state without + * a filesystem, like the HCL renderer it mirrors. + */ +import { hclType } from "../export/hcl.js"; +import { EXPORTABLE_TYPES } from "../export/layout.js"; +import type { IdMapEntry } from "./idMap.js"; + +/** Terraform resource type → ct resource type, derived from the exporter's own table so the two + * directions cannot drift. */ +const CT_TYPE_BY_HCL_TYPE: ReadonlyMap = new Map( + EXPORTABLE_TYPES.map((ctType) => [hclType(ctType), ctType] as const), +); + +export interface TfStateReadResult { + entries: IdMapEntry[]; + /** Provider resources in the state whose type ct has no mapping for — reported, never fatal. */ + unmapped: string[]; + /** `for_each`/`count` blocks, whose label is not a resource key — reported, never guessed at. */ + multiInstance: string[]; + /** Terraform's own serial, if present, so a sync can say which state version it read. */ + serial: number | null; +} + +interface TfResource { + mode?: string; + type?: string; + name?: string; + instances?: { attributes?: Record; index_key?: unknown }[]; +} + +/** + * Turn a parsed `terraform.tfstate` into id-map entries. + * + * `keysByLabel` maps a `type\0label` back to the ct key it was exported from. The exporter relabels + * keys that are not valid HCL identifiers (`3_groupactive` → `g_3_groupactive`) and that mapping is + * many-to-one, so it cannot be inverted by rule — without the previous map's help, the label IS the + * key, which is correct for every key that needed no relabelling. + */ +export function readTfState( + parsed: unknown, + keysByLabel: ReadonlyMap = new Map(), +): TfStateReadResult { + if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) { + throw new Error("Malformed tofu state: expected a JSON object at the top level."); + } + const state = parsed as { resources?: unknown; serial?: unknown }; + if (state.resources !== undefined && !Array.isArray(state.resources)) { + throw new Error('Malformed tofu state: "resources" is not an array.'); + } + const entries: IdMapEntry[] = []; + const unmapped = new Set(); + const multiInstance = new Set(); + for (const raw of (state.resources ?? []) as TfResource[]) { + // `data` blocks describe reads, not managed objects: their id is whatever the data source + // happened to return, and nothing in the config keys off it. + if (raw.mode !== undefined && raw.mode !== "managed") continue; + if (typeof raw.type !== "string" || typeof raw.name !== "string") continue; + const ctType = CT_TYPE_BY_HCL_TYPE.get(raw.type); + if (!ctType) { + // Anything the provider manages that ct has no resource type for — a churchtools_group once + // the provider grows one, or another provider's resources in a shared state. + if (raw.type.startsWith("churchtools_")) unmapped.add(raw.type); + continue; + } + const instances = raw.instances ?? []; + // A `for_each`/`count` block is ONE resource with many instances, and its `name` is the block's + // (`campuses`), not any resource's key. Taking `[0]` would map that one label to one arbitrary + // id and report every real key as removed — and because the result is non-empty, the + // empty-map guard in `ct ids sync` would not catch it. Report it and map nothing. + if (instances.length > 1 || instances[0]?.index_key !== undefined) { + multiInstance.add(`${raw.type}.${raw.name}`); + continue; + } + const attributes = instances[0]?.attributes ?? {}; + // NOT `Number(attributes.id)`: `Number(null)`, `Number("")`, `Number([])` and `Number(false)` are + // all 0, and `Number.isFinite(0)` is true — so a resource that is mid-create, tainted or + // hand-edited would be written into the map as id 0. Zero is a REAL ChurchTools id here (the + // comment-viewer "Alle", and campus 0 in ct's own fixtures), so the reference would resolve + // silently to the wrong live object instead of failing. Only a genuine number or numeric string. + const rawId = attributes.id; + if (typeof rawId !== "number" && !(typeof rawId === "string" && rawId.trim() !== "")) continue; + const id = Number(rawId); + // Mid-create or tainted: an unresolvable reference is a better outcome than a wrong one. + if (!Number.isFinite(id)) continue; + const key = keysByLabel.get(`${ctType}\u0000${raw.name}`) ?? raw.name; + entries.push(key === raw.name ? { type: ctType, key, id } : { type: ctType, key, id, label: raw.name }); + } + return { + entries, + unmapped: [...unmapped].sort(), + multiInstance: [...multiInstance].sort(), + serial: typeof state.serial === "number" ? state.serial : null, + }; +} diff --git a/tests/auth-token-command.test.ts b/tests/auth-token-command.test.ts new file mode 100644 index 0000000..bbcbbae --- /dev/null +++ b/tests/auth-token-command.test.ts @@ -0,0 +1,131 @@ +/** + * `ct auth token` — a credential helper for the OpenTofu provider (#179). + * + * The provider declares `token` as required, so the tier-0 cutover meant writing a personal + * ChurchTools login token to local disk: permanent, unscopable, and an admin credential on prod. + * What this command emits instead is the SESSION that token buys — the one credential in the system + * that expires and can be revoked (`ct auth logout`), so a copy that leaks into a `tofu` debug log + * dies in hours rather than never. + * + * The contract under test is the one a `credential_process`-style consumer depends on: + * the credential on stdout and NOTHING else there, everything human on stderr, and an empty stdout + * plus a non-zero exit when nothing resolves. + */ +import { describe, it, expect, beforeEach, afterEach, vi } from "vitest"; +import type * as TokenStore from "../src/auth/tokenStore.js"; + +const authedSession = vi.fn(); +const LOGIN_TOKEN = "permanent-login-token-xyz"; +const readToken = vi.fn(async () => LOGIN_TOKEN); +vi.mock("../src/api/session.js", () => ({ authedSession })); +vi.mock("../src/auth/tokenStore.js", async (importOriginal) => ({ + ...(await importOriginal()), + readToken, +})); + +const { authCommand } = await import("../src/commands/auth.js"); + +const HOST = "https://mychurch.church.tools"; +const originalHost = process.env.CT_HOST; +let stdout: string[]; +let stderr: string[]; + +function session(overrides: Partial<{ cookie: string; source: "cache" | "handshake" }> = {}): void { + authedSession.mockResolvedValue({ + me: { id: 1 }, + client: { + sessionCredential: () => ({ + cookie: overrides.cookie ?? "ChurchTools_mychurch=abc", + csrfToken: "csrf-1", + obtainedAt: Date.parse("2026-09-21T00:00:00.000Z"), + source: overrides.source ?? "cache", + }), + }, + }); +} + +async function run(args: string[]): Promise { + await authCommand().parseAsync(["token", ...args], { from: "user" }); +} + +beforeEach(() => { + process.env.CT_HOST = HOST; + stdout = []; + stderr = []; + vi.spyOn(process.stdout, "write").mockImplementation((chunk) => { + stdout.push(String(chunk)); + return true; + }); + vi.spyOn(process.stderr, "write").mockImplementation((chunk) => { + stderr.push(String(chunk)); + return true; + }); + // A terminal is refused on purpose; every test below is the piped case unless it says otherwise. + Object.defineProperty(process.stdout, "isTTY", { value: false, configurable: true }); + process.exitCode = 0; + readToken.mockResolvedValue(LOGIN_TOKEN); + session(); +}); + +afterEach(() => { + vi.restoreAllMocks(); + process.exitCode = 0; + if (originalHost === undefined) delete process.env.CT_HOST; + else process.env.CT_HOST = originalHost; +}); + +describe("ct auth token", () => { + it("puts the session on stdout and nothing else there", async () => { + await run([]); + expect(stdout).toHaveLength(1); + const payload = JSON.parse(stdout[0]!) as Record; + expect(payload).toMatchObject({ + host: HOST, + cookie: "ChurchTools_mychurch=abc", + csrfToken: "csrf-1", + source: "cache", + }); + // The reuse ceiling, so a consumer can cache it rather than call ct on every single request. + expect(payload.expiresAt).toBe("2026-09-21T12:00:00.000Z"); + // The permanent credential is never part of the answer. + expect(stdout[0]).not.toContain(LOGIN_TOKEN); + }); + + it("emits the bare cookie under --raw, so command substitution is safe", async () => { + await run(["--raw"]); + expect(stdout).toEqual(["ChurchTools_mychurch=abc\n"]); + // Progress belongs on stderr: `$(ct auth token --raw)` must capture the credential alone. + expect(stderr.join("")).toContain(HOST); + expect(stderr.join("")).not.toContain("ChurchTools_mychurch=abc"); + }); + + it("writes nothing to stdout and exits non-zero when no credential resolves", async () => { + readToken.mockResolvedValue(null as never); + await run([]); + expect(stdout).toEqual([]); + expect(process.exitCode).toBe(1); + // The remedy, not just the refusal — this is the case a helper hits most often. + expect(stderr.join("")).toMatch(/ct auth login/); + }); + + it("refuses a terminal, where the credential would stay in the scrollback", async () => { + Object.defineProperty(process.stdout, "isTTY", { value: true, configurable: true }); + await run([]); + expect(stdout).toEqual([]); + expect(process.exitCode).toBe(1); + expect(stderr.join("")).toMatch(/Refusing to print a credential to a terminal/); + }); + + it("prints to a terminal when explicitly asked", async () => { + Object.defineProperty(process.stdout, "isTTY", { value: true, configurable: true }); + await run(["--allow-tty", "--raw"]); + expect(stdout).toEqual(["ChurchTools_mychurch=abc\n"]); + expect(process.exitCode).toBe(0); + }); + + it("says whether a login handshake was spent", async () => { + session({ source: "handshake" }); + await run(["--raw"]); + expect(stderr.join("")).toMatch(/fresh login handshake/); + }); +}); diff --git a/tests/export/export-tf.test.ts b/tests/export/export-tf.test.ts index 8125f50..9ff453f 100644 --- a/tests/export/export-tf.test.ts +++ b/tests/export/export-tf.test.ts @@ -211,3 +211,96 @@ describe("writeVersions: false", () => { expect(files).toContain("versions.tf"); }); }); + +/** + * The id map written beside the export (#181): the same key→id table, for ct's own resolution once + * these resources leave its state. Written under `.ct/`, not into the tofu root module — it is ct's + * input, not part of what tofu reads. + */ +describe("runExportTf — the id map", () => { + it("writes one entry per exported resource, keyed by host", async () => { + const result = await exportInto({ + mainz: row("campus", "mainz", 0), + struktur: row("group-type", "struktur", 9), + jugend: row("group", "jugend", 99), // not exportable → not in the map either + }); + expect(result.value.idMapPath).toBe(join(dir, ".ct", "ids.example.church.tools.json")); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map.campus).toEqual({ mainz: { id: 0 } }); + expect(map["group-type"]).toEqual({ struktur: { id: 9 } }); + expect(map.group).toBeUndefined(); + expect(map.$meta).toMatchObject({ host: HOST, source: "ct export tf", entries: 2 }); + }); + + it("records the HCL relabelling, which cannot be inverted by rule", async () => { + const result = await exportInto({ "3_active": row("group-type", "3_active", 4) }); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map["group-type"]).toEqual({ "3_active": { id: 4, label: "g_3_active" } }); + }); + + // A partial export describes the types it was asked for, but the map it writes is the whole file. + // Rewriting it from one run's entries dropped every id the previous export put there — and a + // `personStatus: "status_unbekannt"` reference cannot fall back to the live name (the key is not + // name-derived), so the next `ct plan` hard-errors on a map the operator never meant to shrink. + it("carries over the types a --only export never looked at", async () => { + const rows = { + mainz: row("campus", "mainz", 4), + status_unbekannt: row("person-status", "status_unbekannt", 9), + }; + await exportInto(rows); + const result = await exportInto(rows, ["campus"]); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map.campus).toEqual({ mainz: { id: 4 } }); + expect(map["person-status"]).toEqual({ status_unbekannt: { id: 9 } }); + expect(map.$meta).toMatchObject({ entries: 2 }); + }); + + // The other half of the same rule: a type the export DID cover is rewritten wholesale, because a + // resource that left ct's state has left ct's ownership and its id must go with it. + it("drops an entry whose resource left state, for a type the export covered", async () => { + await exportInto({ + mainz: row("campus", "mainz", 4), + horgen: row("campus", "horgen", 5), + status_unbekannt: row("person-status", "status_unbekannt", 9), + }); + const result = await exportInto({ mainz: row("campus", "mainz", 4) }, ["campus"]); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map.campus).toEqual({ mainz: { id: 4 } }); + expect(map["person-status"]).toEqual({ status_unbekannt: { id: 9 } }); + }); + + // The end state the map exists to serve: tier-0 has left ct.config.ts AND ct-state..json, so + // a full `ct export tf` (a `make export`, a CI regen, a second env) selects every type, finds + // nothing, and would write `entries: 0` over the only thing still resolving every leftover + // `campus:` / `personStatus:` reference. An export that exported nothing has learned nothing. + it("never empties a populated map, however complete the export", async () => { + await exportInto({ + mainz: row("campus", "mainz", 4), + status_unbekannt: row("person-status", "status_unbekannt", 9), + }); + const result = await exportInto({}); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map.campus).toEqual({ mainz: { id: 4 } }); + expect(map["person-status"]).toEqual({ status_unbekannt: { id: 9 } }); + expect(result.warnings.map((w) => w.code)).toContain("IDS_KEPT"); + }); + + it("still writes an empty map when there was nothing to keep", async () => { + const result = await exportInto({}); + const map = JSON.parse(await readFile(result.value.idMapPath!, "utf8")); + expect(map.$meta).toMatchObject({ entries: 0 }); + expect(result.warnings.map((w) => w.code)).not.toContain("IDS_KEPT"); + }); + + it("writes nothing under .ct when --no-ids is passed", async () => { + await stateWith({ mainz: row("campus", "mainz", 0) }); + const result = await runExportTf({ + cwd: dir, + statePath: "ct-state.json", + outDir: "tofu", + writeIds: false, + }); + expect(result.value.idMapPath).toBeNull(); + await expect(readdir(join(dir, ".ct"))).rejects.toThrow(/ENOENT/); + }); +}); diff --git a/tests/ids-sync.test.ts b/tests/ids-sync.test.ts new file mode 100644 index 0000000..2111f18 --- /dev/null +++ b/tests/ids-sync.test.ts @@ -0,0 +1,185 @@ +/** + * `ct ids sync` — refreshing the id map from OpenTofu's own state (#181). + * + * `ct export tf` can only write what ct still holds. After the cutover, tofu is the only place new + * ids appear: a campus created by `tofu apply` exists in no ct state file, so no export will ever + * mention it. Reading tofu's state closes that loop — from a file or stdin, so it works against any + * backend (the real one is S3) without ct learning to speak S3. + */ +import { describe, it, expect, afterEach, beforeEach, vi } from "vitest"; +import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { runIdsList, runIdsSync } from "../src/application/operations/ids.js"; +import { writeIdMap } from "../src/resolve/idMap.js"; + +const HOST = "https://eqrm.church.tools"; +let dir: string; + +function tfstate(resources: unknown[], serial = 3): string { + return JSON.stringify({ version: 4, serial, resources }); +} + +const CAMPUS_MAINZ = { + mode: "managed", + type: "churchtools_campus", + name: "mainz", + instances: [{ attributes: { id: 0 } }], +}; + +beforeEach(async () => { + dir = await mkdtemp(join(tmpdir(), "ct-ids-sync-")); + await mkdir(join(dir, ".ct"), { recursive: true }); + vi.stubEnv("CT_HOST", HOST); +}); + +afterEach(async () => { + vi.unstubAllEnvs(); + await rm(dir, { recursive: true, force: true }); +}); + +async function sync(state: string, opts: { dryRun?: boolean } = {}) { + const path = join(dir, "terraform.tfstate"); + await writeFile(path, state, "utf8"); + return runIdsSync({ cwd: dir, tofuState: path, dryRun: opts.dryRun }); +} + +describe("runIdsSync", () => { + it("writes the map from the state and reports what moved", async () => { + await writeIdMap( + HOST, + [ + { type: "campus", key: "mainz", id: 0 }, + { type: "campus", key: "horgen", id: 4 }, + ], + join(dir, ".ct"), + ); + const result = await sync( + tfstate([ + CAMPUS_MAINZ, + { ...CAMPUS_MAINZ, name: "egc", instances: [{ attributes: { id: 3 } }] }, + { + mode: "managed", + type: "churchtools_person_status", + name: "status_5_core", + instances: [{ attributes: { id: 6 } }], + }, + ]), + ); + expect(result.value.written).toBe(true); + expect(result.value.serial).toBe(3); + expect(result.value.added.map((e) => e.key)).toEqual(["egc", "status_5_core"]); + // Gone from tofu's state, so ct must stop claiming to resolve it: a kept id would point every + // reference at whatever holds that number now. + expect(result.value.removed.map((e) => e.key)).toEqual(["horgen"]); + const map = JSON.parse(await readFile(result.value.path, "utf8")); + expect(map.campus).toEqual({ egc: { id: 3 }, mainz: { id: 0 } }); + expect(map.$meta).toMatchObject({ host: HOST, source: "ct ids sync", entries: 3 }); + }); + + it("reports an id that changed under the same key", async () => { + await writeIdMap(HOST, [{ type: "campus", key: "mainz", id: 6 }], join(dir, ".ct")); + const result = await sync(tfstate([CAMPUS_MAINZ])); + expect(result.value.changed).toEqual([{ entry: { type: "campus", key: "mainz", id: 0 }, previousId: 6 }]); + }); + + it("maps a relabelled resource back to its ct key via the existing map", async () => { + await writeIdMap( + HOST, + [{ type: "group-type", key: "3_active", id: 4, label: "g_3_active" }], + join(dir, ".ct"), + ); + const result = await sync( + tfstate([ + { + mode: "managed", + type: "churchtools_group_type", + name: "g_3_active", + instances: [{ attributes: { id: 4 } }], + }, + ]), + ); + expect(result.value.entries).toEqual([ + { type: "group-type", key: "3_active", id: 4, label: "g_3_active" }, + ]); + expect(result.value.added).toEqual([]); + }); + + it("reads the state from stdin, so `tofu state pull | ct ids sync` works on any backend", async () => { + const result = await runIdsSync( + { cwd: dir, tofuState: "-" }, + { readStdin: async () => tfstate([CAMPUS_MAINZ]) }, + ); + expect(result.value.entries).toEqual([{ type: "campus", key: "mainz", id: 0 }]); + expect(result.value.written).toBe(true); + }); + + it("never replaces a good map with an empty one", async () => { + // The likeliest cause is the wrong workspace, and overwriting 50 working ids with nothing would + // break every reference at once — the command reports and refuses instead. + await writeIdMap(HOST, [{ type: "campus", key: "mainz", id: 0 }], join(dir, ".ct")); + const result = await sync(tfstate([])); + expect(result.value.written).toBe(false); + expect(result.warnings.map((w) => w.code)).toContain("IDS_EMPTY"); + const map = JSON.parse(await readFile(result.value.path, "utf8")); + expect(map.campus).toEqual({ mainz: { id: 0 } }); + }); + + it("warns about provider resources ct has no type for", async () => { + const result = await sync( + tfstate([ + CAMPUS_MAINZ, + { mode: "managed", type: "churchtools_group", name: "kids", instances: [{ attributes: { id: 7 } }] }, + ]), + ); + const warning = result.warnings.find((w) => w.code === "IDS_TYPE_UNSUPPORTED"); + expect(warning?.message).toContain("churchtools_group"); + }); + + it("writes nothing under --dry-run", async () => { + const result = await sync(tfstate([CAMPUS_MAINZ]), { dryRun: true }); + expect(result.value.written).toBe(false); + expect(result.value.added.map((e) => e.key)).toEqual(["mainz"]); + await expect(readFile(join(dir, ".ct", "ids.eqrm.church.tools.json"), "utf8")).rejects.toThrow(/ENOENT/); + // The path it REPORTS is the one it would have written. Built from the raw host, it read + // `.ct/ids.https:/eqrm.church.tools.json` — a filename that exists nowhere, in the one mode + // whose entire output is "here is what I would do". + expect(result.value.path).toBe(join(dir, ".ct", "ids.eqrm.church.tools.json")); + }); + + it("says where to look rather than parsing garbage", async () => { + await expect(sync("not json")).rejects.toThrow(/not valid JSON/); + }); + + // `loadIdMap` throws on a malformed or foreign-host map, and this is the one command able to + // replace one. Refusing to run until the broken file is deleted by hand makes the repair tool + // need the repair. + it("regenerates over a malformed map instead of refusing to run", async () => { + await writeFile(join(dir, ".ct", "ids.eqrm.church.tools.json"), "not json at all", "utf8"); + const result = await sync(tfstate([CAMPUS_MAINZ])); + expect(result.value.written).toBe(true); + expect(result.value.entries.map((e) => e.key)).toEqual(["mainz"]); + expect(result.warnings.map((w) => w.code)).toContain("IDS_PREVIOUS_UNREADABLE"); + }); + + // Every other path in this operation is anchored to project.cwd; this one read from process.cwd(), + // so an embedded or HTTP caller passing `cwd` read a different file than it asked for. + it("resolves a relative --tofu-state against the project cwd", async () => { + await writeFile(join(dir, "terraform.tfstate"), tfstate([CAMPUS_MAINZ]), "utf8"); + const result = await runIdsSync({ cwd: dir, tofuState: "terraform.tfstate" }); + expect(result.value.entries.map((e) => e.key)).toEqual(["mainz"]); + }); +}); + +describe("runIdsList", () => { + it("reports no map rather than an empty one", async () => { + const result = await runIdsList({ cwd: dir }); + expect(result.value).toEqual({ path: null, entries: [] }); + }); + + it("lists what the resolver would use", async () => { + await writeIdMap(HOST, [{ type: "campus", key: "mainz", id: 0 }], join(dir, ".ct")); + const result = await runIdsList({ cwd: dir }); + expect(result.value.entries).toEqual([{ type: "campus", key: "mainz", id: 0 }]); + }); +}); diff --git a/tests/login-throttle.test.ts b/tests/login-throttle.test.ts new file mode 100644 index 0000000..63316ad --- /dev/null +++ b/tests/login-throttle.test.ts @@ -0,0 +1,120 @@ +/** + * The login-handshake brake (#179). + * + * `ct` is one-shot, so nothing inside an invocation knows how many handshakes the last minute + * already spent — and `ct auth token` is designed to be called by another tool on every run, with no + * session cache at all off macOS. ChurchTools rate-limits logins per instance, so the count has to + * outlive the process: it lives in a cache file holding nothing but timestamps. + */ +import { describe, it, expect, afterEach } from "vitest"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + fileLoginThrottle, + MAX_PER_HOUR, + MIN_INTERVAL_MS, + throttleDir, + throttlePath, +} from "../src/auth/loginThrottle.js"; + +const HOST = "https://mychurch.church.tools"; +let dir: string | undefined; + +function cacheDir(): NodeJS.ProcessEnv { + dir = mkdtempSync(join(tmpdir(), "ct-throttle-")); + const env = { XDG_CACHE_HOME: dir }; + mkdirSync(throttleDir(env), { recursive: true }); // so a test can seed the file the throttle reads + return env; +} + +afterEach(() => { + if (dir) rmSync(dir, { recursive: true, force: true }); + dir = undefined; +}); + +describe("fileLoginThrottle", () => { + it("keeps one file per host, holding no credential", async () => { + const env = cacheDir(); + await fileLoginThrottle({ env, now: () => 1000 }).acquire(HOST); + const path = throttlePath(HOST, env); + expect(path).toContain("login-throttle.mychurch.church.tools.json"); + expect(JSON.parse(readFileSync(path, "utf8"))).toEqual({ host: HOST, recent: [1000] }); + }); + + it("waits out the minimum spacing rather than letting the instance answer 429", async () => { + const env = cacheDir(); + const slept: number[] = []; + const sleep = async (ms: number): Promise => void slept.push(ms); + const throttle = fileLoginThrottle({ env, sleep, now: () => 1000 }); + await throttle.acquire(HOST); + expect(slept).toEqual([]); + // A second handshake 500ms later waits the remaining 2.5s — a short sleep, not a failure: the + // caller gets its session either way. + await fileLoginThrottle({ env, sleep, now: () => 1500 }).acquire(HOST); + expect(slept).toEqual([MIN_INTERVAL_MS - 500]); + }); + + it("refuses once a rolling hour is full, naming when the window frees up", async () => { + const env = cacheDir(); + const start = Date.parse("2026-09-21T10:00:00.000Z"); + writeFileSync( + throttlePath(HOST, env), + JSON.stringify({ host: HOST, recent: Array.from({ length: MAX_PER_HOUR }, (_, i) => start + i) }), + "utf8", + ); + await expect(fileLoginThrottle({ env, now: () => start + 60_000 }).acquire(HOST)).rejects.toThrow( + /2026-09-21T11:00:00.000Z/, + ); + }); + + it("forgets handshakes older than the window", async () => { + const env = cacheDir(); + const start = Date.parse("2026-09-21T10:00:00.000Z"); + writeFileSync( + throttlePath(HOST, env), + JSON.stringify({ host: HOST, recent: Array.from({ length: MAX_PER_HOUR }, (_, i) => start + i) }), + "utf8", + ); + const later = start + 2 * 60 * 60 * 1000; + await fileLoginThrottle({ env, now: () => later }).acquire(HOST); + expect(JSON.parse(readFileSync(throttlePath(HOST, env), "utf8")).recent).toEqual([later]); + }); + + it("drops a timestamp from the future instead of holding the brake on", async () => { + // A clock change or a copied cache file would otherwise throttle every login until the skew ran + // out — an unreadable cache must degrade to "no throttle", never to "no logins". + const env = cacheDir(); + const now = Date.parse("2026-09-21T10:00:00.000Z"); + writeFileSync( + throttlePath(HOST, env), + JSON.stringify({ host: HOST, recent: [now + 10 * 60 * 1000] }), + "utf8", + ); + const slept: number[] = []; + await fileLoginThrottle({ env, sleep: async (ms) => void slept.push(ms), now: () => now }).acquire(HOST); + expect(slept).toEqual([]); + }); + + it("does nothing under CT_NO_LOGIN_THROTTLE=1", async () => { + const env = { ...cacheDir(), CT_NO_LOGIN_THROTTLE: "1" }; + const start = Date.parse("2026-09-21T10:00:00.000Z"); + writeFileSync( + throttlePath(HOST, env), + JSON.stringify({ host: HOST, recent: Array.from({ length: MAX_PER_HOUR }, (_, i) => start + i) }), + "utf8", + ); + await expect(fileLoginThrottle({ env, now: () => start + 1 }).acquire(HOST)).resolves.toBeUndefined(); + }); + + it("never fails a login because the cache is unwritable", async () => { + // No such directory, and none can be created under a file — the throttle is a courtesy, and a + // courtesy that breaks the tool is a bug. + const file = join(mkdtempSync(join(tmpdir(), "ct-throttle-")), "not-a-dir"); + writeFileSync(file, "x", "utf8"); + dir = file; + await expect( + fileLoginThrottle({ env: { XDG_CACHE_HOME: file }, now: () => 1 }).acquire(HOST), + ).resolves.toBeUndefined(); + }); +}); diff --git a/tests/permission-catalog-host-difference.test.ts b/tests/permission-catalog-host-difference.test.ts new file mode 100644 index 0000000..0c3315b --- /dev/null +++ b/tests/permission-catalog-host-difference.test.ts @@ -0,0 +1,221 @@ +/** + * A right absent from THIS host's catalog is a host difference, not a config error (#178). + * + * One config serves an estate whose instances do not all have the same modules installed. Before + * this, a declared right the active catalog did not define aborted the whole plan: 13 of 113 rights + * did not exist on dev (Flow and Report are not installed there), so planning dev failed entirely — + * including for the 100 rights that do apply. `ct` already handles the mirror case exactly the right + * way (a LIVE grant whose authId the catalog cannot name is reported and left untouched, never + * revoked); this pins the same posture in the declaring direction. + * + * The oracle for "is this a real right?" is ct's BUNDLED catalog, which is why every test here loads + * a per-instance capture first: without one, the active catalog IS the bundled one and "missing here" + * and "missing everywhere" are the same statement. + */ +import { describe, it, expect, afterEach, vi } from "vitest"; +import { buildPermissionPlan, desiredTuples } from "../src/permissions/plan.js"; +import { + CATALOG, + catalogVerdict, + scopeFieldVerdict, + setStrictCatalog, + useBundledCatalog, + useCatalog, +} from "../src/permissions/catalog.js"; +import { evaluateConfig } from "../src/config/context.js"; +import type { State } from "../src/state/state.js"; + +/** A right the bundled catalog defines — the stand-in for `jpmFlowManager:*` on a Flow-less host. */ +const ABSENT_HERE = "churchgroup:administer groups"; +/** A dimension the bundled catalog defines, for the `preserveUnknown` half of the same story. */ +const ABSENT_DIMENSION_HERE = "cdb_comment_viewer"; + +const state: State = { version: 1, host: "h", resources: {} }; + +/** + * A per-instance capture standing in for eqrm-dev: it has ONE right, and deliberately not the one + * the config below declares. + */ +function useHostCatalogWithout(): void { + useCatalog( + { + $meta: { + capturedFrom: "dev.church.tools", + ctVersion: "3.137.0-RC13", + capturedAt: "2026-09-21", + rightCount: 1, + }, + "churchcore:administer settings": { authId: 1, scopeField: null, revocable: true, desc: "" }, + }, + { perInstance: true }, + ); +} + +afterEach(() => { + useBundledCatalog(); // the catalog is process-global — never leak a capture into another test +}); + +describe("catalogVerdict", () => { + it("separates a host difference from a typo", () => { + useHostCatalogWithout(); + expect(catalogVerdict("churchcore:administer settings")).toBe("known"); + expect(catalogVerdict(ABSENT_HERE)).toBe("host-missing"); + // Nothing ct has ever seen defines this, so it is a typo or a right ChurchTools deleted — + // the case the hard error was written for, and the one `churchreport:edit masterdata` is. + expect(catalogVerdict("jpmFlowManager:nosuchright")).toBe("unknown"); + }); + + it("keeps every absence fatal while the bundled catalog is the active one", () => { + // No capture loaded: "absent from the active catalog" and "absent from every catalog" are the + // same statement, so a repo that has not captured its host's catalog sees today's behaviour. + expect(CATALOG[ABSENT_HERE]).toBeDefined(); + expect(catalogVerdict("churchgroup:nosuchright")).toBe("unknown"); + }); + + it("makes --strict-catalog fatal again", () => { + useHostCatalogWithout(); + setStrictCatalog(true); + expect(catalogVerdict(ABSENT_HERE)).toBe("unknown"); + expect(scopeFieldVerdict(ABSENT_DIMENSION_HERE)).toBe("unknown"); + }); +}); + +describe("desiredTuples", () => { + it("emits no tuple for a right this host does not have, and reports it", () => { + useHostCatalogWithout(); + const skips: unknown[] = []; + const tuples = desiredTuples( + { + key: "mitglied", + domainType: "group_role", + domainId: 5, + grants: ["churchcore:administer settings", ABSENT_HERE], + }, + state, + new Set(), + new Map(), + (skip) => skips.push(skip), + ); + // No tuple is the whole point: no tuple means no PUT, and because the host cannot carry a live + // row under a name it does not define, nothing lands in toDelete either. + expect(tuples).toEqual([{ authId: 1, dataId: [], type: "grant" }]); + expect(skips).toEqual([{ domainType: "group_role", key: "mitglied", name: ABSENT_HERE, kind: "right" }]); + }); + + it("still throws for a name no catalog defines", () => { + useHostCatalogWithout(); + expect(() => + desiredTuples( + { key: "mitglied", domainType: "group_role", domainId: 5, grants: ["churchreport:no such right"] }, + state, + ), + ).toThrow(/Unknown permission/); + }); + + it("throws for a host-missing right under --strict-catalog", () => { + useHostCatalogWithout(); + setStrictCatalog(true); + expect(() => + desiredTuples({ key: "mitglied", domainType: "group_role", domainId: 5, grants: [ABSENT_HERE] }, state), + ).toThrow(/Unknown permission/); + }); +}); + +describe("buildPermissionPlan", () => { + it("plans everything else and warns once per skipped declaration", async () => { + useHostCatalogWithout(); + const client = { get: vi.fn(async () => []) }; + const { items, warnings } = await buildPermissionPlan(client as never, state, [ + { + key: "mitglied", + domainType: "group_role", + domainId: 5, + grants: ["churchcore:administer settings", ABSENT_HERE], + }, + ]); + // The declaration is still planned — the skip costs it one grant, not the whole role. + expect(items[0]?.diff.toPut).toEqual([{ authId: 1, dataId: [], type: "grant" }]); + const skipWarning = warnings.find((w) => w.includes(ABSENT_HERE)); + expect(skipWarning).toMatch(/group_role "mitglied"/); + expect(skipWarning).toMatch(/absent from this host's permission catalog/); + // Both catalogs are named: without the bundled half the warning reads like a typo. + expect(skipWarning).toMatch(/3\.137\.0-RC13/); + expect(skipWarning).toMatch(/bundled catalog/); + expect(skipWarning).toMatch(/never granted, never revoked/); + expect(skipWarning).toMatch(/--strict-catalog/); + }); + + it("does not revoke the live grants of a domain whose declaration was partly skipped", async () => { + useHostCatalogWithout(); + // The host grants authId 1 directly; the declaration names it plus a right the host lacks. + const client = { + get: vi.fn(async () => [ + { + domainType: "group_role", + domainId: 5, + authId: 1, + dataId: null, + type: "grant", + meta: { modifiedPid: 9 }, + }, + ]), + }; + const { items } = await buildPermissionPlan(client as never, state, [ + { + key: "mitglied", + domainType: "group_role", + domainId: 5, + grants: ["churchcore:administer settings", ABSENT_HERE], + }, + ]); + expect(items[0]?.diff.toPut).toEqual([]); + expect(items[0]?.diff.toDelete).toEqual([]); + }); + + it("reports a preserveUnknown dimension this host has no right for", async () => { + useHostCatalogWithout(); + const client = { get: vi.fn(async () => []) }; + const { warnings } = await buildPermissionPlan(client as never, state, [ + { + key: "mitglied", + domainType: "group_role", + domainId: 5, + grants: ["churchcore:administer settings"], + preserveUnknown: [ABSENT_DIMENSION_HERE], + }, + ]); + const warning = warnings.find((w) => w.includes(ABSENT_DIMENSION_HERE)); + expect(warning).toMatch(/preserveUnknown/); + expect(warning).toMatch(/can preserve nothing here/); + }); +}); + +describe("config evaluation", () => { + it("accepts a preserveUnknown dimension that exists on the other host", async () => { + useHostCatalogWithout(); + const { permissions } = await evaluateConfig((ct) => { + ct.groupRole({ + key: "mitglied", + id: 5, + grants: ["churchcore:administer settings"], + preserveUnknown: [ABSENT_DIMENSION_HERE], + }); + }); + // Kept verbatim in the declaration: the verdict belongs to the plan, which can name the host. + expect(permissions[0]?.preserveUnknown).toEqual([ABSENT_DIMENSION_HERE]); + }); + + it("still rejects a dimension no catalog defines", async () => { + useHostCatalogWithout(); + await expect( + evaluateConfig((ct) => { + ct.groupRole({ + key: "mitglied", + id: 5, + grants: ["churchcore:administer settings"], + preserveUnknown: ["cdb_nosuchdimension"], + }); + }), + ).rejects.toThrow(/no right in the permission catalog scopes by/); + }); +}); diff --git a/tests/state-rm-command.test.ts b/tests/state-rm-command.test.ts index 72e6f79..ce644fe 100644 --- a/tests/state-rm-command.test.ts +++ b/tests/state-rm-command.test.ts @@ -15,6 +15,7 @@ import { mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { emptyState, loadState, saveState } from "../src/state/state.js"; +import { removeStateEntry } from "../src/application/operations/state.js"; const authedSession = vi.fn(async () => { throw new Error("ct state rm must never contact ChurchTools"); @@ -109,25 +110,81 @@ describe("ct state rm (#122)", () => { // A key can be named by a PERMISSION declaration without being declared as a resource. A // resources-only guard waves those through, and the breakage surfaces one command later as a // `ct plan` hard error ("does not resolve to a managed group") — after the state file was written. + // + // Still refused after #180, which stopped treating a reference as a declaration: a GROUP reference + // is the one class that genuinely cannot survive the removal, because groups are managed-only and + // have no live catalog to fall back to. The refusal now says that instead of claiming the key is + // declared — it is not — so the two cases can be told apart. it("refuses a key a permission DOMAIN still names, not just a declared resource", async () => { await writeFile( configPath, `export default (ct) => { ct.groupRole({ key: "youth_leiter", group: "youth", role: "Leiter", grants: ["churchcore:administer settings"] }); };`, ); await expect(run(["group", "youth", "--state", statePath])).rejects.toThrow( - /still declared in the config/, + /still referenced by the config as a resource that only ct's state can resolve/, ); const state = await loadState(statePath, HOST); expect(state.resources.youth).toBeDefined(); }); + // #180: the tier-0 OpenTofu cutover leaves 232 `campus:`/`groupType:`/`department:` REFERENCES + // behind in the config with the declarations gone. Those are logical refs — they ask the host for + // the id — so `--force` was being demanded for 49 of 50 entries, which suppresses the check for the + // one case it is actually meant to catch. + it("removes a campus the config only references, and says the refs now resolve live", async () => { + await writeFile( + configPath, + `export default (ct) => { ct.group({ key: "kids", name: "Kids", campus: "mainz", groupTypeId: 1 }); };`, + ); + const result = await removeStateEntry({ type: "campus", key: "mainz", statePath, configPath }); + expect(result.value.removed).toBe(true); + const warning = result.warnings.find((w) => w.code === "STILL_REFERENCED"); + expect(warning?.message).toMatch(/references "mainz" 1 time\(s\) but does not declare it/); + expect(warning?.message).toMatch(/resolve against https:\/\/mychurch\.church\.tools by name/); + const state = await loadState(statePath, HOST); + expect(state.resources.mainz).toBeUndefined(); + }); + + // The guard's original purpose, unchanged: a real declaration means the next plan would propose + // CREATING something that already exists on this host. + it("still refuses a campus the config DECLARES", async () => { + await writeFile(configPath, `export default (ct) => { ct.campus({ key: "mainz", name: "Mainz" }); };`); + await expect(run(["campus", "mainz", "--state", statePath])).rejects.toThrow( + /still declared in the config/, + ); + const state = await loadState(statePath, HOST); + expect(state.resources.mainz).toBeDefined(); + }); + + // Declarations are matched on type AND key: a `campus` declaration says nothing about what + // removing a group-type of the same name would do. + it("does not let a declaration of another type block a removal", async () => { + await writeFile(configPath, `export default (ct) => { ct.campus({ key: "mainz", name: "Mainz" }); };`); + const state = await loadState(statePath, HOST); + state.resources["mainz-type"] = { + type: "group-type", + id: 12, + key: "mainz", + fields: {}, + adoptedAt: "t", + updatedAt: "t", + }; + // Re-key the entry under the colliding key so the removal targets group-type/mainz. + delete state.resources.mainz; + state.resources.mainz = { ...state.resources["mainz-type"]! }; + delete state.resources["mainz-type"]; + await saveState(statePath, state); + await run(["group-type", "mainz", "--state", statePath]); + expect((await loadState(statePath, HOST)).resources.mainz).toBeUndefined(); + }); + it("refuses a key a permission SCOPE still names", async () => { await writeFile( configPath, `export default (ct) => { ct.groupRole({ key: "p", id: 77, grants: [{ right: "churchgroup:view group", scope: ["youth"] }] }); };`, ); await expect(run(["group", "youth", "--state", statePath])).rejects.toThrow( - /still declared in the config/, + /still referenced by the config as a resource that only ct's state can resolve/, ); const state = await loadState(statePath, HOST); expect(state.resources.youth).toBeDefined(); diff --git a/tests/tofu-id-map.test.ts b/tests/tofu-id-map.test.ts new file mode 100644 index 0000000..f4119bd --- /dev/null +++ b/tests/tofu-id-map.test.ts @@ -0,0 +1,256 @@ +/** + * Resolving OpenTofu-owned resources through a committed id map (#181). + * + * This is the last step of the tier-0 cutover, and the one that could not be done by hand. When the + * five tier-0 tables leave `ct.config.ts` AND their 50 entries leave `ct-state..json`, the + * references that stay behind fall back to matching the key against the live object's NAME — and ct's + * tier-0 keys were never name-derived, so for person statuses that can never work: + * + * key live name + * status_unbekannt Unbekannt + * status_5_core 5 - Core + * + * A literal id is no escape either: one config serves two hosts and 39 of 43 tier-0 ids differ. + * So the map carries the exact key→id table, per host, committed. + */ +import { describe, it, expect, afterEach } from "vitest"; +import { mkdirSync, mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { idMapPath, loadIdMap, parseIdMap, renderIdMap, writeIdMap } from "../src/resolve/idMap.js"; +import { readTfState } from "../src/resolve/tfstate.js"; +import { Resolver } from "../src/resolve/resolver.js"; +import { ref } from "../src/resolve/refs.js"; +import type { State } from "../src/state/state.js"; + +const HOST = "https://eqrm.church.tools"; +let dir: string | undefined; + +const state: State = { version: 1, host: HOST, resources: {} }; + +afterEach(() => { + if (dir) rmSync(dir, { recursive: true, force: true }); + dir = undefined; +}); + +function tempDir(): string { + dir = mkdtempSync(join(tmpdir(), "ct-ids-")); + mkdirSync(join(dir, ".ct"), { recursive: true }); + return join(dir, ".ct"); +} + +/** The tier-0 shape the exporter produces, with one relabelled key. */ +const ENTRIES = [ + { type: "campus", key: "mainz", id: 0 }, + { type: "person-status", key: "status_unbekannt", id: 0 }, + { type: "person-status", key: "status_5_core", id: 6 }, + { type: "group-type", key: "3_groupactive", id: 9, label: "g_3_groupactive" }, +]; + +describe("the id map file", () => { + it("is one file per host, beside the per-instance permission catalog", () => { + expect(idMapPath(HOST, ".ct")).toBe(".ct/ids.eqrm.church.tools.json"); + expect(idMapPath("https://eqrm-dev.church.tools/", ".ct")).toBe(".ct/ids.eqrm-dev.church.tools.json"); + }); + + it("renders byte-stably, sorted by type then key", () => { + const first = renderIdMap(HOST, ENTRIES, "ct export tf"); + const shuffled = renderIdMap(HOST, [...ENTRIES].reverse(), "ct export tf"); + expect(first).toBe(shuffled); + const parsed = JSON.parse(first) as Record; + expect(Object.keys(parsed)).toEqual(["$meta", "campus", "group-type", "person-status"]); + // The label rides along only where it differs — a relabelling is many-to-one, so it cannot be + // re-derived from the label when reading a tofu state back. + expect(parsed["group-type"]).toEqual({ "3_groupactive": { id: 9, label: "g_3_groupactive" } }); + expect(parsed.campus).toEqual({ mainz: { id: 0 } }); + }); + + it("id 0 survives the round trip", async () => { + // `campus/mainz` is id 0 on prod and `person-status/status_unbekannt` is 0 too: any code that + // treats a falsy id as "missing" breaks precisely the references this feature exists for. + const d = tempDir(); + await writeIdMap(HOST, ENTRIES, d); + const loaded = await loadIdMap(HOST, d); + expect(loaded?.ids.get("campus\u0000mainz")).toBe(0); + expect(loaded?.ids.get("person-status\u0000status_unbekannt")).toBe(0); + }); + + it("returns null when the repo has no map, so nothing changes for anyone else", async () => { + expect(await loadIdMap(HOST, tempDir())).toBeNull(); + }); + + it("refuses a map generated for another host", () => { + // 39 of 43 tier-0 ids differ between the two hosts, so applying dev's map to prod would resolve + // every reference to a real, wrong resource — undetectable downstream. + const raw = renderIdMap("https://eqrm-dev.church.tools", ENTRIES, "ct export tf"); + expect(() => parseIdMap(raw, ".ct/ids.json", HOST)).toThrow(/was generated for https:\/\/eqrm-dev/); + }); + + it("refuses a malformed entry rather than resolving to NaN", () => { + const raw = JSON.stringify({ campus: { mainz: { id: "0" } } }); + expect(() => parseIdMap(raw, ".ct/ids.json", HOST)).toThrow(/campus "mainz" has no numeric id/); + }); +}); + +describe("Resolver", () => { + const client = { + get: async () => [], + getAll: async () => ({ data: [{ id: 99, name: "Unbekannt" }] }), + }; + + it("resolves a key the live name cannot produce", async () => { + const map = parseIdMap(renderIdMap(HOST, ENTRIES, "ct export tf"), ".ct/ids.json", HOST); + const resolver = new Resolver({ client: client as never, state, desired: [], host: HOST, idMap: map }); + expect(await resolver.resolve(ref.personStatus("status_5_core"), "site")).toBe(6); + expect(await resolver.resolve(ref.campus("mainz"), "site")).toBe(0); + }); + + it("still prefers ct's own state over the map", async () => { + // ct never stops trusting what it owns: a resource still in ct's state is ct's, and a stale map + // entry must not overrule it. + const owned: State = { + version: 1, + host: HOST, + resources: { + mainz: { type: "campus", id: 42, key: "mainz", fields: {}, adoptedAt: "t", updatedAt: "t" }, + }, + }; + const map = parseIdMap(renderIdMap(HOST, ENTRIES, "ct export tf"), ".ct/ids.json", HOST); + const resolver = new Resolver({ + client: client as never, + state: owned, + desired: [], + host: HOST, + idMap: map, + }); + expect(await resolver.resolve(ref.campus("mainz"), "site")).toBe(42); + }); + + it("prefers the map over a live name match", async () => { + // The map is exact; the catalog match is a guess at a slug. `status_unbekannt` would resolve to + // the live "Unbekannt" row (#99) only by accident of naming — and does not on the real instance. + const map = parseIdMap( + renderIdMap(HOST, [{ type: "person-status", key: "unbekannt", id: 7 }], "ct export tf"), + ".ct/ids.json", + HOST, + ); + const resolver = new Resolver({ client: client as never, state, desired: [], host: HOST, idMap: map }); + expect(await resolver.resolve(ref.personStatus("unbekannt"), "site")).toBe(7); + }); + + it("names the map, and how to refresh it, when a key is in neither", async () => { + const map = parseIdMap(renderIdMap(HOST, ENTRIES, "ct export tf"), ".ct/ids.json", HOST); + const resolver = new Resolver({ client: client as never, state, desired: [], host: HOST, idMap: map }); + await expect(resolver.resolve(ref.campus("horgen"), "site")).rejects.toThrow( + /\.ct\/ids\.json has no campus "horgen" either.*ct ids sync/s, + ); + }); +}); + +describe("readTfState", () => { + const tfstate = { + version: 4, + serial: 12, + resources: [ + { + mode: "managed", + type: "churchtools_person_status", + name: "status_unbekannt", + instances: [{ attributes: { id: 0, name: "Unbekannt" } }], + }, + { + mode: "managed", + type: "churchtools_group_type", + name: "g_3_groupactive", + instances: [{ attributes: { id: 9 } }], + }, + // A data source describes a read, not a managed object. + { + mode: "data", + type: "churchtools_campus", + name: "elsewhere", + instances: [{ attributes: { id: 5 } }], + }, + // Mid-create or tainted: no id to map. + { mode: "managed", type: "churchtools_campus", name: "pending", instances: [{ attributes: {} }] }, + // A provider resource ct has no type for — reported, never fatal. + { mode: "managed", type: "churchtools_group", name: "kids", instances: [{ attributes: { id: 3 } }] }, + { mode: "managed", type: "aws_s3_bucket", name: "state", instances: [{ attributes: { id: "b" } }] }, + ], + }; + + it("maps managed churchtools resources back to ct keys", () => { + // tofu's label is mapped back through the previous map, because the exporter's relabelling + // (`3_groupactive` → `g_3_groupactive`) is many-to-one and cannot be inverted by rule. + const previous = parseIdMap(renderIdMap(HOST, ENTRIES, "ct export tf"), ".ct/ids.json", HOST); + const { entries, unmapped, serial } = readTfState(tfstate, previous.keysByLabel); + expect(entries).toEqual([ + { type: "person-status", key: "status_unbekannt", id: 0 }, + { type: "group-type", key: "3_groupactive", id: 9, label: "g_3_groupactive" }, + ]); + expect(unmapped).toEqual(["churchtools_group"]); + expect(serial).toBe(12); + }); + + it("takes the label as the key when no previous map recorded a relabelling", () => { + const { entries } = readTfState(tfstate); + expect(entries.map((e) => e.key)).toEqual(["status_unbekannt", "g_3_groupactive"]); + }); + + it("rejects something that is not a tofu state", () => { + expect(() => readTfState([])).toThrow(/expected a JSON object/); + expect(() => readTfState({ resources: {} })).toThrow(/"resources" is not an array/); + }); +}); + +/** + * Reading a tofu state defensively (#181 review): the two ways `readTfState` could put a + * CONFIDENTLY WRONG row into a committed map, which is worse than putting none there. + */ +describe("readTfState — ids it must refuse to guess", () => { + const managed = (name: string, id: unknown) => ({ + mode: "managed", + type: "churchtools_campus", + name, + instances: [{ attributes: { id } }], + }); + + // `Number(null)`, `Number("")`, `Number([])` and `Number(false)` are all 0, and `Number.isFinite(0)` + // is true — so the finite check alone let a mid-create or tainted resource through as id 0. Zero is + // a REAL ChurchTools id, so the reference would have resolved silently to the wrong live object. + it("does not turn a null or empty id into the real id 0", () => { + const { entries } = readTfState({ + resources: [managed("mainz", null), managed("koeln", ""), managed("bonn", [])], + }); + expect(entries).toEqual([]); + }); + + it("still accepts a genuine 0 and a numeric string", () => { + const { entries } = readTfState({ resources: [managed("mainz", 0), managed("koeln", "7")] }); + expect(entries).toEqual([ + { type: "campus", key: "mainz", id: 0 }, + { type: "campus", key: "koeln", id: 7 }, + ]); + }); + + // A for_each/count block is ONE resource with many instances, and its name is the block's, not any + // resource's key. Taking instances[0] mapped that label to one arbitrary id and reported every real + // key as removed — and being non-empty, it would sail past the empty-map guard in `ct ids sync`. + it("reports a for_each block instead of mapping its label to one arbitrary id", () => { + const { entries, multiInstance } = readTfState({ + resources: [ + { + mode: "managed", + type: "churchtools_campus", + name: "campuses", + instances: [ + { index_key: "mainz", attributes: { id: 4 } }, + { index_key: "koeln", attributes: { id: 5 } }, + ], + }, + ], + }); + expect(entries).toEqual([]); + expect(multiInstance).toEqual(["churchtools_campus.campuses"]); + }); +});