diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 44132e4..c1bc85f 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -30,3 +30,7 @@ jobs: - name: Functional cache-staleness tests (envtest) run: make test-envtest + + lint-docs: + name: lint-docs + uses: krateo-platformops/.github/.github/workflows/lint-docs.yaml@main diff --git a/README.md b/README.md index cf85302..e384af1 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,54 @@ -# `plumbing` +# plumbing -[![Go Report Card](https://goreportcard.com/badge/github.com/krateoplatformops/plumbing)](https://goreportcard.com/report/github.com/krateoplatformops/plumbing)     [![Reference](https://pkg.go.dev/badge/github.com/krateoplatformops/plumbing)](https://pkg.go.dev/github.com/krateoplatformops/plumbing) +The shared Go library of the Krateo platform — reusable building blocks and the cross-repo contracts (helm reconcile engine, crdgen, composition labels) consumed by core-provider, composition-dynamic-controller and the platform services. +[![test](https://github.com/krateo-platformops/plumbing/actions/workflows/test.yaml/badge.svg)](https://github.com/krateo-platformops/plumbing/actions/workflows/test.yaml) +[![Reference](https://pkg.go.dev/badge/github.com/krateo-platformops/plumbing)](https://pkg.go.dev/github.com/krateo-platformops/plumbing) +## What is this -Provides reusable building blocks and common helpers to reduce boilerplate across [Krateo Platformops](https://krateo.io/) projects. +A flat collection of independent Go packages with one module identity. It carries +the Helm client with the fork-free apply-if-changed `Reconcile` engine that +composition-dynamic-controller runs on, the `crdgen` JSON-Schema → CRD transpiler +core-provider uses, the `labels` package that pins the core-provider ↔ CDC label +contract at compile time, plus `kubeutil/*`, `jwtutil` and small utilities. +Library only: no binary, no image, no chart. Full picture: +[docs/index.md](docs/index.md). -It includes general-purpose utilities aimed at simplifying day-to-day development while promoting consistency and maintainability. \ No newline at end of file +## Install + +```sh +go get github.com/krateo-platformops/plumbing@v1.13.2 +``` + +## Configure + +There is almost nothing to configure — see +[docs/configuration.md](docs/configuration.md). Most used: + +| Setting | Default | Effect | +|---|---|---| +| `HELM_DRIVER` | `secret` | Helm release-storage driver used by `helm/v3`. | +| `CLUSTER_NAME` | unset | Overrides `kubeutil.DetectClusterName`. | +| build tag `envtest` | off | Selects the real-apiserver functional tests (`make test-envtest`). | + +## Examples + +- [examples/minimal](examples/minimal) — `values.schema.json` → CRD via + `crdgen.Generate`; `go run ./examples/minimal`. + +## Docs + +- [docs/index.md](docs/index.md) — the map +- [docs/overview.md](docs/overview.md) — package design + the Reconcile engine +- [docs/usage.md](docs/usage.md) — go get + version-pinning conventions +- [docs/configuration.md](docs/configuration.md) — the whole config surface +- [docs/api.md](docs/api.md) — the exported package map +- [docs/examples.md](docs/examples.md) — examples index +- [docs/release.md](docs/release.md) — how a release ships (tag-only) +- [docs/log.md](docs/log.md) — curated history + +## Develop & release + +`go build ./... && make test-race` (CI adds `make test-envtest`). Releases are +plain `vX.Y.Z` git tags — see [docs/release.md](docs/release.md). diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..aecc53e --- /dev/null +++ b/docs/api.md @@ -0,0 +1,98 @@ +--- +type: API +title: plumbing exported API +description: The exported package map — one line per package, grouped by theme, derived from the package docs and code; full godoc lives on pkg.go.dev. +resource: https://pkg.go.dev/github.com/krateo-platformops/plumbing +tags: [go-api, packages] +timestamp: 2026-08-07T00:00:00Z +--- + +# Exported API + +The contract is the exported surface of every Go package below. This page is the +**map**, not the reference — full signatures and doc comments live at +[pkg.go.dev/github.com/krateo-platformops/plumbing](https://pkg.go.dev/github.com/krateo-platformops/plumbing) +(or `go doc github.com/krateo-platformops/plumbing/`). One line per package, +derived from the actual package docs and code. + +## Helm engine + +| Package | What it exports | +|---|---| +| `helm` | The provider-agnostic `Client` interface (Install/Upgrade/**Reconcile**/Uninstall/Rollback/GetRelease/ListReleases) plus config, `Release`/`ReconcileResult` and `PostRenderer` types. | +| `helm/v3` | The Helm-v3-SDK implementation: cached action configs, CRD-informer discovery invalidation (`NewCRDInformer`), duplicate-resource post-render validation, and the fork-free apply-if-changed `Reconcile` (see [overview](./overview.md)). | +| `helm/v3/tracer` | An `http.RoundTripper` that logs each Kubernetes request a Helm operation makes and collects the touched `Resource`s. | +| `helm/getter` | Chart fetching (`Get`, `NewOCIGetter`) with credentials, TLS, timeout, repo and disk-cache options. | +| `helm/getter/cache` | `DiskCache` — content-addressed on-disk cache for fetched charts. | +| `helm/getter/repo` | Classic HTTP chart-repository index types (`IndexFile`, `ChartVersions`) and URL helpers. | +| `helm/utils` | Post-render helpers: composition `LabelsPostRender`, the `krateo.io/traceparent` stamp (scoped to `*.krateo.io`), the `krateo.io/gracefully-paused` annotation key. | + +## Kubernetes utilities + +| Package | What it exports | +|---|---| +| `kubeutil` | Misc helpers: DNS-1123 name mangling, `DetectClusterName`, `ServiceAccountNamespace`, `ConfigMapData`, in-cluster CA cert. | +| `kubeutil/objectclient` | Retrying, `dynamic.Interface`-based create-or-update (`Apply` — writes the server response back into the object), `Get`, delete-if-present. | +| `kubeutil/hasher` | Cumulative, order-dependent hash over JSON-marshalable values — desired-vs-deployed drift detection. | +| `kubeutil/rbacgen` | Builds least-privilege Role/RoleBinding sets granting one ServiceAccount access to an exact named resource set. | +| `kubeutil/rbac` | Batched `UserCan` permission checks (SelfSubjectAccessReview) with a TTL cache. | +| `kubeutil/dynamicwatch` | Registers a controller-runtime watch on a GVK not known until runtime, deduping repeat registrations. | +| `kubeutil/secretref` | Reads one key out of a Secret via `dynamic.Interface` (for schema-unknown-at-compile-time reconcilers). | +| `kubeutil/plurals` | GVK → plural/singular resource-name resolution (`Get`, `ResolveAPINames`). | +| `kubeutil/event` | `APIRecorder` + `Normal`/`Warning` event constructors for controller runtimes. | +| `kubeutil/eventrecorder` | `record.EventRecorder` factories, including a state-aware throttled recorder. | +| `kubeutil/discoveryevents` | Publishes API-discovery resource added/changed/removed events onto an `eventbus.Bus`. | +| `labels` | THE shared composition label keys coupling core-provider and composition-dynamic-controller (compile-time cross-repo contract). | + +## CRD generation + +| Package | What it exports | +|---|---| +| `crdgen` | `Generate(Options)` — direct JSON-Schema → structural-OpenAPI-v3 CRD transpiler, gated on the apiextensions validation library; `NormalizeVersionName`. | +| `crdgen/schemas` | JSON-Schema resolution primitives and type-name constants used by the transpiler. | + +## Identity, endpoints, auth + +| Package | What it exports | +|---|---| +| `jwtutil` | HS256 Krateo JWT create/verify with `UserInfo` claims; the shared `AUTHN_JWT_SECRET` env-key constant. | +| `endpoints` | The Krateo `Endpoint` (user API-server credential record) stored/loaded as a Secret. | +| `kubeconfig` | `Endpoint` → kubeconfig YAML (`Marshal`) and → `*rest.Config` (`NewClientConfig`). | +| `signup` | Creates a certificate-based cluster user (CSR flow) and stores its `Endpoint`. | +| `certs` | CertificateSigningRequest helpers: create, approve, wait, fetch the issued cert. | + +## HTTP client & server + +| Package | What it exports | +|---|---| +| `http/request` | Outbound request building incl. AWS Signature v4 header computation. | +| `http/response` | The `Status` response envelope (`Success`/`Failure`) + typed error writers (`BadRequest`, `Forbidden`, …). | +| `http/util` | `RetryClient` — retrying, rate-limited `http.Client` wrapper. | +| `server/probes` | `/livez` + `/readyz` handlers and a `HealthServer`. | +| `server/use` | Middleware: access logging, CORS, trace-id propagation. | +| `server/use/cors` | The CORS handler implementation used by `server/use`. | + +## General utilities + +| Package | What it exports | +|---|---| +| `bufferpool` | Sized `sync.Pool`-backed byte-buffer pool. | +| `cache` | Generic `TTLCache[K,V]` with cleanup interval and max-entries options. | +| `codegen` | Fluent Go source-code generation builders (`Package`, `Function`, `If`, …). | +| `context` | Krateo request context: trace-id (`X-Krateo-TraceId`), per-request logger, access token, user config/info. | +| `deps` | Small dependency `Graph` with topological resolution. | +| `env` | Typed env readers (`String`, `Bool`, `Int`, `Duration`, `ServicePort`, …) with defaults. | +| `eventbus` | In-process publish/subscribe `Bus`. | +| `jqutil` | gojq evaluation helpers: `Eval`, `Extract`, `ForEach`, module loading, type inference. | +| `logger` | JSON `slog.Logger` factory with a service-name attribute. | +| `maps` | Nested-map access/copy (`NestedString`, `DeepCopyJSON`, `LeafPaths`, …). | +| `pgutil` | PostgreSQL connection-URL building and wait-for-ready pool creation (pgx). | +| `ptr` | Generic pointer helpers (`To`, `Deref`, `Equal`). | +| `shortid` | Short, unique, non-sequential URL-friendly id generation. | +| `wait` | Generic backoff retry `Until` / `UntilWithOptions`. | +| `e2e` | e2e-framework helpers for consumer test suites: namespaces, coverage, sign-up step. | + +Compatibility note: within the v1 line, exported APIs may still move between minors +(e.g. the slogs→slog removal, the crdgen transpiler replacement) — this is an +internal platform library, and consumers pin exact tags precisely for this reason +(see [usage](./usage.md)). diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..1239a1c --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,38 @@ +--- +type: Configuration +title: plumbing configuration +description: The whole config surface the library itself reads — two runtime env vars, one conventional env-key constant, and the test-only build tags. +resource: github.com/krateo-platformops/plumbing +tags: [env, build-tags] +timestamp: 2026-08-07T00:00:00Z +--- + +# Configuration + +plumbing is a library: it has no config file, no flags, no values. Configuration is +whatever each package's `Options`/functional-option arguments say (see +[api](./api.md)). The library itself reads exactly this from the environment: + +| Env var | Read by | Effect | +|---|---|---| +| `HELM_DRIVER` | `helm/v3` ([client.go](../helm/v3/client.go)) | Helm storage driver for release records (`secret` when unset — the Helm SDK default). | +| `CLUSTER_NAME` | `kubeutil` ([detect_cluster.go](../kubeutil/detect_cluster.go)) | First choice for `DetectClusterName`; falls back to the rest-config host, then the local hostname. | +| `KUBERNETES_SERVICE_HOST` / `KUBERNETES_SERVICE_PORT` | `signup` ([signup.go](../signup/signup.go)) | In-cluster apiserver URL when `Options.ServerURL` is empty (standard in-cluster env). | + +One near-miss worth stating: `jwtutil.JwtSecretEnvKey` (`AUTHN_JWT_SECRET`) is a +**shared env-key NAME constant**, not an env read — `CreateToken` takes the signing +key via `CreateTokenOptions.SigningKey` and errors if it is empty; consumers +(authn, snowplow) read the env var themselves under this agreed name. + +The `env` package (`env.Bool`, `env.String`, `env.Duration`, …) is a typed +env-reading helper **for consumers**; it does not make the library read anything on +its own. + +## Build tags + +No non-test code carries a build tag. Test-only: + +| Tag | Selects | Needs | +|---|---|---| +| `envtest` | the functional cache-staleness tests in `helm/v3` | a kubebuilder envtest control plane (`make test-envtest` installs it, pinned in the [Makefile](../Makefile)) | +| `integration` | tests in `endpoints` and `helm/v3` that hit a real cluster | a reachable kubeconfig cluster; not run in CI | diff --git a/docs/examples.md b/docs/examples.md new file mode 100644 index 0000000..34664cb --- /dev/null +++ b/docs/examples.md @@ -0,0 +1,17 @@ +--- +type: ExampleIndex +title: plumbing examples +description: Index of the runnable examples under examples/ — one line each. +resource: github.com/krateo-platformops/plumbing +tags: [examples] +timestamp: 2026-08-07T00:00:00Z +--- + +# Examples + +- [examples/minimal](../examples/minimal/README.md) — feed a chart-style + `values.schema.json` to `crdgen.Generate` and print the resulting CRD YAML + (no cluster needed): `go run ./examples/minimal`. + +The example is part of the module, so `go build ./...` at the repo root compiles it +and CI keeps it honest. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..762b7c1 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,42 @@ +--- +type: Library +title: plumbing — index +description: The map of the plumbing doc bundle — the shared Go library (helm client + reconcile engine, kubeutil, crdgen, jwtutil, labels and friends) consumed by the Krateo platform components. +resource: github.com/krateo-platformops/plumbing +tags: [library, helm, crdgen, kubeutil] +timestamp: 2026-08-07T00:00:00Z +--- + +# plumbing + +plumbing is **the shared Go library of the Krateo platform**: reusable building +blocks that reduce boilerplate and — more importantly — pin cross-repo contracts in +one place. It carries the Helm client with the fork-free **apply-if-changed +`Reconcile` engine** that composition-dynamic-controller runs on, the +**`crdgen`** JSON-Schema → CRD transpiler that core-provider uses to turn a chart's +`values.schema.json` into a versioned CRD, the **`labels`** package that makes the +core-provider ↔ CDC composition-label contract a compile-time guarantee, plus +`kubeutil/*` controller helpers, `jwtutil`, and small utilities. It is a library +only: no binary, no image, no chart — releases are plain git tags. + +## The bundle (start here) + +- [overview](./overview.md) — package design: the helm engine (and the + `Reconcile` flow step by step), crdgen, the kubeutil family, and how + core-provider/CDC consume them. +- [usage](./usage.md) — `go get`, the version-pinning conventions consumers follow, + and the bump chain into core-provider and the installer. +- [configuration](./configuration.md) — the (small) config surface the library + itself reads: two env vars and two test build tags. +- [api](./api.md) — the exported package map, one line per package, with + pkg.go.dev links. +- [examples](./examples.md) — the runnable example under `examples/`. +- [release](./release.md) — how a release ships (tag-only, no OCI artifact). +- [log](./log.md) — curated history. +- [llms.txt](./llms.txt) — the version-pinned agent index of this bundle. + +## Deeper, code-adjacent docs + +- [crdgen/docs/ref-resolution-redesign.md](../crdgen/docs/ref-resolution-redesign.md) + — the design record (RFC, implemented) of the direct JSON-Schema → + structural-OpenAPI-v3 transpiler that replaced the legacy crdgen path. diff --git a/docs/llms.txt b/docs/llms.txt new file mode 100644 index 0000000..c36aabe --- /dev/null +++ b/docs/llms.txt @@ -0,0 +1,12 @@ +# plumbing — LLM doc index (pinned: v1.13.2) +# One line per file; regenerate the pin on release. +docs/index.md: the map of this bundle — what the shared Go library is, links to everything +docs/overview.md: package design — the helm apply-if-changed Reconcile engine, crdgen, kubeutil, labels, and how core-provider/CDC consume them +docs/usage.md: go get + the exact-tag version-pinning conventions consumers follow + the bump chain +docs/configuration.md: the config surface the library itself reads — two env vars, the AUTHN_JWT_SECRET key-name constant, test build tags +docs/api.md: the exported package map, one line per package, grouped by theme (full godoc on pkg.go.dev) +docs/examples.md: examples index +docs/release.md: how a release ships — tag-only Go module (vX.Y.Z git tags, no OCI artifact) +docs/log.md: curated history, newest first +examples/minimal/README.md: runnable example — values.schema.json to CRD via crdgen.Generate +crdgen/docs/ref-resolution-redesign.md: design record of the direct JSON-Schema to structural-OpenAPI-v3 crdgen transpiler diff --git a/docs/log.md b/docs/log.md new file mode 100644 index 0000000..5fb6b8c --- /dev/null +++ b/docs/log.md @@ -0,0 +1,57 @@ +--- +type: Log +title: plumbing history +description: Curated chronological history of the plumbing library — notable changes, decisions and incidents, newest first. Release notes stay in git tags. +resource: github.com/krateo-platformops/plumbing +tags: [history] +timestamp: 2026-08-07T00:00:00Z +--- + +# History (curated, newest first) + +- **2026-08-07 — docs**: adopted the Krateo Documentation Standard (this bundle); + `lint-docs` wired into CI. +- **2026-08-05 — envtest CI**: functional cache-staleness coverage for `helm/v3` + runs against a real kube-apiserver (`make test-envtest`, `-tags envtest`) in CI + alongside the race-enabled unit suite (#18). +- **2026-08-04 — v1.13.2**: `helm/v3` `Install` retries once on a stale-discovery + REST-mapping miss (chart installed immediately after its CRD lands). +- **2026-08-04 — v1.13.1**: `kubeutil/objectclient.Apply` writes the server + response back into the object — the fix for core-provider's + CompositionDefinition deployed-digest gate never converging (desired hash was + computed over the bare render instead of the server-defaulted object). +- **2026-08-03 — v1.13.0**: module identity migrated to + `github.com/krateo-platformops/plumbing` (org independence; no `/v2` suffix). +- **2026-07-27 — v1.12.0**: five new `kubeutil` packages — `hasher`, + `objectclient`, `secretref`, `rbacgen`, `dynamicwatch` — extracted as shared + controller tooling. +- **2026-07-24 — v1.10.7 / v1.11.0**: crdgen sanitizes type-less object nodes so + rich third-party schemas yield valid CRDs (#11); `NormalizeVersionName` exported + (#12); the **direct JSON-Schema → structural-OpenAPI-v3 transpiler** replaced + the legacy crdgen path (#15 RFC, #16) — see + [crdgen/docs/ref-resolution-redesign.md](../crdgen/docs/ref-resolution-redesign.md). +- **2026-07-12 — v1.10.6**: `Upgrade` propagates `TakeOwnership` (self-healing + adoption was silently dropped). +- **2026-07-11 — v1.10.5**: `labels` package created — the shared home for the + cross-repo composition label keys (compile-time contract between core-provider + and composition-dynamic-controller). +- **2026-07-09..10 — v1.10.0..v1.10.3**: the **apply-if-changed `Reconcile`** + landed with semantic change-detection, then hardened: don't wedge on a child's + CRD-version migration, adopt live-owned children missing from the stored + manifest, repair the stored manifest against unserved GVKs. This is the engine + behind CDC's no-churn self-healing Observe. +- **2026-07 — traceparent scoping**: the `krateo.io/traceparent` post-render stamp + was scoped to `*.krateo.io` resources only — stamping every child made GKE + re-ensure LoadBalancer Services on every reconcile (IP reserve/release thrash). + Shipped on the `v1.7.x` maintenance line (extended through `v1.7.16`) while + `main` was ahead. +- **2026-04-22 — v1.8.0**: the pretty logger was removed in favor of stdlib + `slog` JSON output (a brief v1.10.x restore of `slogs/pretty` was reverted). +- **2026-02..06 — v1.0.0..v1.9.0**: the Helm `Client` introduced (v1.0.0) and + grown: discovery cache with CRD informer (v1.6.0), REST-config + chart caching + (v1.7.1), duplicate-resource validation (v1.7.2); crdgen fixes (concurrent + generation race, int-or-string, min/max lengths); AWS Signature v4 request + support; eventbus + discovery events; throttled event recorder; probes, wait, + pgutil. +- **pre-v1.0.0**: utility-belt era (`v0.x`) — jqutil, maps, endpoints, jwtutil, + shortid and friends accumulated as the platform services were extracted. diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..fcc961c --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,114 @@ +--- +type: Architecture +title: plumbing architecture +description: Package design of the shared Go library — the helm apply-if-changed Reconcile engine, crdgen, the kubeutil family, the shared label contract — and how core-provider and composition-dynamic-controller consume them. +resource: github.com/krateo-platformops/plumbing +tags: [helm, reconcile, crdgen, labels] +timestamp: 2026-08-07T00:00:00Z +--- + +# Architecture + +plumbing is a flat collection of small, independent Go packages with one module +identity (`github.com/krateo-platformops/plumbing`, see [go.mod](../go.mod)). There is +no binary and no shared framework: each package is importable on its own, and the +library's real job is to be the **single home for code and contracts that would +otherwise drift across the platform repos**. Three packages carry that weight +explicitly; the rest are conventional utilities (mapped one-per-line in +[api](./api.md)). + +## 1. The helm engine (`helm`, `helm/v3`, `helm/getter`, `helm/utils`) + +`helm` defines the provider-agnostic `Client` interface (Install / Upgrade / +**Reconcile** / Uninstall / Rollback / GetRelease / ListReleases) plus its config and +result types. `helm/v3` implements it on the unforked Helm v3 SDK, adding what a +controller needs and upstream Helm does not provide: + +- **cached, invalidation-aware clients** — action configs and discovery are cached + per target; a CRD informer (`NewCRDInformer` / `WithCRDInformer`) invalidates the + discovery/REST-mapper cache when CRDs change, and `Install` retries once on a + stale-discovery REST-mapping miss (v1.13.2) so a chart installed right after its + CRD lands does not fail spuriously; +- **post-render validation** — duplicate-resource detection in rendered manifests; +- **`helm/utils` post-renderers** — the composition labels post-renderer, and the + `krateo.io/traceparent` stamp deliberately scoped to `*.krateo.io` children only + (stamping everything made GKE re-ensure LoadBalancer Services every minute); +- **`helm/getter`** — OCI/repo chart fetching with credentials, TLS options and a + content-addressed disk cache (`helm/getter/cache`). + +### The apply-if-changed `Reconcile` (why CDC does not fork helm) + +`Client.Reconcile` ([helm/v3/reconcile.go](../helm/v3/reconcile.go)) is a fork-free, +self-healing reconcile built ONLY on Helm's exported API. Per cycle: + +1. Fetch the stored release (its `.Manifest` = last-applied state). No release yet → + delegate to `Upgrade(Install:true)`, report `Changed`. +2. Repair the stored manifest against GVKs the cluster no longer serves (a + CompositionDefinition version bump prunes the old served CRD version; without the + repair, helm's own current-manifest build deadlocks hard). +3. Render the **target** manifest via a server-side dry-run Upgrade (real render + pipeline — post-renderer, lookups, server validation — no revision, no hooks). +4. Snapshot each target object's live state; neutralize volatile non-semantic diffs + (copy the live `traceparent` onto the target, fold Secret `stringData` into + `data` as the apiserver does); adopt live-owned children missing from the stored + manifest. +5. `KubeClient.UpdateThreeWayMerge(current, target)` — Helm's own 3-way merge + **converges the cluster**: recreates children deleted out-of-band, patches + drifted fields. +6. Decide `changed` by **semantic** before/after comparison with write-volatile and + server-owned fields stripped — not `resourceVersion` deltas, which over-count and + churned a Helm revision every cycle at steady state. +7. Only if changed: run ONE real `Upgrade` to write the revision and run hooks with + correct ordering. Otherwise return `Changed:false` — no revision, no hooks. + +The steady-state guarantee is "no cluster mutation → no Helm revision, no hooks", +which is what stopped the platform's per-60s revision churn. It is only as no-churn +as the charts are idempotent (e.g. `lookup`-guarded random passwords). + +## 2. `crdgen` — JSON Schema → CRD + +`crdgen.Generate` transpiles a chart's `values.schema.json` directly into a +structural OpenAPI-v3 CRD ([crdgen/transpile.go](../crdgen/transpile.go)): `$ref`s +inlined by JSON pointer, cycles broken with `x-kubernetes-preserve-unknown-fields`, +tractable validation keywords carried over (some as generated CEL), and the output +gated in-process on the `k8s.io/apiextensions-apiserver` validation library — so a +generated CRD is structurally valid by construction. Design record: +[crdgen/docs/ref-resolution-redesign.md](../crdgen/docs/ref-resolution-redesign.md). +`NormalizeVersionName` maps a chart semver to the CRD version name (`1.0.0` → +`v1-0-0`). See it run in [examples/minimal](../examples/minimal/README.md). + +## 3. `labels` — the cross-repo contract + +[labels](../labels/labels.go) declares, once, the composition label keys that couple +core-provider and composition-dynamic-controller: core-provider stamps each +composition instance with its owning CompositionDefinition coordinates and served +version; the per-version CDC selects instances by the same keys. One byte of drift +would make version migration silently select nothing — importing the keys from one +package turns the agreement into a compile-time guarantee. + +## 4. The kubeutil family and the rest + +`kubeutil/*` holds controller-grade helpers: `objectclient` (retrying dynamic +create-or-update whose `Apply` writes the server response back into the object — +the v1.13.1 fix that unwedged core-provider's deployed-digest gate), `hasher` +(cumulative drift hash), `rbacgen` (least-privilege Role/RoleBinding generation), +`dynamicwatch` (watch a GVK unknown until runtime), `secretref`, `plurals`, +`eventrecorder` (throttled), `rbac` (batched SelfSubjectAccessReview-style checks +with cache). Alongside: identity plumbing (`jwtutil`, `kubeconfig`, `endpoints`, +`signup`, `certs`) used by authn/snowplow, HTTP server/client helpers +(`server/*`, `http/*`), and generic utilities (`cache`, `maps`, `jqutil`, `env`, +`ptr`, `wait`, `shortid`, `logger`, `eventbus`, `pgutil`, `e2e`). + +## How the platform consumes it + +- **core-provider** (engine, v1.13.2 pin): `crdgen` for CRD generation per + CompositionDefinition; `kubeutil/objectclient` + `hasher` + `rbacgen` in its + deploy tooling; `plurals`, `eventrecorder`, `labels`. +- **composition-dynamic-controller**: `helm/v3` end to end — `NewClient` per + target, `Client.Reconcile` as its Observe-phase apply-if-changed self-heal, + `helm/utils` post-renderers, `helm/getter` for chart fetch; plus `labels`, + `dynamicwatch`, `secretref`. +- **chart-inspector, snowplow, authn, and the other platform services** use the + identity, HTTP and utility packages. + +Consumers pin explicit tags and bump deliberately — see [usage](./usage.md). diff --git a/docs/release.md b/docs/release.md new file mode 100644 index 0000000..15f3d5b --- /dev/null +++ b/docs/release.md @@ -0,0 +1,52 @@ +--- +type: Runbook +title: Releasing plumbing +description: How a plumbing release ships — tag-only Go module (plain vX.Y.Z git tags, no OCI artifact, no release workflow), and how a bump reaches consumers. +resource: github.com/krateo-platformops/plumbing +tags: [release, tags] +timestamp: 2026-08-07T00:00:00Z +--- + +# Release runbook + +plumbing is a **tag-only Go module**. There is no release workflow, no image, no +chart, no OCI artifact — a release is a semver git tag on `main`, served to +consumers by the Go module proxy. This is the reality of the repo today: the only +workflow is [test.yaml](../.github/workflows/test.yaml) (unit + envtest on every PR +and push to `main`). + +## Convention (derived from the existing tags) + +- Tags are `vMAJOR.MINOR.PATCH`, **with** the `v` prefix (`v1.13.2` is current; + the line runs back through `v1.0.0`). Note this differs from the platform's + chart/image repos, whose tags carry no `v` — Go modules require it. +- Minor bumps for new packages/features (`v1.12.0` added five kubeutil packages), + patch bumps for fixes (`v1.13.1` objectclient Apply fix). Contract-affecting + changes (e.g. `labels`) always get a tag consumers can sweep to. +- Historical maintenance tags exist off-`main` (the `v1.7.x` line was extended + through `v1.7.16` while `main` was ahead); prefer fixing on `main` and bumping + consumers unless a consumer is pinned to an old line for cause. +- The module identity is `github.com/krateo-platformops/plumbing` since `v1.13.0` + (org-independence migration) with no `/v2` path suffix. + +## Ship a release + +1. Land the change on `main` via PR; `test` (unit+race, envtest) and `lint-docs` + must be green. +2. Tag and push: + + ```sh + git tag vX.Y.Z + git push origin vX.Y.Z + ``` + +3. There is no step 3 in this repo. Propagation is the consumer chain: bump + `go.mod` in core-provider / composition-dynamic-controller (and any other + consumer), release those, then bump the chart/installer pins — see + [usage](./usage.md). + +## Docs freshness + +`docs/llms.txt` pins this bundle to the release tag; when tagging, update the pin +and review the core files' `timestamp` (CI warns when a doc timestamp trails the +latest tag). diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..dd20a5d --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,71 @@ +--- +type: Usage +title: Using plumbing +description: go get, the explicit version-pinning conventions Krateo consumers follow, and how a plumbing bump propagates through core-provider into the installer. +resource: github.com/krateo-platformops/plumbing +tags: [go-get, versioning] +timestamp: 2026-08-07T00:00:00Z +--- + +# Usage + +plumbing is consumed only as a Go module. There is nothing to deploy: no image, no +chart, no CRD. + +## Install + +```sh +go get github.com/krateo-platformops/plumbing@latest +``` + +or pin an exact tag (what every platform repo actually does): + +```sh +go get github.com/krateo-platformops/plumbing@v1.13.2 +``` + +The repo is public — no `GOPRIVATE`/auth setup is needed. Import any package +directly, e.g.: + +```go +import ( + "github.com/krateo-platformops/plumbing/helm/v3" + "github.com/krateo-platformops/plumbing/labels" +) +``` + +A compilable starting point is [examples/minimal](../examples/minimal/README.md) +(`go run ./examples/minimal` from the repo root). + +## Version-pinning conventions (how Krateo consumers depend on this library) + +- **Pin an exact release tag, never a pseudo-version of `main`.** Every consumer's + `go.mod` carries `github.com/krateo-platformops/plumbing vX.Y.Z` (core-provider + pins `v1.13.2`, composition-dynamic-controller `v1.13.0` today). `main` is + release-worthy but unreleased commits are not consumed. +- **Contract packages must be the SAME version on both sides.** `labels` (and any + other cross-repo contract) only delivers its compile-time guarantee if + core-provider and CDC resolve the same plumbing version; bumps to contract + packages are propagated to all parties in one sweep. +- **Module identity is `krateo-platformops` (since v1.13.0).** The library migrated + from the dead org as a major-independence move without a `/v2` suffix — the module + path in `go.mod` is the truth; old-org import paths in downstream code are bugs to + fix, not aliases. +- **A fix ships as a chain, not a commit.** The pattern for a plumbing fix reaching + a cluster: tag plumbing `vX.Y.Z` → bump the consumer's `go.mod` (core-provider / + CDC) and release it → bump the chart / installer pin. Nothing consumes plumbing + at run time; only through released consumers. + +## Build & test locally + +```sh +go build ./... +make test # fast unit suite +make test-race # unit suite with -race (what CI runs) +make test-envtest # functional cache-staleness tests against a real kube-apiserver +``` + +`make test-envtest` downloads a kubebuilder envtest control plane (pinned via the +[Makefile](../Makefile), see [configuration](./configuration.md) for the build +tags). Tests tagged `integration` additionally expect a reachable cluster and are +not run by CI. diff --git a/examples/minimal/README.md b/examples/minimal/README.md new file mode 100644 index 0000000..b8b398b --- /dev/null +++ b/examples/minimal/README.md @@ -0,0 +1,30 @@ +--- +type: Example +title: minimal — generate a CRD from a chart's values.schema.json +description: Runnable example that feeds a JSON Schema to crdgen.Generate and prints the resulting CustomResourceDefinition YAML, the same transformation core-provider performs per CompositionDefinition. +resource: github.com/krateo-platformops/plumbing/crdgen +tags: [example, crdgen] +timestamp: 2026-08-07T00:00:00Z +--- + +# minimal + +Feeds a chart-style `values.schema.json` (inline JSON Schema) to +[`crdgen.Generate`](../../crdgen/crdgen.go) and prints the resulting +`CustomResourceDefinition` YAML — the exact transformation the core-provider engine +performs for every `CompositionDefinition`. It also shows +`crdgen.NormalizeVersionName` (chart version `1.0.0` → CRD version `v1-0-0`) and the +`Managed` option (conditioned status subresource). + +## Preconditions + +- Go (version per [`go.mod`](../../go.mod)). No cluster, no network beyond the module cache. + +## Run + +```sh +go run ./examples/minimal +``` + +Expected output: a structurally valid CRD named `dummyapps.examples.krateo.io` with a +single served version `v1-0-0`. diff --git a/examples/minimal/main.go b/examples/minimal/main.go new file mode 100644 index 0000000..6063b8b --- /dev/null +++ b/examples/minimal/main.go @@ -0,0 +1,50 @@ +// Command minimal is the runnable example for the plumbing library: it feeds a +// chart-style values.schema.json (JSON Schema) to crdgen.Generate and prints the +// resulting CustomResourceDefinition YAML — the exact transformation core-provider +// performs for every CompositionDefinition. It needs no cluster and no network. +// +// Run from the repo root: +// +// go run ./examples/minimal +package main + +import ( + "fmt" + "log" + + "github.com/krateo-platformops/plumbing/crdgen" +) + +// specSchema is a minimal chart values.schema.json: two typed fields, one with a +// default, one required. +const specSchema = `{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "required": ["host"], + "properties": { + "host": { + "type": "string", + "description": "Hostname the app is served on." + }, + "replicas": { + "type": "integer", + "default": 1, + "description": "Number of replicas." + } + } +}` + +func main() { + crd, err := crdgen.Generate(crdgen.Options{ + Group: "examples.krateo.io", + Version: crdgen.NormalizeVersionName("1.0.0"), // "v1-0-0" + Kind: "DummyApp", + Categories: []string{"compositions"}, + SpecSchema: []byte(specSchema), + Managed: true, // adds the conditioned status subresource + }) + if err != nil { + log.Fatalf("crdgen: %v", err) + } + fmt.Print(string(crd)) +}