Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,7 @@ jobs:

- name: Functional cache-staleness tests (envtest)
run: make test-envtest

lint-docs:
name: lint-docs
uses: krateo-platformops/.github/.github/workflows/lint-docs.yaml@main
53 changes: 49 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,54 @@
# `plumbing`
# plumbing

[![Go Report Card](https://goreportcard.com/badge/github.com/krateoplatformops/plumbing)](https://goreportcard.com/report/github.com/krateoplatformops/plumbing)     [![Reference](https://pkg.go.dev/badge/github.com/krateoplatformops/plumbing)](https://pkg.go.dev/github.com/krateoplatformops/plumbing)
The shared Go library of the Krateo platform — reusable building blocks and the cross-repo contracts (helm reconcile engine, crdgen, composition labels) consumed by core-provider, composition-dynamic-controller and the platform services.

[![test](https://github.com/krateo-platformops/plumbing/actions/workflows/test.yaml/badge.svg)](https://github.com/krateo-platformops/plumbing/actions/workflows/test.yaml)
[![Reference](https://pkg.go.dev/badge/github.com/krateo-platformops/plumbing)](https://pkg.go.dev/github.com/krateo-platformops/plumbing)

## What is this

Provides reusable building blocks and common helpers to reduce boilerplate across [Krateo Platformops](https://krateo.io/) projects.
A flat collection of independent Go packages with one module identity. It carries
the Helm client with the fork-free apply-if-changed `Reconcile` engine that
composition-dynamic-controller runs on, the `crdgen` JSON-Schema → CRD transpiler
core-provider uses, the `labels` package that pins the core-provider ↔ CDC label
contract at compile time, plus `kubeutil/*`, `jwtutil` and small utilities.
Library only: no binary, no image, no chart. Full picture:
[docs/index.md](docs/index.md).

It includes general-purpose utilities aimed at simplifying day-to-day development while promoting consistency and maintainability.
## Install

```sh
go get github.com/krateo-platformops/plumbing@v1.13.2
```

## Configure

There is almost nothing to configure — see
[docs/configuration.md](docs/configuration.md). Most used:

| Setting | Default | Effect |
|---|---|---|
| `HELM_DRIVER` | `secret` | Helm release-storage driver used by `helm/v3`. |
| `CLUSTER_NAME` | unset | Overrides `kubeutil.DetectClusterName`. |
| build tag `envtest` | off | Selects the real-apiserver functional tests (`make test-envtest`). |

## Examples

- [examples/minimal](examples/minimal) — `values.schema.json` → CRD via
`crdgen.Generate`; `go run ./examples/minimal`.

## Docs

- [docs/index.md](docs/index.md) — the map
- [docs/overview.md](docs/overview.md) — package design + the Reconcile engine
- [docs/usage.md](docs/usage.md) — go get + version-pinning conventions
- [docs/configuration.md](docs/configuration.md) — the whole config surface
- [docs/api.md](docs/api.md) — the exported package map
- [docs/examples.md](docs/examples.md) — examples index
- [docs/release.md](docs/release.md) — how a release ships (tag-only)
- [docs/log.md](docs/log.md) — curated history

## Develop & release

`go build ./... && make test-race` (CI adds `make test-envtest`). Releases are
plain `vX.Y.Z` git tags — see [docs/release.md](docs/release.md).
98 changes: 98 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
---
type: API
title: plumbing exported API
description: The exported package map — one line per package, grouped by theme, derived from the package docs and code; full godoc lives on pkg.go.dev.
resource: https://pkg.go.dev/github.com/krateo-platformops/plumbing
tags: [go-api, packages]
timestamp: 2026-08-07T00:00:00Z
---

# Exported API

The contract is the exported surface of every Go package below. This page is the
**map**, not the reference — full signatures and doc comments live at
[pkg.go.dev/github.com/krateo-platformops/plumbing](https://pkg.go.dev/github.com/krateo-platformops/plumbing)
(or `go doc github.com/krateo-platformops/plumbing/<pkg>`). One line per package,
derived from the actual package docs and code.

## Helm engine

| Package | What it exports |
|---|---|
| `helm` | The provider-agnostic `Client` interface (Install/Upgrade/**Reconcile**/Uninstall/Rollback/GetRelease/ListReleases) plus config, `Release`/`ReconcileResult` and `PostRenderer` types. |
| `helm/v3` | The Helm-v3-SDK implementation: cached action configs, CRD-informer discovery invalidation (`NewCRDInformer`), duplicate-resource post-render validation, and the fork-free apply-if-changed `Reconcile` (see [overview](./overview.md)). |
| `helm/v3/tracer` | An `http.RoundTripper` that logs each Kubernetes request a Helm operation makes and collects the touched `Resource`s. |
| `helm/getter` | Chart fetching (`Get`, `NewOCIGetter`) with credentials, TLS, timeout, repo and disk-cache options. |
| `helm/getter/cache` | `DiskCache` — content-addressed on-disk cache for fetched charts. |
| `helm/getter/repo` | Classic HTTP chart-repository index types (`IndexFile`, `ChartVersions`) and URL helpers. |
| `helm/utils` | Post-render helpers: composition `LabelsPostRender`, the `krateo.io/traceparent` stamp (scoped to `*.krateo.io`), the `krateo.io/gracefully-paused` annotation key. |

## Kubernetes utilities

| Package | What it exports |
|---|---|
| `kubeutil` | Misc helpers: DNS-1123 name mangling, `DetectClusterName`, `ServiceAccountNamespace`, `ConfigMapData`, in-cluster CA cert. |
| `kubeutil/objectclient` | Retrying, `dynamic.Interface`-based create-or-update (`Apply` — writes the server response back into the object), `Get`, delete-if-present. |
| `kubeutil/hasher` | Cumulative, order-dependent hash over JSON-marshalable values — desired-vs-deployed drift detection. |
| `kubeutil/rbacgen` | Builds least-privilege Role/RoleBinding sets granting one ServiceAccount access to an exact named resource set. |
| `kubeutil/rbac` | Batched `UserCan` permission checks (SelfSubjectAccessReview) with a TTL cache. |
| `kubeutil/dynamicwatch` | Registers a controller-runtime watch on a GVK not known until runtime, deduping repeat registrations. |
| `kubeutil/secretref` | Reads one key out of a Secret via `dynamic.Interface` (for schema-unknown-at-compile-time reconcilers). |
| `kubeutil/plurals` | GVK → plural/singular resource-name resolution (`Get`, `ResolveAPINames`). |
| `kubeutil/event` | `APIRecorder` + `Normal`/`Warning` event constructors for controller runtimes. |
| `kubeutil/eventrecorder` | `record.EventRecorder` factories, including a state-aware throttled recorder. |
| `kubeutil/discoveryevents` | Publishes API-discovery resource added/changed/removed events onto an `eventbus.Bus`. |
| `labels` | THE shared composition label keys coupling core-provider and composition-dynamic-controller (compile-time cross-repo contract). |

## CRD generation

| Package | What it exports |
|---|---|
| `crdgen` | `Generate(Options)` — direct JSON-Schema → structural-OpenAPI-v3 CRD transpiler, gated on the apiextensions validation library; `NormalizeVersionName`. |
| `crdgen/schemas` | JSON-Schema resolution primitives and type-name constants used by the transpiler. |

## Identity, endpoints, auth

| Package | What it exports |
|---|---|
| `jwtutil` | HS256 Krateo JWT create/verify with `UserInfo` claims; the shared `AUTHN_JWT_SECRET` env-key constant. |
| `endpoints` | The Krateo `Endpoint` (user API-server credential record) stored/loaded as a Secret. |
| `kubeconfig` | `Endpoint` → kubeconfig YAML (`Marshal`) and → `*rest.Config` (`NewClientConfig`). |
| `signup` | Creates a certificate-based cluster user (CSR flow) and stores its `Endpoint`. |
| `certs` | CertificateSigningRequest helpers: create, approve, wait, fetch the issued cert. |

## HTTP client & server

| Package | What it exports |
|---|---|
| `http/request` | Outbound request building incl. AWS Signature v4 header computation. |
| `http/response` | The `Status` response envelope (`Success`/`Failure`) + typed error writers (`BadRequest`, `Forbidden`, …). |
| `http/util` | `RetryClient` — retrying, rate-limited `http.Client` wrapper. |
| `server/probes` | `/livez` + `/readyz` handlers and a `HealthServer`. |
| `server/use` | Middleware: access logging, CORS, trace-id propagation. |
| `server/use/cors` | The CORS handler implementation used by `server/use`. |

## General utilities

| Package | What it exports |
|---|---|
| `bufferpool` | Sized `sync.Pool`-backed byte-buffer pool. |
| `cache` | Generic `TTLCache[K,V]` with cleanup interval and max-entries options. |
| `codegen` | Fluent Go source-code generation builders (`Package`, `Function`, `If`, …). |
| `context` | Krateo request context: trace-id (`X-Krateo-TraceId`), per-request logger, access token, user config/info. |
| `deps` | Small dependency `Graph` with topological resolution. |
| `env` | Typed env readers (`String`, `Bool`, `Int`, `Duration`, `ServicePort`, …) with defaults. |
| `eventbus` | In-process publish/subscribe `Bus`. |
| `jqutil` | gojq evaluation helpers: `Eval`, `Extract`, `ForEach`, module loading, type inference. |
| `logger` | JSON `slog.Logger` factory with a service-name attribute. |
| `maps` | Nested-map access/copy (`NestedString`, `DeepCopyJSON`, `LeafPaths`, …). |
| `pgutil` | PostgreSQL connection-URL building and wait-for-ready pool creation (pgx). |
| `ptr` | Generic pointer helpers (`To`, `Deref`, `Equal`). |
| `shortid` | Short, unique, non-sequential URL-friendly id generation. |
| `wait` | Generic backoff retry `Until` / `UntilWithOptions`. |
| `e2e` | e2e-framework helpers for consumer test suites: namespaces, coverage, sign-up step. |

Compatibility note: within the v1 line, exported APIs may still move between minors
(e.g. the slogs→slog removal, the crdgen transpiler replacement) — this is an
internal platform library, and consumers pin exact tags precisely for this reason
(see [usage](./usage.md)).
38 changes: 38 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
type: Configuration
title: plumbing configuration
description: The whole config surface the library itself reads — two runtime env vars, one conventional env-key constant, and the test-only build tags.
resource: github.com/krateo-platformops/plumbing
tags: [env, build-tags]
timestamp: 2026-08-07T00:00:00Z
---

# Configuration

plumbing is a library: it has no config file, no flags, no values. Configuration is
whatever each package's `Options`/functional-option arguments say (see
[api](./api.md)). The library itself reads exactly this from the environment:

| Env var | Read by | Effect |
|---|---|---|
| `HELM_DRIVER` | `helm/v3` ([client.go](../helm/v3/client.go)) | Helm storage driver for release records (`secret` when unset — the Helm SDK default). |
| `CLUSTER_NAME` | `kubeutil` ([detect_cluster.go](../kubeutil/detect_cluster.go)) | First choice for `DetectClusterName`; falls back to the rest-config host, then the local hostname. |
| `KUBERNETES_SERVICE_HOST` / `KUBERNETES_SERVICE_PORT` | `signup` ([signup.go](../signup/signup.go)) | In-cluster apiserver URL when `Options.ServerURL` is empty (standard in-cluster env). |

One near-miss worth stating: `jwtutil.JwtSecretEnvKey` (`AUTHN_JWT_SECRET`) is a
**shared env-key NAME constant**, not an env read — `CreateToken` takes the signing
key via `CreateTokenOptions.SigningKey` and errors if it is empty; consumers
(authn, snowplow) read the env var themselves under this agreed name.

The `env` package (`env.Bool`, `env.String`, `env.Duration`, …) is a typed
env-reading helper **for consumers**; it does not make the library read anything on
its own.

## Build tags

No non-test code carries a build tag. Test-only:

| Tag | Selects | Needs |
|---|---|---|
| `envtest` | the functional cache-staleness tests in `helm/v3` | a kubebuilder envtest control plane (`make test-envtest` installs it, pinned in the [Makefile](../Makefile)) |
| `integration` | tests in `endpoints` and `helm/v3` that hit a real cluster | a reachable kubeconfig cluster; not run in CI |
17 changes: 17 additions & 0 deletions docs/examples.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
type: ExampleIndex
title: plumbing examples
description: Index of the runnable examples under examples/ — one line each.
resource: github.com/krateo-platformops/plumbing
tags: [examples]
timestamp: 2026-08-07T00:00:00Z
---

# Examples

- [examples/minimal](../examples/minimal/README.md) — feed a chart-style
`values.schema.json` to `crdgen.Generate` and print the resulting CRD YAML
(no cluster needed): `go run ./examples/minimal`.

The example is part of the module, so `go build ./...` at the repo root compiles it
and CI keeps it honest.
42 changes: 42 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
type: Library
title: plumbing — index
description: The map of the plumbing doc bundle — the shared Go library (helm client + reconcile engine, kubeutil, crdgen, jwtutil, labels and friends) consumed by the Krateo platform components.
resource: github.com/krateo-platformops/plumbing
tags: [library, helm, crdgen, kubeutil]
timestamp: 2026-08-07T00:00:00Z
---

# plumbing

plumbing is **the shared Go library of the Krateo platform**: reusable building
blocks that reduce boilerplate and — more importantly — pin cross-repo contracts in
one place. It carries the Helm client with the fork-free **apply-if-changed
`Reconcile` engine** that composition-dynamic-controller runs on, the
**`crdgen`** JSON-Schema → CRD transpiler that core-provider uses to turn a chart's
`values.schema.json` into a versioned CRD, the **`labels`** package that makes the
core-provider ↔ CDC composition-label contract a compile-time guarantee, plus
`kubeutil/*` controller helpers, `jwtutil`, and small utilities. It is a library
only: no binary, no image, no chart — releases are plain git tags.

## The bundle (start here)

- [overview](./overview.md) — package design: the helm engine (and the
`Reconcile` flow step by step), crdgen, the kubeutil family, and how
core-provider/CDC consume them.
- [usage](./usage.md) — `go get`, the version-pinning conventions consumers follow,
and the bump chain into core-provider and the installer.
- [configuration](./configuration.md) — the (small) config surface the library
itself reads: two env vars and two test build tags.
- [api](./api.md) — the exported package map, one line per package, with
pkg.go.dev links.
- [examples](./examples.md) — the runnable example under `examples/`.
- [release](./release.md) — how a release ships (tag-only, no OCI artifact).
- [log](./log.md) — curated history.
- [llms.txt](./llms.txt) — the version-pinned agent index of this bundle.

## Deeper, code-adjacent docs

- [crdgen/docs/ref-resolution-redesign.md](../crdgen/docs/ref-resolution-redesign.md)
— the design record (RFC, implemented) of the direct JSON-Schema →
structural-OpenAPI-v3 transpiler that replaced the legacy crdgen path.
12 changes: 12 additions & 0 deletions docs/llms.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# plumbing — LLM doc index (pinned: v1.13.2)
# One line per file; regenerate the pin on release.
docs/index.md: the map of this bundle — what the shared Go library is, links to everything
docs/overview.md: package design — the helm apply-if-changed Reconcile engine, crdgen, kubeutil, labels, and how core-provider/CDC consume them
docs/usage.md: go get + the exact-tag version-pinning conventions consumers follow + the bump chain
docs/configuration.md: the config surface the library itself reads — two env vars, the AUTHN_JWT_SECRET key-name constant, test build tags
docs/api.md: the exported package map, one line per package, grouped by theme (full godoc on pkg.go.dev)
docs/examples.md: examples index
docs/release.md: how a release ships — tag-only Go module (vX.Y.Z git tags, no OCI artifact)
docs/log.md: curated history, newest first
examples/minimal/README.md: runnable example — values.schema.json to CRD via crdgen.Generate
crdgen/docs/ref-resolution-redesign.md: design record of the direct JSON-Schema to structural-OpenAPI-v3 crdgen transpiler
57 changes: 57 additions & 0 deletions docs/log.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
type: Log
title: plumbing history
description: Curated chronological history of the plumbing library — notable changes, decisions and incidents, newest first. Release notes stay in git tags.
resource: github.com/krateo-platformops/plumbing
tags: [history]
timestamp: 2026-08-07T00:00:00Z
---

# History (curated, newest first)

- **2026-08-07 — docs**: adopted the Krateo Documentation Standard (this bundle);
`lint-docs` wired into CI.
- **2026-08-05 — envtest CI**: functional cache-staleness coverage for `helm/v3`
runs against a real kube-apiserver (`make test-envtest`, `-tags envtest`) in CI
alongside the race-enabled unit suite (#18).
- **2026-08-04 — v1.13.2**: `helm/v3` `Install` retries once on a stale-discovery
REST-mapping miss (chart installed immediately after its CRD lands).
- **2026-08-04 — v1.13.1**: `kubeutil/objectclient.Apply` writes the server
response back into the object — the fix for core-provider's
CompositionDefinition deployed-digest gate never converging (desired hash was
computed over the bare render instead of the server-defaulted object).
- **2026-08-03 — v1.13.0**: module identity migrated to
`github.com/krateo-platformops/plumbing` (org independence; no `/v2` suffix).
- **2026-07-27 — v1.12.0**: five new `kubeutil` packages — `hasher`,
`objectclient`, `secretref`, `rbacgen`, `dynamicwatch` — extracted as shared
controller tooling.
- **2026-07-24 — v1.10.7 / v1.11.0**: crdgen sanitizes type-less object nodes so
rich third-party schemas yield valid CRDs (#11); `NormalizeVersionName` exported
(#12); the **direct JSON-Schema → structural-OpenAPI-v3 transpiler** replaced
the legacy crdgen path (#15 RFC, #16) — see
[crdgen/docs/ref-resolution-redesign.md](../crdgen/docs/ref-resolution-redesign.md).
- **2026-07-12 — v1.10.6**: `Upgrade` propagates `TakeOwnership` (self-healing
adoption was silently dropped).
- **2026-07-11 — v1.10.5**: `labels` package created — the shared home for the
cross-repo composition label keys (compile-time contract between core-provider
and composition-dynamic-controller).
- **2026-07-09..10 — v1.10.0..v1.10.3**: the **apply-if-changed `Reconcile`**
landed with semantic change-detection, then hardened: don't wedge on a child's
CRD-version migration, adopt live-owned children missing from the stored
manifest, repair the stored manifest against unserved GVKs. This is the engine
behind CDC's no-churn self-healing Observe.
- **2026-07 — traceparent scoping**: the `krateo.io/traceparent` post-render stamp
was scoped to `*.krateo.io` resources only — stamping every child made GKE
re-ensure LoadBalancer Services on every reconcile (IP reserve/release thrash).
Shipped on the `v1.7.x` maintenance line (extended through `v1.7.16`) while
`main` was ahead.
- **2026-04-22 — v1.8.0**: the pretty logger was removed in favor of stdlib
`slog` JSON output (a brief v1.10.x restore of `slogs/pretty` was reverted).
- **2026-02..06 — v1.0.0..v1.9.0**: the Helm `Client` introduced (v1.0.0) and
grown: discovery cache with CRD informer (v1.6.0), REST-config + chart caching
(v1.7.1), duplicate-resource validation (v1.7.2); crdgen fixes (concurrent
generation race, int-or-string, min/max lengths); AWS Signature v4 request
support; eventbus + discovery events; throttled event recorder; probes, wait,
pgutil.
- **pre-v1.0.0**: utility-belt era (`v0.x`) — jqutil, maps, endpoints, jwtutil,
shortid and friends accumulated as the platform services were extracted.
Loading
Loading