diff --git a/docs/deployment.md b/docs/deployment.md index 9f89c0fe..15c69f36 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -218,7 +218,7 @@ The chart ships with a `values.schema.json` that validates all values at `helm i | Value | Default | Description | |---|---|---| -| `web.enabled` | `false` | Enable web UI deployment | +| `web.enabled` | `true` | Enable web UI deployment (the BFF ingress entry point) | | `web.replicas` | `2` | Number of web replicas | | `web.image.repository` | `ghcr.io/mattrobinsonsre/terrapod-web` | Web Docker image | | `web.resources.requests.cpu` | `100m` | CPU request | diff --git a/docs/getting-started.md b/docs/getting-started.md index b7dc8946..06438da2 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,6 +1,8 @@ # Getting Started -This guide walks through setting up Terrapod locally, creating your first workspace, and running your first plan and apply against it. +This guide deploys Terrapod onto a Kubernetes cluster with the Helm chart, then creates your first workspace and runs your first plan and apply against it. + +> Terrapod runs **only on Kubernetes** — the runner uses the Jobs API to schedule plan/apply executions. You don't need a managed cloud cluster; a single-node [k3s](https://k3s.io/) VM works. If you instead want to run Terrapod **from source** for development, see [Local Development](local-development.md). > **OpenTofu is the recommended execution backend.** Terrapod supports both [OpenTofu](https://opentofu.org/) (`tofu`) and Terraform (`terraform`) as execution backends. OpenTofu is recommended because it is open-source under the MPL-2.0 license. All CLI examples in this guide use `tofu`, but `terraform` commands work identically. @@ -8,89 +10,86 @@ This guide walks through setting up Terrapod locally, creating your first worksp ## Prerequisites -### Required Software - -| Tool | Purpose | Install | -|---|---|---| -| Docker | Container runtime | [docker.com](https://www.docker.com/) | -| Kubernetes cluster | Local K8s (any of the below) | See below | -| Tilt | Local K8s dev environment | `brew install tilt` | -| mkcert | Local TLS certificates | `brew install mkcert` | -| tofu (recommended) or terraform | Infrastructure CLI | [opentofu.org](https://opentofu.org/) (recommended) or [terraform.io](https://www.terraform.io/) | - -### Supported Local Kubernetes Clusters - -Any of these will work: -- [Rancher Desktop](https://rancherdesktop.io/) -- [Docker Desktop](https://www.docker.com/products/docker-desktop/) (with Kubernetes enabled) -- [minikube](https://minikube.sigs.k8s.io/) -- [kind](https://kind.sigs.k8s.io/) -- [OrbStack](https://orbstack.dev/) -- [colima](https://github.com/abiosoft/colima) - ---- +| Need | Notes | +|---|---| +| A Kubernetes cluster (1.27+) | Any conformant cluster. No cluster? Spin up [k3s](https://k3s.io/) on a single VM (below). | +| Helm 3.x | `brew install helm` | +| PostgreSQL 14+ and Redis 7+ | **External** — the chart does not bundle them. Provide a connection URL for each (a managed service, or run them on the cluster/VM). See [Deployment → Database Setup](deployment.md#database-setup). | +| Storage | Defaults to a PVC-backed filesystem (no extra setup). For S3/Azure/GCS see [Deployment → Storage Backend Setup](deployment.md#storage-backend-setup). | +| `tofu` (recommended) or `terraform` | Infrastructure CLI — [opentofu.org](https://opentofu.org/) or [terraform.io](https://www.terraform.io/). | -## Local Development Setup +### Don't have a cluster? Use k3s -### Step 1: Install mkcert and create local CA +[k3s](https://k3s.io/) is a fully conformant single-binary Kubernetes that runs on one VM (~512 MB RAM) and ships with an ingress controller (Traefik) and local-path storage: ```zsh -brew install mkcert -mkcert -install +curl -sfL https://get.k3s.io | sh - # single-node cluster +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash ``` -This installs a local Certificate Authority into your system trust store so that `https://terrapod.local` is trusted by your browser and the terraform CLI. +Run PostgreSQL and Redis as managed services or on a separate VM so your data survives cluster recreation. + +--- -### Step 2: Add hosts entry +## Deploy Terrapod -```zsh -sudo sh -c 'echo "127.0.0.1 terrapod.local" >> /etc/hosts' -``` +Terrapod is published to GitHub Container Registry as an OCI Helm chart at `oci://ghcr.io/mattrobinsonsre/terrapod`. -### Step 3: Start Terrapod +Pick a hostname and install: ```zsh -make dev +export TERRAPOD_HOST=terrapod.example.com + +helm install terrapod oci://ghcr.io/mattrobinsonsre/terrapod \ + --namespace terrapod --create-namespace \ + --set ingress.enabled=true \ + --set ingress.hostname="$TERRAPOD_HOST" \ + --set ingress.className=traefik \ + --set postgresql.url="postgresql+asyncpg://terrapod:PASSWORD@PGHOST:5432/terrapod" \ + --set redis.url="redis://REDISHOST:6379" \ + --set bootstrap.adminEmail="admin@example.com" \ + --set bootstrap.adminPassword="change-me-now" ``` -This runs `tilt up --port 10352` which: +What the defaults give you: filesystem storage on a PVC, local password auth enabled, a pre-install migrations job, and a post-install bootstrap job that creates the admin user from `bootstrap.adminEmail` / `bootstrap.adminPassword`. Set `ingress.className` to your cluster's ingress controller (`traefik` on k3s, often `nginx` on managed clusters). Pin a version with `--version X.Y.Z`. -1. Creates the `terrapod` namespace -2. Generates TLS certificates via mkcert -3. Deploys PostgreSQL and Redis in-cluster -4. Builds the API and web Docker images -5. Runs database migrations (Alembic) -6. Bootstraps the initial admin user -7. Deploys the API server, web UI, and runner listener +> For production, put the admin password in a Kubernetes Secret (`bootstrap.existingSecret`) rather than on the command line, terminate TLS at the ingress (e.g. cert-manager), and set `api.config.external_url` so outbound links (VCS status, notifications) are correct. The full value reference — storage backends, external DB, SSO, scaling, TLS, runners — is in [Deployment](deployment.md), and [Production Checklist](production-checklist.md) covers hardening. -Open the Tilt UI at http://localhost:10352 to monitor the deployment. +Watch it come up: -### Step 4: Access Terrapod +```zsh +kubectl -n terrapod get pods -w +``` -Open https://terrapod.local in your browser. +### Access -![Login](images/login.png) +Point your hostname's DNS at the ingress controller's external IP (on k3s, the VM's IP — Traefik listens on ports 80/443). For a quick local test you can add a hosts entry instead: -The bootstrap job creates an initial admin user. The default credentials are `admin` / `admin` (configured in `values-local.yaml`). Check the Tilt logs for the `terrapod-bootstrap-1` resource to confirm. +```zsh +sudo sh -c 'echo " terrapod.example.com" >> /etc/hosts' +``` + +Then open `https://terrapod.example.com` and log in with the bootstrap admin email + password. -### Step 5: Get an API Token +> TLS: the ingress expects a certificate by default (`ingress.tls=true`). For a quick HTTP-only evaluation, add `--set ingress.tls=false`; for real TLS see [Deployment → Ingress](deployment.md#ingress). +> +> No DNS available at all? Port-forward the web service for a quick look: `kubectl -n terrapod port-forward svc/terrapod-web 8080:3000`, then open `http://localhost:8080`. -After logging in to the web UI, navigate to **Settings > API Tokens** and create a new token. Copy the token value -- it is only shown once. +### Get an API token -![API Tokens](images/api-tokens.png) +In the web UI, go to **Settings → API Tokens**, create a token, and copy the value (shown once): ```zsh export TERRAPOD_TOKEN="" ``` -Alternatively, use `terraform login`: +Or let the CLI mint one for you via the browser login flow: ```zsh -terraform login terrapod.local +tofu login terrapod.example.com ``` -This opens a browser window, authenticates via the configured identity provider, and stores the token automatically. - --- ## Creating Your First Workspace @@ -98,7 +97,7 @@ This opens a browser window, authenticates via the configured identity provider, ### Via the API ```zsh -curl -s -X POST https://terrapod.local/api/v2/organizations/default/workspaces \ +curl -s -X POST https://terrapod.example.com/api/v2/organizations/default/workspaces \ -H "Authorization: Bearer $TERRAPOD_TOKEN" \ -H "Content-Type: application/vnd.api+json" \ -d '{ @@ -134,7 +133,7 @@ Create a configuration that uses Terrapod as its backend: # main.tf terraform { cloud { - hostname = "terrapod.local" + hostname = "terrapod.example.com" organization = "default" workspaces { @@ -165,12 +164,14 @@ If you'd rather select a workspace at run time (typical when a single repo backs ```hcl workspaces { - tags = ["network"] # any workspace whose labels include "network" - # or, for an exact match: - tags = { repo = "aws-infra" } + tags = ["network"] # any workspace with the label key "network" + # or, for an exact key:value match: + tags = ["repo:aws-infra"] # workspaces whose `repo` label is `aws-infra` } ``` +> OpenTofu only accepts the **set-of-strings** `tags` form, so use the `key:value` string above for an exact match. The Terraform 1.10+ **map** form (`tags = { repo = "aws-infra" }`) is rejected by `tofu` at config validation with `set of string required`. + Then pick a specific workspace per invocation: ```zsh @@ -182,7 +183,7 @@ tofu workspace select network-staging If you have not already run `tofu login`: ```zsh -tofu login terrapod.local +tofu login terrapod.example.com ``` Then initialize: @@ -239,7 +240,7 @@ Agent workspaces support two workflows depending on whether VCS is connected: 1. Set the workspace execution mode to `agent`: ```zsh -curl -X PATCH https://terrapod.local/api/v2/workspaces/ws-{id} \ +curl -X PATCH https://terrapod.example.com/api/v2/workspaces/ws-{id} \ -H "Authorization: Bearer $TERRAPOD_TOKEN" \ -H "Content-Type: application/vnd.api+json" \ -d '{ @@ -256,7 +257,7 @@ curl -X PATCH https://terrapod.local/api/v2/workspaces/ws-{id} \ ```zsh # Create configuration version -CV_RESPONSE=$(curl -s -X POST https://terrapod.local/api/v2/workspaces/ws-{id}/configuration-versions \ +CV_RESPONSE=$(curl -s -X POST https://terrapod.example.com/api/v2/workspaces/ws-{id}/configuration-versions \ -H "Authorization: Bearer $TERRAPOD_TOKEN" \ -H "Content-Type: application/vnd.api+json" \ -d '{"data": {"type": "configuration-versions", "attributes": {"auto-queue-runs": true}}}') @@ -274,7 +275,7 @@ curl -X PUT "$UPLOAD_URL" \ 3. A run is automatically queued (plan-only). Monitor it: ```zsh -curl -s https://terrapod.local/api/v2/workspaces/ws-{id}/runs \ +curl -s https://terrapod.example.com/api/v2/workspaces/ws-{id}/runs \ -H "Authorization: Bearer $TERRAPOD_TOKEN" | jq '.data[0].attributes.status' ``` @@ -289,7 +290,7 @@ Variables can be set per-workspace via the API or web UI. ### Terraform Variables ```zsh -curl -X POST https://terrapod.local/api/v2/workspaces/ws-{id}/vars \ +curl -X POST https://terrapod.example.com/api/v2/workspaces/ws-{id}/vars \ -H "Authorization: Bearer $TERRAPOD_TOKEN" \ -H "Content-Type: application/vnd.api+json" \ -d '{ @@ -309,7 +310,7 @@ curl -X POST https://terrapod.local/api/v2/workspaces/ws-{id}/vars \ ### Environment Variables ```zsh -curl -X POST https://terrapod.local/api/v2/workspaces/ws-{id}/vars \ +curl -X POST https://terrapod.example.com/api/v2/workspaces/ws-{id}/vars \ -H "Authorization: Bearer $TERRAPOD_TOKEN" \ -H "Content-Type: application/vnd.api+json" \ -d '{ @@ -330,7 +331,7 @@ curl -X POST https://terrapod.local/api/v2/workspaces/ws-{id}/vars \ Set `"sensitive": true` for secrets. The value is protected by database encryption-at-rest and never returned in API responses: ```zsh -curl -X POST https://terrapod.local/api/v2/workspaces/ws-{id}/vars \ +curl -X POST https://terrapod.example.com/api/v2/workspaces/ws-{id}/vars \ -H "Authorization: Bearer $TERRAPOD_TOKEN" \ -H "Content-Type: application/vnd.api+json" \ -d '{ @@ -363,7 +364,7 @@ For managing variables across multiple workspaces, see variable sets (admin-only 1. Create the module: ```zsh -curl -X POST https://terrapod.local/api/terrapod/v1/registry-modules \ +curl -X POST https://terrapod.example.com/api/terrapod/v1/registry-modules \ -H "Authorization: Bearer $TERRAPOD_TOKEN" \ -H "Content-Type: application/vnd.api+json" \ -d '{ @@ -380,7 +381,7 @@ curl -X POST https://terrapod.local/api/terrapod/v1/registry-modules \ 2. Create a version: ```zsh -curl -X POST https://terrapod.local/api/terrapod/v1/registry-modules/private/default/vpc/aws/versions \ +curl -X POST https://terrapod.example.com/api/terrapod/v1/registry-modules/private/default/vpc/aws/versions \ -H "Authorization: Bearer $TERRAPOD_TOKEN" \ -H "Content-Type: application/vnd.api+json" \ -d '{ @@ -399,7 +400,7 @@ curl -X POST https://terrapod.local/api/terrapod/v1/registry-modules/private/def ```hcl module "vpc" { - source = "terrapod.local/default/vpc/aws" + source = "terrapod.example.com/default/vpc/aws" version = "1.0.0" } ``` @@ -419,25 +420,19 @@ See [vcs-integration.md](vcs-integration.md) for detailed setup instructions. --- -## Running Tests and Linting - -All tests run in Docker -- no local Python or Node.js install needed: +## Uninstall ```zsh -make test # Run pytest with LocalStack for S3 testing -make lint # Run ruff check + format check + mypy -make test-down # Tear down test containers +helm uninstall terrapod --namespace terrapod ``` ---- - -## Stopping the Dev Environment - -```zsh -make dev-down -``` +PersistentVolumeClaims (filesystem storage, ephemeral runner scratch) are not +removed by `helm uninstall` — delete them explicitly if you want to reclaim the +storage, and drop the namespace with `kubectl delete namespace terrapod`. -This stops all Tilt-managed resources and cleans up the namespace. +> Building and testing Terrapod from source — Tilt, live-reload, `make test` — +> is covered in [Local Development](local-development.md). You don't need any of +> that to run Terrapod. --- diff --git a/docs/index.md b/docs/index.md index 26be7035..771fd8ff 100644 --- a/docs/index.md +++ b/docs/index.md @@ -41,14 +41,21 @@ Terrapod is **not** a fork of Terraform or OpenTofu. It orchestrates them. ## Quick Start +Deploy onto any Kubernetes cluster (or a single-node [k3s](https://k3s.io/) VM) with the Helm chart: + ```zsh -# Prerequisites: Docker, local K8s cluster, Tilt, mkcert -brew install mkcert && mkcert -install -sudo sh -c 'echo "127.0.0.1 terrapod.local" >> /etc/hosts' -make dev +helm install terrapod oci://ghcr.io/mattrobinsonsre/terrapod \ + --namespace terrapod --create-namespace \ + --set ingress.enabled=true \ + --set ingress.hostname="terrapod.example.com" \ + --set ingress.className=traefik \ + --set postgresql.url="postgresql+asyncpg://terrapod:PASSWORD@PGHOST:5432/terrapod" \ + --set redis.url="redis://REDISHOST:6379" \ + --set bootstrap.adminEmail="admin@example.com" \ + --set bootstrap.adminPassword="change-me-now" ``` -Open `https://terrapod.local` in your browser. See [Getting Started](getting-started.md) for the full walkthrough. +PostgreSQL and Redis are external (not bundled). See [Getting Started](getting-started.md) for the full walkthrough, or [Local Development](local-development.md) if you want to run Terrapod from source. --- @@ -80,7 +87,8 @@ See [Architecture](architecture.md) for the full breakdown. | Guide | Description | |---|---| -| [Getting Started](getting-started.md) | Local development setup, first workspace, first plan/apply | +| [Getting Started](getting-started.md) | Deploy the Helm chart on Kubernetes (or k3s), first workspace, first plan/apply | +| [Local Development](local-development.md) | Run Terrapod from source with Tilt (contributors only) | | [Architecture](architecture.md) | System components, storage, runners, auth flows | | [Authentication](authentication.md) | Local auth, OIDC, SAML, terraform login, API tokens, scoped service tokens (bound/detached) + offboarding idle guard | | [RBAC](rbac.md) | Permission model, label-based access control, custom roles | diff --git a/docs/local-development.md b/docs/local-development.md new file mode 100644 index 00000000..33d4d3ff --- /dev/null +++ b/docs/local-development.md @@ -0,0 +1,77 @@ +# Local Development + +This page is for **contributors** who want to run Terrapod from source on a +local Kubernetes cluster. If you just want to **deploy and use** Terrapod, see +[Getting Started](getting-started.md) — you do not need any of this. + +Terrapod's local dev stack is driven by [Tilt](https://tilt.dev/): it builds the +images, deploys the Helm chart against a local cluster, runs the database +migrations, bootstraps an admin user, and live-syncs source changes into the +running pods. + +## Prerequisites + +| Tool | Purpose | Install | +|---|---|---| +| Docker | Container runtime | [docker.com](https://www.docker.com/) | +| A local Kubernetes cluster | Run the stack | Rancher Desktop, Docker Desktop (K8s enabled), minikube, kind, OrbStack, or colima | +| Tilt | Local dev orchestration | `brew install tilt` | +| mkcert | Local TLS certificate | `brew install mkcert` | +| tofu (recommended) or terraform | Exercise the CLI flows | [opentofu.org](https://opentofu.org/) | + +The local stack is deliberately pinned to its own namespace (`terrapod`), Tilt +port (`10352`), and hostname (`terrapod.local`) so it can run alongside other +local projects. + +## Setup + +### 1. Local CA + hosts entry + +```zsh +brew install mkcert && mkcert -install +sudo sh -c 'echo "127.0.0.1 terrapod.local" >> /etc/hosts' +``` + +`mkcert -install` adds a local CA to your system trust store so +`https://terrapod.local` is trusted by your browser and the terraform/tofu CLI. + +### 2. Start the stack + +```zsh +make dev # runs `tilt up --port 10352` +``` + +This creates the `terrapod` namespace, generates the TLS cert, deploys +PostgreSQL and Redis in-cluster, builds the API/web images, runs the Alembic +migrations, bootstraps the admin user, and deploys the API, web UI, and runner +listener. Watch progress in the Tilt UI at . + +Tear it down with `make dev-down`. + +### 3. Access + +Open . The bootstrap job creates an admin user — the +default local credentials are `admin` / `admin` (set in +`helm/terrapod/values-local.yaml`). Check the `terrapod-bootstrap-1` resource in +the Tilt UI to confirm it ran. + +From there the workflow is the same as a real deployment — see +[Getting Started](getting-started.md) for creating a workspace and running your +first plan/apply (substitute `terrapod.local` for the hostname). + +## Day-to-day + +- **Live reload** — `tilt up` live-syncs `services/terrapod` and `web/src` into + the running pods; the API auto-reloads (uvicorn) and the web hot-reloads + (Next.js). If a change doesn't take, force it: + `tilt trigger --port 10352 terrapod-api` (or `terrapod-web`). +- **Migrations** — after adding an Alembic revision, trigger the migration job: + `tilt trigger --port 10352 terrapod-migrations-1`. +- **Tests & lint** (containerised — no local Python needed): + `make test`, `make lint`. Tear down test containers with `make test-down`. +- **Never** use `kubectl cp`, `kubectl apply`, or `docker build` against + Tilt-managed resources — it corrupts Tilt's state. Let Tilt manage its + resources; use `tilt trigger` to force a rebuild. + +For the architecture and conventions behind the stack, see +[Architecture](architecture.md) and the repository `CLAUDE.md`. diff --git a/helm/terrapod/values.yaml b/helm/terrapod/values.yaml index 3a0a10b4..c83e8224 100644 --- a/helm/terrapod/values.yaml +++ b/helm/terrapod/values.yaml @@ -546,8 +546,11 @@ api: annotations: {} # ── Web UI (Next.js) ────────────────────────────────────────────────────────── +# The web frontend is the BFF (Backend-For-Frontend): all browser, CLI, and +# listener traffic enters through it and is proxied to the API. The Ingress +# routes here, so it is enabled by default alongside the API and listener. web: - enabled: false + enabled: true replicas: 2 diff --git a/mkdocs.yml b/mkdocs.yml index 788ec07e..dd5cfa32 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -50,6 +50,7 @@ markdown_extensions: nav: - Home: index.md - Getting Started: getting-started.md + - Local Development: local-development.md - Architecture: architecture.md - Features: - VCS Integration: vcs-integration.md