diff --git a/.editorconfig b/.editorconfig index 7a8a801..068b09a 100644 --- a/.editorconfig +++ b/.editorconfig @@ -16,10 +16,6 @@ indent_size = 2 indent_style = space indent_size = 2 -[config.ru] -indent_style = tab -indent_size = 4 - [*.rb] indent_style = space indent_size = 2 diff --git a/apiserver/README.md b/apiserver/README.md index 6cf9ca5..2d528dd 100644 --- a/apiserver/README.md +++ b/apiserver/README.md @@ -1,17 +1,58 @@ # API server -The API server is a service that allows performing limited management operations on the infrastructure. It mainly exists to securely allow the Server Edition's CI to tell Caddy about the fact that new packages have been deployed. +The API server is a small service that exposes a couple of privileged management +operations to the project's GitHub Actions workflows — primarily so that CI can +trigger restarts of Caddy and self-upgrades of the API server itself, without +needing SSH access to the backend host. -Presently, there is only one endpoint: `POST /admin/restart_web_server`. This endpoint initiates restarting of Caddy, but does not wait for it to finish. +It runs as a systemd-managed Puma process (`apiserver.service`) on the Hetzner +backend host, listening on a Unix socket at `/run/apiserver/server.sock`. Caddy +reverse-proxies the `/admin/*` paths from `apt.fullstaqruby.org` and +`yum.fullstaqruby.org` to that socket — there is no dedicated `apiserver.*` +hostname. -## Calling the production instance +## Endpoints -The production instance is deployed at https://apiserver-f7awo4fcoa-uk.a.run.app/. To call it, you need to include your Google Cloud identity token in the Authorization header. +- `GET /` — health check, returns `ok`. +- `POST /admin/upgrade_apiserver` — kicks off `apiserver-deployer` (which fetches + the latest API server release from GitHub and activates it), then restarts + `apiserver` itself. Callable only from `fullstaq-ruby/infra`'s `deploy` + GitHub Actions environment. +- `POST /admin/restart_web_server` — restarts Caddy. Callable only from + `fullstaq-ruby/server-edition`'s `deploy` GitHub Actions environment. -```bash -curl -v -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://apiserver-f7awo4fcoa-uk.a.run.app/ -``` +## Authentication + +The API server authenticates callers using **GitHub Actions OIDC**. Every +request must carry an `Authorization: Bearer ` header where the token +is an ID token minted by GitHub's OIDC provider with audience claim +`backend.fullstaqruby.org`. The server verifies the JWT signature against +GitHub's JWKS and rejects the request unless the token's `repository`, `sub`, +`runner_environment`, and `environment` claims match the calling repo's +expected `deploy` environment for that endpoint. + +Because the audience and claim shape are tied to GitHub-hosted runners, the +endpoints are not callable directly by a human or from a local machine. ## Continuous deployment -New API server code changes, when pushed to main, are automatically deployed by the Infrastructure project's CI. +API server builds and deployments are driven by two workflows in +`.github/workflows/`: + +- **Test and build** (`test-and-build.yml`) runs on every push. When a push + touches `apiserver/**`, its `apiserver-build` job packages the API server + into a tarball and uploads it as a workflow artifact. +- **Deploy** (`deploy.yml`) runs after a successful Test and build run on + `main`. It tags the commit, publishes a GitHub release with the tarball, and + calls `POST https://apt.fullstaqruby.org/admin/upgrade_apiserver` with a + freshly minted OIDC token. + +On the host, `apiserver-deployer` (a oneshot systemd unit running +`/usr/local/bin/apiserver-deployer`) handles the actual install: it fetches +the latest release metadata from the GitHub API, downloads the asset matching +the host's distribution and architecture, extracts it into +`/opt/apiserver/versions/---`, installs any runtime +dependencies declared in `dpkg-dependencies.txt`, prunes all but the last five +versions, and atomically swaps the `/opt/apiserver/versions/latest` symlink. +The API server's working directory points at that symlink, so the subsequent +`systemctl restart apiserver` brings the new version online. diff --git a/docs/deploy.md b/docs/deploy.md index b3bd8ec..223eaf2 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -56,20 +56,12 @@ This guide explains how to deploy infrastructure updates. This guide is not for cd .. ~~~ - 4. Get the credentials for the Kubernetes cluster: + 4. Apply Ansible to the backend VM: ~~~bash - gcloud container clusters get-credentials fullstaq-ruby-autopilot --configuration fullstaq-ruby --region us-east4 + cd ansible + ansible-playbook -i hosts.ini -v main.yml + cd .. ~~~ - 5. Set the default namespace: - - ~~~bash - kubectl config set-context --current --namespace=fullstaq-ruby - ~~~ - - 6. Apply the Kustomization: - - ~~~bash - kubectl apply --context=gke_fullstaq-ruby_us-east4_fullstaq-ruby-autopilot -k ../kubernetes - ~~~ +> The API server itself is not deployed by this playbook. Code changes under `apiserver/` are built and deployed by the Github Actions workflows in `.github/workflows/` when pushed to `main`. They package a tarball, attach it to a GitHub Release, and trigger `POST /admin/upgrade_apiserver` on the live host. diff --git a/docs/infrastructure-as-code.md b/docs/infrastructure-as-code.md index 3ccaf76..97d21b1 100644 --- a/docs/infrastructure-as-code.md +++ b/docs/infrastructure-as-code.md @@ -3,19 +3,19 @@ We define as much infrastructure as possible in the form of code, using: * [Terraform](https://terraform.io) - * Kubernetes YAML, managed with [Kustomize](https://kustomize.io/) + * [Ansible](https://www.ansible.com/) * Github Actions The infrastructure-as-code is stored in the following directories: - * `terraform/` — Infrastructure administered by [Infra Maintainers](roles.md), except for resources inside Kubernetes. Most of the infrastructure is defined here. + * `terraform/` — Infrastructure administered by [Infra Maintainers](roles.md). Most of the cloud-side infrastructure is defined here. - * `terraform-hisec/` — Infrastructure administered by [Infra Owners](roles.md). This covers for example resources in the `fullstaq-ruby-hisec` Google Cloud project. + * `terraform-hisec/` — Infrastructure administered by [Infra Owners](roles.md). This covers for example sensitive resources such as the GPG signing key in Azure Key Vault, and the high-security Terraform state backend. - Because we don't expect the infrastructure in this directory to change very often, we've chosen — for security reasons — not to run Terraform in a CI/CD pipeline. This way we don't have to worry about the security of the CI/CD pipeline's service account. Instead, an [Infra Owner](roles.md) runs Terraform manually, using that person's personal Google Cloud credentials. + Because we don't expect the infrastructure in this directory to change very often, we've chosen — for security reasons — not to run Terraform in a CI/CD pipeline. This way we don't have to worry about the security of any CI/CD pipeline credentials. Instead, an [Infra Owner](roles.md) runs Terraform manually, using their personal cloud credentials. - * `kubernetes/` — Kubernetes resources administered by [Infra Maintainers](roles.md). + * `ansible/` — Configuration of the backend VM (Caddy, the API server, Prometheus, and OS hardening). Administered by [Infra Maintainers](roles.md) and applied manually; see [Deployment guide](deploy.md). - * `.github/workflows/apiserver.yml` — Deploys the API server. + * `.github/workflows/` — Github Actions workflows that together test the infrastructure code, and build and deploy the API server. Note that not all infrastructure can, or (for security reasons) should, be managed via code. Learn more at [Infrastructure bootstrapping](infrastructure-bootstrapping.md). diff --git a/docs/infrastructure-bootstrapping.md b/docs/infrastructure-bootstrapping.md index c8c8510..c1805ab 100644 --- a/docs/infrastructure-bootstrapping.md +++ b/docs/infrastructure-bootstrapping.md @@ -1,6 +1,6 @@ # Infrastructure bootstrapping -We try to codify infrastructure as much as possible using Terraform and Kubernetes YAML. However: +We try to codify infrastructure as much as possible using Terraform and Ansible. However: - Not everything _can_ be automated. For example, we need to setup Azure Blob Storage for storing Terraform state, before we can use Terraform. - Not everything _should_ be automated. For example, the `fullstaq-ruby-hisec` project contains such sensitive data, that giving access to CI/CD systems would pose a security risk. @@ -148,6 +148,8 @@ Register the domain `fullstaqruby.org`. Configure it to use the Azure DNS zone. ## Step 11: Run initial Ansible +The playbook deploys the API server from the latest `apiserver-*` release of the [fullstaq-ruby/infra](https://github.com/fullstaq-ruby/infra/releases) repo, so at least one such release must exist. + Make sure the Azure CLI is logged in, then: ```bash @@ -156,7 +158,21 @@ ansible-playbook -i hosts.ini -v main.yml cd .. ``` -## Step 12: Populate Github Actions secrets and variables +## Step 12: Deploy the API server + +The Ansible run in step 11 installs `apiserver-deployer`, which downloads and activates the latest API server release from the [fullstaq-ruby/infra](https://github.com/fullstaq-ruby/infra/releases) repo. Verify on the VM that the API server is running: + +```bash +sudo systemctl status apiserver-deployer apiserver +``` + +If it isn't, install and start the latest release manually: + +```bash +sudo systemctl restart apiserver-deployer apiserver +``` + +## Step 13: Populate Github Actions secrets and variables In the [fullstaq-ruby/server-edition](https://github.com/fullstaq-ruby/server-edition/settings/secrets) repo, create the following environments: @@ -186,6 +202,6 @@ Create these environment-specific variables: - `AZURE_CLIENT_ID` ('test' environment): fetch using `pushd terraform-hisec >/dev/null && terraform show -json | jq -r '.values.root_module.resources[] | select(.address == "azuread_application.server-edition-github-ci-test") | .values.application_id'; popd >/dev/null` - `AZURE_CLIENT_ID` ('deploy' environment): fetch using `pushd terraform-hisec >/dev/null && terraform show -json | jq -r '.values.root_module.resources[] | select(.address == "azuread_application.server-edition-github-ci-deploy") | .values.application_id'; popd >/dev/null` -## Step 13: Onboard everybody +## Step 14: Onboard everybody Onboard everybody in the [members list](members.md) according to the [onboarding instructions](onboarding.md). diff --git a/docs/infrastructure-overview.md b/docs/infrastructure-overview.md index f95bab9..651be15 100644 --- a/docs/infrastructure-overview.md +++ b/docs/infrastructure-overview.md @@ -4,79 +4,75 @@ The following diagram shows which infrastructure components exist and how they r ![Infrastructure overview diagram](infrastructure-overview.drawio.svg) -## Google Cloud projects +## Google Cloud project -- Administrated by role: Infra Owners - -All Google Cloud resources are contained in two projects: +- Administered by role: Infra Owners (project resource); Infra Maintainers (resources within) -- `fullstaq-ruby` for normal resources. Infra Maintainers have full access to all resources _inside_ this project. -- `fullstaq-ruby-hisec` for especially sensitive resources. Only Infra Owners have access to the resources inside this project. +All Google Cloud resources live in a single project, `fsruby-server-edition2` (display name "Fullstaq Ruby Server Edition"). The `google_project` resource itself is provisioned in `terraform-hisec/gcloud_project.tf` so that creating/deleting the project requires Infra Owner access, but resources _inside_ the project (buckets, IAM, Workload Identity Federation) are managed in `terraform/` by Infra Maintainers. -## Google Cloud service account for Server Edition CI/CD +## CI/CD authentication - Administered by role: Infra Maintainers -The `fullstaq-ruby` Google Cloud project has a service account. It's used by the Server Edition's CI/CD system. This service account: - -- Has full access to the Server Edition CI artifacts store. -- Has full access to the container registry, in order to access and store [build environment images](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/build-environments.md). -- Has full access to the APT and YUM repo buckets, in order to publish new packages. +CI/CD authenticates using short-lived GitHub-issued OIDC tokens — there are no long-lived service-account keys. -## Google Cloud service account for Infrastructure CI/CD - -- Administered by role: Infra Maintainers - -The `fullstaq-ruby` Google Cloud project has a service account. It's used by the Infrastructure repository's CI/CD system. This service account: - -- Has full access to the container registry, in order to publish new API server images. -- Can deploy new versions of the API server. +- **`fullstaq-ruby/server-edition` → Google Cloud** uses [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) (defined in `terraform/gcloud_auth.tf`). Two pools (`github-ci-test`, `github-ci-deploy`) gate access by GitHub repository owner and Actions environment. Through these pools, server-edition CI jobs gain write access to the APT/YUM repo buckets and the GCP CI artifacts bucket — see `terraform/repo_buckets.tf` and `terraform/ci_storage.tf`. The CI cache lives in Azure (see below), not on GCP. +- **`fullstaq-ruby/server-edition` → Azure** uses [Federated Identity Credentials](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation) on Entra ID applications (defined in `terraform-hisec/`). These authenticate workflows that read or write Azure Blob Storage (the CI artifacts and CI cache containers) and Azure Key Vault (the GPG signing key). +- **`fullstaq-ruby/infra` → API server** uses a GitHub-issued OIDC JWT (audience `backend.fullstaqruby.org`) sent as a bearer token to `POST /admin/upgrade_apiserver`. The infra repo's workflows do **not** authenticate to GCP or Azure APIs — the rollout mechanism is entirely on the VM (see [API server](#api-server)). The same JWT mechanism is used by `server-edition` to call `/admin/restart_web_server` after a publish. ## API server - Administered by role: Infra Maintainers -The API server is a service that allows performing limited management operations on the infrastructure. It mainly exists to securely allow the Server Edition's CI to tell Nginx about the fact that new packages have been deployed. It's hosted on Google Cloud Run. +The API server is a small Sinatra service that exposes a narrow set of `/admin/*` endpoints used by CI/CD: notably `restart_web_server` (called by Server Edition's CI after a publish so Caddy picks up the new repo version) and `upgrade_apiserver` (called by this repo's CI to roll out a new API server build). All endpoints authenticate the caller via GitHub Actions OIDC JWTs. -The API server's source code lives in the Infrastructure repository, and is deployed by that repository's CI/CD system. +The API server runs on the backend VM (see [VM (Hetzner)](#vm-hetzner)) under systemd, with Puma listening on a Unix socket that Caddy proxies to. It is provisioned by Ansible (`ansible/tasks/apiserver.yml`, unit file `ansible/files/apiserver.service`). -The API server is not defined in Terraform, but is defined in the Infrastructure's CI/CD system in the form of a `gcloud` call. +Releases are built and published by the workflows in `.github/workflows/`: the "Test and build" workflow packages the source as a tarball, and the "Deploy" workflow attaches it to a GitHub Release tagged `apiserver-N` and then calls `POST /admin/upgrade_apiserver`. That request triggers a sibling `apiserver-deployer.service` (provisioned alongside), which fetches the release tarball into `/opt/apiserver/versions/`, after which the API server is restarted. The deployer exists so that the API server can be replaced without leaving an unreachable gap, and for security: the API server itself runs as a separate user without the ability to modify its own code. -## Nginx web server +## Caddy web server - Administered by role: Infra Maintainers -We run one Nginx instance, which serves these virtual hosts: +Caddy runs on the backend VM (see [VM (Hetzner)](#vm-hetzner)) and serves two virtual hosts: `apt.fullstaqruby.org` and `yum.fullstaqruby.org` (see `ansible/files/Caddyfile`). Each vhost has the same shape: + +- `/admin/*` is reverse-proxied over a local Unix socket to the API server. +- All other paths redirect to the corresponding GCS repo bucket, with the current published version baked into the redirect target. + +There is no separate `backend.fullstaqruby.org` virtual host — the hostname exists as a DNS record (and as the OIDC token audience claim) but is not terminated by Caddy. CI/CD calls the `/admin/*` endpoints via `https://apt.fullstaqruby.org/admin/*`. -- `apt.fullstaqruby.org` — redirects everything to the APT repo bucket. -- `yum.fullstaqruby.org` — redirects everything to the YUM repo bucket. +The redirect target's version is read from each bucket's `latest_version.txt` once at startup (via the `query-latest-repo-versions.rb` `ExecStartPre` in the systemd unit). Caddy must therefore be restarted after a publish so it picks up the new version — that restart is what the API server's `restart_web_server` endpoint exists to trigger. See the [APT/YUM repo design](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/apt-yum-repo.md) for how `latest_version.txt` fits into the repo layout. -Nginx's only purpose is to redirect traffic. Users interact primarily with `{apt,yum}.fullstaqruby.org` instead of with the APT and YUM repo buckets directly. This decouples users from our APT and YUM repos are actually hosted, allowing us to change the hosting mechanism without breaking users' URLs. +Users interact with `{apt,yum}.fullstaqruby.org` rather than the buckets directly. This decouples users from where the repos are actually hosted, allowing us to change the hosting mechanism without breaking users' repository URLs. > Historic note: our APT and YUM repos used to be hosted on Bintray. But Bintray shut down on March 1 2021. The HTTP redirection mechanism allowed us to move away from Bintray with minimal downtime, and without breaking users' repository URLs. -TLS is configured through the Ingress, using [Google-managed TLS certificates](https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs). +TLS certificates are obtained via the ACME DNS-01 challenge against Azure DNS. Caddy authenticates to Azure DNS using a service principal whose credentials are stored in Azure Key Vault and injected into the Caddy systemd unit via an environment file. -## Kubernetes cluster +## VM (Hetzner) - Administered by role: Infra Maintainers -The Kubernetes cluster runs our Nginx web server. This cluster is in Autopilot mode. +A single Ubuntu (≥ 24.04) VPS hosted at Hetzner runs Caddy and the API server. Its forward DNS records (`backend.fullstaqruby.org`, `apt.fullstaqruby.org`, `yum.fullstaqruby.org`) are managed in `terraform/dns.tf`; its static IPs are referenced from `terraform/variables.tf`. The PTR record (`backend.fullstaqruby.org`) is set manually at the Hetzner provider during bootstrapping (see [bootstrapping](infrastructure-bootstrapping.md) Step 7), not via Terraform. -## DNS, static IPs, Ingresses +The VM is configured entirely by Ansible (`ansible/main.yml`), covering both OS hardening and the services it runs. + +## DNS - Administered by role: Infra Maintainers -All DNS entries for `fullstaqruby.org` are managed through Google Cloud DNS, inside the `fullstaq-ruby` project. +All DNS records for `fullstaqruby.org` are managed in Azure DNS (`terraform/dns.tf`). + +- `fullstaqruby.org` and `www.fullstaqruby.org` point to GitHub Pages, where we host the [website](https://github.com/fullstaq-ruby/website). +- `backend.fullstaqruby.org`, `apt.fullstaqruby.org`, and `yum.fullstaqruby.org` all resolve to the backend VM's IP addresses. -- `fullstaqruby.org` points to Github Pages, where we host the [website](https://github.com/fullstaq-ruby/website). -- `{apt,yum}.fullstaqruby.org` point to Nginx's two virtual hosts, via two different static IPs and two different Kubernetes Ingresses. +`apt.fullstaqruby.org` and `yum.fullstaqruby.org` are also delegated as their own Azure DNS zones so that Caddy's ACME service principal has DNS-write access scoped only to those subdomains, not the parent zone. ## Domain name - Administrated by role: Infra Owners -The `fullstaqruby.org` domain is registered at [TransIP](https://www.transip.nl/). It's registered using [Fullstaq](https://www.fullstaq.com)'s account. The DNS zone is not managed at TransIP, but at Google Cloud DNS. +The `fullstaqruby.org` domain is registered at [TransIP](https://www.transip.nl/). It's registered using [Fullstaq](https://www.fullstaq.com)'s account. The DNS zone is not managed at TransIP, but at Azure DNS (see [DNS](#dns)). ## Github CI bot account @@ -88,13 +84,13 @@ This Github bot account is used by the Server Edition's CI/CD system. The accoun - Administered by role: Infra Maintainers -The Terraform state for normal infrastructure is stored in a Google Cloud Storage bucket, inside the `fullstaq-ruby` project. +The Terraform state for normal infrastructure is stored in an Azure Blob Storage container in the `fullstaq-ruby-terraform` resource group (see `terraform/backend.tf`). ## Terraform state (hisec) - Administered by role: Infra Owners -The Terraform state for sensitive infrastructure is stored in a Google Cloud Storage bucket, inside the `fullstaq-ruby-hisec` project. +The Terraform state for sensitive infrastructure is stored in an Azure Blob Storage container in the `fullstaq-ruby-terraform-hisec` resource group (see `terraform-hisec/backend.tf`). ## Server Edition APT & YUM repo buckets @@ -102,31 +98,25 @@ The Terraform state for sensitive infrastructure is stored in a Google Cloud Sto The Server Edition's APT and YUM repositories are stored inside these buckets. These buckets are publicly readable. -Users don't access these buckets directly. Instead, they access `apt.fullstaqruby.org` and `yum.fullstaqruby.org` (served by the Nginx web servers), which redirect to these buckets. - -## Container registry - -- Administered by role: Infra Maintainers - -The `fullstaq-ruby` project has a container registry. This registry has two uses: - -- It's used by developers' CI/CD systems, for example to store [build environment images](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/build-environments.md). -- It's used to store the API server's images. +Users don't access these buckets directly. Instead, they access `apt.fullstaqruby.org` and `yum.fullstaqruby.org` (served by Caddy on the backend VM), which redirect to these buckets. ## Server Edition CI artifacts store - Administered by role: Infra Maintainers -The Server Edition's CI/CD system stores artifacts in this bucket, for the purpose of implementing [resumption](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/ci-cd-resumption.md). Objects in this bucket only live for 30 days. +The Server Edition's CI/CD system stores artifacts for [CI/CD resumption](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/ci-cd-resumption.md) in two buckets (see `terraform/ci_storage.tf`): + +- A **GCS bucket** (`fsruby-server-edition-ci-artifacts`) — publicly readable; the `test` environment writes via Workload Identity Federation (WIF), the `deploy` environment reads. Objects expire automatically. +- An **Azure Blob container** (`server-edition-ci-artifacts` inside the `fsruby2seredci1` storage account) — private. Provisioned for a future migration of CI artifacts off GCS, but **not currently in use**; server-edition CI still writes artifacts only to the GCS bucket above. The cache container in the same storage account *is* actively used (see "Server Edition CI cache store" below). ## Server Edition CI cache store - Administered by role: Infra Maintainers -The Server Edition's CI/CD system stores caches in this bucket. Objects are automatically deleted 90 days after last access. +The Server Edition's CI/CD system stores caches in an **Azure Blob container** (`server-edition-ci-cache` inside the same `fsruby2seredci1` storage account; see `terraform/ci_storage.tf`). Only the `test` environment writes; objects expire automatically based on last access time. There is no equivalent cache bucket on GCP. ## GPG private key - Administered by role: Infra Owners, Infra Maintainers -The GPG private key is used to sign APT and YUM repositories. We store the canonical copy in Secrets Manager in the `fullstaq-ruby-hisec` Google Cloud project. We store a secondary copy in the Secret Manager in the `fullstaq-ruby` Google Cloud project. +The GPG private key is used to sign APT and YUM repositories. It is stored in the Azure Key Vault for Infra Owners, `fsruby2infraowners` (see `terraform-hisec/key_vault.tf`). GitHub Actions in the `test` and `deploy` environments are granted read access via GitHub OIDC (see [CI/CD authentication](#cicd-authentication)).