From 05f89668d06923c30fdf869ad78d43793a3040f3 Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 28 Apr 2026 21:56:42 -0500 Subject: [PATCH 1/8] docs: refresh infra docs for post-Hetzner architecture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #55. Brings docs/ in line with the post-July-2024 architecture (single Hetzner VM running Caddy + Sinatra/Puma API server + Prometheus, provisioned by Ansible), replacing references to the previous GKE Autopilot + Nginx Ingress + Cloud Run apiserver setup. Files changed: - docs/infrastructure-overview.md — rewritten section by section. Every claim is grounded in current IaC. The two GCP-service-account sections are folded into a single "CI/CD authentication" section that splits per-caller: server-edition uses GCP WIF (APT/YUM repo buckets + GCS CI artifacts bucket) and Azure Federated Identity Credentials (Azure Blob CI artifacts + CI cache + Key Vault GPG key); infra repo's apiserver workflow only mints a GitHub OIDC JWT (audience backend.fullstaqruby.org) and POSTs to /admin/upgrade_apiserver — it does not authenticate to GCP or Azure APIs. The Caddy section is corrected: there is no backend.fullstaqruby.org vhost; both apt. and yum. vhosts handle /admin/* via reverse_proxy to the apiserver Unix socket. The "Google Cloud projects" claim of two projects is corrected — there is one project, fsruby-server-edition2, provisioned by terraform-hisec/gcloud_project.tf and populated by terraform/; the hisec/non-hisec separation lives at the Terraform-state and access-group layer. Container registry section dropped (no registry resources are managed in this repo). Key Vault name uses the templated form ${var.key_vault_prefix}infraowners (currently fsruby2infraowners). CI artifacts/cache split is now explicit (artifacts dual-cloud, cache Azure-only). VM section distinguishes Terraform-managed forward DNS from the manually-set Hetzner PTR record. - docs/infrastructure-overview.drawio.svg — deleted. Replaced by an inline Mermaid diagram in infrastructure-overview.md so future diagram changes are reviewable as text diffs. - docs/editing-diagrams.md — deleted (no longer needed without the drawio round-trip). - docs/deploy.md — replaces the gcloud-clusters/kubectl steps with a single ansible-playbook step matching bootstrapping Step 11. Adds a callout that apiserver code changes deploy via the GitHub Actions workflow. - docs/infrastructure-as-code.md — drops Kustomize and the kubernetes/ directory bullet; adds Ansible to the tool list and an ansible/ directory bullet. - docs/infrastructure-bootstrapping.md — intro updated to mention Terraform + Ansible (not Kubernetes/Kustomize); the rest of the file already reflected the post-migration setup. - docs/pull_request_template.md — diagram-update checkbox now points to the Mermaid block instead of the deleted drawio file. - README.md — drops the link to the deleted editing-diagrams.md. - .editorconfig — removes the duplicate [config.ru] block (the tab/4 one); only the correct space/2 rule remains. Note: the "Github CI bot account" section is kept as-is. Retiring that PAT-based bot is already tracked in #18 and is therefore out of scope here. --- .editorconfig | 4 - README.md | 1 - docs/deploy.md | 18 +- docs/editing-diagrams.md | 12 - docs/infrastructure-as-code.md | 12 +- docs/infrastructure-bootstrapping.md | 2 +- docs/infrastructure-overview.drawio.svg | 1846 ----------------------- docs/infrastructure-overview.md | 162 +- docs/pull_request_template.md | 3 +- 9 files changed, 117 insertions(+), 1943 deletions(-) delete mode 100644 docs/editing-diagrams.md delete mode 100644 docs/infrastructure-overview.drawio.svg diff --git a/.editorconfig b/.editorconfig index 7a8a801..068b09a 100644 --- a/.editorconfig +++ b/.editorconfig @@ -16,10 +16,6 @@ indent_size = 2 indent_style = space indent_size = 2 -[config.ru] -indent_style = tab -indent_size = 4 - [*.rb] indent_style = space indent_size = 2 diff --git a/README.md b/README.md index 0d8720c..dbfba64 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,6 @@ Concepts: Tasks: * [Deployment guide](docs/deploy.md) - * [Editing diagrams](docs/editing-diagrams.md) Organizational (for team members): diff --git a/docs/deploy.md b/docs/deploy.md index b3bd8ec..4c942ff 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -56,20 +56,12 @@ This guide explains how to deploy infrastructure updates. This guide is not for cd .. ~~~ - 4. Get the credentials for the Kubernetes cluster: + 4. Apply Ansible to the backend VM: ~~~bash - gcloud container clusters get-credentials fullstaq-ruby-autopilot --configuration fullstaq-ruby --region us-east4 + cd ansible + ansible-playbook -i hosts.ini -v main.yml + cd .. ~~~ - 5. Set the default namespace: - - ~~~bash - kubectl config set-context --current --namespace=fullstaq-ruby - ~~~ - - 6. Apply the Kustomization: - - ~~~bash - kubectl apply --context=gke_fullstaq-ruby_us-east4_fullstaq-ruby-autopilot -k ../kubernetes - ~~~ +> The API server itself is not deployed by this playbook. Code changes under `apiserver/` are released by the `.github/workflows/apiserver.yml` workflow, which packages a tarball, attaches it to a GitHub Release, and triggers `POST /admin/upgrade_apiserver` on the live host. diff --git a/docs/editing-diagrams.md b/docs/editing-diagrams.md deleted file mode 100644 index c032294..0000000 --- a/docs/editing-diagrams.md +++ /dev/null @@ -1,12 +0,0 @@ -# Editing diagrams - -All diagrams' sources are stored in `.drawio.svg` files — _SVG files with embedded Diagrams.net content_. They are made using [Diagrams.net](https://diagrams.net) (formerly Draw.io). - -The recommended way to edit these files locally is with [Visual Studio Code's Draw.io integration](https://marketplace.visualstudio.com/items?itemName=hediet.vscode-drawio). - -If you want to use Diagram.net's web app instead of the Visual Studio Code integration, then be sure to save it by exporting the diagram as SVG, using the following settings: - - * Zoom: 110% - * Border width: 12 - * Check: Transparent Background - * Uncheck: Include a copy of my diagram diff --git a/docs/infrastructure-as-code.md b/docs/infrastructure-as-code.md index 3ccaf76..0f3159c 100644 --- a/docs/infrastructure-as-code.md +++ b/docs/infrastructure-as-code.md @@ -3,19 +3,19 @@ We define as much infrastructure as possible in the form of code, using: * [Terraform](https://terraform.io) - * Kubernetes YAML, managed with [Kustomize](https://kustomize.io/) + * [Ansible](https://www.ansible.com/) * Github Actions The infrastructure-as-code is stored in the following directories: - * `terraform/` — Infrastructure administered by [Infra Maintainers](roles.md), except for resources inside Kubernetes. Most of the infrastructure is defined here. + * `terraform/` — Infrastructure administered by [Infra Maintainers](roles.md). Most of the cloud-side infrastructure is defined here. - * `terraform-hisec/` — Infrastructure administered by [Infra Owners](roles.md). This covers for example resources in the `fullstaq-ruby-hisec` Google Cloud project. + * `terraform-hisec/` — Infrastructure administered by [Infra Owners](roles.md). This covers for example sensitive resources such as the GPG signing key in Azure Key Vault, and the high-security Terraform state backend. - Because we don't expect the infrastructure in this directory to change very often, we've chosen — for security reasons — not to run Terraform in a CI/CD pipeline. This way we don't have to worry about the security of the CI/CD pipeline's service account. Instead, an [Infra Owner](roles.md) runs Terraform manually, using that person's personal Google Cloud credentials. + Because we don't expect the infrastructure in this directory to change very often, we've chosen — for security reasons — not to run Terraform in a CI/CD pipeline. This way we don't have to worry about the security of any CI/CD pipeline credentials. Instead, an [Infra Owner](roles.md) runs Terraform manually, using their personal cloud credentials. - * `kubernetes/` — Kubernetes resources administered by [Infra Maintainers](roles.md). + * `ansible/` — Configuration of the backend VM (Caddy, the API server, Prometheus, and OS hardening). Administered by [Infra Maintainers](roles.md) and applied manually; see [Deployment guide](deploy.md). - * `.github/workflows/apiserver.yml` — Deploys the API server. + * `.github/workflows/apiserver.yml` — Builds and deploys the API server. Note that not all infrastructure can, or (for security reasons) should, be managed via code. Learn more at [Infrastructure bootstrapping](infrastructure-bootstrapping.md). diff --git a/docs/infrastructure-bootstrapping.md b/docs/infrastructure-bootstrapping.md index c8c8510..6efceda 100644 --- a/docs/infrastructure-bootstrapping.md +++ b/docs/infrastructure-bootstrapping.md @@ -1,6 +1,6 @@ # Infrastructure bootstrapping -We try to codify infrastructure as much as possible using Terraform and Kubernetes YAML. However: +We try to codify infrastructure as much as possible using Terraform and Ansible. However: - Not everything _can_ be automated. For example, we need to setup Azure Blob Storage for storing Terraform state, before we can use Terraform. - Not everything _should_ be automated. For example, the `fullstaq-ruby-hisec` project contains such sensitive data, that giving access to CI/CD systems would pose a security risk. diff --git a/docs/infrastructure-overview.drawio.svg b/docs/infrastructure-overview.drawio.svg deleted file mode 100644 index 6362db3..0000000 --- a/docs/infrastructure-overview.drawio.svg +++ /dev/null @@ -1,1846 +0,0 @@ - - - - - - - - - - -
-
-
- logs to -
-
-
- - - logs to - - - - - - - - -
-
-
- Kubernetes Engine (autopilot) -
-
-
- - - Kubernetes... - - - - - - - - -
-
-
- Deployment -
-
-
- - - Deployment - - - - - - - -
-
-
- Nginx web server -
- ("gateway") -
-
-
- - - Nginx web server... - - - - - - - - -
-
-
- Virtual host -
-
-
- - - Virtual ho... - - - - - - -
-
-
- apt.fullstaqruby.org -
-
-
- - - apt.fullstaqr... - - - - - - - -
-
-
-

- - - Static IP - - -
- gateway -

-
-
-
- - - Static IP... - - - - - - - - - -
-
-
-

- - - A record - - -
- apt.fullstaqruby.org -

-
-
-
- - - A record... - - - - - - - - -
-
-
-

- - - A record - - -
- yum.fullstaqruby.org -

-
-
-
- - - A record... - - - - - - -
-
-
-

- - - - Cloud DNS -
- - - - - DNS zone -
- - - fullstaqruby.org - -

-
-
-
- - - Cloud DNS... - - - - - - - - -
-
-
- HTTP redirect to path: -
- /versions/$LATEST_VERSION/public/$URI -
-
-
- - - HTTP redirect to path:... - - - - - - - -
-
-
- Virtual host -
-
-
- - - Virtual ho... - - - - - - -
-
-
- yum.fullstaqruby.org -
-
-
- - - yum.fullstaqr... - - - - - - - - -
-
-
- HTTP redirect to path: -
- /versions/$LATEST_VERSION/public/$URI -
-
-
- - - HTTP redirect to path:... - - - - - - -
-
-
-

- - Google Cloud project - -
- - fullstaq-ruby - -

-
-
-
- - - Google Cloud project... - - - - - - - -
-
-
-

- - Google Cloud project - -
- - fullstaq-ruby-hisec - -

-
-
-
- - - Google Cloud project... - - - - - - - -
-
-
-

- - Secret Manager - -
-

-
-
-
- - - Secret Manager - - - - - - - -
-
-
- Resources inside the project are editable by - - Infra Maintainers - -
-
-
- - - Resources inside the pro... - - - - - - - -
-
-
-

- - TransIP - -
- - Account: fullstaq - -

-
-
-
- - - TransIP... - - - - - - - - -
-
-
-

-
-

-
-
-
- - - - - - - -
-
-
-

- - Cloud Storage bucket -
- -
- - - Terraform state - -
-
- - fullstaq-ruby-infra- -
- terraform-state - -
- -

-
-
-
- - - Cloud Storage bucket... - - - - - - - -
-
-
-

- Domain name -
- - fullstaqruby.org - -
-

-
-
-
- - - Domain name... - - - - - - - -
-
-
-

- - - A record - - -
- fullstaqruby.org -

-
-
-
- - - A record... - - - - - - - -
-
-
- infra owners -
-
-
- - - infra owners - - - - - - - -
-
-
- infra owners -
-
-
- - - infra owners - - - - - - - -
-
-
- infra owners -
-
-
- - - infra owners - - - - - - - -
-
-
- infra maintainers -
-
-
- - - infra maintainers - - - - - - - - -
-
-
-

- - Github pages - -

-

- fullstaq-ruby-website -

-
-
-
- - - Github pages... - - - - - - - - -
-
-
-

-
-

-
-
-
- - - - - - - -
-
-
-

- - Container Registry - -

-
-
-
- - - Container Regist... - - - - - - - - - - - - - - -
-
-
- can call -
-
-
- - - can call - - - - - - -
-
-
- has object -
- admin access to -
-
-
- - - has object... - - - - - - - - -
-
-
- has object -
- admin access to -
-
-
- - - has object... - - - - - - - -
-
-
- - GPG private key - -
- - (canonical copy) - -
-
-
- - - GPG private key... - - - - - - - - -
-
-
-

-
-

-
-
-
- - - - - - - -
-
-
-

- - Cloud Storage bucket -
- -
- - - fullstaq-ruby-server-edition-ci-artifacts - -
- -

-
-
-
- - - Cloud Storage bucket... - - - - - - - - -
-
-
- has r/w -
- access to -
-
-
- - - has r/w... - - - - - - - -
-
-
-

- - Email: -
- fullstaq-ruby-ci-bot -
- @fullstaq.com - -

-
-
-
- - - Email:... - - - - - - - -
-
-
- - fullstaq-ruby-ci-bot Github password - -
-
-
- - - fullstaq-ruby-ci-bot G... - - - - - - - -
-
-
- infra owners -
-
-
- - - infra owners - - - - - - - - -
-
-
- is copy -
- of -
-
-
- - - is copy... - - - - - - - -
-
-
- - Personal access token -
- -
-
-
- - - Personal... - - - - - - - - -
-
-
-

-
-

-
-
-
- - - - - - - -
-
-
-

- - Cloud Storage bucket -
- -
- - - - fullstaq-ruby-server-edition-apt-repo - - -
- -

-
-
-
- - - Cloud Storage bucket... - - - - - - - - -
-
-
-

-
-

-
-
-
- - - - - - - -
-
-
-

- - Cloud Storage bucket -
- -
- - fullstaq-ruby-server-edition-yum-repo -
- -

-
-
-
- - - Cloud Storage bucket... - - - - - - - -
-
-
- Ingress -
-
-
- - - Ingress - - - - - - - -
-
-
- TLS enabled -
-
-
- - - TLS enabled - - - - - - - -
-
-
- - fullstaq-ruby-ci-bot Github personal access token - -
-
-
- - - fullstaq-ruby-ci-bot G... - - - - - - - - -
-
-
- is copy -
- of -
-
-
- - - is copy... - - - - - - - -
-
-
- - Password - -
-
-
- - - Password - - - - - - -
-
-
-

- - Github account - -
- - fullstaq-labs - -

-
-
-
- - - Github account... - - - - - - - - -
-
-
-

- - Github account - - -
- fullstaq-ruby-ci-bot -
- -

-
-
-
- - - Github account... - - - - - - - - - - -
-
-
- can deploy -
-
-
- - - can deploy - - - - - - - - -
-
-
- has object -
- admin access to -
-
-
- - - has object... - - - - - - - - -
-
-
- has developer -
- access to -
-
-
- - - has developer... - - - - - - - - -
-
-
- logs to -
-
-
- - - logs to - - - - - - - - - - - - - -
-
-
- - apiserver - -
- Cloud Run -
-
-
- - - apiserve... - - - - - - - - -
-
-
- - NS record - -
-
-
- - - NS record - - - - - - - - - - -
-
-
-

- - - Service account - - -

-

- - infra-ci-bot - -

-
-
-
- - - Service accou... - - - - - - - - - - -
-
-
- - Cloud Logging - -
-
-
- - - Cloud Logging - - - - - - - - -
-
-
-

-
-

-
-
-
- - - - - - - -
-
-
-

- - Cloud Storage bucket -
- -
- - - Terraform state - -
-
- - fullstaq-ruby-infra- -
- hisec-terraform-state - -
- -

-
-
-
- - - Cloud Storage bucket... - - - - - - - -
-
-
-

-
-

-
-
-
- - - - - - - -
-
-
- - - - Github repo - - -
- fullstaq-ruby-server-edition -
- -
-
-
- - - Github repo... - - - - - - - - -
-
-
- Github Actions -
-
-
- - - Github Actions - - - - - - - - -
-
-
- has private key of -
-
-
- - - has private key of - - - - - - - - -
-
-
- has -
- copy of -
-
-
- - - has... - - - - - - - -
-
-
-

-
-

-
-
-
- - - - - - - -
-
-
- - - - Github repo - - -
- fullstaq-ruby-infra -
- -
-
-
- - - Github repo... - - - - - - - - -
-
-
- Github Actions -
-
-
- - - Github Actions - - - - - - - - -
-
-
- has private key of -
-
-
- - - has private key of - - - - - - - -
-
-
-

- - Secret Manager - -
-

-
-
-
- - - Secret Manager - - - - - - - -
-
-
- - GPG private key - -
-
-
- - - GPG private key - - - - - - - - -
-
-
- - has access to - -
-
-
- - - has access to - - - - - - - -
-
-
-

- - - Service account - - -

-

- server-edition-ci-bot -

-
-
-
- - - Service accou... - - - - - - - -
-
-
- - copy of each other - -
-
-
- - - copy of each other - - - - - - - -
-
-
-

- - Azure tenant - -
- - Fullstaq B.V. - -

-
-
-
- - - Azure tenant... - - - - - - - -
-
-
- infra owners -
-
-
- - - infra owners - - - - - - - -
-
-
-

- - Resource group - -
- - fullstaq-ruby-server-edition - -

-
-
-
- - - Resource group... - - - - - - - -
-
-
- infra maintainers -
-
-
- - - infra maintainers - - - - - - - -
-
-
-

- - Storage account - -
- fsrubyseredci1 -
-

-
-
-
- - - Storage account... - - - - - - - - -
-
-
-

-
-

-
-
-
- - - - - - - -
-
-
-

- - Blob Storage container -
- - - - server-edition-ci-cache - -
- -

-
-
-
- - - Blob Storage container... - - - - - - - -
-
-
-

- - Shared access key - -

-
-
-
- - - Shared access... - - - - - - - - -
-
-
- has copy of -
-
-
- - - has copy of - - - - - - - - - Viewer does not support full SVG 1.1 - - - - diff --git a/docs/infrastructure-overview.md b/docs/infrastructure-overview.md index f95bab9..938b337 100644 --- a/docs/infrastructure-overview.md +++ b/docs/infrastructure-overview.md @@ -1,82 +1,134 @@ ## Infrastructure overview -The following diagram shows which infrastructure components exist and how they relate to each other. The bubbles on the corner of a component shows which [role](roles.md) administers that component. - -![Infrastructure overview diagram](infrastructure-overview.drawio.svg) - -## Google Cloud projects - -- Administrated by role: Infra Owners - -All Google Cloud resources are contained in two projects: - -- `fullstaq-ruby` for normal resources. Infra Maintainers have full access to all resources _inside_ this project. -- `fullstaq-ruby-hisec` for especially sensitive resources. Only Infra Owners have access to the resources inside this project. - -## Google Cloud service account for Server Edition CI/CD - -- Administered by role: Infra Maintainers - -The `fullstaq-ruby` Google Cloud project has a service account. It's used by the Server Edition's CI/CD system. This service account: - -- Has full access to the Server Edition CI artifacts store. -- Has full access to the container registry, in order to access and store [build environment images](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/build-environments.md). -- Has full access to the APT and YUM repo buckets, in order to publish new packages. - -## Google Cloud service account for Infrastructure CI/CD +The following diagram shows the major infrastructure components and how they relate to each other. The role that administers each component is given in the section heading below. + +```mermaid +flowchart LR + U([End users
apt-get / yum]) + + subgraph GH["GitHub Actions"] + SE[server-edition CI] + INF[infra CI
apiserver workflow] + end + + subgraph HZ["Hetzner VM  ·  Ubuntu ≥ 24.04"] + Caddy[Caddy
apt./yum. vhosts] + API[API server
Sinatra + Puma] + DEPL[apiserver-deployer] + Prom[Prometheus
+ node_exporter] + end + + subgraph GCP["Google Cloud  ·  fsruby-server-edition2"] + APT[(APT repo bucket)] + YUM[(YUM repo bucket)] + CIA[(CI artifacts bucket
public read)] + WIF[Workload Identity
Federation pools] + end + + subgraph AZ["Azure  ·  Entra ID tenant"] + DNS[Azure DNS
fullstaqruby.org
+ delegated apt./yum. zones] + KV[Key Vault
GPG key, ACME SP creds] + BS[(Blob Storage
Terraform state,
CI artifacts, CI cache)] + FIC[Entra ID FIC apps] + end + + GP[GitHub Pages
fullstaqruby.org / www.] + TR[TransIP
domain registrar] + + U -->|HTTPS| Caddy + Caddy -->|redirect| APT + Caddy -->|redirect| YUM + Caddy -->|/admin/*| API + Caddy -. ACME DNS-01 .-> DNS + API --> DEPL + DEPL -. systemctl restart .-> API + DEPL -. systemctl restart .-> Caddy + + SE -->|OIDC| WIF + WIF --> APT + WIF --> YUM + WIF --> CIA + SE -->|OIDC JWT| Caddy + INF -->|OIDC JWT| Caddy + SE -->|OIDC| FIC + FIC --> KV + + TR -. NS delegation .-> DNS + GP -. A/AAAA .-> DNS + HZ -. A/AAAA .-> DNS +``` + +## Google Cloud project + +- Administered by role: Infra Owners (project resource); Infra Maintainers (resources within) + +All Google Cloud resources live in a single project, `fsruby-server-edition2` (display name "Fullstaq Ruby Server Edition"). The `google_project` resource itself is provisioned in `terraform-hisec/gcloud_project.tf` so that creating/deleting the project requires Infra Owner access, but resources _inside_ the project (buckets, IAM, Workload Identity Federation) are managed in `terraform/` by Infra Maintainers. + +The hisec / non-hisec separation is enforced at the **Terraform state and access-group layer**, not via separate GCP projects. See [Terraform state (normal)](#terraform-state-normal) and [Terraform state (hisec)](#terraform-state-hisec). + +## CI/CD authentication - Administered by role: Infra Maintainers -The `fullstaq-ruby` Google Cloud project has a service account. It's used by the Infrastructure repository's CI/CD system. This service account: +CI/CD authenticates using short-lived GitHub-issued OIDC tokens — there are no long-lived service-account keys. -- Has full access to the container registry, in order to publish new API server images. -- Can deploy new versions of the API server. +- **`fullstaq-ruby/server-edition` → Google Cloud** uses [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) (defined in `terraform/gcloud_auth.tf`). Two pools (`github-ci-test`, `github-ci-deploy`) gate access by GitHub repository owner and Actions environment. Through these pools, server-edition CI jobs gain write access to the APT/YUM repo buckets and the GCP CI artifacts bucket — see `terraform/repo_buckets.tf` and `terraform/ci_storage.tf`. The CI cache lives in Azure (see below), not on GCP. +- **`fullstaq-ruby/server-edition` → Azure** uses [Federated Identity Credentials](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation) on Entra ID applications (defined in `terraform-hisec/`). These authenticate workflows that read or write Azure Blob Storage (the CI artifacts and CI cache containers) and Azure Key Vault (the GPG signing key). +- **`fullstaq-ruby/infra` → API server** uses a GitHub-issued OIDC JWT (audience `backend.fullstaqruby.org`) sent as a bearer token to `POST /admin/upgrade_apiserver`. The infra repo's `apiserver.yml` workflow does **not** authenticate to GCP or Azure APIs — the rollout mechanism is entirely on the VM (see [API server](#api-server)). The same JWT mechanism is used by `server-edition` to call `/admin/restart_web_server` after a publish. ## API server - Administered by role: Infra Maintainers -The API server is a service that allows performing limited management operations on the infrastructure. It mainly exists to securely allow the Server Edition's CI to tell Nginx about the fact that new packages have been deployed. It's hosted on Google Cloud Run. +The API server is a small Sinatra service that exposes a narrow set of `/admin/*` endpoints used by CI/CD: notably `restart_web_server` (called by Server Edition's CI after a publish so Caddy picks up the new repo version) and `upgrade_apiserver` (called by this repo's CI to roll out a new API server build). All endpoints authenticate the caller via GitHub Actions OIDC JWTs. -The API server's source code lives in the Infrastructure repository, and is deployed by that repository's CI/CD system. +The API server runs on the backend VM (see [VM (Hetzner)](#vm-hetzner)) under systemd, with Puma listening on a Unix socket that Caddy proxies to. It is provisioned by Ansible (`ansible/tasks/apiserver.yml`, unit file `ansible/files/apiserver.service`). -The API server is not defined in Terraform, but is defined in the Infrastructure's CI/CD system in the form of a `gcloud` call. +Releases are built and published by `.github/workflows/apiserver.yml`: the workflow packages the source as a tarball, attaches it to a GitHub Release tagged `apiserver-N`, and then calls `POST /admin/upgrade_apiserver`. That request is handled by a sibling `apiserver-deployer.service` (provisioned alongside) which fetches the release tarball into `/opt/apiserver/versions/` and restarts the API server. The deployer exists so the API server can replace itself without leaving an unreachable gap. -## Nginx web server +## Caddy web server - Administered by role: Infra Maintainers -We run one Nginx instance, which serves these virtual hosts: +Caddy runs on the backend VM (see [VM (Hetzner)](#vm-hetzner)) and serves two virtual hosts: `apt.fullstaqruby.org` and `yum.fullstaqruby.org` (see `ansible/files/Caddyfile`). Each vhost has the same shape: + +- `/admin/*` is reverse-proxied over a local Unix socket to the API server. +- All other paths redirect to the corresponding GCS repo bucket, with the current published version baked into the redirect target. + +There is no separate `backend.fullstaqruby.org` virtual host — the hostname exists as a DNS record (and as the OIDC token audience claim) but is not terminated by Caddy. CI/CD calls the `/admin/*` endpoints via `https://apt.fullstaqruby.org/admin/*`. -- `apt.fullstaqruby.org` — redirects everything to the APT repo bucket. -- `yum.fullstaqruby.org` — redirects everything to the YUM repo bucket. +The redirect target's version is read from each bucket's `latest_version.txt` once at startup (via the `query-latest-repo-versions.rb` `ExecStartPre` in the systemd unit). Caddy must therefore be restarted after a publish so it picks up the new version — that restart is what the API server's `restart_web_server` endpoint exists to trigger. -Nginx's only purpose is to redirect traffic. Users interact primarily with `{apt,yum}.fullstaqruby.org` instead of with the APT and YUM repo buckets directly. This decouples users from our APT and YUM repos are actually hosted, allowing us to change the hosting mechanism without breaking users' URLs. +Users interact with `{apt,yum}.fullstaqruby.org` rather than the buckets directly. This decouples users from where the repos are actually hosted, allowing us to change the hosting mechanism without breaking users' repository URLs. > Historic note: our APT and YUM repos used to be hosted on Bintray. But Bintray shut down on March 1 2021. The HTTP redirection mechanism allowed us to move away from Bintray with minimal downtime, and without breaking users' repository URLs. -TLS is configured through the Ingress, using [Google-managed TLS certificates](https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs). +TLS certificates are obtained via the ACME DNS-01 challenge against Azure DNS. Caddy authenticates to Azure DNS using a service principal whose credentials are stored in Azure Key Vault and injected into the Caddy systemd unit via an environment file. -## Kubernetes cluster +## VM (Hetzner) - Administered by role: Infra Maintainers -The Kubernetes cluster runs our Nginx web server. This cluster is in Autopilot mode. +A single Ubuntu (≥ 24.04) VPS hosted at Hetzner runs every backend service (Caddy, the API server, the API server deployer, Prometheus + node_exporter, fail2ban, AppArmor, unattended-upgrades, ufw). Its forward DNS records (`backend.fullstaqruby.org`, `apt.fullstaqruby.org`, `yum.fullstaqruby.org`) are managed in `terraform/dns.tf`; its static IPs are referenced from `terraform/variables.tf`. The PTR record (`backend.fullstaqruby.org`) is set manually at the Hetzner provider during bootstrapping (see [bootstrapping](infrastructure-bootstrapping.md) Step 7), not via Terraform. -## DNS, static IPs, Ingresses +The VM is configured entirely by Ansible (`ansible/main.yml`). The playbook covers OS hardening (SSH, fail2ban, AppArmor, ufw, autoreboot, unattended-upgrades) and the service stack (Prometheus, Caddy, apiserver-deployer, apiserver). There is no Kubernetes — the previous GKE Autopilot setup was replaced by this VM in the July 2024 rearchitecture. + +## DNS - Administered by role: Infra Maintainers -All DNS entries for `fullstaqruby.org` are managed through Google Cloud DNS, inside the `fullstaq-ruby` project. +All DNS records for `fullstaqruby.org` are managed in Azure DNS (`terraform/dns.tf`). + +- `fullstaqruby.org` and `www.fullstaqruby.org` point to GitHub Pages, where we host the [website](https://github.com/fullstaq-ruby/website). +- `backend.fullstaqruby.org`, `apt.fullstaqruby.org`, and `yum.fullstaqruby.org` all resolve to the backend VM's IP addresses. -- `fullstaqruby.org` points to Github Pages, where we host the [website](https://github.com/fullstaq-ruby/website). -- `{apt,yum}.fullstaqruby.org` point to Nginx's two virtual hosts, via two different static IPs and two different Kubernetes Ingresses. +`apt.fullstaqruby.org` and `yum.fullstaqruby.org` are also delegated as their own Azure DNS zones so that Caddy's ACME service principal has DNS-write access scoped only to those subdomains, not the parent zone. ## Domain name - Administrated by role: Infra Owners -The `fullstaqruby.org` domain is registered at [TransIP](https://www.transip.nl/). It's registered using [Fullstaq](https://www.fullstaq.com)'s account. The DNS zone is not managed at TransIP, but at Google Cloud DNS. +The `fullstaqruby.org` domain is registered at [TransIP](https://www.transip.nl/). It's registered using [Fullstaq](https://www.fullstaq.com)'s account. The DNS zone is not managed at TransIP, but at Azure DNS (see [DNS](#dns)). ## Github CI bot account @@ -88,13 +140,13 @@ This Github bot account is used by the Server Edition's CI/CD system. The accoun - Administered by role: Infra Maintainers -The Terraform state for normal infrastructure is stored in a Google Cloud Storage bucket, inside the `fullstaq-ruby` project. +The Terraform state for normal infrastructure is stored in an Azure Blob Storage container in the `fullstaq-ruby-terraform` resource group (see `terraform/backend.tf`). ## Terraform state (hisec) - Administered by role: Infra Owners -The Terraform state for sensitive infrastructure is stored in a Google Cloud Storage bucket, inside the `fullstaq-ruby-hisec` project. +The Terraform state for sensitive infrastructure is stored in an Azure Blob Storage container in the `fullstaq-ruby-terraform-hisec` resource group (see `terraform-hisec/backend.tf`). ## Server Edition APT & YUM repo buckets @@ -102,31 +154,25 @@ The Terraform state for sensitive infrastructure is stored in a Google Cloud Sto The Server Edition's APT and YUM repositories are stored inside these buckets. These buckets are publicly readable. -Users don't access these buckets directly. Instead, they access `apt.fullstaqruby.org` and `yum.fullstaqruby.org` (served by the Nginx web servers), which redirect to these buckets. - -## Container registry - -- Administered by role: Infra Maintainers - -The `fullstaq-ruby` project has a container registry. This registry has two uses: - -- It's used by developers' CI/CD systems, for example to store [build environment images](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/build-environments.md). -- It's used to store the API server's images. +Users don't access these buckets directly. Instead, they access `apt.fullstaqruby.org` and `yum.fullstaqruby.org` (served by Caddy on the backend VM), which redirect to these buckets. ## Server Edition CI artifacts store - Administered by role: Infra Maintainers -The Server Edition's CI/CD system stores artifacts in this bucket, for the purpose of implementing [resumption](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/ci-cd-resumption.md). Objects in this bucket only live for 30 days. +The Server Edition's CI/CD system stores artifacts for [CI/CD resumption](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/ci-cd-resumption.md) in two buckets (see `terraform/ci_storage.tf`): + +- A **GCS bucket** (`${var.gcloud_bucket_prefix}-server-edition-ci-artifacts`) — publicly readable; the `test` environment writes via WIF, the `deploy` environment reads. Objects expire after 30 days. +- An **Azure Blob container** (`server-edition-ci-artifacts` inside the `${var.storage_account_prefix}seredci1` storage account) — private; the `test` environment writes via Federated Identity Credentials, the `deploy` environment reads. Objects expire after 30 days. ## Server Edition CI cache store - Administered by role: Infra Maintainers -The Server Edition's CI/CD system stores caches in this bucket. Objects are automatically deleted 90 days after last access. +The Server Edition's CI/CD system stores caches in an **Azure Blob container** (`server-edition-ci-cache` inside the same `${var.storage_account_prefix}seredci1` storage account; see `terraform/ci_storage.tf`). Only the `test` environment writes; objects are automatically deleted 90 days after last access. There is no equivalent cache bucket on GCP. ## GPG private key - Administered by role: Infra Owners, Infra Maintainers -The GPG private key is used to sign APT and YUM repositories. We store the canonical copy in Secrets Manager in the `fullstaq-ruby-hisec` Google Cloud project. We store a secondary copy in the Secret Manager in the `fullstaq-ruby` Google Cloud project. +The GPG private key is used to sign APT and YUM repositories. It is stored in the Azure Key Vault for Infra Owners — `${var.key_vault_prefix}infraowners`, currently `fsruby2infraowners` (see `terraform-hisec/key_vault.tf`). GitHub Actions in the `test` and `deploy` environments are granted read access via Entra ID Federated Identity Credentials. diff --git a/docs/pull_request_template.md b/docs/pull_request_template.md index 96bd077..b7bb3ee 100644 --- a/docs/pull_request_template.md +++ b/docs/pull_request_template.md @@ -28,8 +28,7 @@ Check a box by filling in 'x', like so... --> - [ ] Documentation updated - - [ ] Infrastructure overview diagram updated - + - [ ] Infrastructure overview diagram updated (Mermaid block in `docs/infrastructure-overview.md`) - [ ] The following content/files are kept in sync with each other: + The Terraform version specified in `**/providers.tf`, `.github/workflows/code-reviews.yml` and `docs/required-devtools.md` + The list of responsibilities specified in `.github/ISSUE_TEMPLATE/apply_join_team.md` and `docs/responsibilities-expectations.md` From f754aa2d5452977964abdee72b7f1da7b742b9cd Mon Sep 17 00:00:00 2001 From: abtreece Date: Fri, 22 May 2026 09:20:32 -0500 Subject: [PATCH 2/8] docs: clarify CI artifacts storage section - Use interpolated bucket/storage account names instead of Terraform vars - Spell out Workload Identity Federation on first use in the section - Note that the Azure Blob artifacts container is provisioned but not yet used by CI; artifacts still go to the GCS bucket. The cache container in the same storage account is actively used (sccache). Addresses FooBarWidget review on PR #57. --- docs/infrastructure-overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/infrastructure-overview.md b/docs/infrastructure-overview.md index 938b337..5472968 100644 --- a/docs/infrastructure-overview.md +++ b/docs/infrastructure-overview.md @@ -162,8 +162,8 @@ Users don't access these buckets directly. Instead, they access `apt.fullstaqrub The Server Edition's CI/CD system stores artifacts for [CI/CD resumption](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/ci-cd-resumption.md) in two buckets (see `terraform/ci_storage.tf`): -- A **GCS bucket** (`${var.gcloud_bucket_prefix}-server-edition-ci-artifacts`) — publicly readable; the `test` environment writes via WIF, the `deploy` environment reads. Objects expire after 30 days. -- An **Azure Blob container** (`server-edition-ci-artifacts` inside the `${var.storage_account_prefix}seredci1` storage account) — private; the `test` environment writes via Federated Identity Credentials, the `deploy` environment reads. Objects expire after 30 days. +- A **GCS bucket** (`fsruby-server-edition-ci-artifacts`) — publicly readable; the `test` environment writes via Workload Identity Federation (WIF), the `deploy` environment reads. Objects expire after 30 days. +- An **Azure Blob container** (`server-edition-ci-artifacts` inside the `fsruby2seredci1` storage account) — private. Provisioned for a future migration of CI artifacts off GCS, but **not currently in use**; server-edition CI still writes artifacts only to the GCS bucket above. The cache container in the same storage account *is* actively used (see "Server Edition CI cache store" below). ## Server Edition CI cache store From b13c9095bfdf2a08f35b6c014aa2dedceccb1902 Mon Sep 17 00:00:00 2001 From: abtreece Date: Fri, 22 May 2026 09:39:12 -0500 Subject: [PATCH 3/8] docs(apiserver): rewrite README for post-Hetzner hosting The previous README described a Google Cloud Run-hosted API server authenticated with Google Cloud identity tokens. Post-Hetzner, the service runs as a systemd-managed Puma process on the backend host behind Caddy on a Unix socket, authenticates via GitHub Actions OIDC, and is deployed via the .github/workflows/apiserver.yml workflow plus the apiserver-deployer oneshot unit. The README now reflects all four. Addresses FooBarWidget review on PR #57. --- apiserver/README.md | 57 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 49 insertions(+), 8 deletions(-) diff --git a/apiserver/README.md b/apiserver/README.md index 6cf9ca5..2d528dd 100644 --- a/apiserver/README.md +++ b/apiserver/README.md @@ -1,17 +1,58 @@ # API server -The API server is a service that allows performing limited management operations on the infrastructure. It mainly exists to securely allow the Server Edition's CI to tell Caddy about the fact that new packages have been deployed. +The API server is a small service that exposes a couple of privileged management +operations to the project's GitHub Actions workflows — primarily so that CI can +trigger restarts of Caddy and self-upgrades of the API server itself, without +needing SSH access to the backend host. -Presently, there is only one endpoint: `POST /admin/restart_web_server`. This endpoint initiates restarting of Caddy, but does not wait for it to finish. +It runs as a systemd-managed Puma process (`apiserver.service`) on the Hetzner +backend host, listening on a Unix socket at `/run/apiserver/server.sock`. Caddy +reverse-proxies the `/admin/*` paths from `apt.fullstaqruby.org` and +`yum.fullstaqruby.org` to that socket — there is no dedicated `apiserver.*` +hostname. -## Calling the production instance +## Endpoints -The production instance is deployed at https://apiserver-f7awo4fcoa-uk.a.run.app/. To call it, you need to include your Google Cloud identity token in the Authorization header. +- `GET /` — health check, returns `ok`. +- `POST /admin/upgrade_apiserver` — kicks off `apiserver-deployer` (which fetches + the latest API server release from GitHub and activates it), then restarts + `apiserver` itself. Callable only from `fullstaq-ruby/infra`'s `deploy` + GitHub Actions environment. +- `POST /admin/restart_web_server` — restarts Caddy. Callable only from + `fullstaq-ruby/server-edition`'s `deploy` GitHub Actions environment. -```bash -curl -v -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://apiserver-f7awo4fcoa-uk.a.run.app/ -``` +## Authentication + +The API server authenticates callers using **GitHub Actions OIDC**. Every +request must carry an `Authorization: Bearer ` header where the token +is an ID token minted by GitHub's OIDC provider with audience claim +`backend.fullstaqruby.org`. The server verifies the JWT signature against +GitHub's JWKS and rejects the request unless the token's `repository`, `sub`, +`runner_environment`, and `environment` claims match the calling repo's +expected `deploy` environment for that endpoint. + +Because the audience and claim shape are tied to GitHub-hosted runners, the +endpoints are not callable directly by a human or from a local machine. ## Continuous deployment -New API server code changes, when pushed to main, are automatically deployed by the Infrastructure project's CI. +API server builds and deployments are driven by two workflows in +`.github/workflows/`: + +- **Test and build** (`test-and-build.yml`) runs on every push. When a push + touches `apiserver/**`, its `apiserver-build` job packages the API server + into a tarball and uploads it as a workflow artifact. +- **Deploy** (`deploy.yml`) runs after a successful Test and build run on + `main`. It tags the commit, publishes a GitHub release with the tarball, and + calls `POST https://apt.fullstaqruby.org/admin/upgrade_apiserver` with a + freshly minted OIDC token. + +On the host, `apiserver-deployer` (a oneshot systemd unit running +`/usr/local/bin/apiserver-deployer`) handles the actual install: it fetches +the latest release metadata from the GitHub API, downloads the asset matching +the host's distribution and architecture, extracts it into +`/opt/apiserver/versions/---`, installs any runtime +dependencies declared in `dpkg-dependencies.txt`, prunes all but the last five +versions, and atomically swaps the `/opt/apiserver/versions/latest` symlink. +The API server's working directory points at that symlink, so the subsequent +`systemctl restart apiserver` brings the new version online. From b0b5fe7a52f323bc8a5e85c50bd5313985c78967 Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 20:32:22 -0500 Subject: [PATCH 4/8] docs: add API server deployment step to bootstrapping guide --- docs/infrastructure-bootstrapping.md | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/docs/infrastructure-bootstrapping.md b/docs/infrastructure-bootstrapping.md index 6efceda..6f817fe 100644 --- a/docs/infrastructure-bootstrapping.md +++ b/docs/infrastructure-bootstrapping.md @@ -156,7 +156,26 @@ ansible-playbook -i hosts.ini -v main.yml cd .. ``` -## Step 12: Populate Github Actions secrets and variables +## Step 12: Deploy the API server + +The Ansible run in step 11 installs `apiserver-deployer`, which downloads the latest API server release from the [fullstaq-ruby/infra](https://github.com/fullstaq-ruby/infra/releases) repo. If a release already exists, the API server is now running. Verify on the VM: + +```bash +sudo systemctl status apiserver-deployer apiserver +``` + +If the repo has no release yet (e.g. it was freshly created in step 6): + +1. In the [fullstaq-ruby/infra](https://github.com/fullstaq-ruby/infra/settings/environments) repo, create a `deploy` environment. +2. Push a commit that touches `apiserver/` to `main`. This triggers the "Test and build" workflow, followed by the "Deploy" workflow, which publishes a release. The Deploy workflow's final step (calling `/admin/upgrade_apiserver`) fails because no API server is running yet; that is expected. +3. On the VM, install and start the released API server: + + ```bash + sudo systemctl restart apiserver-deployer + sudo systemctl restart apiserver + ``` + +## Step 13: Populate Github Actions secrets and variables In the [fullstaq-ruby/server-edition](https://github.com/fullstaq-ruby/server-edition/settings/secrets) repo, create the following environments: @@ -186,6 +205,6 @@ Create these environment-specific variables: - `AZURE_CLIENT_ID` ('test' environment): fetch using `pushd terraform-hisec >/dev/null && terraform show -json | jq -r '.values.root_module.resources[] | select(.address == "azuread_application.server-edition-github-ci-test") | .values.application_id'; popd >/dev/null` - `AZURE_CLIENT_ID` ('deploy' environment): fetch using `pushd terraform-hisec >/dev/null && terraform show -json | jq -r '.values.root_module.resources[] | select(.address == "azuread_application.server-edition-github-ci-deploy") | .values.application_id'; popd >/dev/null` -## Step 13: Onboard everybody +## Step 14: Onboard everybody Onboard everybody in the [members list](members.md) according to the [onboarding instructions](onboarding.md). From dcb67009315f4d56a0dca2072397a8ebc50e867f Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 20:32:23 -0500 Subject: [PATCH 5/8] docs: reference .github/workflows/ instead of apiserver.yml --- docs/deploy.md | 2 +- docs/infrastructure-as-code.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deploy.md b/docs/deploy.md index 4c942ff..223eaf2 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -64,4 +64,4 @@ This guide explains how to deploy infrastructure updates. This guide is not for cd .. ~~~ -> The API server itself is not deployed by this playbook. Code changes under `apiserver/` are released by the `.github/workflows/apiserver.yml` workflow, which packages a tarball, attaches it to a GitHub Release, and triggers `POST /admin/upgrade_apiserver` on the live host. +> The API server itself is not deployed by this playbook. Code changes under `apiserver/` are built and deployed by the Github Actions workflows in `.github/workflows/` when pushed to `main`. They package a tarball, attach it to a GitHub Release, and trigger `POST /admin/upgrade_apiserver` on the live host. diff --git a/docs/infrastructure-as-code.md b/docs/infrastructure-as-code.md index 0f3159c..97d21b1 100644 --- a/docs/infrastructure-as-code.md +++ b/docs/infrastructure-as-code.md @@ -16,6 +16,6 @@ The infrastructure-as-code is stored in the following directories: * `ansible/` — Configuration of the backend VM (Caddy, the API server, Prometheus, and OS hardening). Administered by [Infra Maintainers](roles.md) and applied manually; see [Deployment guide](deploy.md). - * `.github/workflows/apiserver.yml` — Builds and deploys the API server. + * `.github/workflows/` — Github Actions workflows that together test the infrastructure code, and build and deploy the API server. Note that not all infrastructure can, or (for security reasons) should, be managed via code. Learn more at [Infrastructure bootstrapping](infrastructure-bootstrapping.md). From fd09d2faf3f36a804b1252654deff656a4b3786a Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 22:56:34 -0500 Subject: [PATCH 6/8] docs: simplify API server bootstrapping step --- docs/infrastructure-bootstrapping.md | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/docs/infrastructure-bootstrapping.md b/docs/infrastructure-bootstrapping.md index 6f817fe..c1805ab 100644 --- a/docs/infrastructure-bootstrapping.md +++ b/docs/infrastructure-bootstrapping.md @@ -148,6 +148,8 @@ Register the domain `fullstaqruby.org`. Configure it to use the Azure DNS zone. ## Step 11: Run initial Ansible +The playbook deploys the API server from the latest `apiserver-*` release of the [fullstaq-ruby/infra](https://github.com/fullstaq-ruby/infra/releases) repo, so at least one such release must exist. + Make sure the Azure CLI is logged in, then: ```bash @@ -158,22 +160,17 @@ cd .. ## Step 12: Deploy the API server -The Ansible run in step 11 installs `apiserver-deployer`, which downloads the latest API server release from the [fullstaq-ruby/infra](https://github.com/fullstaq-ruby/infra/releases) repo. If a release already exists, the API server is now running. Verify on the VM: +The Ansible run in step 11 installs `apiserver-deployer`, which downloads and activates the latest API server release from the [fullstaq-ruby/infra](https://github.com/fullstaq-ruby/infra/releases) repo. Verify on the VM that the API server is running: ```bash sudo systemctl status apiserver-deployer apiserver ``` -If the repo has no release yet (e.g. it was freshly created in step 6): - -1. In the [fullstaq-ruby/infra](https://github.com/fullstaq-ruby/infra/settings/environments) repo, create a `deploy` environment. -2. Push a commit that touches `apiserver/` to `main`. This triggers the "Test and build" workflow, followed by the "Deploy" workflow, which publishes a release. The Deploy workflow's final step (calling `/admin/upgrade_apiserver`) fails because no API server is running yet; that is expected. -3. On the VM, install and start the released API server: +If it isn't, install and start the latest release manually: - ```bash - sudo systemctl restart apiserver-deployer - sudo systemctl restart apiserver - ``` +```bash +sudo systemctl restart apiserver-deployer apiserver +``` ## Step 13: Populate Github Actions secrets and variables From cfb8b153db2fee074278a189cfdf9621ea6f0661 Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 22:59:59 -0500 Subject: [PATCH 7/8] docs: trim drift-prone detail and fix stale refs in infra overview --- docs/infrastructure-overview.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/infrastructure-overview.md b/docs/infrastructure-overview.md index 5472968..de4839a 100644 --- a/docs/infrastructure-overview.md +++ b/docs/infrastructure-overview.md @@ -64,8 +64,6 @@ flowchart LR All Google Cloud resources live in a single project, `fsruby-server-edition2` (display name "Fullstaq Ruby Server Edition"). The `google_project` resource itself is provisioned in `terraform-hisec/gcloud_project.tf` so that creating/deleting the project requires Infra Owner access, but resources _inside_ the project (buckets, IAM, Workload Identity Federation) are managed in `terraform/` by Infra Maintainers. -The hisec / non-hisec separation is enforced at the **Terraform state and access-group layer**, not via separate GCP projects. See [Terraform state (normal)](#terraform-state-normal) and [Terraform state (hisec)](#terraform-state-hisec). - ## CI/CD authentication - Administered by role: Infra Maintainers @@ -74,7 +72,7 @@ CI/CD authenticates using short-lived GitHub-issued OIDC tokens — there are no - **`fullstaq-ruby/server-edition` → Google Cloud** uses [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) (defined in `terraform/gcloud_auth.tf`). Two pools (`github-ci-test`, `github-ci-deploy`) gate access by GitHub repository owner and Actions environment. Through these pools, server-edition CI jobs gain write access to the APT/YUM repo buckets and the GCP CI artifacts bucket — see `terraform/repo_buckets.tf` and `terraform/ci_storage.tf`. The CI cache lives in Azure (see below), not on GCP. - **`fullstaq-ruby/server-edition` → Azure** uses [Federated Identity Credentials](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation) on Entra ID applications (defined in `terraform-hisec/`). These authenticate workflows that read or write Azure Blob Storage (the CI artifacts and CI cache containers) and Azure Key Vault (the GPG signing key). -- **`fullstaq-ruby/infra` → API server** uses a GitHub-issued OIDC JWT (audience `backend.fullstaqruby.org`) sent as a bearer token to `POST /admin/upgrade_apiserver`. The infra repo's `apiserver.yml` workflow does **not** authenticate to GCP or Azure APIs — the rollout mechanism is entirely on the VM (see [API server](#api-server)). The same JWT mechanism is used by `server-edition` to call `/admin/restart_web_server` after a publish. +- **`fullstaq-ruby/infra` → API server** uses a GitHub-issued OIDC JWT (audience `backend.fullstaqruby.org`) sent as a bearer token to `POST /admin/upgrade_apiserver`. The infra repo's workflows do **not** authenticate to GCP or Azure APIs — the rollout mechanism is entirely on the VM (see [API server](#api-server)). The same JWT mechanism is used by `server-edition` to call `/admin/restart_web_server` after a publish. ## API server @@ -84,7 +82,7 @@ The API server is a small Sinatra service that exposes a narrow set of `/admin/* The API server runs on the backend VM (see [VM (Hetzner)](#vm-hetzner)) under systemd, with Puma listening on a Unix socket that Caddy proxies to. It is provisioned by Ansible (`ansible/tasks/apiserver.yml`, unit file `ansible/files/apiserver.service`). -Releases are built and published by `.github/workflows/apiserver.yml`: the workflow packages the source as a tarball, attaches it to a GitHub Release tagged `apiserver-N`, and then calls `POST /admin/upgrade_apiserver`. That request is handled by a sibling `apiserver-deployer.service` (provisioned alongside) which fetches the release tarball into `/opt/apiserver/versions/` and restarts the API server. The deployer exists so the API server can replace itself without leaving an unreachable gap. +Releases are built and published by the workflows in `.github/workflows/`: the "Test and build" workflow packages the source as a tarball, and the "Deploy" workflow attaches it to a GitHub Release tagged `apiserver-N` and then calls `POST /admin/upgrade_apiserver`. That request triggers a sibling `apiserver-deployer.service` (provisioned alongside), which fetches the release tarball into `/opt/apiserver/versions/`, after which the API server is restarted. The deployer exists so that the API server can be replaced without leaving an unreachable gap, and for security: the API server itself runs as a separate user without the ability to modify its own code. ## Caddy web server @@ -97,7 +95,7 @@ Caddy runs on the backend VM (see [VM (Hetzner)](#vm-hetzner)) and serves two vi There is no separate `backend.fullstaqruby.org` virtual host — the hostname exists as a DNS record (and as the OIDC token audience claim) but is not terminated by Caddy. CI/CD calls the `/admin/*` endpoints via `https://apt.fullstaqruby.org/admin/*`. -The redirect target's version is read from each bucket's `latest_version.txt` once at startup (via the `query-latest-repo-versions.rb` `ExecStartPre` in the systemd unit). Caddy must therefore be restarted after a publish so it picks up the new version — that restart is what the API server's `restart_web_server` endpoint exists to trigger. +The redirect target's version is read from each bucket's `latest_version.txt` once at startup (via the `query-latest-repo-versions.rb` `ExecStartPre` in the systemd unit). Caddy must therefore be restarted after a publish so it picks up the new version — that restart is what the API server's `restart_web_server` endpoint exists to trigger. See the [APT/YUM repo design](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/apt-yum-repo.md) for how `latest_version.txt` fits into the repo layout. Users interact with `{apt,yum}.fullstaqruby.org` rather than the buckets directly. This decouples users from where the repos are actually hosted, allowing us to change the hosting mechanism without breaking users' repository URLs. @@ -109,9 +107,9 @@ TLS certificates are obtained via the ACME DNS-01 challenge against Azure DNS. C - Administered by role: Infra Maintainers -A single Ubuntu (≥ 24.04) VPS hosted at Hetzner runs every backend service (Caddy, the API server, the API server deployer, Prometheus + node_exporter, fail2ban, AppArmor, unattended-upgrades, ufw). Its forward DNS records (`backend.fullstaqruby.org`, `apt.fullstaqruby.org`, `yum.fullstaqruby.org`) are managed in `terraform/dns.tf`; its static IPs are referenced from `terraform/variables.tf`. The PTR record (`backend.fullstaqruby.org`) is set manually at the Hetzner provider during bootstrapping (see [bootstrapping](infrastructure-bootstrapping.md) Step 7), not via Terraform. +A single Ubuntu (≥ 24.04) VPS hosted at Hetzner runs Caddy and the API server. Its forward DNS records (`backend.fullstaqruby.org`, `apt.fullstaqruby.org`, `yum.fullstaqruby.org`) are managed in `terraform/dns.tf`; its static IPs are referenced from `terraform/variables.tf`. The PTR record (`backend.fullstaqruby.org`) is set manually at the Hetzner provider during bootstrapping (see [bootstrapping](infrastructure-bootstrapping.md) Step 7), not via Terraform. -The VM is configured entirely by Ansible (`ansible/main.yml`). The playbook covers OS hardening (SSH, fail2ban, AppArmor, ufw, autoreboot, unattended-upgrades) and the service stack (Prometheus, Caddy, apiserver-deployer, apiserver). There is no Kubernetes — the previous GKE Autopilot setup was replaced by this VM in the July 2024 rearchitecture. +The VM is configured entirely by Ansible (`ansible/main.yml`), covering both OS hardening and the services it runs. ## DNS @@ -162,17 +160,17 @@ Users don't access these buckets directly. Instead, they access `apt.fullstaqrub The Server Edition's CI/CD system stores artifacts for [CI/CD resumption](https://github.com/fullstaq-ruby/server-edition/blob/main/dev-handbook/ci-cd-resumption.md) in two buckets (see `terraform/ci_storage.tf`): -- A **GCS bucket** (`fsruby-server-edition-ci-artifacts`) — publicly readable; the `test` environment writes via Workload Identity Federation (WIF), the `deploy` environment reads. Objects expire after 30 days. +- A **GCS bucket** (`fsruby-server-edition-ci-artifacts`) — publicly readable; the `test` environment writes via Workload Identity Federation (WIF), the `deploy` environment reads. Objects expire automatically. - An **Azure Blob container** (`server-edition-ci-artifacts` inside the `fsruby2seredci1` storage account) — private. Provisioned for a future migration of CI artifacts off GCS, but **not currently in use**; server-edition CI still writes artifacts only to the GCS bucket above. The cache container in the same storage account *is* actively used (see "Server Edition CI cache store" below). ## Server Edition CI cache store - Administered by role: Infra Maintainers -The Server Edition's CI/CD system stores caches in an **Azure Blob container** (`server-edition-ci-cache` inside the same `${var.storage_account_prefix}seredci1` storage account; see `terraform/ci_storage.tf`). Only the `test` environment writes; objects are automatically deleted 90 days after last access. There is no equivalent cache bucket on GCP. +The Server Edition's CI/CD system stores caches in an **Azure Blob container** (`server-edition-ci-cache` inside the same `fsruby2seredci1` storage account; see `terraform/ci_storage.tf`). Only the `test` environment writes; objects expire automatically based on last access time. There is no equivalent cache bucket on GCP. ## GPG private key - Administered by role: Infra Owners, Infra Maintainers -The GPG private key is used to sign APT and YUM repositories. It is stored in the Azure Key Vault for Infra Owners — `${var.key_vault_prefix}infraowners`, currently `fsruby2infraowners` (see `terraform-hisec/key_vault.tf`). GitHub Actions in the `test` and `deploy` environments are granted read access via Entra ID Federated Identity Credentials. +The GPG private key is used to sign APT and YUM repositories. It is stored in the Azure Key Vault for Infra Owners, `fsruby2infraowners` (see `terraform-hisec/key_vault.tf`). GitHub Actions in the `test` and `deploy` environments are granted read access via GitHub OIDC (see [CI/CD authentication](#cicd-authentication)). From 0d01a9986e110b58ad8f82a3640af76a34fc204d Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 23:06:53 -0500 Subject: [PATCH 8/8] docs: restore drawio infrastructure diagram in place of Mermaid --- README.md | 1 + docs/editing-diagrams.md | 12 + docs/infrastructure-overview.drawio.svg | 1846 +++++++++++++++++++++++ docs/infrastructure-overview.md | 60 +- docs/pull_request_template.md | 3 +- 5 files changed, 1864 insertions(+), 58 deletions(-) create mode 100644 docs/editing-diagrams.md create mode 100644 docs/infrastructure-overview.drawio.svg diff --git a/README.md b/README.md index dbfba64..0d8720c 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ Concepts: Tasks: * [Deployment guide](docs/deploy.md) + * [Editing diagrams](docs/editing-diagrams.md) Organizational (for team members): diff --git a/docs/editing-diagrams.md b/docs/editing-diagrams.md new file mode 100644 index 0000000..c032294 --- /dev/null +++ b/docs/editing-diagrams.md @@ -0,0 +1,12 @@ +# Editing diagrams + +All diagrams' sources are stored in `.drawio.svg` files — _SVG files with embedded Diagrams.net content_. They are made using [Diagrams.net](https://diagrams.net) (formerly Draw.io). + +The recommended way to edit these files locally is with [Visual Studio Code's Draw.io integration](https://marketplace.visualstudio.com/items?itemName=hediet.vscode-drawio). + +If you want to use Diagram.net's web app instead of the Visual Studio Code integration, then be sure to save it by exporting the diagram as SVG, using the following settings: + + * Zoom: 110% + * Border width: 12 + * Check: Transparent Background + * Uncheck: Include a copy of my diagram diff --git a/docs/infrastructure-overview.drawio.svg b/docs/infrastructure-overview.drawio.svg new file mode 100644 index 0000000..6362db3 --- /dev/null +++ b/docs/infrastructure-overview.drawio.svg @@ -0,0 +1,1846 @@ + + + + + + + + + + +
+
+
+ logs to +
+
+
+ + + logs to + + + + + + + + +
+
+
+ Kubernetes Engine (autopilot) +
+
+
+ + + Kubernetes... + + + + + + + + +
+
+
+ Deployment +
+
+
+ + + Deployment + + + + + + + +
+
+
+ Nginx web server +
+ ("gateway") +
+
+
+ + + Nginx web server... + + + + + + + + +
+
+
+ Virtual host +
+
+
+ + + Virtual ho... + + + + + + +
+
+
+ apt.fullstaqruby.org +
+
+
+ + + apt.fullstaqr... + + + + + + + +
+
+
+

+ + + Static IP + + +
+ gateway +

+
+
+
+ + + Static IP... + + + + + + + + + +
+
+
+

+ + + A record + + +
+ apt.fullstaqruby.org +

+
+
+
+ + + A record... + + + + + + + + +
+
+
+

+ + + A record + + +
+ yum.fullstaqruby.org +

+
+
+
+ + + A record... + + + + + + +
+
+
+

+ + + + Cloud DNS +
+ + + + + DNS zone +
+ + + fullstaqruby.org + +

+
+
+
+ + + Cloud DNS... + + + + + + + + +
+
+
+ HTTP redirect to path: +
+ /versions/$LATEST_VERSION/public/$URI +
+
+
+ + + HTTP redirect to path:... + + + + + + + +
+
+
+ Virtual host +
+
+
+ + + Virtual ho... + + + + + + +
+
+
+ yum.fullstaqruby.org +
+
+
+ + + yum.fullstaqr... + + + + + + + + +
+
+
+ HTTP redirect to path: +
+ /versions/$LATEST_VERSION/public/$URI +
+
+
+ + + HTTP redirect to path:... + + + + + + +
+
+
+

+ + Google Cloud project + +
+ + fullstaq-ruby + +

+
+
+
+ + + Google Cloud project... + + + + + + + +
+
+
+

+ + Google Cloud project + +
+ + fullstaq-ruby-hisec + +

+
+
+
+ + + Google Cloud project... + + + + + + + +
+
+
+

+ + Secret Manager + +
+

+
+
+
+ + + Secret Manager + + + + + + + +
+
+
+ Resources inside the project are editable by + + Infra Maintainers + +
+
+
+ + + Resources inside the pro... + + + + + + + +
+
+
+

+ + TransIP + +
+ + Account: fullstaq + +

+
+
+
+ + + TransIP... + + + + + + + + +
+
+
+

+
+

+
+
+
+ + + + + + + +
+
+
+

+ + Cloud Storage bucket +
+ +
+ + + Terraform state + +
+
+ + fullstaq-ruby-infra- +
+ terraform-state + +
+ +

+
+
+
+ + + Cloud Storage bucket... + + + + + + + +
+
+
+

+ Domain name +
+ + fullstaqruby.org + +
+

+
+
+
+ + + Domain name... + + + + + + + +
+
+
+

+ + + A record + + +
+ fullstaqruby.org +

+
+
+
+ + + A record... + + + + + + + +
+
+
+ infra owners +
+
+
+ + + infra owners + + + + + + + +
+
+
+ infra owners +
+
+
+ + + infra owners + + + + + + + +
+
+
+ infra owners +
+
+
+ + + infra owners + + + + + + + +
+
+
+ infra maintainers +
+
+
+ + + infra maintainers + + + + + + + + +
+
+
+

+ + Github pages + +

+

+ fullstaq-ruby-website +

+
+
+
+ + + Github pages... + + + + + + + + +
+
+
+

+
+

+
+
+
+ + + + + + + +
+
+
+

+ + Container Registry + +

+
+
+
+ + + Container Regist... + + + + + + + + + + + + + + +
+
+
+ can call +
+
+
+ + + can call + + + + + + +
+
+
+ has object +
+ admin access to +
+
+
+ + + has object... + + + + + + + + +
+
+
+ has object +
+ admin access to +
+
+
+ + + has object... + + + + + + + +
+
+
+ + GPG private key + +
+ + (canonical copy) + +
+
+
+ + + GPG private key... + + + + + + + + +
+
+
+

+
+

+
+
+
+ + + + + + + +
+
+
+

+ + Cloud Storage bucket +
+ +
+ + + fullstaq-ruby-server-edition-ci-artifacts + +
+ +

+
+
+
+ + + Cloud Storage bucket... + + + + + + + + +
+
+
+ has r/w +
+ access to +
+
+
+ + + has r/w... + + + + + + + +
+
+
+

+ + Email: +
+ fullstaq-ruby-ci-bot +
+ @fullstaq.com + +

+
+
+
+ + + Email:... + + + + + + + +
+
+
+ + fullstaq-ruby-ci-bot Github password + +
+
+
+ + + fullstaq-ruby-ci-bot G... + + + + + + + +
+
+
+ infra owners +
+
+
+ + + infra owners + + + + + + + + +
+
+
+ is copy +
+ of +
+
+
+ + + is copy... + + + + + + + +
+
+
+ + Personal access token +
+ +
+
+
+ + + Personal... + + + + + + + + +
+
+
+

+
+

+
+
+
+ + + + + + + +
+
+
+

+ + Cloud Storage bucket +
+ +
+ + + + fullstaq-ruby-server-edition-apt-repo + + +
+ +

+
+
+
+ + + Cloud Storage bucket... + + + + + + + + +
+
+
+

+
+

+
+
+
+ + + + + + + +
+
+
+

+ + Cloud Storage bucket +
+ +
+ + fullstaq-ruby-server-edition-yum-repo +
+ +

+
+
+
+ + + Cloud Storage bucket... + + + + + + + +
+
+
+ Ingress +
+
+
+ + + Ingress + + + + + + + +
+
+
+ TLS enabled +
+
+
+ + + TLS enabled + + + + + + + +
+
+
+ + fullstaq-ruby-ci-bot Github personal access token + +
+
+
+ + + fullstaq-ruby-ci-bot G... + + + + + + + + +
+
+
+ is copy +
+ of +
+
+
+ + + is copy... + + + + + + + +
+
+
+ + Password + +
+
+
+ + + Password + + + + + + +
+
+
+

+ + Github account + +
+ + fullstaq-labs + +

+
+
+
+ + + Github account... + + + + + + + + +
+
+
+

+ + Github account + + +
+ fullstaq-ruby-ci-bot +
+ +

+
+
+
+ + + Github account... + + + + + + + + + + +
+
+
+ can deploy +
+
+
+ + + can deploy + + + + + + + + +
+
+
+ has object +
+ admin access to +
+
+
+ + + has object... + + + + + + + + +
+
+
+ has developer +
+ access to +
+
+
+ + + has developer... + + + + + + + + +
+
+
+ logs to +
+
+
+ + + logs to + + + + + + + + + + + + + +
+
+
+ + apiserver + +
+ Cloud Run +
+
+
+ + + apiserve... + + + + + + + + +
+
+
+ + NS record + +
+
+
+ + + NS record + + + + + + + + + + +
+
+
+

+ + + Service account + + +

+

+ + infra-ci-bot + +

+
+
+
+ + + Service accou... + + + + + + + + + + +
+
+
+ + Cloud Logging + +
+
+
+ + + Cloud Logging + + + + + + + + +
+
+
+

+
+

+
+
+
+ + + + + + + +
+
+
+

+ + Cloud Storage bucket +
+ +
+ + + Terraform state + +
+
+ + fullstaq-ruby-infra- +
+ hisec-terraform-state + +
+ +

+
+
+
+ + + Cloud Storage bucket... + + + + + + + +
+
+
+

+
+

+
+
+
+ + + + + + + +
+
+
+ + + + Github repo + + +
+ fullstaq-ruby-server-edition +
+ +
+
+
+ + + Github repo... + + + + + + + + +
+
+
+ Github Actions +
+
+
+ + + Github Actions + + + + + + + + +
+
+
+ has private key of +
+
+
+ + + has private key of + + + + + + + + +
+
+
+ has +
+ copy of +
+
+
+ + + has... + + + + + + + +
+
+
+

+
+

+
+
+
+ + + + + + + +
+
+
+ + + + Github repo + + +
+ fullstaq-ruby-infra +
+ +
+
+
+ + + Github repo... + + + + + + + + +
+
+
+ Github Actions +
+
+
+ + + Github Actions + + + + + + + + +
+
+
+ has private key of +
+
+
+ + + has private key of + + + + + + + +
+
+
+

+ + Secret Manager + +
+

+
+
+
+ + + Secret Manager + + + + + + + +
+
+
+ + GPG private key + +
+
+
+ + + GPG private key + + + + + + + + +
+
+
+ + has access to + +
+
+
+ + + has access to + + + + + + + +
+
+
+

+ + + Service account + + +

+

+ server-edition-ci-bot +

+
+
+
+ + + Service accou... + + + + + + + +
+
+
+ + copy of each other + +
+
+
+ + + copy of each other + + + + + + + +
+
+
+

+ + Azure tenant + +
+ + Fullstaq B.V. + +

+
+
+
+ + + Azure tenant... + + + + + + + +
+
+
+ infra owners +
+
+
+ + + infra owners + + + + + + + +
+
+
+

+ + Resource group + +
+ + fullstaq-ruby-server-edition + +

+
+
+
+ + + Resource group... + + + + + + + +
+
+
+ infra maintainers +
+
+
+ + + infra maintainers + + + + + + + +
+
+
+

+ + Storage account + +
+ fsrubyseredci1 +
+

+
+
+
+ + + Storage account... + + + + + + + + +
+
+
+

+
+

+
+
+
+ + + + + + + +
+
+
+

+ + Blob Storage container +
+ + + + server-edition-ci-cache + +
+ +

+
+
+
+ + + Blob Storage container... + + + + + + + +
+
+
+

+ + Shared access key + +

+
+
+
+ + + Shared access... + + + + + + + + +
+
+
+ has copy of +
+
+
+ + + has copy of + + + + + + + + + Viewer does not support full SVG 1.1 + + + + diff --git a/docs/infrastructure-overview.md b/docs/infrastructure-overview.md index de4839a..651be15 100644 --- a/docs/infrastructure-overview.md +++ b/docs/infrastructure-overview.md @@ -1,62 +1,8 @@ ## Infrastructure overview -The following diagram shows the major infrastructure components and how they relate to each other. The role that administers each component is given in the section heading below. - -```mermaid -flowchart LR - U([End users
apt-get / yum]) - - subgraph GH["GitHub Actions"] - SE[server-edition CI] - INF[infra CI
apiserver workflow] - end - - subgraph HZ["Hetzner VM  ·  Ubuntu ≥ 24.04"] - Caddy[Caddy
apt./yum. vhosts] - API[API server
Sinatra + Puma] - DEPL[apiserver-deployer] - Prom[Prometheus
+ node_exporter] - end - - subgraph GCP["Google Cloud  ·  fsruby-server-edition2"] - APT[(APT repo bucket)] - YUM[(YUM repo bucket)] - CIA[(CI artifacts bucket
public read)] - WIF[Workload Identity
Federation pools] - end - - subgraph AZ["Azure  ·  Entra ID tenant"] - DNS[Azure DNS
fullstaqruby.org
+ delegated apt./yum. zones] - KV[Key Vault
GPG key, ACME SP creds] - BS[(Blob Storage
Terraform state,
CI artifacts, CI cache)] - FIC[Entra ID FIC apps] - end - - GP[GitHub Pages
fullstaqruby.org / www.] - TR[TransIP
domain registrar] - - U -->|HTTPS| Caddy - Caddy -->|redirect| APT - Caddy -->|redirect| YUM - Caddy -->|/admin/*| API - Caddy -. ACME DNS-01 .-> DNS - API --> DEPL - DEPL -. systemctl restart .-> API - DEPL -. systemctl restart .-> Caddy - - SE -->|OIDC| WIF - WIF --> APT - WIF --> YUM - WIF --> CIA - SE -->|OIDC JWT| Caddy - INF -->|OIDC JWT| Caddy - SE -->|OIDC| FIC - FIC --> KV - - TR -. NS delegation .-> DNS - GP -. A/AAAA .-> DNS - HZ -. A/AAAA .-> DNS -``` +The following diagram shows which infrastructure components exist and how they relate to each other. The bubbles on the corner of a component shows which [role](roles.md) administers that component. + +![Infrastructure overview diagram](infrastructure-overview.drawio.svg) ## Google Cloud project diff --git a/docs/pull_request_template.md b/docs/pull_request_template.md index b7bb3ee..96bd077 100644 --- a/docs/pull_request_template.md +++ b/docs/pull_request_template.md @@ -28,7 +28,8 @@ Check a box by filling in 'x', like so... --> - [ ] Documentation updated - - [ ] Infrastructure overview diagram updated (Mermaid block in `docs/infrastructure-overview.md`) + - [ ] Infrastructure overview diagram updated + - [ ] The following content/files are kept in sync with each other: + The Terraform version specified in `**/providers.tf`, `.github/workflows/code-reviews.yml` and `docs/required-devtools.md` + The list of responsibilities specified in `.github/ISSUE_TEMPLATE/apply_join_team.md` and `docs/responsibilities-expectations.md`