docs: refresh infra docs for post-Hetzner architecture

.editorconfig

@@ -16,10 +16,6 @@ indent_size = 2
 indent_style = space
 indent_size = 2
 
-[config.ru]
-indent_style = tab
-indent_size = 4
-
 [*.rb]
 indent_style = space
 indent_size = 2

apiserver/README.md

@@ -1,17 +1,58 @@
 # API server
 
-The API server is a service that allows performing limited management operations on the infrastructure. It mainly exists to securely allow the Server Edition's CI to tell Caddy about the fact that new packages have been deployed.
+The API server is a small service that exposes a couple of privileged management
+operations to the project's GitHub Actions workflows — primarily so that CI can
+trigger restarts of Caddy and self-upgrades of the API server itself, without
+needing SSH access to the backend host.
 
-Presently, there is only one endpoint: `POST /admin/restart_web_server`. This endpoint initiates restarting of Caddy, but does not wait for it to finish.
+It runs as a systemd-managed Puma process (`apiserver.service`) on the Hetzner
+backend host, listening on a Unix socket at `/run/apiserver/server.sock`. Caddy
+reverse-proxies the `/admin/*` paths from `apt.fullstaqruby.org` and
+`yum.fullstaqruby.org` to that socket — there is no dedicated `apiserver.*`
+hostname.
 
-## Calling the production instance
+## Endpoints
 
-The production instance is deployed at https://apiserver-f7awo4fcoa-uk.a.run.app/. To call it, you need to include your Google Cloud identity token in the Authorization header.
+- `GET /` — health check, returns `ok`.
+- `POST /admin/upgrade_apiserver` — kicks off `apiserver-deployer` (which fetches
+  the latest API server release from GitHub and activates it), then restarts
+  `apiserver` itself. Callable only from `fullstaq-ruby/infra`'s `deploy`
+  GitHub Actions environment.
+- `POST /admin/restart_web_server` — restarts Caddy. Callable only from
+  `fullstaq-ruby/server-edition`'s `deploy` GitHub Actions environment.
 
-```bash
-curl -v -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://apiserver-f7awo4fcoa-uk.a.run.app/
-```
+## Authentication
+
+The API server authenticates callers using **GitHub Actions OIDC**. Every
+request must carry an `Authorization: Bearer <token>` header where the token
+is an ID token minted by GitHub's OIDC provider with audience claim
+`backend.fullstaqruby.org`. The server verifies the JWT signature against
+GitHub's JWKS and rejects the request unless the token's `repository`, `sub`,
+`runner_environment`, and `environment` claims match the calling repo's
+expected `deploy` environment for that endpoint.
+
+Because the audience and claim shape are tied to GitHub-hosted runners, the
+endpoints are not callable directly by a human or from a local machine.
 
 ## Continuous deployment
 
-New API server code changes, when pushed to main, are automatically deployed by the Infrastructure project's CI.
+API server builds and deployments are driven by two workflows in
+`.github/workflows/`:
+
+- **Test and build** (`test-and-build.yml`) runs on every push. When a push
+  touches `apiserver/**`, its `apiserver-build` job packages the API server
+  into a tarball and uploads it as a workflow artifact.
+- **Deploy** (`deploy.yml`) runs after a successful Test and build run on
+  `main`. It tags the commit, publishes a GitHub release with the tarball, and
+  calls `POST https://apt.fullstaqruby.org/admin/upgrade_apiserver` with a
+  freshly minted OIDC token.
+
+On the host, `apiserver-deployer` (a oneshot systemd unit running
+`/usr/local/bin/apiserver-deployer`) handles the actual install: it fetches
+the latest release metadata from the GitHub API, downloads the asset matching
+the host's distribution and architecture, extracts it into
+`/opt/apiserver/versions/<tag>-<dist>-<version>-<arch>`, installs any runtime
+dependencies declared in `dpkg-dependencies.txt`, prunes all but the last five
+versions, and atomically swaps the `/opt/apiserver/versions/latest` symlink.
+The API server's working directory points at that symlink, so the subsequent
+`systemctl restart apiserver` brings the new version online.

docs/deploy.md

@@ -56,20 +56,12 @@ This guide explains how to deploy infrastructure updates. This guide is not for
         cd ..
         ~~~
 
- 4. Get the credentials for the Kubernetes cluster:
+ 4. Apply Ansible to the backend VM:
 
     ~~~bash
-    gcloud container clusters get-credentials fullstaq-ruby-autopilot --configuration fullstaq-ruby --region us-east4
+    cd ansible
+    ansible-playbook -i hosts.ini -v main.yml
+    cd ..
     ~~~
 
- 5. Set the default namespace:
-
-    ~~~bash
-    kubectl config set-context --current --namespace=fullstaq-ruby
-    ~~~
-
- 6. Apply the Kustomization:
-
-    ~~~bash
-    kubectl apply --context=gke_fullstaq-ruby_us-east4_fullstaq-ruby-autopilot -k ../kubernetes
-    ~~~
+> The API server itself is not deployed by this playbook. Code changes under `apiserver/` are built and deployed by the Github Actions workflows in `.github/workflows/` when pushed to `main`. They package a tarball, attach it to a GitHub Release, and trigger `POST /admin/upgrade_apiserver` on the live host.

docs/infrastructure-as-code.md

@@ -3,19 +3,19 @@
 We define as much infrastructure as possible in the form of code, using:
 
  * [Terraform](https://terraform.io)
- * Kubernetes YAML, managed with [Kustomize](https://kustomize.io/)
+ * [Ansible](https://www.ansible.com/)
  * Github Actions
 
 The infrastructure-as-code is stored in the following directories:
 
- * `terraform/` — Infrastructure administered by [Infra Maintainers](roles.md), except for resources inside Kubernetes. Most of the infrastructure is defined here.
+ * `terraform/` — Infrastructure administered by [Infra Maintainers](roles.md). Most of the cloud-side infrastructure is defined here.
 
- * `terraform-hisec/` — Infrastructure administered by [Infra Owners](roles.md). This covers for example resources in the `fullstaq-ruby-hisec` Google Cloud project.
+ * `terraform-hisec/` — Infrastructure administered by [Infra Owners](roles.md). This covers for example sensitive resources such as the GPG signing key in Azure Key Vault, and the high-security Terraform state backend.
 
-   Because we don't expect the infrastructure in this directory to change very often, we've chosen — for security reasons — not to run Terraform in a CI/CD pipeline. This way we don't have to worry about the security of the CI/CD pipeline's service account. Instead, an [Infra Owner](roles.md) runs Terraform manually, using that person's personal Google Cloud credentials.
+   Because we don't expect the infrastructure in this directory to change very often, we've chosen — for security reasons — not to run Terraform in a CI/CD pipeline. This way we don't have to worry about the security of any CI/CD pipeline credentials. Instead, an [Infra Owner](roles.md) runs Terraform manually, using their personal cloud credentials.
 
- * `kubernetes/` — Kubernetes resources administered by [Infra Maintainers](roles.md).
+ * `ansible/` — Configuration of the backend VM (Caddy, the API server, Prometheus, and OS hardening). Administered by [Infra Maintainers](roles.md) and applied manually; see [Deployment guide](deploy.md).
 
- * `.github/workflows/apiserver.yml` — Deploys the API server.
+ * `.github/workflows/` — Github Actions workflows that together test the infrastructure code, and build and deploy the API server.
 
 Note that not all infrastructure can, or (for security reasons) should, be managed via code. Learn more at [Infrastructure bootstrapping](infrastructure-bootstrapping.md).

docs/infrastructure-bootstrapping.md

@@ -1,6 +1,6 @@
 # Infrastructure bootstrapping
 
-We try to codify infrastructure as much as possible using Terraform and Kubernetes YAML. However:
+We try to codify infrastructure as much as possible using Terraform and Ansible. However:
 
 - Not everything _can_ be automated. For example, we need to setup Azure Blob Storage for storing Terraform state, before we can use Terraform.
 - Not everything _should_ be automated. For example, the `fullstaq-ruby-hisec` project contains such sensitive data, that giving access to CI/CD systems would pose a security risk.
@@ -148,6 +148,8 @@ Register the domain `fullstaqruby.org`. Configure it to use the Azure DNS zone.
 
 ## Step 11: Run initial Ansible
 
+The playbook deploys the API server from the latest `apiserver-*` release of the [fullstaq-ruby/infra](https://github.com/fullstaq-ruby/infra/releases) repo, so at least one such release must exist.
+
 Make sure the Azure CLI is logged in, then:
 
 ```bash
@@ -156,7 +158,21 @@ ansible-playbook -i hosts.ini -v main.yml
 cd ..
 ```
 
-## Step 12: Populate Github Actions secrets and variables
+## Step 12: Deploy the API server
+
+The Ansible run in step 11 installs `apiserver-deployer`, which downloads and activates the latest API server release from the [fullstaq-ruby/infra](https://github.com/fullstaq-ruby/infra/releases) repo. Verify on the VM that the API server is running:
+
+```bash
+sudo systemctl status apiserver-deployer apiserver
+```
+
+If it isn't, install and start the latest release manually:
+
+```bash
+sudo systemctl restart apiserver-deployer apiserver
+```
+
+## Step 13: Populate Github Actions secrets and variables
 
 In the [fullstaq-ruby/server-edition](https://github.com/fullstaq-ruby/server-edition/settings/secrets) repo, create the following environments:
 
@@ -186,6 +202,6 @@ Create these environment-specific variables:
 - `AZURE_CLIENT_ID` ('test' environment): fetch using `pushd terraform-hisec >/dev/null && terraform show -json | jq -r '.values.root_module.resources[] | select(.address == "azuread_application.server-edition-github-ci-test") | .values.application_id'; popd >/dev/null`
 - `AZURE_CLIENT_ID` ('deploy' environment): fetch using `pushd terraform-hisec >/dev/null && terraform show -json | jq -r '.values.root_module.resources[] | select(.address == "azuread_application.server-edition-github-ci-deploy") | .values.application_id'; popd >/dev/null`
 
-## Step 13: Onboard everybody
+## Step 14: Onboard everybody
 
 Onboard everybody in the [members list](members.md) according to the [onboarding instructions](onboarding.md).