Skip to content

docs: refresh infra docs for post-Hetzner architecture - #57

Open
abtreece wants to merge 8 commits into
fullstaq-ruby:mainfrom
abtreece:docs/post-hetzner-refresh
Open

abtreece wants to merge 8 commits into
fullstaq-ruby:mainfrom
abtreece:docs/post-hetzner-refresh

Conversation

@abtreece

@abtreece abtreece commented Apr 29, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #55.

Summary

Refreshes the infra docs to describe the current infrastructure: a single Hetzner VM running Caddy and the Sinatra/Puma API server, configured by Ansible. The old docs described the pre-July-2024 architecture (GKE Autopilot, Nginx Ingress, Cloud Run apiserver). Text is written for readers who only know the current setup, and avoids repeating details that live in Terraform/Ansible (expiry periods, hardening specifics) so it doesn't drift from the code.

Files changed

  • docs/infrastructure-overview.md: rewritten section by section against current IaC.
    • Google Cloud project: a single project (fsruby-server-edition2); the project resource lives in terraform-hisec/, resources inside it in terraform/.
    • CI/CD authentication: replaces the two old GCP service-account sections, split per caller:
      • server-edition → Google Cloud via Workload Identity Federation.
      • server-edition → Azure via Federated Identity Credentials.
      • infra → API server via a GitHub OIDC JWT (audience backend.fullstaqruby.org) sent to POST /admin/upgrade_apiserver. The infra workflows don't authenticate to GCP or Azure.
    • API server: runs under systemd on the VM. Releases are built by the "Test and build" workflow and published by the "Deploy" workflow, which calls /admin/upgrade_apiserver. The separate deployer exists both so the server can be replaced without a gap and for security (the API server can't modify its own code).
    • Caddy: apt. and yum. vhosts only; both proxy /admin/* to the API server socket. Links the server-edition APT/YUM repo design doc for latest_version.txt.
    • VM (Hetzner): runs Caddy and the API server, configured entirely by Ansible. Forward DNS is Terraform-managed; the PTR record is set manually.
    • CI artifacts / cache: artifacts go to the GCS bucket. The Azure artifacts container is provisioned for a future migration but not in use. The Azure cache container is in use and expires on last access time.
    • GPG private key: stored in fsruby2infraowners; CI reads it via GitHub OIDC.
    • Container registry: section dropped (no registry resources are managed in this repo).
  • apiserver/README.md: rewritten. It previously described Google Cloud hosting. Now covers hosting on the VM behind Caddy, the three endpoints, GitHub OIDC auth, and the Test and build → Deploy → apiserver-deployer flow.
  • docs/infrastructure-bootstrapping.md: intro mentions Terraform + Ansible instead of Kubernetes. New Step 12: Deploy the API server (later steps renumbered), and Step 11 notes that at least one apiserver-* release must exist.
  • docs/deploy.md: replaces the gcloud container clusters get-credentials + kubectl apply steps with an ansible-playbook step, and notes that API server changes deploy via .github/workflows/.
  • docs/infrastructure-as-code.md: drops Kustomize and kubernetes/; adds Ansible and ansible/; references .github/workflows/.
  • .editorconfig: removes the duplicate [config.ru] block (the tab/4 one).

Diagram (pending)

Per review, the hand-drawn diagram is kept. The earlier Mermaid replacement has been reverted, so docs/infrastructure-overview.drawio.svg, docs/editing-diagrams.md, and their links in README.md and docs/pull_request_template.md are unchanged from main. The drawio itself still depicts the pre-Hetzner architecture; updating it is pending.

Verification

  • eclint check $(git ls-files) passes.
  • Claims were checked against current IaC: terraform/{dns,gcloud_auth,repo_buckets,ci_storage,variables}.tf, terraform-hisec/{gcloud_project,key_vault}.tf, ansible/main.yml, ansible/files/{Caddyfile,apiserver.service,apiserver-deployer.service,apiserver-deployer.sh,apiserver-sudoers}, apiserver/app.rb, .github/workflows/{test-and-build,deploy}.yml.

Out of scope

@abtreece
abtreece force-pushed the docs/post-hetzner-refresh branch from 66e915d to cfbe8a1 Compare April 29, 2026 02:56
@abtreece
abtreece requested a review from FooBarWidget May 15, 2026 03:30
@abtreece
abtreece force-pushed the docs/post-hetzner-refresh branch from cfbe8a1 to 85b83bf Compare May 15, 2026 03:48
@FooBarWidget

Copy link
Copy Markdown
Member

I'll have a good look. So far my first impression is that the new diagram lacks a lot of detail that was in the older diagram. I'm also not sure whether a detailed but automatically rendered diagram is still readable compared to a manually drawn one.

Comment thread docs/infrastructure-bootstrapping.md Resolved
Comment thread docs/infrastructure-as-code.md Resolved
Comment thread docs/deploy.md Resolved
Comment thread docs/infrastructure-overview.md Resolved
Comment thread docs/infrastructure-overview.md Resolved
Comment thread docs/infrastructure-overview.md Resolved
Comment thread docs/infrastructure-overview.md Resolved
Comment thread docs/infrastructure-overview.md Resolved
Comment thread docs/infrastructure-overview.md Resolved
Comment thread docs/infrastructure-overview.md Outdated
## Google Cloud service account for Infrastructure CI/CD
The following diagram shows the major infrastructure components and how they relate to each other. The role that administers each component is given in the section heading below.

```mermaid

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rendered the Mermaid diagram but I don't think it's easier to read (nor better-looking) than a hand-drawn one, so I prefer an update based on the hand-drawn diagram.

Comment thread docs/infrastructure-overview.md Resolved
Comment thread docs/infrastructure-overview.md Resolved
abtreece added a commit to abtreece/infra that referenced this pull request May 22, 2026
- Use interpolated bucket/storage account names instead of Terraform vars
- Spell out Workload Identity Federation on first use in the section
- Note that the Azure Blob artifacts container is provisioned but not yet
  used by CI; artifacts still go to the GCS bucket. The cache container
  in the same storage account is actively used (sccache).

Addresses FooBarWidget review on PR fullstaq-ruby#57.
abtreece added a commit to abtreece/infra that referenced this pull request May 22, 2026
The previous README described a Google Cloud Run-hosted API server
authenticated with Google Cloud identity tokens. Post-Hetzner, the
service runs as a systemd-managed Puma process on the backend host
behind Caddy on a Unix socket, authenticates via GitHub Actions OIDC,
and is deployed via the .github/workflows/apiserver.yml workflow plus
the apiserver-deployer oneshot unit. The README now reflects all four.

Addresses FooBarWidget review on PR fullstaq-ruby#57.
abtreece added a commit to abtreece/infra that referenced this pull request May 22, 2026
Documents the new APT/YUM archive infrastructure introduced in this PR:
the two new public-read GCS buckets, the deliberate absence of CI write
access (frozen-mirror invariant enforced in IAM), the Azure DNS zones
and apex NS delegation for the archive subdomains, and the 404 fallback
behavior in query-latest-repo-versions.rb that lets the web server
start cleanly before the first migration runs.

Addresses FooBarWidget's note on PR fullstaq-ruby#57 that fullstaq-ruby#47's changes should ship
with their own documentation.
Closes fullstaq-ruby#55.

Brings docs/ in line with the post-July-2024 architecture (single
Hetzner VM running Caddy + Sinatra/Puma API server + Prometheus,
provisioned by Ansible), replacing references to the previous
GKE Autopilot + Nginx Ingress + Cloud Run apiserver setup.

Files changed:

- docs/infrastructure-overview.md — rewritten section by section.
  Every claim is grounded in current IaC. The two GCP-service-account
  sections are folded into a single "CI/CD authentication" section
  that splits per-caller: server-edition uses GCP WIF (APT/YUM repo
  buckets + GCS CI artifacts bucket) and Azure Federated Identity
  Credentials (Azure Blob CI artifacts + CI cache + Key Vault GPG
  key); infra repo's apiserver workflow only mints a GitHub OIDC JWT
  (audience backend.fullstaqruby.org) and POSTs to
  /admin/upgrade_apiserver — it does not authenticate to GCP or
  Azure APIs. The Caddy section is corrected: there is no
  backend.fullstaqruby.org vhost; both apt. and yum. vhosts handle
  /admin/* via reverse_proxy to the apiserver Unix socket. The
  "Google Cloud projects" claim of two projects is corrected — there
  is one project, fsruby-server-edition2, provisioned by
  terraform-hisec/gcloud_project.tf and populated by terraform/; the
  hisec/non-hisec separation lives at the Terraform-state and
  access-group layer. Container registry section dropped (no
  registry resources are managed in this repo). Key Vault name
  uses the templated form ${var.key_vault_prefix}infraowners
  (currently fsruby2infraowners). CI artifacts/cache split is now
  explicit (artifacts dual-cloud, cache Azure-only). VM section
  distinguishes Terraform-managed forward DNS from the
  manually-set Hetzner PTR record.

- docs/infrastructure-overview.drawio.svg — deleted. Replaced by an
  inline Mermaid diagram in infrastructure-overview.md so future
  diagram changes are reviewable as text diffs.

- docs/editing-diagrams.md — deleted (no longer needed without the
  drawio round-trip).

- docs/deploy.md — replaces the gcloud-clusters/kubectl steps with
  a single ansible-playbook step matching bootstrapping Step 11.
  Adds a callout that apiserver code changes deploy via the
  GitHub Actions workflow.

- docs/infrastructure-as-code.md — drops Kustomize and the
  kubernetes/ directory bullet; adds Ansible to the tool list and
  an ansible/ directory bullet.

- docs/infrastructure-bootstrapping.md — intro updated to mention
  Terraform + Ansible (not Kubernetes/Kustomize); the rest of the
  file already reflected the post-migration setup.

- docs/pull_request_template.md — diagram-update checkbox now
  points to the Mermaid block instead of the deleted drawio file.

- README.md — drops the link to the deleted editing-diagrams.md.

- .editorconfig — removes the duplicate [config.ru] block (the
  tab/4 one); only the correct space/2 rule remains.

Note: the "Github CI bot account" section is kept as-is. Retiring
that PAT-based bot is already tracked in fullstaq-ruby#18 and is therefore out
of scope here.
- Use interpolated bucket/storage account names instead of Terraform vars
- Spell out Workload Identity Federation on first use in the section
- Note that the Azure Blob artifacts container is provisioned but not yet
  used by CI; artifacts still go to the GCS bucket. The cache container
  in the same storage account is actively used (sccache).

Addresses FooBarWidget review on PR fullstaq-ruby#57.
The previous README described a Google Cloud Run-hosted API server
authenticated with Google Cloud identity tokens. Post-Hetzner, the
service runs as a systemd-managed Puma process on the backend host
behind Caddy on a Unix socket, authenticates via GitHub Actions OIDC,
and is deployed via the .github/workflows/apiserver.yml workflow plus
the apiserver-deployer oneshot unit. The README now reflects all four.

Addresses FooBarWidget review on PR fullstaq-ruby#57.
abtreece added a commit to abtreece/infra that referenced this pull request Sep 30, 2026
Documents the new APT/YUM archive infrastructure introduced in this PR:
the two new public-read GCS buckets, the deliberate absence of CI write
access (frozen-mirror invariant enforced in IAM), the Azure DNS zones
and apex NS delegation for the archive subdomains, and the 404 fallback
behavior in query-latest-repo-versions.rb that lets the web server
start cleanly before the first migration runs.

Addresses FooBarWidget's note on PR fullstaq-ruby#57 that fullstaq-ruby#47's changes should ship
with their own documentation.
@abtreece
abtreece force-pushed the docs/post-hetzner-refresh branch from 109b1b4 to dcb6700 Compare September 30, 2026 03:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update documentation to reflect current architecture

2 participants