feat: Helm chart + deploy runbook/SLOs for hosted environments

.github/workflows/helm.yml

@@ -0,0 +1,39 @@
+name: helm
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'deploy/helm/**'
+      - '.github/workflows/helm.yml'
+  pull_request:
+    paths:
+      - 'deploy/helm/**'
+      - '.github/workflows/helm.yml'
+
+jobs:
+  chart:
+    name: lint + template
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+
+      - uses: azure/setup-helm@v4
+        with:
+          version: v3.16.3
+
+      - name: helm lint
+        run: |
+          helm lint deploy/helm/flakemetry \
+            --set database.url='postgresql://u:p@db:5432/flakemetry?schema=public' \
+            --set auth.secret='ci-placeholder-secret-value-0001'
+
+      - name: helm template (renders all resources)
+        run: |
+          helm template ci deploy/helm/flakemetry \
+            --set database.url='postgresql://u:p@db:5432/flakemetry?schema=public' \
+            --set auth.secret='ci-placeholder-secret-value-0001' \
+            --set ai.enabled=true \
+            --set ingress.enabled=true \
+            > /tmp/rendered.yaml
+          test "$(grep -c '^kind:' /tmp/rendered.yaml)" -ge 10

.prettierignore

@@ -5,3 +5,4 @@ coverage
 .next
 pnpm-lock.yaml
 *.md
+deploy/helm/**/templates/

README.md

@@ -83,6 +83,10 @@ an OAuth app with callback `http://localhost:3000/api/auth/callback/github` and
 `AUTH_GITHUB_ID` / `AUTH_GITHUB_SECRET` in `.env`. The first account to sign in adopts the seeded
 workspace.
 
+For a horizontally scaled hosted environment, [`deploy/`](deploy) ships a Helm chart
+(stateless `api`/`worker`/`web` with autoscaling, a migration hook, and ingress) plus an
+operations [runbook](deploy/RUNBOOK.md) with SLOs — see the [deploy guide](deploy/README.md).
+
 ## See it in 60 seconds
 
 Load the demo dataset — one project's worth of history with a stable test, two flaky tests, and a

apps/docs/guide/self-hosting.md

@@ -53,6 +53,12 @@ hardening story.
 
 ## Production deployment
 
-The compose stack is aimed at local and small self-hosted use. Helm charts and Terraform
-modules for a horizontally scaled hosted environment are tracked on the
-[roadmap board](https://github.com/users/AKogut/projects/14).
+The compose stack is aimed at local and small self-hosted use. For a horizontally scaled
+hosted environment there is a **Helm chart** in
+[`deploy/helm/flakemetry`](https://github.com/AKogut/flakemetry/tree/main/deploy/helm/flakemetry):
+stateless `api`/`worker`/`web` with HorizontalPodAutoscalers, a pre-install migration hook,
+and ingress — running against a managed Postgres and object store. The
+[deploy guide](https://github.com/AKogut/flakemetry/blob/main/deploy/README.md) walks the
+path from zero to a running environment, and the
+[runbook](https://github.com/AKogut/flakemetry/blob/main/deploy/RUNBOOK.md) covers SLOs,
+scaling, and upgrades.

deploy/README.md

@@ -0,0 +1,67 @@
+# Deploying Flakemetry
+
+`docker compose up` (see the root README) is the path for local and small self-hosted use.
+This directory is the path for a **hosted, horizontally scaled** environment: a Helm chart
+for Kubernetes plus an operations runbook.
+
+- [`helm/flakemetry`](helm/flakemetry) — the chart (api, worker, web, migrations, HPAs, ingress).
+- [`helm/example-values.yaml`](helm/example-values.yaml) — a documented production values file.
+- [`RUNBOOK.md`](RUNBOOK.md) — SLOs, scaling, upgrade/rollback, and symptom→action runbook.
+
+## From zero to a running environment
+
+Flakemetry needs a managed **Postgres** and an S3-compatible **object store**. The chart
+never bundles a database — production uses your managed services. The queue is a Postgres
+table, so there is no separate broker.
+
+### 1. Provision dependencies
+
+- A Postgres 16 database (with the `pgvector` extension for AI RCA).
+- An S3 bucket (or compatible) plus access credentials.
+
+### 2. Publish the images
+
+The chart references four images (`flakemetry-api`, `flakemetry-worker`, `flakemetry-web`,
+`flakemetry-migrate`), built from the repository [`Dockerfile`](../Dockerfile) targets. Build
+and push them to your registry, then set `image.registry` / `image.repository` / `image.tag`.
+
+```bash
+for target in api worker web migrate; do
+  docker build --target "$target" -t ghcr.io/akogut/flakemetry-$target:v0.1.0 .
+  docker push ghcr.io/akogut/flakemetry-$target:v0.1.0
+done
+```
+
+### 3. Configure values
+
+Copy [`helm/example-values.yaml`](helm/example-values.yaml) and fill in the database URL,
+object-store credentials, `auth.secret` (`openssl rand -base64 32`), GitHub OAuth app, and
+your hostnames. Inject secrets from your secret manager, or point `existingSecret` at a
+pre-created Kubernetes Secret with the expected keys and set nothing sensitive in values.
+
+### 4. Install
+
+```bash
+helm upgrade --install flakemetry deploy/helm/flakemetry \
+  -n flakemetry --create-namespace \
+  -f my-values.yaml
+```
+
+Migrations run automatically as a pre-install hook before the app pods start. When the
+release is ready, the dashboard is on your `ingress.web.host` and the ingest API on
+`ingress.api.host`. Create a project and its ingest token in the dashboard, point your
+reporters at the ingest host, and runs start flowing.
+
+## What scales, and how
+
+`api` and `worker` are stateless and ship with CPU-target HorizontalPodAutoscalers, so
+ingestion and processing scale independently with load. The durable Postgres queue lets the
+worker fleet lag under a spike and catch up without dropping data. See
+[`RUNBOOK.md`](RUNBOOK.md) for scaling guidance and SLOs.
+
+## Not yet here
+
+Terraform modules for the managed dependencies and reference OTel dashboards for the
+platform's own telemetry are tracked as a follow-up on the
+[roadmap](https://github.com/users/AKogut/projects/14). Today the documented path is the
+Helm chart against managed Postgres and object storage.

deploy/RUNBOOK.md

@@ -0,0 +1,108 @@
+# Flakemetry operations runbook
+
+Operating a hosted or serious self-hosted Flakemetry instance: what to run, what to watch,
+and what to do when it breaks. Deployment is via the [Helm chart](helm/flakemetry), against
+a managed Postgres and an S3-compatible object store.
+
+## Architecture recap
+
+Three workloads, one governing constraint — **ingestion never blocks CI**:
+
+- **api** — validates and enqueues runs, returns `202` immediately. Stateless; scales
+  horizontally behind an HPA.
+- **worker** — drains the Postgres-backed queue (`SKIP LOCKED`) and runs identity, flaky
+  scoring, signature clustering, and AI RCA. Stateless; scales horizontally.
+- **web** — Next.js dashboard and query API. Stateless.
+
+Managed dependencies: **Postgres** (relational + JSONB, pgvector for RCA) and an
+**object store** (artifacts). The queue is a table in Postgres, so there is no separate
+broker to run.
+
+## Service level objectives
+
+| SLO                          | Target                        | Measured by                                   |
+| ---------------------------- | ----------------------------- | --------------------------------------------- |
+| Ingestion availability       | 99.9% of `POST /v1/ingest`    | non-5xx responses / total                     |
+| Ingestion latency            | p99 `< 300ms` (enqueue only)  | api request duration                          |
+| Processing lag               | p95 run processed `< 60s`     | worker dequeue-to-complete                    |
+| Dashboard availability       | 99.5%                         | non-5xx on web health + key queries           |
+| Data durability              | no acknowledged run lost      | queue depth vs. processed count reconciliation |
+
+The ingestion SLO is the important one: the `202` contract means a CI pipeline must never
+wait on Flakemetry. Everything downstream (scoring, RCA) is allowed to lag under load and
+catch up.
+
+## Error budget policy
+
+- Ingestion availability burns from a **0.1%** monthly budget. If a rolling 1-hour burn
+  would exhaust more than 5% of the month's budget, page.
+- Processing lag is a **latency** objective, not availability: sustained lag drains no
+  budget as long as the queue is draining. Alert (do not page) when p95 lag exceeds 60s for
+  10 minutes; page only if the queue depth is monotonically increasing for 30 minutes
+  (workers not keeping up — see below).
+
+## Scaling
+
+Ingestion and processing scale independently:
+
+- **api** and **worker** ship with HPAs (CPU-target) in the chart. Ingestion spikes with CI
+  volume; processing spikes with backlog. Because the queue is durable, the worker fleet can
+  lag and recover without data loss.
+- Raise `worker.autoscaling.maxReplicas` when queue depth is the bottleneck; raise
+  `api.autoscaling.maxReplicas` when ingestion latency is. Watch Postgres connection count
+  as you scale workers — each worker holds a small pool, so cap replicas below the
+  database's `max_connections` (or front it with a pooler such as PgBouncer).
+
+## Common operations
+
+### Deploy / upgrade
+
+```bash
+helm upgrade --install flakemetry deploy/helm/flakemetry \
+  -n flakemetry --create-namespace \
+  -f deploy/helm/example-values.yaml
+```
+
+Migrations run as a **pre-install/pre-upgrade hook** (`prisma migrate deploy`) before the
+new pods roll. Migrations are additive by design, so a rolling upgrade never requires
+downtime. The hook never seeds — production data is untouched.
+
+### Roll back
+
+```bash
+helm rollback flakemetry -n flakemetry
+```
+
+Because migrations are additive and backward-compatible, rolling the app back one release
+is safe without a schema rollback.
+
+### Inspect the queue
+
+Processing lag almost always traces to the queue. Check depth and the oldest unprocessed
+job in Postgres, and confirm workers are running and not crash-looping
+(`kubectl get pods -l app.kubernetes.io/component=worker`).
+
+## Runbook: symptoms → actions
+
+| Symptom                              | Likely cause                          | Action                                                                 |
+| ------------------------------------ | ------------------------------------- | ---------------------------------------------------------------------- |
+| `202` latency rising, 5xx on ingest  | api saturated or DB writes slow       | Confirm api HPA scaled; check DB CPU/connections; raise api max replicas |
+| Queue depth climbing, lag rising     | worker fleet undersized or stuck      | Check worker pods healthy; raise worker max replicas; check DB pool     |
+| Migrations hook failing on upgrade   | bad migration or DB unreachable       | Read the migrate Job logs; fix connectivity; migrations are idempotent  |
+| Dashboard 5xx                        | web ↔ DB or web ↔ object store issue  | Check web pod logs; verify S3 public endpoint reachable from browser    |
+| Artifacts 404 in the UI              | wrong `storage.publicEndpoint`        | Set a browser-reachable public endpoint; re-check bucket CORS           |
+| AI RCA silent                        | budget spent or provider misconfigured | Expected once the daily token budget is spent; else check provider/key  |
+
+## Backups & disaster recovery
+
+Postgres is the system of record — the queue, all history, identities, and scores live
+there. Object storage holds only artifacts (screenshots, video, traces), which are
+regenerable. Back up Postgres with your managed provider's point-in-time recovery; artifact
+loss degrades the UI but never the intelligence. Full backup/DR automation is tracked on
+the [roadmap](https://github.com/users/AKogut/projects/14).
+
+## Platform observability
+
+Flakemetry is OpenTelemetry-native and should dogfood its own telemetry: api and worker
+export traces/metrics so ingestion latency, queue lag, and processing throughput are
+first-class dashboards. Reference OTel dashboards and alert rules are a tracked follow-up.

deploy/helm/example-values.yaml

@@ -0,0 +1,73 @@
+# Example production values for the Flakemetry Helm chart.
+#
+#   helm install flakemetry deploy/helm/flakemetry -n flakemetry --create-namespace \
+#     -f deploy/helm/example-values.yaml
+#
+# Secrets below are placeholders — inject the real values from your secret
+# manager (or point `existingSecret` at a pre-created Secret) rather than
+# committing them.
+
+image:
+  registry: ghcr.io
+  repository: akogut
+  tag: v0.1.0
+
+database:
+  url: postgresql://flakemetry:CHANGE_ME@db.internal:5432/flakemetry?schema=public
+
+storage:
+  bucket: flakemetry-artifacts
+  endpoint: https://s3.us-east-1.amazonaws.com
+  publicEndpoint: https://artifacts.flakemetry.example.com
+  region: us-east-1
+  accessKeyId: CHANGE_ME
+  secretAccessKey: CHANGE_ME
+  forcePathStyle: false
+
+auth:
+  secret: CHANGE_ME_openssl_rand_base64_32
+  url: https://flakemetry.example.com
+  githubId: CHANGE_ME
+  githubSecret: CHANGE_ME
+
+ai:
+  enabled: true
+  provider: claude
+  model: claude-sonnet-4-5
+  apiKey: CHANGE_ME
+  dailyTokenBudget: '2000000'
+
+api:
+  autoscaling:
+    enabled: true
+    minReplicas: 3
+    maxReplicas: 20
+
+worker:
+  autoscaling:
+    enabled: true
+    minReplicas: 2
+    maxReplicas: 12
+
+web:
+  autoscaling:
+    enabled: true
+    minReplicas: 2
+    maxReplicas: 6
+
+ingress:
+  enabled: true
+  className: nginx
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+  web:
+    host: flakemetry.example.com
+  api:
+    host: ingest.flakemetry.example.com
+  tls:
+    - secretName: flakemetry-web-tls
+      hosts:
+        - flakemetry.example.com
+    - secretName: flakemetry-api-tls
+      hosts:
+        - ingest.flakemetry.example.com

deploy/helm/flakemetry/.helmignore

@@ -0,0 +1,7 @@
+.DS_Store
+.git
+.gitignore
+*.tmp
+*.orig
+*.swp
+ci/

deploy/helm/flakemetry/Chart.yaml

@@ -0,0 +1,16 @@
+apiVersion: v2
+name: flakemetry
+description: OpenTelemetry-native test intelligence — ingestion API, processing worker, and dashboard
+type: application
+version: 0.1.0
+appVersion: '0.1.0'
+home: https://akogut.github.io/flakemetry/
+sources:
+  - https://github.com/AKogut/flakemetry
+maintainers:
+  - name: Andrii Kohut
+keywords:
+  - testing
+  - observability
+  - opentelemetry
+  - flaky-tests

deploy/helm/flakemetry/templates/NOTES.txt

@@ -0,0 +1,24 @@
+Flakemetry {{ .Chart.AppVersion }} is installed as release "{{ .Release.Name }}".
+
+Components:
+  api     — ingestion API   ({{ if .Values.api.autoscaling.enabled }}HPA {{ .Values.api.autoscaling.minReplicas }}–{{ .Values.api.autoscaling.maxReplicas }}{{ else }}{{ .Values.api.replicaCount }} replicas{{ end }})
+  worker  — processing      ({{ if .Values.worker.autoscaling.enabled }}HPA {{ .Values.worker.autoscaling.minReplicas }}–{{ .Values.worker.autoscaling.maxReplicas }}{{ else }}{{ .Values.worker.replicaCount }} replicas{{ end }})
+  web     — dashboard        ({{ if .Values.web.autoscaling.enabled }}HPA {{ .Values.web.autoscaling.minReplicas }}–{{ .Values.web.autoscaling.maxReplicas }}{{ else }}{{ .Values.web.replicaCount }} replicas{{ end }})
+
+Database migrations ran as a pre-install/pre-upgrade hook (they never seed).
+
+{{- if .Values.ingress.enabled }}
+
+Dashboard:  https://{{ .Values.ingress.web.host }}{{ .Values.ingress.web.path }}
+Ingest API: https://{{ .Values.ingress.api.host }}{{ .Values.ingress.api.path }}
+{{- else }}
+
+No ingress is enabled. Reach the dashboard by port-forwarding:
+
+  kubectl --namespace {{ .Release.Namespace }} port-forward svc/{{ include "flakemetry.fullname" . }}-web 3000:{{ .Values.web.service.port }}
+
+then open http://localhost:3000
+{{- end }}
+
+Point your reporters at the ingest API and create a project token in the dashboard.
+See the runbook: https://github.com/AKogut/flakemetry/blob/main/deploy/RUNBOOK.md