diff --git a/.github/workflows/helm.yml b/.github/workflows/helm.yml new file mode 100644 index 0000000..5dd618c --- /dev/null +++ b/.github/workflows/helm.yml @@ -0,0 +1,39 @@ +name: helm + +on: + push: + branches: [main] + paths: + - 'deploy/helm/**' + - '.github/workflows/helm.yml' + pull_request: + paths: + - 'deploy/helm/**' + - '.github/workflows/helm.yml' + +jobs: + chart: + name: lint + template + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + + - uses: azure/setup-helm@v4 + with: + version: v3.16.3 + + - name: helm lint + run: | + helm lint deploy/helm/flakemetry \ + --set database.url='postgresql://u:p@db:5432/flakemetry?schema=public' \ + --set auth.secret='ci-placeholder-secret-value-0001' + + - name: helm template (renders all resources) + run: | + helm template ci deploy/helm/flakemetry \ + --set database.url='postgresql://u:p@db:5432/flakemetry?schema=public' \ + --set auth.secret='ci-placeholder-secret-value-0001' \ + --set ai.enabled=true \ + --set ingress.enabled=true \ + > /tmp/rendered.yaml + test "$(grep -c '^kind:' /tmp/rendered.yaml)" -ge 10 diff --git a/.prettierignore b/.prettierignore index 4a9c11c..fa156b3 100644 --- a/.prettierignore +++ b/.prettierignore @@ -5,3 +5,4 @@ coverage .next pnpm-lock.yaml *.md +deploy/helm/**/templates/ diff --git a/README.md b/README.md index c5696f0..c4c6537 100644 --- a/README.md +++ b/README.md @@ -83,6 +83,10 @@ an OAuth app with callback `http://localhost:3000/api/auth/callback/github` and `AUTH_GITHUB_ID` / `AUTH_GITHUB_SECRET` in `.env`. The first account to sign in adopts the seeded workspace. +For a horizontally scaled hosted environment, [`deploy/`](deploy) ships a Helm chart +(stateless `api`/`worker`/`web` with autoscaling, a migration hook, and ingress) plus an +operations [runbook](deploy/RUNBOOK.md) with SLOs — see the [deploy guide](deploy/README.md). + ## See it in 60 seconds Load the demo dataset — one project's worth of history with a stable test, two flaky tests, and a diff --git a/apps/docs/guide/self-hosting.md b/apps/docs/guide/self-hosting.md index bfb1a1f..ec62a37 100644 --- a/apps/docs/guide/self-hosting.md +++ b/apps/docs/guide/self-hosting.md @@ -53,6 +53,12 @@ hardening story. ## Production deployment -The compose stack is aimed at local and small self-hosted use. Helm charts and Terraform -modules for a horizontally scaled hosted environment are tracked on the -[roadmap board](https://github.com/users/AKogut/projects/14). +The compose stack is aimed at local and small self-hosted use. For a horizontally scaled +hosted environment there is a **Helm chart** in +[`deploy/helm/flakemetry`](https://github.com/AKogut/flakemetry/tree/main/deploy/helm/flakemetry): +stateless `api`/`worker`/`web` with HorizontalPodAutoscalers, a pre-install migration hook, +and ingress — running against a managed Postgres and object store. The +[deploy guide](https://github.com/AKogut/flakemetry/blob/main/deploy/README.md) walks the +path from zero to a running environment, and the +[runbook](https://github.com/AKogut/flakemetry/blob/main/deploy/RUNBOOK.md) covers SLOs, +scaling, and upgrades. diff --git a/deploy/README.md b/deploy/README.md new file mode 100644 index 0000000..202ac5d --- /dev/null +++ b/deploy/README.md @@ -0,0 +1,67 @@ +# Deploying Flakemetry + +`docker compose up` (see the root README) is the path for local and small self-hosted use. +This directory is the path for a **hosted, horizontally scaled** environment: a Helm chart +for Kubernetes plus an operations runbook. + +- [`helm/flakemetry`](helm/flakemetry) — the chart (api, worker, web, migrations, HPAs, ingress). +- [`helm/example-values.yaml`](helm/example-values.yaml) — a documented production values file. +- [`RUNBOOK.md`](RUNBOOK.md) — SLOs, scaling, upgrade/rollback, and symptom→action runbook. + +## From zero to a running environment + +Flakemetry needs a managed **Postgres** and an S3-compatible **object store**. The chart +never bundles a database — production uses your managed services. The queue is a Postgres +table, so there is no separate broker. + +### 1. Provision dependencies + +- A Postgres 16 database (with the `pgvector` extension for AI RCA). +- An S3 bucket (or compatible) plus access credentials. + +### 2. Publish the images + +The chart references four images (`flakemetry-api`, `flakemetry-worker`, `flakemetry-web`, +`flakemetry-migrate`), built from the repository [`Dockerfile`](../Dockerfile) targets. Build +and push them to your registry, then set `image.registry` / `image.repository` / `image.tag`. + +```bash +for target in api worker web migrate; do + docker build --target "$target" -t ghcr.io/akogut/flakemetry-$target:v0.1.0 . + docker push ghcr.io/akogut/flakemetry-$target:v0.1.0 +done +``` + +### 3. Configure values + +Copy [`helm/example-values.yaml`](helm/example-values.yaml) and fill in the database URL, +object-store credentials, `auth.secret` (`openssl rand -base64 32`), GitHub OAuth app, and +your hostnames. Inject secrets from your secret manager, or point `existingSecret` at a +pre-created Kubernetes Secret with the expected keys and set nothing sensitive in values. + +### 4. Install + +```bash +helm upgrade --install flakemetry deploy/helm/flakemetry \ + -n flakemetry --create-namespace \ + -f my-values.yaml +``` + +Migrations run automatically as a pre-install hook before the app pods start. When the +release is ready, the dashboard is on your `ingress.web.host` and the ingest API on +`ingress.api.host`. Create a project and its ingest token in the dashboard, point your +reporters at the ingest host, and runs start flowing. + +## What scales, and how + +`api` and `worker` are stateless and ship with CPU-target HorizontalPodAutoscalers, so +ingestion and processing scale independently with load. The durable Postgres queue lets the +worker fleet lag under a spike and catch up without dropping data. See +[`RUNBOOK.md`](RUNBOOK.md) for scaling guidance and SLOs. + +## Not yet here + +Terraform modules for the managed dependencies and reference OTel dashboards for the +platform's own telemetry are tracked as a follow-up on the +[roadmap](https://github.com/users/AKogut/projects/14). Today the documented path is the +Helm chart against managed Postgres and object storage. diff --git a/deploy/RUNBOOK.md b/deploy/RUNBOOK.md new file mode 100644 index 0000000..5f3d30f --- /dev/null +++ b/deploy/RUNBOOK.md @@ -0,0 +1,108 @@ +# Flakemetry operations runbook + +Operating a hosted or serious self-hosted Flakemetry instance: what to run, what to watch, +and what to do when it breaks. Deployment is via the [Helm chart](helm/flakemetry), against +a managed Postgres and an S3-compatible object store. + +## Architecture recap + +Three workloads, one governing constraint — **ingestion never blocks CI**: + +- **api** — validates and enqueues runs, returns `202` immediately. Stateless; scales + horizontally behind an HPA. +- **worker** — drains the Postgres-backed queue (`SKIP LOCKED`) and runs identity, flaky + scoring, signature clustering, and AI RCA. Stateless; scales horizontally. +- **web** — Next.js dashboard and query API. Stateless. + +Managed dependencies: **Postgres** (relational + JSONB, pgvector for RCA) and an +**object store** (artifacts). The queue is a table in Postgres, so there is no separate +broker to run. + +## Service level objectives + +| SLO | Target | Measured by | +| ---------------------------- | ----------------------------- | --------------------------------------------- | +| Ingestion availability | 99.9% of `POST /v1/ingest` | non-5xx responses / total | +| Ingestion latency | p99 `< 300ms` (enqueue only) | api request duration | +| Processing lag | p95 run processed `< 60s` | worker dequeue-to-complete | +| Dashboard availability | 99.5% | non-5xx on web health + key queries | +| Data durability | no acknowledged run lost | queue depth vs. processed count reconciliation | + +The ingestion SLO is the important one: the `202` contract means a CI pipeline must never +wait on Flakemetry. Everything downstream (scoring, RCA) is allowed to lag under load and +catch up. + +## Error budget policy + +- Ingestion availability burns from a **0.1%** monthly budget. If a rolling 1-hour burn + would exhaust more than 5% of the month's budget, page. +- Processing lag is a **latency** objective, not availability: sustained lag drains no + budget as long as the queue is draining. Alert (do not page) when p95 lag exceeds 60s for + 10 minutes; page only if the queue depth is monotonically increasing for 30 minutes + (workers not keeping up — see below). + +## Scaling + +Ingestion and processing scale independently: + +- **api** and **worker** ship with HPAs (CPU-target) in the chart. Ingestion spikes with CI + volume; processing spikes with backlog. Because the queue is durable, the worker fleet can + lag and recover without data loss. +- Raise `worker.autoscaling.maxReplicas` when queue depth is the bottleneck; raise + `api.autoscaling.maxReplicas` when ingestion latency is. Watch Postgres connection count + as you scale workers — each worker holds a small pool, so cap replicas below the + database's `max_connections` (or front it with a pooler such as PgBouncer). + +## Common operations + +### Deploy / upgrade + +```bash +helm upgrade --install flakemetry deploy/helm/flakemetry \ + -n flakemetry --create-namespace \ + -f deploy/helm/example-values.yaml +``` + +Migrations run as a **pre-install/pre-upgrade hook** (`prisma migrate deploy`) before the +new pods roll. Migrations are additive by design, so a rolling upgrade never requires +downtime. The hook never seeds — production data is untouched. + +### Roll back + +```bash +helm rollback flakemetry -n flakemetry +``` + +Because migrations are additive and backward-compatible, rolling the app back one release +is safe without a schema rollback. + +### Inspect the queue + +Processing lag almost always traces to the queue. Check depth and the oldest unprocessed +job in Postgres, and confirm workers are running and not crash-looping +(`kubectl get pods -l app.kubernetes.io/component=worker`). + +## Runbook: symptoms → actions + +| Symptom | Likely cause | Action | +| ------------------------------------ | ------------------------------------- | ---------------------------------------------------------------------- | +| `202` latency rising, 5xx on ingest | api saturated or DB writes slow | Confirm api HPA scaled; check DB CPU/connections; raise api max replicas | +| Queue depth climbing, lag rising | worker fleet undersized or stuck | Check worker pods healthy; raise worker max replicas; check DB pool | +| Migrations hook failing on upgrade | bad migration or DB unreachable | Read the migrate Job logs; fix connectivity; migrations are idempotent | +| Dashboard 5xx | web ↔ DB or web ↔ object store issue | Check web pod logs; verify S3 public endpoint reachable from browser | +| Artifacts 404 in the UI | wrong `storage.publicEndpoint` | Set a browser-reachable public endpoint; re-check bucket CORS | +| AI RCA silent | budget spent or provider misconfigured | Expected once the daily token budget is spent; else check provider/key | + +## Backups & disaster recovery + +Postgres is the system of record — the queue, all history, identities, and scores live +there. Object storage holds only artifacts (screenshots, video, traces), which are +regenerable. Back up Postgres with your managed provider's point-in-time recovery; artifact +loss degrades the UI but never the intelligence. Full backup/DR automation is tracked on +the [roadmap](https://github.com/users/AKogut/projects/14). + +## Platform observability + +Flakemetry is OpenTelemetry-native and should dogfood its own telemetry: api and worker +export traces/metrics so ingestion latency, queue lag, and processing throughput are +first-class dashboards. Reference OTel dashboards and alert rules are a tracked follow-up. diff --git a/deploy/helm/example-values.yaml b/deploy/helm/example-values.yaml new file mode 100644 index 0000000..dd589e9 --- /dev/null +++ b/deploy/helm/example-values.yaml @@ -0,0 +1,73 @@ +# Example production values for the Flakemetry Helm chart. +# +# helm install flakemetry deploy/helm/flakemetry -n flakemetry --create-namespace \ +# -f deploy/helm/example-values.yaml +# +# Secrets below are placeholders — inject the real values from your secret +# manager (or point `existingSecret` at a pre-created Secret) rather than +# committing them. + +image: + registry: ghcr.io + repository: akogut + tag: v0.1.0 + +database: + url: postgresql://flakemetry:CHANGE_ME@db.internal:5432/flakemetry?schema=public + +storage: + bucket: flakemetry-artifacts + endpoint: https://s3.us-east-1.amazonaws.com + publicEndpoint: https://artifacts.flakemetry.example.com + region: us-east-1 + accessKeyId: CHANGE_ME + secretAccessKey: CHANGE_ME + forcePathStyle: false + +auth: + secret: CHANGE_ME_openssl_rand_base64_32 + url: https://flakemetry.example.com + githubId: CHANGE_ME + githubSecret: CHANGE_ME + +ai: + enabled: true + provider: claude + model: claude-sonnet-4-5 + apiKey: CHANGE_ME + dailyTokenBudget: '2000000' + +api: + autoscaling: + enabled: true + minReplicas: 3 + maxReplicas: 20 + +worker: + autoscaling: + enabled: true + minReplicas: 2 + maxReplicas: 12 + +web: + autoscaling: + enabled: true + minReplicas: 2 + maxReplicas: 6 + +ingress: + enabled: true + className: nginx + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod + web: + host: flakemetry.example.com + api: + host: ingest.flakemetry.example.com + tls: + - secretName: flakemetry-web-tls + hosts: + - flakemetry.example.com + - secretName: flakemetry-api-tls + hosts: + - ingest.flakemetry.example.com diff --git a/deploy/helm/flakemetry/.helmignore b/deploy/helm/flakemetry/.helmignore new file mode 100644 index 0000000..49bcb72 --- /dev/null +++ b/deploy/helm/flakemetry/.helmignore @@ -0,0 +1,7 @@ +.DS_Store +.git +.gitignore +*.tmp +*.orig +*.swp +ci/ diff --git a/deploy/helm/flakemetry/Chart.yaml b/deploy/helm/flakemetry/Chart.yaml new file mode 100644 index 0000000..ac726c7 --- /dev/null +++ b/deploy/helm/flakemetry/Chart.yaml @@ -0,0 +1,16 @@ +apiVersion: v2 +name: flakemetry +description: OpenTelemetry-native test intelligence — ingestion API, processing worker, and dashboard +type: application +version: 0.1.0 +appVersion: '0.1.0' +home: https://akogut.github.io/flakemetry/ +sources: + - https://github.com/AKogut/flakemetry +maintainers: + - name: Andrii Kohut +keywords: + - testing + - observability + - opentelemetry + - flaky-tests diff --git a/deploy/helm/flakemetry/templates/NOTES.txt b/deploy/helm/flakemetry/templates/NOTES.txt new file mode 100644 index 0000000..ac0e11a --- /dev/null +++ b/deploy/helm/flakemetry/templates/NOTES.txt @@ -0,0 +1,24 @@ +Flakemetry {{ .Chart.AppVersion }} is installed as release "{{ .Release.Name }}". + +Components: + api — ingestion API ({{ if .Values.api.autoscaling.enabled }}HPA {{ .Values.api.autoscaling.minReplicas }}–{{ .Values.api.autoscaling.maxReplicas }}{{ else }}{{ .Values.api.replicaCount }} replicas{{ end }}) + worker — processing ({{ if .Values.worker.autoscaling.enabled }}HPA {{ .Values.worker.autoscaling.minReplicas }}–{{ .Values.worker.autoscaling.maxReplicas }}{{ else }}{{ .Values.worker.replicaCount }} replicas{{ end }}) + web — dashboard ({{ if .Values.web.autoscaling.enabled }}HPA {{ .Values.web.autoscaling.minReplicas }}–{{ .Values.web.autoscaling.maxReplicas }}{{ else }}{{ .Values.web.replicaCount }} replicas{{ end }}) + +Database migrations ran as a pre-install/pre-upgrade hook (they never seed). + +{{- if .Values.ingress.enabled }} + +Dashboard: https://{{ .Values.ingress.web.host }}{{ .Values.ingress.web.path }} +Ingest API: https://{{ .Values.ingress.api.host }}{{ .Values.ingress.api.path }} +{{- else }} + +No ingress is enabled. Reach the dashboard by port-forwarding: + + kubectl --namespace {{ .Release.Namespace }} port-forward svc/{{ include "flakemetry.fullname" . }}-web 3000:{{ .Values.web.service.port }} + +then open http://localhost:3000 +{{- end }} + +Point your reporters at the ingest API and create a project token in the dashboard. +See the runbook: https://github.com/AKogut/flakemetry/blob/main/deploy/RUNBOOK.md diff --git a/deploy/helm/flakemetry/templates/_helpers.tpl b/deploy/helm/flakemetry/templates/_helpers.tpl new file mode 100644 index 0000000..b44a21f --- /dev/null +++ b/deploy/helm/flakemetry/templates/_helpers.tpl @@ -0,0 +1,84 @@ +{{- define "flakemetry.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{- define "flakemetry.fullname" -}} +{{- if .Values.fullnameOverride -}} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- $name := default .Chart.Name .Values.nameOverride -}} +{{- if contains $name .Release.Name -}} +{{- .Release.Name | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +{{- end -}} +{{- end -}} + +{{- define "flakemetry.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{- define "flakemetry.labels" -}} +helm.sh/chart: {{ include "flakemetry.chart" . }} +{{ include "flakemetry.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end -}} + +{{- define "flakemetry.selectorLabels" -}} +app.kubernetes.io/name: {{ include "flakemetry.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "flakemetry.serviceAccountName" -}} +{{- if .Values.serviceAccount.create -}} +{{- default (include "flakemetry.fullname" .) .Values.serviceAccount.name -}} +{{- else -}} +{{- default "default" .Values.serviceAccount.name -}} +{{- end -}} +{{- end -}} + +{{- define "flakemetry.secretName" -}} +{{- if .Values.existingSecret -}} +{{- .Values.existingSecret -}} +{{- else -}} +{{- printf "%s-secrets" (include "flakemetry.fullname" .) -}} +{{- end -}} +{{- end -}} + +{{/* component image ref: (dict "root" $ "name" "flakemetry-api" "tag" "") */}} +{{- define "flakemetry.image" -}} +{{- $root := .root -}} +{{- $tag := .tag | default $root.Values.image.tag -}} +{{- printf "%s/%s/%s:%s" $root.Values.image.registry $root.Values.image.repository .name $tag -}} +{{- end -}} + +{{/* Shared env referencing the secret; used by api, worker, web. */}} +{{- define "flakemetry.commonEnv" -}} +- name: DATABASE_URL + valueFrom: + secretKeyRef: + name: {{ include "flakemetry.secretName" . }} + key: database-url +- name: FLAKEMETRY_S3_BUCKET + value: {{ .Values.storage.bucket | quote }} +- name: FLAKEMETRY_S3_ENDPOINT + value: {{ .Values.storage.endpoint | quote }} +- name: FLAKEMETRY_S3_REGION + value: {{ .Values.storage.region | quote }} +- name: FLAKEMETRY_S3_FORCE_PATH_STYLE + value: {{ .Values.storage.forcePathStyle | quote }} +- name: FLAKEMETRY_S3_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: {{ include "flakemetry.secretName" . }} + key: s3-access-key-id +- name: FLAKEMETRY_S3_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: {{ include "flakemetry.secretName" . }} + key: s3-secret-access-key +{{- end -}} diff --git a/deploy/helm/flakemetry/templates/api-deployment.yaml b/deploy/helm/flakemetry/templates/api-deployment.yaml new file mode 100644 index 0000000..0878088 --- /dev/null +++ b/deploy/helm/flakemetry/templates/api-deployment.yaml @@ -0,0 +1,62 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "flakemetry.fullname" . }}-api + labels: + {{- include "flakemetry.labels" . | nindent 4 }} + app.kubernetes.io/component: api +spec: + {{- if not .Values.api.autoscaling.enabled }} + replicas: {{ .Values.api.replicaCount }} + {{- end }} + selector: + matchLabels: + {{- include "flakemetry.selectorLabels" . | nindent 6 }} + app.kubernetes.io/component: api + template: + metadata: + labels: + {{- include "flakemetry.selectorLabels" . | nindent 8 }} + app.kubernetes.io/component: api + {{- with .Values.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + serviceAccountName: {{ include "flakemetry.serviceAccountName" . }} + {{- with .Values.pullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + securityContext: + {{- toYaml .Values.podSecurityContext | nindent 8 }} + containers: + - name: api + image: {{ include "flakemetry.image" (dict "root" $ "name" .Values.api.image.name "tag" .Values.api.image.tag) }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + securityContext: + {{- toYaml .Values.securityContext | nindent 12 }} + ports: + - name: http + containerPort: {{ .Values.api.port }} + env: + - name: PORT + value: {{ .Values.api.port | quote }} + {{- include "flakemetry.commonEnv" . | nindent 12 }} + {{- with .Values.api.extraEnv }} + {{- toYaml . | nindent 12 }} + {{- end }} + readinessProbe: + httpGet: + path: /health + port: http + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: + path: /health + port: http + initialDelaySeconds: 15 + periodSeconds: 20 + resources: + {{- toYaml .Values.api.resources | nindent 12 }} diff --git a/deploy/helm/flakemetry/templates/api-hpa.yaml b/deploy/helm/flakemetry/templates/api-hpa.yaml new file mode 100644 index 0000000..44b068c --- /dev/null +++ b/deploy/helm/flakemetry/templates/api-hpa.yaml @@ -0,0 +1,23 @@ +{{- if .Values.api.autoscaling.enabled }} +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: {{ include "flakemetry.fullname" . }}-api + labels: + {{- include "flakemetry.labels" . | nindent 4 }} + app.kubernetes.io/component: api +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: {{ include "flakemetry.fullname" . }}-api + minReplicas: {{ .Values.api.autoscaling.minReplicas }} + maxReplicas: {{ .Values.api.autoscaling.maxReplicas }} + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: {{ .Values.api.autoscaling.targetCPUUtilizationPercentage }} +{{- end }} diff --git a/deploy/helm/flakemetry/templates/api-service.yaml b/deploy/helm/flakemetry/templates/api-service.yaml new file mode 100644 index 0000000..403c09c --- /dev/null +++ b/deploy/helm/flakemetry/templates/api-service.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ include "flakemetry.fullname" . }}-api + labels: + {{- include "flakemetry.labels" . | nindent 4 }} + app.kubernetes.io/component: api +spec: + type: {{ .Values.api.service.type }} + ports: + - name: http + port: {{ .Values.api.service.port }} + targetPort: http + selector: + {{- include "flakemetry.selectorLabels" . | nindent 4 }} + app.kubernetes.io/component: api diff --git a/deploy/helm/flakemetry/templates/ingress.yaml b/deploy/helm/flakemetry/templates/ingress.yaml new file mode 100644 index 0000000..9093d37 --- /dev/null +++ b/deploy/helm/flakemetry/templates/ingress.yaml @@ -0,0 +1,41 @@ +{{- if .Values.ingress.enabled }} +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: {{ include "flakemetry.fullname" . }} + labels: + {{- include "flakemetry.labels" . | nindent 4 }} + {{- with .Values.ingress.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + {{- with .Values.ingress.className }} + ingressClassName: {{ . }} + {{- end }} + {{- with .Values.ingress.tls }} + tls: + {{- toYaml . | nindent 4 }} + {{- end }} + rules: + - host: {{ .Values.ingress.web.host | quote }} + http: + paths: + - path: {{ .Values.ingress.web.path }} + pathType: {{ .Values.ingress.web.pathType }} + backend: + service: + name: {{ include "flakemetry.fullname" . }}-web + port: + number: {{ .Values.web.service.port }} + - host: {{ .Values.ingress.api.host | quote }} + http: + paths: + - path: {{ .Values.ingress.api.path }} + pathType: {{ .Values.ingress.api.pathType }} + backend: + service: + name: {{ include "flakemetry.fullname" . }}-api + port: + number: {{ .Values.api.service.port }} +{{- end }} diff --git a/deploy/helm/flakemetry/templates/migrate-job.yaml b/deploy/helm/flakemetry/templates/migrate-job.yaml new file mode 100644 index 0000000..a6ee9c0 --- /dev/null +++ b/deploy/helm/flakemetry/templates/migrate-job.yaml @@ -0,0 +1,44 @@ +{{- if .Values.migrate.enabled }} +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ include "flakemetry.fullname" . }}-migrate + labels: + {{- include "flakemetry.labels" . | nindent 4 }} + app.kubernetes.io/component: migrate + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: '-5' + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +spec: + backoffLimit: {{ .Values.migrate.backoffLimit }} + template: + metadata: + labels: + {{- include "flakemetry.selectorLabels" . | nindent 8 }} + app.kubernetes.io/component: migrate + spec: + restartPolicy: Never + serviceAccountName: {{ include "flakemetry.serviceAccountName" . }} + {{- with .Values.pullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + securityContext: + {{- toYaml .Values.podSecurityContext | nindent 8 }} + containers: + - name: migrate + image: {{ include "flakemetry.image" (dict "root" $ "name" .Values.migrateImage.name "tag" .Values.migrateImage.tag) }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + securityContext: + {{- toYaml .Values.securityContext | nindent 12 }} + command: ['sh', '-c', 'corepack pnpm exec prisma migrate deploy'] + env: + - name: DATABASE_URL + valueFrom: + secretKeyRef: + name: {{ include "flakemetry.secretName" . }} + key: database-url + resources: + {{- toYaml .Values.migrate.resources | nindent 12 }} +{{- end }} diff --git a/deploy/helm/flakemetry/templates/secret.yaml b/deploy/helm/flakemetry/templates/secret.yaml new file mode 100644 index 0000000..3c3a535 --- /dev/null +++ b/deploy/helm/flakemetry/templates/secret.yaml @@ -0,0 +1,17 @@ +{{- if not .Values.existingSecret }} +apiVersion: v1 +kind: Secret +metadata: + name: {{ include "flakemetry.secretName" . }} + labels: + {{- include "flakemetry.labels" . | nindent 4 }} +type: Opaque +stringData: + database-url: {{ required "database.url is required (or set existingSecret)" .Values.database.url | quote }} + s3-access-key-id: {{ .Values.storage.accessKeyId | quote }} + s3-secret-access-key: {{ .Values.storage.secretAccessKey | quote }} + auth-secret: {{ required "auth.secret is required (or set existingSecret)" .Values.auth.secret | quote }} + auth-github-id: {{ .Values.auth.githubId | quote }} + auth-github-secret: {{ .Values.auth.githubSecret | quote }} + ai-api-key: {{ .Values.ai.apiKey | quote }} +{{- end }} diff --git a/deploy/helm/flakemetry/templates/serviceaccount.yaml b/deploy/helm/flakemetry/templates/serviceaccount.yaml new file mode 100644 index 0000000..3c88b26 --- /dev/null +++ b/deploy/helm/flakemetry/templates/serviceaccount.yaml @@ -0,0 +1,12 @@ +{{- if .Values.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "flakemetry.serviceAccountName" . }} + labels: + {{- include "flakemetry.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/deploy/helm/flakemetry/templates/web-deployment.yaml b/deploy/helm/flakemetry/templates/web-deployment.yaml new file mode 100644 index 0000000..2ef5b83 --- /dev/null +++ b/deploy/helm/flakemetry/templates/web-deployment.yaml @@ -0,0 +1,109 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "flakemetry.fullname" . }}-web + labels: + {{- include "flakemetry.labels" . | nindent 4 }} + app.kubernetes.io/component: web +spec: + {{- if not .Values.web.autoscaling.enabled }} + replicas: {{ .Values.web.replicaCount }} + {{- end }} + selector: + matchLabels: + {{- include "flakemetry.selectorLabels" . | nindent 6 }} + app.kubernetes.io/component: web + template: + metadata: + labels: + {{- include "flakemetry.selectorLabels" . | nindent 8 }} + app.kubernetes.io/component: web + {{- with .Values.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + serviceAccountName: {{ include "flakemetry.serviceAccountName" . }} + {{- with .Values.pullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + securityContext: + {{- toYaml .Values.podSecurityContext | nindent 8 }} + containers: + - name: web + image: {{ include "flakemetry.image" (dict "root" $ "name" .Values.web.image.name "tag" .Values.web.image.tag) }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + securityContext: + {{- toYaml .Values.securityContext | nindent 12 }} + ports: + - name: http + containerPort: {{ .Values.web.port }} + env: + - name: PORT + value: {{ .Values.web.port | quote }} + - name: HOSTNAME + value: '0.0.0.0' + - name: DATABASE_URL + valueFrom: + secretKeyRef: + name: {{ include "flakemetry.secretName" . }} + key: database-url + - name: AUTH_URL + value: {{ .Values.auth.url | quote }} + - name: AUTH_TRUST_HOST + value: {{ .Values.auth.trustHost | quote }} + - name: AUTH_SECRET + valueFrom: + secretKeyRef: + name: {{ include "flakemetry.secretName" . }} + key: auth-secret + - name: AUTH_GITHUB_ID + valueFrom: + secretKeyRef: + name: {{ include "flakemetry.secretName" . }} + key: auth-github-id + - name: AUTH_GITHUB_SECRET + valueFrom: + secretKeyRef: + name: {{ include "flakemetry.secretName" . }} + key: auth-github-secret + - name: FLAKEMETRY_S3_BUCKET + value: {{ .Values.storage.bucket | quote }} + - name: FLAKEMETRY_S3_ENDPOINT + value: {{ .Values.storage.publicEndpoint | default .Values.storage.endpoint | quote }} + - name: FLAKEMETRY_S3_REGION + value: {{ .Values.storage.region | quote }} + - name: FLAKEMETRY_S3_FORCE_PATH_STYLE + value: {{ .Values.storage.forcePathStyle | quote }} + - name: FLAKEMETRY_S3_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: {{ include "flakemetry.secretName" . }} + key: s3-access-key-id + - name: FLAKEMETRY_S3_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: {{ include "flakemetry.secretName" . }} + key: s3-secret-access-key + {{- if .Values.ai.enabled }} + - name: FLAKEMETRY_AI_RCA + value: 'true' + {{- end }} + {{- with .Values.web.extraEnv }} + {{- toYaml . | nindent 12 }} + {{- end }} + readinessProbe: + httpGet: + path: /health + port: http + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: + path: /health + port: http + initialDelaySeconds: 15 + periodSeconds: 20 + resources: + {{- toYaml .Values.web.resources | nindent 12 }} diff --git a/deploy/helm/flakemetry/templates/web-service.yaml b/deploy/helm/flakemetry/templates/web-service.yaml new file mode 100644 index 0000000..0794a8c --- /dev/null +++ b/deploy/helm/flakemetry/templates/web-service.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ include "flakemetry.fullname" . }}-web + labels: + {{- include "flakemetry.labels" . | nindent 4 }} + app.kubernetes.io/component: web +spec: + type: {{ .Values.web.service.type }} + ports: + - name: http + port: {{ .Values.web.service.port }} + targetPort: http + selector: + {{- include "flakemetry.selectorLabels" . | nindent 4 }} + app.kubernetes.io/component: web diff --git a/deploy/helm/flakemetry/templates/worker-deployment.yaml b/deploy/helm/flakemetry/templates/worker-deployment.yaml new file mode 100644 index 0000000..15b3f5e --- /dev/null +++ b/deploy/helm/flakemetry/templates/worker-deployment.yaml @@ -0,0 +1,68 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "flakemetry.fullname" . }}-worker + labels: + {{- include "flakemetry.labels" . | nindent 4 }} + app.kubernetes.io/component: worker +spec: + {{- if not .Values.worker.autoscaling.enabled }} + replicas: {{ .Values.worker.replicaCount }} + {{- end }} + selector: + matchLabels: + {{- include "flakemetry.selectorLabels" . | nindent 6 }} + app.kubernetes.io/component: worker + template: + metadata: + labels: + {{- include "flakemetry.selectorLabels" . | nindent 8 }} + app.kubernetes.io/component: worker + {{- with .Values.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + serviceAccountName: {{ include "flakemetry.serviceAccountName" . }} + {{- with .Values.pullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + securityContext: + {{- toYaml .Values.podSecurityContext | nindent 8 }} + containers: + - name: worker + image: {{ include "flakemetry.image" (dict "root" $ "name" .Values.worker.image.name "tag" .Values.worker.image.tag) }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + securityContext: + {{- toYaml .Values.securityContext | nindent 12 }} + env: + - name: POLL_INTERVAL_MS + value: {{ .Values.worker.pollIntervalMs | quote }} + - name: FLAKEMETRY_ARTIFACT_RETENTION_DAYS + value: {{ .Values.worker.retention.artifactDays | quote }} + - name: FLAKEMETRY_EXECUTION_RETENTION_DAYS + value: {{ .Values.worker.retention.executionDays | quote }} + {{- include "flakemetry.commonEnv" . | nindent 12 }} + {{- if .Values.ai.enabled }} + - name: FLAKEMETRY_AI_RCA + value: 'true' + - name: FLAKEMETRY_AI_PROVIDER + value: {{ .Values.ai.provider | quote }} + - name: FLAKEMETRY_AI_MODEL + value: {{ .Values.ai.model | quote }} + - name: FLAKEMETRY_AI_ENDPOINT + value: {{ .Values.ai.endpoint | quote }} + - name: FLAKEMETRY_AI_DAILY_TOKEN_BUDGET + value: {{ .Values.ai.dailyTokenBudget | quote }} + - name: FLAKEMETRY_AI_API_KEY + valueFrom: + secretKeyRef: + name: {{ include "flakemetry.secretName" . }} + key: ai-api-key + {{- end }} + {{- with .Values.worker.extraEnv }} + {{- toYaml . | nindent 12 }} + {{- end }} + resources: + {{- toYaml .Values.worker.resources | nindent 12 }} diff --git a/deploy/helm/flakemetry/templates/worker-hpa.yaml b/deploy/helm/flakemetry/templates/worker-hpa.yaml new file mode 100644 index 0000000..894a18d --- /dev/null +++ b/deploy/helm/flakemetry/templates/worker-hpa.yaml @@ -0,0 +1,23 @@ +{{- if .Values.worker.autoscaling.enabled }} +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: {{ include "flakemetry.fullname" . }}-worker + labels: + {{- include "flakemetry.labels" . | nindent 4 }} + app.kubernetes.io/component: worker +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: {{ include "flakemetry.fullname" . }}-worker + minReplicas: {{ .Values.worker.autoscaling.minReplicas }} + maxReplicas: {{ .Values.worker.autoscaling.maxReplicas }} + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: {{ .Values.worker.autoscaling.targetCPUUtilizationPercentage }} +{{- end }} diff --git a/deploy/helm/flakemetry/values.yaml b/deploy/helm/flakemetry/values.yaml new file mode 100644 index 0000000..2e5ee5c --- /dev/null +++ b/deploy/helm/flakemetry/values.yaml @@ -0,0 +1,158 @@ +nameOverride: '' +fullnameOverride: '' + +image: + registry: ghcr.io + repository: akogut + # Applied to every component image unless overridden per component. + tag: latest + pullPolicy: IfNotPresent +pullSecrets: [] + +serviceAccount: + create: true + name: '' + annotations: {} + +podAnnotations: {} +podSecurityContext: + runAsNonRoot: true + runAsUser: 1000 + fsGroup: 1000 +securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: false + capabilities: + drop: ['ALL'] + +# Managed dependencies. Flakemetry expects an external managed Postgres and an +# S3-compatible object store — the chart never bundles a database. Provide the +# connection details here (or via existingSecret) and the chart wires them into +# every component. +database: + # postgresql://user:password@host:5432/flakemetry?schema=public + url: '' +storage: + bucket: flakemetry-artifacts + endpoint: '' + publicEndpoint: '' + region: us-east-1 + accessKeyId: '' + secretAccessKey: '' + forcePathStyle: false + +auth: + # Generate with: openssl rand -base64 32 + secret: '' + url: '' + trustHost: true + githubId: '' + githubSecret: '' + +ai: + enabled: false + provider: '' + model: '' + apiKey: '' + endpoint: '' + dailyTokenBudget: '' + +# Supply all sensitive values through a pre-existing Secret instead of the +# values above. When set, the chart references this Secret and creates none. +existingSecret: '' + +# Runs `prisma migrate deploy` as a pre-install/pre-upgrade hook before the +# app pods roll. It never seeds — production data is never touched. +migrate: + enabled: true + backoffLimit: 3 + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + memory: 512Mi + +api: + replicaCount: 2 + port: 4000 + image: + name: flakemetry-api + tag: '' + service: + type: ClusterIP + port: 80 + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + memory: 512Mi + autoscaling: + enabled: true + minReplicas: 2 + maxReplicas: 10 + targetCPUUtilizationPercentage: 70 + extraEnv: [] + +worker: + replicaCount: 2 + image: + name: flakemetry-worker + tag: '' + pollIntervalMs: 1000 + resources: + requests: + cpu: 250m + memory: 384Mi + limits: + memory: 768Mi + autoscaling: + enabled: true + minReplicas: 2 + maxReplicas: 8 + targetCPUUtilizationPercentage: 75 + retention: + artifactDays: 90 + executionDays: 90 + extraEnv: [] + +web: + replicaCount: 2 + port: 3000 + image: + name: flakemetry-web + tag: '' + service: + type: ClusterIP + port: 80 + resources: + requests: + cpu: 250m + memory: 384Mi + limits: + memory: 768Mi + autoscaling: + enabled: true + minReplicas: 2 + maxReplicas: 6 + targetCPUUtilizationPercentage: 70 + extraEnv: [] + +migrateImage: + name: flakemetry-migrate + tag: '' + +ingress: + enabled: false + className: '' + annotations: {} + web: + host: flakemetry.example.com + path: / + pathType: Prefix + api: + host: ingest.flakemetry.example.com + path: / + pathType: Prefix + tls: []