From e00201811fe98733033ecefbd10514cb3f0b5184 Mon Sep 17 00:00:00 2001 From: Krstan Vjestica Date: Wed, 19 Aug 2026 09:10:00 +0200 Subject: [PATCH 1/7] chore: pin the OPA and OpenTelemetry Collector images Both tracked floating tags, so the same chart version installed different code over time. Neither pin changes what runs today. The collector's `latest` genuinely moves -- it was rebuilt on 2026-08-18 -- and 0.159.0 is the release it points at now. OPA's `latest-rootless` turned out to be the opposite problem: upstream stopped publishing `-rootless` variants after 0.58.0, so it has been a frozen orphan since 2023-10-26. Pinning it to 0.58.0-rootless makes that visible instead of implying the image is current. The TODO this replaces suggested 0.70.0-rootless, which does not exist. Moving off 0.58.0 means choosing a different image, and OPA 1.x defaults to Rego v1, so it needs the backend's generated policies checked first -- a separate decision, not a version bump. --- values.yaml | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/values.yaml b/values.yaml index 8f5384a..8c48834 100644 --- a/values.yaml +++ b/values.yaml @@ -43,16 +43,26 @@ images: tag: 7-alpine opa: repository: openpolicyagent/opa - # needtofix L15: pin to an immutable version tag (e.g. 0.70.0-rootless) - # before shipping — a floating tag drifts and undermines supply-chain trust. - tag: latest-rootless + # Pinned. This is the same image `latest-rootless` has served since + # 2023-10-26 -- upstream stopped publishing `-rootless` variants after + # 0.58.0, so that tag is a frozen orphan rather than a moving target, and + # this pin changes nothing about what runs. (The earlier TODO here suggested + # 0.70.0-rootless; that tag does not exist.) + # + # Moving off 0.58.0 means choosing a different image, not bumping a number, + # and OPA 1.x defaults to Rego v1 -- so it needs the policies the backend + # pushes to be checked first. Deliberately a separate decision. + tag: 0.58.0-rootless nginx: repository: nginx tag: 1.27-alpine otelCollector: repository: otel/opentelemetry-collector-contrib - # needtofix L15: pin to an immutable version tag (e.g. 0.110.0) before shipping. - tag: latest + # Pinned. Unlike OPA's, this `latest` really does move -- it was rebuilt on + # 2026-08-18 -- so the same chart version was installing different collector + # builds over time. 0.159.0 is the release `latest` points at today, so this + # pin also changes nothing about what runs right now. + tag: 0.159.0 # Public ghcr.io images do not require pull secrets. # Override only when mirroring images to a private registry. From 8348453623f2d975fbe773957cc5d123351d061d Mon Sep 17 00:00:00 2001 From: Krstan Vjestica Date: Wed, 19 Aug 2026 10:35:00 +0200 Subject: [PATCH 2/7] fix: Redis ran with no authentication `redis-server --appendonly yes` and nothing else. Its Service is a ClusterIP, so every pod that can route to it could read and write the cache, the Celery task queue and the websocket channel layer -- or empty all three with one FLUSHALL. The NetworkPolicy that would have limited the blast radius is off by default, because k3s' flannel does not enforce policies, so a stock install had no control at either layer. `requirepass` is now always set, from a password generated on first install and reused on upgrade like the other infrastructure secrets. It is passed through the environment rather than through args, since an argument shows up in the pod spec, in `kubectl describe` and in the container's own /proc. The liveness and readiness probes authenticate via REDISCLI_AUTH, which redis-cli reads on its own, so the password stays off their command line too. `redis_settings.py` builds the URL with the password when REDIS_PASSWORD is set and without it when it is not, so the compose deployment -- where Redis is not published and this stays empty -- is unaffected. That file is one of the ones `make sync-from-deploy` copies from forail-deploy, so the same change belongs there or the next sync will revert it. --- files/settings/redis_settings.py | 20 +++++++++++++++++--- templates/_helpers.tpl | 5 +++++ templates/redis.yaml | 18 +++++++++++++++++- templates/secret.yaml | 2 ++ values.yaml | 5 +++++ 5 files changed, 46 insertions(+), 4 deletions(-) diff --git a/files/settings/redis_settings.py b/files/settings/redis_settings.py index f4e1b35..1a40e45 100644 --- a/files/settings/redis_settings.py +++ b/files/settings/redis_settings.py @@ -1,14 +1,28 @@ import os +from urllib.parse import quote _redis_host = os.environ.get('REDIS_HOST', 'redis') _redis_port = os.environ.get('REDIS_PORT', '6379') -BROKER_URL = f'redis://{_redis_host}:{_redis_port}/0' +# Redis has no authentication unless it is asked for. On a single-host compose +# deployment it is not published and this stays empty; in Kubernetes the chart +# sets requirepass and passes the value here, because a ClusterIP Service is +# reachable by every pod that can route to it -- which is enough to read the +# cache and the task queue, or to FLUSHALL them. +# +# Quoted, because a generated password may contain characters that would +# otherwise end the userinfo section of the URL. +_redis_password = os.environ.get('REDIS_PASSWORD', '') +_redis_auth = f':{quote(_redis_password, safe="")}@' if _redis_password else '' + +_redis_url = f'redis://{_redis_auth}{_redis_host}:{_redis_port}' + +BROKER_URL = f'{_redis_url}/0' CACHES = { 'default': { 'BACKEND': 'forail.main.cache.AWXRedisCache', - 'LOCATION': f'redis://{_redis_host}:{_redis_port}/1', + 'LOCATION': f'{_redis_url}/1', } } @@ -16,7 +30,7 @@ 'default': { 'BACKEND': 'channels_redis.core.RedisChannelLayer', 'CONFIG': { - 'hosts': [f'redis://{_redis_host}:{_redis_port}/0'], + 'hosts': [f'{_redis_url}/0'], 'capacity': 10000, 'group_expiry': 157784760, }, diff --git a/templates/_helpers.tpl b/templates/_helpers.tpl index 51c24c8..f040823 100644 --- a/templates/_helpers.tpl +++ b/templates/_helpers.tpl @@ -57,6 +57,11 @@ Wraps DB, Redis, secrets, OTel, admin into one block to avoid drift. value: forail-redis - name: REDIS_PORT value: "6379" +- name: REDIS_PASSWORD + valueFrom: + secretKeyRef: + name: forail-secrets + key: redisPassword - name: FORAIL_SECRET_KEY valueFrom: secretKeyRef: diff --git a/templates/redis.yaml b/templates/redis.yaml index b551a41..a875809 100644 --- a/templates/redis.yaml +++ b/templates/redis.yaml @@ -53,7 +53,23 @@ spec: containers: - name: redis image: "{{ .Values.images.redis.repository }}:{{ .Values.images.redis.tag }}" - args: ["redis-server", "--appendonly", "yes"] + # requirepass is read from the environment rather than written into + # args: an argument is visible in the pod spec, in `kubectl describe` + # and in the container's own /proc. + args: ["sh", "-c", "exec redis-server --appendonly yes --requirepass \"$REDIS_PASSWORD\""] + env: + - name: REDIS_PASSWORD + valueFrom: + secretKeyRef: + name: forail-secrets + key: redisPassword + # redis-cli picks the password up from REDISCLI_AUTH, so the probes + # below authenticate without putting it on their command line. + - name: REDISCLI_AUTH + valueFrom: + secretKeyRef: + name: forail-secrets + key: redisPassword ports: - name: redis containerPort: 6379 diff --git a/templates/secret.yaml b/templates/secret.yaml index 4a6f031..a404005 100644 --- a/templates/secret.yaml +++ b/templates/secret.yaml @@ -13,6 +13,7 @@ required. {{- $pgPass := .Values.secrets.postgresPassword | default (index $existingData "postgresPassword" | default "" | b64dec) | default (randAlphaNum 32) -}} {{- $secretKey := .Values.secrets.forailSecretKey | default (index $existingData "forailSecretKey" | default "" | b64dec) | default (randAlphaNum 50) -}} {{- $bcastSecret := .Values.secrets.forailBroadcastWebsocketSecret | default (index $existingData "forailBroadcastWebsocketSecret" | default "" | b64dec) | default (randAlphaNum 50) -}} +{{- $redisPass := .Values.secrets.redisPassword | default (index $existingData "redisPassword" | default "" | b64dec) | default (randAlphaNum 32) -}} {{- $adminPass := .Values.secrets.forailAdminPassword | default (index $existingData "forailAdminPassword" | default "" | b64dec) -}} {{- if not $adminPass -}} {{- fail "secrets.forailAdminPassword is required — set it, e.g. --set secrets.forailAdminPassword=\"$(openssl rand -base64 24)\" (see needtofix H3)" -}} @@ -29,4 +30,5 @@ stringData: postgresPassword: {{ $pgPass | quote }} forailSecretKey: {{ $secretKey | quote }} forailBroadcastWebsocketSecret: {{ $bcastSecret | quote }} + redisPassword: {{ $redisPass | quote }} forailAdminPassword: {{ $adminPass | quote }} diff --git a/values.yaml b/values.yaml index 8c48834..257a5a5 100644 --- a/values.yaml +++ b/values.yaml @@ -82,6 +82,11 @@ secrets: postgresPassword: "" forailSecretKey: "" forailBroadcastWebsocketSecret: "" + # Redis requirepass. A ClusterIP Service is reachable by every pod that can + # route to it, and an unauthenticated Redis holds the cache, the task queue + # and the websocket channel layer -- readable, writable, and one FLUSHALL from + # empty. Generated on first install and reused on upgrade, like the others. + redisPassword: "" # Required — no default; install fails if unset: forailAdminPassword: "" From 0913aea98e28853ae75cf41366f887de46d105e8 Mon Sep 17 00:00:00 2001 From: Krstan Vjestica Date: Wed, 19 Aug 2026 12:00:00 +0200 Subject: [PATCH 3/7] fix: enabling tenancy in the chart enforced nothing `forail.tenancyEnabled` was the only tenancy value the chart offered, and it maps to `TENANCY_ENABLED` -- the flag that turns on quotas, branding and isolation auditing. The flags that enforce a boundary, `TENANCY_RLS_ENABLED` and `TENANCY_STRICT_ISOLATION_ENABLED`, default to False in the backend and had no Helm value at all. An operator who set the one switch they were offered got a multi-tenant-looking install with no row-level security behind it. Adds `forail.tenancy.{rls,strictIsolation,rateLimiting}`, all passed through, and refuses the combination that caused the problem: `tenancyEnabled=true` with `rls=false` fails the render with an explanation rather than installing something that only looks isolated. RLS defaults to true, so turning tenancy on now scopes rows by default and the operator opts *out* rather than having to know to opt in. All three are ANDed with `tenancyEnabled`, so leaving them set while turning tenancy off does not leave stray flags on. Needs the matching backend change: only `TENANCY_ENABLED` was mirrored from the environment into the Setting registry, so these variables would otherwise be set and ignored. --- templates/_helpers.tpl | 9 +++++++++ values.yaml | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/templates/_helpers.tpl b/templates/_helpers.tpl index f040823..cb60f0e 100644 --- a/templates/_helpers.tpl +++ b/templates/_helpers.tpl @@ -101,8 +101,17 @@ Wraps DB, Redis, secrets, OTel, admin into one block to avoid drift. value: {{ .Values.forail.node.name | quote }} - name: FORAIL_NODE_TYPE value: {{ .Values.forail.node.type | quote }} +{{- if and .Values.forail.tenancyEnabled (not .Values.forail.tenancy.rls) }} +{{- fail "forail.tenancyEnabled=true requires forail.tenancy.rls=true — without row-level security the tenancy features run with no boundary behind them. Set --set forail.tenancy.rls=true, or turn tenancy off." }} +{{- end }} - name: TENANCY_ENABLED value: {{ .Values.forail.tenancyEnabled | quote }} +- name: TENANCY_RLS_ENABLED + value: {{ and .Values.forail.tenancyEnabled .Values.forail.tenancy.rls | quote }} +- name: TENANCY_STRICT_ISOLATION_ENABLED + value: {{ and .Values.forail.tenancyEnabled .Values.forail.tenancy.strictIsolation | quote }} +- name: TENANCY_RATE_LIMITING_ENABLED + value: {{ and .Values.forail.tenancyEnabled .Values.forail.tenancy.rateLimiting | quote }} - name: OTEL_ENABLED value: {{ .Values.forail.otel.enabled | quote }} - name: OTEL_EXPORTER_ENDPOINT diff --git a/values.yaml b/values.yaml index 257a5a5..a58f0f1 100644 --- a/values.yaml +++ b/values.yaml @@ -121,7 +121,26 @@ forail: node: name: forail-node type: hybrid + # ── Multi-tenancy ─────────────────────────────────────── + # tenancyEnabled alone used to be the only switch the chart offered, while the + # controls that actually enforce isolation defaulted to off in the backend and + # had no value here at all -- so turning it on gave the tenancy features + # (quotas, branding, isolation auditing) with nothing enforcing a boundary. + # + # The chart now refuses that combination: tenancy.enabled=true requires + # tenancy.rls=true. Set tenancy.strictIsolation=true to block cross-tenant + # access rather than only audit it. tenancyEnabled: false + tenancy: + # Row-Level Security. Postgres policies scope rows to the requesting + # tenant's organization; without it "multi-tenant" means only that the UI + # shows a tenant name. + rls: true + # False audits cross-tenant access and allows it; true returns 403. Still + # requires the organization's own tenant_isolation_strict flag. + strictIsolation: false + # Per-tenant API rate limiting. Needs Redis. + rateLimiting: false otel: enabled: true endpoint: "http://forail-otel-collector:4317" From 8684e402d562ae9f72e6b42424e46cef431cf8d4 Mon Sep 17 00:00:00 2001 From: Krstan Vjestica Date: Wed, 19 Aug 2026 14:15:00 +0200 Subject: [PATCH 4/7] fix: the assistant answered anyone who could reach its Service `FORAIL_ASSISTANT_CHAT_TOKEN` defaults to empty, which the assistant treats as "no authentication required" -- and the chart passed only the model name and log level, so enabling `assistant.enabled=true` gave a chat endpoint open to every pod that can route to its ClusterIP. There is no NetworkPolicy by default, so that is the whole cluster. Beyond unmetered use of the model and holding all four concurrency slots, the answers carry indexed documentation back out. The chart now always sets a token, generated on first install and reused on upgrade like the other infrastructure secrets. Whatever proxies `/assistant` to the Service must send it as `Authorization: Bearer `; the value is in the same Secret and both values.yaml and the deployment say how to read it. The chart still does not route the assistant through the Ingress. That stays deliberate -- nothing outside the cluster reaches it until an operator adds the path and configures the header, rather than the token being the only thing between the internet and the model. --- templates/forail-assistant.yaml | 14 ++++++++++++++ templates/secret.yaml | 2 ++ values.yaml | 7 +++++++ 3 files changed, 23 insertions(+) diff --git a/templates/forail-assistant.yaml b/templates/forail-assistant.yaml index f9aa35e..3e1d100 100644 --- a/templates/forail-assistant.yaml +++ b/templates/forail-assistant.yaml @@ -190,6 +190,20 @@ spec: value: {{ .Values.assistant.model | quote }} - name: FORAIL_ASSISTANT_LOG_LEVEL value: {{ .Values.assistant.logLevel | quote }} + # Without this the chat endpoint answers anyone who can reach the + # Service -- which is every pod in the cluster that can route to it, + # since there is no NetworkPolicy by default. Beyond unmetered use of + # the model, the answers carry indexed documentation back out. + # + # Whatever proxies /assistant to this Service has to send + # `Authorization: Bearer `; read it from the same Secret: + # kubectl -n {{ include "forail.namespace" . }} get secret forail-secrets \ + # -o jsonpath='{.data.assistantChatToken}' | base64 -d + - name: FORAIL_ASSISTANT_CHAT_TOKEN + valueFrom: + secretKeyRef: + name: forail-secrets + key: assistantChatToken # First boot waits for Ollama, pulls the model (gemma3:1b ≈ 800MB) # over its API, and indexes docs; tolerate up to ~5 min before # declaring the pod unhealthy. diff --git a/templates/secret.yaml b/templates/secret.yaml index a404005..75a4a03 100644 --- a/templates/secret.yaml +++ b/templates/secret.yaml @@ -14,6 +14,7 @@ required. {{- $secretKey := .Values.secrets.forailSecretKey | default (index $existingData "forailSecretKey" | default "" | b64dec) | default (randAlphaNum 50) -}} {{- $bcastSecret := .Values.secrets.forailBroadcastWebsocketSecret | default (index $existingData "forailBroadcastWebsocketSecret" | default "" | b64dec) | default (randAlphaNum 50) -}} {{- $redisPass := .Values.secrets.redisPassword | default (index $existingData "redisPassword" | default "" | b64dec) | default (randAlphaNum 32) -}} +{{- $assistantToken := .Values.secrets.assistantChatToken | default (index $existingData "assistantChatToken" | default "" | b64dec) | default (randAlphaNum 40) -}} {{- $adminPass := .Values.secrets.forailAdminPassword | default (index $existingData "forailAdminPassword" | default "" | b64dec) -}} {{- if not $adminPass -}} {{- fail "secrets.forailAdminPassword is required — set it, e.g. --set secrets.forailAdminPassword=\"$(openssl rand -base64 24)\" (see needtofix H3)" -}} @@ -31,4 +32,5 @@ stringData: forailSecretKey: {{ $secretKey | quote }} forailBroadcastWebsocketSecret: {{ $bcastSecret | quote }} redisPassword: {{ $redisPass | quote }} + assistantChatToken: {{ $assistantToken | quote }} forailAdminPassword: {{ $adminPass | quote }} diff --git a/values.yaml b/values.yaml index a58f0f1..ab69c3f 100644 --- a/values.yaml +++ b/values.yaml @@ -87,6 +87,13 @@ secrets: # and the websocket channel layer -- readable, writable, and one FLUSHALL from # empty. Generated on first install and reused on upgrade, like the others. redisPassword: "" + # Bearer token the assistant requires on its chat endpoint. Only used when + # assistant.enabled=true; generated on first install and reused on upgrade. + # Whatever proxies /assistant to the Service must send it as + # `Authorization: Bearer ` -- the chart does not expose the assistant + # through the Ingress, deliberately, so nothing outside the cluster reaches it + # until an operator routes it and configures that header. + assistantChatToken: "" # Required — no default; install fails if unset: forailAdminPassword: "" From 1bb860ac94aa7f9a7dd81694ef01a3fab2a34bda Mon Sep 17 00:00:00 2001 From: Krstan Vjestica Date: Wed, 19 Aug 2026 16:05:00 +0200 Subject: [PATCH 5/7] fix: most workloads ran with no pod or container hardening `podSecurityContext` and the three `securityContext.*` keys were all empty, and postgres, redis, OPA, the OTel collector, the init Job and the Ollama container had no key to set at all. Every container therefore kept its image's default UID and full capability set, and could gain privileges through a setuid binary. Every workload now has a key, and the default is the subset that is safe to assert without knowing what is inside the image: all capabilities dropped, privilege escalation refused, and the runtime's default seccomp profile at pod level. Nothing here takes away something these services legitimately need at these ports. The frontend is the exception and gets NET_BIND_SERVICE back, because nginx binds :80. `runAsNonRoot` and `readOnlyRootFilesystem` are deliberately left off. Both depend on what an image writes and which user it starts as -- the postgres and redis entrypoints begin as root and step down themselves -- so asserting them blind turns a working install into CrashLoopBackOff. values.yaml documents the full profile to apply once each image has been checked, which is a cluster exercise rather than a template one. The task pod skips the hardened block entirely when `task.privileged=true`: privileged and dropped capabilities cannot both be what the operator asked for, and the flag they set explicitly wins. Renders with 9 hardened containers and 10 pods carrying the seccomp profile; `helm lint` clean. Not validated on a running cluster -- the dev-cluster VM is currently unusable -- which is exactly why the two settings that can break a pod are not on by default. --- templates/forail-assistant.yaml | 3 ++ templates/forail-init-job.yaml | 6 ++++ templates/forail-task.yaml | 7 ++++ templates/opa.yaml | 6 ++++ templates/otel-collector.yaml | 6 ++++ templates/postgres.yaml | 6 ++++ templates/redis.yaml | 6 ++++ values.yaml | 57 ++++++++++++++++++++++++++------- 8 files changed, 86 insertions(+), 11 deletions(-) diff --git a/templates/forail-assistant.yaml b/templates/forail-assistant.yaml index 3e1d100..7f03008 100644 --- a/templates/forail-assistant.yaml +++ b/templates/forail-assistant.yaml @@ -105,6 +105,9 @@ spec: containers: - name: ollama image: "{{ .Values.images.assistantOllama.repository }}:{{ .Values.images.assistantOllama.tag }}" + {{- with .Values.securityContext.assistantOllama }} + securityContext: {{- toYaml . | nindent 12 }} + {{- end }} imagePullPolicy: {{ .Values.images.assistantOllama.pullPolicy }} {{- with .Values.securityContext.assistantOllama }} securityContext: {{- toYaml . | nindent 12 }} diff --git a/templates/forail-init-job.yaml b/templates/forail-init-job.yaml index 75c3fcc..5b5c2d3 100644 --- a/templates/forail-init-job.yaml +++ b/templates/forail-init-job.yaml @@ -32,9 +32,15 @@ spec: echo "Waiting for redis..." until nc -z forail-redis 6379; do sleep 2; done echo "Dependencies ready." + {{- with .Values.podSecurityContext }} + securityContext: {{- toYaml . | nindent 8 }} + {{- end }} containers: - name: forail-init image: "{{ .Values.images.backend.repository }}:{{ .Values.images.backend.tag }}" + {{- with .Values.securityContext.init }} + securityContext: {{- toYaml . | nindent 12 }} + {{- end }} imagePullPolicy: {{ .Values.images.backend.pullPolicy }} command: ["/bin/bash", "/etc/forail/init.sh"] env: diff --git a/templates/forail-task.yaml b/templates/forail-task.yaml index e805f9c..32c7082 100644 --- a/templates/forail-task.yaml +++ b/templates/forail-task.yaml @@ -22,6 +22,9 @@ spec: {{- if .Values.task.hostCgroup }} hostPID: false {{- end }} + {{- with .Values.podSecurityContext }} + securityContext: {{- toYaml . | nindent 8 }} + {{- end }} containers: - name: forail-task image: "{{ .Values.images.backend.repository }}:{{ .Values.images.backend.tag }}" @@ -30,6 +33,10 @@ spec: {{- if .Values.task.privileged }} securityContext: privileged: true + {{- else }} + {{- with .Values.securityContext.task }} + securityContext: {{- toYaml . | nindent 12 }} + {{- end }} {{- end }} env: {{- include "forail.backendEnv" . | nindent 12 }} diff --git a/templates/opa.yaml b/templates/opa.yaml index 536c2fb..0c2f84d 100644 --- a/templates/opa.yaml +++ b/templates/opa.yaml @@ -32,9 +32,15 @@ spec: labels: {{- include "forail.componentLabels" (dict "root" . "component" "opa") | nindent 8 }} spec: + {{- with .Values.podSecurityContext }} + securityContext: {{- toYaml . | nindent 8 }} + {{- end }} containers: - name: opa image: "{{ .Values.images.opa.repository }}:{{ .Values.images.opa.tag }}" + {{- with .Values.securityContext.opa }} + securityContext: {{- toYaml . | nindent 12 }} + {{- end }} args: ["run", "--server", "--addr", ":8181", "--log-level", "error"] ports: - name: opa diff --git a/templates/otel-collector.yaml b/templates/otel-collector.yaml index a4898f0..c12a5b0 100644 --- a/templates/otel-collector.yaml +++ b/templates/otel-collector.yaml @@ -35,9 +35,15 @@ spec: labels: {{- include "forail.componentLabels" (dict "root" . "component" "otel-collector") | nindent 8 }} spec: + {{- with .Values.podSecurityContext }} + securityContext: {{- toYaml . | nindent 8 }} + {{- end }} containers: - name: otel-collector image: "{{ .Values.images.otelCollector.repository }}:{{ .Values.images.otelCollector.tag }}" + {{- with .Values.securityContext.otelCollector }} + securityContext: {{- toYaml . | nindent 12 }} + {{- end }} args: ["--config=/etc/otel/config.yaml"] ports: - { name: otlp-grpc, containerPort: 4317 } diff --git a/templates/postgres.yaml b/templates/postgres.yaml index 2f18f3d..d951324 100644 --- a/templates/postgres.yaml +++ b/templates/postgres.yaml @@ -33,9 +33,15 @@ spec: labels: {{- include "forail.componentLabels" (dict "root" . "component" "postgres") | nindent 8 }} spec: + {{- with .Values.podSecurityContext }} + securityContext: {{- toYaml . | nindent 8 }} + {{- end }} containers: - name: postgres image: "{{ .Values.images.postgres.repository }}:{{ .Values.images.postgres.tag }}" + {{- with .Values.securityContext.postgres }} + securityContext: {{- toYaml . | nindent 12 }} + {{- end }} ports: - name: postgres containerPort: 5432 diff --git a/templates/redis.yaml b/templates/redis.yaml index a875809..3716d04 100644 --- a/templates/redis.yaml +++ b/templates/redis.yaml @@ -50,9 +50,15 @@ spec: labels: {{- include "forail.componentLabels" (dict "root" . "component" "redis") | nindent 8 }} spec: + {{- with .Values.podSecurityContext }} + securityContext: {{- toYaml . | nindent 8 }} + {{- end }} containers: - name: redis image: "{{ .Values.images.redis.repository }}:{{ .Values.images.redis.tag }}" + {{- with .Values.securityContext.redis }} + securityContext: {{- toYaml . | nindent 12 }} + {{- end }} # requirepass is read from the environment rather than written into # args: an argument is visible in the pod spec, in `kubectl describe` # and in the container's own /proc. diff --git a/values.yaml b/values.yaml index ab69c3f..2bbd1c3 100644 --- a/values.yaml +++ b/values.yaml @@ -291,18 +291,53 @@ assistant: networkPolicy: enabled: false -# ── Pod hardening (needtofix M10) ───────────────────────── -# Per-workload securityContext. Empty by default (no behaviour change) so the -# chart stays deployable while each context is validated per image — note the -# frontend binds :80 and needs NET_BIND_SERVICE or a non-root port before caps -# can be dropped. The operator chart is the reference model to copy: -# runAsNonRoot: true, allowPrivilegeEscalation: false, -# capabilities: { drop: ["ALL"] }, readOnlyRootFilesystem: true -podSecurityContext: {} +# ── Pod hardening (needtofix M10, Codex M5) ─────────────── +# These were empty, which meant every container kept its image's default UID and +# full capability set and could gain privileges through a setuid binary. Three +# workloads even had no value to set. +# +# What is on by default is the subset that is safe to assert without knowing the +# image: dropping all capabilities, refusing privilege escalation, and asking for +# the runtime's default seccomp profile. None of these grant anything an image +# legitimately needs at these ports -- the frontend is the one exception and gets +# NET_BIND_SERVICE back, because it binds :80. +# +# runAsNonRoot and readOnlyRootFilesystem are deliberately NOT defaulted. Both +# depend on what the image writes and which user it starts as (the postgres and +# redis entrypoints start as root and step down themselves), and asserting them +# blind turns a working install into CrashLoopBackOff. The recommended profile, +# once validated per image against your registry, is: +# +# runAsNonRoot: true +# runAsUser: +# readOnlyRootFilesystem: true # plus emptyDir mounts on its writable paths +# +# Set a workload's key to {} to opt out entirely. +podSecurityContext: + seccompProfile: + type: RuntimeDefault securityContext: - web: {} - frontend: {} - assistant: {} + web: &hardened + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + # The task pod runs automation. When task.privileged=true (the podman-in-pod + # execution path) the chart skips this block entirely -- the two cannot both + # apply, and the privileged flag is the one the operator asked for. + task: *hardened + # nginx binds :80, which needs the one capability back. + frontend: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + add: ["NET_BIND_SERVICE"] + assistant: *hardened + assistantOllama: *hardened + postgres: *hardened + redis: *hardened + opa: *hardened + otelCollector: *hardened + init: *hardened assistantOllama: {} # ── Ingress ─────────────────────────────────────────────── From aeed6cde2c2b3017216c428aa25375b43de8af6f Mon Sep 17 00:00:00 2001 From: Krstan Vjestica Date: Wed, 19 Aug 2026 17:50:00 +0200 Subject: [PATCH 6/7] fix: the default install could not run a job `forail.node.type` defaulted to `hybrid` -- the task pod runs jobs itself, through podman inside the container -- while `task.privileged` and `task.hostCgroup` both defaulted to false. That is the one combination that cannot execute anything: podman fails on the overlay mount, and every job sits in Pending. The dev-cluster install script already documented this. Each default was the safer of its pair; together they made the product's main function unusable out of the box. The default is now `control`, which runs each job as its own Kubernetes pod via receptor's `kubernetes-incluster-auth` work type and needs no privileges anywhere. Nothing new was required for it -- the chart already shipped every piece: the worktype in receptor.conf (whose comment is written for exactly this arrangement), the namespaced pod RBAC in rbac.yaml, and MY_POD_NAMESPACE from the downward API. The chart's own default contradicted its own receptor config. `init.sh` asserted the hybrid shape unconditionally -- it forced the default group back to a regular instance group on every run, so `node.type=control` would have been undone by the init Job even when set explicitly. The assertions now follow the node type: a container group with no member instance for control (a container group dispatches to Kubernetes, so a member makes the scheduler try to run the job on that node instead), a regular group containing this instance for hybrid and execution. That also keeps the file correct for the compose deployment, which is hybrid and shares it. `hybrid` without `task.privileged` now fails the render with both working configurations spelled out, rather than installing the broken pairing quietly. Renders and lints clean, and the guard fires on exactly the combination it should. NOT validated against a running cluster -- the dev-cluster VM is currently unusable -- so the container-group path should get a live job launch before this ships in a release. --- README.md | 29 ++++++++++++++- files/scripts/init.sh | 84 ++++++++++++++++++++++++++++-------------- templates/_helpers.tpl | 3 ++ values.yaml | 16 +++++++- 4 files changed, 102 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index dd64c7d..050f47d 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,22 @@ This chart ships **no working secret defaults**: - `podSecurityContext` and per-workload `securityContext.{web,frontend,assistant}` are available for pod hardening (empty by default; validate per image — the frontend binds `:80` and needs `NET_BIND_SERVICE` or a non-root port). +- **Job execution defaults to Kubernetes container groups** (`forail.node.type` + is now `control`, was `hybrid`). Jobs run as separate pods rather than through + podman inside the task pod, so no workload needs privileges. An install that + wants the podman path must now say so *and* enable it: `hybrid` without + `task.privileged=true` fails the render instead of producing an install where + every job stays `Pending`. +- **Redis now requires a password.** Generated on first install, reused on + upgrade. An existing install picks it up on the next `helm upgrade`; nothing + outside the chart should be talking to that Service, but anything that is will + need the credential from `forail-secrets`. +- **`forail.tenancyEnabled=true` now requires `forail.tenancy.rls=true`.** The + previous single switch turned on the tenancy features with no row-level + security behind them. +- **Every workload now drops all capabilities and refuses privilege escalation** + by default, with the frontend keeping `NET_BIND_SERVICE`. Override per workload + under `securityContext.*`; set a key to `{}` to opt out. - **`assistant.storage.size` dropped 20Gi → 5Gi** when the model server moved to its own claim. PVCs cannot shrink, so an existing install with the assistant enabled keeps its 20Gi claim and the upgrade fails on the immutable field — @@ -82,8 +98,17 @@ in place, all shipped by the chart: `kubernetes-incluster-auth` (`authmethod: incluster`). Without it launches fail at 0s with `unknown work type kubernetes-incluster-auth`. -The podman-in-pod execution path additionally needs `--set task.privileged=true ---set task.hostCgroup=true` (see the secure defaults above). +This is the default (`forail.node.type=control`) and needs no privileges +anywhere. + +The alternative, `forail.node.type=hybrid`, runs jobs through podman *inside* the +task pod and requires `--set task.privileged=true --set task.hostCgroup=true`. +Without both, podman fails on the overlay mount and every job stays `Pending`. + +Those were previously the defaults in the wrong combination — `hybrid` with +`privileged: false` — which is the one pairing that cannot run a job at all. The +render now refuses it and says which of the two configurations to pick, rather +than installing something whose main function is broken. ## AI assistant (optional) diff --git a/files/scripts/init.sh b/files/scripts/init.sh index 9f6fa35..503c7dc 100644 --- a/files/scripts/init.sh +++ b/files/scripts/init.sh @@ -24,6 +24,9 @@ forail-manage update_password --skip-checks \ echo "==> Provisioning instance..." NODE_NAME="${FORAIL_NODE_NAME:-$(hostname)}" +# Fallback is hybrid for the compose deployment, which runs jobs through +# podman on the host VM. The Helm chart always sets this explicitly, and +# defaults it to control. NODE_TYPE="${FORAIL_NODE_TYPE:-hybrid}" forail-manage provision_instance --skip-checks \ @@ -34,45 +37,72 @@ echo "==> Registering queues..." forail-manage register_queue --skip-checks --queuename=controlplane --instance_percent=100 forail-manage register_queue --skip-checks --queuename=default --instance_percent=100 -# Three things the 'default' group needs that the commands above do not -# reliably leave behind: +# What the 'default' instance group has to look like depends on where jobs run, +# and register_queue does not reliably leave either shape behind. # -# 1. is_container_group=false. A post_migrate signal auto-creates 'default' as -# a ContainerGroup on k8s, and without a work type that resolves locally -# every launch errors with 'unknown work type kubernetes-incluster-auth'. -# 2. Membership. On an UPGRADE the group already exists, so register_queue -# prints "Instance Group already registered" and assigns no instance. The -# group is left empty and every job sits in "pending" forever, with nothing -# in the UI or the logs to say why. -# 3. node_type, as a backstop. The real fix for that one is in the backend — -# the task pod re-runs provision_instance on every start and used to -# re-register as 'control' unconditionally, undoing whatever this Job set; -# it now honours FORAIL_NODE_TYPE. Keep the assertion so a newer chart -# paired with an older backend image still converges. +# NODE_TYPE=control (the chart default): jobs run as separate pods, submitted to +# receptor as the "kubernetes-incluster-auth" work type, which is configured in +# receptor.conf and backed by the namespaced pod RBAC. The group must be a +# ContainerGroup and must NOT contain this instance -- a container group +# dispatches to Kubernetes, not to a member node. +# +# NODE_TYPE=hybrid / execution (compose, or a k8s install that opted into +# podman-in-pod): this node runs jobs itself through the local receptor work +# command, so the group must be a regular instance group that contains it. +# Both halves matter -- a regular group with no execution-capable member +# accepts launches and never runs them, and the job sits in "pending" with +# nothing but "not enough available capacity" to go on. +# +# Membership also has to be asserted on every run, not just at creation: on an +# UPGRADE the group already exists, so register_queue prints "Instance Group +# already registered" and assigns nothing. +# +# node_type is re-asserted as a backstop. The real fix is in the backend -- the +# task pod re-runs provision_instance on every start and used to re-register as +# 'control' unconditionally, undoing whatever this Job set; it now honours +# FORAIL_NODE_TYPE. Keep the assertion so a newer chart paired with an older +# backend image still converges. forail-manage shell -c " from forail.main.models import Instance, InstanceGroup +from django.conf import settings + +node_type = '${NODE_TYPE}' +runs_jobs_locally = node_type in ('hybrid', 'execution') + ig = InstanceGroup.objects.filter(name='default').first() if not ig: print('default IG missing — register_queue did not create it') else: - if ig.is_container_group: - ig.is_container_group = False - ig.pod_spec_override = '' - ig.save(update_fields=['is_container_group', 'pod_spec_override']) - print('default IG: is_container_group -> False') i = Instance.objects.filter(hostname='${NODE_NAME}').first() - if not i: - print('instance ${NODE_NAME} missing — cannot assign to default IG') - else: - if i.node_type != '${NODE_TYPE}': - i.node_type = '${NODE_TYPE}' - i.save(update_fields=['node_type']) - print('instance node_type ->', i.node_type) - if not ig.instances.filter(pk=i.pk).exists(): + if i and i.node_type != node_type: + i.node_type = node_type + i.save(update_fields=['node_type']) + print('instance node_type ->', i.node_type) + + if runs_jobs_locally: + if ig.is_container_group: + ig.is_container_group = False + ig.pod_spec_override = '' + ig.save(update_fields=['is_container_group', 'pod_spec_override']) + print('default IG: is_container_group -> False') + if not i: + print('instance ${NODE_NAME} missing — cannot assign to default IG') + elif not ig.instances.filter(pk=i.pk).exists(): ig.instances.add(i) print('default IG: added', i.hostname) else: print('default IG already contains', i.hostname) + else: + if not ig.is_container_group: + ig.is_container_group = True + ig.pod_spec_override = settings.DEFAULT_EXECUTION_QUEUE_POD_SPEC_OVERRIDE + ig.save(update_fields=['is_container_group', 'pod_spec_override']) + print('default IG: is_container_group -> True') + # A container group dispatches to Kubernetes; a member instance here + # would make the scheduler try to run the job on this node instead. + if i and ig.instances.filter(pk=i.pk).exists(): + ig.instances.remove(i) + print('default IG: removed member', i.hostname, '(container group)') " echo "==> Creating preload data..." diff --git a/templates/_helpers.tpl b/templates/_helpers.tpl index cb60f0e..2370dec 100644 --- a/templates/_helpers.tpl +++ b/templates/_helpers.tpl @@ -101,6 +101,9 @@ Wraps DB, Redis, secrets, OTel, admin into one block to avoid drift. value: {{ .Values.forail.node.name | quote }} - name: FORAIL_NODE_TYPE value: {{ .Values.forail.node.type | quote }} +{{- if and (has .Values.forail.node.type (list "hybrid" "execution")) (not .Values.task.privileged) }} +{{- fail "forail.node.type=hybrid runs jobs through podman inside the task pod, which needs --set task.privileged=true --set task.hostCgroup=true. Without them podman fails on the overlay mount and every job stays Pending. Either set both, or use the default forail.node.type=control, which runs jobs as separate Kubernetes pods and needs no privileges." }} +{{- end }} {{- if and .Values.forail.tenancyEnabled (not .Values.forail.tenancy.rls) }} {{- fail "forail.tenancyEnabled=true requires forail.tenancy.rls=true — without row-level security the tenancy features run with no boundary behind them. Set --set forail.tenancy.rls=true, or turn tenancy off." }} {{- end }} diff --git a/values.yaml b/values.yaml index 2bbd1c3..526721d 100644 --- a/values.yaml +++ b/values.yaml @@ -127,7 +127,21 @@ forail: cookieSecure: "true" node: name: forail-node - type: hybrid + # control — jobs run as separate Kubernetes pods, submitted to receptor as + # the "kubernetes-incluster-auth" work type. Everything this needs + # already ships in the chart: the worktype in receptor.conf, the + # namespaced pod RBAC in rbac.yaml, and MY_POD_NAMESPACE from the + # downward API. Needs no privileges anywhere. + # hybrid — the task pod runs jobs itself, through podman inside the + # container. That path REQUIRES task.privileged=true and + # task.hostCgroup=true below; without them podman fails on the + # overlay mount and every job stays Pending. + # + # This defaulted to hybrid while task.privileged defaulted to false, which is + # the one combination that cannot run a job at all -- the safer half of each + # pair, adding up to an install where the product's main function is broken. + # The render now refuses that combination outright (see _helpers.tpl). + type: control # ── Multi-tenancy ─────────────────────────────────────── # tenancyEnabled alone used to be the only switch the chart offered, while the # controls that actually enforce isolation defaulted to off in the backend and From 1b744171e9447f1e8ca5b6cbbe9e3c2e254e2ec0 Mon Sep 17 00:00:00 2001 From: Krstan Vjestica Date: Wed, 19 Aug 2026 18:40:00 +0200 Subject: [PATCH 7/7] changelog: the security and default-install fixes from the Codex review Groups them by what an operator has to know before upgrading: Redis and the assistant start refusing unauthenticated callers, tenancy refuses the enabled-without-RLS combination, and job execution changes shape by default. --- CHANGELOG.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e7b9b4d..23e1299 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,46 @@ and the chart uses SemVer (`version`) plus the upstream Forail CalVer ## [Unreleased] +### Security +- **Redis now requires a password.** It ran `redis-server --appendonly yes` and + nothing else, on a ClusterIP Service, so any pod that could route to it could + read and write the cache, the task queue and the websocket channel layer — or + empty all three with one `FLUSHALL`. The password is generated on first install + and reused on upgrade; it is passed through the environment rather than args, + and the probes authenticate via `REDISCLI_AUTH`, so it appears in neither the + pod spec nor a command line. +- **The assistant requires a bearer token.** `FORAIL_ASSISTANT_CHAT_TOKEN` was + unset, which the assistant reads as "no authentication", and the chart passed + only the model and log level — so `assistant.enabled=true` gave a chat endpoint + open to every pod in the cluster, answering with indexed documentation. The + chart still does not route it through the Ingress. +- **Every workload drops all capabilities and refuses privilege escalation.** + `podSecurityContext` and the `securityContext.*` keys were empty, and six + workloads had no key at all. The frontend keeps `NET_BIND_SERVICE` for `:80`. + `runAsNonRoot` and `readOnlyRootFilesystem` are documented but not defaulted — + both depend on what an image writes and which user it starts as. +- **`forail.tenancyEnabled=true` now requires `forail.tenancy.rls=true`.** The + single switch the chart offered turned on quotas, branding and isolation + auditing while row-level security stayed off with no value to set, so an + install could look multi-tenant with nothing enforcing a boundary. Adds + `forail.tenancy.{rls,strictIsolation,rateLimiting}`; RLS defaults to true. + +### Fixed +- **The default install could not run a job.** `forail.node.type` defaulted to + `hybrid` (podman inside the task pod) while `task.privileged` defaulted to + false — the one combination where podman fails on the overlay mount and every + job stays `Pending`. The default is now `control`: each job runs as its own + Kubernetes pod through receptor's `kubernetes-incluster-auth` work type, which + the chart already shipped everything for. `hybrid` without privileges now fails + the render instead of installing quietly. `init.sh` no longer forces the + default instance group back to a regular group regardless of node type. + +### Changed +- `images.opa` and `images.otelCollector` are pinned. Neither pin changes what + runs today: the collector's `latest` genuinely moves (rebuilt 2026-08-18, now + `0.159.0`), while OPA's `latest-rootless` turned out to be a frozen orphan — + upstream stopped publishing `-rootless` after `0.58.0` in October 2023. + ### Added - **`forail-assistant-ollama` Deployment, Service and PVC.** The model server is no longer part of the assistant image; it runs beside it from