Chart: a default install that can run jobs, plus Redis auth, assistant auth, tenancy enforcement and pod hardening

CHANGELOG.md

@@ -9,6 +9,46 @@ and the chart uses SemVer (`version`) plus the upstream Forail CalVer
 
 ## [Unreleased]
 
+### Security
+- **Redis now requires a password.** It ran `redis-server --appendonly yes` and
+  nothing else, on a ClusterIP Service, so any pod that could route to it could
+  read and write the cache, the task queue and the websocket channel layer — or
+  empty all three with one `FLUSHALL`. The password is generated on first install
+  and reused on upgrade; it is passed through the environment rather than args,
+  and the probes authenticate via `REDISCLI_AUTH`, so it appears in neither the
+  pod spec nor a command line.
+- **The assistant requires a bearer token.** `FORAIL_ASSISTANT_CHAT_TOKEN` was
+  unset, which the assistant reads as "no authentication", and the chart passed
+  only the model and log level — so `assistant.enabled=true` gave a chat endpoint
+  open to every pod in the cluster, answering with indexed documentation. The
+  chart still does not route it through the Ingress.
+- **Every workload drops all capabilities and refuses privilege escalation.**
+  `podSecurityContext` and the `securityContext.*` keys were empty, and six
+  workloads had no key at all. The frontend keeps `NET_BIND_SERVICE` for `:80`.
+  `runAsNonRoot` and `readOnlyRootFilesystem` are documented but not defaulted —
+  both depend on what an image writes and which user it starts as.
+- **`forail.tenancyEnabled=true` now requires `forail.tenancy.rls=true`.** The
+  single switch the chart offered turned on quotas, branding and isolation
+  auditing while row-level security stayed off with no value to set, so an
+  install could look multi-tenant with nothing enforcing a boundary. Adds
+  `forail.tenancy.{rls,strictIsolation,rateLimiting}`; RLS defaults to true.
+
+### Fixed
+- **The default install could not run a job.** `forail.node.type` defaulted to
+  `hybrid` (podman inside the task pod) while `task.privileged` defaulted to
+  false — the one combination where podman fails on the overlay mount and every
+  job stays `Pending`. The default is now `control`: each job runs as its own
+  Kubernetes pod through receptor's `kubernetes-incluster-auth` work type, which
+  the chart already shipped everything for. `hybrid` without privileges now fails
+  the render instead of installing quietly. `init.sh` no longer forces the
+  default instance group back to a regular group regardless of node type.
+
+### Changed
+- `images.opa` and `images.otelCollector` are pinned. Neither pin changes what
+  runs today: the collector's `latest` genuinely moves (rebuilt 2026-08-18, now
+  `0.159.0`), while OPA's `latest-rootless` turned out to be a frozen orphan —
+  upstream stopped publishing `-rootless` after `0.58.0` in October 2023.
+
 ### Added
 - **`forail-assistant-ollama` Deployment, Service and PVC.** The model server is
   no longer part of the assistant image; it runs beside it from

README.md

@@ -57,6 +57,22 @@ This chart ships **no working secret defaults**:
 - `podSecurityContext` and per-workload `securityContext.{web,frontend,assistant}`
   are available for pod hardening (empty by default; validate per image — the
   frontend binds `:80` and needs `NET_BIND_SERVICE` or a non-root port).
+- **Job execution defaults to Kubernetes container groups** (`forail.node.type`
+  is now `control`, was `hybrid`). Jobs run as separate pods rather than through
+  podman inside the task pod, so no workload needs privileges. An install that
+  wants the podman path must now say so *and* enable it: `hybrid` without
+  `task.privileged=true` fails the render instead of producing an install where
+  every job stays `Pending`.
+- **Redis now requires a password.** Generated on first install, reused on
+  upgrade. An existing install picks it up on the next `helm upgrade`; nothing
+  outside the chart should be talking to that Service, but anything that is will
+  need the credential from `forail-secrets`.
+- **`forail.tenancyEnabled=true` now requires `forail.tenancy.rls=true`.** The
+  previous single switch turned on the tenancy features with no row-level
+  security behind them.
+- **Every workload now drops all capabilities and refuses privilege escalation**
+  by default, with the frontend keeping `NET_BIND_SERVICE`. Override per workload
+  under `securityContext.*`; set a key to `{}` to opt out.
 - **`assistant.storage.size` dropped 20Gi → 5Gi** when the model server moved to
   its own claim. PVCs cannot shrink, so an existing install with the assistant
   enabled keeps its 20Gi claim and the upgrade fails on the immutable field —
@@ -82,8 +98,17 @@ in place, all shipped by the chart:
    `kubernetes-incluster-auth` (`authmethod: incluster`). Without it launches
    fail at 0s with `unknown work type kubernetes-incluster-auth`.
 
-The podman-in-pod execution path additionally needs `--set task.privileged=true
---set task.hostCgroup=true` (see the secure defaults above).
+This is the default (`forail.node.type=control`) and needs no privileges
+anywhere.
+
+The alternative, `forail.node.type=hybrid`, runs jobs through podman *inside* the
+task pod and requires `--set task.privileged=true --set task.hostCgroup=true`.
+Without both, podman fails on the overlay mount and every job stays `Pending`.
+
+Those were previously the defaults in the wrong combination — `hybrid` with
+`privileged: false` — which is the one pairing that cannot run a job at all. The
+render now refuses it and says which of the two configurations to pick, rather
+than installing something whose main function is broken.
 
 ## AI assistant (optional)
 

files/scripts/init.sh

@@ -24,6 +24,9 @@ forail-manage update_password --skip-checks \
 
 echo "==> Provisioning instance..."
 NODE_NAME="${FORAIL_NODE_NAME:-$(hostname)}"
+# Fallback is hybrid for the compose deployment, which runs jobs through
+# podman on the host VM. The Helm chart always sets this explicitly, and
+# defaults it to control.
 NODE_TYPE="${FORAIL_NODE_TYPE:-hybrid}"
 
 forail-manage provision_instance --skip-checks \
@@ -34,45 +37,72 @@ echo "==> Registering queues..."
 forail-manage register_queue --skip-checks --queuename=controlplane --instance_percent=100
 forail-manage register_queue --skip-checks --queuename=default      --instance_percent=100
 
-# Three things the 'default' group needs that the commands above do not
-# reliably leave behind:
+# What the 'default' instance group has to look like depends on where jobs run,
+# and register_queue does not reliably leave either shape behind.
 #
-#  1. is_container_group=false. A post_migrate signal auto-creates 'default' as
-#     a ContainerGroup on k8s, and without a work type that resolves locally
-#     every launch errors with 'unknown work type kubernetes-incluster-auth'.
-#  2. Membership. On an UPGRADE the group already exists, so register_queue
-#     prints "Instance Group already registered" and assigns no instance. The
-#     group is left empty and every job sits in "pending" forever, with nothing
-#     in the UI or the logs to say why.
-#  3. node_type, as a backstop. The real fix for that one is in the backend —
-#     the task pod re-runs provision_instance on every start and used to
-#     re-register as 'control' unconditionally, undoing whatever this Job set;
-#     it now honours FORAIL_NODE_TYPE. Keep the assertion so a newer chart
-#     paired with an older backend image still converges.
+# NODE_TYPE=control (the chart default): jobs run as separate pods, submitted to
+#   receptor as the "kubernetes-incluster-auth" work type, which is configured in
+#   receptor.conf and backed by the namespaced pod RBAC. The group must be a
+#   ContainerGroup and must NOT contain this instance -- a container group
+#   dispatches to Kubernetes, not to a member node.
+#
+# NODE_TYPE=hybrid / execution (compose, or a k8s install that opted into
+#   podman-in-pod): this node runs jobs itself through the local receptor work
+#   command, so the group must be a regular instance group that contains it.
+#   Both halves matter -- a regular group with no execution-capable member
+#   accepts launches and never runs them, and the job sits in "pending" with
+#   nothing but "not enough available capacity" to go on.
+#
+# Membership also has to be asserted on every run, not just at creation: on an
+# UPGRADE the group already exists, so register_queue prints "Instance Group
+# already registered" and assigns nothing.
+#
+# node_type is re-asserted as a backstop. The real fix is in the backend -- the
+# task pod re-runs provision_instance on every start and used to re-register as
+# 'control' unconditionally, undoing whatever this Job set; it now honours
+# FORAIL_NODE_TYPE. Keep the assertion so a newer chart paired with an older
+# backend image still converges.
 forail-manage shell -c "
 from forail.main.models import Instance, InstanceGroup
+from django.conf import settings
+
+node_type = '${NODE_TYPE}'
+runs_jobs_locally = node_type in ('hybrid', 'execution')
+
 ig = InstanceGroup.objects.filter(name='default').first()
 if not ig:
     print('default IG missing — register_queue did not create it')
 else:
-    if ig.is_container_group:
-        ig.is_container_group = False
-        ig.pod_spec_override = ''
-        ig.save(update_fields=['is_container_group', 'pod_spec_override'])
-        print('default IG: is_container_group -> False')
     i = Instance.objects.filter(hostname='${NODE_NAME}').first()
-    if not i:
-        print('instance ${NODE_NAME} missing — cannot assign to default IG')
-    else:
-        if i.node_type != '${NODE_TYPE}':
-            i.node_type = '${NODE_TYPE}'
-            i.save(update_fields=['node_type'])
-            print('instance node_type ->', i.node_type)
-        if not ig.instances.filter(pk=i.pk).exists():
+    if i and i.node_type != node_type:
+        i.node_type = node_type
+        i.save(update_fields=['node_type'])
+        print('instance node_type ->', i.node_type)
+
+    if runs_jobs_locally:
+        if ig.is_container_group:
+            ig.is_container_group = False
+            ig.pod_spec_override = ''
+            ig.save(update_fields=['is_container_group', 'pod_spec_override'])
+            print('default IG: is_container_group -> False')
+        if not i:
+            print('instance ${NODE_NAME} missing — cannot assign to default IG')
+        elif not ig.instances.filter(pk=i.pk).exists():
             ig.instances.add(i)
             print('default IG: added', i.hostname)
         else:
             print('default IG already contains', i.hostname)
+    else:
+        if not ig.is_container_group:
+            ig.is_container_group = True
+            ig.pod_spec_override = settings.DEFAULT_EXECUTION_QUEUE_POD_SPEC_OVERRIDE
+            ig.save(update_fields=['is_container_group', 'pod_spec_override'])
+            print('default IG: is_container_group -> True')
+        # A container group dispatches to Kubernetes; a member instance here
+        # would make the scheduler try to run the job on this node instead.
+        if i and ig.instances.filter(pk=i.pk).exists():
+            ig.instances.remove(i)
+            print('default IG: removed member', i.hostname, '(container group)')
 "
 
 echo "==> Creating preload data..."

files/settings/redis_settings.py

@@ -1,22 +1,36 @@
 import os
+from urllib.parse import quote
 
 _redis_host = os.environ.get('REDIS_HOST', 'redis')
 _redis_port = os.environ.get('REDIS_PORT', '6379')
 
-BROKER_URL = f'redis://{_redis_host}:{_redis_port}/0'
+# Redis has no authentication unless it is asked for. On a single-host compose
+# deployment it is not published and this stays empty; in Kubernetes the chart
+# sets requirepass and passes the value here, because a ClusterIP Service is
+# reachable by every pod that can route to it -- which is enough to read the
+# cache and the task queue, or to FLUSHALL them.
+#
+# Quoted, because a generated password may contain characters that would
+# otherwise end the userinfo section of the URL.
+_redis_password = os.environ.get('REDIS_PASSWORD', '')
+_redis_auth = f':{quote(_redis_password, safe="")}@' if _redis_password else ''
+
+_redis_url = f'redis://{_redis_auth}{_redis_host}:{_redis_port}'
+
+BROKER_URL = f'{_redis_url}/0'
 
 CACHES = {
     'default': {
         'BACKEND': 'forail.main.cache.AWXRedisCache',
-        'LOCATION': f'redis://{_redis_host}:{_redis_port}/1',
+        'LOCATION': f'{_redis_url}/1',
     }
 }
 
 CHANNEL_LAYERS = {
     'default': {
         'BACKEND': 'channels_redis.core.RedisChannelLayer',
         'CONFIG': {
-            'hosts': [f'redis://{_redis_host}:{_redis_port}/0'],
+            'hosts': [f'{_redis_url}/0'],
             'capacity': 10000,
             'group_expiry': 157784760,
         },

templates/_helpers.tpl

@@ -57,6 +57,11 @@ Wraps DB, Redis, secrets, OTel, admin into one block to avoid drift.
   value: forail-redis
 - name: REDIS_PORT
   value: "6379"
+- name: REDIS_PASSWORD
+  valueFrom:
+    secretKeyRef:
+      name: forail-secrets
+      key: redisPassword
 - name: FORAIL_SECRET_KEY
   valueFrom:
     secretKeyRef:
@@ -96,8 +101,20 @@ Wraps DB, Redis, secrets, OTel, admin into one block to avoid drift.
   value: {{ .Values.forail.node.name | quote }}
 - name: FORAIL_NODE_TYPE
   value: {{ .Values.forail.node.type | quote }}
+{{- if and (has .Values.forail.node.type (list "hybrid" "execution")) (not .Values.task.privileged) }}
+{{- fail "forail.node.type=hybrid runs jobs through podman inside the task pod, which needs --set task.privileged=true --set task.hostCgroup=true. Without them podman fails on the overlay mount and every job stays Pending. Either set both, or use the default forail.node.type=control, which runs jobs as separate Kubernetes pods and needs no privileges." }}
+{{- end }}
+{{- if and .Values.forail.tenancyEnabled (not .Values.forail.tenancy.rls) }}
+{{- fail "forail.tenancyEnabled=true requires forail.tenancy.rls=true — without row-level security the tenancy features run with no boundary behind them. Set --set forail.tenancy.rls=true, or turn tenancy off." }}
+{{- end }}
 - name: TENANCY_ENABLED
   value: {{ .Values.forail.tenancyEnabled | quote }}
+- name: TENANCY_RLS_ENABLED
+  value: {{ and .Values.forail.tenancyEnabled .Values.forail.tenancy.rls | quote }}
+- name: TENANCY_STRICT_ISOLATION_ENABLED
+  value: {{ and .Values.forail.tenancyEnabled .Values.forail.tenancy.strictIsolation | quote }}
+- name: TENANCY_RATE_LIMITING_ENABLED
+  value: {{ and .Values.forail.tenancyEnabled .Values.forail.tenancy.rateLimiting | quote }}
 - name: OTEL_ENABLED
   value: {{ .Values.forail.otel.enabled | quote }}
 - name: OTEL_EXPORTER_ENDPOINT

templates/forail-assistant.yaml

@@ -105,6 +105,9 @@ spec:
       containers:
         - name: ollama
           image: "{{ .Values.images.assistantOllama.repository }}:{{ .Values.images.assistantOllama.tag }}"
+          {{- with .Values.securityContext.assistantOllama }}
+          securityContext: {{- toYaml . | nindent 12 }}
+          {{- end }}
           imagePullPolicy: {{ .Values.images.assistantOllama.pullPolicy }}
           {{- with .Values.securityContext.assistantOllama }}
           securityContext: {{- toYaml . | nindent 12 }}
@@ -190,6 +193,20 @@ spec:
               value: {{ .Values.assistant.model | quote }}
             - name: FORAIL_ASSISTANT_LOG_LEVEL
               value: {{ .Values.assistant.logLevel | quote }}
+            # Without this the chat endpoint answers anyone who can reach the
+            # Service -- which is every pod in the cluster that can route to it,
+            # since there is no NetworkPolicy by default. Beyond unmetered use of
+            # the model, the answers carry indexed documentation back out.
+            #
+            # Whatever proxies /assistant to this Service has to send
+            # `Authorization: Bearer <token>`; read it from the same Secret:
+            #   kubectl -n {{ include "forail.namespace" . }} get secret forail-secrets \
+            #     -o jsonpath='{.data.assistantChatToken}' | base64 -d
+            - name: FORAIL_ASSISTANT_CHAT_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: forail-secrets
+                  key: assistantChatToken
           # First boot waits for Ollama, pulls the model (gemma3:1b ≈ 800MB)
           # over its API, and indexes docs; tolerate up to ~5 min before
           # declaring the pod unhealthy.

templates/forail-init-job.yaml

@@ -32,9 +32,15 @@ spec:
               echo "Waiting for redis..."
               until nc -z forail-redis 6379; do sleep 2; done
               echo "Dependencies ready."
+      {{- with .Values.podSecurityContext }}
+      securityContext: {{- toYaml . | nindent 8 }}
+      {{- end }}
       containers:
         - name: forail-init
           image: "{{ .Values.images.backend.repository }}:{{ .Values.images.backend.tag }}"
+          {{- with .Values.securityContext.init }}
+          securityContext: {{- toYaml . | nindent 12 }}
+          {{- end }}
           imagePullPolicy: {{ .Values.images.backend.pullPolicy }}
           command: ["/bin/bash", "/etc/forail/init.sh"]
           env:

templates/forail-task.yaml

@@ -22,6 +22,9 @@ spec:
       {{- if .Values.task.hostCgroup }}
       hostPID: false
       {{- end }}
+      {{- with .Values.podSecurityContext }}
+      securityContext: {{- toYaml . | nindent 8 }}
+      {{- end }}
       containers:
         - name: forail-task
           image: "{{ .Values.images.backend.repository }}:{{ .Values.images.backend.tag }}"
@@ -30,6 +33,10 @@ spec:
           {{- if .Values.task.privileged }}
           securityContext:
             privileged: true
+          {{- else }}
+          {{- with .Values.securityContext.task }}
+          securityContext: {{- toYaml . | nindent 12 }}
+          {{- end }}
           {{- end }}
           env:
             {{- include "forail.backendEnv" . | nindent 12 }}

templates/opa.yaml

@@ -32,9 +32,15 @@ spec:
       labels:
         {{- include "forail.componentLabels" (dict "root" . "component" "opa") | nindent 8 }}
     spec:
+      {{- with .Values.podSecurityContext }}
+      securityContext: {{- toYaml . | nindent 8 }}
+      {{- end }}
       containers:
         - name: opa
           image: "{{ .Values.images.opa.repository }}:{{ .Values.images.opa.tag }}"
+          {{- with .Values.securityContext.opa }}
+          securityContext: {{- toYaml . | nindent 12 }}
+          {{- end }}
           args: ["run", "--server", "--addr", ":8181", "--log-level", "error"]
           ports:
             - name: opa

templates/otel-collector.yaml

@@ -35,9 +35,15 @@ spec:
       labels:
         {{- include "forail.componentLabels" (dict "root" . "component" "otel-collector") | nindent 8 }}
     spec:
+      {{- with .Values.podSecurityContext }}
+      securityContext: {{- toYaml . | nindent 8 }}
+      {{- end }}
       containers:
         - name: otel-collector
           image: "{{ .Values.images.otelCollector.repository }}:{{ .Values.images.otelCollector.tag }}"
+          {{- with .Values.securityContext.otelCollector }}
+          securityContext: {{- toYaml . | nindent 12 }}
+          {{- end }}
           args: ["--config=/etc/otel/config.yaml"]
           ports:
             - { name: otlp-grpc, containerPort: 4317 }

templates/postgres.yaml

@@ -33,9 +33,15 @@ spec:
       labels:
         {{- include "forail.componentLabels" (dict "root" . "component" "postgres") | nindent 8 }}
     spec:
+      {{- with .Values.podSecurityContext }}
+      securityContext: {{- toYaml . | nindent 8 }}
+      {{- end }}
       containers:
         - name: postgres
           image: "{{ .Values.images.postgres.repository }}:{{ .Values.images.postgres.tag }}"
+          {{- with .Values.securityContext.postgres }}
+          securityContext: {{- toYaml . | nindent 12 }}
+          {{- end }}
           ports:
             - name: postgres
               containerPort: 5432