Skip to content

Chart: a default install that can run jobs, plus Redis auth, assistant auth, tenancy enforcement and pod hardening - #7

Merged
krlex merged 7 commits into
developfrom
fix/chart-security-and-defaults
Aug 19, 2026
Merged

Chart: a default install that can run jobs, plus Redis auth, assistant auth, tenancy enforcement and pod hardening#7
krlex merged 7 commits into
developfrom
fix/chart-security-and-defaults

Conversation

@krlex

@krlex krlex commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Closes H4, M2, M3, M4, M5 and L1 from the 2026-08-19 Codex review — every
chart finding.

H4 — the default install could not run a job

forail.node.type defaulted to hybrid (the task pod runs jobs itself, through
podman inside the container) while task.privileged and task.hostCgroup both
defaulted to false. That is the one combination that cannot execute anything:
podman fails on the overlay mount and every job stays Pending. Each default was
the safer of its pair; together they made the product's main function unusable
out of the box.

Default is now control — each job runs as its own Kubernetes pod through
receptor's kubernetes-incluster-auth work type, needing no privileges anywhere.
Nothing new was required: the chart already shipped the worktype in
receptor.conf (whose comment is written for exactly this arrangement), the
namespaced pod RBAC in rbac.yaml, and MY_POD_NAMESPACE from the downward API.
The chart's default contradicted its own receptor config.

init.sh asserted the hybrid shape unconditionally — it forced the default group
back to a regular instance group on every run, so node.type=control would have
been undone by the init Job even when set explicitly. The assertions now follow
the node type, which also keeps the file correct for the compose deployment that
shares it.

hybrid without task.privileged now fails the render, with both working
configurations spelled out.

M4 — Redis had no authentication

redis-server --appendonly yes on a ClusterIP Service: any pod that could route
to it could read and write the cache, the Celery queue and the websocket channel
layer, or empty all three with one FLUSHALL. The NetworkPolicy that would have
limited this is off by default because flannel does not enforce policies, so a
stock install had no control at either layer.

requirepass is now always set, from a generated secret. Passed through the
environment rather than args (an argument is visible in the pod spec and in
kubectl describe), and the probes authenticate via REDISCLI_AUTH, so it stays
off their command line too.

M3 — the assistant answered anyone who could reach its Service

FORAIL_ASSISTANT_CHAT_TOKEN defaults to empty, which the assistant reads as "no
authentication", and the chart passed only the model and log level. Beyond
unmetered model use, the answers carry indexed documentation back out. The chart
now always sets a token and documents how a proxy supplies it — and still does
not route the assistant through the Ingress.

M2 — enabling tenancy enforced nothing

forail.tenancyEnabled was the only tenancy value, and it maps to the flag that
turns on quotas, branding and auditing. The flags that enforce a boundary
defaulted to False in the backend with no Helm value at all. Adds
forail.tenancy.{rls,strictIsolation,rateLimiting} and refuses
tenancyEnabled=true with rls=false. RLS defaults to true, so the operator now
opts out rather than having to know to opt in.

Needs forail-backend#7, which mirrors the other three flags from the environment
into the Setting registry — only TENANCY_ENABLED was synced, so these would
otherwise be set and ignored.

M5 — most workloads had no hardening

podSecurityContext and the three securityContext.* keys were empty, and
postgres, redis, OPA, the OTel collector, the init Job and Ollama had no key to
set. Every workload now has one, defaulting to the subset safe to assert without
knowing the image: all capabilities dropped, privilege escalation refused,
RuntimeDefault seccomp. The frontend keeps NET_BIND_SERVICE for :80; the
task pod skips the block entirely when task.privileged=true.

runAsNonRoot and readOnlyRootFilesystem are documented but not defaulted —
both depend on what an image writes and which user it starts as (postgres and
redis start as root and step down themselves), and asserting them blind turns a
working install into CrashLoopBackOff.

L1 — floating image tags

Neither pin changes what runs today. The collector's latest genuinely moves
(rebuilt 2026-08-18, currently 0.159.0). OPA's latest-rootless was the
opposite problem — upstream stopped publishing -rootless after 0.58.0 in
October 2023, so it has been a frozen orphan; pinning makes that visible. The
TODO this replaces suggested 0.70.0-rootless, which does not exist.

Verified

  • helm lint clean; renders across default, assistant.enabled=true,
    tenancy.*, and task.privileged=true.
  • Both new guards fire on exactly the combination they should and on nothing else.
  • 9 hardened containers and 10 pods carrying the seccomp profile in a default
    render.

Not validated against a running cluster — the dev-cluster VM is currently
unusable (vagrant status fails on a UID mismatch). The container-group
execution path in particular should get a live job launch before this goes into a
release, and that is why runAsNonRoot / readOnlyRootFilesystem are not
switched on here.

Upgrade notes

  • Redis starts requiring a password; anything outside the chart talking to that
    Service needs the credential from forail-secrets.
  • Job execution changes shape. An install that wants the podman path must set
    forail.node.type=hybrid and task.privileged=true --set task.hostCgroup=true.
  • assistant.storage.size 20Gi → 5Gi still applies from the Ollama split; PVCs
    cannot shrink.

krlex added 7 commits August 19, 2026 09:10
Both tracked floating tags, so the same chart version installed different code
over time. Neither pin changes what runs today.

The collector's `latest` genuinely moves -- it was rebuilt on 2026-08-18 -- and
0.159.0 is the release it points at now.

OPA's `latest-rootless` turned out to be the opposite problem: upstream stopped
publishing `-rootless` variants after 0.58.0, so it has been a frozen orphan
since 2023-10-26. Pinning it to 0.58.0-rootless makes that visible instead of
implying the image is current. The TODO this replaces suggested 0.70.0-rootless,
which does not exist. Moving off 0.58.0 means choosing a different image, and
OPA 1.x defaults to Rego v1, so it needs the backend's generated policies
checked first -- a separate decision, not a version bump.
`redis-server --appendonly yes` and nothing else. Its Service is a ClusterIP, so
every pod that can route to it could read and write the cache, the Celery task
queue and the websocket channel layer -- or empty all three with one FLUSHALL.
The NetworkPolicy that would have limited the blast radius is off by default,
because k3s' flannel does not enforce policies, so a stock install had no
control at either layer.

`requirepass` is now always set, from a password generated on first install and
reused on upgrade like the other infrastructure secrets. It is passed through
the environment rather than through args, since an argument shows up in the pod
spec, in `kubectl describe` and in the container's own /proc. The liveness and
readiness probes authenticate via REDISCLI_AUTH, which redis-cli reads on its
own, so the password stays off their command line too.

`redis_settings.py` builds the URL with the password when REDIS_PASSWORD is set
and without it when it is not, so the compose deployment -- where Redis is not
published and this stays empty -- is unaffected. That file is one of the ones
`make sync-from-deploy` copies from forail-deploy, so the same change belongs
there or the next sync will revert it.
`forail.tenancyEnabled` was the only tenancy value the chart offered, and it maps
to `TENANCY_ENABLED` -- the flag that turns on quotas, branding and isolation
auditing. The flags that enforce a boundary, `TENANCY_RLS_ENABLED` and
`TENANCY_STRICT_ISOLATION_ENABLED`, default to False in the backend and had no
Helm value at all. An operator who set the one switch they were offered got a
multi-tenant-looking install with no row-level security behind it.

Adds `forail.tenancy.{rls,strictIsolation,rateLimiting}`, all passed through, and
refuses the combination that caused the problem: `tenancyEnabled=true` with
`rls=false` fails the render with an explanation rather than installing something
that only looks isolated. RLS defaults to true, so turning tenancy on now scopes
rows by default and the operator opts *out* rather than having to know to opt in.

All three are ANDed with `tenancyEnabled`, so leaving them set while turning
tenancy off does not leave stray flags on.

Needs the matching backend change: only `TENANCY_ENABLED` was mirrored from the
environment into the Setting registry, so these variables would otherwise be set
and ignored.
`FORAIL_ASSISTANT_CHAT_TOKEN` defaults to empty, which the assistant treats as
"no authentication required" -- and the chart passed only the model name and log
level, so enabling `assistant.enabled=true` gave a chat endpoint open to every
pod that can route to its ClusterIP. There is no NetworkPolicy by default, so
that is the whole cluster. Beyond unmetered use of the model and holding all four
concurrency slots, the answers carry indexed documentation back out.

The chart now always sets a token, generated on first install and reused on
upgrade like the other infrastructure secrets. Whatever proxies `/assistant` to
the Service must send it as `Authorization: Bearer <token>`; the value is in the
same Secret and both values.yaml and the deployment say how to read it.

The chart still does not route the assistant through the Ingress. That stays
deliberate -- nothing outside the cluster reaches it until an operator adds the
path and configures the header, rather than the token being the only thing
between the internet and the model.
`podSecurityContext` and the three `securityContext.*` keys were all empty, and
postgres, redis, OPA, the OTel collector, the init Job and the Ollama container
had no key to set at all. Every container therefore kept its image's default
UID and full capability set, and could gain privileges through a setuid binary.

Every workload now has a key, and the default is the subset that is safe to
assert without knowing what is inside the image: all capabilities dropped,
privilege escalation refused, and the runtime's default seccomp profile at pod
level. Nothing here takes away something these services legitimately need at
these ports. The frontend is the exception and gets NET_BIND_SERVICE back,
because nginx binds :80.

`runAsNonRoot` and `readOnlyRootFilesystem` are deliberately left off. Both
depend on what an image writes and which user it starts as -- the postgres and
redis entrypoints begin as root and step down themselves -- so asserting them
blind turns a working install into CrashLoopBackOff. values.yaml documents the
full profile to apply once each image has been checked, which is a cluster
exercise rather than a template one.

The task pod skips the hardened block entirely when `task.privileged=true`:
privileged and dropped capabilities cannot both be what the operator asked for,
and the flag they set explicitly wins.

Renders with 9 hardened containers and 10 pods carrying the seccomp profile;
`helm lint` clean. Not validated on a running cluster -- the dev-cluster VM is
currently unusable -- which is exactly why the two settings that can break a
pod are not on by default.
`forail.node.type` defaulted to `hybrid` -- the task pod runs jobs itself,
through podman inside the container -- while `task.privileged` and
`task.hostCgroup` both defaulted to false. That is the one combination that
cannot execute anything: podman fails on the overlay mount, and every job sits
in Pending. The dev-cluster install script already documented this. Each default
was the safer of its pair; together they made the product's main function
unusable out of the box.

The default is now `control`, which runs each job as its own Kubernetes pod via
receptor's `kubernetes-incluster-auth` work type and needs no privileges
anywhere. Nothing new was required for it -- the chart already shipped every
piece: the worktype in receptor.conf (whose comment is written for exactly this
arrangement), the namespaced pod RBAC in rbac.yaml, and MY_POD_NAMESPACE from
the downward API. The chart's own default contradicted its own receptor config.

`init.sh` asserted the hybrid shape unconditionally -- it forced the default
group back to a regular instance group on every run, so `node.type=control`
would have been undone by the init Job even when set explicitly. The assertions
now follow the node type: a container group with no member instance for control
(a container group dispatches to Kubernetes, so a member makes the scheduler try
to run the job on that node instead), a regular group containing this instance
for hybrid and execution. That also keeps the file correct for the compose
deployment, which is hybrid and shares it.

`hybrid` without `task.privileged` now fails the render with both working
configurations spelled out, rather than installing the broken pairing quietly.

Renders and lints clean, and the guard fires on exactly the combination it
should. NOT validated against a running cluster -- the dev-cluster VM is
currently unusable -- so the container-group path should get a live job launch
before this ships in a release.
Groups them by what an operator has to know before upgrading: Redis and the
assistant start refusing unauthenticated callers, tenancy refuses the
enabled-without-RLS combination, and job execution changes shape by default.
@krlex
krlex force-pushed the fix/chart-security-and-defaults branch from f8c442f to 1b74417 Compare August 19, 2026 21:17
@krlex
krlex merged commit ceb0979 into develop Aug 19, 2026
1 check passed
@krlex
krlex deleted the fix/chart-security-and-defaults branch August 19, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant