Chart: a default install that can run jobs, plus Redis auth, assistant auth, tenancy enforcement and pod hardening - #7
Merged
Conversation
Both tracked floating tags, so the same chart version installed different code over time. Neither pin changes what runs today. The collector's `latest` genuinely moves -- it was rebuilt on 2026-08-18 -- and 0.159.0 is the release it points at now. OPA's `latest-rootless` turned out to be the opposite problem: upstream stopped publishing `-rootless` variants after 0.58.0, so it has been a frozen orphan since 2023-10-26. Pinning it to 0.58.0-rootless makes that visible instead of implying the image is current. The TODO this replaces suggested 0.70.0-rootless, which does not exist. Moving off 0.58.0 means choosing a different image, and OPA 1.x defaults to Rego v1, so it needs the backend's generated policies checked first -- a separate decision, not a version bump.
`redis-server --appendonly yes` and nothing else. Its Service is a ClusterIP, so every pod that can route to it could read and write the cache, the Celery task queue and the websocket channel layer -- or empty all three with one FLUSHALL. The NetworkPolicy that would have limited the blast radius is off by default, because k3s' flannel does not enforce policies, so a stock install had no control at either layer. `requirepass` is now always set, from a password generated on first install and reused on upgrade like the other infrastructure secrets. It is passed through the environment rather than through args, since an argument shows up in the pod spec, in `kubectl describe` and in the container's own /proc. The liveness and readiness probes authenticate via REDISCLI_AUTH, which redis-cli reads on its own, so the password stays off their command line too. `redis_settings.py` builds the URL with the password when REDIS_PASSWORD is set and without it when it is not, so the compose deployment -- where Redis is not published and this stays empty -- is unaffected. That file is one of the ones `make sync-from-deploy` copies from forail-deploy, so the same change belongs there or the next sync will revert it.
`forail.tenancyEnabled` was the only tenancy value the chart offered, and it maps
to `TENANCY_ENABLED` -- the flag that turns on quotas, branding and isolation
auditing. The flags that enforce a boundary, `TENANCY_RLS_ENABLED` and
`TENANCY_STRICT_ISOLATION_ENABLED`, default to False in the backend and had no
Helm value at all. An operator who set the one switch they were offered got a
multi-tenant-looking install with no row-level security behind it.
Adds `forail.tenancy.{rls,strictIsolation,rateLimiting}`, all passed through, and
refuses the combination that caused the problem: `tenancyEnabled=true` with
`rls=false` fails the render with an explanation rather than installing something
that only looks isolated. RLS defaults to true, so turning tenancy on now scopes
rows by default and the operator opts *out* rather than having to know to opt in.
All three are ANDed with `tenancyEnabled`, so leaving them set while turning
tenancy off does not leave stray flags on.
Needs the matching backend change: only `TENANCY_ENABLED` was mirrored from the
environment into the Setting registry, so these variables would otherwise be set
and ignored.
`FORAIL_ASSISTANT_CHAT_TOKEN` defaults to empty, which the assistant treats as "no authentication required" -- and the chart passed only the model name and log level, so enabling `assistant.enabled=true` gave a chat endpoint open to every pod that can route to its ClusterIP. There is no NetworkPolicy by default, so that is the whole cluster. Beyond unmetered use of the model and holding all four concurrency slots, the answers carry indexed documentation back out. The chart now always sets a token, generated on first install and reused on upgrade like the other infrastructure secrets. Whatever proxies `/assistant` to the Service must send it as `Authorization: Bearer <token>`; the value is in the same Secret and both values.yaml and the deployment say how to read it. The chart still does not route the assistant through the Ingress. That stays deliberate -- nothing outside the cluster reaches it until an operator adds the path and configures the header, rather than the token being the only thing between the internet and the model.
`podSecurityContext` and the three `securityContext.*` keys were all empty, and postgres, redis, OPA, the OTel collector, the init Job and the Ollama container had no key to set at all. Every container therefore kept its image's default UID and full capability set, and could gain privileges through a setuid binary. Every workload now has a key, and the default is the subset that is safe to assert without knowing what is inside the image: all capabilities dropped, privilege escalation refused, and the runtime's default seccomp profile at pod level. Nothing here takes away something these services legitimately need at these ports. The frontend is the exception and gets NET_BIND_SERVICE back, because nginx binds :80. `runAsNonRoot` and `readOnlyRootFilesystem` are deliberately left off. Both depend on what an image writes and which user it starts as -- the postgres and redis entrypoints begin as root and step down themselves -- so asserting them blind turns a working install into CrashLoopBackOff. values.yaml documents the full profile to apply once each image has been checked, which is a cluster exercise rather than a template one. The task pod skips the hardened block entirely when `task.privileged=true`: privileged and dropped capabilities cannot both be what the operator asked for, and the flag they set explicitly wins. Renders with 9 hardened containers and 10 pods carrying the seccomp profile; `helm lint` clean. Not validated on a running cluster -- the dev-cluster VM is currently unusable -- which is exactly why the two settings that can break a pod are not on by default.
`forail.node.type` defaulted to `hybrid` -- the task pod runs jobs itself, through podman inside the container -- while `task.privileged` and `task.hostCgroup` both defaulted to false. That is the one combination that cannot execute anything: podman fails on the overlay mount, and every job sits in Pending. The dev-cluster install script already documented this. Each default was the safer of its pair; together they made the product's main function unusable out of the box. The default is now `control`, which runs each job as its own Kubernetes pod via receptor's `kubernetes-incluster-auth` work type and needs no privileges anywhere. Nothing new was required for it -- the chart already shipped every piece: the worktype in receptor.conf (whose comment is written for exactly this arrangement), the namespaced pod RBAC in rbac.yaml, and MY_POD_NAMESPACE from the downward API. The chart's own default contradicted its own receptor config. `init.sh` asserted the hybrid shape unconditionally -- it forced the default group back to a regular instance group on every run, so `node.type=control` would have been undone by the init Job even when set explicitly. The assertions now follow the node type: a container group with no member instance for control (a container group dispatches to Kubernetes, so a member makes the scheduler try to run the job on that node instead), a regular group containing this instance for hybrid and execution. That also keeps the file correct for the compose deployment, which is hybrid and shares it. `hybrid` without `task.privileged` now fails the render with both working configurations spelled out, rather than installing the broken pairing quietly. Renders and lints clean, and the guard fires on exactly the combination it should. NOT validated against a running cluster -- the dev-cluster VM is currently unusable -- so the container-group path should get a live job launch before this ships in a release.
Groups them by what an operator has to know before upgrading: Redis and the assistant start refusing unauthenticated callers, tenancy refuses the enabled-without-RLS combination, and job execution changes shape by default.
krlex
force-pushed
the
fix/chart-security-and-defaults
branch
from
August 19, 2026 21:17
f8c442f to
1b74417
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes H4, M2, M3, M4, M5 and L1 from the 2026-08-19 Codex review — every
chart finding.
H4 — the default install could not run a job
forail.node.typedefaulted tohybrid(the task pod runs jobs itself, throughpodman inside the container) while
task.privilegedandtask.hostCgroupbothdefaulted to false. That is the one combination that cannot execute anything:
podman fails on the overlay mount and every job stays
Pending. Each default wasthe safer of its pair; together they made the product's main function unusable
out of the box.
Default is now
control— each job runs as its own Kubernetes pod throughreceptor's
kubernetes-incluster-authwork type, needing no privileges anywhere.Nothing new was required: the chart already shipped the worktype in
receptor.conf(whose comment is written for exactly this arrangement), thenamespaced pod RBAC in
rbac.yaml, andMY_POD_NAMESPACEfrom the downward API.The chart's default contradicted its own receptor config.
init.shasserted the hybrid shape unconditionally — it forced the default groupback to a regular instance group on every run, so
node.type=controlwould havebeen undone by the init Job even when set explicitly. The assertions now follow
the node type, which also keeps the file correct for the compose deployment that
shares it.
hybridwithouttask.privilegednow fails the render, with both workingconfigurations spelled out.
M4 — Redis had no authentication
redis-server --appendonly yeson a ClusterIP Service: any pod that could routeto it could read and write the cache, the Celery queue and the websocket channel
layer, or empty all three with one
FLUSHALL. The NetworkPolicy that would havelimited this is off by default because flannel does not enforce policies, so a
stock install had no control at either layer.
requirepassis now always set, from a generated secret. Passed through theenvironment rather than args (an argument is visible in the pod spec and in
kubectl describe), and the probes authenticate viaREDISCLI_AUTH, so it staysoff their command line too.
M3 — the assistant answered anyone who could reach its Service
FORAIL_ASSISTANT_CHAT_TOKENdefaults to empty, which the assistant reads as "noauthentication", and the chart passed only the model and log level. Beyond
unmetered model use, the answers carry indexed documentation back out. The chart
now always sets a token and documents how a proxy supplies it — and still does
not route the assistant through the Ingress.
M2 — enabling tenancy enforced nothing
forail.tenancyEnabledwas the only tenancy value, and it maps to the flag thatturns on quotas, branding and auditing. The flags that enforce a boundary
defaulted to False in the backend with no Helm value at all. Adds
forail.tenancy.{rls,strictIsolation,rateLimiting}and refusestenancyEnabled=truewithrls=false. RLS defaults to true, so the operator nowopts out rather than having to know to opt in.
Needs forail-backend#7, which mirrors the other three flags from the environment
into the Setting registry — only
TENANCY_ENABLEDwas synced, so these wouldotherwise be set and ignored.
M5 — most workloads had no hardening
podSecurityContextand the threesecurityContext.*keys were empty, andpostgres, redis, OPA, the OTel collector, the init Job and Ollama had no key to
set. Every workload now has one, defaulting to the subset safe to assert without
knowing the image: all capabilities dropped, privilege escalation refused,
RuntimeDefaultseccomp. The frontend keepsNET_BIND_SERVICEfor:80; thetask pod skips the block entirely when
task.privileged=true.runAsNonRootandreadOnlyRootFilesystemare documented but not defaulted —both depend on what an image writes and which user it starts as (postgres and
redis start as root and step down themselves), and asserting them blind turns a
working install into CrashLoopBackOff.
L1 — floating image tags
Neither pin changes what runs today. The collector's
latestgenuinely moves(rebuilt 2026-08-18, currently
0.159.0). OPA'slatest-rootlesswas theopposite problem — upstream stopped publishing
-rootlessafter0.58.0inOctober 2023, so it has been a frozen orphan; pinning makes that visible. The
TODO this replaces suggested
0.70.0-rootless, which does not exist.Verified
helm lintclean; renders across default,assistant.enabled=true,tenancy.*, andtask.privileged=true.render.
Not validated against a running cluster — the dev-cluster VM is currently
unusable (
vagrant statusfails on a UID mismatch). The container-groupexecution path in particular should get a live job launch before this goes into a
release, and that is why
runAsNonRoot/readOnlyRootFilesystemare notswitched on here.
Upgrade notes
Service needs the credential from
forail-secrets.forail.node.type=hybridandtask.privileged=true --set task.hostCgroup=true.assistant.storage.size20Gi → 5Gi still applies from the Ollama split; PVCscannot shrink.