The pod hardening stopped postgres, web and the frontend from starting - #9
Open
krlex wants to merge 1 commit into
Open
The pod hardening stopped postgres, web and the frontend from starting#9krlex wants to merge 1 commit into
krlex wants to merge 1 commit into
Conversation
…arting
Every stateful component in the chart failed to start. Found by
installing the chart into a k3s cluster -- the hardening had only ever
been rendered, never run.
postgres chown: /var/lib/postgresql/data/pgdata: Operation not permitted
frontend chown("/var/cache/nginx/client_temp", 101) failed
web chown("/var/lib/nginx/body", 65534) failed
All three deliberately start as root, fix ownership on their own runtime
directories, and then drop to their own user. capabilities.drop: [ALL]
takes CHOWN away along with everything else, so they never reach the
point of dropping privileges -- they die in the entrypoint, and the
install is left with nothing serving and a database that never comes up.
Grant each one exactly what its entrypoint needs (CHOWN, SETUID, SETGID,
plus FOWNER and DAC_OVERRIDE for re-owning an existing PGDATA) and keep
the blanket drop for everything else. The components that already run as
their own user from PID 1 -- task, redis, opa, the collector, init, the
assistant -- are unchanged and still drop everything.
Also: assistantOllama was listed twice in the same map, and the second
entry ({}) silently won, so the Ollama container had no hardening at all
while appearing to have it. Now listed once.
Verified on a two-node k3s cluster: postgres, web and frontend all reach
1/1 Running with the fix, from CrashLoopBackOff without it.
krlex
force-pushed
the
fix/hardening-broke-every-stateful-component
branch
from
August 20, 2026 17:53
450a2eb to
1a9debe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by installing the chart into a real k3s cluster for the first time. The hardening from #7 had been rendered and reviewed, but never run — and it takes down three of the three stateful components.
All three start as root on purpose, fix ownership on their own runtime directories, then drop to their own user.
capabilities.drop: [ALL]takesCHOWNwith everything else, so they die in the entrypoint before they ever get to dropping privileges. The result is an install with nothing serving and a database that never comes up —CrashLoopBackOffon postgres, web and frontend, and a task pod that never goes Ready because the schema is never created.The fix
Each of the three gets exactly what its entrypoint needs, and keeps the blanket drop for the rest:
web,frontendCHOWN,SETUID,SETGID,NET_BIND_SERVICEpostgresCHOWN,FOWNER,DAC_OVERRIDE,SETUID,SETGIDComponents that already run as their own user from PID 1 —
task,redis,opa,otelCollector,init,assistant— are untouched and still drop everything.Second bug in the same block
assistantOllamawas listed twice, and the second entry ({}) silently won. The Ollama container had no hardening at all while appearing to have it. Now listed once.Verified
Two-node k3s cluster (1 server + 1 agent), chart installed with
task.privileged=true task.hostCgroup=true:CrashLoopBackOff, 7 restarts1/1 Running; redis, opa and the collector unaffected throughoutNote
This is exactly the gap PLAN-2026-08 warns about: the chart's security work was never validated against a cluster. It is worth deciding whether a smoke install belongs in CI before the next chart change.