Skip to content

The pod hardening stopped postgres, web and the frontend from starting - #9

Open
krlex wants to merge 1 commit into
developfrom
fix/hardening-broke-every-stateful-component
Open

The pod hardening stopped postgres, web and the frontend from starting#9
krlex wants to merge 1 commit into
developfrom
fix/hardening-broke-every-stateful-component

Conversation

@krlex

@krlex krlex commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Found by installing the chart into a real k3s cluster for the first time. The hardening from #7 had been rendered and reviewed, but never run — and it takes down three of the three stateful components.

postgres  chown: /var/lib/postgresql/data/pgdata: Operation not permitted
frontend  chown("/var/cache/nginx/client_temp", 101) failed
web       chown("/var/lib/nginx/body", 65534) failed

All three start as root on purpose, fix ownership on their own runtime directories, then drop to their own user. capabilities.drop: [ALL] takes CHOWN with everything else, so they die in the entrypoint before they ever get to dropping privileges. The result is an install with nothing serving and a database that never comes up — CrashLoopBackOff on postgres, web and frontend, and a task pod that never goes Ready because the schema is never created.

The fix

Each of the three gets exactly what its entrypoint needs, and keeps the blanket drop for the rest:

added
web, frontend CHOWN, SETUID, SETGID, NET_BIND_SERVICE
postgres CHOWN, FOWNER, DAC_OVERRIDE, SETUID, SETGID

Components that already run as their own user from PID 1 — task, redis, opa, otelCollector, init, assistant — are untouched and still drop everything.

Second bug in the same block

assistantOllama was listed twice, and the second entry ({}) silently won. The Ollama container had no hardening at all while appearing to have it. Now listed once.

Verified

Two-node k3s cluster (1 server + 1 agent), chart installed with task.privileged=true task.hostCgroup=true:

  • without the fix — postgres, web, frontend all CrashLoopBackOff, 7 restarts
  • with the fix — all three 1/1 Running; redis, opa and the collector unaffected throughout

Note

This is exactly the gap PLAN-2026-08 warns about: the chart's security work was never validated against a cluster. It is worth deciding whether a smoke install belongs in CI before the next chart change.

…arting

Every stateful component in the chart failed to start. Found by
installing the chart into a k3s cluster -- the hardening had only ever
been rendered, never run.

  postgres  chown: /var/lib/postgresql/data/pgdata: Operation not permitted
  frontend  chown("/var/cache/nginx/client_temp", 101) failed
  web       chown("/var/lib/nginx/body", 65534) failed

All three deliberately start as root, fix ownership on their own runtime
directories, and then drop to their own user. capabilities.drop: [ALL]
takes CHOWN away along with everything else, so they never reach the
point of dropping privileges -- they die in the entrypoint, and the
install is left with nothing serving and a database that never comes up.

Grant each one exactly what its entrypoint needs (CHOWN, SETUID, SETGID,
plus FOWNER and DAC_OVERRIDE for re-owning an existing PGDATA) and keep
the blanket drop for everything else. The components that already run as
their own user from PID 1 -- task, redis, opa, the collector, init, the
assistant -- are unchanged and still drop everything.

Also: assistantOllama was listed twice in the same map, and the second
entry ({}) silently won, so the Ollama container had no hardening at all
while appearing to have it. Now listed once.

Verified on a two-node k3s cluster: postgres, web and frontend all reach
1/1 Running with the fix, from CrashLoopBackOff without it.
@krlex
krlex force-pushed the fix/hardening-broke-every-stateful-component branch from 450a2eb to 1a9debe Compare August 20, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant