Incident (2026-07-12, superaseo #77 run 29192609643)
Local-lane engine: codex runs die at implement spawn with an opaque AgentError: codex exited with code 1 and no reason file — same signature as the quota runbook, but quota was green.
Root cause
The local lane bind-mounts host ~/.codex-afk → /home/agent/.codex (container UID 1000). codex 0.144.x rewrites its home with hardened perms (700 dir / 600 auth.json, host UID 501). Under VirtioFS the container user can no longer read the mount → codex login status = "Not logged in" → exit 1 at spawn. Any host-side codex upgrade/login/token refresh can re-trigger this fleet-wide (all 5 local images).
Repro: docker run --rm --entrypoint sh -v ~/.codex-afk:/home/agent/.codex sandcastle-<repo> -c 'codex login status' → Permission denied / Not logged in.
Mitigation applied (fragile)
chmod 755 ~/.codex-afk && chmod 644 auth.json config.toml → in-container chat smoke passes. This regresses on codex's next perm-hardening write.
Proposed durable fix
Replace the bind-mount with the cloud lane's copy-in seed pattern: at phase start, copy auth.json/config.toml into a container-local /home/agent/.codex owned by UID 1000 (flock the host copy during read; write refreshed auth back to host under the existing host lock, or accept in-container refresh loss like the cloud lane does). Bonus: also removes the "two concurrent local runs share one writable codex home" hazard.
Secondary: the image installs @openai/codex unpinned, so image codex version drifts from host (0.143.0 vs 0.144.1 today) — consider stamping the version at build and surfacing it in the run header for diagnosability.
Found while dispatching superaseo #77/#78; superaseo image rebuilt to 0.144.1 same day (rebuild alone did NOT fix it — perms did).
🤖 Generated with Claude Code
Incident (2026-07-12, superaseo #77 run 29192609643)
Local-lane
engine: codexruns die at implement spawn with an opaqueAgentError: codex exited with code 1and no reason file — same signature as the quota runbook, but quota was green.Root cause
The local lane bind-mounts host
~/.codex-afk→/home/agent/.codex(container UID 1000). codex 0.144.x rewrites its home with hardened perms (700dir /600auth.json, host UID 501). Under VirtioFS the container user can no longer read the mount →codex login status= "Not logged in" → exit 1 at spawn. Any host-side codex upgrade/login/token refresh can re-trigger this fleet-wide (all 5 local images).Repro:
docker run --rm --entrypoint sh -v ~/.codex-afk:/home/agent/.codex sandcastle-<repo> -c 'codex login status'→ Permission denied / Not logged in.Mitigation applied (fragile)
chmod 755 ~/.codex-afk && chmod 644 auth.json config.toml→ in-container chat smoke passes. This regresses on codex's next perm-hardening write.Proposed durable fix
Replace the bind-mount with the cloud lane's copy-in seed pattern: at phase start, copy
auth.json/config.tomlinto a container-local/home/agent/.codexowned by UID 1000 (flock the host copy during read; write refreshed auth back to host under the existing host lock, or accept in-container refresh loss like the cloud lane does). Bonus: also removes the "two concurrent local runs share one writable codex home" hazard.Secondary: the image installs
@openai/codexunpinned, so image codex version drifts from host (0.143.0 vs 0.144.1 today) — consider stamping the version at build and surfacing it in the run header for diagnosability.Found while dispatching superaseo #77/#78; superaseo image rebuilt to 0.144.1 same day (rebuild alone did NOT fix it — perms did).
🤖 Generated with Claude Code