feat(sandbox): isolate per-candidate agent state, and stop candidates racing to refresh one credential

[KE7]

This PR does two related things to the one shared login volume. Both come out of the same fact — every candidate container mounts helix-auth-<backend> at /home/node read-write, and that mount is deliberate and unchanged — and each is its own commit.

1. Stop candidates inheriting each other's state. The CLIs write their working state into that volume, so candidate N started life reading candidate N-1's transcripts and session databases.
2. Stop candidates competing to refresh one credential. When the shared credential goes stale, every candidate in a generation independently decides a refresh is due and posts the same single-use refresh token. One wins; the rest are told it was already used, and they fail silently.

---

1. Per-candidate agent state

The problem

Every candidate container mounts the shared helix-auth-<backend> login volume at /home/node read-write. That is deliberate and stays exactly as it is — it is what lets a CLI refresh its token and take its cross-process refresh lock.

The problem is what else rides along in that volume. The CLIs also write their working state there, so candidate N started life reading candidate N-1's transcripts, session databases, memories and to-do lists. For an evolutionary optimizer whose candidates are meant to be independent samples, that contaminates the experiment — the primary motivation here. Secondarily, some of that state (opencode's opencode.db) carries access_token / refresh_token columns, so it is a second credential store lying around in a shared location.

The change

Mount a second, per-candidate directory at /helix-state — deliberately outside /home/node, so the shared auth mount is untouched — and point each backend's state at it. It lives in the sandbox's existing temporary tree (helix.sandbox.run_sandboxed_commands), so it is created and removed with the candidate by the existing _safe_rmtree.

Backend	Knob	Moves	Credential
codex	-c sqlite_home=…	state_5.sqlite, logs_2.sqlite (+-wal/-shm)	.codex/auth.json stays shared
opencode	OPENCODE_DB=…	opencode.db (+-wal/-shm)	auth.json and the lock dir stay shared
cursor	CURSOR_CONFIG_DIR=…	the whole ~/.cursor tree	~/.config/cursor/auth.json stays shared
claude	none — see below	—	—

Picking the knob is the whole problem

The obvious environment variable is usually the wrong one, because it relocates the credential too and silently makes an existing login invisible. Verified against the real CLIs:

• XDG_DATA_HOME on opencode → opencode auth list reports 0 credentials
• XDG_CONFIG_HOME on cursor → cursor-agent status reports Not logged in
• CURSOR_DATA_DIR is accepted but relocates nothing

These are recorded in helix.agent_state.REJECTED_AGENT_STATE_KNOBS so they are not re-tried, and pinned by container tests. HELIX never sets either variable; if a user routes XDG_CONFIG_HOME through passthrough_env or [env], the cursor backend now logs a warning.

Drive-by fix

Unsandboxed opencode runs used XDG_DATA_HOME for per-candidate SQLite isolation, which hid any existing opencode login. Switched to OPENCODE_DB; the on-disk layout is unchanged (<worktree>/.helix_opencode_state/opencode/opencode.db).

claude is deliberately not isolated

CLAUDE_CONFIG_DIR is all-or-nothing — it moves .credentials.json together with the transcripts, and pulls .claude.json in as well. The masking alternative was evaluated against Claude Code 2.1.138 and rejected for four reasons, in docs/agent-state-isolation.md:

1. The mask list is already stale against the shipped CLI — no todos/, but telemetry/ and backups/ both carry per-session identifiers. Future drift is silent.
2. .claude.json sits at $HOME, outside .claude/, so it needs a file-level bind mount, and the CLI rewrites it via backup-and-replace.
3. Masking writes to the shared volume (5 new entries, including turning .claude.json into a 0-byte file). The knob-based approach leaves it byte-for-byte unchanged.
4. It silently breaks _copy_claude_transcript_from_auth_volume, which reads from the auth volume in a separate container that does not carry the masks — and whose [ -f "$src" ] || exit 0 guard means it fails silently.

Separately, requirement "the CLI still reports itself authenticated" cannot be met for claude without a real grant, since it validates the credential shape. Proving isolation by risking the credential it protects is a bad trade.

Residue

Relocation is partial, and what still crosses candidates is named per backend in helix.agent_state.UNRELOCATED_AGENT_STATE rather than left implicit. The significant case is codex's session rollout transcript (.codex/sessions/<date>/rollout-*.jsonl, ~28 KB/run) plus shell_snapshots/ and memories/: sqlite_home does not cover them and the CLI exposes no separate knob — only CODEX_HOME, which moves auth.json too.

Verification

Real containers, synthetic credentials in throwaway volumes, no login, no model calls, network none. Before/after listings are names/modes/sizes only.

• codex — 6 sqlite files land in /helix-state/codex/; no .sqlite in the shared volume; codex login status → Logged in using an API key
• cursor — cli-config.json lands in /helix-state/cursor/; shared volume byte-for-byte unchanged; status reports authenticated
• opencode — opencode.db + -wal/-shm land in /helix-state/opencode/; auth list reads ~/.local/share/opencode/auth.json → 1 credentials; the locks/ dir stays in the shared volume

The five real helix-auth-* volumes were confirmed intact by name count (5 → 5); every volume created during verification was removed.

New: tests/integration/test_agent_state_isolation.py (marker docker_integration) pins all of the above, including both rejected knobs, and asserts by source inspection that no test can name a real login volume.

---

2. Warm the shared credential once per generation, and name credential failures

The problem

The shared mount is what lets a CLI refresh its token — but not every CLI serialises that refresh. Measured against the shipped codex CLI (codex-cli 0.130.0) with a synthetic credential in a throwaway volume and a local single-use token endpoint, five simultaneous candidates against one stale credential produce:

5 token exchanges   1 granted   4 rejected as "your refresh token was already used"
...and all five processes exit 0 with empty stderr, even at RUST_LOG=info.

A whole generation can die without saying anything. Today the only thing keeping that window narrow is codex's own last_refresh field acting as an accidental serialiser.

The change — warm once per generation

Run the refresh once, before the generation dispatches anything, through the sandboxed auth container that already exists and is already a single writer (helix.sandbox.run_sandbox_auth_command). Same measurement with the warm in front:

1 token exchange    1 granted   0 rejected
...and the five candidates then perform zero exchanges: nothing left to refresh.

Per generation, not per run. A run outlives any refresh interval, so a single warm at startup stops protecting it the moment the credential next goes stale mid-flight. It sits above the merge/mutate split so it covers every path that dispatches a candidate.

It must be free. This runs on the operator's paid account every generation, so a status/whoami command that quietly cost a request would be worse than the race. That check is what decides whether a backend is warmed at all.

Which backends are warmed, and why the others are not

Backend	Warmed	Why
codex	yes	codex debug models — see below
claude	no	Takes a real cross-process lock file, retries while another process holds it, and re-reads afterwards. Candidates cannot consume the same refresh token.
cursor	no	Never spends its stored refresh token; it re-exchanges an API key. There is no single-use grant to race for.
opencode	no	Refreshes an oauth-type credential only from inside the fetch wrapper that issues a model request (read from opencode-ai 1.14.24). No command performs that refresh without also invoking a model. opencode providers list was measured free — it completes with --network none against an expired credential and leaves auth.json byte-identical — and for exactly that reason refreshes nothing. api-type credentials never refresh.
gemini	no	The registered status command is gemini --version, which touches no credential; no free command is known to take the refresh path, and there is no credential to measure one against. Not warmed on a guess.

Every reason is recorded in helix.backends.CREDENTIAL_WARM_SKIP_REASONS, and a test asserts each backend is either warmed or explained — a new backend cannot fall through silently.

The warm command is not codex login status

codex login status never takes the refresh path. Measured with a synthetic credential in a throwaway volume, it prints Logged in using ChatGPT and exits 0 without issuing a single request, whether the stored last_refresh is minutes or 30 days old. Warming with it would look like protection while providing none.

codex debug models renders the CLI's built-in model catalog and loads auth through the refreshing path. It is free:

• with a fresh credential it completes under --network none, writes nothing to the login volume, and makes no request at all — three consecutive warms produced 0 token exchanges and a byte-identical auth.json;
• with a stale credential its only request is the OAuth token exchange, and the refreshed credential is written back from it.

No model is invoked and no quota is consumed either way. stdout is discarded (~200 KB of catalog) since the command is run for its side effect; stderr is kept so a failure stays diagnosable.

A failed warm is logged and the run continues — candidates may still succeed on the credential already stored — but the log says what protection was lost, not just that a command exited non-zero.

The second half — make the failure visible

There was no auth-failure detection at all, and the structured result envelope's is_error field was never read anywhere in src/. A credential-exhaustion failure was indistinguishable from "the agent wrote bad code": the mutation was abandoned, the slot dropped, and nothing said why.

CredentialRefreshError is now its own failure kind — a sibling of MutationError (a code failure) and RateLimitError (a quota failure that clears on its own). Detection is anchored on wording read out of the shipped binaries, not guessed:

Marker	Source
your access token could not be refreshed	codex-cli 0.130.0 — one prefix covering all five suffixes it appends, including "…because your refresh token was already used."
failed to refresh token while getting account	codex-cli 0.130.0
chatgpt account id not available, please re-run `codex login`	codex-cli 0.130.0
token refresh failed:	opencode-ai 1.14.24 — thrown by the provider fetch wrapper
user oauth refresh failed	Claude Code 2.1.138
api error: 401 invalid api key	Claude Code 2.1.138

Every marker is a whole distinctive clause. Matching a bare number or a common word (401, token, auth) is the false-positive trap this repo has already paid for once.

is_error narrows, it never classifies. A tool_result carrying is_error is usually just the agent's own failing shell command — an ordinary code failure. So on a zero exit only is_error-flagged envelope text is scanned; scanning the raw streams there would let a candidate that merely edited an OAuth code path talk HELIX into declaring the operator's login broken. On a non-zero exit the raw streams are read too: the invocation already failed, and the only question left is which kind of failure it was.

The verdict surfaces where an operator sees it — the proposal slot says the login failed rather than the code, and the permanent end-of-run summary repeats it after the live display is gone, naming the affected candidates and the remedy.

No config knob: a credential that cannot be refreshed is never the behaviour anyone wants, and a warm that costs nothing needs no opt-out.

Verification

Real containers, synthetic credentials in throwaway volumes, a local fake token endpoint, no login and no model calls. New tests:

• tests/unit/test_credential_warm.py — registry completeness, single-writer args, failure is never fatal
• tests/unit/test_credential_warm_loop.py — one warm per generation, warm precedes every mutation, the run survives a credential failure and says so
• tests/unit/test_credential_failure_classification.py — every real CLI string classified; ordinary candidate output (401, token, a failing test_oauth.py, a diff adding refresh_token) classified as not a credential failure
• tests/integration/test_credential_warm_docker.py (docker_integration) — the warm is free (clean exit under --network none), is a no-op on a fresh credential (auth.json digest unchanged, no files added), and leaves no residue when repeated

The five real helix-auth-* volumes were confirmed intact by name; every volume, network and container created during verification was removed.

---

Gates

uv run python -m pytest                                    1094 passed  (baseline 1022)
HELIX_DOCKER_TESTS_STRICT=1 ... -m docker_integration      10 passed    (baseline 6)
uv run ruff check src/ tests/                              All checks passed
uv run mypy --strict src/helix/                            no issues, 28 files
git diff --shortstat origin/main...HEAD                    19 files changed, 2661 insertions(+), 44 deletions(-)

Review fixes

Five focused commits on top of 3af85bd, one per review finding:

1. CredentialRefreshError escaping merge() (18a19fb) — merger.merge() now labels the operation, removes the merge worktree and re-raises, exactly as mutate() does. The merge call site in run_evolution catches it, records the slot in CredentialFailureLog, prints the panel and falls through to mutation. cli.evolve / cli.resume gain a last-resort handler (panel + resume hint, exit 2) so it can never surface as a raw traceback. Tests: test_merger.py, test_credential_warm_loop.py (end-to-end via the merge gate), new test_cli_credential_failure.py.
2. Unbounded credential warm (c69ff7c) — run_sandbox_auth_command accepts timeout and container_name, emits --name, and on TimeoutExpired force-removes the named container before re-raising (killing the docker client does not stop the container). warm_backend_credential always runs under credential_warm_timeout() — a 300 s cap that sandbox.timeout_seconds can only tighten — and reports a hang as a distinct, non-fatal CredentialWarmResult(timed_out=True); the loop-level warning says "timed out" instead of "(exit None)". Tests mock subprocess.run raising TimeoutExpired.
3. "refresh token was already used" treated as a dead login (0f66bc7) — that suffix is now a separate transient marker matched ahead of the generic prefix, and CredentialRefreshError.transient carries it. invoke_claude_code retries exactly once against the credential the race winner just stored (both the exit-1 stderr path and the exit-0 is_error envelope path). A second loss is raised with a suggestion that points at helix resume and does not tell the operator to re-login; the per-slot message and end-of-run summary use the same distinction (re-login advice is kept for expired/revoked/bare wording). Tests cover the marker split, retry-then-success, retry-then-failure, and that non-transient wording is never retried.
4. Lower-severity items
   • (a) ac9da92 — the per-candidate agent-state tree (where opencode.db with its OAuth tokens lives while a candidate runs) is created mode 0700 in its own right, on top of the mkdtemp (0700) parent it already sat in and the finally-time removal it already had. Documented in docs/agent-state-isolation.md; mode and removal pinned in test_agent_state.py.
   • (b) CURSOR_CONFIG_DIR seeding from login-time config: not changed. The condition in the finding ("if that is what the credential-copy path already does for other backends") does not hold — this PR deliberately copies no credential or config for any backend; everything stays in the shared volume. Seeding would need an extra docker run against the auth volume per candidate and verification against the real cursor-agent, so it is left for a follow-up.
   • (c) 18fd2a0 — addopts = "-m 'not docker_integration'" in pyproject.toml, so a bare pytest no longer collects tests/integration/; pytest -m docker_integration tests/integration/ still opts in (verified: 10 collected). README says how.

Gates (after fixes)

uv run python -m pytest -q                       1112 passed, 10 deselected in 12.73s
uv run ruff check src/ tests/                    All checks passed!
uv run mypy --strict src/helix/                  Success: no issues found in 28 source files

Branch already contains origin/main (0 commits behind). Conflicts with PR #71 (feat/agy-backend-and-sandbox-fixes) in README.md, src/helix/mutator.py, tests/unit/test_mutator.py — to be sequenced by the owner.